Artificial Intelligence

ARTIFICIAL INTELLIGENCE

Also in this issue: > Risks > Moving to Autonomous and Self-Migrating Containers for Cloud Applications

APRIL 2017 www.computer.org Focus on Your Job Search

IEEE Computer Society Jobs helps you easily find a new job in IT, software development, computer engineering, research, programming, architecture, cloud computing, consulting, databases, and many other computer-related areas. New feature: Find jobs recommending or requiring the IEEE CS CSDA or CSDP certifications!

Visit www.computer.org/jobs to search technical job openings, plus internships, from employers worldwide.

http://www.computer.org/jobs

The IEEE Computer Society is a partner in the AIP Career Network, a collection of online job sites for scientists, engineers, and computing professionals. Other partners include Physics Today, the American Association of Physicists in Medicine (AAPM), American Association of Physics Teachers (AAPT), American Physical Society (APS), AVS Science and Technology, and the Society of Physics Students (SPS) and Sigma Pi Sigma. IEEE COMPUTER SOCIETY http://computer.org • +1 714 821 8380

STAFF

Editor Acting Editorial Content Manager Lee Garber Carrie Clark

Contributing Staff Senior Manager, Editorial Services Christine Anthony, Brian Brannon, Lori Cameron, Cathy Martin, Robin Baldwin Chris Nelson, Meghan O’Dell, Dennis Taylor, Rebecca Torres, Bonnie Wylie Director, Products and Services Evan Butterfield Production & Design Carmen Flores-Garvey, Monette Velasco, Jennie Zhu-Mai, Senior Advertising Coordinator Mark Bartosik Debbie Sims

Circulation: ComputingEdge (ISSN 2469-7087) is published monthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720; voice +1 714 821 8380; fax +1 714 821 4010; IEEE Computer Society Headquarters, 2001 L Street NW, Suite 700, Washington, DC 20036. Postmaster: Send address changes to ComputingEdge-IEEE Membership Processing Dept., 445 Hoes Lane, Piscataway, NJ 08855. Periodicals Postage Paid at New York, New York, and at additional mailing offices. Printed in USA. Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in ComputingEdge does not necessarily constitute endorsement by the IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and space. Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own Web servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first scree n of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to: http://www.ieee .org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or pubs-permissions@ieee .org. Copyright © 2017 IEEE. All rights reserved. Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per- copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923. Unsubscribe: If you no longer wish to receive this ComputingEdge mailing, please email IEEE Computer Society Customer Service at [email protected] and type “unsubscribe ComputingEdge” in your subject line. IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Computer Society Magazine Editors in Chief

Computer IEEE Micro IEEE Intelligent Systems Sumi Helal, University of Florida Lieven Eeckhout, Ghent V.S. Subrahmanian, University University of Maryland IEEE Software IEEE MultiMedia Diomidis Spinellis, Athens IEEE Computer Graphics Yong Rui, Lenovo Research University of Economics and and Applications and Technology Business L. Miguel Encarnação, ACT, Inc. IEEE Annals of the History IEEE Internet Computing IEEE Pervasive Computing of Computing M. Brian Blake, University of Maria Ebling, IBM T.J. Watson Nathan Ensmenger, Indiana Miami Research Center University Bloomington

IT Professional Computing in Science IEEE Cloud Computing San Murugesan, BRITE & Engineering Mazin Yousif, T-Systems Professional Services Jim X. Chen, George Mason International University IEEE Security & Privacy Ahmad-Reza Sadeghi, Technical University of Darmstadt www.computer.org/computingedge 1 APRIL 2017 • VOLUME 3, NUMBER 4

THEME HERE 14 26 36 Artifi cial There Is Masterminds Intelligence as No AI of Artifi cial an Eff ective Without IA Intelligence: Classroom Marvin Minsky and Assistant Seymour Papert 7 Editor’s Note: A Close Look at Artificial Intelligence 8 Machine Learning PANOS LOURIDAS AND CHRISTOF EBERT

14 Artificial Intelligence as an Effective Classroom Assistant BENEDICT DU BOULAY

20 TJBot and Zooids: The Connection between Pervasive Computing and AI MARIA R. EBLING

23 Engineering the New Boundaries of AI AMIR BANIFATEMI AND JEAN-LUC GAUDIOT

26 There Is No AI Without IA SETH EARLEY

33 Unreliable Machine Learning VINTON G. CERF

36 Masterminds of Artificial Intelligence: Marvin Minsky and Seymour Papert GEORGE STRAWN AND CANDACE STRAWN

39 Risks MATTHEW TURK

42 Moving to Autonomous and Self-Migrating Containers for Cloud Applications DAVID S. LINTHICUM

Departments 39 4 Magazine Roundup 46 Computing Careers: Risks Careers in Artificial Intelligence 48 Career Opportunities

Subscribe to ComputingEdge for free at www.computer.org/computingedge. CS FOCUS

Magazine Roundup

Current approaches to teaching software engineering are out- dated and don’t address today’s conditions. In “How Best to Teach Global Software Engineering? Educators Are Divided,” from IEEE Software’s January/February 2017 issue, pioneering educators discuss how they inject realism he IEEE Computer ensure their effi ciency, security, into global-software engineer- Society’s lineup of 13 and dependability is an impor- ing education. T peer-reviewed technical tant research area. Meanwhile, as magazines covers cutting-edge essential, life-sustaining systems IEEE Internet Computing topics ranging from software increasingly connect in cyber- design and computer graphics space, we must fi gure out how Fog computing could alleviate to Internet computing and secu- to identify, assess, anticipate, many of the Internet of Things’ rity, from scientifi c applications and manage risk. Computer’s challenges by, for example, and machine intelligence to April 2017 issue has special sec- reducing latency and enhancing cloud migration and microchip tions on these topics. security. IEEE Internet Comput- design. Here are highlights from ing’s March/April 2017 special recent issues. IEEE Software issue explores the technology’s potential to both form distrib- Computer With global software engineer- uted and virtualized platforms ing becoming standard practice, that support computation- Our society increasingly relies on today’s software engineering intensive tasks and distribute the successful marriage of digital students will become tomor- advanced computing, storage, and physical systems to perform row’s global software engineers. networking, and management advanced automation and control The education systems under- services to the network edge. tasks. Thus, engineering these pinning the profession will thus The issue also looks at the cyber-physical systems to need to change accordingly. obstacles fog computing faces.

4 April 2017 Published by the IEEE Computer Society 2469-7087/17/$33.00 © 2017 IEEE Computing in Science & IEEE Intelligent Systems before the pervasive-health fi eld Engineering can advance: technological chal- “An Open Semantic Framework for lenges and opportunities, adoption CiSE’s March/April 2017 special the Industrial Internet of Things,” and adherence, open data, ethical issue provides a mix of perspec- from IEEE Intelligent Systems’ issues, and education. tives for understanding more deeply January/February 2017 issue, intro- where Moore’s law stands today duces the Open Semantic Frame- IT Professional and the prospects for continued work (OSF), which lets users create, exponential performance growth in deploy, and manage semantic appli- To achieve digital transformation, the upcoming computing era. cations. The OSF represents an organizations must control the important step toward the ability to avalanche of data generated by IEEE Security & Privacy integrate various knowledge models. supplier, customer, and employee interactions. This complex data IEEE S&P’s January/February 2017 IEEE MultiMedia environment has led enterprises to special issue looks at how the lat- realize that new leadership—in the est security and privacy research The authors of “The Continu- form of a chief data offi cer (CDO)— is moving the fi eld forward. Articles ing Reinvention of Content-Based is needed to oversee this task and in the issue address topics such as Retrieval: Multimedia Is Not Dead,” to take responsibility for extracting the use of containerization to secure from IEEE MultiMedia’s January– value from data. “The Evolving Role mobile devices, cloud data-auditing March 2017 special issue, review of the CDO,” from IT Pro’s January/ techniques, and a novel security- how content-based retrieval has February 2017 issue, examines this development lifecycle model. evolved over the years and discuss relatively new position. the technology’s future goals and IEEE Cloud Computing research directions. IEEE Micro

IEEE Cloud Computing’s January/ IEEE Annals of the History of IEEE Micro’s January/February February 2017 issue includes arti- Computing 2017 special issue on cognitive cles on subjects such as combat- architectures includes articles on ing distributed denial-of-service As part of a broader look at early spacetime computation and the neo- attacks in the cloud, secure data intrusion-detection systems, “The cortex, low-power automatic speech sharing and searching at the edge March of IDES: Early History of recognition via a mobile GPU and of the cloud-assisted Internet of Intrusion-Detection Expert Sys- Viterbi accelerator, effi cient situ- Things, and trusting cloud service tems,” from IEEE Annals’ October– ational scheduling of graph work- providers. December 2016 issue, focuses loads on single-chip multicores and on the fi rst such application, the GPUs, and graph-analytics acceler- IEEE Computer Graphics and Intrusion Detection Expert System ators for cognitive systems. Applications (IDES), which SRI International developed in the 1980s. Computing Now Articles from CG&A’s March/April 2017 special issue on information IEEE Pervasive Computing The Computing Now website visualization examine topics such (computingnow.computer.org) as layered graph drawing for visu- The authors of “The Future of features up-to-the-minute com- alizing evaluation structures, the Pervasive Health,” from IEEE Per- puting news and blogs, along visualization of rank time series of vasive Computing’s January–March with articles ranging from peer- top-viewed Wikipedia pages, and 2017 issue, discuss fi ve important reviewed research to opinion spatial analytic interfaces. matters that must be addressed pieces by industry leaders. www.computer.org/computingedge 5 Move Your Career Forward IEEE Computer Society Membership

Explore These Artificial Intelligence Resources

Rock Stars of Deep Learning/Machine Learning 12 September 2017 Mountain View, CA, USA Join us in Silicon Valley with Fortune 100 C-suite and upper management executives and others to learn about deep learning and machine learning opportunities for 2017. Hear insights on how they use analytics, algorithms, and data structures, and learn how to trust semisupervised or unsupervised analysis and how to contract for or build your deep learning talent pool.

Rock Stars of Smart Robo Technologies 17 October 2017 Dallas, TX, USA This conference will explore how companies are using smart robo systems for financial services, insurance, healthcare, manufacturing, publishing, and marketing. Join us in the heart of Texas, as executive speakers will discuss their current implementations and expectations for future applications. They’ll address key issues, such as integration strategies and challenges; ROI; impact on the workforce; accuracy; and security.

IEEE Transactions on Learning Technologies The quarterly journal TLT covers research on such topics as innovative online learning systems, intelligent tutors, educational software applications and games, and simulation systems for education and training.

FOR DIRECT LINKS TO THESE RESOURCES, VISIT The Community for Technology Leaders www.computer.org/edge-resources EDITOR’S NOTE

A Close Look at Artifi cial Intelligence

ong the subject of science fi ction, artifi - interdisciplinary teams advance AI technologies cial intelligence (AI) began as a formal while competing for the grand prize. L research discipline at a 1956 Dartmouth The author of “There Is No AI Without IA,” College workshop. Since then, optimism about from IT Professional, argues that artifi cial intelli- artifi cial intelligence has risen and fallen, based in gence needs high-quality, structured data—in part part on the limitations of AI itself and the technolo- via eff ective information architectures (IAs)—to be gies it uses. Now, AI is being employed extensively useful to organizations and their customers. in business applications and consumer products, In IEEE Internet Computing’s “Unreliable giving the technology an economic foundation. Machine Learning,” Internet pioneer Vinton Cerf This ComputingEdge issue examines today’s speculates about whether neural networks might important AI-related developments. experience problems as users ask them to work IEEE Software’s “Machine Learning” presents with more complex and diverse parameters and a brief overview of machine-learning technologies, with larger decisions and responsibilities at stake. with a case study involving code analysis. IT Professional’s “Masterminds of Artifi cial “Artifi cial Intelligence as an Eff ective Class- Intelligence: Marvin Minsky and Seymour Papert” room Assistant,” from IEEE Intelligent Systems, highlights the contributions of two AI pioneers. reviews metareviews and meta-analyses to make ComputingEdge articles on topics other than AI the case for blended learning, in which a teacher include the following: offl oads some work to systems that use AIand cognitive science to help with the process. • Computing in Science & Engineering’s “Risks” The author of “TJBot and Zooids: The Connec- focuses on scientifi c computing’s potential tion between Pervasive Computing and AI,” from hazards. IEEE Pervasive Computing, looks at a robot and a • The author of “Moving to Autonomous and robot kit to show how AI and pervasive computing Self-Migrating Containers for Cloud Applica- could reshape the way that humans interact with tions,” from IEEE Cloud Computing, says con- computers. tainers should be able to migrate from one type Computer’s “Engineering the New Boundar- of public cloud to another in automated and ies of AI” discusses the XPRIZE Foundation’s autonomous ways. He then describes the man- launching of the IBM Watson AI XPRIZE, in which ner in which this could be done.

Machine Learning

Panos Louridas and Christof Ebert

Machine learning is the major success factor in the ongoing digital transformation across industries. Startups and behemoths alike announce new products that will learn to perform tasks that previously only humans could do, and perform those tasks better, faster, and more intelligently. But how do they do it? What does it mean for IT developers and software engineers? Here, Panos Louridas and I present a brief overview of machine-learning technologies, with a concrete case study from code analysis. I look forward to hearing from both readers and prospective column authors. —Christof Ebert

MACHINE LEARNING ISN’T new; it multisensor fusion for autonomous has been around at least since the 1970s, driving; and when the rst related algorithms ap- • pattern recognition to analyze code peared. What has changed is that the ex- for weaknesses such as criticality plosion in computing power has allowed and code smells (for a related case us to use machine learning to tackle ever- study, see the sidebar). more-complex problems, while the explo- sion of data being captured and stored The general idea behind most ma- has allowed us to apply machine learning chine learning is that a computer learns to an ever-expanding range of domains. to perform a task by studying a training Machine learning is used in different set of examples. The computer (or sys- domains. Here are a few examples: tem of distributed or embedded computers and controllers) then performs the • security heuristics that distill attack same task with data it hasn’t encoun- patterns to protect, for instance, tered before. ports or networks; • image analysis to identify distinct Learning Strategies forms and shapes, such as for medi- Machine learning employs the following cal analyses or face and ngerprint two strategies (see Figure 1). recognition; • deep learning to generate rules for Supervised Learning data analytics and big data handing, In supervised learning, the training set such as are used in marketing and contains data and the correct output of sales promotions; the task with that data. This is like giving • object recognition and predictions a student a set of problems and their solu- from combined video streams and tions and telling that student to gure out

8 110 IEEE SOFTWAREApril 2017 | PUBLISHED BY THE IEEE COMPUTERPublished by the SOCIETY IEEE Computer Society 0740-7459/16/$33.002469-7087/17/$33.00 © 2016 © IEEE 2017 IEEE Editor: Christof Ebert SOFTWARE Vector Consulting Services SOFTWARE TECHNOLOGY TECHNOLOGY [email protected]

Machine Learning CASE STUDY: MACHINE LEARNING FOR CODE ANALYSIS Machine learning has many practical applications. As be ts Step 3: Provide a change history classi cation (that is, the this magazine, we’ll present an example that shows how number of compiles or deliveries) for each learning module. Panos Louridas and Christof Ebert machine learning can manage quality risks and improve Step 4: With static code analysis, assemble for each quality assurance productivity. learning module a complexity classi cation such as hot spots from code analysis. Machine learning is the major success factor in the ongoing digital A CRITICALITY ASSESSMENT TOOL Step 5: With the machine-learning system, construct transformation across industries. Startups and behemoths alike Project managers and product owners often wonder when an initial criticality list that takes into account the inputs would be the right release point and how to assess the from steps 2, 3, and 4, mapped to the list from step 1. announce new products that will learn to perform tasks that previously criticality of the code to be delivered. Static-analysis tools Evaluate the criticality list’s validity—for example, by only humans could do, and perform those tasks better, faster, and exist but give numerous warnings that might be dif cult to screening for the identi ed critical modules, outliers, and more intelligently. But how do they do it? What does it mean for IT link to actual weak spots. At Vector Consulting Services, potential misleading effects. Such screening aims to nd we offer clients a machine-learning-based criticality as- undesired in uences from the defect or change histo- developers and software engineers? Here, Panos Louridas and I sessment tool for software release management. Crite- ries. The screening and ranking must primarily ensure the present a brief overview of machine-learning technologies, with a ria include hard factors from static code analysis, such as fewest possible type-I prediction errors. (In type-I errors, cyclomatic complexity, the degree of reuse, and the history defect-prone components are misclassi ed as uncritical concrete case study from code analysis. I look forward to hearing from of defects in preceding versions and variants. We also use components.) both readers and prospective column authors. —Christof Ebert soft factors such as designer competences, experience with Step 6: For the current project, repeat steps 1 to 5 to get similar projects, and architectural decisions that might incur a predictive result on each new module’s criticality. Then, technical debt. present the rankings so that the developers can decide on Using these criteria, the machine-learning tool builds further actions. MACHINE LEARNING ISN’T new; it multisensor fusion for autonomous a criticality prediction model. On the basis of a ranked list Step 7: Manually prepare suggestions based on the new has been around at least since the 1970s, driving; and of the criticality of the modules used in a build, develop- modules ranked the most critical. Critical modules should when the rst related algorithms ap- • pattern recognition to analyze code ers can apply different mechanisms to improve quality— at least undergo a ash review and subsequent refactoring, peared. What has changed is that the ex- for weaknesses such as criticality refactoring, redesign, thorough static analysis, and unit redesign, or rewriting—depending on their complexity, age, plosion in computing power has allowed and code smells (for a related case testing with increased coverage schemes. and reuse in other projects. Refactoring includes reducing us to use machine learning to tackle ever- study, see the sidebar). Instead of predicting the number of defects or changes size, improving modularity, balancing cohesion and cou- more-complex problems, while the explo- (algorithmic relationships), the tool considers assignments pling, and so on. For instance, apply thorough unit testing sion of data being captured and stored The general idea behind most ma- to classes (for example, “defect prone”). The training and with 100 percent C0 coverage (statement coverage) to the has allowed us to apply machine learning chine learning is that a computer learns test data come from nished projects that had been under modules. Investigate the details of those modules’ complex- to an ever-expanding range of domains. to perform a task by studying a training con guration control since coding started. After the ma- ity measurements to determine the redesign approach. Typ- Machine learning is used in different set of examples. The computer (or sys- chine-learning process, the data used operationally comes ically, the different complexity measurements will indicate domains. Here are a few examples: tem of distributed or embedded comput- from active projects. the approach to follow. ers and controllers) then performs the To achieve feedback to improve predictions, this ap- Step 8: After the new project is nished, validate and • security heuristics that distill attack same task with data it hasn’t encoun- proach is integrated throughout development (requirements, improve the prediction model on the basis of postmortem patterns to protect, for instance, tered before. design, code, system test, and deployment). studies with all collected defects and the population of a ports or networks; “real” criticality list. Then, compare the actual defect rank- • image analysis to identify distinct Learning Strategies STEP-BY-STEP CRITICALITY PREDICTION ing with the predicted ranking. Investigate the reasons for forms and shapes, such as for medi- Machine learning employs the following Figure A illustrates criticality classi cation and validation, deviations, and tune the implemented automatic classi ca- cal analyses or face and ngerprint two strategies (see Figure 1). which consists of eight steps. tion approaches. Improve the screening rules to ensure that recognition; Step 1: For the nished projects, provide a list of all the type-II prediction errors will be reduced the next time. (In • deep learning to generate rules for Supervised Learning modules used for learning from the con guration system. type-II errors, uncritical components are misclassi ed as data analytics and big data handing, In supervised learning, the training set Step 2: Provide a defect list for each learning mod- defect-prone components.) such as are used in marketing and contains data and the correct output of ule. For high-ranking defects, you might add a root-cause sales promotions; the task with that data. This is like giving analy sis that allows for a Pareto-based mitigation list. continued on next page • object recognition and predictions a student a set of problems and their solu- from combined video streams and tions and telling that student to gure out

continued from previous page

Con guration 4 Static code data analysis, technical 5 debt, and so on Machine learning: statistical 1 analysis and construction of a Soft factors Finished criticality prediction model projects 2 Defect data and heuristics 3 Change history Feedback for learning 8 Static code Criticality analysis, prediction and technical heat map 7 debt, and so on Active 6 Veri cation, (running) Change history, actual versus planned effort, Proposals for refactoring, and projects degree of reuse, and defect analysis improvement operational use

FIGURE A. Machine learning for criticality prediction of source code as implemented in this case study. The numbers refer to the steps described in the sidebar text.

THIS APPROACH’S EFFECTIVENESS business case: Criticality prediction doesn’t aim to detect all defects. Instead, it aims to optimize resource allocation by focusing • On average, for each project, 20 percent of the mod- resources on areas with critical defects that would affect ules were selected as the most critical (after coding). the delivered product’s utility. We estimate the trade-off of •Those modules contained over 40 percent of all the costs of applying complexity-based predictive quality defects (up to release time). models and the eventual code changes versus the improved quality, on the following basis: In addition, we’ve determined that 60 percent of all defects theoretically can be detected until the end of unit •Effort—limited resources are assigned to the high-risk testing. Also, defect correction with unit testing and static components. analysis costs 10 to 50 percent less than defect correc- •Effort—gray-box testing strategies are applied to only tion in subsequent testing activities. So, we calculate that the high-risk components. developers can detect 24 percent of all defects early by • Benefits—the risk assessment of changes is eased investigating 20 percent of all modules more intensively, with code analysis on the basis of the affected or with over 10 percent reduced effort than with late defect changed complexity. correction. This yields at least a 20 percent cost reduction • Benefits—fewer customers reported failures than for defect correction. in previous releases, and security and maintain- The necessary tools, such as Coverity, Klocwork, Lat- ability improved. tix, Structure 101, SonarX, and SourceMeter, are off the shelf and account for even less per project. These criticality On the basis of the results from many of our client analyses provide numerous other bene ts, such as the re- projects (and taking a conservative ratio of only 40 per- moval of speci c code-related risks and defects that other- cent defects in critical components), we can calculate the wise are hard to identify (for example, security aws).

10112 IEEE SOFTWAREComputing | Edge WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE April 2017 SOFTWARE TECHNOLOGY SOFTWARE TECHNOLOGY

how to solve other problems he or she continued from previous page will have to deal with in the future. Machine learning Supervised learning includes classification algorithms, which take as Supervised learning Unsupervised learning Con guration 4 Static code input a dataset and the class of each data analysis, piece of data so that the computer technical Classi cation Regression Clustering Dimension 5 can learn how to classify new data. debt, and so on reduction Machine learning: statistical For example, the input might be a set 1 analysis and construction of a Soft factors k-means Principal component Finished criticality prediction model of past loan applications with an in- Logic regression Linear regression projects 2 Defect data and heuristics clustering analysis dication of which of them went bad. 3 Change history Hierarchical Tensor Feedback for learning On the basis of this information, the Classi cation trees Decision trees clustering decomposition 8 computer classifies new loan appli- Static code Criticality cations. Classification can employ Support vector Bayesian Gaussian mixture Multidimensional analysis, prediction and machines networks models statistics technical logic regression, classification trees, heat map 7 debt, and so on support vector machines, random Random Active 6 Random forests Fuzzy classi cation Genetic algorithms Veri cation, forests, artificial neural networks projection (running) Change history, actual versus planned effort, Proposals for refactoring, and projects degree of reuse, and defect analysis improvement (ANNs), or other algorithms. ANNs Arti cial neural Arti cial neural Arti cial neural Arti cial neural operational use are a major topic on their own; we networks networks networks networks discuss them in more detail later. Regression algorithms predict a FIGURE A. Machine learning for criticality prediction of source code as implemented in this case study. The numbers value of an entity’s attribute (“re- FIGURE 1. Machine-learning approaches. In machine learning, a computer first learns refer to the steps described in the sidebar text. gression” here has a wider sense to perform a task by studying a training set of examples. The computer then performs than merely statistical regression). the same task with data it hasn’t encountered before. THIS APPROACH’S EFFECTIVENESS business case: Regression algorithms include linear Criticality prediction doesn’t aim to detect all defects. regression, decision trees, Bayesian Instead, it aims to optimize resource allocation by focusing • On average, for each project, 20 percent of the mod- networks, fuzzy classification, and best way for a task through artificial camps: one that prefers R and one resources on areas with critical defects that would affect ules were selected as the most critical (after coding). ANNs. selection), and ANNs. that prefers Python. Of course, any the delivered product’s utility. We estimate the trade-off of •Those modules contained over 40 percent of all Dimensionality reduction algo- absolute division makes no sense. the costs of applying complexity-based predictive quality defects (up to release time). Unsupervised Learning rithms take the initial dataset cover- For a field as wide as machine learn- models and the eventual code changes versus the improved In unsupervised learning, the training various dimensions and project ing, no single tool will do. The best quality, on the following basis: In addition, we’ve determined that 60 percent of all ing set contains data but no solutions; the data to fewer dimensions. These a software engineer can do is to be- defects theoretically can be detected until the end of unit the computer must find the solutions fewer dimensions try to better cap- come acquainted with many different •Effort—limited resources are assigned to the high-risk testing. Also, defect correction with unit testing and static on its own. This is like giving a stu- ture the data’s fundamental aspects. tools and learn which one is the most components. analysis costs 10 to 50 percent less than defect correc- dent a set of patterns and asking him Dimensionality reduction algorithms appropriate for a given situation. •Effort—gray-box testing strategies are applied to only tion in subsequent testing activities. So, we calculate that or her to figure out the underlying include principal component analy- That said, R is more popular with the high-risk components. developers can detect 24 percent of all defects early by motifs that generated the patterns. sis, tensor reduction, multidimen- people with a somewhat stronger • Benefits—the risk assessment of changes is eased investigating 20 percent of all modules more intensively, Unsupervised learning includes sional statistics, random projection, statistical background. It has a su- with code analysis on the basis of the affected or with over 10 percent reduced effort than with late defect clustering algorithms, which take as and ANNs. perb collection of machine-learning changed complexity. correction. This yields at least a 20 percent cost reduction input a dataset covering various di- and statistical-inference libraries. • Benefits—fewer customers reported failures than for defect correction. mensions and partition it into clus- Essential Tools Chances are, if you find a fancy al- in previous releases, and security and maintain- The necessary tools, such as Coverity, Klocwork, Lat- ters satisfying certain criteria. A Machine learning’s popularity has gorithm somewhere and want to ability improved. tix, Structure 101, SonarX, and SourceMeter, are off the popular algorithm is k-means clus- brought along a wealth of tools. try it on your data, an implementa- shelf and account for even less per project. These criticality tering, which aims to partition the Most of them are open source, so us- tion in R exists for it. R boasts the On the basis of the results from many of our client analyses provide numerous other bene ts, such as the re- dataset so that each observation lies ers can easily experiment with them ggplot2 visualization library, which projects (and taking a conservative ratio of only 40 per- moval of speci c code-related risks and defects that other- closest to the mean of its cluster. and learn how to use them. Table 1 can produce excellent graphs. cent defects in critical components), we can calculate the wise are hard to identify (for example, security aws). Other clustering approaches include compares some popular machine- Python is more popular with peo- hierarchical clustering, Gaussian learning tools. ple with a computer science back- mixture models, genetic algorithms The numerical and statistical ground. Although not made spe- (in which the computer learns the communities are divided into two cifically for machine learning or

112 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge SEPTEMBER/OCTOBER 2016 | IEEE SOFTWARE 113 11 SOFTWARE TECHNOLOGY

Some popular machine-learning tools.

Tool

Python R Spark Matlab TensorFlow TABLE 1 TABLE License Open source Open source Open source Proprietary Open source

Distributed No No Yes No No

Visualization Yes Yes No Yes No

Neural nets Yes Yes Multilayer perceptron classifier Yes Yes

Supported languages Python R Scala, Java, Python, and R Matlab Python and C++

Variety of machine- High High Medium High Low learning models

Suitability as a general- High Medium Medium High Low purpose tool

Maturity High Very high Medium Very high Low

statistics, Python has extensive librar- puter’s main memory. If that’s not Julia programming language for ies for numerical computing (NumPy), possible, you must use a distributed technical computing, which aims at scientific computing (SciPy), statistics platform. The most well-known is top performance. Because Julia is (StatsModels), and machine learning Hadoop, but Hadoop isn’t the most new, it doesn’t have nearly as many (scikit-learn). These are largely wrap- convenient for machine learning. libraries as Python or R. Yet, thanks pers of C code, so you get Python’s Making even simple algorithms run to its impressive speed, its popularity convenience with C’s speed. on it can be a struggle. might grow. Although there are fewer machine- So, many people prefer to work Strong commercial players in- learning libraries for Python than at the higher level of abstraction that clude Matlab and SAS, which both there are for R, many programmers Spark offers. Spark leverages Ha- have a distinguished history. Matlab find working with Python easier. doop but looks like a scripting envi- has long offered solid tools for nu- They might already know the lan- ronment. You can interact with it us- merical computation, to which it has guage or find it easier to learn than ing Scala, Java, Python, or R. Spark added machine-learning algorithms R. They also find Python convenient has a machine-learning library that and implementations. For engineers for preprocessing data: reading it implements key algorithms, so for familiar with Matlab, it might be a from various sources, cleaning it, many purposes you don’t need to im- natural fit. SAS is a software suite and bringing it to the required for- plement anything yourself. for advanced statistical analysis; it mats. For visualization, Python re- H2O is a relatively newer entrant also has added machine-learning ca- lies on matplotlib. You can do pretty in the machine-learning scene. It’s a pabilities and is popular for business much everything on matplotlib, but platform for descriptive and predic- intelligence tasks. you might discover you have to put tive analytics that uses Hadoop and in some effort. The seaborn library Spark; you can use it with R and Py- ANNs and Deep Learning is built on top of it, letting you pro- thon. It implements supervised- and Cynics might roll their eyes, arguing duce elegant visualizations with little unsupervised-learning algorithms and that ANNs’ resurgence is déjà vu. code. a Web interface through which you It’s true; ANNs’ fundamental com- In general, R and Python work can organize your workflow. ponents have been around for about when the dataset fits in the com- A promising development is the half a century. However, it’s also true

12114 IEEE SOFTWAREComputing | Edge WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE April 2017 SOFTWARE TECHNOLOGY SOFTWARE TECHNOLOGY

that you can now architect them in those rules, you can easily under- Inference, and Prediction, 2nd Some popular machine-learning tools. new ways. ANNs can be used across stand how a classification tree classi- ed., Springer, 2009. the spectrum of machine learning: fies data. ANNs don’t produce any- • C.M. Bishop, Pattern Recogni- Tool classification, regression, clustering, thing their users can interpret. An tion and Machine Learning, Python R Spark Matlab TensorFlow and dimensionality reduction. ANN that classifies images doesn’t Springer, 2006. TABLE 1 TABLE License Open source Open source Open source Proprietary Open source Innovations in ANN architec- produce any rules; the network itself • K.P. Murphy, Machine Learn- tures and the availability of cheap embodies everything it has learned ing: A Probabilistic Perspective, Distributed No No Yes No No computing resources to run ANNs about image classification. MIT Press, 2012. Visualization Yes Yes No Yes No has brought about the burgeoning • E. Alpaydın, Introduction to of deep learning—using big ANNs Machine Learning, 3rd ed., MIT Neural nets Yes Yes Multilayer perceptron classifier Yes Yes to perform machine learning. Over any machine-learning Press, 2014. the last few years, deep learning has books have a practical Supported languages Python R Scala, Java, Python, and R Matlab Python and C++ chalked up headline-grabbing suc- M slant, aiming to intro- PANOS LOURIDAS is an associate professor cesses by beating humans in Jeop- duce machine learning on a particu- teaching algorithms and software at the Athens Variety of machine- High High Medium High Low ardy! and Go, learning to play ar- lar platform. As technologies quickly University of Economics and Business. He’s also learning models cade games, showing an uncanny evolve, it’s better to focus on getting an active developer. Contact him at louridas@ capability to recognize images, per- a solid grasp of the fundamentals. aueb.gr. Suitability as a general- High Medium Medium High Low forming automatic translation, and After all, using a machine-learning purpose tool so on. Deep learning is particularly platform isn’t difficult; knowing CHRISTOF EBERT is the managing director Maturity High Very high Medium Very high Low good at general tasks requiring the when to use a particular algorithm of Vector Consulting Services. He is on the IEEE elicitation of higher-level, abstract and how to use it well requires quite Software editorial board and teaches at the concepts from the input data, which a bit of background knowledge. University of Stuttgart and the Sorbonne in Paris. statistics, Python has extensive librar- puter’s main memory. If that’s not Julia programming language for is what the many layers of an ANN Here are four popular books: Contact him at [email protected]. ies for numerical computing (NumPy), possible, you must use a distributed technical computing, which aims at excel at. scientific computing (SciPy), statistics platform. The most well-known is top performance. Because Julia is Deep learning is usually imple- • T. Hastie, R. Tibshirani, and J. Selected CS articles and columns (StatsModels), and machine learning Hadoop, but Hadoop isn’t the most new, it doesn’t have nearly as many mented through matrices, so work- Friedman, The Elements of Sta- This articleare also originallyavailable for freeappeared at in http://ComputingNow.computer.org. (scikit-learn). These are largely wrap- convenient for machine learning. libraries as Python or R. Yet, thanks ing with it requires efficient matrix tistical Learning: Data Mining, IEEE Software, vol. 33, no. 5, 2016. pers of C code, so you get Python’s Making even simple algorithms run to its impressive speed, its popularity operations and manipulation. Usu- convenience with C’s speed. on it can be a struggle. might grow. ally the implementations are in C Although there are fewer machine- So, many people prefer to work Strong commercial players in- or C++, but designing ANNs at that learning libraries for Python than at the higher level of abstraction that clude Matlab and SAS, which both level is unwieldy. Python program- there are for R, many programmers Spark offers. Spark leverages Ha- have a distinguished history. Matlab mers can use the Theano library to find working with Python easier. doop but looks like a scripting envi- has long offered solid tools for nu- define ANNs, which are compiled to They might already know the lan- ronment. You can interact with it us- merical computation, to which it has C code that’s then compiled to ma- guage or find it easier to learn than ing Scala, Java, Python, or R. Spark added machine-learning algorithms chine language. Recently, Google re- Take the CS Library R. They also find Python convenient has a machine-learning library that and implementations. For engineers leased as open source its TensorFlow for preprocessing data: reading it implements key algorithms, so for familiar with Matlab, it might be a library for working with ANNs. wherever you go! from various sources, cleaning it, many purposes you don’t need to im- natural fit. SAS is a software suite You can interact with TensorFlow IEEE Computer Society magazines and Transactions are now and bringing it to the required for- plement anything yourself. for advanced statistical analysis; it through a Python API. A C++ API is available to subscribers in the portable ePub format. mats. For visualization, Python re- H2O is a relatively newer entrant also has added machine-learning ca- also available; although not as easy Just download the articles from the IEEE Computer Society Digital lies on matplotlib. You can do pretty in the machine-learning scene. It’s a pabilities and is popular for business to use, it might give some perfor- Library, and you can read them on any device that supports ePub. For much everything on matplotlib, but platform for descriptive and predic- intelligence tasks. mance benefits. more information, including a list of compatible devices, visit you might discover you have to put tive analytics that uses Hadoop and Before jumping on the deep- www.computer.org/epub in some effort. The seaborn library Spark; you can use it with R and Py- ANNs and Deep Learning learning bandwagon, keep in mind is built on top of it, letting you pro- thon. It implements supervised- and Cynics might roll their eyes, arguing that all machine-learning approaches duce elegant visualizations with little unsupervised-learning algorithms and that ANNs’ resurgence is déjà vu. lie on a spectrum based on the ease code. a Web interface through which you It’s true; ANNs’ fundamental com- of interpreting their results. For ex- In general, R and Python work can organize your workflow. ponents have been around for about ample, classification trees produce when the dataset fits in the com- A promising development is the half a century. However, it’s also true rules that classify data. By reading

114 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge SEPTEMBER/OCTOBER 2016 | IEEE SOFTWARE 115 13 AI AND EDUCATION Editor: Judy Kay, University of Sydney, [email protected]

Artiﬁ cial Intelligence as an Effective Classroom Assistant

Benedict du Boulay, University of Sussex

he fi eld of artifi cial intelligence in education • a model of pedagogy, so that the system can (AIED) has been in existence for about 40 make sensible tutorial moves, such as providing T effective feedback or adjusting the nature of the years and has operated under various other names, next task; and the most common of which is intelligent tutoring sys- • one or more interfaces through which the systems (ITSs). The fi eld was initially brought to wider tem and the learner can communicate to explore attention by papers in a special issue of the Inter- and learn about the domain in question. national Journal of Man-Machine Studies,1 in a book based on that special issue,2 and in AI books Over the years, many systems using various of the era.3 It continues to use techniques from AI pedagogical techniques and topics have been built and cognitive science to attempt to understand the and evaluated. To illustrate the scope of the work, nature of learning and teaching and to build sys- I discuss four diverse systems, which range from tems to assist learners to master new skills or un- classic teaching in a formal subject and a pro- derstand new concepts in ways that mimic the ac- cedural skill, to learning by creating external- tions of a skilled human tutor working one-on-one ized forms of knowledge for a highly conceptual with the learner. That is, such systems attempt to learning task, to rich, natural user interaction via adapt the way they teach to the learner’s knowl- speech for learning complex, culture-laden skills. edge, skill, and preferred ways of learning, and to The fi rst AIED example is a system to help learn- consider the learners’ affective trajectory as they ers understand basic algebra by being given prob- deal with the expected setbacks and impasses of lems and then provided with step-by-step feedback mastering new material. There is clearly some and guidance for their solution.4 The second exam- overlap with other uses of computing technology ple is a system that helps learners gain a concep- in education, although the commitment to individ- tual understanding of river ecosystems by building ual adaptation through modeling different parts of a concept map of that domain, as if for another the educational process is key. learner, and having that simulated other learner For such systems to adapt to the learner and provide take tests on the concept map’s adequacy.5 The a personalized experience, a typical conceptual archi- third example is a system that helps military per- tecture has evolved. This consists of sonnel learn to speak Arabic and understand the social and cultural norms needed to interact with • a model of the domain being learned, so that people in the country in which they are operating.6 the system can reason about and judge whether The fourth example illustrates the increas- a student’s answer or a problem-solving step is ing importance of the interface in AIED systems appropriate; and their use in informal learning environments, • a model of the learner’s current understanding such as museums, as well as formal ones. Figure 1 or skill level, so that tasks of appropriate com- shows Coach Mike, a pedagogical agent designed plexity can be posed; to help children visiting a museum learn about

76 1541-1672/16/$33.00 © 2016 IEEE Ieee InTeLLIGenT SYSTemS 14 April 2017 Published by thePublished IEEE Computer by the IEEE Society Computer Society 2469-7087/17/$33.00 © 2017 IEEE AI AND EDUCATION Editor: Judy Kay, University of Sydney, [email protected]

Answer-, Step-, and Substep-Based Tutoring

magine that a student must solve the following equation: In a step-based system, the student might be invited to 2(14 − x) = 23 + 3x. multiply out the bracket expression as a first step, and thus will I An answer-based system would expect the student to give 28 − 2x as the answer to that step. If the student requests Artiﬁ cial Intelligence do all the work offline and then provide the answer x = 1. a hint or gives a wrong answer to this step, the tutor can give If asked for a hint, the tutor can suggest broad ways of help about how to work that step. Once the step is completed going about the problem, such as to collect all the terms in correctly, the tutor would invite an answer to the next step, as an Effective x on one side of the equation, but the tutor has no way of such as reordering terms in the equation, and then on through knowing that this advice is being followed. If the answer further steps to the final answer. provided is wrong—for example, x = 1.25—the tutor might In a substep-based system, there might be a remedial Classroom Assistant be able to hypothesize that the student multiplied out the dialogue at a finer level than an individual step, for instance, bracket incorrectly, but if the answer provided is, say, x = 14, about what expressions such as 2x or 3x mean, if that seems it probably will not offer much in the way of specific help. warranted by the request for a hint or by a wrong step answer.

Benedict du Boulay, University of Sussex he fi eld of artifi cial intelligence in education • a model of pedagogy, so that the system can (AIED) has been in existence for about 40 make sensible tutorial moves, such as providing T effective feedback or adjusting the nature of the years and has operated under various other names, next task; and the most common of which is intelligent tutoring sys- • one or more interfaces through which the systems (ITSs). The fi eld was initially brought to wider tem and the learner can communicate to explore attention by papers in a special issue of the Inter- and learn about the domain in question. national Journal of Man-Machine Studies,1 in a book based on that special issue,2 and in AI books Over the years, many systems using various of the era.3 It continues to use techniques from AI pedagogical techniques and topics have been built and cognitive science to attempt to understand the and evaluated. To illustrate the scope of the work, nature of learning and teaching and to build sys- I discuss four diverse systems, which range from tems to assist learners to master new skills or un- classic teaching in a formal subject and a pro- derstand new concepts in ways that mimic the ac- cedural skill, to learning by creating external- tions of a skilled human tutor working one-on-one ized forms of knowledge for a highly conceptual with the learner. That is, such systems attempt to learning task, to rich, natural user interaction via adapt the way they teach to the learner’s knowl- speech for learning complex, culture-laden skills. edge, skill, and preferred ways of learning, and to The fi rst AIED example is a system to help learn- Figure 1. Coach Mike, a pedagogical agent, in three different poses.7 Source: H. Chad Lane; used with permission. consider the learners’ affective trajectory as they ers understand basic algebra by being given prob- deal with the expected setbacks and impasses of lems and then provided with step-by-step feedback mastering new material. There is clearly some and guidance for their solution.4 The second exam- robotics. This kind of application ex- tors’ work, including helping orientate duced a small but significant effect overlap with other uses of computing technology ple is a system that helps learners gain a concep- tends the role of classroom teaching7: visitors, encouraging them to explore, on learning.”11 in education, although the commitment to individ- tual understanding of river ecosystems by building and providing problem-solving chal- This column is not intended as sup- ual adaptation through modeling different parts of a concept map of that domain, as if for another It means that such systems need to go lenges and support. port for an argument about getting the educational process is key. learner, and having that simulated other learner beyond simply focusing on knowledge Some researchers have recently ar- rid of human teachers, but rather For such systems to adapt to the learner and provide take tests on the concept map’s adequacy.5 The outcomes. They must take seriously gued the benefits of AI systems in ed- as support for blended learning, in a personalized experience, a typical conceptual archi- third example is a system that helps military per- goals such as convincing a visitor to en- ucation,8,9 whereas others have been which some of the human teacher’s tecture has evolved. This consists of sonnel learn to speak Arabic and understand the gage, promoting curiosity and interest, more skeptical.10 This column looks work can be offloaded to AIED sys- social and cultural norms needed to interact with and ensuring that a visitor has a posi- at the evidence derived from meta- tems, as if to a classroom assistant. • a model of the domain being learned, so that people in the country in which they are operating.6 tive learning experience. In other words, reviews and meta-analyses conducted the system can reason about and judge whether The fourth example illustrates the increas- pedagogical agents for informal learning over the past five years. Its main fo- Meta-Analysis and a student’s answer or a problem-solving step is ing importance of the interface in AIED systems need to not only act as coach (or teacher), cus is on the comparative effective- Metareviews appropriate; and their use in informal learning environments, but also as advocate (or salesperson). ness of AIED systems versus human Since 2011, several metareviews and • a model of the learner’s current understanding such as museums, as well as formal ones. Figure 1 tutoring. Note that a metareview of meta-analyses have attempted to deter- or skill level, so that tasks of appropriate com- shows Coach Mike, a pedagogical agent designed Coach Mike was designed to emu- the use of pedagogical agents (not mine the degree to which a whole host plexity can be posed; to help children visiting a museum learn about late some of the human museum cura- necessarily in AIED systems) “proof systems have been educationally

76 1541-1672/16/$33.00 © 2016 IEEE Ieee InTeLLIGenT SYSTemS november/december 2016 www.computer.org/intelligent 77 Published by the IEEE Computer Society www.computer.org/computingedge 15 Table 1. Effect sizes adapted from work by Kurt VanLehn.12

Comparison No. studies Mean effect size Reliability (%) Answer based vs. no tutoring* 165 0.31 40 Step based vs. no tutoring 28 0.76 68 In a metareview of 107 studies, Substep based vs. no tutoring 26 0.40 54 Wenting Ma and colleagues found similar results to VanLehn for step-based Human vs. no tutoring 10 0.79 80 ITSs both when compared to a no- Step based vs. answer based 2 0.40 50 tutoring condition (that is, just a text- Substep based vs. answer based 6 0.32 33 book; mean effect size = 0.36) and, Human vs. answer based 1 –0.04 0 more positively than VanLehn, when Substep based vs. step based 11 0.16 0 compared to large-group instruction led Human vs. step based 10 0.21 30 by a human teacher (mean effect size 14 Human vs. substep based 5 –0.12 0 = 0.44). They found no differences when compared to small group human 13 *Row 1 was taken by VanLehn from a separate study. tutoring or one-on-one tutoring. The same authors analyzed 22 sys- effective. Typically, this has meant Given these granularity levels, tems for teaching programming and comparing them in terms of learning VanLehn derived 10 pairwise com- also found a “a significant advantage gains with other instructional meth- parisons of effect sizes (see Table of ITS over teacher-led classroom ods, such as whole-class teaching by 1). The rightmost column shows the instruction and non-ITS computer- a human teacher or the use of a text- proportion of the results for that row based instruction.”15 A larger version book without a teacher. where the individual study compari- of a similar study involving 280 stud- son was statistically reliable at the ies is currently in progress.16 VanLehn’s Meta-Analysis level p < 0.05. In a metareview of 50 studies in- Kurt VanLehn analyzed papers com- For the purposes of this review, the volving 63 comparisons, James Kulik paring five types of tutoring: no tutor- most interesting comparison is that and J.D. Fletcher found comparable im- ing (for example, learning with just between one-on-one human tutor- provements (mean effect size = 0.65),17 a textbook), answer-based tutoring, ing and step-based tutors (effect size but they distinguished studies that step-based tutoring, substep-based tu- = 0.21). By collating all the results in used standardized tests from those in toring, and human tutoring.12 Table 1, VanLehn found that step- which the tests were more specifically The difference between answer- based tutors were, within the limita- tuned to the system providing tuition, based, step-based, and substep-based tions of his review, “just as effective as with smaller effect sizes when stan- tutoring resides in the granularity of adult, one-on-one tutoring for increas- dardized tests were employed. Over- the interaction between the tutor and ing learning gains in STEM topics.”12 all, they concluded that “this meta- student (see the sidebar, “Answer-, He also found that although increasing analysis shows that ITSs can be very Step-, and Substep-Based Tutoring”). the granularity of instruction from an- effective instructional tools … Devel- Answer-based systems can provide swer-based to step-based yielded sig- opers of ITSs long ago set out to im- hints and feedback only at the level nificant gains, going to the finer level prove on the success of CAI tutoring of the overall answer. Step-based sys- of substep-based tutoring did not add and to match the success of human tems can provide hints, scaffolding, further value. (Note that this latter tutoring. Our results suggest that ITS and feedback on every step that the stu- finding was based on a small number developers have already met both of dent makes in solving the problem. By of studies only.) these goals.”17 They also found better contrast, substep-based systems work results for substep-based systems than at a finer granularity level still and “can Six Metareviews VanLehn, which they ascribed to dif- give scaffolding and feedback at a level Since VanLehn’s meta-analysis, six fering comparison methodologies. of detail that is even finer than the steps metareviews have been published, as Much smaller effect sizes were students would normally enter when well as a large-scale study of a spe- found by Saiying Steenbergen-Hu and solving a problem.”12 AI techniques are cific tutor (see Table 2). In the table, Harris Cooper in their meta-analysis required to underpin both step-based the “number of comparisons” col- of pupils using ITSs in school set- and substep-based tutors, whereas an- umn shows the number of instances tings.18 Kulik and Fletcher put this swer-based systems would typically fall for the given comparison in that row, down to the weaker study inclusion under the heading of computer-based not the total number of studies in the criteria (for example, the inclusion of or computer-assisted instruction (CAI). overall metareview. answer-based systems as if they were

78 www.computer.org/intelligent Ieee InTeLLIGenT SYSTemS 16 Computing Edge April 2017 Table 1. Effect sizes adapted from work by Kurt VanLehn.12 Table 2. Six metareviews and a large-scale study.

Comparison No. studies Mean effect size Reliability (%) Row number Metareview Comparison No. comparisons Mean effect size Standard error Answer based vs. no tutoring* 165 0.31 40 1 VanLehn12 Step based vs. one-on-one human 10 –0.21 0.19*† tutoring Step based vs. no tutoring 28 0.76 68 In a metareview of 107 studies, Wenting Ma and colleagues found simi- 2 Wenting Ma and colleagues14 Step based vs. one-on-one human 5 –0.11 0.10 Substep based vs. no tutoring 26 0.40 54 tutoring lar results to VanLehn for step-based Human vs. no tutoring 10 0.79 80 14 ITSs both when compared to a no- 3 Ma and colleagues Step based vs. large group human 66 0.44 0.05 Step based vs. answer based 2 0.40 50 instruction tutoring condition (that is, just a text- Substep based vs. answer based 6 0.32 33 4 John Nesbit and colleagues15 Step based vs. teacher-led group 11 0.67 0.09 book; mean effect size = 0.36) and, instruction Human vs. answer based 1 –0.04 0 more positively than VanLehn, when 5 James Kulik and J.D. Fletcher17 Step based and substep based vs. 63 0.65 0.07† Substep based vs. step based 11 0.16 0 compared to large-group instruction led conventional classes by a human teacher (mean effect size Human vs. step based 10 0.21 30 6 Saiying Steenbergen-Hu and Step based vs. one-on-one human 3 –0.25 0.24 14 Human vs. substep based 5 –0.12 0 = 0.44). They found no differences Harris Cooper (2014)19 tutoring when compared to small group human 7 Steenbergen-Hu and Cooper Step based vs. traditional classroom 16 0.37 0.07 *Row 1 was taken by VanLehn from a separate study.13 tutoring or one-on-one tutoring. (2014)19 instruction The same authors analyzed 22 sys- 8 Steenbergen-Hu and Cooper Step based and answer based vs. 26 0.09 0.01 effective. Typically, this has meant Given these granularity levels, tems for teaching programming and (2013)18 traditional classroom instruction comparing them in terms of learning VanLehn derived 10 pairwise com- also found a “a significant advantage 9 John Pane and colleagues20 Blended learning including a 147 schools –0.1 0.10 gains with other instructional meth- parisons of effect sizes (see Table of ITS over teacher-led classroom step-based system vs. traditional classroom instruction ods, such as whole-class teaching by 1). The rightmost column shows the instruction and non-ITS computer- 0.21 0.10 a human teacher or the use of a text- proportion of the results for that row based instruction.”15 A larger version book without a teacher. where the individual study compari- of a similar study involving 280 stud- 0.01 0.11 16 son was statistically reliable at the ies is currently in progress. 0.19 0.14 VanLehn’s Meta-Analysis level p < 0.05. In a metareview of 50 studies in- 10 Weighted mean AIED system vs. one-on-one human 18 –0.19 N/A Kurt VanLehn analyzed papers com- For the purposes of this review, the volving 63 comparisons, James Kulik tutoring paring five types of tutoring: no tutor- most interesting comparison is that and J.D. Fletcher found comparable im- 11 Weighted mean AIED system vs. conventional 182 0.47 N/A ing (for example, learning with just between one-on-one human tutor- provements (mean effect size = 0.65),17 classes a textbook), answer-based tutoring, ing and step-based tutors (effect size but they distinguished studies that * The standard error in row 1 is based on all 10 studies, not just the 30% that produced reliable results (see Table 1). step-based tutoring, substep-based tu- = 0.21). By collating all the results in used standardized tests from those in † Standard errors computed by this article’s author. toring, and human tutoring.12 Table 1, VanLehn found that step- which the tests were more specifically The difference between answer- based tutors were, within the limita- tuned to the system providing tuition, based, step-based, and substep-based tions of his review, “just as effective as with smaller effect sizes when stan- step-based systems) used by Steen- Rows 10 and 11 of Table 2 summa- lion students use the courses each tutoring resides in the granularity of adult, one-on-one tutoring for increas- dardized tests were employed. Over- bergen-Hu and Cooper, who also rize the results of the metareviews, ex- year.”21 They represent the most the interaction between the tutor and ing learning gains in STEM topics.”12 all, they concluded that “this meta- noted that lower achievers seemed cluding the evaluation of the Cogni- successful transition, in terms of student (see the sidebar, “Answer-, He also found that although increasing analysis shows that ITSs can be very to do worse with ITSs than did the tive Algebra Tutor (row 9), and show numbers of students, of AIED work Step-, and Substep-Based Tutoring”). the granularity of instruction from an- effective instructional tools … Devel- broad spectrum of school pupils, al- a weighted mean effect size of 0.47 from the laboratory to the class- Answer-based systems can provide swer-based to step-based yielded sig- opers of ITSs long ago set out to im- though Kulik and Fletcher disputed for AIED systems versus conven- room. They provide scaffolded help hints and feedback only at the level nificant gains, going to the finer level prove on the success of CAI tutoring this result.17 However, in a parallel tional classroom teaching. We use the with step-by-step problem-solving in of the overall answer. Step-based sys- of substep-based tutoring did not add and to match the success of human study of university students, Steen- term AIED system to cover all the sys- various domains, mostly mathemati- tems can provide hints, scaffolding, further value. (Note that this latter tutoring. Our results suggest that ITS bergen-Hu and Cooper found more tems—step-based, substep-based, cal, and are designed to be used in and feedback on every step that the stu- finding was based on a small number developers have already met both of positive effect sizes (in the range of and answer-based—looked at in the a blended learning manner, thus dent makes in solving the problem. By of studies only.) these goals.”17 They also found better 0.32 to 0.37) for ITSs as compared to metareviews. The comparison with freeing up the teacher to work with contrast, substep-based systems work results for substep-based systems than conventional instruction.19 They con- one-on-one human tutoring shows that other children. Teachers are trained at a finer granularity level still and “can Six Metareviews VanLehn, which they ascribed to dif- clude that ITSs “have demonstrated AIED systems do slightly worse, with a to make the best of these systems’ ar- give scaffolding and feedback at a level Since VanLehn’s meta-analysis, six fering comparison methodologies. their ability to outperform many in- mean effect size of –0.19. In both cases, rival in their classrooms in terms of of detail that is even finer than the steps metareviews have been published, as Much smaller effect sizes were structional methods or learning ac- the means are weighted in terms of the managing all the pupils in the class- students would normally enter when well as a large-scale study of a spe- found by Saiying Steenbergen-Hu and tivities in facilitating college-level number of comparisons in the metare- room before, during, and after the solving a problem.”12 AI techniques are cific tutor (see Table 2). In the table, Harris Cooper in their meta-analysis students’ learning of a wide range of view, not in terms of the original N val- use of the tutors.4 Individual evalu- required to underpin both step-based the “number of comparisons” col- of pupils using ITSs in school set- subjects, although they are not as ef- ues in the studies themselves. ations of various Cognitive Tutors and substep-based tutors, whereas an- umn shows the number of instances tings.18 Kulik and Fletcher put this fective as human tutors. ITSs appear are included in the reviews already swer-based systems would typically fall for the given comparison in that row, down to the weaker study inclusion to have a more pronounced effect on Cognitive Tutors described. under the heading of computer-based not the total number of studies in the criteria (for example, the inclusion of college-level learners than on K–12 Cognitive Tutors “are found in about A large-scale US study of the Cog- or computer-assisted instruction (CAI). overall metareview. answer-based systems as if they were students.”19 3,000 schools, and over a half mil- nitive Algebra Tutor undertook a

78 www.computer.org/intelligent Ieee InTeLLIGenT SYSTemS november/december 2016 www.computer.org/intelligent 79 www.computer.org/computingedge 17 between-schools project involv- dle schools and in the second year 5. K. Leelawong and G. Biswas, “Design- ing 73 high schools and 74 middle of the evaluation as opposed to the ing Learning by Teaching Agents: The schools across seven states.20 The first year. For various reasons, the Betty’s Brain System,” Int’l J. Artificial schools were matched in pairs: half way forward for AIED systems in Intelligence in Education, vol. 18, no. 3, received the Cognitive Algebra Tu- the classroom must be the blended 2008, pp. 181–208. tor and adjusted their teaching to in- model—classroom assistants, if you 6. W.L. Johnson, “Serious Use of a Serious clude it as they saw fit, whereas the like—in order to provide detailed Game for Language Learning,” Int’l J. others carried on with their regular one-on-one tutoring for some stu- Artificial Intelligence in Education, vol. method of teaching algebra. The dents while the human teacher at- 20, no. 2, 2010, pp. 175–195. study ran for two years and found tends to others, as well as having 7. H.C. Lane et al., “The Effects of a no significant differences on post- overall responsibility for all the stu- Pedagogical Agent for Informal Science test scores in the first year of the dents’ progress. Education on Learner Behaviors and study, but in the second year, the Of course, good post-test results Self-Efficacy,” Proc. 16th Int’l Conf. high schools that used the Cognitive are not the only criteria for judg- Artificial Intelligence in Education, Tutor showed a small but significant ing whether an educational technol- 2013, pp. 309–318. effect size of 0.21 (see the bolded ogy will be or should be adopted.10 8. R. Luckin et al., Intelligence Unleashed: data in row 9 of Table 2). However, the overall message of An Argument for AI in Education, Pear- Note that how the Cognitive Tutor these evaluations is that blending son, 2016. was actually used in the classrooms AIED technology with other forms of 9. B.P. Woolf et al., “AI Grand Challenges was not controlled, although post hoc teaching is beneficial, particularly for for Education,” AI Magazine, vol. 34, analyses showed that teachers did not older pupils and college-level students no. 4, 2014, pp. 66–84. generally use the Tutor exactly as rec- studying STEM subjects. 10. N. Enyedy, “Personalized Instruction: ommended by its developers. New Interest, Old Rhetoric, Limited Results, and the Need for a New Direc- Acknowledgments tion for Computer-Mediated Learning,” The overall conclusion of these meta- This column is an adapted and expanded policy brief, National Education Policy reviews and analyses is that AIED sys- version of a letter to the editor of the Inter- Center, 2014. tems perform better than both CAI national Journal of Artificial Intelligence in 11. N.L. Schroeder, O.O. Adesope, and R.B. Education.22 systems and human teachers work- Gilbert, “How Effective Are Pedagogical ing in large classes. They perform Agents for Learning? A Meta-Analytic slightly worse than one-on-one human References Review,” J. Educational Computing tutors. Most of the systems taught 1. J.S. Brown, R.R. Burton, and A.G. Bell, Research, vol. 49, no. 1, 2013, pp. 1–39. mathematics or STEM subjects, be- “SOPHIE: A Step Towards a Reactive 12. K. VanLehn, “The Relative Effective- cause these are the kinds of subject Learning Environment,” Int’l J. Man- ness of Human Tutoring, Intelligent for which it is easier to build the do- Machine Studies, vol. 7, 1975, Tutoring Systems, and Other Tutoring main and student models mentioned pp. 675–696. Systems,” Educational Psychologist, in the introduction. Note that there 2. T. O’Shea, “A Self-Improving Quadratic vol. 46, no. 4, 2011, pp. 197–221. was a degree of overlap between these Tutor,” Intelligent Tutoring Systems, D. 13. C.-L.C. Kulik and J.A. Kulik, “Effective- metareviews and analyses in terms of Sleeman and J.S. Brown, eds., Academic ness of Computer-Based Instruction: An the collections of individual evalu- Press, 1982. Updated Analysis,” Computers in Human ations from which they have drawn 3. J.S. Brown and R.R. Burton, “Multiple Behavior, vol. 7, nos. 1–2, 1991, pp. 75–94. their conclusions. Representations of Knowledge for 14. W. Ma et al., “Intelligent Tutoring Sys- The specific study of the Cogni- Tutorial Reasoning,” Representation tems and Learning Outcomes: A Meta- tive Tutor for Algebra evaluated its and Understanding, D.G. Bobrow and Analysis,” J. Educational Psychology, use as a blended addition to the reg- A. Collins, eds., Academic Press, 1975, vol. 106, no. 4, 2014, pp. 901–918. ular algebra teaching in the schools pp. 311–349. 15. J.C. Nesbit et al., “How Effective Are in which it was tried, rather 4. K.R. Koedinger et al., “Intelligent Tu- Intelligent Tutoring Systems in Com- than as a total replacement for the toring Goes to School in the Big City,” puter Science Education?” Proc. IEEE teachers, and found good results Int’l J. Artificial Intelligence in Educa- 14th Int’l Conf. Advanced Learning in high schools as opposed to mid- tion, vol. 8, no. 1, 1997, pp. 30–43. Technologies, 2014, pp. 99–103.

80 www.computer.org/intelligent Ieee InTeLLIGenT SYSTemS 18 Computing Edge April 2017 16. J.C. Nesbit et al., “Work in Progress: 19. S. Steenbergen-Hu and H. Cooper, “A 22. B. du Boulay, “Recent Meta-Reviews Intelligent Tutoring Systems in Com- Meta-Analysis of the Effectiveness of and Meta-Analyses of AIED Systems,” puter Science and Software Engineer- Intelligent Tutoring Systems on College Int’l J. Artifi cial Intelligence in Educa- ing Education,” Proc. 122nd Am. Soc. Students’ Academic Learning,” J. Edu- tion, vol. 26, no. 1, 2016, pp. 536–537. Eng. Education Ann. Conf., 2015, cational Psychology, vol. 106, no. 2, paper 11861. 2014, pp. 331–347.S. benedict du boulay is an emeritus profes- 17. J.A. Kulik and J.D. Fletcher, “Effective- 20. J.F. Pane et al., “Effectiveness of sor of artifi cial intelligence and a member of ness of Intelligent Tutoring Systems: A Cognitive Tutor Algebra I at Scale,” the Human Centered Technology Research Meta-Analytic Review,” Rev. Educa- Educational Evaluation and Policy Group at the University of Sussex. Contact tional Research, vol. 86, no. 1, 2016, Analysis, vol. 36, no. 2, 2014, him at [email protected]. pp. 42–78. pp. 127–144. 18. S. Steenbergen-Hu and H. Cooper, “A 21. K.R. Koedinger and V. Aleven, “An Meta-Analysis of the Effectiveness of Interview Refl ection on ‘Intelligent Intelligent Tutoring Systems on K–12 Tutoring Goes to School in the Big Read your subscriptions Students’ Mathematical Learning,” J. City,’” Int’l J. Artifi cial Intelligence in This article originallythrough the appeared myCS pub- in Educational Psychology, vol. 105, Education, vol. 16, no. 1, 2016, IEEE Intelligentlications Systems portal, vol. at http:// 31, mycs.computer.org. no. 4, 2013, pp. 970–987. pp. 13–24. no. 6, 2016.

EXECUTIVE STAFF Executive Director: Angela R. Burgess; Director, Governance & Associate Executive Director: Anne Marie Kelly; Director, Finance & Accounting: PURPOSE: The IEEE Computer Society is the world’s largest association Sunny Hwang; Director, Information Technology & Services: Sumit of computingKeeping professionals and is the leading provider of technical Kacker; Director, Membership Development: Eric Berkowitz; Director, information in the field. Products & Services: Evan M.What’s Butterfield; Director,the Buzz? Sales & Marketing: MEMBERSHIP: Members receive the monthly magazine Computer, Chris Jensen discounts, and opportunities to serve (all activities are led by volunteer If it’s worth knowing, it will be members). Membership is open to all IEEE members, affiliate society COMPUTER SOCIETY OFFICEScovered in an IEEE Computer Society at the Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928 members, andYOU others interested in the computer field. Webinar! Check out these topics and OMBUDSMAN: Email [email protected]. Phone: +1 202 371 0101 • Fax: more+1 202 by 728 industry’s 9614 • Email: leading [email protected] experts: Los Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720 COMPUTER SOCIETY WEBSITE: www.computer.org • Hidden “Gotcha’s” Most IT Experts Phone: +1 714 821 8380 • Email: [email protected] Next BoardCenter Meeting: 12–17 June 2017, Phoenix, AZ, USA Miss in Hybrid Cloud MEMBERSHIP & PUBLICATION• Achieving ORDERS Application Delivery EXECUTIVE COMMITTEE Phone: +1 800 272 6657 • Fax: +1Velocity 714 821 with 4641 a• Email:482% [email protected] President: Jean-Luc Gaudiot; President-Elect: Hironori Kasahara; Past Asia/Pacific: Watanabe Building,• Preparing 1-4-2 Minami-Aoyama, for the Next Minato-ku,Generation Tokyo President: Rogerof U. Fujii; Secretary: Forrest Shull; First VP, Treasurer: 107-0062, Japan • Phone: +81 3 of3408 Enterprise 3118 • Fax: Applications +81 3 3408 3553 • TechnologyEmail: [email protected] David Lomet; Second VP, Publications: Gregory T. Byrd; VP, Member & • Preventing Defects With Test Geographic Activities: Cecilia Metra; VP, Professional & Educational IEEE BOARD OF DIRECTORSDriven Development Activities: Andy T. Chen; VP, Standards Activities: Jon Rosdahl; V P, • Four Strategies for Responsive Technical &IEEE Conference Activities:Computer Hausi A. Müller; Society2017–2018 IEEE President & CEO: Karen Bartleson; President-Elect: James Jefferies; Past President: Barry L. Shoop; Secretary:Design William Walsh; Treasurer: John Director & Delegate Division VIII: Dejan S. Milojičić; 2016–2017 IEEE Director & WebinarsDelegate Division V: Harold Javid; 2017 IEEE Director-Elect & W. Walz; Director & President,Learn IEEE-USA: something Karen new.Pedersen; Check Director out a & Delegate Division V-Elect: John W. Walz President, Standards Association:Computer Forrest Society Don Wright; webinar Director today! & VP, Educational Activities: S.K. Ramesh; Director & VP, Membership and BOARD OF GOVERNORS Geographic Activities: Mary Ellen Randall; Director & VP, Publication Term Expiring 2017: Alfredo Benso, Sy-Yen Kuo, Ming C. Lin, Fabrizio Services and Products: Samir El-Ghazaly; Director & VP, Technical Lombardi, Hausi A. Müller, Dimitrios Serpanos, Forrest J. Shull Activities: Marina Ruggieri; Director & Delegate Division V: Harold Javid; Stay relevant with the IEEE Computer Society Term Expiring 2018: Ann DeMarle, Fred Douglis, Vladimir Getov, Bruce Director & Delegate Division VIII: Dejan S. Milojičić M. McMillin,More Cecilia atMetra, www.computer.org/webinars Kunio Uchiyama, Stefano Zanero Term Expiring 2019: Saurabh Bagchi, Leila De Floriani, David S. Ebert, Jill I. Gostin, William Gropp, Sumi Helal, Avi Mendelson revised 26 Jan. 2017

november/december 2016 www.computer.org/intelligent 81 www.computer.org/computingedge 19 From the Editor in Chief Editor in Chief: Maria R. Ebling n IBM T.J. Watson Research Center n [email protected]

TJBot and Zooids: The Connection between Pervasive Computing and AI

Maria R. Ebling, IBM T.J. Watson Research Center

ervasive computing and AI have EDIToRIAl BoARD CHAngEs P always struck me as a match made in heaven. The notion that technology I’m sorry to announce that Anind Dey will be stepping down as Associate Editor- will fade into the background of our in-Chief. His expertise and leadership in the area of human-computer interaction, lives necessarily requires understand- especially in context-aware computing, will be missed. We thank him for his many ing and anticipating our needs, which years of service to IEEE Pervasive Computing. requires some level of AI. Zooids are small, ant-like robots with Both of these projects also leverage BrIngIng AI to PErvASIvE little wheels, a touch sensor, gyros, and internal AI capabilities—either explic- ComPutIng an optical sensor on top to monitor itly or implicitly. These kits highlight The maker community has recently instructions from a projector overhead. that our industry is on the cusp of fun- rolled out kits to bring these two areas Zooids work cooperatively, in a swarm, damental changes in the way we will of computer science together. Two to accomplish things like moving your interact with technology. notable examples are the TJBot and phone closer to you or coming out of a the Zooids. mouse hole in the wall at night to tidy up In thIS ISSuE TJBot is a DIY kit for creating a card- while you sleep (I like this one the best Drones, our theme for this issue, are board robot with Watson inside. Inside but it’s still just a vision). Like TJBot, another good example of technology TJBot is a Raspberry Pi with a camera, Zooids software is available on GitHub, that falls at the intersection of perva- microphone, speaker, and LED light and the authors invite interested com- sive computing with AI. Many thanks with easy connection to Watson services. munity members to try them out. You to Albrecht Schmidt and Floyd Mueller The TJBot kit provides three starter can find the software at https://github. for focusing our attention on drones. In recipes. The first is a sentiment analysis com/ShapeLab/SwarmUI. this issue, we also introduce two new that changes the color of TJBot’s LED Interestingly, both of these maker departments: IoT News and Human based on the emotional sentiment of a projects are exploring fundamental Augmentation. particular topic on Twitter. The second questions about human-computer Our first new department, IoT News, lets you use your voice to control TJBot. interaction. TJBot is looking to under- is led by Florian Michahelles from Sie- The third lets you have a conversation stand how humans will (want to) inter- mens and presents news and research with TJBot. This open source project act with cognitive objects, especially on the Internet of Things. The coverage is available on GitHub. Directions for using voice commands. Zooids are will range from industrial use cases of printing your very own TJBot, as well exploring how humans might exploit IoT to social and business implications. as the initial recipes and recipes contrib- swarm-based user interfaces consisting Michahelles is looking for contribu- uted by the community, are available at of small, tangible robots working in tions from the community and outlines, https://ibmtjbot.github.io. concert with one another and a human. in more detail, what he is looking for

mISSIon StAtEmEnt: IEEE Pervasive Computing is a catalyst for advancing research and practice in mobile and ubiquitous computing. It is the premier publishing forum for peer-reviewed articles, industry news, surveys, and tutorials for a broad, multidisciplinary community.

April 2017 Published by the IEEE Computer Society 2469-7087/17/$33.00 © 2017 IEEE 2 20PERVASIVE computing Published by the IEEE CS n 1536-1268/17/$33.00 © 2017 IEEE From the Editor in Chief Editor in Chief: Maria R. Ebling n IBM T.J. Watson Research Center n [email protected] CAll FoR VolunTEERs IEEE Computer Society IEEE Pervasive Computing is launching a open positions include department drive to recruit new volunteers. Through editors for three departments: Confer- Publications office a position on the IEEE Pervasive Comput- ences, Education, and Wearable Com- 10662 Los Vaqueros Circle ing editorial or initiatives team, you can puting. We’re also open to proposals Los Alamitos, CA 90720 serve your profession, interact closely for new departments. In addition, we’re with leaders in our field, affect our field’s creating a new organization within our TJBot and Zooids: direction, and gain recognition and team of volunteers, the initiatives team. visibility. Members of this team will help the StAff We are reaching out to our readers to magazine in other roles by contribut- Lead Editor The Connection between expand our editorial team beyond our ing their knowledge, experience, and Brian Brannon immediate networks. We encourage connections. The open positions for the [email protected] Pervasive Computing and AI researchers from all demographics, loca- initiatives team include the following: tions, organizations and areas of research online presence and engagement, so- Content Editor Shani Murray Maria R. Ebling, IBM T.J. Watson Research Center relevant to pervasive computing, mobile cial network ambassador, constituency computing, and the Internet of Things to ambassador manager, constituency Assoc. Peer Review Manager join IEEE Pervasive Computing’s volunteer ambassador, awards, IEEE Computer Hilda Carman network. you can see the current board society project liaison, and member Publications Coordinator composition at www.computer.org/ at-large. [email protected] pervasive, under “About.” Anyone interested in volunteering is ervasive computing and AI have EDIToRIAl BoARD CHAngEs We’re looking for volunteers to join asked to fill out the application found at: Contributors always struck me as a match made our editorial board as well as to join Keri Schreiner, P http://tinyurl.com/zgwrzj5. our new initiatives team. Members of Joan Taylor, and Teri Sullivan in heaven. The notion that technology I’m sorry to announce that Anind Dey will be stepping down as Associate Editor- the editorial board contribute, edit, or Full consideration will be given to appli- Director, Products & Services will fade into the background of our in-Chief. His expertise and leadership in the area of human-computer interaction, manage the magazine’s content. The cations received by 1 February 2017. Evan Butterfield lives necessarily requires understand- especially in context-aware computing, will be missed. We thank him for his many years of service to IEEE Pervasive Computing. Senior Manager, Editorial Services ing and anticipating our needs, which Robin Baldwin requires some level of AI. Senior Business Development Manager Zooids are small, ant-like robots with Both of these projects also leverage from community submissions. Please built a system that follows human ges- Sandra Brown BrIngIng AI to PErvASIvE little wheels, a touch sensor, gyros, and internal AI capabilities—either explic- read through it and consider submitting tures from the phone, even when the CallMembership Development Manager ComPutIng an optical sensor on top to monitor itly or implicitly. These kits highlight material for future issues. phone is in your pocket. Imagine being Cecelia Huffman The maker community has recently instructions from a projector overhead. that our industry is on the cusp of fun- Our second new department, Human able to use gestures to control your for Senior Advertising Coordinator rolled out kits to bring these two areas Zooids work cooperatively, in a swarm, damental changes in the way we will Augmentation, is led by Albrecht phone without having to dig it out of Marian Anderson [email protected] of computer science together. Two to accomplish things like moving your interact with technology. Schmidt. In this department, Schmidt your pocket! I found the article fasci- notable examples are the TJBot and phone closer to you or coming out of a plans to discuss technologies designed to nating and seemingly right out of a sci- IEEE Software seeks practical, readable the Zooids. mouse hole in the wall at night to tidy up In thIS ISSuE augment the human intellect, including ence fiction novel. articles that will appeal to experts and TJBot is a DIY kit for creating a card- while you sleep (I like this one the best Drones, our theme for this issue, are cognition and perception. In this issue, In our Pervasive Health depart- nonexpertsIEEE Pervasive alike. Computing The magazine (ISSN 1536-1268) aims is published quarterly by the IEEE Computer Society. IEEE board robot with Watson inside. Inside but it’s still just a vision). Like TJBot, another good example of technology he argues that the ongoing technical rev- ment, Kay Connelly, Oscar Mayora, toHeadquarters, deliver reliable, Three Park useful, Ave., 17th leading-edge Floor, New York, NY 10016-5997; IEEE Computer Society Publications TJBot is a Raspberry Pi with a camera, Zooids software is available on GitHub, that falls at the intersection of perva- olution is the modern equivalent of the Jesus Favela, Maia Jacobs, Aleksandar information to software developers, engineers, Office, 10662 Los Vaqueros Circle, PO Box 3014, Los microphone, speaker, and LED light and the authors invite interested com- sive computing with AI. Many thanks industrial revolution. He explains how Matic, Chris Nugent, and Stefan Wag- andAlamitos, managers CA 90720-1314, to help phonethem +1stay 714 on 821 top 8380; of with easy connection to Watson services. munity members to try them out. You to Albrecht Schmidt and Floyd Mueller today’s technologies enhance our cogni- ner report on the outcomes of a set of IEEE Computer Society Headquarters, 2001 L St., rapidSte. 700, technology Washington, change. DC 20036. Topics Subscribe include to IEEE The TJBot kit provides three starter can find the software at https://github. for focusing our attention on drones. In tive abilities much the way technologies discussions held at the Future of Per- requirements,Pervasive Computing design, by visiting construction, www.computer.org/ tools, pervasive. recipes. The first is a sentiment analysis com/ShapeLab/SwarmUI. this issue, we also introduce two new of the industrial revolution enhanced our vasive Health Workshop. During this project management, process improvement, that changes the color of TJBot’s LED Interestingly, both of these maker departments: IoT News and Human physical abilities. This department will workshop, they identified five major maintenance, testing, education and training, based on the emotional sentiment of a projects are exploring fundamental Augmentation. help us understand the societal implica- themes: technological challenges and quality, standards, and more. particular topic on Twitter. The second questions about human-computer Our first new department, IoT News, tions of pervasive computing and IoT. opportunities, adoption and adherence Postmaster: Send undelivered copies and address changes to IEEE Pervasive Computing, Membership lets you use your voice to control TJBot. interaction. TJBot is looking to under- is led by Florian Michahelles from Sie- In our Smartphones department, of pervasive health solutions, open data Processing Dept., IEEE Service Center, 445 Hoes Author guidelines: The third lets you have a conversation stand how humans will (want to) inter- mens and presents news and research Rajalakshmi Nandakumar and Shyam- for pervasive health, pervasive health Lane, Piscataway, NJ 08854-4141. Periodicals www.computer.org/software/authorpostage paid at New York, NY, and at additional mailing with TJBot. This open source project act with cognitive objects, especially on the Internet of Things. The coverage nath Gollakota discuss the prototype methods and ethical issues, and the offices. Canadian GST #125634188. Canada Post Further details: [email protected] is available on GitHub. Directions for using voice commands. Zooids are will range from industrial use cases of system they built, demonstrating some road to education on pervasive health. Publications Mail Agreement Number 40013885. www.computer.org/softwareReturn undeliverable Canadian addresses to PO Box printing your very own TJBot, as well exploring how humans might exploit IoT to social and business implications. compelling use cases for active sonar I found the summary of the discussion 122, Niagara Falls, ON L2E 6S8. Printed in the USA. as the initial recipes and recipes contrib- swarm-based user interfaces consisting Michahelles is looking for contribu- implementations on smartphones. They very interesting, especially the need for uted by the community, are available at of small, tangible robots working in tions from the community and outlines, describe two major use cases. First, they open data and a formal curriculum. https://ibmtjbot.github.io. concert with one another and a human. in more detail, what he is looking for built and studied a system that detects Finally, our Notes from the Commu- IEEE prohibits discrimination, harassment and sleep disorders using active sonar on a nity department highlights some really bullying: For more information, visit www.ieee.org/ web/aboutus/whatis/policies/p9-26.html. smartphone. What makes this compel- interesting technologies. As someone who IEEE mISSIon StAtEmEnt: IEEE Pervasive Computing is a catalyst for advancing research and practice in mobile and ubiquitous computing. ling is that you no longer have to attach has experienced physical therapy, includ- It is the premier publishing forum for peer-reviewed articles, industry news, surveys, and tutorials for a broad, multidisciplinary community. all kinds of wires to the patient using ing for repetitive strain injuries of the white goo in their hair! Second, they arms, I found the skintillates temporary

www.computer.org/computingedge 2 PERVASIVE computing Published by the IEEE CS n 1536-1268/17/$33.00 © 2017 IEEE JAnuARy–MARCH 2017 PERVASIVE computing21 3 From the editor in ChieF From the editor in ChieF

tattoos particularly intriguing. The about users’ destinations. The authors he ability to bring AI together with potential of this technology for use in recognize the importance of location T pervasive computing will help us physical therapy seems enormous! My predictions as well as the challenges to create more compelling user experi- physical therapist could see the potential faced in making such predictions on ences in the years to come. As we move in helping change people’s habits, espe- mobile devices, including power con- forward, I anticipate these two fields cially when they sit in front of a keyboard. sumption, privacy, and data sparsity. coming together more and more, in We also have three feature articles To overcome these challenges, they applications from health, to drones, to this issue, starting with “A Survey of propose dividing the geographic area smarter cities. Diet Monitoring Technology,” by Haik into regions that uniquely differentiate Kalantarian, Nabil Alshurafa, and Majid users’ paths. This approach addresses Sarrafzadeh. This article provides a deep- the power consumption problem maria r. Ebling is a director at the IBM T.J. dive analysis of different technologies because updates are needed much less Watson Research Center. she manages a team available for monitoring eating behav- frequently. It addresses the privacy building systems capable of supporting a iors. These technologies range from problem because the user’s exact loca- smarter Planet while not forgetting about the manual tracking, to acoustic monitoring, tion within the region is unimportant. people who use such systems. Ebling received to gesture recognition, to instrumented It addresses the data sparsity problem her PhD in computer science from Carnegie Mel- objects, to visual approaches, to sensors by reducing the overall state space. lon university. she’s a member of the IBM Acad- attached to the skin. The authors also Although this approach shows prom- emy of Technology, a distinguished member of performed a survey designed to measure ise, more research is needed before it is the ACM, and a senior member of IEEE. Contact user acceptance of these technologies. applied in “the real world.” her at [email protected]. This article provides a thorough review of the many ways to do automated tracking of eating behaviors. This article originally appeared in The next article is “Semi-Automated IEEE Pervasive Computing, vol. 16, no. 1, 2017. Tracking: A Balanced Approach for Self-Monitoring Applications,” by Eun Kyoung Choe, Saeed Abdullah, Mash- fiqui Rabbi, Edison Thomaz, Daniel A. Epstein, Felicia Cordeiro, Matthew Kay, Gregory D. Abowd, Tanzeem Call Choudhury, James Fogarty, Bong- shin Lee, Mark Matthews, and Julie for Articles A. Kientz. The authors make the case for semi-automated tracking of health behaviors. Semi-automated tracking IEEE Pervasive Computing balances the convenience of automated tracking with the awareness of manual seeks accessible, useful papers on the latest peer- tracking. The authors define semi- reviewed developments in pervasive, mobile, and automated tracking and walk readers through its characteristics, contrast- ubiquitous computing. Topics include hardware ing the strengths and weaknesses with technology, software infrastructure, real-world those of both manual and automated sensing and interaction, human-computer tracking. They then examine three Further important design considerations, and interaction, and systems considerations, including details: give examples of semi-automated track- deployment, scalability, security, and privacy. ing of food, mood, and sleep to dem- pervasive@ onstrate these ideas. Semi-automated computer.org Author guidelines: tracking strikes me as a promising com- www. promise between fully automated and www.computer.org/mc/pervasive/author.htm computer. fully manual tracking approaches. Finally, in “Region Formation for org/pervasive Efficient Offline Location Prediction,” Ian Craig and Mark Whitty explore the use of regions to make predictions

Computing Edge April 2017 4 PERVASIVE22 computing www.computer.org/pervasive EDITOR ANN E.K. SOBEL COMPUTING EDUCATION Miami University; [email protected]

Engineering the New Boundaries of AI

Amir Banifatemi, XPRIZE Foundation Jean-Luc Gaudiot, University of California, Irvine

Recognizing the increasingly critical role that AI plays in all aspects of modern society, the

XPRIZE Foundation launched the IBM Watson ENTER XPRIZE AI XPRIZE. Interdisciplinary teams will advance It’s this question of the impact of AI on mankind that inspired the IBM current AI technologies as they compete for the Watson AI XPRIZE, the latest innovation challenge from the XPRIZE grand prize. Foundation. IBM sponsored the competition with a prize purse of million, including a grand prize of I surrounds us. It’s in our search engines, auto- million, a second- place prize of million, and a third- mobiles, phones, video-streaming websites, place prize of ,. The goal of the competition is to and even the systems that run our nancial accelerate the development of scalable AI solutions to ad- markets. We interact with and rely on intelli- dress humanity’s greatest challenges. The four-year com- gentA machines throughout our daily activities. petition will have milestone completions in , , AI has made steady, linear progress toward adoption and , with the top three nalists competing for the and integration over the past years. It’s now at an im- grand prize in . portant in ection point—the adoption curve could soon Recognizing the competition’s humanitarian impor- transcend the linear growth pattern and take an exponen- tance and global reach, the IEEE Computer Society (CS) tial leap. With machine learning, AI technologies have the formed a partnership agreement in August with potential to make sense of vast, complex datasets. This the XPRIZE Foundation. Select CS members will join begs the question: How can we best leverage both human the XPRIZE scienti c advisory board, and the CS will intelligence and these powerful AI technologies to bene t draw from its global network of experts to help judge humanity now and in the future? and advise the competitors. The CS will also encourage

2469-7087/17/$33.00 © 2017 IEEE Published by the IEEE Computer Society April 2017 23 COMPUTER 0018-9162/16/$33.00 © 2016 IEEE PUBLISHED BY THE IEEE COMPUTER SOCIETY NOVEMBER 2016 77 COMPUTING EDUCATION

driving competitors to invest every multimillion-dollar competitions tend intellectual and financial resource at to have a greater impact on the market XPRIZE ORIGIN STORY their disposal to solve the problem, and innovation than smaller, more JOIN THE COMPETITION reach the goal, and win the prize, in- localized versions. However, their he history of XPRIZE is unique. The first was inspired by the $25,000 Orteig novation can take center stage. A large greater popularity has brought them evelopers, engineers, entrepreneurs, and innovators throughout the world TPrize, which was awarded in 1927 to American aviator Charles Lindbergh for number of groups can work in paral- increasingly into other domains—in- Dare encouraged to form or join an AI XPRIZE team. To read more or submit an flying the Spirit of St. Louis on the first nonstop flight between New York and Paris. lel toward the goal, and the resulting cluding the classroom. Educators have entry, visit ai.xprize.org. The original XPRIZE, announced in 1996, was a $10 million prize to the first solution or benefit can be quickly ad- been enthusiastic to explore chal- privately financed team that could build and fly a three‐passenger vehicle 100 opted and realized by society. lenge-based learning, appreciating km into space twice within two weeks. The prize, later titled the Ansari XPRIZE their potential for developing stu- for sub orbital spaceflight, motivated 26 teams from 7 nations to invest more INCENTIVE-DRIVEN dents’ abilities to work in teams, to be than $100 million in pursuit of the $10 million prize. On 4 October 2004, the prize COMPETITIONS SPUR inventive and resourceful, and to push experience, reason toward specific of stakeholders to help define the was awarded to Mojave Aerospace Ventures, kicking off the personal spaceflight INNOVATION beyond traditional lines of thought goals, and interact naturally with hu- guidelines and mechanisms by which revolution. Challenge-based models offering sub- and institutional boundaries. Indeed, man collaborators. Thus, AI applica- AI could or should be used. The ben- The first competition created a funding model to stimulate broad investment stantial financial prizes have histor- leveraging such challenges in the tions benefit greatly from expertise in efits of this kind of heterogeneous, in innovation that produces 10-fold return on the prize as well as a 100-fold in- ically succeed in driving innovation. learning environment imparts some data science and machine intelligence team-based approach to address im- crease in follow‐on investment and social benefit. This departure from traditional re- unique skillsets that are invaluable as well as more humanities- focused portant humanitarian challenges is The XPRIZE Foundation recognized that this extraordinary leverage could be search models enables competitors for students as they transition to the disciplines. that many voices can be heard, allow- beneficial for a range of global grand challenges in which these inducement prizes to focus on a solution’s impact, rather modern workplace. To foster increasing democratiza- ing us to more knowledgably shape would lead to drastic and important improvements in quality of life. than the particulars of the methods tion and availability in AI technologies our future through technology. Active competitions include the $30 million Google Lunar XPRIZE, the $20 that are used to achieve it. Contribu- THE AI XPRIZE and allow individuals from many dis- Because this competition is, at its million NRG COSIA Carbon XPRIZE, the $15 million Global Learning XPRIZE, the $10 tors are inspired to explore all possible As the first “open” challenge in AI, the ciplines to participate in technological core, a large-scale collaboration em- million Qualcomm Tricorder XPRIZE, the $7 million Shell Ocean Discovery XPRIZE, and creative ways to meet that objec- IBM Watson AI XPRIZE is unlike its and social AI innovation, the competi- ploying interdisciplinary approaches the $7 million Barbara Bush Foundation Adult Literacy XPRIZE, and the $5 million tives and win. predecessors. Rather than set a single tion is structured to attract large, col- to problem solving, it’s a tremendous IBM Watson AI XPRIZE. Of course, prizes for innovation universal goal for all teams, this com- laborative teams. In fact, institutions learning opportunity—indeed, the and technological achievements aren’t petition invites teams to each create and organizations with computing first real benefit is learning. new: The British government’s sim - their own goal: an application of AI to resources, datasets, tools, and domain ple and practical method to precisely a meaningful problem. Once goals are expertise are encouraged to join the determine a ship’s longitude; Charles defined, the teams will use their own competition as supporting partners to its members to serve as mentors or to academic or corporate settings. Uni- Lindbergh won the Orteig Prize for best available resources, along with help teams with the development and provide other technical expertise to versity- and corporation-based re- the first transatlantic flight from New assistance from selected experts in the execution of their plans. Similarly, in- entrants in support of improving their search and development traditionally York to Paris; and, more recently, El- professional AI and engineering com- dividual engineers, researchers, and application or approach as well as in have incentivized research advances bert “Burt” Rutan’s team won the An- munities, to develop their plans for the specific domain experts (healthcare, showcasing their team’s progressive through promotion- or salary-based sari XPRIZE for developing the first duration of the competition. environment, education, art, policy, achievements. structures. XPRIZE funding and pub- reusable private spaceship. Each team must demonstrate prog- economics, and so on) can join teams The XPRIZE Foundation launched licity are untethered to any one cor- With tech-focused prizes, awards ress and annual milestone achieve- as advisors or hands-on support— its first competition in 1996 in an effort poration or institution, so they offer are tied to achievements versus at- ments for their plan, which will ulti- indeed, it’s this cross- pollination of AMIR BANIFATEMI is the lead to spur privately funded innovation global recognition and opportunities tempts, and teams are driven to lever- mately qualify them to compete for the knowledge, techniques, social con- for the XPRIZE Foundation’s IBM and research for space exploration for successful competitors. age any and all resources and support, grand prize on the TED 2020 stage in texts, and methodologies that will Watson AI XPRIZE. Contact him at (learn more about XPRIZE’s history in XPRIZE competitions draw sig- pulling together an impressive matrix front of a live in-person and online au- contribute to accelerating education [email protected]. the “XPRIZE Origin Story” sidebar). nificant numbers of nontraditional of expertise, capital, collaboration sup- dience. The audience and a panel of ex- and skills development as it drives With the success of this first compe- innovators and entrepreneurs and port, and achievement-oriented struc- pert judges will crown the grand-prize groups toward meeting their goals and JEAN-LUC GAUDIOT is a profes- tition, the foundation has attracted enable them to focus completely on ture. In addition, recent incentive- winner and rank the runners-up from winning the prize. sor of electrical engineering and continued private funding for addi- one area of innovation. With its goal driven competitions have shown that the three finalists. computer science at University tional XPRIZE competitions in health, to bring about “radical breakthroughs teams invest their own resources in As AI encompasses a wide range of California, Irvine. He is IEEE learning, energy, environment, global for the benefit of humanity” through these competitions to the tune of 10 to of disciplines, teams can utilize sev- he XPRIZE’s public visibility Computer Society’s president-elect. development and exploration. incentivized competition, the XPRIZE 100 times the prize itself. eral types of specialized technology will have a tremendous impact Contact him at [email protected]. These privately funded, prize- approach doesn’t impose budget re- Competitions can come in all sizes— in achieving their goals. Technologies on increasing our everyday based scientific challenges are highly quirements or other significant ad- from the smaller Kaggle Competitions that drive or support intelligent be- Tawareness of AI, particularly the ways attractive to individuals, compa- ministrative overhead. Researchers, (kaggle.com/competition) to the larger haviors also integrate sensory inputs in which it benefits society. Through nies, and organizations, and they’re engineers, and entrepreneurs are $1 million Netflix Prize—thus there’s and cognitive capabilities. These sys- this kind of highly publicized, global, Read your subscriptions clearly inspiring a solution-focused free from constraints that might typ- more awareness and acceptance of tems support human–machine collab- interdisciplinary, public/private com- through the myCS publications portal at approach to grand challenges that is ically limit them with government- or such incentive prizes than ever. Of oration through their ability to possess petition, the XPRIZE Foundation also http://mycs.computer.org less research oriented than traditional consortia- based funding sources. By course, scale and size matter, and domain understanding, learn from opens up a crucial forum for a variety

24 Computing Edge April 2017 78 COMPUTER WWW.COMPUTER.ORG/COMPUTER NOVEMBER 2016 79 COMPUTING EDUCATION

driving competitors to invest every multimillion-dollar competitions tend intellectual and financial resource at to have a greater impact on the market XPRIZE ORIGIN STORY their disposal to solve the problem, and innovation than smaller, more JOIN THE COMPETITION reach the goal, and win the prize, in- localized versions. However, their he history of XPRIZE is unique. The first was inspired by the $25,000 Orteig novation can take center stage. A large greater popularity has brought them evelopers, engineers, entrepreneurs, and innovators throughout the world TPrize, which was awarded in 1927 to American aviator Charles Lindbergh for number of groups can work in paral- increasingly into other domains—in- Dare encouraged to form or join an AI XPRIZE team. To read more or submit an flying the Spirit of St. Louis on the first nonstop flight between New York and Paris. lel toward the goal, and the resulting cluding the classroom. Educators have entry, visit ai.xprize.org. The original XPRIZE, announced in 1996, was a $10 million prize to the first solution or benefit can be quickly ad- been enthusiastic to explore chal- privately financed team that could build and fly a three‐passenger vehicle 100 opted and realized by society. lenge-based learning, appreciating km into space twice within two weeks. The prize, later titled the Ansari XPRIZE their potential for developing stu- for sub orbital spaceflight, motivated 26 teams from 7 nations to invest more INCENTIVE-DRIVEN dents’ abilities to work in teams, to be than $100 million in pursuit of the $10 million prize. On 4 October 2004, the prize COMPETITIONS SPUR inventive and resourceful, and to push experience, reason toward specific of stakeholders to help define the was awarded to Mojave Aerospace Ventures, kicking off the personal spaceflight INNOVATION beyond traditional lines of thought goals, and interact naturally with hu- guidelines and mechanisms by which revolution. Challenge-based models offering sub- and institutional boundaries. Indeed, man collaborators. Thus, AI applica- AI could or should be used. The ben- The first competition created a funding model to stimulate broad investment stantial financial prizes have histor- leveraging such challenges in the tions benefit greatly from expertise in efits of this kind of heterogeneous, in innovation that produces 10-fold return on the prize as well as a 100-fold in- ically succeed in driving innovation. learning environment imparts some data science and machine intelligence team-based approach to address im- crease in follow‐on investment and social benefit. This departure from traditional re- unique skillsets that are invaluable as well as more humanities- focused portant humanitarian challenges is The XPRIZE Foundation recognized that this extraordinary leverage could be search models enables competitors for students as they transition to the disciplines. that many voices can be heard, allow- beneficial for a range of global grand challenges in which these inducement prizes to focus on a solution’s impact, rather modern workplace. To foster increasing democratiza- ing us to more knowledgably shape would lead to drastic and important improvements in quality of life. than the particulars of the methods tion and availability in AI technologies our future through technology. Active competitions include the $30 million Google Lunar XPRIZE, the $20 that are used to achieve it. Contribu- THE AI XPRIZE and allow individuals from many dis- Because this competition is, at its million NRG COSIA Carbon XPRIZE, the $15 million Global Learning XPRIZE, the $10 tors are inspired to explore all possible As the first “open” challenge in AI, the ciplines to participate in technological core, a large-scale collaboration em- million Qualcomm Tricorder XPRIZE, the $7 million Shell Ocean Discovery XPRIZE, and creative ways to meet that objec- IBM Watson AI XPRIZE is unlike its and social AI innovation, the competi- ploying interdisciplinary approaches the $7 million Barbara Bush Foundation Adult Literacy XPRIZE, and the $5 million tives and win. predecessors. Rather than set a single tion is structured to attract large, col- to problem solving, it’s a tremendous IBM Watson AI XPRIZE. Of course, prizes for innovation universal goal for all teams, this com- laborative teams. In fact, institutions learning opportunity—indeed, the and technological achievements aren’t petition invites teams to each create and organizations with computing first real benefit is learning. new: The British government’s sim - their own goal: an application of AI to resources, datasets, tools, and domain ple and practical method to precisely a meaningful problem. Once goals are expertise are encouraged to join the determine a ship’s longitude; Charles defined, the teams will use their own competition as supporting partners to its members to serve as mentors or to academic or corporate settings. Uni- Lindbergh won the Orteig Prize for best available resources, along with help teams with the development and provide other technical expertise to versity- and corporation-based re- the first transatlantic flight from New assistance from selected experts in the execution of their plans. Similarly, in- entrants in support of improving their search and development traditionally York to Paris; and, more recently, El- professional AI and engineering com- dividual engineers, researchers, and This article originally appeared in application or approach as well as in have incentivized research advances bert “Burt” Rutan’s team won the An- munities, to develop their plans for the specific domain experts (healthcare, Computer, vol. 49, no. 11, 2016. showcasing their team’s progressive through promotion- or salary-based sari XPRIZE for developing the first duration of the competition. environment, education, art, policy, achievements. structures. XPRIZE funding and pub- reusable private spaceship. Each team must demonstrate prog- economics, and so on) can join teams The XPRIZE Foundation launched licity are untethered to any one cor- With tech-focused prizes, awards ress and annual milestone achieve- as advisors or hands-on support— its first competition in 1996 in an effort poration or institution, so they offer are tied to achievements versus at- ments for their plan, which will ulti- indeed, it’s this cross- pollination of AMIR BANIFATEMI is the lead to spur privately funded innovation global recognition and opportunities tempts, and teams are driven to lever- mately qualify them to compete for the knowledge, techniques, social con- for the XPRIZE Foundation’s IBM and research for space exploration for successful competitors. age any and all resources and support, grand prize on the TED 2020 stage in texts, and methodologies that will Watson AI XPRIZE. Contact him at (learn more about XPRIZE’s history in XPRIZE competitions draw sig- pulling together an impressive matrix front of a live in-person and online au- contribute to accelerating education [email protected]. the “XPRIZE Origin Story” sidebar). nificant numbers of nontraditional of expertise, capital, collaboration sup- dience. The audience and a panel of ex- and skills development as it drives With the success of this first compe- innovators and entrepreneurs and port, and achievement-oriented struc- pert judges will crown the grand-prize groups toward meeting their goals and JEAN-LUC GAUDIOT is a profes- tition, the foundation has attracted enable them to focus completely on ture. In addition, recent incentive- winner and rank the runners-up from winning the prize. sor of electrical engineering and continued private funding for addi- one area of innovation. With its goal driven competitions have shown that the three finalists. computer science at University tional XPRIZE competitions in health, to bring about “radical breakthroughs teams invest their own resources in As AI encompasses a wide range of California, Irvine. He is IEEE learning, energy, environment, global for the benefit of humanity” through these competitions to the tune of 10 to of disciplines, teams can utilize sev- he XPRIZE’s public visibility Computer Society’s president-elect. development and exploration. incentivized competition, the XPRIZE 100 times the prize itself. eral types of specialized technology will have a tremendous impact Contact him at [email protected]. These privately funded, prize- approach doesn’t impose budget re- Competitions can come in all sizes— in achieving their goals. Technologies on increasing our everyday based scientific challenges are highly quirements or other significant ad- from the smaller Kaggle Competitions that drive or support intelligent be- Tawareness of AI, particularly the ways attractive to individuals, compa- ministrative overhead. Researchers, (kaggle.com/competition) to the larger haviors also integrate sensory inputs in which it benefits society. Through nies, and organizations, and they’re engineers, and entrepreneurs are $1 million Netflix Prize—thus there’s and cognitive capabilities. These sys- this kind of highly publicized, global, Read your subscriptions clearly inspiring a solution-focused free from constraints that might typ- more awareness and acceptance of tems support human–machine collab- interdisciplinary, public/private com- through the myCS publications portal at approach to grand challenges that is ically limit them with government- or such incentive prizes than ever. Of oration through their ability to possess petition, the XPRIZE Foundation also http://mycs.computer.org less research oriented than traditional consortia- based funding sources. By course, scale and size matter, and domain understanding, learn from opens up a crucial forum for a variety

www.computer.org/computingedge 25 78 COMPUTER WWW.COMPUTER.ORG/COMPUTER NOVEMBER 2016 79 Data analytics EDT itI OOR:R: Ssetheth Earley, Earley Iinformationnformation Ssccience,ience, [email protected]

There Is No AI Without IA

Seth Earley, Earley Information Science

rtificial intelligence (AI) ence in 2015,4 uses deep learning customers. These initiatives in- is increasingly hyped by and neural nets. DigitalGenius first clude improving personalization vendors of all shapes and classifies the incoming questions of the user experience by present- sizes—from well-funded into one of several categories for ing more relevant content; tuning Astartups to the well-known software further processing: product infor- search results to return exactly brands. Financial organizations are mation, account information, ac- what the user is interested in; and building AI-driven investment advi- tion request, comparison question, improving the effectiveness of of- sors.1 Chat bots provide everything recommendation question, and so fers and promotions. Organiza- from customer service2 to sales as- on. These classifications qualify tions might also strive to increase sistance.3 Although AI is receiv- as a foundational element of infor- response rates to email communi- ing a lot of visibility, the fact that mation architecture. The starting cations; provide better customer these technologies all require some point is contextualizing the query, self-service; increase participation element of knowledge engineer- which is then passed to other mod- in user communities and other ing, information architecture, and ules, including product informa- social media venues; and gener- high-quality data sources is not tion systems and other databases ally enhance the product experi- well known. Many vendors sidestep and APIs. Each of these systems ence through various other online this question or claim that their al- and sources in turn needs to be mechanisms. In each of these gorithms operate on unstructured well architected to return the cor- cases, the means of engagement is information sources, “understand” rect information. If the informa- providing a relevant piece of data those sources, interpret the user tion is not structured and curated or content (in the form of promo- query, and present the result within some way, there is nothing for tions, offers, sales, next-best ac- out predefined architectures or hu- the system to return. The ability of tion, products for cross sell and man intervention. That very well DigitalGenius to use AI to engage upsell, answers to questions, and might be true in certain circum- customers is predicated on having so on) at exactly the right time and stances, but most applications re- high-quality, structured data. in the context that is most mean- quire a significant amount of hard ingful and valuable to the user. work on the part of humans before Digital Engagement: This is done by interpreting the neural nets, machine learning, and The Right Information various signals that users provide natural language processors can in Context through their current and past work their magic. Organizations are in a never- interactions with the organiza- DigitalGenius, a product that ending cycle of improving their tion. These signals include prior received press coverage at a confer- digital means of engaging with purchases, real-time click-stream

26 April 2017 Published by the IEEE Computer Society 2469-7087/17/$33.00 © 2017 IEEE 58 IT Pro May/June 2016 Published by the IEEE Computer Society 1520-9202/16/$33.00 © 2016 IEEE Data analytics EDitITOR: Ssetheth Earley, Earley Iinformationnformation Sscience, [email protected]

data, support center interactions, program is. A corpus of informa- ent the right content for the user’s consumed content, preferences, tion contains the answers that the context, and requires not only buying characteristics, demo- program is attempting to process that our customer data is clean, graphics, firmographics, social and interpret. Structuring that in- properly structured, and inte- media information, and any other formation for retrieval is referred grated across multiple systems “electronic body language” that is to as knowledge engineering and the and processes but also that the captured by marketing automa- structures are called knowledge system understand the relation- tion and integration technolo- representation. ship between the user, his or her gies. A search query could return specific task, the product, and the a different result for a technical Ontologies as Knowledge content needed—all assembled There Is No AI user than a nontechnical user, for Representations dynamically in real time. Building example. At its core, search is a Knowledge representation con- these structures and relationships recommendation engine. The sig- sists of taxonomies, controlled and harmonizing the architec- Without IA nal is the search phrase, and the vocabularies, thesaurus structure across the various back-end recommendation is the result set. tures, and all of the relationships platforms and front-end systems The more that is known about the between terms and concepts. results in an enterprise ontology user, the more the recommenda- These elements collectively make that enables a personalized, om- tion can be tailored. If the recom- up the ontology. An ontology rep- nichannel experience. Some might Seth Earley, Earley Information Science mendation is related to a product, resents a domain of knowledge call this an enterprise informa- then clean, well-structured prod- and the information architecture tion architecture; however, there uct data is a precondition. structures and mechanisms for is more to it than the data struc- accessing and retrieving answers tures. Recall that the definition of rtificial intelligence (AI) ence in 2015,4 uses deep learning customers. These initiatives in- Personalization, in specific contexts. Ontologies an ontology includes real-world is increasingly hyped by and neural nets. DigitalGenius first clude improving personalization User Signals, and vendors of all shapes and classifies the incoming questions of the user experience by present- Recommendations sizes—from well-funded into one of several categories for ing more relevant content; tuning Getting these recommendations Every AI program interfaces with information, and Astartups to the well-known software further processing: product infor- search results to return exactly right and truly personalizing the the better that information is structured, the more brands. Financial organizations are mation, account information, ac- what the user is interested in; and user experience requires that building AI-driven investment advi- tion request, comparison question, improving the effectiveness of of- product data be correctly struc- effective the program is. sors.1 Chat bots provide everything recommendation question, and so fers and promotions. Organiza- tured and organized, that con- from customer service2 to sales as- on. These classifications qualify tions might also strive to increase tent processes be integrated into sistance.3 Although AI is receiv- as a foundational element of infor- response rates to email communi- product onboarding, and that can also capture “common sense” logic and relationships. The ontol- ing a lot of visibility, the fact that mation architecture. The starting cations; provide better customer associations can be made among knowledge of objects, processes, ogy can contain knowledge about these technologies all require some point is contextualizing the query, self-service; increase participation products, content, and user intent materials, actions, events, and processes, customer needs, and element of knowledge engineer- which is then passed to other mod- in user communities and other signals. The relationship of prod- myriad more classes of real-world content relationships.5 ing, information architecture, and ules, including product informa- social media venues; and gener- ucts to content is based on knowl- logical relationships. In this way, high-quality data sources is not tion systems and other databases ally enhance the product experi- edge of the user’s task and what an ontology forms a foundation Mining Content for well known. Many vendors sidestep and APIs. Each of these systems ence through various other online content would help him or her for computer reasoning even if the Product Relationships this question or claim that their al- and sources in turn needs to be mechanisms. In each of these accomplish it. The task might be answer to a question is not explic- Consumer and industrial prod- gorithms operate on unstructured well architected to return the cor- cases, the means of engagement is a review, a how-to, product speci- itly contained in the corpus. The ucts need to be associated with information sources, “understand” rect information. If the informa- providing a relevant piece of data fications, reference materials, in- answer can be inferred from the content and user context, but those sources, interpret the user tion is not structured and curated or content (in the form of promo- structions, diagrams and images, facts, terms, and relationships in content can also be mined to sug- query, and present the result within some way, there is nothing for tions, offers, sales, next-best ac- or other content that helps the the ontology. In a practical sense, gest products for a user context. out predefined architectures or hu- the system to return. The ability of tion, products for cross sell and user decide on the purchase. this can make the system more For an industrial application, a man intervention. That very well DigitalGenius to use AI to engage upsell, answers to questions, and AI encompasses a class of ap- user friendly and forgiving when user might need parts and tools might be true in certain circum- customers is predicated on having so on) at exactly the right time and plication that allows for easier the user makes requests using to complete maintenance on a stances, but most applications re- high-quality, structured data. in the context that is most mean- interaction with computers and phrase variations, and more capa- hydraulic system. Using adap- quire a significant amount of hard ingful and valuable to the user. also allows computers to take on ble when it encounters use cases tive pattern-recognition software work on the part of humans before Digital Engagement: This is done by interpreting the more of the types of problems that were not completely defined to mine reference manuals about neural nets, machine learning, and The Right Information various signals that users provide that were typically in the realm of when it was developed. In effect, hydraulic systems and repair, the natural language processors can in Context through their current and past human cognition. Every AI pro- the system can “reason” and make system can extract a list of needed work their magic. Organizations are in a never- interactions with the organiza- gram interfaces with information, logical deductions. tools and related parts. A search DigitalGenius, a product that ending cycle of improving their tion. These signals include prior and the better that information is Correctly interpreting user sig- on hydraulic repair will present received press coverage at a confer- digital means of engaging with purchases, real-time click-stream structured, the more effective the nals enables the system to pres- a dynamically generated product

page based on the product rela- first car phone in the ’80s, in ad- In many cases, the data being tionships and correlated with the dition to costing several thousand processed or corpus being ana- company’s offerings. To some dollars, left little room for any- lyzed by AI systems is typically information professionals, this thing else in the spacious trunk). less structured than better-orga- might sound complex and cum- Most AI is quietly taken for grant- nized sources such as financial bersome to implement; however, ed today. The word processor I am and transactional data. Learn- there are emerging approaches using was once considered an ad- ing algorithms can be used to that bring these aspirations closer vanced AI application! both extract meaning from am- to reality. biguous queries and attempt to Simplicity Is Hidden make sense of unstructured data If It Works, It’s Not AI Complexity inputs. Humans might phrase The concept of what constitutes Under the covers, AI is complex; questions using different termi- AI has evolved as technology has however, this complexity is hid- nology, or they might ask overly evolved. A colleague of mine has den from the user and is, in fact, broad questions. They are not said, “It’s artificial intelligence the enabler of an easy, intuitive always clear about their objec- until you know how it works.” An experience. It is not magic, and it tives—they don’t necessarily interesting perspective indeed. I requires foundational structures know what they are looking for. found support for this in mate- that can be reused across many This is why human sales people rial from an MIT AI course (see different processes, depart- typically engage prospects in http://bit.ly/1WTCUXK): ments, and applications. These conversations about their overall structures are generally first de- needs, rather than asking them There’s another part of AI […] veloped in silos and standalone what they are looking for. (At that’s fundamentally about appli- tools; however, the true power least, the good ones do.) cations. Some of these applications will be realized when consid- Inserting AI into the process is you might not want to call “intel- ered in a holistic framework of more effective when users know ligent” […] For instance, compilers machine-intelligence-enabled what they want and can clearly used to be considered AI, because infrastructure. AI will change articulate it, and where there is […] statements [were in a] high- the business landscape but will a relatively straightforward an- level language; and how could a require investments in product swer. The algorithm does what computer possibly understand? and content architecture, cus- it does best in terms of handling [The] work to make a computer un- tomer data, and analytics, and variations in how those ques- derstand […] was taken to be AI. the harmonization of tools in tions are asked, interpreting the Now […] we understand compilers, the customer engagement eco- meaning of the question, and and there’s a theory of how to build system. The organizations that processing other unstructured compilers […] well, it’s not AI any- adopt these approaches will gain signals that help further con- more. […] When they finally get significant advantages over the textualize the user’s intent. something working, it gets co-opted competition. There are many flavors of AI by some other part of the field. So, and many classes of algorithm by definition, no AI ever works; if it Clean Data Is the Price that comprise AI systems. How- works, it’s not AI. of Admission ever, even when AI systems are AI approaches are proposed as the used to find structure from Seemingly intractable problems answer to enterprise challenges completely unstructured in- have been solved by advances in around improving customer en- formation, they still require processing power and capabilities. gagement by dealing with infor- structure at the data layer. Not long ago, autonomous vehicles mation overload. Before those Given that the data being were considered technologically approaches can be leveraged, searched by the AI system is infeasible due to the volume of however, organizations need to unstructured, why do we need data that needed to be processed in possess the data needed as inputs information architecture? Un- real time. Speech recognition was to machine learning algorithms structured information is typi- unreliable and required extensive that can in turn process these di- cally in the form of text from pages, speaker-dependent training ses- verse signals from unstructured documents, comments, surveys, sions. Mobile phones were once and structured sources. Clean, social media, or some other source. “auto-mobile” phones, requiring well-structured, managed data is Though it is unstructured, there a car trunk full of equipment (my assumed. are still parameters associated with

28 Computing Edge April 2017 60 IT Pro May/June 2016 Data analytics Table 1. Example applications for AI technology. Application/problem Typical approach Why it will not work How AI helps page based on the product rela- first car phone in the ’80s, in ad- In many cases, the data being tionships and correlated with the dition to costing several thousand processed or corpus being ana- Omnichannel selling Integrations across systems Content is typically out Harvesting signals of user intent that are inflexible of synch, and systems across channels and devices; company’s offerings. To some dollars, left little room for any- lyzed by AI systems is typically have brittle integrations. intelligent integration across information professionals, this thing else in the spacious trunk). less structured than better-orga- diverse classes of customer- might sound complex and cum- Most AI is quietly taken for grant- nized sources such as financial engagement technologies. bersome to implement; however, ed today. The word processor I am and transactional data. Learn- Personalization Classification of users Assumptions are Incorporating multiple data there are emerging approaches using was once considered an ad- ing algorithms can be used to and recommendation according to personas rudimentary and simplistic, sources, ontologies, and reasoning that bring these aspirations closer vanced AI application! both extract meaning from am- engines and presentation of limit user choice, and rarely algorithms. to reality. biguous queries and attempt to content based on use cases anticipate needs. Simplicity Is Hidden make sense of unstructured data Dynamic content Limited assembly based on The number of permutations Assembling content that meets If It Works, It’s Not AI Complexity inputs. Humans might phrase assembly curated content cannot be tracked, managed, a user’s preferences while or understood. incorporating real-time data, such The concept of what constitutes Under the covers, AI is complex; questions using different termi- as news and market performance AI has evolved as technology has however, this complexity is hid- nology, or they might ask overly supported by knowledge evolved. A colleague of mine has den from the user and is, in fact, broad questions. They are not embedded in ontologies. said, “It’s artificial intelligence the enabler of an easy, intuitive always clear about their objec- Field service View of maintenance The volume of information Processing performance data to until you know how it works.” An experience. It is not magic, and it tives—they don’t necessarily equipment maintenance schedules and analysis from sensors and hidden predict or prevent maintenance interesting perspective indeed. I requires foundational structures know what they are looking for. of field reports. signals from degrading events. found support for this in mate- that can be reused across many This is why human sales people performance is not perceivable by humans. rial from an MIT AI course (see different processes, depart- typically engage prospects in http://bit.ly/1WTCUXK): ments, and applications. These conversations about their overall Customer self-service Knowledge bases There are insufficient Enabling intelligent agent and FAQs considerations of user context, knowledge base access using structures are generally first de- needs, rather than asking them language variations, fast- multiple variables to provide There’s another part of AI […] veloped in silos and standalone what they are looking for. (At changing content, and a high specific answers with metrics- that’s fundamentally about appli- tools; however, the true power least, the good ones do.) volume of possible use cases. driven feedback loops. cations. Some of these applications will be realized when consid- Inserting AI into the process is Call center support Reps trained through For technical and specialized Enabling intelligent agent access you might not want to call “intel- ered in a holistic framework of more effective when users know on-the-job experience, knowledge, the level of integrating answers and content ligent” […] For instance, compilers machine-intelligence-enabled what they want and can clearly knowledge bases experience needed is too from multiple knowledge and data used to be considered AI, because infrastructure. AI will change articulate it, and where there is costly. sources using natural language interfaces. […] statements [were in a] high- the business landscape but will a relatively straightforward an- level language; and how could a require investments in product swer. The algorithm does what Product and tool Hand-built landing pages Maintenance of hand-curated Harvesting product relationships presentation and assembled landing through knowledge acquisition computer possibly understand? and content architecture, cus- it does best in terms of handling pages becomes costly and and mining of reference materials [The] work to make a computer un- tomer data, and analytics, and variations in how those ques- unmanageable. based on maintenance tasks and derstand […] was taken to be AI. the harmonization of tools in tions are asked, interpreting the repair use cases. Now […] we understand compilers, the customer engagement eco- meaning of the question, and and there’s a theory of how to build system. The organizations that processing other unstructured compilers […] well, it’s not AI any- adopt these approaches will gain signals that help further con- the source and context. Social identified, but the input needs to Some vendors might debate the more. […] When they finally get significant advantages over the textualize the user’s intent. media information requires vari- have structure. value of this approach, insisting something working, it gets co-opted competition. There are many flavors of AI ous parameters to describe users, A common fallacy in consider- that their algorithms can handle by some other part of the field. So, and many classes of algorithm their posts, relationships, time and ing big data sources that form in- whatever you throw at them— by definition, no AI ever works; if it Clean Data Is the Price that comprise AI systems. How- location of posts, links, hashtags, puts for machine learning is that however, I would argue that this works, it’s not AI. of Admission ever, even when AI systems are and so on. The information ar- because the data is “schema-less” is only the case when the ontolo- AI approaches are proposed as the used to find structure from chitecture question in this case (does not have a predefined struc- gies are self-contained in the tool. Seemingly intractable problems answer to enterprise challenges completely unstructured in- characterizes the structures of the ture), no structure is required. Even so, there will always be gaps have been solved by advances in around improving customer en- formation, they still require input data, so that the system can Data still requires attribute defini- between what a tool developed for processing power and capabilities. gagement by dealing with infor- structure at the data layer. be programmed to find patterns tions, normalization, and cleans- broad adoption can contain and Not long ago, autonomous vehicles mation overload. Before those Given that the data being of interest. Even in the case of un- ing to apply machine learning and the specialized needs of an enter- were considered technologically approaches can be leveraged, searched by the AI system is supervised machine learning pattern-identification algorithms.6 prise. Even if the tool is developed infeasible due to the volume of however, organizations need to unstructured, why do we need (a class of application that derives As enterprises embark on the path for a specific industry, the differ- data that needed to be processed in possess the data needed as inputs information architecture? Un- signals from data that are not to machine learning and AI, they ences in processes found from or- real time. Speech recognition was to machine learning algorithms structured information is typi- predefined by a human), the pro- should, first and foremost, be de- ganization to organization require unreliable and required extensive that can in turn process these di- cally in the form of text from pages, grammer still needs to describe veloping an enterprise ontology specialized vocabularies and con- speaker-dependent training ses- verse signals from unstructured documents, comments, surveys, the data in the first place with at- that represents all of the knowl- textual knowledge relationships. sions. Mobile phones were once and structured sources. Clean, social media, or some other source. tributes and values. There might edge that any AI system they This is a significant undertaking; “auto-mobile” phones, requiring well-structured, managed data is Though it is unstructured, there not be predefined categories for deploy would process, analyze, le- however, not doing so misses an a car trunk full of equipment (my assumed. are still parameters associated with outliers and patterns that are verage, or require. essential step in the process.

www.computer.org/computingedge 29 60 IT Pro May/June 2016 computer.org/ITPro 61 Data analytics

Table 2. Examples of AI tools with representative applications, limitations, considerations, and data sources.

Class of tool Best suited for Limitations Decision factors Example data I nference engine Deriving product and data relationships from S ignificant effort required in certain cases for Volume of content, quality of corpus, structure of data within Structured documents such as catalogs, engineering unstructured content development of adaptive pattern-recognition content (use of tables, headers, other identifiers, and delimiters) specifications, knowledge bases, ERP data algorithms I ntelligent agent Highly selective search and information retrieval Breadth of possible use cases, variability of Narrow scenarios, defined repeatable Policy and procedure manuals, chat logs, customer for definable processes queries and questions, degree of manual tasks, sophisticated audience, standard terminology for domain service databases, support tickets, knowledge harvested term-relationship mapping from human subject matter experts A uto-classifier L arge volumes of higher-quality content where Rules bases can become very complex, High-value content that has a predictable structure, Semi-structured content such as regulations, journal training sets or clear rules can be applied high volumes of training content required, curated with editorial standards articles, news articles, policies and procedures, support variability of content documentation, engineering documents Entity extractors Predictable formats for data (Social Security A mbiguity of entity values (for “Washington,” Security, compliance, and reference-based applications (looking Same semi-structured content sources mentioned numbers, addresses, names, phone numbers, can be Washington State, George Washington, for personally identifiable information, sensitive account above, along with controlled vocabularies and account numbers) Washington, DC, and so on), variations in information, intellectual property protection, fact identification) thesaurus structures to identify entity values format, quality of content Unsupervised Pattern detection, recognition, and prediction; Depends on creation of foundational hypothesis Determination of type of patterns that are being detected; not Unsupervised learning can be applied to virtually any machine learning anomaly detection, outliers, hidden attributes, to define type of outlier or pattern (though typically selected in isolation because learning algorithms are data source as the tools look for patterns; can be applied and relationships; discovering new patterns or not necessarily details of pattern); selection embedded in many aspects of AI application; requires guidance of to unstructured and semi-structured content as well as segmenting audiences, content, or data into of class of algorithm requires technical data science specialist knowledgeable in specific type of algorithm transactional data, images, audio and video, scientific clusters and groupings sophistication and iterative testing across datasets, sensor data, social media sources, and so on many classes of unsupervised learning from data signals to content, images, or transactions S upervised Pattern detection based on training data and Requires training sets and example data that Similar challenges to unsupervised learning: typically embedded Same as unsupervised; the difference is having training machine learning used for well-understood patterns; discovering teach the algorithm what type of information in AI applications rather than being applied as standalone tool sets that contain patterns sought data, content, and relationships based on is being sought; needs large enough samples and requires specialized knowledge; can be used to find types examples; finding similar documents, purchase to test various hypotheses and an understanding of images, content or data, purchase patterns, audiences, users, patterns, audiences with like characteristics; of the specific outcome sought; risk of finding combinations of products, or other patterns depending predicting outcomes patterns where none exist (over fitting) on data sources Hybrid unsupervised Finding hidden patterns (unsupervised); using Similar in concept to creating a survey with Most AI programs contain hybrid implementations of supervised Same data used for unsupervised learning can be used and supervised those patterns to train an algorithm to locate open-ended questions, then taking the answers and unsupervised and therefore the complexity of each approach as input for supervised learning after representative learning more instances of the data or content to the survey and using those results to form can be compounded; requires research and experimentation to patterns have been identified closed-ended surveys find correct combinations and fine-tune algorithms

Much of what is described as AI that will allow for continuous AI approach will require a greater is an extension of well-known ap- improvement of AI and cognitive level of investment, sponsorship at proaches to addressing informa- computing systems. the executive level, program-level tion management problems, all of governance, and an enterprise which require clean foundational Identifying Use Cases span of influence. It will also re- data and information structures as Differentiating AI use cases from quire a longer-term commitment a starting point. The difference be- standard information manage- than a typical information man- tween standard information man- ment use cases requires consid- agement project. Although there agement and practical AI lies in ering the sources of data that are opportunities to deploy limit- understanding the limits of these comprise the “signals” being pro- ed scope AI, fully leveraging it as a technologies and where they can cessed, the type of task the user transformative class of technology best be applied to address enter- faces, and the systems that will be should be part of an overall digital prise challenges. part of the solution. The differ- transformation strategy in some The remainder of this article ence in approaches to these prob- cases on the scale of enterprise re- describes how your organization lems lies in how the sources of source planning (ERP) programs, can identify use cases that will data will be curated and ingested, with commensurate support, benefit from AI, identifies data how organizing principles need funding, and commitment. (Some sources that offer reliable and to be derived and applied, the ERP programs can cost US$50 to meaningful insights to train and sophistication of the functional- $100 million or more). Although guide AI, and defines governance, ity desired, and the limitations of no organization will approach an curation, and scalable processes the current solution in place. An unproven set of technologies with

30 Computing Edge April 2017 62 IT Pro May/June 2016 Data analytics

Table 2. Examples of AI tools with representative applications, limitations, considerations, and data sources.

Class of tool Best suited for Limitations Decision factors Example data InferenceInference engine Deriving product and data relationships from SignificantSignificant effort required in certain cases for Volume of content, quality of corpus, structure of data within StructuredStructured documents such as catalogs, engineering unstructured content development of adaptive pattern-recognition content (use of tables, headers, other identifiers, and delimiters) specifications, knowledge bases, ERP data algorithms IntelligentIntelligent agent Highly selective search and information retrieval Breadth of possible use cases, variability of NarrowNarrow scenarios, defined repeatable Policy and procedure manuals, chat logs, customer for definable processes queries and questions, degree of manual tasks, sophisticated audience, standard terminology for domain service databases, support tickets, knowledge harvested term-relationship mapping from human subject matter experts Auto-classifierAuto-classifier LargeLarge volumes of higher-quality content where Rules bases can become very complex, High-value content that has a predictable structure, Semi-structuredSemi-structured content such as regulations, journal training sets or clear rules can be applied high volumes of training content required, curated with editorial standards articles, news articles, policies and procedures, support variability of content documentation, engineering documents Entity extractors Predictable formats for data (Social(Social SecuritySecurity AmbiguityAmbiguity of entity values (for “Washington,” Security,Security, compliance, and reference-based applications (looking SameSame semi-structured content sources mentioned numbers, addresses, names, phone numbers, can be Washington State,State, George Washington, for personally identifiable information, sensitive account above, along with controlled vocabularies and account numbers) Washington, DC,DC, and so on), variations in information, intellectual property protection, fact identification) thesaurus structures to identify entity values format, quality of content Unsupervised Pattern detection, recognition, and prediction; Depends on creation of foundational hypothesis Determination of type of patterns that are being detected; not Unsupervised learning can be applied to virtually any machine learning anomaly detection, outliers, hidden attributes, to define type of outlier or pattern (though typically selected in isolation because learning algorithms are data source as the tools look for patterns; can be applied and relationships; discovering new patterns or not necessarily details of pattern); selection embedded in many aspects of AI application; requires guidance of to unstructured and semi-structured content as well as segmenting audiences, content, or data into of class of algorithm requires technical data science specialist knowledgeable in specific type of algorithm transactional data, images, audio and video, scientific clusters and groupings sophistication and iterative testing across datasets, sensor data, social media sources, and so on many classes of unsupervised learning from data signals to content, images, or transactions SupervisedSupervised Pattern detection based on training data and Requires training sets and example data that SimilarSimilar challenges to unsupervised learning: typically embedded SameSame as unsupervised; the difference is having training machine learning used for well-understood patterns; discovering teach the algorithm what type of information in AI applications rather than being applied as standalone tool sets that contain patterns sought data, content, and relationships based on is being sought; needs large enough samples and requires specialized knowledge; can be used to find types examples; finding similar documents, purchase to test various hypotheses and an understanding of images, content or data, purchase patterns, audiences, users, patterns, audiences with like characteristics; of the specific outcome sought; risk of finding combinations of products, or other patterns depending predicting outcomes patterns where none exist (over fitting) on data sources Hybrid unsupervised Finding hidden patterns (unsupervised); using SimilarSimilar in concept to creating a survey with Most AI programs contain hybrid implementations of supervised SameSame data used for unsupervised learning can be used and supervised those patterns to train an algorithm to locate open-ended questions, then taking the answers and unsupervised and therefore the complexity of each approach as input for supervised learning after representative learning more instances of the data or content to the survey and using those results to form can be compounded; requires research and experimentation to patterns have been identified closed-ended surveys find correct combinations and fine-tune algorithms

Much of what is described as AI that will allow for continuous AI approach will require a greater that level of commitment, fund- organization’s knowledge landscape user problems. Customer account is an extension of well-known ap- improvement of AI and cognitive level of investment, sponsorship at ing needs to be allocated to extend and implementation of new gov- data and purchase history can be proaches to addressing informa- computing systems. the executive level, program-level proven approaches with emerging ernance, metrics, and data quality processed to look for similarities in tion management problems, all of governance, and an enterprise AI technologies. programs—governance to make buyers and predict responses to of- which require clean foundational Identifying Use Cases span of influence. It will also re- The roadmap for AI transforma- decisions, metrics to monitor the fers; email response metrics can be data and information structures as Differentiating AI use cases from quire a longer-term commitment tion includes continuous evaluation effectiveness of those decisions, and processed with text content of offers a starting point. The difference be- standard information manage- than a typical information man- of payback and ROI and focuses data quality to fuel the AI engine. to surface buyer segments. Product tween standard information man- ment use cases requires consid- agement project. Although there on short-term wins while pursuing Table 1 presents example appli- catalogs and data sheets are sources agement and practical AI lies in ering the sources of data that are opportunities to deploy limit- longer-term goals. Most organiza- cations for AI technology. of attributes and attribute values. understanding the limits of these comprise the “signals” being pro- ed scope AI, fully leveraging it as a tions are attempting to solve the Public references can be used for technologies and where they can cessed, the type of task the user transformative class of technology problems described in Table 1 with Identifying Data Sources procedures, tool lists, and product best be applied to address enter- faces, and the systems that will be should be part of an overall digital limited approaches, departmental- Training data can come from typi- associations. YouTube video con- prise challenges. part of the solution. The differ- transformation strategy in some level solu tions, standalone tools, cal knowledge bases, the more tent audio tracks can be converted The remainder of this article ence in approaches to these prob- cases on the scale of enterprise re- and insufficient funding. These highly curated the better. Call cen- to text and mined for product as- describes how your organization lems lies in how the sources of source planning (ERP) programs, problem classes are facing most enter recordings and chat logs can sociations. User website behaviors can identify use cases that will data will be curated and ingested, with commensurate support, terprises, and though progress can be mined for content and data re- can be correlated with offers and benefit from AI, identifies data how organizing principles need funding, and commitment. (Some be made with limited resources and lationships as well as answers to dynamic content. Sentiment analy- sources that offer reliable and to be derived and applied, the ERP programs can cost US$50 to siloed approaches, this would be an questions. Streaming sensor data sis, user-generated content, social meaningful insights to train and sophistication of the functional- $100 million or more). Although extension of business as usual. Tru- can be correlated with historical graph data, and other external data guide AI, and defines governance, ity desired, and the limitations of no organization will approach an ly transformative applications will maintenance records, and search sources can all be mined and re- curation, and scalable processes the current solution in place. An unproven set of technologies with require an enterprise view of the logs can be mined for use cases and combined to yield knowledge and

www.computer.org/computingedge 31 62 IT Pro May/June 2016 computer.org/ITPro 63 Data analytics

user-intent signals. The correct data will occur in subtle ways—such 17 Nov. 2015; http://blogs.wsj.com/ sources depend on the application, as improved usability of applica- digits/2015/11/17/can-artificial use cases, and objectives. tions and findability of informa- -intelligence-sell-shoes/. Table 2 describes examples of tion. These will not necessarily 4. R. Miller, “DigitalGenius Brings AI tools with representative ap- appear on the surface to be AI. Artificial Intelligence to Customer plications, limitations, consider- Over time, AI-driven intelligent Service via SMS,” Tech Crunch, 5 ations, and data sources. Though virtual assistants will become May 2015; http://techcrunch.com/ not meant to be an exhaustive more fluent and capable, and will 2015/05/05/digitalgenius-brings list, and recognizing that one become the preferred mechanism -artificial-intelligence-to-customer class of tool is frequently lever- for interacting with technology. -service-via-sms/. aged in other tools and applica- Humans create knowledge, while 5. S. Earley, “Lessons from Alexa: tions (an intelligent agent can use machines process, store, and act Artificial Intelligence and Machine inference engines, which in turn on that knowledge. AI is applied Learning Use Cases,” blog, Earley can leverage learning algorithms, human knowledge. Organizations Information Science, 24 Mar. 2016; for example), the table articulates need to develop the foundations www.earley.com/blog/lessons considerations for exploring one for advancing AI by capturing -alexa-artificial-intelligence-and approach versus another. and curating that knowledge and -machine-learning-use-cases. by building the foundational data 6. J. Brownlee, “How to Prepare Data Defining Governance, structures that form the scaffold- for Machine Learning,” Machine Curation, and Scalable ing for that knowledge. Without Learning Mastery, 25 Dec. 2013; Processes those components, the algorithms http://machinelearningmastery.com/ AI and cognitive computing are have nothing to run on. how-to-prepare-data-for-machine managed in the same way as many -learning/. other information and technology References governance programs. They require 1. J. Vögeli, “UBS Turns to Artificial Seth Earley is CEO of Earley Informa- executive sponsorship, charters, Intelligence to Advise Clients,” tion Science (www.earley.com). He’s an roles and responsibilities, decision- Bloomberg, 7 Dec. 2014; www. expert in knowledge processes, enterprise making protocols, escalation pro- bloomberg.com/news/articles/ data architecture, and customer experi- cesses, defined agendas, and linkage 2014-12-07/ubs-turns-to-artificial ence management strategies. His interests to specific business objectives and -intelligence-to-advise-wealthy include customer experience analytics, processes. These initiatives are a -clients. knowledge management, structured and subset of digital transformation and 2. C. Green, “Is Artificial Intelligence unstructured data systems and strategy, are linked to customer life cycles and the Future of Customer Service?” and machine learning. Contact him at internal value chains. Because the MyCustomer, 3 Dec. 2015; www. [email protected]. objective is always to affect a pro- mycustomer.com/service/channels/ cess outcome, all AI and cogni- is-artificial-intelligence-the-future tive computing programs are closely -of-customer-service. Selected CS articles and aligned with ongoing metrics at mul- 3. E. Dwoskin, “Can Artificial Intelli- Thiscolumns article originallyare available appeared for free inat tiple levels of detail—from content gence Sell Shoes?” blog, Wall Street J., http://ComputingNow.computer.org.IT Professional, vol. 18, no. 3, 2016. and data quality to process effectiveness and satisfaction of business im- peratives—and ultimately are linked IT Professional (ISSN 1520-9202) is published bimonthly by the IEEE Computer Society. IEEE to the organizational competitive Headquarters, Three Park Avenue, 17th Floor, New York, NY 10016-5997; IEEE Computer So- ciety Publications Office, 10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720- and market strategy. Milestones and 1314; voice +714 821 8380; fax +714 821 4010; IEEE Computer Society Headquarters, 1828 stages are defined to release fund- L St. NW, Suite 1202, Washington, DC 20036. Annual subscription: $49 in addition to any IEEE Computer Society dues. Nonmember rates are available on request. Back issues: $20 for ing for program phases, each with members, $143 for nonmembers. defined success criteria and measur- Postmaster: Send undelivered copies and address changes to IT Professional, Member- ship Processing Dept.,Technology IEEE Service SolutionsCenter, 445 Hoes for Ltheane, EnterprisePiscataway, NJ 08854-4141. able outcomes. Periodicals Postage Paid at New York, NY, and at additional mailing offices. Canadian GST #125634188. Canada Post Publications Mail Agreement Number 40013885. Return undeliverable Canadian addresses to PO Box 122, Niagara Falls, ON L2E 6S8, Canada. Printed in the USA. I will no doubt continue Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IT Professional does not to impact every aspect of necessarily constitute endorsement by the IEEE or the Computerwww.computer.org/itpro Society. All A our personal and profes- submissions are subject to editing for style, clarity, and space. sional lives. Much of this impact

32 Computing Edge April 2017 64 IT Pro May/June 2016 Backspace Unreliable Machine Learning

Vinton G. Cerf • Google

’m not an expert in machine learning, so this There’s something mildly unsettling about essay is likely to expose this fact. Machine this — it seems to suggest that we don’t have I learning has been in the news increasingly in a theory of operation that allows us to make the past few years with the occasional spectacular direct adjustments in a confident way. When in headline, such as “AlphaGo Beats World Cham- doubt, Google! So I did, and discovered a very pion Go Player!” or “Self-Driving Cars Pass Two cogent paper from nearly 20 years ago on the Million Miles on the Road!” We can’t help being subject of neural networks and Bayesian meth- impressed by the successes of neural networks ods. My interpretation of this paper, as applied and their uncanny ability to perform tasks that in to my worry about theory of operation, is that some cases surpass human capacity. So how does the Bayesian methods can be applied to the this work and why am I worried about it? operation of the neural network to corral some John Giannandrea, Google’s Senior Vice Pres- of the side effects of overfitting in consequence ident for Search, Research, and Machine Intelli- of limited input data. While this doesn’t offer gence, likens neural networks to a box with about a mechanism for direct adjustment of the neu- a million dials on it. As the box is exposed to ral model parameters, short of training, it does inputs, it makes choices or decisions, and is given seem to improve the neural network’s perfor- feedback as to the correctness or desirability mance. Presumably, some progress has been of its output. As these lessons proceed, the box made in the past 20 years so that Bishop’s paper adjusts the dials until it performs to the train- doesn’t represent the state of the art, although I er’s satisfaction. On the other hand, if someone found it oddly comforting. said, “What happens if you adjust THIS dial by If we presume that neural networks will find 2 percent?” it’s not clear that anyone (including their way into an increasing variety of processes, the trainer) would have a credible answer. This we might wonder how these systems will behave is, of course, where my general lack of knowl- when they interact with each other in the real edge about these mechanisms presumably shows. world. Reading the headlines again, my impres- My impression is that if such devices are put into sion is that there are a number of companies out to operation and make a poor decision under some invent self-driving cars, Google included. It would set of conditions, the only option is to analyze be wrong to assume that these cars rely solely on the conditions leading to the bad choice and to neural methods for their operation. A great deal of invent new training sequences to get the system hard information is available for navigation, state to magically adjust its dials for that case. After information (traffic lights, for example), detection which, someone then needs to test to see that of moving and fixed objects, collision avoidance, other inputs still produce satisfactory results. and so on. If there’s a significant neural component

to the operation of these cars, we spots” with regard to particular situ- president of ACM. He’s widely known as might wonder how they will interact ations. Will the training feedback be one of the “fathers of the Internet.” He’s a with each other. based on accidents? “Back to driver fellow of IEEE and ACM. Contact him at In the stock market, programmed training for you, Robot 45792B!” [email protected]. trading is now a major component of all the trades done and multiple parties run these systems. I’ve been s with all my essays, I hope that I told that they sometimes behave like Aprovoke some discussion so that I This article originally appeared in young kids playing soccer: they all can learn, as well. Of course, I might IEEE Internet Computing, vol. run after the ball, leading to various provoke outrage among those who 21, no. 1, 2017. kinds of instability. I wonder about know a lot more than I do on this streets lled with self-driving cars, topic, but that’s the risk you take in each running its own brand of neu- writing for public consumption. Read your subscriptions ral network. Each will have been through the myCS publications portal at http:// trained with different scenarios Vinton G. Cerf is vice president and chief mycs.computer.org. and each might have its own “blind Internet evangelist at Google, and past

80 www.computer.org/internet/ IEEE INTERNET COMPUTING 34 Computing Edge April 2017 ADAPTIVE CYBERSECURITY

June 6, 2017 | Waltham, MA

CYBERSECURITY ATTACKS - HOW WILL YOU

RESPOND? Caleb Barlow Edna Conway Jim Routh V.P. Threat CSO, Global CSO, Aetna Intelligence, Value Chain, The landscape is ever-changing. IBM Security Cisco Find out how to win the cyber war at Rock Stars of Adaptive Cybersecurity.

Gary Hayslip Shon Lyublanovits CISO, Deputy IT Security Subcategory Director, Manager, U.S. General www.computer.org/AdaptiveCyber City of San Diego Services Administration MASTERMIND EDITOR: George Strawn, US National Academies of Sciences, Engineering, and Medicine, [email protected]

Masterminds of Artificial Intelligence Marvin Minsky and Seymour Papert

George Strawn, US National Academies of Sciences, Engineering, and Medicine Candace Strawn

he world lost two great Bronx High School of Science and 1952 (see https://en.m.wikipedia. scientists this year. Mar- the Phillips Academy. By the last org/wiki/Seymour_Papert). He vin Minsky passed away years of World War II, he was old then moved to England, where he on 24 January 2016, and enough to be drafted and spent worked at St. John’s College, Cam- TSeymour Papert passed away on 1944–1945 in the US Navy. At the bridge, and earned a second PhD 31 July. They were both 88 years end of hostilities, he returned to in mathematics in 1959. Papert was old. Their careers at MIT were his schooling, obtaining a bach- a prominent activist against South closely joined, both in artificial in- elor’s degree from Harvard in 1950 African apartheid policies during telligence (AI) and in other fields. and a PhD from Princeton in 1954, his university education. He also This article will highlight some of both in mathematics. His thesis worked at the Henri Poincaré In- their contributions and one “con- was titled “Theory of Neural-An- stitute at the University of Paris, troversy” that they jointly sparked. alog Reinforcement Systems and the University of Geneva (where There are a number of YouTube Its Application to the Brain Model he worked with Jean Piaget; see videos of talks and lectures given Problem.” This was an early con- https://en.m.wikipedia.org/wiki/ by both of these masterminds that tribution to the field of artificial Jean_Piaget), and the National are well worth watching (see, for neural networks (ANNs) which is Physical Laboratory in London be- example, https://youtu.be/-pb3z the same field in which he and Pa- fore becoming a research associate 2w9gDg). Moreover, a friend’s pert created the aforementioned at MIT in 1963. chapter-length biography of Min- controversy, which we discuss sky recently appeared in Stephen later. He joined the MIT faculty Careers at MIT Wolfram’s Idea Makers.1 in 1958 (see http://web.media.mit. Minsky is best known as a father edu/~minsky/minskybiog.html). of AI, having cofounded the MIT Early Life and Education Papert was born in 1927 in South AI laboratory in 1959 (with John Minsky was born in 1927 in New Africa and was educated there at the McCarthy, who subsequently York City to prosperous parents. University of the Witwatersrand, moved to Stanford). However, He received a good education from receiving a BA in philosophy in he made contributions in many the beginning, including at the 1949 and a PhD in mathematics in domains, including graphics,

36 April 2017 Published by the IEEE Computer Society 2469-7087/17/$33.00 © 2017 IEEE 62 IT Pro November/December 2016 Published by the IEEE Computer Society 1520-9202/16/$33.00 © 2016 IEEE MASTERMIND EDITOR: George Strawn, US National Academies of Sciences, Engineering, and Medicine, [email protected]

symbolic mathematical computa- feelings, goals, emotions, and uct was realized by constructing tion, knowledge representation, conscious thoughts in terms of LOGO programs and robotic de- commonsensical semantics, ma- multiple levels of processes, some vices that could be controlled by chine perception, and both sym- of which can reflect on the oth- LOGO programs. Papert’s coinventor bolic and connectionist learning. ers. This view of consciousness of LOGO, Wally Feurzeig (https:// He was also a pioneer of robotics is process-oriented, in that con- en.m.wikipedia.org/wiki/Wally_ and telepresence. He designed sciousness is seen to be the result Feurzeig), had been developing and built some of the first visu- of more than 20 processes going conversational programming lan- Masterminds al scanners—mechanical hands on in the human brain. guages at BBN based on Fortran, with tactile sensors. He also built If The Society of Mind and The which was designed for mathe- one of the first “turtles” for Pa- Emotion Machine contain ideas that matical problems. By focusing on of Artificial pert’s LOGO programming lan- can be implemented in comput- nonmathematical problems, they guage, including the software ers, then Minsky has indeed pro- defined a language simple enough and hardware interfaces. He vided a path for AI researchers for children to learn. And by con- Intelligence published a major work summa- to follow. If cognitive neurosci- necting the language to the control rizing many years of thought in ence discovers that the agents and of “turtles” and other robotic devic- 1985, titled The Society of Mind processes he identifies are indeed es, they provided additional moti- Marvin Minsky and Seymour (https://en.m.wikipedia.org/wiki/ present in the structures of the vation (fun) for young learners. Society_of_Mind).2 brain, then he has identified ma- Papert expressed these ideas in Papert Papert is perhaps best known jor components of “natural intel- Mindstorms: Children, Computers, for his contributions to education, ligence.” Perhaps the time is not and Powerful Ideas.3 He also col- and he also made contributions too far off when these conjectures laborated with the construction George Strawn, US National Academies of Sciences, Engineering, and Medicine in other fields. Minsky appointed will be supported or refuted. toy manufacturer Lego on their Candace Strawn him codirector of the MIT AI lab- LOGO-programmable Lego Mind- oratory in 1967. Based on Piaget’s Papert’s LOGO storms robotics kits, which were constructivist learning theory, As mentioned, Papert had worked named after his book. The third for which he was a principle ad- with Piaget in Geneva and was an ad- generation of Lego Mindstorms he world lost two great Bronx High School of Science and 1952 (see https://en.m.wikipedia. vocate, Papert codesigned the vocate for his constructivist learning was released in 2013 (see https:// scientists this year. Mar- the Phillips Academy. By the last org/wiki/Seymour_Papert). He LOGO programming language, theory (https://en.m.wikipedia.org/ en.m.wikipedia.org/wiki/Lego_ vin Minsky passed away years of World War II, he was old then moved to England, where he which influenced later languages wiki/Constructivism_(philosophy_ Mindstorms_EV3). A Lego Mind- on 24 January 2016, and enough to be drafted and spent worked at St. John’s College, Cam- such as Smalltalk and Scratch (see of_education)). Papert’s ideas were storms kit contains software and TSeymour Papert passed away on 1944–1945 in the US Navy. At the bridge, and earned a second PhD https://en.m.wikipedia.org/wiki/ given the confusingly similar name hardware to create customizable, 31 July. They were both 88 years end of hostilities, he returned to in mathematics in 1959. Papert was Logo_(programming_language)). of constructionist learning theory. programmable robots. It includes old. Their careers at MIT were his schooling, obtaining a bach- a prominent activist against South In 1980, he published Mindstorms: In a US National Science Foun- an intelligent brick computer closely joined, both in artificial in- elor’s degree from Harvard in 1950 African apartheid policies during Children, Computers, and Powerful dation proposal, he defined this that controls the system, a set of telligence (AI) and in other fields. and a PhD from Princeton in 1954, his university education. He also Ideas (https://en.m.wikipedia.org/ theory as follows (see https://en.m. modular sensors and motors, and This article will highlight some of both in mathematics. His thesis worked at the Henri Poincaré In- wiki/Mindstorms_(book)).3 wikipedia.org/wiki/Constructionism_ Lego parts from the Technic line their contributions and one “con- was titled “Theory of Neural-An- stitute at the University of Paris, (learning_theory)): to create the mechanical systems. troversy” that they jointly sparked. alog Reinforcement Systems and the University of Geneva (where Minsky’s Society of Mind (Several different programming There are a number of YouTube Its Application to the Brain Model he worked with Jean Piaget; see The Society of Mind is more specu- Constructionism is a mnemonic for languages can now be used in videos of talks and lectures given Problem.” This was an early con- https://en.m.wikipedia.org/wiki/ lative cognitive science and philos- two aspects of the theory of science Lego Mindstorms.) by both of these masterminds that tribution to the field of artificial Jean_Piaget), and the National ophy than AI. That is, it proposes education underlying this project. are well worth watching (see, for neural networks (ANNs) which is Physical Laboratory in London be- that the human mind is a “society” From constructivist theories of psy- Artificial Neural Net example, https://youtu.be/-pb3z the same field in which he and Pa- fore becoming a research associate of independent agents that evolu- chology, we take a view of learning Controversy and 2w9gDg). Moreover, a friend’s pert created the aforementioned at MIT in 1963. tion produced over time to deal as a reconstruction rather than as Deep Learning chapter-length biography of Min- controversy, which we discuss with various problems that hu- a transmission of knowledge. Then The subject of ANNs goes back sky recently appeared in Stephen later. He joined the MIT faculty Careers at MIT mans needed to solve. The book we extend the idea of manipulative to 1943, when Warren McCull- Wolfram’s Idea Makers.1 in 1958 (see http://web.media.mit. Minsky is best known as a father consists of 270 one-page chapters, materials to the idea that learning och and Walter Pitts created a edu/~minsky/minskybiog.html). of AI, having cofounded the MIT each dealing with a brain agent, is most effective when part of an computational model for neural Early Life and Education Papert was born in 1927 in South AI laboratory in 1959 (with John process, or interconnection issue. activity the learner experiences as networks (https://en.m.wikipedia. Minsky was born in 1927 in New Africa and was educated there at the McCarthy, who subsequently In 2006, Minsky published a se- constructing a meaningful product. org/wiki/Artificial_neural_network). York City to prosperous parents. University of the Witwatersrand, moved to Stanford). However, quel, The Emotion Machine,4 which As mentioned, Minsky’s PhD He received a good education from receiving a BA in philosophy in he made contributions in many proposes theories that could ac- The idea of the (child) learner dissertation had contributed to the beginning, including at the 1949 and a PhD in mathematics in domains, including graphics, count for human higher-level constructing a meaningful prod- the early study of ANNs. But in

1969, Minsky and Papert coau- (See www.ucs.louisiana.edu/ 3. S.A. Papert, Mindstorms: Children, thored a book titled Perceptrons,5 ~isb9112/dept/phil341/histconn. Computers, and Powerful Ideas, Basic which gave rise to a controversy html for a second view of this Books, 1980. (over what they really said) that is history.) 4. M. Minsky, The Emotion Machine: still being argued today. In their By the 2000s, hidden-layer Commonsense Thinking, Artificial In- book, they showed that “simple ANNs had proven to be very telligence, and the Future of the Human ANNs” were quite limited in powerful and in fact were pow- Mind, Simon & Schuster, 2006. computational power. The as- ering many real AI applications. 5. M. Minsky and S.A. Papert, Percep- sumption that “complex ANNs” Hidden-layer ANNs are now part trons, MIT Press, 1969. (technically called hidden-layer of the family of deep learning al- 6. M. Minsky and S.A. Papert, Percep- ANNs) were also of limited pow- gorithms (https://en.m.wikipedia. trons: An Introduction to Computational er might have been made by Min- org/wiki/Deep_learning), which Geometry, expanded ed., MIT Press, sky and Papert, but not in this has propelled us from an “AI win- 1987. book. One commentator stated ter” into an “AI summer.” that “the often-miscited Minsky/ George Strawn is the director of the Papert text caused a significant Board for Research Data and Informa- decline in interest and fund- he authority of Minsky tion at the US National Academies of ing of neural network research,” and Papert was evident in Sciences, Engineering, and Medicine. and that it would take 10 years T that their negative view He is the former director of the National for neural network research to of ANNs was probably a factor Coordination Office for the Networking experience a resurgence in the in discouraging researchers and and Information Technology Research 1980s. (An expanded edition of funders of research from further and Development Program (NITRD). Perceptrons6 was reprinted in 1987 study of ANNs at that time. Their Contact him at [email protected]. with some errors from the origi- authority arose, of course, from nal text shown and corrected; see their many important contribu- Candace Strawn is a retired high school, https://en.m.wikipedia.org/wiki/ tions to AI and allied fields. They community college, and university teacher. Perceptron). Another commen- will be remembered more for Contact her at [email protected]. tator, a researcher at the MIT those contributions than for the AI lab, quoted Minsky and Pap- controversy. ert in the 1971 “Report of Proj- ect MAC,”—directed at funding References agencies—on “gamba networks” 1. S. Wolfram, Idea Makers: Personal (see https://en.m.wikipedia.org/ Perspectives on the Lives & Ideas of Some This article originally appeared in wiki/Perceptrons_(book)): Notable People, Wolfram Media, Read your subscriptions IT Professional, vol. 18, no. 6, 2016. through the myCS publi- 2016. Virtually nothing is known about the 2. M. Minsky, The Society of Mind, Si- cations portal at http:// mycs.computer.org. computational capabilities of [these mon & Schuster, 1988. machines]. We believe that [they] can do little more than can a low order perceptron.

IT Professional (ISSN 1520-9202) is published bimonthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th Floor, New York, NY 10016-5997; IEEE Computer So- ciety Publications Office, 10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720- 1314; voice +714 821 8380; fax +714 821 4010; IEEE Computer Society Headquarters, 1828 L St. NW, Suite 1202, Washington, DC 20036. Annual subscription: $49 in addition to any IEEE Computer Society dues. NonmemberThis rates series are ofavailable in-depth on interviewsrequest. Back with issues: prominent $20 for members, $143 for nonmembers. security experts features Gary McGraw as anchor. Postmaster: Send undelivered copies and addressIEEE Security changes & to Privacy IT Professional magazine, Member- ship Processing Dept., IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854-4141. Periodicals Postage Paid at New York, NY, andpublishes at additional excerpts mailing of theoffices. 20-minute Canadian GST #125634188. Canada Post Publications Mailconversations Agreement Numberin article 40013885. format each Return issue. unde- FOLLOW US liverable Canadian addresses to PO Box 122, Niagara Falls, ON L2E 6S8, Canada. Printed in the USA. Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect thewww.computer.org/silverbullet author’s or firm’s opinion. Inclusion in IT Professional does not necessarily constitute endorsement*Also by available the IEEE or the at ComputeriTunes Society. All submissions are subject to editing for style, clarity, and space. www.computer.org/itpro @securityprivacy 38 Computing Edge April 2017 64 IT Pro November/December 2016 Scientific Programming Editors: Konrad Hinsen, [email protected]@cnrs.fr | Matthew | Matthew Turk, [email protected] Turk, [email protected]

Risks

Matthew Turk | National Center for Supercomputing Applications

y thoughts about the topic of this department If you can’t mitigate risks out of existence, you still have to are usually in a project context—for me, “sci- deal with—and attempt to counter—them in some fashion. entific programming” typically takes the form Three particular risk scenarios have been on my mind re- M of something related to computational physics, cently, and I have a few strategies for addressing them. These or data analysis and visualization. I proceed through a few won’t work for everyone—they don’t even always work for stages when evaluating a new or existing project: deciding me—but they could be of some use, depending on the situation. what needs to happen, sketching out a rough plan, evaluating my options, and determining the manageable risks. Par- Risk of Making a Mistake ticularly when preparing open source scientific projects or The risk that I worry about the most, the one that keeps me up those that consume large quantities of resources, risk evaluat night, is the risk of making a mistake. The dangers are both ation is crucial, both to commit to a project and to harden it overt and subtle. The vagaries of transcription, the translation of against possible weaknesses. an algorithm into a programming language, for example, can I’ve spent a fair amount of time evaluating how I think result in changes in functionality—what if there’s a stray paren- about risk and how it has influenced my projects and plans. thesis, a missing decimal point, or something even more insidi- As I’ve tried to be more self-aware about this process (espe- ous, such as a line of code extending past the character limit in cially considering that “being aware of risk” is often a nice a Fortran routine? Fortunately, an algorithm isn’t the same as an way of saying “being a tad bit afraid of it”), I’ve also realized implementation; for all our lofty goals of translating formulas that implicit in the decision-making process is risk mitigation. into machine code, they still remain separate. But beyond minor

2469-7087/17/$33.00 © 2017 IEEE Published by the IEEE Computer Society April 2017 39 November/December 2016 Copublished by the IEEE CS and the AIP 1521-9615/16/$33.00 © 2016 IEEE Computing in Science & Engineering 63 Scientific Programming

Perfection can’t be reached; this must be taken as a given. Instead, we should focus on mistaken identification, mitigation, and reduction. How can we find mistakes in software? And, from the perspective of mitigating risks going forward, how can we guard against adding new mistakes? mechanical errors, larger errors still lurk—perhaps a method Risk of Depending on Something doesn’t converge with sufficiently low error, terms can’t be split The rate of change in the scientific software ecosystem can in some regimes, or a corner case has gone unnoticed. These mis- be very difficult to predict. Libraries interoperate through takes, too, are risks. I’ve heard far too often from colleagues that APIs that provide clearly defined relationships for what they’re apprehensive of releasing code for fear of exposing mis- serves as input (and what in return serves as output from takes. The implicit irony is that they aren’t apprehensive about functions and routines). Although these APIs often remain utilizing their software for their own purposes but of exposure. stable, they’re fundamentally outside the control of down- I’m reluctant to attribute this quotation, as I’m sure I’ll stream end users—in this case, scientific programmers us- misattribute it, but someone once said, “Are you afraid of re- ing those APIs. This type of dependency brings with it its leasing your software because it might have bugs in it? I’ll save own risks, with the software developer required to maintain you some time. It does.” This really stuck with me, because it and keep up with changes (or forego gaining access to im- says not only that this is to be expected and you aren’t alone, provements and bug fixes) as well as ensure that any behav- but also that it’s something you shouldn’t be in denial about. ioral changes are understood and accounted for. The most self-aware expression of this that I’ve seen is the ver- Utilizing an external package or project requires giving sion numbering system used in TeX, which will converge to up a degree of autonomy and self-direction. The converse, pi during the lifetime of its inventor, Donald Knuth. of course, is that simply implementing every item necessary The most obvious answer, of course, is to test things, which requires an incredible amount of dedication and hard work is relatively straightforward when the code being tested is mod- that could be much more easily spent on improving the spe- ular, has few execution paths, and conforms to a result that can cific project being worked on. be identified a priori. But when an algorithm, module, or proj- Mitigating this risk is in many ways more difficult than ect doesn’t possess these attributes, ever-increasing circles around mitigating the risk of errors, as it involves people rather than them can still be identified and tested. You start by breaking code. I’ve found that the most productive way of mitigating the system down into its smallest parts, testing them in multiple the risks in adding a dependency, building in an ecosystem, combinations, and then testing those combinations. You need or interoperating with a tool is to understand the alignment to search input space parameters and guard against unknown of interests. Are the motivations of the external library or soft- behavior—in contrast with a silent, implicit failure that can be ware developers aligned with my own? Or at the very least, misinterpreted as a result, an explicit failure when applied out- are they not antialigned? What’s on my road map, what’s on side of known input space can save confusion and heartache. theirs, and do they allow for long-term interoperability? When explicit, analytic, or prior solutions can’t be determined For many pieces of software, this is a simple evaluation: in advance, we often have to resort to testing against regressions API contracts are stable unless specifically noted, and in through so-called “answer testing”: if an answer is agreed on some fields of scientific software, these notes come once a as being a gold standard, subsequent iterations of development decade. But in other areas, particularly the parts of the eco- shouldn’t deviate from it. For more on this, see P.F. Dubois, system undergoing rapid change (or that are well-suited to “Testing Scientific Programs,” Computing in Science & Eng., rapid change, such as dynamic languages or anything relat- vol. 14, no. 4, 2012, pp. 69–73, doi:10.1109/MCSE.2012.84; ed to Web services), these situations must be more carefully and K. Hinsen, “Technical Debt in Computational Science,” monitored. This can take the form of extensive integration Computing in Science & Eng., vol. 17, no. 6, 2015, pp. 103–107, tests between systems, “pinning” specific version numbers doi:10.1109/MCSE.2015.113. as dependencies, and building in contingency plans for if The best way to mitigate—but not eliminate—risk is to a piece of software shifts in focus or no longer provides the openly share, discuss, and examine written code. Sharing necessary functionality. code doesn’t immediately and magically ensure that it’s error free (it doesn’t even guarantee that it will monotonically Risk of Starting Something Big decrease in error count), but it does increase the opportunity Being wrong and building on an unreliable foundation are for discovering mistakes. equally scary, but they both have mitigation paths that are

40 Computing Edge April 2017 64 November/December 2016 Scientific Programming

Perfection can’t be reached; this must be taken as a given. Instead, These are the easy parts to implement; best practices of software we should focus on mistaken identification, mitigation, and reduction. development help guide these types of invasive projects and changes. How can we find mistakes in software? And, from the perspective of But how do we convince ourselves that this is the right path? mitigating risks going forward, how can we guard against adding new mistakes? easy to see and straightforward to implement. But there’s How do we undertake something large, intimidating, another risk that lurks in the minds of scientific program- overwhelming, knowing that it might be a failure down the mers, and that’s the risk of starting something all-caps BIG. road, or that it may never pay off? mechanical errors, larger errors still lurk—perhaps a method Risk of Depending on Something Major undertakings, refactorings, implementations—these To this, I don’t have a ready, easy answer. For some rea- doesn’t converge with sufficiently low error, terms can’t be split The rate of change in the scientific software ecosystem can are the investments of time, resources, and energy that son, computational work seems to draw out different emotions in some regimes, or a corner case has gone unnoticed. These mis- be very difficult to predict. Libraries interoperate through might or might not flourish. If your fantastic new project than experimental work; experiments are by their nature “ex- takes, too, are risks. I’ve heard far too often from colleagues that APIs that provide clearly defined relationships for what idea results in enormous benefits, that’s marvelous, but what perimental,” sometimes without a known endpoint, and some- they’re apprehensive of releasing code for fear of exposing mis- serves as input (and what in return serves as output from if it results in the alienation of collaborators? If your project times carrying with them the risk of the unknown result. takes. The implicit irony is that they aren’t apprehensive about functions and routines). Although these APIs often remain brings in new people, ideas, and results that are outstanding I can only share the strategies that work for me: faced utilizing their software for their own purposes but of exposure. stable, they’re fundamentally outside the control of down- and exciting, it’s a success, but how do you mitigate against with a large project, overwhelming and scary, focusing on I’m reluctant to attribute this quotation, as I’m sure I’ll stream end users—in this case, scientific programmers us- the very real possibility that it might simply go nowhere or the creative urge can be comforting. But this only lasts for a misattribute it, but someone once said, “Are you afraid of re- ing those APIs. This type of dependency brings with it its that it might not work as you expect? while, until infrastructure, maintenance, and integration are leasing your software because it might have bugs in it? I’ll save own risks, with the software developer required to maintain I’ve struggled the most with this particular risk. The required to keep pushing the project forward to its conclu- you some time. It does.” This really stuck with me, because it and keep up with changes (or forego gaining access to im- others I understand better how to mitigate; there are worst- sion. In these cases, I find myself struggling, and the most says not only that this is to be expected and you aren’t alone, provements and bug fixes) as well as ensure that any behav- case scenarios that can be dealt with. But this one is more successful strategy I’ve found has been to foster collaboration. but also that it’s something you shouldn’t be in denial about. ioral changes are understood and accounted for. existential, and it cuts across a broader swathe of profession- Building something can be isolating and lonely, but reducing The most self-aware expression of this that I’ve seen is the ver- Utilizing an external package or project requires giving al responsibilities. If I start this project, and it doesn’t work that isolation can build social capital, share the burden of the sion numbering system used in TeX, which will converge to up a degree of autonomy and self-direction. The converse, out, how can I justify the time I spent? project, and even guide things forward in unforeseen ways. pi during the lifetime of its inventor, Donald Knuth. of course, is that simply implementing every item necessary Even more to the point, if I start a new project, how The most obvious answer, of course, is to test things, which requires an incredible amount of dedication and hard work can I justify spending the time that it needs to get boot- is relatively straightforward when the code being tested is mod- that could be much more easily spent on improving the spe- strapped, to reach a minimum viability, to even demon- hese risks and strategies are by no means the only ones ular, has few execution paths, and conforms to a result that can cific project being worked on. strate some of its promise? Many of us have seen this before, T we encounter in the process of scientific programming, be identified a priori. But when an algorithm, module, or proj- Mitigating this risk is in many ways more difficult than and I confess I’ve even participated, for good and ill. In my and I certainly don’t think that I’ve identified all the pos- ect doesn’t possess these attributes, ever-increasing circles around mitigating the risk of errors, as it involves people rather than case, I’ve seen this conflict arise around scientific platforms. sible strategies for addressing them. I invite readers to share them can still be identified and tested. You start by breaking code. I’ve found that the most productive way of mitigating Imagine a piece of software, used for knowledge production their thoughts on these and other approaches: When faced the system down into its smallest parts, testing them in multiple the risks in adding a dependency, building in an ecosystem, (and thus aligned with the economic and reputational inter- with difficulties or challenges, how have you addressed combinations, and then testing those combinations. You need or interoperating with a tool is to understand the alignment ests of its users), that perhaps hasn’t been updated with new them? What strategies have you undertaken? What worked to search input space parameters and guard against unknown of interests. Are the motivations of the external library or soft- design patterns, new languages, or modern libraries. Maybe and what didn’t? behavior—in contrast with a silent, implicit failure that can be ware developers aligned with my own? Or at the very least, it has started to buckle under design decisions made early in misinterpreted as a result, an explicit failure when applied out- are they not antialigned? What’s on my road map, what’s on the project’s career. Matthew Turk is a research scientist at NCSA at the University side of known input space can save confusion and heartache. theirs, and do they allow for long-term interoperability? The opportunity to rebuild, redesign, and start afresh of Illinois. His research interests include data analysis, computa- When explicit, analytic, or prior solutions can’t be determined For many pieces of software, this is a simple evaluation: is enticing. But for me, at least, there’s an undercurrent tional science, and scientific software communities. Turk received in advance, we often have to resort to testing against regressions API contracts are stable unless specifically noted, and in of dread: if I can’t undertake this entire project by my- a PhD in physics from Stanford. Contact him at matthewturk@ through so-called “answer testing”: if an answer is agreed on some fields of scientific software, these notes come once a self, how do I best set it up for success if I can’t commit gmail.com. as being a gold standard, subsequent iterations of development decade. But in other areas, particularly the parts of the eco- time to it? How do I build it out in such a way that others shouldn’t deviate from it. For more on this, see P.F. Dubois, system undergoing rapid change (or that are well-suited to can participate, that it can show immediate results and be “Testing Scientific Programs,” Computing in Science & Eng., rapid change, such as dynamic languages or anything relat- minimally disruptive? vol. 14, no. 4, 2012, pp. 69–73, doi:10.1109/MCSE.2012.84; ed to Web services), these situations must be more carefully Mitigating this type of risk is much more difficult be- and K. Hinsen, “Technical Debt in Computational Science,” monitored. This can take the form of extensive integration cause it requires both external and internal preparation. We Computing in Science & Eng., vol. 17, no. 6, 2015, pp. 103–107, tests between systems, “pinning” specific version numbers have to design our projects in such a way as to minimize This article originally appeared in doi:10.1109/MCSE.2015.113. as dependencies, and building in contingency plans for if disruption to those who might be using them—chiefly, our- Computing in Science & Engineering, vol. 18, The best way to mitigate—but not eliminate—risk is to a piece of software shifts in focus or no longer provides the selves—which requires discipline in development, ensuring no. 6, 2016. openly share, discuss, and examine written code. Sharing necessary functionality. changes are as modular as possible and that each new state of code doesn’t immediately and magically ensure that it’s er- the project is as working and functional as possible. If your Read your subscriptions through ror free (it doesn’t even guarantee that it will monotonically Risk of Starting Something Big project has an existing user base, it requires that disruptions Selected articles and columns from IEEE Computer Society pub- the myCS publications portal at decrease in error count), but it does increase the opportunity Being wrong and building on an unreliable foundation are are minimal and well-communicated, with stakeholders in- lications are also availablehttp://mycs.computer.org. for free at http://ComputingNow. for discovering mistakes. equally scary, but they both have mitigation paths that are vested in the changes and the reasoning behind them. computer.org.

www.computer.org/computingedge 41 64 November/December 2016 www.computer.org/cise 65 CLOUD TIDBITS

Moving to Autonomous and Self-Migrating Containers for Cloud Applications

THE TROUBLE WITH EXISTING APPROACH- vide a common abstraction layer that allows applica- ES TO CLOUD COMPUTING, INCLUDING tions to be localized within the container, and then LEVERAGING INFRASTRUCTURE AS A SER- ported to other public and private cloud providers VICE (IAAS) AND PLATFORM AS A SERVICE that support the container standard. (PAAS), IS THAT THEY TEND TO COME WITH RightScale’s new State of the Cloud Report con- PLATFORM LOCK-IN. Once you’ve ported an firms that containers (exemplified by Docker and application to a cloud-based platform, including CoreOS) are undergoing rapid growth.1 The quick Google, Amazon Web Services (AWS), IBM, and uptake of containers makes a lot of sense given Microsoft, it’s tough, risky, and expensive to move what they offer. At a high level, containers provide that application from one cloud to another. lightweight platform abstraction without using This isn’t by design. The market moved so virtualization. quickly that public and private cloud providers Containers are also much more efficient for cre- couldn’t build portability into their platform and ating workload bundles that are transportable from still keep pace with demand. There’s also the fact cloud to cloud. In many cases, virtualization is too that portability isn’t in the best interests of cloud cumbersome for workload migration. Thus, contain- providers. ers provide a real foundation for moving workloads Enter new approaches based on old approaches, around hybrid clouds and multiclouds without hav- namely containers, and thus Docker and container ing to alter much, if any, of the application. cluster managers, such as Google’s Kubernetes, as More specifically, containers provide these well as hundreds of upstarts. The promise is to pro- advantages:

• reduced complexity through container abstractions; • the ability to use automation with containers to maximize their portability; • better security and governance from placing services around, rather than inside, containers; • better distributed computing capabilities, be- DAVID S. cause an application can be divided into many LINTHICUM separate domains, all residing within containers; and Cloud Technology Partners • the ability to provide automation servic- [email protected] es that offer policy-based optimization and self-configuration.

d6tid.indd 6 3/16/17 12:45 PM CLOUD TIDBITS

Containers provide something we’ve been trying to achieve for years: a standard application architec- Cloud orchestration Moving to Autonomous and ture that offers both managed distribution and service orientation. Most compelling right now is containers’ portability advantage. However, I suspect we’ll discover Containers Self-Migrating Containers more value over time. In fact, I suspect that containers will become a part of most IT shops, no matter whether they’re moving to the cloud or not.2 Cloud platform and data services for Cloud Applications Defining a New Value for Containers

Cloud A Cloud B Containers are predicated on the goal of deploying and managing n-tier application THE TROUBLE WITH EXISTING APPROACH- vide a common abstraction layer that allows applica- designs. By their nature, containers manage FIGURE 1. A cloud orchestration layer oversees the infrastructure ES TO CLOUD COMPUTING, INCLUDING tions to be localized within the container, and then n-tier application components, e.g., database supporting live migration of containers. LEVERAGING INFRASTRUCTURE AS A SER- ported to other public and private cloud providers servers, application servers, web servers, VICE (IAAS) AND PLATFORM AS A SERVICE that support the container standard. etc., at the operating system level. Indeed, (PAAS), IS THAT THEY TEND TO COME WITH RightScale’s new State of the Cloud Report con- portability is inherent because all operating vices, which reduces the complexity of dealing with PLATFORM LOCK-IN. Once you’ve ported an firms that containers (exemplified by Docker and system and application configuration depen- those platforms. Containers are truly small plat- application to a cloud-based platform, including CoreOS) are undergoing rapid growth.1 The quick dencies are packaged and delivered inside a forms that support an application or an application’s Google, Amazon Web Services (AWS), IBM, and uptake of containers makes a lot of sense given container to any other operating system plat- services that sit inside of a well-defined domain. Microsoft, it’s tough, risky, and expensive to move what they offer. At a high level, containers provide form. Containers are preferable to virtual The second advantage is the ability to leverage that application from one cloud to another. lightweight platform abstraction without using machines here because they share compute automation with containers to maximize their por- This isn’t by design. The market moved so virtualization. platform resources very well whereas virtual tability, and thus their value. Through the use of quickly that public and private cloud providers Containers are also much more efficient for cre- machine platforms tend to acquire and hold automation, we script things we could also do manu- couldn’t build portability into their platform and ating workload bundles that are transportable from resources on a machine-by-machine basis.3 ally, such as migrating containers from one cloud to still keep pace with demand. There’s also the fact cloud to cloud. In many cases, virtualization is too another. We can also reconfigure communications that portability isn’t in the best interests of cloud cumbersome for workload migration. Thus, contain- In essence, containers can move from cloud to between containers, such as tiered services, or data providers. ers provide a real foundation for moving workloads cloud and system to system, and thus can also pro- service access. However, today it’s much harder to Enter new approaches based on old approaches, around hybrid clouds and multiclouds without hav- vide automation for this process. In other words, guarantee portability and application behavior when namely containers, and thus Docker and container ing to alter much, if any, of the application. we not only can leverage containers, but also can using automation. Indeed, automation often relies cluster managers, such as Google’s Kubernetes, as More specifically, containers provide these have them automatically “live migrate” from cloud on many external dependencies that can break at well as hundreds of upstarts. The promise is to pro- advantages: to cloud as needed to support the application’s any time, and thus remains a problem. However, it’s requirements. indeed solvable. • reduced complexity through container At the center of the container evolution is a Another advantage is the ability to provide bet- abstractions; cloud orchestration layer that can provision the inter security and governance services by placing • the ability to use automation with containers to frastructure required to support the containers, as those services around, rather than within, contain- maximize their portability; well as perform the live migration and monitor their ers. In many instances, security and governance ser- • better security and governance from placing ser- health after the migration occurs (see Figure 1). vices are platform-specific, not application-specific. vices around, rather than inside, containers; The concepts of autoprovisioning and automi- Placing security and governance services outside • better distributed computing capabilities, be- gration are often promoted within modern cloud of the application domain provides better portabil- DAVID S. cause an application can be divided into many computing development but are elusive in prac- ity and less complexity during implementation and LINTHICUM separate domains, all residing within contain- tice. These concepts have a few basic features and operations. ers; and advantages. Better distributed computing capabilities can Cloud Technology Partners • the ability to provide automation servic- First is the ability to reduce complexity by lever- also be provided since an application can be divided [email protected] es that offer policy-based optimization and aging container abstractions. Containers remove the into many different domains, all residing with con- self-configuration. dependencies on the underlying infrastructure ser- tainers. These containers can be run on any number

d6tid.indd 6 3/16/17 12:45 PM d6tid.indd 7 3/16/17 12:45 PM CLOUD TIDBITS

of cloud platforms, including those that provide the noted in an email, portability of container cluster- most cost and performance efficiencies, and therefore ing and orchestration is likely to quickly become the applications can be distributed and optimized as to bottleneck. their use of the platform from within the container. For example, an I/O-intensive portion of the applica- Making the Business Case tion could run on a bare metal cloud that provides The problem with technical assertions is that they the best performance, while a compute-intensive por- need to define a business benefit to be accepted by tion of the application runs on a public cloud that the industry as a best practice. The technical benefits provides the proper scaling and load balancing. Per- I’ve defined need to be translated into direct business haps even a portion of the application could run on benefits that provide a quick return on investment. traditional hardware and software. They all work One business benefit is the ability to automati- together to form the application, and the application cally find least-cost cloud providers. Part of the ben- is separated into components that can be optimized. efit of moving from cloud to cloud is that you can Finally, there’s the ability to provide automation leverage this portability to find the least-cost pro- services that offer policy-based optimization and vider. Assuming most things are equal, the applica- self-configuration. None of this works without pro- tions within a set of containers can live migrate to a viding an automation layer that can “automagically” cloud that offers price advantages for similar types find the best place to run the container, as well as of cloud services, such as storage. For example, an inventory control application that exists within a doz- en or so containers might have some storage-intensive components that cost The problem with technical $100,000 a month on AWS. However, Google charges $50,000 a month for assertions is that they need to define the same types of resources. Under- a business benefit to be accepted by standing this configuration possibility the industry as a best practice. within the orchestration layer, the containers can automigrate/live migrate to the new cloud where there’s a 50 percent savings. If Google raises its pric- es and AWS lowers theirs, the reverse could occur. deal with the changes in the configurations, and These automation concepts also support better other things specific to the cloud platforms where reliability. We’ve all done business cases around up- the containers reside. time and down-time. In some instances, businesses However, we’ve learned that n-tier applications can lose as much as $1 million an hour when sys- have inherent limitations. “They are designed to tems aren’t operating. Even if the performance issue scale up with very little focus paid on scaling down lasts for only an hour or two, the lost productivity and no attention paid to scaling out or in. They typi- can move costs well into thousands of dollars per cally are rife with single points of failure and tend to minute. manage their own state via the use of cluster-style This architecture shown in Figure 1 can help computing. Each tier of the n-tiered architecture avoid outages and related performance issues by must be scaled independently of the other tiers.”3 opening other cloud platforms where the container Also, keep in mind that the automation/orches- workloads can relocate if issues occur on the pri- tration required will not always be portable. Indeed, mary clouds. For example, if AWS suffers an out- that’s likely the new lock-in layer; once you’ve built age, the containers can be relocated to Google in out the operational side, how easy is it to migrate a matter of minutes, where they can operate once from cloud to cloud? As Lori MacVittie of F5.com again until the problem is resolved. You might

44 Computing Edge April 2017 8 IEEE CLOUD COMPUTING WWW.COMPUTER.ORG/CLOUDCOMPUTING

d6tid.indd 8 3/16/17 12:45 PM CLOUD TIDBITS

of cloud platforms, including those that provide the noted in an email, portability of container cluster- choose to run redundant versions of the containers time, moving from a true platform to good contain- most cost and performance efficiencies, and therefore ing and orchestration is likely to quickly become the on both clouds, supporting an active/active type of er hosts. It will be interesting to see if the larger applications can be distributed and optimized as to bottleneck. recovery platform. providers want to take on that role. Considering their use of the platform from within the container. provider interest in Docker, that indeed could be For example, an I/O-intensive portion of the applica- Making the Business Case Facing Realities their direction. tion could run on a bare metal cloud that provides The problem with technical assertions is that they Containers might sound like distributed application The core question now: if this is the destination the best performance, while a compute-intensive por- need to define a business benefit to be accepted by nirvana. They certainly offer a better way to utilize of this technology and application hosting on cloud- tion of the application runs on a public cloud that the industry as a best practice. The technical benefits emerging cloud-based platforms. However, there are based platforms, should I redirect resources toward provides the proper scaling and load balancing. Per- I’ve defined need to be translated into direct business many roadblocks in front of us and a lot of work to this new vision? I suspect that most enterprises haps even a portion of the application could run on benefits that provide a quick return on investment. be done. already have their hands full with the great cloud traditional hardware and software. They all work One business benefit is the ability to automati- We need to consider the fact that current tech- migration. However, as we get better at cloud ap- together to form the application, and the application cally find least-cost cloud providers. Part of the ben- nology can’t provide this type of automation. Al- plication architectures using approaches that better is separated into components that can be optimized. efit of moving from cloud to cloud is that you can though it can certainly manage machine instances, account for both automation and portability, we’ll Finally, there’s the ability to provide automation leverage this portability to find the least-cost pro- even containers, using basic policy and scripting eventually land on containers. services that offer policy-based optimization and vider. Assuming most things are equal, the applica- approaches, automatically moving containers from self-configuration. None of this works without pro- tions within a set of containers can live migrate to a cloud to cloud using policy-driven automation, in- Acknowledgments viding an automation layer that can “automagically” cloud that offers price advantages for similar types cluding autoconfiguration and autolocalization, isn’t Part of this article was derived from research I’ve find the best place to run the container, as well as of cloud services, such as storage. there yet. done at Gigaom Pro, which is now out of business. For example, an inventory control Also, we’ve only just begun our Docker con- application that exists within a doz- tainer journey. We still have a lot to learn about the References en or so containers might have some technology’s potential as well as its limitations. As 1. Rightscale, State of the Cloud Report, 2016; storage-intensive components that cost we learned from the use of containers and distrib- https://www.rightscale.com/lp/state-of-the The problem with technical $100,000 a month on AWS. However, uted objects from years ago, the only way this tech- -cloud?campaign=701700000015euX. Google charges $50,000 a month for nology can provide value is through coordinating 2. D. Linthicum, “Fad? No, Containers Are Here to assertions is that they need to define the same types of resources. Under- clouds that support containers. Although having a Stay,” InfoWorld, 12 Feb. 2016; www.infoworld a business benefit to be accepted by standing this configuration possibility standard here is great, history shows that vendors .com/article/3032164/cloud-computing/fad-no the industry as a best practice. within the orchestration layer, the con- and providers tend to march off in their own propri- -containers-are-here-to-stay.html. tainers can automigrate/live migrate to etary directions for the sake of market share. If that 3. D. Linthicum, “Containers Are Designed for an An- the new cloud where there’s a 50 per- occurs, all is lost. tiquated Application Architecture,” Container J., cent savings. If Google raises its pric- The final issue is complexity. It only seems like 5 June 2015; http://containerjournal.com/2015/ es and AWS lowers theirs, the reverse we’re making things less complex. Over time, the 06/05/containers-are-designed-for-an-antiquated could occur. use of containers as the means of platform abstrac- -application-architecture. deal with the changes in the configurations, and These automation concepts also support better tion will result in applications that morph toward other things specific to the cloud platforms where reliability. We’ve all done business cases around up- architectures that are much more complex and dis- the containers reside. time and down-time. In some instances, businesses tributed. Moving forward, it might not be unusual DAVID S. LINTHICUM is senior vice president of However, we’ve learned that n-tier applications can lose as much as $1 million an hour when sys- to find applications that exist in hundreds of con- Cloud Technology Partners. He also frequently writes have inherent limitations. “They are designed to tems aren’t operating. Even if the performance issue tainers, running on dozens of different models and for InfoWorld on deep technology subjects. His re- scale up with very little focus paid on scaling down lasts for only an hour or two, the lost productivity brands of cloud computing. The more complex these search interests include complex distributed systems, and no attention paid to scaling out or in. They typi- can move costs well into thousands of dollars per things become, the more vulnerable they are to op- including cloud computing, data integration, service- cally are rife with single points of failure and tend to minute. erational issues. oriented architecture, Internet of Things, and big data manage their own state via the use of cluster-style This architecture shown in Figure 1 can help systems. Contact him at [email protected]. computing. Each tier of the n-tiered architecture avoid outages and related performance issues by must be scaled independently of the other tiers.”3 opening other cloud platforms where the container ALL THINGS CONSIDERED, CONTAINERS Also, keep in mind that the automation/orches- workloads can relocate if issues occur on the pri- MIGHT BE A MUCH BETTER APPROACH TO tration required will not always be portable. Indeed, mary clouds. For example, if AWS suffers an out- BUILDING APPLICATIONS ON THE CLOUD. that’s likely the new lock-in layer; once you’ve built age, the containers can be relocated to Google in PaaS and IaaS clouds will still provide the plat- This article originallyRead yourappeared subscriptions in through the myCS publications portal at out the operational side, how easy is it to migrate a matter of minutes, where they can operate once form foundations and even development capabilities. IEEE Cloud Computinghttp://mycs.computer.org., vol. 3, no. 6, 2016. from cloud to cloud? As Lori MacVittie of F5.com again until the problem is resolved. You might However, these things will likely commoditize over

www.computer.org/computingedge 45 8 IEEE CLOUD COMPUTING WWW.COMPUTER.ORG/CLOUDCOMPUTING NOVEMBER/DECEMBER 2016 IEEE CLOUD COMPUTING 9

d6tid.indd 8 3/16/17 12:45 PM d6tid.indd 9 3/16/17 12:45 PM COMPUTING CAREERS

Careers in Artifi cial Intelligence

or this ComputingEdge issue, we asked to do tasks that were previously done by humans Vanderbilt University professor of com- or AI alone is probably the future. F puter science and computer engineering So, AI professionals will be working on teams Douglas Fisher about career opportunities in arti- with HCI (human-computer interaction) profes- fi cial intelligence. Fisher also serves as director sionals, and it’s likely that expertise in this area for outreach, education, diversity, and synthesis will be a plus. The products they produce will for CompSustNet, a US National Science Founda- include mobile devices, as well as intelligent per- tion–sponsored research network used to explore sonal assistants that work with, rather than frus- new directions in computational sustainability. He trate, their owners. authored the article “Recent Advances in AI for Machine learning will also continue to be a Computational Sustainability” in IEEE Intelligent dominant AI subfi eld for some time. And there Systems’ July/August 2016 issue. is plenty of room to improve AI in games and in social and environmental simulations. ComputingEdge: What careers in artifi cial intelligence will see the most growth during the next ComputingEdge: What would you tell college several years? students to give them an advantage over the competition? Fisher: Many AI jobs will be broadly concerned with designing and implementing automation. Fisher: Students should think about their interests They’ll be distributed across many sectors, includ- and passions. Students who are in computer sci- ing retail, transportation, and healthcare. However, ence for “guaranteed” jobs may get work but not for the most part, they won’t be about full automa- necessarily a career they love. Well-meaning but, tion because experience suggests this is fraught I think, misguided faculty members sometimes with problems. Rather, integrating humans and AI promote computer science for its job potential,

46 April 2017 Published by the IEEE Computer Society 2469-7087/17/$33.00 © 2017 IEEE without considering other sources of motivation Thomas Standish in computer science and a such as a love of technology or the possibility of class in cognition and memory from Tarow Indow advancing the social good. in psychology. Studying how computers and Students should consider pairing their AI and humans represented and processed information computer-science interests with something else— led to small revelations that changed the direction perhaps a love of foreign languages and cultures, of both my studies and my career. This goes to the medicine and health, psychology, neuroscience, point I made that combining computing and other the law, journalism and writing, music, fi lm, or fi elds can be a big win. other types of art. This can open students’ eyes to what’s possible in an AI career and make them more attractive to companies. omputingEdge’s Lori Cameron inter- Internships are very important, too. These can viewed Fisher for this article. Contact include summer undergraduate research projects C her at [email protected] if you with faculty at their school or at another institu- would like to contribute to a future ComputingEdge tion. Underclass students who don’t feel ready for article on computing careers. Contact Fisher at a full-blown computer-science internship might douglas.h.fi [email protected]. want to do something like teaching programming to kids at a summer camp. I think many companies appreciate a diversity of internship experience. Read your subscriptions through the myCS publications portal at http://mycs.computer.org. ComputingEdge: What should applicants keep in mind when applying for artifi cial-intelligence- related jobs?

Fisher: Understand the job, including its requirements, obligations, and freedoms. Is AI the only necessary area of expertise? Will they be trained 2017 B. Ramakrishna Rau Award in other skills on the job? Call for Nominations

ComputingEdge: Name one critical mistake for Honoring contributions to the computer microarchitecture field young graduates to avoid when starting their New Deadline: 1 May 2017 careers? Established in memory of Dr. B. (Bob) Ramakrishna Rau, the award recognizes his distinguished career in promoting and expanding the use of innovative comput- Fisher: Don’t forget about the future—both an er microarchitecture techniques, including his innovation in complier technology, his leadership in academic and individual’s fi nancial and family future, and a industrial computer architecture, and his extremely high societal and environmental future that requires personal and ethical standards. WHO IS ELIGIBLE?: The candidate will have made an active contributions by people with brains, skills, outstanding innovative contribution or contributions to microarchitecture and use of novel microarchitectural techniques or compiler/architecture and caring. interfacing. It is hoped, but not required, that the winner will have also contributed to the computer microarchitecture community through teaching, mentoring, or community service. ComputingEdge: Do you have any learning expe- AWARD: Certificate and a $2,000 honorarium riences you could share that could benefi t those PRESENTATION: Annually presented at the ACM/IEEE International Symposium on Microarchitecture just starting their careers? NOMINATION SUBMISSION: This award requires 3 endorsements. Nominations are being accepted electronically: www.computer.org/web /awards/rau Fisher: During the same semester as an under- CONTACT US: Send any award-related questions to [email protected]. graduate at the University of California, Irvine, I www.computer.org/web/awards/rau took a class in data structure techniques from www.computer.org/computingedge 47 CAREER OPPORTUNITIES

DATA MESSAGING MIDDLEWARE SPE- MC:482-C32-D44, Detroit, MI 48265, Transform raw data into insightful infor- CIALIST, Warren, MI, General Motors. Ref#35904. mation for critical decision making using Monitor, support, debug &troubleshoot Tableau &MS Power BI. Run data qlty GSSM (Global Sales, Services & Market- HUMAN RESOURCES DATA STRATE- scans &conduct data profiling using IBM ing), GPSC (Global Part Supply Chain) GIST, General Motors, Detroit, MI. Desg, Cognos, IBM InphoSphere Information departments’ app interfaces supporting dvlp &execute apps architectures (using Servers, Information Analyzer &Atac- vehicle mfg &aftersales. Support ICVM PeopleSoft &Oracle Developer), policies, cama Data Qlty analyzer tools, to identify (Inter Continent Vehicle Movement) in- practices &procedures to properly man- outlier patterns or anomalies in systems voice processing &GWM (Global Warranty age the full lifecycle for GM People’s Data data &determine structured solutions for Management) warranty claim processing. (employee, contractor, suppliers, &per- unstructured fuzzy data problems. Bach- Execute day to day operation tasks incldg son of interest) over main sys of record elor, Computer Engrg, Systems Engrg, In- anlys of failed interfaces in ETL tools such of all GM ops. Strategize &ensure imple- formation Technology or related. 12 mos as IBM Datastage, ICAN, SRE &JCAPS. mentation of global HR data qlty mgmt exp as HR Data Strategist, HR Data Scien- Perform &manage day-today monitoring rules’ definitions, discovery processes, tist or related, executing apps architec- sys, implementing responsive actions, remediation policies, &standards incldg tures using PeopleSoft, &procedures to &reporting on the IBM Information Bus data dictionary with global HR, Finance manage full lifecycle People’s Data (em- messaging sys. Deploy interfaces on IBM &IT, ensuring one global data definition ployee, contractor, suppliers, &person Datastage Information Server 11.3. Re- regardless of sys of record. Establish of interest) over main sys of record for new certificates on EAI operated &owned global Oracle PeopleSoft Data Gover- a region, establishing MDM &DQM pro- servers. Master, Computer Science, Com- nance, MDM (Master Data Mgmt) &DQM cesses, &running data qlty scans &con- puter Networking or related. 6 mos exp (Data Qlty Monitoring) processes to en- ducting data profiling using IBM Cognos as Computer Programmer, Information sure qlty, integrity &security of people to identify outlier patterns or anomalies Technology Analyst or related, execut- data in HR systems. Establish standards in systems data &determine structured ing day to day operation tasks incldg an- &procedures for people data life cycle solutions for unstructured fuzzy data lys of failed interfaces in ETL tools such mgmt CRUD standards to ensure data problems. Mail resume to Ref#46806, GM as IBM Datastage, ICAN, SRE &JCAPS. is recorded, stored, extracted &retired Global Mobility, 300 Renaissance Center, Mail resume to Alicia Scott-Wears, GM from HR systems in a consistent manner. MC:482-C32-D44, Detroit, MI 48265. Global Mobility, 300 Renaissance Center,

TECHNICAL SOFTWARE TECHNICAL Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for has openings for TECHNICAL ANALYST- SOFTWARE TECHNICAL SUPPORT DEVELOPER ANALYST positions in Orlando, FL. positions in Morrisville, NC. positions in Orlando, FL. Job duties include: Deliver post-sales Job duties include: Design, develop, Job duties include: Analyze user support and solutions to the Oracle troubleshoot and/or test/QA requirements to develop, implement, customer base while serving as an software. and/or support Oracle’s global advocate for customer needs. infrastructure. Apply by e-mailing resume to Apply by e-mailing resume to [email protected], Apply by e-mailing resume to [email protected], referencing 385.16880. [email protected], referencing 385.19791. referencing 385.19760. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity.

IT PROJECT MGRS & SOFTWARE DEVS 7. Solid understanding of SQL, Oracle SWL up to 100% of the time anywhere in the SOUGHT. IT Proj Mgrs req MS or equiv preferred. Exp w/ project implmtn using United States. Must have Bach deg or in CS, Comp Apps, Engg or rltd & 12 mos’ technologies such as Java, JSP, Servlet, the foreign equiv in Comp Sci, or related relevant indus exp. Will also accept BS JDBC, XML/XSLT, & HTML. Multiple po- w/2 yrs of exp Designing, Building & Sup- or equiv in above fields & 5 yrs’ progres- sitions avail. Resumes to: Sapna Rao, HR porting integration interfaces that meets sively responsible relevant indus exp. S/ Manager, 850 Technology Way, Liber- business requirements using WebSphere ware Devs req BS or equiv in above fields tyville, IL 60048. Process Server, WebSphere Application & 2yrs’ relevant indus exp. All positions Server, WebSphere Portal Server v8.x, based out of San Jose, CA HQ & subj to re- PROGRAMMER ANALYST: Analyze, IBM Rational Software, BlueMix Cloud Plat- loc to various unanticipated US locations. Develop & Design User Interface us- form, DevOps Tools, IBM Sterling Platform, Mail resumes to: HR Mgr, Happiest Minds ing knowledge in Oracle Apps(11i & R12) Web Services, SOAP, Databases (IBM DB2 Technologies Pvt. Ltd., 2051 Junction Ave, AP, AR, GL, FA, CM modules, Oracle 8.x,9.x), scripting skills in JavaScript. Basic Ste 208, San Jose, CA 95131. Developer 9i/10g/11g, OAF, Forms10g, knowledge of Java J2EE (JSF, Struts, Hi- Reports10g, XML publisher, PL/SQL de- bernate, EJB, JDBC, REST, JSP, JMS, JUnit) INTERSHOP DEVELOPER, Brightstar US, veloper, SQL*Loader, UNIX, Java, TOAD, and different communication protocols Inc. (Libertyville, IL) to assist in the dvlpmt WF Builder ,video/audio streaming API’s, (TCP/IP, HTTP, FTP, SMTP). Send resume: of Brightstar’s next generation e-Com- MarkView, Advisor, Informatica, Visual Ba- Indotronix Int’l Corp, Recruiting (AE), 331 merce processes & srvcs for customers. sic, ASP .Net, C#.Net, SQL Server & Busi- Main St, Poughkeepsie, NY 12601. Align to dsgn reqmts of the Brightstar sys- ness Objects. Must be willing to travel & tms (applicable) to clients are managed & relocate to unanticipated client locations SYSTEM ANALYST: BA in comp science successful for applics locally, regionally throughout US. Reqs MS in comp sci, eng + 6 mo. exp in web dev., online-store, or globally dvlpd. Identify techn’l issues or rel. Mail resumes to Nitya Software mngmt system, CRM req’d. Design syst & provide resolution steps. Dsgn & dvlp Solutions Inc. 3100 Mowry Avenue, Suite for inventory, price, orders, dev. online systms using Intershop platform. Reqmts: 205, Fremont, CA 94538. store mngmt syst, config new softw, im- Bachelor’s deg, or foreign equiv, in Comp plement quality standards, analyze data, Sci or rltd field. Min. 5 yrs exp in job or over- LEAD INTEGRATION DEVELOPER F/T design procedures. Mail CV to UF Corpo- all s/ware dvlpmt exp w/ a min. 3 yrs work- (Poughkeepsie, NY) Position involves ration, Inc. 1601 S Main Street, High Point, ing w/ Intershop Enfinity 6.x or Intershop travel to various unanticipated worksites NC 27260.

QA ANALYST SOFTWARE Oracle America, Inc. Oracle America, Inc. has openings for has openings for CLOUDERA, INC. QA SOFTWARE IS RECRUITING FOR OUR PALO ALTO, CA ANALYST DEVELOPER OFFICE: positions in Redwood Shores, CA positions in Bedford, MA. SOFTWARE ENGINEER #37532: Design, develop, support & Job duties include: Responsible for maintain software components developing, applying and maintaining Job duties include: Design, develop, in Cloudera’s product offering. quality standards for company products troubleshoot and/or test/QA CUSTOMER OPERATION EN- GINEER #37441: Investigate with adherence to both internal and software. product related issues both for external standards. individual customers & for common trends that may arise Apply by e-mailing resume to Apply by e-mailing resume to MAIL RESUME WITH JOB [email protected], [email protected], CODE # TO referencing 385.19772. CLOUDERA referencing 385.19855. ATTN: HR 1001 PAGE MILL RD., BLDG. 3 Oracle supports workforce diversity. Oracle supports workforce diversity. PALO ALTO, CA 94304

www.computer.org/computingedge 49 CAREER OPPORTUNITIES

SENIOR QUALITY ASSURANCE AN- Shell Scripts for UNIX platform to de- &feature functionality on passenger ALYST, General Motors, Detroit, MI. ploy &monitor application. Master, Elec- vehicles, using Java/J2EE, Oracle, Test, validate, &perform soft updates trical, Electronics or, Computer Engrg Mysql, AngularJS, &Ajax. Perform soft- Over the Air on General Motors vehicle or related. 36 mos exp as Quality Con- ware, code &reqmts analysis, software internal module such as OnStar core sultant, Software Engineer, Quality As- review, identification of code metrics, module, &external modules such as the surance Analyst or related, performing system risk, software reliability &OOPS center stack module &the engine con- soft test estimation, test planning, test analysis, &aspect oriented program- trol unit through soft platform. Perform designing, test optimization &test exe- ming. Mentor younger dvlpmt teams, functional testing, back-end testing, cution. Will accept bachelor’s or foreign &review software being dvlpd to ensure installation testing &sys integration equiv degree, in Electrical, Electronic, the highest standards in code qlty, per- testing. Train Qlty Assurance Analysts Computer Engrg or related, followed by formance, security, &overall delivery to &Jr Analysts on vehicle domains, soft at least 5 yrs of progressive exp in the reqmts. Collaborate with Software Ar- updates reqmts, &Vehicle soft platform. specialty, in lieu of required education chitects to contribute to multilayered Document &recommend test process, &exp. Will also accept any equally suit- software solution architecture &drive design vehicle module test artifacts, able combination of education, training, design framework alignment. Dvlp, de- &functional flows. Dvlp test plans, test &/or exp which would qualify applicant sign &unit test software applications. reports, &qlty metrics. Design vehicle to perform job offered. Mail resume to Dvlp, design &unit test software appli- module test artifacts such as test plan, Ref#35608, GM Global Mobility, 300 cations. Master, Information Technol- test report &qlty metrics. Maintain man- Renaissance Center, MC:482-C32-D44, ogy, Computer Science, or related. 6 ual &automation test suites &update test Detroit, MI 48265. mos exp as Software Developer, Com- scripts for changes &enhancements. puter Programmer, or related, gathering Mentor team members on performing SENIOR SOFTWARE ENGINEER-CON- reqmts &executing &supporting full life- test activities. Execute web service NECTED CAR, General Motors, Detroit, cycle vehicle diagnostic data collection testing using SOAP user interface tool, MI. Gather reqmts, perform, &execute sys &OTA reflash applications dvlpmt, &automation testing using Selenium full lifecycle Vehicle Service Mgmt Sys- using Java, Oracle &Ajax. Mail resume WebDriver 2.0, TestNG, JUnit, &Jen- tem &Over the Air (OTA) reflash appli- to Ref#2875, GM Global Mobility, 300 kins tools. Perform database validation cations dvlpmt incldg vehicle modules Renaissance Center, MC:482-C32-D44, by writing SQL queries &PL/SQL. Dvlp Detroit, MI 48265.

SOFTWARE TECHNOLOGY TECHNOLOGY Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for SYSTEMS has openings for SOFTWARE ANALYST- CONSULTING DEVELOPERS SUPPORT PROJECT positions in Broomfield, Colorado. positions in Redwood Shores, CA. DIRECTOR positions in Redwood Shores, CA. Job duties include: As a member of the Job duties include: Analyze user requirements, procedures, and problems to software engineering division, assist in automate or improve existing systems and Job duties include: Plan, initiate, and defining and developing software for review computer system capabilities, manage information technology (IT) tasks associated with the developing, workflow, and scheduling limitations. Some projects. Lead and guide the work of debugging or designing of software travel may be required to work on projects technical staff. Travel to various, applications or operating systems. at various, unanticipated sites throughout unanticipated sites throughout the U.S. the United States. May telecommute from required. May telecommute from home. Apply by e-mailing resume to home. [email protected], Apply by e-mailing resume to Apply by e-mailing resume to referencing 385.19628. [email protected], [email protected], Oracle supports workforce diversity. referencing 385.16903. referencing 385.7516. Oracle supports Oracle supports workforce diversity. workforce diversity.

50 ComputingEdge April 2017 CAREER OPPORTUNITIES

Cisco Systems, Inc. is accepting resumes for the following positions: ALPHARETTA, GA: Network for the support and delivery of Advanced manufacturing designs. Technical Engineer (Ref.# ALP3): Responsible Services to company’s major accounts. Marketing Engineer (Ref.# SJ15): for the operational support of internal IRVINE, CA: Consulting Systems Responsible for enlarging company’s network systems. Engineer (Ref.# IRV15): Provide market and increasing revenue by AUSTIN, TX: Customer Support specific end-to-end solutions andmarketing, supporting, and promoting Engineer (Ref.# AUS5): Responsible architecture consulting, technical and sales company’s technology to customers. IT for providing technical support regarding support for major account opportunities Engineer (Ref.# SJ7): Responsible for the company’s proprietary systems and at the theater, area, or operation level. development, support and implementation software. Telecommuting Permitted. of major system functionality of company’s proprietary networking products. BELLEVUE/SEATTLE, WA: Product JACKSONVILLE, FL: Solutions Manager, Engineering (Ref.# Manager (Ref.# SEA384): Develop Architect (Ref.# JAC2): Responsible SJ4): Schedule and evaluate the the internal business case and customer for IT advisory and technical consulting resources required for multiple projects in value proposition for new offers and services development and delivery. Travel terms of human resources and hardware changes to existing offers, based on an may be required to various unanticipated equipment allocation. Network understanding of the market conditions, locations throughout the United States. Consulting Engineer (Ref.# customer decision drivers, and competitive LAWRENCEVILLE, GA: Network SJ107): Responsible for the support dynamics. Consulting Engineer (Ref.# LV16): and delivery of Advanced Services to BEAVERTON, OR: Software Responsible for the support and delivery company’s major accounts. Travel may be Engineer (Ref.# BEA1): Responsible of Advanced Services to company’s major required to various unanticipated locations for the definition, design, development, accounts. Telecommuting permitted. throughout the United States. Technical test, debugging, release, enhancement or RESEARCH TRIANGLE PARK, NC: Marketing Engineer (Ref.# maintenance of networking software. Software Engineer (Ref.# RTP3): SJ178): Responsible for enlarging BOXBOROUGH, MA: Technical Responsible for the definition, design, company’s market and increasing revenue Lead/Leader (Ref.# BOX3): Lead development, test, debugging, release, by marketing, supporting, and promoting engineering groups on projects to design, enhancement or maintenance of networking company’s technology to customers. Travel develop or test hardware or software software. Software Engineer (Ref.# may be required to various unanticipated products. Technical Lead/Leader RTP773): Responsible for the definition, locations throughout the United States. (Ref.# BOX23): Lead engineering design, development, test, debugging, Network Engineer (Ref.# SJ57): groups on projects to design, develop release, enhancement or maintenance Responsible for the operational support or test hardware or software products. of networking software. Telecommuting of internal network systems. Network Telecommuting permitted. Software permitted. Customer Support Consulting Engineer (Ref.# SJ9): Engineer (Ref.# BOX1): Responsible Engineer (Ref.# RTP1): Responsible Responsible for the support and delivery for the definition, design, development, for providing technical support regarding of Advanced Services to company’s major test, debugging, release, enhancement or the company’s proprietary systems and accounts. Data Engineer (Ref.# maintenance of networking software. software. SJ829): Implement complex big data projects with large sets of data collected BOSTON, MA: Product Manager, RICHARDSON, TX: Software from various endpoints in the network Engineering (Ref.# BOS4): Engineer (Ref.# RIC3): Responsible and turn this information into useful Responsible for managing the development for the definition, design, development, insights. Corporate Development and implementation of new product test, debugging, release, enhancement or Engineer (Ref.# SJ368): Configure introduction engineering activities to meet maintenance of networking software. and troubleshoot routers and networking production launch schedules, quality and SAN DIEGO, CA: User Experience equipment. Network Consulting cost objectives. Telecommuting permitted. Specialist (Ref.# SD4): Identify user Engineer (Ref.# SJ525): Responsible DENVER, CO: Software/QA interaction requirements and develop user for the support and delivery of Advanced Engineer (Ref.# DEN2): Debug experience interface specifications and Services to company’s major accounts. software products through the use of guidelines. Telecommuting permitted. systematic tests to develop, apply, and SAN FRANCISCO, CA: CNG WALTHAM, MA: Software Engineer maintain quality standards for company Member of Technical Staff (Ref.# (Ref.# WAL2): Responsible for the products. Software Engineer (Ref.# SF9): Design, implement, and test definition, design, development, test, DEN7): Responsible for the definition, software for a web application used by our debugging, release, enhancement or design, development, test, debugging, customers for IT management. maintenance of networking software. release, enhancement or maintenance of networking software. Network SAN JOSE/MILPITAS/SANTA PLEASE MAIL RESUMES WITH Consulting Engineer (Ref.# CLARA, CA: Customer Support REFERENCE NUMBER TO CISCO DEN8): Responsible for the support and Engineer (Ref.# SJ3): Responsible SYSTEMS, INC., ATTN: G51G, 170 W. delivery of Advanced Services to company’s for providing technical support regarding Tasman Drive, Mail Stop: SJC 5/1/4, San major accounts. the company’s proprietary systems and Jose, CA 95134. No phone calls please. software. Test Engineer (Ref.# Must be legally authorized to work in the HOUSTON, TX: Network Consulting SJ16): Build test equipment and test U.S. without sponsorship. EOE. Engineer (Ref.# HOU4): Responsible diagnostics for new products based on www.cisco.com

www.computer.org/computingedge 51 CAREER OPPORTUNITIES

Microsoft Corporation currently has the following openings (job opportunities available at all levels, e.g., Principal, Senior and Lead levels):

ALPHARETTA, GA and international travel up to 25%. https:// tic travel up to 25%. Telecommuting per- Consultant: Deliver design, planning, and jobs-microsoft.icims.com/jobs/6753/go/job mitted. https://jobs-microsoft.icims.com/ jobs/7131/go/job implementation services that provide IT MOUNTAIN VIEW, PALO ALTO, solutions to customers and partners. Re- SUNNYVALE, CA Solutions Sales Specialist: Enhance the quires travel throughout U.S. up to 75% with Microsoft customer relationship from a ca- Applied Scientist: Utilize knowledge in ap- work to be performed at various worksites pability development perspective by articu- plied statistics and mathematics to handle throughout the U.S. https://jobs-microsoft. lating the value of our services and solutions large amounts of data using various tools. icims.com/jobs/7181/go/job and identifying competition gaps in target- http://bit.ly/MSJobs_Data_Applied_Science ed accounts. Requires domestic travel up TEMPE, AZ Data Scientist: Manipulate large volumes of to 25%. Telecommuting permitted. https:// Cloud Solution Architects/Solution Archi- data, create new and improved techniques jobs-microsoft.icims.com/jobs/7927/go/job tects: Architect software, platform, services, and/or solutions for data collection, manage- hardware or technology solutions. http://bit. ment and usage. http://bit.ly/MSJobs_Data_ SAN FRANCISCO, CA ly/MSJobs_Tech_Solns Applied_Science Data Scientist: Manipulate large volumes of data, create new and improved techniques IRVINE, CA Design Verification/Validation Engineers: Responsible for ensuring the quality of Mi- and/or solutions for data collection, manage- Premier Field Engineers: Provide technical crosoft hardware products. http://bit.ly/ ment and usage. http://bit.ly/MSJobs_Data_ support to enterprise customers, partners, MSJobs_Hardware_Design_Verification_Eng Applied_Science internal staff or others on mission critical Designers: Develop user interface and user issues experienced with Microsoft tech- Hardware Dev., Test or Design Engineers, interaction designs, prototypes and/or con- nologies. Requires domestic travel up to Hardware Engineers, Electrical Engineers, cepts for business productivity, entertain- 75% with work to be performed at various Design Engineers (all levels, including ment or other software or hardware applica- unknown worksites throughout the U.S. Leads and Managers): Design, implement tions. http://bit.ly/MSJobs_Design Telecommuting permitted. http://bit.ly/MS- and test computer hardware that add stra- Jobs_Support_Delivery tegic value to the company. (http://bit.ly/ Premier Field Engineers: Provide technical MSJobs_Hardware_Dev_Eng)(http://bit.ly/ support to enterprise customers, partners, Premier Field Engineer: Provide technical MSJobs_Electrical_Eng) internal staff or others on mission critical support to enterprise customers, partners, issues experienced with Microsoft tech- internal staff or others on mission critical Hardware Dev., Test or Design Engineers, nologies. Requires domestic travel up to issues experienced with Microsoft technolo- Hardware Engineers, Electrical Engineers, 75% with work to be performed at various gies. Requires travel up to 50% with work to Design Engineers (all levels, including unknown worksites throughout the U.S. be performed at various unknown worksites Leads and Managers): Design, implement Telecommuting permitted. http://bit.ly/MS- throughout the U.S. Telecommuting per- and test computer hardware. Requires Do- Jobs_Support_Delivery mitted. https://jobs-microsoft.icims.com/ mestic and International travel up to 25%. jobs/7246/go/job (http://bit.ly/MSJobs_Hardware_Dev_Eng) Solutions Sales Professional/Specialist/ (http://bit.ly/MSJobs_Electrical_Eng) Technology Solutions Professionals: En- Technology Solutions Professional: Improve hance the Microsoft customer relationship the Enterprise Mobility business metrics Solutions Sales Professional/Specialist/ from a capability development perspective (revenue and scorecard) through excellence Technology Solutions Professionals: En- by articulating the value of our services and in technical sales strategy and execution. hance the Microsoft customer relationship solutions and identifying competition gaps Requires travel throughout U.S. up to 50% from a capability development perspective in targeted accounts. http://bit.ly/MSJobs_ with work to be performed at various un- by articulating the value of our services and Solution_Sales known worksites throughout the U.S. Tele- solutions and identifying competition gaps commuting permitted. https://jobs-micro- in targeted accounts. http://bit.ly/MSJobs_ Solution Specialist: Enhance the Microsoft soft.icims.com/jobs/7144/go/job Solution_Sales customer relationship from a capability development perspective by articulating the value LOS ANGELES, CA Solutions Specialist: Enhance the Micro- soft customer relationship from a capabili- of our services and solutions and identifying Technical Account Managers: Assure pro- ty development perspective by articulating competition gaps in targeted accounts. Re- ductive use of Microsoft technologies, fo- the value of our services and solutions and quires domestic and international travel up cusing on delivery quality through planning identifying competition gaps in targeted ac- to 25%. Telecommuting permitted. https:// and governance. Telecommuting permitted. counts. Telecommuting permitted. https:// jobs-microsoft.icims.com/jobs/7708/go/job http://bit.ly/MSJobs_Delivery_Relation- jobs-microsoft.icims.com/jobs/7383/go/job Solutions Specialist: Enhance the Micro- ship_Mgmt Sr. Quality and Reliability Engineer – Pack- soft customer relationship from a capabili- Senior Consultant: Deliver design, planning, aging: Design, implement and test computer ty development perspective by articulating and implementation services that provide IT hardware products that add strategic val- the value of our services and solutions and solutions to customers and partners. Re- ue to the company. https://jobs-microsoft. identifying competition gaps in targeted ac- quires domestic and international travel up icims.com/jobs/7704/go/job counts. Requires domestic travel up to 25%. to 50%. Telecommuting permitted. https:// https://jobs-microsoft.icims.com/jobs/7672/ jobs-microsoft.icims.com/jobs/7139/go/job CAMBRIDGE, MA go/job Technical Evangelist: Collaborate with sales Senior Technical Evangelist: Collaborate GUAYNABO, PR teams to understand customer require- with sales teams to understand customer Premier Field Engineer, Senior: Provide ments, promote the sale of products, and requirements, promote the sale of products, technical support to enterprise customers, provide sales support. Requires domestic and provide sales support. Requires domes-

52 ComputingEdge April 2017 CAREER OPPORTUNITIES

partners, internal staff or others on mission CHICAGO, IL cusing on delivery quality through planning critical issues experienced with Microsoft Cloud Solution Architects/Solution Archi- and governance. http://bit.ly/MSJobs_De- technologies. Travel up to 50% with work to tects: Architect software, platform, ser- livery_Relationship_Mgmt be performed at various unknown worksites vices, hardware or technology solutions. FARGO, ND throughout the U.S. Telecommuting per- Telecommuting permitted. http://bit.ly/ Support Engineers / Escalation Engi- mitted. https://jobs-microsoft.icims.com/ MSJobs_Tech_Solns jobs/7201/go/job neers: Install, configure, support and trou- Cloud Solution Architect: Responsible for bleshoot issues related to Microsoft technol- FT. LAUDERDALE, FL architecting software, platform, services, ogies. http://bit.ly/MSJobs_Support_Eng Business Managers and Business Devel- hardware or technology solutions. Requires ISELIN, NJ opment Managers/Business Development domestic and international travel up to 50%. and Strategy Analyst/Manager: Develop Telecommuting permitted. https://jobs-mic- Cloud Solution Architects/Solution Archi- business opportunities for sales of soft- rosoft.icims.com/jobs/6757/go/job tects: Architect software, platform, services, ware and services. http://bit.ly/MSJobs_ Solution Specialist: Enhance Microsoft hardware or technology solutions. http://bit. Business_Development customer relationship from capability ly/MSJobs_Tech_Solns Solutions Sales Professional/Specialist/ development perspective by articulat- Cloud Solution Architects/Solution Archi- Technology Solutions Professionals: En- ing value of services and solutions and tects: Architect software, platform, ser- hance the Microsoft customer relationship identifying competition gaps in targeted vices, hardware or technology solutions. from a capability development perspective accounts. Requires domestic travel up to Telecommuting permitted. http://bit.ly/ by articulating the value of our services and 25%. https://jobs-microsoft.icims.com/ MSJobs_Tech_Solns solutions and identifying competition gaps jobs/6786/go/job Cloud Solution Architect: Architect and de- in targeted accounts. http://bit.ly/MSJobs_ DOWNERS GROVE, IL ploy Microsoft cloud solutions for custom- Solution_Sales ers. Requires travel up to 25% with work to Associate Architect: Architect software, be performed at various unknown worksites Business Operations Associate: Analyze platform, services, hardware, or technolo- throughout the U.S. https://jobs-microsoft. data to identify business insights related to gy solutions. Domestic travel required up icims.com/jobs/7164/go/job business performance and growth. Requires to 50%. Telecommuting permitted. https:// international travel up to 25%. https:// jobs-microsoft.icims.com/jobs/6776/go/job Premier Field Engineer, SharePoint: Provide jobs-microsoft.icims.com/jobs/7280/job technical support to enterprise customers, Delivery Manager: Responsible for engaging SOUTHFIELD, MI partners, internal staff or others on mission with and managing accounts of customers Solutions Sales Professional/Specialist/ critical issues experienced with Microsoft utilizing Microsoft Dynamics AX and CRM Technology Solutions Professionals: En- technologies. Requires travel up to 50% with business solution. Requires domestic and hance the Microsoft customer relationship work to be performed at various unknown international travel up to 75%. https:// from a capability development perspective worksites throughout the U.S. Telecommut- jobs-microsoft.icims.com/jobs/6867/go/job by articulating the value of our services and ing permitted. https://jobs-microsoft.icims. com/jobs/7350/job Senior Technical Advisor: Provide technical solutions and identifying competition gaps advice and support on issues experienced in targeted accounts. http://bit.ly/MSJobs_ Solution Architect: Engage with strategic with Microsoft technologies. Telecommut- Solution_Sales customers to solve their business needs by ing permitted. https://jobs-microsoft.icims. EDINA, MN conceiving, designing, and implementing com/jobs/7250/go/job digitally transformative technical architec- Solutions Sales Professional/Specialist/ ture on Microsoft’s public cloud offering Solution Architect: Architect software, Technology Solutions Professionals: En- (Azure). Telecommuting permitted. https:// platform, services, hardware or technology hance the Microsoft customer relationship jobs-microsoft.icims.com/jobs/7242/go/job solutions. Requires international travel up from a capability development perspective to 25%. Telecommuting permitted. https:// by articulating the value of our services and NEW YORK, NY jobs-microsoft.icims.com/jobs/6815/job solutions and identifying competition gaps Research Software Development Engi- Solution Specialist: Enhance the Microsoft in targeted accounts. http://bit.ly/MSJobs_ neers (all levels): Responsible for conduct- customer relationship from a capability Solution_Sales ing applied research into new products and development perspective by articulating CHARLOTTE, NC services through software engineering tech- the value of our services and solutions and niques. http://bit.ly/MSJobs_Research_ identifying competition gaps in targeted ac- Support Engineers / Escalation Engineers: Software_Engineer Install, configure, support and troubleshoot counts. Telecommuting permitted. https:// Solutions Sales Professional/Specialist/ jobs-microsoft.icims.com/jobs/7398/go/job issues related to Microsoft technologies. http://bit.ly/MSJobs_Support_Eng Technology Solutions Professionals: En- Solutions Sales Specialist, Education: hance the Microsoft customer relationship Enhance Microsoft customer relationship Senior Technical Account Manager: As- from a capability development perspective from capability development perspective sure productive use of Microsoft technol- by articulating the value of our services and by articulating value of services & solutions ogies, focusing on delivery quality through solutions and identifying competition gaps & identifying competition gaps in targeted planning and governance. Requires travel in targeted accounts. http://bit.ly/MSJobs_ accounts. Requires travel up to 25% with up to 25% with work to be performed at Solution_Sales work to be performed at various unknown various unknown worksites throughout the worksites throughout the U.S. and Latin U.S. Telecommuting permitted. https://jobs America. https://jobs-microsoft.icims.com/ -microsoft.icims.com/jobs/6825/go/job jobs/6821/go/job Technical Account Managers: Assure productive use of Microsoft technologies, fo- CONTINUED ON NEXT PAGE…

www.computer.org/computingedge 53 CAREER OPPORTUNITIES

…CONTINUED FROM PREVIOUS PAGE ing permitted. https://jobs-microsoft.icims. mitted. https://jobs-microsoft.icims.com/ com/jobs/6750/go/job jobs/7178/go/job Technical Evangelist: Collaborate with sales MCALLEN, TX Account Technology Strategist: Identify and teams to understand customer requirements, Hardware Dev., Test or Design Engineers, analyze internal client and partner business promote the sale of products, and provide Hardware Engineers, Electrical Engineers, needs, and translate needs into business re- sales support. Requires domestic travel up Design Engineers (all levels, including quirements and value-added solutions and to 25%. Telecommuting permitted. https:// Leads and Managers): Design, implement solution roadmaps. Requires travel up to jobs-microsoft.icims.com/jobs/7746/go/job and test computer hardware that add stra- 25% with work to be performed at various Technology Solutions Professional Devices: tegic value to the company. (http://bit.ly/ worksites throughout the U.S. Telecommut- Enhance the Microsoft customer relationship M ing permitted. https://jobs-microsoft.icims. SJobs_Hardware_Dev_Eng)(http://bit.ly/ from a capability development perspective MSJobs_Electrical_Eng) com/jobs/6766/job by articulating the value of our services and Hardware Dev., Test or Design Engineers, Associate Architect: Architect solutions by solutions and identifying competition gaps Hardware Engineers, Electrical Engineers, mapping common customer business prob- in targeted accounts. Telecommuting per- Design Engineers (all levels, including lems to reusable end-to-end technology mitted. https://jobs-microsoft.icims.com/ Leads and Managers): Design, implement solutions. Requires domestic and interna- jobs/7065/job and test computer hardware. Requires Do- tional travel up to 50%. Telecommuting per- mestic and International travel up to 25%. mitted. https://jobs-microsoft.icims.com/ WILSONVILLE, OR (http://bit.ly/MSJobs_Hardware_Dev_Eng) jobs/7170/go/job Hardware Dev., Test or Design Engineers, (http://bit.ly/MSJobs_Electrical_Eng) Cloud Solution Architect: Architect and de- Hardware Engineers, Electrical Engineers, ploy Microsoft cloud solutions for customers. Design Engineers (all levels, including RESTON, VA Leads and Managers): Design, implement Requires travel up to 25% with work to be Technical Account Managers: Assure pro- and test computer hardware that add stra- performed at various unknown worksites ductive use of Microsoft technologies, fo- tegic value to the company. (http://bit.ly/ throughout the U.S. Telecommuting per- cusing on delivery quality through planning MSJobs_Hardware_Dev_Eng)(http://bit.ly/ mitted. https://jobs-microsoft.icims.com/ and governance. http://bit.ly/MSJobs_Deliv- MSJobs_Electrical_Eng) jobs/6802/job ery_Relationship_Mgmt Hardware Dev., Test or Design Engineers, Principal Solution Specialist: Enhance the REDMOND, WA Microsoft customer relationship from a ca- Hardware Engineers, Electrical Engineers, pability development perspective by articu- Design Engineers (all levels, including Applied Scientist: Utilize knowledge in ap- Design, implement lating the value of our services and solutions Leads and Managers): plied statistics and mathematics to handle and test computer hardware. Requires Do- and identifying competition gaps in targeted large amounts of data using various tools. mestic and International travel up to 25%. accounts. Requires domestic travel up to http://bit.ly/MSJobs_Data_Applied_Science ( 25% to various unknown locations. Tele- http://bit.ly/MSJobs_Hardware_Dev_Eng) Business Managers and Business Devel- commuting permitted. https://jobs-micro- (http://bit.ly/MSJobs_Electrical_Eng) opment Managers/Business Development soft.icims.com/jobs/7265/go/job HOUSTON, TX and Strategy Analyst/Manager: Develop business opportunities for sales of software Solution Sales Professional: Enhance the Technical Account Managers: Assure produc- and services. http://bit.ly/MSJobs_Busi- Microsoft customer relationship from a ca- tive use of Microsoft technologies, focusing on ness_Development pability development perspective by articu- delivery quality through planning and gover- lating the value of our services and solutions nance. Roving Employee - requires travel up Business Process Manager: Responsible for and identifying competition gaps in targeted to 100% with work to be performed at various the design, implementation, and release of accounts. Requires travel up to 25% with unknown worksites throughout the U.S. http:// programs or projects. http://bit.ly/MSJobs_ work to be performed at various unknown bit.ly/MSJobs_Delivery_Relationship_Mgmt Fin_Plan_Analy_Contr worksites throughout the U.S. Telecommut- AUSTIN, TX Cloud Solution Architects/Solution Archi- ing permitted. https://jobs-microsoft.icims. tects: Architect software, platform, services, com/jobs/7243/go/job Partner Business Evangelist: Research market hardware or technology solutions. http://bit. Solution Specialist: Enhance the Microsoft conditions in regional area and gather infor- ly/MSJobs_Tech_Solns customer relationship from a capability mation to determine sales of Windows Azure. Requires Domestic and International travel up Consultants: Deliver design, planning, and development perspective by articulating implementation services that provide IT solu- the value of our services and solutions and to 25%. Telecommuting permitted. https:// jobs-microsoft.icims.com/jobs/7530/job tions to customers and partners. Requires identifying competition gaps in targeted domestic and international travel up to 25%. accounts. Requires travel up to 25% to per- IRVING, TX http://bit.ly/MSJobs_Technical_Delivery form work at various unknown customer Support Engineers / Escalation Engi- sites. https://jobs-microsoft.icims.com/ Consultants: Deliver design, planning, and neers: Install, configure, support and trou- jobs/7380/job implementation services that provide IT bleshoot issues related to Microsoft technol- solutions to customers and partners. Roving Solutions Specialist: Enhance the Micro- ogies. http://bit.ly/MSJobs_Support_Eng Employee—requires travel up to 100% with soft customer relationship from a capabili- Premier Field Engineer: Provide technical work to be performed at various unknown ty development perspective by articulating support to enterprise customers, partners, worksites throughout the U.S. http://bit.ly/ the value of our services and solutions and internal staff or others on mission critical MSJobs_Technical_Delivery identifying competition gaps in targeted ac- issues experienced with Microsoft technolo- counts. Requires travel up to 50% with work Consultants: Deliver design, planning, and gies. Requires travel up to 75% with work to to be performed at various unanticipated implementation services that provide IT be performed at various unknown worksites worksites throughout the U.S. Telecommut- solutions to customers and partners. Roving throughout the U.S. Telecommuting per- Employee—requires travel up to 100% with

54 ComputingEdge April 2017 CAREER OPPORTUNITIES

work to be performed at various unknown issues experienced with Microsoft technol- mitted. https://jobs-microsoft.icims.com/ worksites throughout the U.S. Telecommut- ogies. Roving Employee—requires travel jobs/6819/job ing permitted. http://bit.ly/MSJobs_Techni- up to 100% with work to be performed at Audience Engagement Development En- cal_Delivery various unknown worksites throughout the gineer: Responsible for building audience Content Developer/Engineer: (All levels, in- U.S. Telecommuting permitted. http://bit.ly/ measurement and engagement tools to cluding Leads and Managers) Responsible MSJobs_Support_Delivery support the organization’s effort in under- for the design, development, deployment, Research Software Development Engi- standing audience experiences relative to vision, and business strategy for content neers (all levels): Responsible for conduct- the communications and marketing objec- creation, acquisition, production, editorial, ing applied research into new products and tives. https://jobs-microsoft.icims.com/ and publishing activities at Microsoft. http:// services through software engineering tech- jobs/7347/job bit.ly/MSJobs_Content_Publishing niques. http://bit.ly/MSJobs_Research_ B2B Global Demand Center Analyst: Data Scientist: Manipulate large volumes of Software_Engineer Responsible for converting high vol- data, create new and improved techniques Researchers/Scientists: Conduct re- umes of complex data into actionable in- and/or solutions for data collection, man- search and lead research collaborations sights. https://jobs-microsoft.icims.com/ agement and usage. http://bit.ly/MSJobs_ that yield new insights, theories, analy- jobs/6862/go/job Data_Applied_Science ses, data, algorithms, and prototypes and Business Analytics Specialist: Provide in- Design Researchers: Develop user inter- that advance state-of-the-art of computer sights, facilitate data-driven testing and face and user interaction designs, proto- science and engineering, as well as gen- learning, and develop ways to optimize Win- types and/or concepts for business pro- eral scientific knowledge. http://bit.ly/MS- dows Devices Group (WDG) performance ductivity, entertainment or other software Jobs_Research marketing campaigns. https://jobs-micro- or hardware applications. http://bit.ly/MS- Service Engineers/Managers, Service soft.icims.com/jobs/7377/go/job Jobs_Design_Research Operations Engineers, and Systems/Op- Business Analytics Specialist: Identify Design Verification/Validation Engineers: erations Engineers/Site Reliability En- strategic areas of focus for insights and an- Responsible for ensuring the quality of Mi- gineer: Research, design, develop, and alytics. https://jobs-microsoft.icims.com/ crosoft hardware products. http://bit.ly/MS- test operating systems-level software, jobs/7137/go/job Jobs_Hardware_Design_Verification_Eng compilers, and network distribution software. (http://bit.ly/MSJobs_Service_Engi- Business Analytics Specialist: Provide deep Designers: Develop user interface and user neering) (http://bit.ly/MSJobs_IT_Serv_Eng) level data analytics and modeling for search interaction designs, prototypes and/or con- (http://bit.ly/MSJobs_IT_Serv_Ops) advertising platform. https://jobs-microsoft. cepts for business productivity, entertain- icims.com/jobs/6806/job Solutions Sales Professional/Specialist/ ment or other software or hardware applica- Business Analytics Specialist, Digital tions. http://bit.ly/MSJobs_Design Technology Solutions Professionals: En- hance the Microsoft customer relationship Technology & Operations: Analyze and opti- Evangelists/Technical Evangelists: Col- from a capability development perspective mize Business Intelligence systems and pro- laborate with sales teams to understand by articulating the value of our services and cedures. https://jobs-microsoft.icims.com/ customer requirements, promote the sale of solutions and identifying competition gaps jobs/7148/go/job products, and provide sales support. http:// in targeted accounts. http://bit.ly/MSJobs_ Business Development Analyst: Devel- bit.ly/MSJobs_Tech_Evangelist Solution_Sales op and analyze business opportunities for Evangelists/Technical Evangelists: Col- Support Engineers / Escalation Engi- sales of software and services. Requires laborate with sales teams to understand neers: Install, configure, support and trou- domestic and international travel up to customer requirements, promote the sale bleshoot issues related to Microsoft technol- 25%. https://jobs-microsoft.icims.com/ of products, and provide sales support. Re- ogies. http://bit.ly/MSJobs_Support_Eng jobs/7230/go/job quires travel throughout the U.S. up to 25%. Business Manager: Develop business op- http://bit.ly/MSJobs_Tech_Evangelist Architect: Deliver design, planning, and implementation services that provide IT solu- portunities for sales of software and ser- Hardware Dev., Test or Design Engineers, tions to customers and partners. https:// vices. Requires domestic and international Hardware Engineers, Electrical Engineers, jobs-microsoft.icims.com/jobs/7389/job travel up to 25%. https://jobs-microsoft. Design Engineers (all levels, including icims.com/jobs/6758/go/job Associate Architect: Plan, develop and Leads and Managers): Design, implement Channel Executive OEM: Build and main- and test computer hardware that add stra- market solutions that enable Microsoft Sell- ers, architects, and consultants to sell and tain business relationships with Original tegic value to the company. (http://bit.ly/ Equipment Manufacturers (OEM) to assess MSJobs_Hardware_Dev_Eng)(http://bit.ly/ deliver our products and services. https:// jobs-microsoft.icims.com/jobs/7257/go/job needs, determine requirements, and deliver MSJobs_Electrical_Eng) new Agreements and net new opportunity Hardware Dev., Test or Design Engineers, Associate Architect: Design and develop revenue. https://jobs-microsoft.icims.com/ Hardware Engineers, Electrical Engineers, business intelligence, big data and advanced jobs/6756/go/job Design Engineers (all levels, including analytics-based IT solutions for Microsoft Leads and Managers): Design, implement customers. Requires travel up to 25% with and test computer hardware. Requires Do- work to be performed at various unknown mestic and International travel up to 25%. worksites throughout the U.S. https:// (http://bit.ly/MSJobs_Hardware_Dev_Eng) jobs-microsoft.icims.com/jobs/6789/job (http://bit.ly/MSJobs_Electrical_Eng) Associate Architect: Architect and deploy Premier Field Engineers: Provide technical complex solutions mapping customer busi- support to enterprise customers, partners, ness problems with end-to-end technology internal staff or others on mission critical solutions. Requires domestic and international travel up to 50%. Telecommuting per- CONTINUED ON NEXT PAGE…

www.computer.org/computingedge 55 CAREER OPPORTUNITIES

…CONTINUED FROM PREVIOUS PAGE ware products that add strategic value to the jobs/7582/go/job company. https://jobs-microsoft.icims.com/ Senior Industry Technology Strategist: jobs/7046/job Channel Executive-Hosting: Enhance the Work with commercial segments to help Microsoft customer relationship from a ca- Premier Field Engineer: Provide technical them successfully transform and deploy Mi- pability development perspective by articu- support to enterprise customers, partners, crosoft technologies. https://jobs-microsoft. lating the value of our services and solutions internal staff or others on mission critical icims.com/jobs/7251/go/job and identifying competition gaps in target- issues experienced with Microsoft technol- Senior IT Service Operations: Partner within ed accounts. Requires travel throughout ogies. Requires travel throughout U.S. up to Engineering organizations and other stake- U.S. up to 25% with work to be performed 50% with work to be performed at various holders to be continually improving the at various unknown worksites throughout unknown worksites throughout the U.S. predictability of our service and enhancing the U.S. https://jobs-microsoft.icims.com/ https://jobs-microsoft.icims.com/jobs/7211/ our service offerings based on customer jobs/6775/go/job go/job feedback balanced against business pri- Cloud Solution Architect: Architect and Premier Field Engineer: Provide technical orities. https://jobs-microsoft.icims.com/ deploy Microsoft Azure cloud solutions for support to enterprise customers, partners, jobs/7400/job portfolio partners. https://jobs-microsoft. internal staff or others on mission critical Senior Patent Engineer: Perform analysis, icims.com/jobs/7684/go/job issues experienced with Microsoft technol- reverse engineering, testing and investiga- ogies. Requires international and domestic tions on software and information technol- Cloud Solution Architect: Architect soft- travel up to 50%. Telecommuting permitted. ware, platform, services, hardware or tech- ogy that are being deployed and used widely https://jobs-microsoft.icims.com/jobs/7037/ in the industry. https://jobs-microsoft.icims. nology solutions. Requires domestic and go/job international travel up to 25%. Telecommut- com/jobs/6773/job ing Permitted. https://jobs-microsoft.icims. Premier Field Engineer: Provide technical Senior Premier Field Engineer: Provide com/jobs/7067/go/job support to enterprise customers, partners, technical support to enterprise customers, internal staff or others on mission critical Cloud Solution Architect: Design, plan, and partners, internal staff or others on mission issues experienced with Microsoft tech- critical issues experienced with Microsoft implement cloud deployments to move cus- nologies. Travel up to 50% with work to be tomer workloads to Azure. Roving Employ- technologies. Position requires travel up performed at various unknown worksites to 25% with work to be performed at vari- ee—requires travel up to 100% with work to throughout the US. Telecommuting per- be performed at various unknown worksites ous unknown worksites throughout the U.S. mitted. https://jobs-microsoft.icims.com/ Telecommuting permitted. https://jobs-mic- throughout the U.S. Telecommuting per- jobs/6751/go/job mitted. https://jobs-microsoft.icims.com/ rosoft.icims.com/jobs/6891/go/job jobs/7184/go/job Premier Field Engineer: Provide technical Service Engineer 2: Responsible for network support to enterprise customers, partners, Data & Applied Scientist; Apply scientific designs, deployments, testing, and certifi- internal staff or others on mission critical cations; and operational escalation support methodology and algorithms to large vol- issues experienced with Microsoft technolo- umes of data using various tools. https:// of complex network infrastructure https:// gies. Requires travel up to 25% with work to jobs-microsoft.icims.com/jobs/7425/go/job jobs-microsoft.icims.com/jobs/7688/go/job be performed at various unknown worksites Design Researcher II: Develop user interface throughout the U.S. https://jobs-microsoft. Solution Specialist: Enhance the Microsoft and user interaction designs, prototypes icims.com/jobs/6752/go/job customer relationship from a capability and/or concepts for business productivity, development perspective by articulating Principal Researcher: Conduct research and the value of our services and solutions and entertainment or other software or hardware lead research collaborations that yield new applications. Requires domestic and inter- identifying competition gaps in targeted ac- insights, theories, analyses, data, algorithms counts. Requires travel up to 25% through- national travel up to 25%. https://jobs-mic- and prototypes and that advance state-of- rosoft.icims.com/jobs/6961/job out the U.S. https://jobs-microsoft.icims. the-art of computer science and engineer- com/jobs/7168/go/job IT Audit Group Manager: Manage a porting, as well as general scientific knowledge. folio of IT audit projects including driving Requires international and domestic travel Solutions Architect: Engage with strategic value to the business, quality, and execution up to 25%. https://jobs-microsoft.icims. customers to solve their business needs by excellence. Requires domestic travel up com/jobs/7064/job conceiving, designing, and implementing transformative technical architecture on to 50%. https://jobs-microsoft.icims.com/ Product Intelligence Manager: Build busi- jobs/7365/go/job data and cloud-based platforms and related ness process and business intelligence technologies. Roving Employee—requires Marketing Operations Automation Special- solutions based upon different telemetry travel up to 100% with work to be performed ist: Utilize modern big data management systems. https://jobs-microsoft.icims.com/ at various unknown worksites throughout tools, marketing analytics methods, CRM jobs/6788/go/job the U.S. Telecommuting permitted. https:// tools, and marketing automation technology Senior Camera Module Engineer: Specify, jobs-microsoft.icims.com/jobs/7387/go/job to prepare findings in marketing trends and design, implement, and verify hardware for Solutions Specialist (Technology Solutions assist the design of marketing plans. https:// microcomputers. Requires domestic and in- jobs-microsoft.icims.com/jobs/7355/job Professional): Enhance the Microsoft cus- ternational travel up to 25%. https://jobs-mi- tomer relationship from a capability devel- Optical Engineer II: Design tools, technolo- crosoft.icims.com/jobs/6949/go/job opment perspective by articulating the value gies and processes for next generation op- Senior Designer: Support the creation and of our services and solutions and identifying tical systems. https://jobs-microsoft.icims. improvement of user experiences for new competition gaps in targeted accounts. Re- com/jobs/7420/go/job products, and mature all successful indus- quires travel throughout the U.S. up to 25%. Packaging Engineer: Develop, qualify and trial design concepts through to product Telecommuting permitted. https://jobs-mic- implement structural packaging for hard- launch. https://jobs-microsoft.icims.com/ rosoft.icims.com/jobs/6840/job

56 ComputingEdge April 2017 CAREER OPPORTUNITIES

SOFTWARE ENGINEER, Warren, MI, Create oracle &mysql database scripts General Motors. Dvlp soft tools to sup- to debug issues related to vehicle data Support Engineer: Install, configure, port the dvlpmt of GM’s in Over the Air collection &reflash. Create Perl script for support and troubleshoot issues relat- (OTA) vehicle soft remote reflash dvlpmt app failover &maintain high availability ed to Microsoft technologies. Requires kit for 2018-2020 &beyond model year of app. Create &request infrastructure domestic travel up to 25% to perform passenger vehicle infotainment/radios team to provide CSS, firewall &IP tables work at various unknown customers’ &telematic systems, using C/C++, &per- rules for app to interact with vehicle &to worksites. https://jobs-microsoft.icims. forming root cause analysis on defects external cmpt svces. Master, Electrical com/jobs/6810/job &provide fixes. Use network app pro- or Electronics Engineering or related. 36 Support Planner: Build assisted support tocols incldg HTTP, HTTPS, FTP, TCP/IP mos exp as Network Infrastructure Engi- experiences for customers inquiring about to troubleshoot, debug &fix OTA issues. neer, Module Lead, Senior Software Engi- our product offerings or about an existing Coordinate &participate in architecture, neer, Software Engineer or related, using order. https://jobs-microsoft.icims.com/ reqmt, design, code, &test case reviews. &improving J2EE &Web Svces-based app jobs/7190/go/job Dvlp soft using C/C++ &porting to Ra- soft to analyze &perform OTA Reflash dio head units &Telematic Devices that &data collection of OnStar ECU in vehi- Technical Data Analyst: Analyze, vali- run on QNX, Linux &Android operating cles communicating through VCP. Will ac- date, and present data findings to answer systems. Use State Machine, Threads, cept bachelor’s or foreign equiv degree, high-impact business and product devel- Socket programming, Mutex, Semaphore in Electrical or Electronics Engineering opment questions. https://jobs-micro- &message queues for soft dvlpmt. Use or related, followed by at least 5 yrs pro- soft.icims.com/jobs/6768/job SqLite for storing sys status over ve- gressive exp in the specialty, in lieu of re- Technology Solutions Professional: hicle ignition cycles. Dvlp automation quired education &exp. Will also accept Enhance the Microsoft customer rela- test scripts using perl for remote reflash any equally suitable combination of ed- tionship from a capability development unit test execution. Dvlp CANoE vector ucation, training, &/or exp which would perspective by articulating the value of tool scripts to simulate Electronic Con- qualify applicant to perform job offered. our services and solutions and identifying trol Units. Use QNX Momentics as inte- Mail resume to Ref#2783, GM Global Mo- competition gaps in targeted accounts. grated dvlpmt environment. Use Vehicle bility, 300 Renaissance Center, MC:482- Requires domestic travel up to 25% to Spy vector tool to monitor messages on C32-D44, Detroit, MI 48265. perform work at unknown customer CAN bus. Bachelor, Computer Science, sites. https://jobs-microsoft.icims.com/ Computer Engrg, or related. 60 mos exp Warren, MI, Gen- jobs/7366/job STORAGE ENGINEER, as Software Engineer, Senior Software eral Motors Company. Perform opera- RENO, NV Engineer, Remote Reflash Engineer, us- tional support for 10PB storage allocation, ing State Machine, Threads, Socket pro- Plan, ini- resolving storage incidents, evaluating Partners OPS Manager Launch: gramming, Mutex, Semaphore &message tiate, and manage information technology team member scripts, reporting on stor- queues for soft dvlpmt, &network app (IT) projects. https://jobs-microsoft.icims. age hardware, &performing regular ca- protocols incldg HTTP, HTTPS, FTP, TCP/ com/jobs/7208/go/job pacity planning. Execute scripts for stor- IP to troubleshoot, debug &fix wireless or age allocation on hardware such as EMC SACRAMENTO, CA OTA issues. Exp must be post baccalau- Vmax, Vplex, XtremIO, DMX-4/DMX-3, Premier Field Engineer, SharePoint Dev: reate &progressive. Any suitable combi- CX3-80 using symcli &Unisphere, cre- Provide technical support to enterprise nation of education, training &exp is ac- ate hyper mapping Luns, Lun masking, customers, partners, internal staff or oth- ceptable. Mail resume to Ref#2675, GM &configure RAID groups on Clariion LUN. ers on mission critical issues experienced Global Mobility, 300 Renaissance Center, Configure Zoning &troubleshoot storage with Microsoft technologies. Requires MC:482-C32-D44, Detroit, MI 48265. &host related issues. Engr storage hard- local travel up to 25%. Telecommuting ware for capacity to ensure storage pro- permitted. https://jobs-microsoft.icims. APPLICATION SUSTAIN ENGINEER, visioning, efficiency &maintenance. Desg com/jobs/7357/go/job General Motors, Detroit, MI. Deploy, op- SRDF/S (Symmetrix Remove Data Facility erate, &enhance OnStar Infotainment Synchronous) &Timefinder Clone/Snap MADISON, WI Advanced Systems Dvlpmt Pilot Apps on production sys for disaster recovery Research Software Development En- &operating environments such as Unix, meeting RTO (Recovery time objectives) gineers (all levels): Responsible for Oracle, Java. Use &improve J2EE &Web &RPO (Recovery Point Objectives) for conducting applied research into new Svces-based app soft to analyze &per- SAP. Master, Computer Science, Infor- products and services through software form OTA Reflash &data collection of mation Systems, or related. 6 mos exp as engineering techniques. http://bit.ly/MS- OnStar Electronic Control Units (ECU) in Storage Administrator, Storage Engr or Jobs_Research_Software_Engineer vehicles communicating through Vehi- related, executing scripts for storage al- cle Computer Platform (VCP). Enhance location on hardware such as EMC Vmax Multiple job openings are available for current Pilot App product cmpts &lead &Vplex using symcli &Unisphere, &creat- each of these categories. To view de- dvlpmt efforts for new service offerings ing Lun masking, or related. Mail resume tailed job descriptions and minimum re- to troubleshoot pilot apps. Create cer- to Ref#3493, GM Global Mobility, 300 quirements, and to apply, visit the web- tificates service request, generate Java Renaissance Center, MC:482-C32-D44, site address listed. EOE. keystore using openssl &keytool IUI for Detroit, MI 48265. OnStar ADPA cmpts for vehicle communication using SMS &packet gateways, VISITING FACULTY POSITION in Com- implementing certificates accordingly puter Science (Non-Tenure Track) The to VSMS environments &test them us- Department of Computer Science at The ing VSMS. Use Jenkins &zuul configura- College of New Jersey invites application tools to build, deploy &operate the tions for a ten-month, non-tenure track apps in N.A., Europe &China. Configure visiting faculty position in computer sci- zabbix, kibana &log stash for log mgmt. ence starting August 2017. www.computer.org/computingedge 57 CAREER OPPORTUNITIES

SENIOR RESEARCHER, Warren, MI, Gen- bounded constraint solvers &CONCOLIC approaches combined with ATG meth- eral Motors. Research, investigate, inno- testing, with traversal approaches com- ods with enhanced model structural vate, &dvlp new methods &techniques bining concrete &symbolic simulation by coverage. Mail resume to Ref#2456, GM to automate (or complement techniques means of local constraint solvers &Monte Global Mobility, 300 Renaissance Center, in) rigorous dvlpmt, verification, valida- Carlo Simulation approaches. Dvlp tech- MC:482-C32-D44, Detroit, MI 48265. tion &testing of high-integrity embedded niques to check code conformance of electronic control unit systems &soft that auto generated code from parallel &hi- SENIOR USER EXPERIENCE & PROD- implement required automotive control erarchical State Flow &Simulink models UCT DESIGN. Position available in New features such as Automated Driving Sys using the Model Based Testing (MBT) ap- York, NY. Provide full stack services from Controllers, Automated steering sub sys, proaches combined with Automatic Test- information architecture, wire frames and Power distribution sub sys, Vehicle con- case Generation (ATG) methods with en- flow charts, User Experience/User Inter- trols, Interface gateways, Automatic Lane hanced model structural coverage. PhD, face and visual design. Lead internal and Changing &Obstacle Deduction &Avoid- Computer Science, Information Tech- external teams on multi-platform projects ance features in autonomous vehicles for nology, Computer Engrg, or related. 12 in the design and development of best smoother &safer ride qlty, &conventional mos exp as Researcher, Scientist or En- user experiences for large scale web- vehicle features such as Body Control gineer, identifying &solving gaps in tools sites and applications. Conduct usability Module, Engine Control Module, Airbag &methods in existing engrg anlys &dvlpg evaluations of existing and proposed de- Control Module. Dvlp first-order predicate techniques, &methods for psgr vehicle signs in-person or using online platform logic model to analyze safety impact of features such as Automated steering usertesting.com. Create wireframes and autonomous systems feature interfaces sub sys, Power distribution sub sys, Au- design prototypes using Sketch, InVision, through concurrent sys predicate &prop- tomatic lane changing &warning &Ob- Adobe Creative Suite and Axure. Apply: ositional modelling, &anlys through Bool- stacle deduction &collision avoidance in L. Sawtelle, MIP F105, Massachusetts Mu- ean satisfiability solver techniques. Per- autonomous vehicles, dvlpg techniques tual Life Insurance Company, 1295 State form verification &validation using Binary to check code conformance of auto gen- Street, Springfield, MA 01111; Please Ref- Decision Diagrams based on advanced erated code from parallel &hierarchical erence Job ID: 708203400 formal verification methodology with State Flow &Simulink models using MBT

Microsoft Corporation currently has the following openings (job opportunities available at all levels, e.g., Principal, Senior and Lead levels): TECHNICAL Oracle America, Inc. IRVINE, CA cations, systems or services working with Business Program Managers: Plan, initi- development and product planning teams. has openings for ate, and manage technology and business (http://bit.ly/MSJobs_ProgMgr) (http://bit. projects. http://bit.ly/MSJobs-Buss_Oper_ ly/MSJobs_HW_ProgMgr)(http://bit.ly/MS- Prog_Mgmt Jobs_ProdQlty_Supp)(http://bit.ly/MSJobs_ IT_ProgMgr) TECHNICAL MOUNTAIN VIEW, PALO ALTO, SUNNYVALE, CA BOSTON, MA Program Managers: (All levels, including Program Managers: (All levels, including ANALYST Leads and Managers) Coordinate program Leads and Managers) Coordinate program positions in development of computer software appli- development of computer software applications, systems or services working with cations, systems or services working with Colorado Springs, CO. development and product planning teams. development and product planning teams. (http://bit.ly/MSJobs_ProgMgr) (http://bit. (http://bit.ly/MSJobs_ProgMgr) (http://bit. ly/MSJobs_HW_ProgMgr)(http://bit.ly/MS- ly/MSJobs_HW_ProgMgr)(http://bit.ly/MS- Job duties include: Deliver solutions to Jobs_ProdQlty_Supp)(http://bit.ly/MSJobs_ Jobs_ProdQlty_Supp)(http://bit.ly/MSJobs_ IT_ProgMgr) IT_ProgMgr) the Oracle customer base while serving SAN FRANCISCO, CA IRVING, TX as an advocate for customer needs. Program Managers: (All levels, including Program Managers: (All levels, including Leads and Managers) Coordinate program Leads and Managers) Coordinate program Offer strategic technical support to development of computer software appli- development of computer software appli- assure the highest level of customer cations, systems or services working with cations, systems or services working with development and product planning teams. development and product planning teams. satisfaction. (http://bit.ly/MSJobs_ProgMgr) (http://bit. (http://bit.ly/MSJobs_ProgMgr) (http://bit. ly/MSJobs_HW_ProgMgr)(http://bit.ly/MS- ly/MSJobs_HW_ProgMgr)(http://bit.ly/MS- Jobs_ProdQlty_Supp)(http://bit.ly/MSJobs_ Jobs_ProdQlty_Supp)(http://bit.ly/MSJobs_ Apply by e-mailing resume to IT_ProgMgr) IT_ProgMgr) Multiple job openings are available for [email protected], CAMBRIDGE, MA each of these categories. To view detailed referencing 385.19923. Program Managers: (All levels, including job descriptions and minimum require- Leads and Managers) Coordinate program ments, and to apply, visit the website ad- Oracle supports workforce diversity. development of computer software appli- dress listed. EOE.

58 ComputingEdge April 2017 CAREER OPPORTUNITIES

SENIOR BIG DATA ENGINEER, General IBM Big Insights, &Oracle ExaData. Dvlp of required education &exp. Will also Motors, Detroit, MI. Use Hadoop platform BI reports using Cognos for automotive accept any equally suitable combination &Kafka, HBASE, Hive &Spark to perform telematics platforms. Install, configure of education, training, &/or experience Big Data analysis incldg complete vehicle Information Server suite on Redhat, Suse which would qualify applicant to perform prognostics &diagnostics. Design, &opti- &Oracle Enterprise Linux servers. Master, job offered. Mail resume to Ref#2251, GM mize infotainment Customer Relationship Computer Science, Computer Science Global Mobility, 300 Renaissance Center, Mgmt, Provisioning &Vehicle Diagnos- &Engrg, or Information Systems. 36 mos MC:482-C32-D44, Detroit, MI 48265. tics Systems, &Billing &Revenue mgmt exp as Sr Application Architect, Solu- business processes. Mentor, train &lead tion Architect, Big Data Engr, or related, ENGINEERING. PROJECT MANAGER 1. internal customers &colleagues in dvlpg performing Data Mgmt using InfoSphere CH2M HILL, Inc. in Kansas City, MO seeks &deploying emerging technologies for Information Server Product Suite for au- a Project Manager 1 to lead and manage business use cases. Perform Data Mgmt tomotive telematics platforms, &Data projects/programs that are aligned to using InfoSphere Information Server Replication using Oracle Golden Gate the water business group markets and Product Suite (incldg Information Gover- &InfoSphere Change Data Delivery. Will underlying business objectives. To ap- nance Catalog) for automotive telematics accept bachelor’s or foreign equiv de- ply, mail resume to: Shelly Saitta, CH2M platforms. Perform Data Replication using gree, in Computer Science, Computer HILL, 9191 S. Jamaica St., Englewood, CO Oracle Golden Gate &InfoSphere Change Science and Engineering, or Information 80112. Must reference job code: 13176. Data Delivery. Use large scale database Systems, followed by at least 5 yrs of platforms such as Hadoop, Teradata, progressive exp in the specialty, in lieu

Microsoft Corporation currently has the following openings (job opportunities available at all levels, e.g., Principal, Senior and Lead levels):

ALISO VIEJO, CA MORRISVILLE, NC Software Engineers and Software Development Engineers in Test Software Engineers and Software Development Engineers in Test (all levels, including Leads and Managers): Responsible for devel- (all levels, including Leads and Managers): Responsible for developing or testing computer software applications, systems or ser- oping or testing computer software applications, systems or services. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) vices. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) LOS ANGELES, CA FARGO, ND Software Engineers and Software Development Engineers in Test Software Engineers and Software Development Engineers in Test (all levels, including Leads and Managers): Responsible for devel- (all levels, including Leads and Managers): Responsible for developing or testing computer software applications, systems or ser- oping or testing computer software applications, systems or services. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) vices. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) MOUNTAIN VIEW, PALO ALTO, SUNNYVALE, CA NEW YORK, NY Software Engineers and Software Development Engineers in Test Software Engineers and Software Development Engineers in Test (all levels, including Leads and Managers): Responsible for devel- (all levels, including Leads and Managers): Responsible for developing or testing computer software applications, systems or ser- oping or testing computer software applications, systems or services. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) vices. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) SAN FRANCISCO, CA RESTON, VA Software Engineers and Software Development Engineers in Test Software Engineers and Software Development Engineers in Test (all levels, including Leads and Managers): Responsible for devel- (all levels, including Leads and Managers): Responsible for developing or testing computer software applications, systems or ser- oping or testing computer software applications, systems or services. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) vices. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) BOSTON, MA REDMOND, WA Software Engineers and Software Development Engineers in Test Software Engineers and Software Development Engineers in Test (all levels, including Leads and Managers): Responsible for devel- (all levels, including Leads and Managers): Responsible for develop- oping or testing computer software applications, systems or ser- ing or testing computer software applications, systems or services. vices. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) Requires domestic and international travel up to 25%. (http://bit.ly/ Program Managers: (All levels, including Leads and Managers) Co- MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) ordinate program development of computer software applications, Software Engineers and Software Development Engineers in Test systems or services working with development and product planning (all levels, including Leads and Managers): Responsible for develop- teams. (http://bit.ly/MSJobs_ProgMgr) (http://bit.ly/MSJobs_HW_ ing or testing computer software applications, systems or services. ProgMgr)(http://bit.ly/MSJobs_ProdQlty_Supp)(http://bit.ly/MSJobs_ Telecommuting permitted. (http://bit.ly/MSJobs_SDE) (http://bit.ly/ IT_ProgMgr) MSJobs_IT_SDE) CAMBRIDGE, MA Software Engineers and Software Development Engineers in Test (all levels, including Leads and Managers): Responsible for devel- Software Engineers and Software Development Engineers in Test oping or testing computer software applications, systems or ser- (all levels, including Leads and Managers): Responsible for devel- vices. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) oping or testing computer software applications, systems or services. (http://bit.ly/MSJobs_SDE) (http://bit.ly/MSJobs_IT_SDE) Multiple job openings are available for each of these categories. To view detailed job descriptions and minimum requirements, and to apply, visit the website address listed. EOE.

www.computer.org/computingedge 59 CAREER OPPORTUNITIES TECHNICAL Oracle America, Inc. has openings for LIGHTBEND SEEKS: Sr. Software Engi- Ericsson Inc. 6300 Legacy Drive, R1-C12 neer Full Stack -Architect, design, de- Plano, TX 75024 and indicate applying velop, deliver company products, plat- for 16-TX-3320. forms and API’s, create automated tests/ TECHNICAL infrastructures; Present at conferences, SOLUTIONS ARCHITECT ASCENSION meetups worldwide. 10% Domestic Travel. HEALTH-IS, INC. is seeking a Solutions Worksite: May telecommute from any US Architect in Austin, Texas to identify is- location or headquarters (San Francisco, sues between the architecture and ven- ANALYST CA 94105). To apply send resume/cov- dor solution implementation; identify er(20170301HE) to Lightbend, 625 Market solutions and formulate implementation positions in Lehi, UT. St., Ste. 1000, San Francisco, CA, 94105. and migration strategies; and provide expertise in technical knowledge, meth- ERICSSON INC. has openings for the fol- odology and framework, data integration Job duties include: Deliver solutions lowing positions: ENGINEER-SERVICES and data flows, Enterprise Architecture, SOFTWARE _ OVERLAND PARK, KS in- and software development and coding. to the Oracle customer base while tegration, certification, & implementation Contact Denise D. Dodge, Project Lead, of 4G LTE technology into customer’s lab 11775 Borman Drive, St. Louis, MO 63146. serving as an advocate for customer facilities & first market appls. Travel req’d. needs. Mail resume to Ericsson Inc. 6300 Legacy VALASSIS COMMUNICATIONS INC. has Dr, R1-C12, Plano, TX 75024 & indicate an opening for the position of Applica- applying for 17-KS-2569. DATA SCIEN- tions Engineer in WINDSOR, CT to ana- Apply by e-mailing resume to TIST/DATA ANALYTICS ARCHITECT _ lyze, write, test, & deploy software appli- PLANO, TX provide analysis of big data, cations & research, design, document, & [email protected], mathematical modeling, & Data Mining modify software specifications. To apply techniques applied to data monetization mail resume to Valassis, Attn: Patty [CT- referencing 385.19075. opportunities for telecomm. Travel req’d. 11307.26], 19975 Victor Parkway, Livonia, Telecommuting available. Mail resume to MI 48152. Oracle supports workforce diversity.

SOFTWARE TECHNICAL SOFTWARE Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for SOFTWARE has openings for SOFTWARE DEVELOPER TECHNICAL DEVELOPER positions in Burlington, Massachusetts. ANALYST positions in Minneapolis, Minnesota. Job duties include: design, develop, trouble- Job duties include: design, develop, troubleshoot and/or test/QA software; as a member positions in Lehi, Utah. shoot and/or test/QA software; as a member of the software engineering division, apply of the software engineering division, apply knowledge of software architecture to Job duties include: Deliver solutions to knowledge of software architecture to perform tasks associated with developing, the Oracle customer base while serving perform tasks associated with developing, debugging, or designing software applica- as an advocate for customer needs. debugging, or designing software applications or operating systems according to tions or operating systems according to provided design specification. Apply by e-mailing resume to provided design specifications. [email protected], Apply by e-mailing resume to referencing 385.19975. Apply by e-mailing resume to [email protected], [email protected], referencing 385.18388. referencing 385.18110. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity.

60 ComputingEdge April 2017 SOFTWARE SOFTWARE TECHNICAL Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for has openings for SOFTWARE TECHNICAL DEVELOPER SOFTWARE ANALYSTS- positions in Irvine, CA. DEVELOPER Job duties include: Design, develop, trouble- positions in Seattle, Washington. SUPPORT shoot and/or test/QA software. As a positions in Colorado Springs, Colorado. Job duties include: design, develop, trouble- member of the software engineering shoot and/or test/QA software; as a division, apply knowledge of software Job duties include: offer strategic technical member of the software engineering support to assure the highest level of customer architecture to perform tasks associated division, apply knowledge of software satisfaction; quickly isolate and clarify customer with developing, debugging, or designing architecture to perform tasks associated issues, apply technical knowledge to reproduce software applications or operating systems with developing, debugging, or designing product issues, identify solutions based on according to provided design specifications. software applications or operating systems testing results and/or comprehensive research, Travel to various unanticipated sites according to provided design specifications. all while meeting multiple customers’ critical throughout the United States required. timelines. Apply by e-mailing resume to Apply by e-mailing resume to Apply by e-mailing resume to [email protected], [email protected], [email protected], referencing 385.19985. referencing 385.20615. referencing 385.19769. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity.

QA ANALYST SOFTWARE TECHNICAL Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for has openings for QA TECHNICAL ANALYST SOFTWARE ANALYST- positions in Bedford, Massachusetts. DEVELOPER SUPPORT Job duties include: responsible for positions in Orlando, FL. positions in Orlando, FL. developing, applying and maintaining Job duties include: Design, develop, Job duties include: deliver solutions to quality standards for company products the Oracle customer base while serving with adherence to both internal and troubleshoot and/or test/QA software. as an advocate for customer needs; offer external standards; develop and execute strategic technical support to assure the software test plans. Analyze and write highest level of customer satisfaction. test standards and procedures. Apply by e-mailing resume to Apply by e-mailing resume to [email protected], Apply by e-mailing resume to [email protected], referencing 385.18437. [email protected], referencing 385.20197. referencing 385.20037. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity. www.computer.org/computingedge 61 CAREER OPPORTUNITIES

Juniper Networks is recruiting for Software Engineer Staff #28235: Par- and SDN. Develop and execute test our Sunnyvale, CA office: ticipate in new product development plans for new functionality and fea- Engineering Program Manager #50766: and contribute technical innovation tures developed in Junos. Design, develop and monitor the execu- to Juniper network security platform. Functional Systems Analyst Staff #38661: tion of software development projects Design and develop solutions for new Design and document solutions to in partnership with engineering team. features and enhancements for exist- address complex business problems. ing Juniper security products. Software Engineer Staff #37399: Ana- Analyze business processes, gather lyze complex business problems and Software Engineer #35871: Design, project requirements, and contribute collaborate with project team mem- develop, troubleshoot and debug to design and review workshops. bers to ensure development and im- high-performance packet forward- Software Engineer Staff #14818: De- plementation of new data driven busi- ing software for the current genera- sign, develop, troubleshoot and de- ness solutions. tion and next generation terabit-core bug software modules for Company’s routers. Test Engineer #34383: Design, develop products and platforms. script and implement sophisticated Hardware Engineer #30838: Plan, de- Software Engineer #37443: Work close- tests to verify software feature func- sign and develop signal integrity de- ly with product marketing software tionality for Company products. sign rules for boards, packages and and hardware development teams to ASICs with high speed electrical Software Engineer #23245: Design, review design specification, then de- interfaces. develop troubleshoot and debug develop functional test plan for review. tailed software functional and design Technical Systems Analyst #38746: Ana- Systems Engineer #11815: Develop and specifications for new software fea- lyze, design, develop and support SAP deliver technical sales strategy and tures on Juniper products. technical enhancements and deliver solutions by providing pre-sales tech- new and complex business solutions Software Engineer Senior Staff #4587: nical support, training, and lab sup- by integrating modules of SAP and ex- Design, develop, troubleshoot and port to meet end customer require- ternal systems to meet business and debug high performance forwarding ments. Telecommuting allowed. user requirements. software for current generation and Hardware Engineer #30222: Respon- next generation Terabit Core Routers. Sales Demonstration Engineer #26725: sible for the testing and character- Work with customers, Company Systems Engineer #27726: Assist Com- ization of optical sub-systems and business units and account teams to pany Account Managers with the de- modules used in Company platforms demonstrate Company products and velopment and delivery of technical to evaluate reliability of materials, technologies through Proof of Con- solutions and sales activities for new properties and techniques used in cept (POC) testing. and existing opportunities. Telecom- production. muting allowed. Software Engineer #29261: Design, de- Software Engineer #39619: Design, velop, troubleshoot and debug func- Software Engineer #38704: Design, de- develop, troubleshoot and debug in- tional and design specifications for velop, troubleshoot and debug com- gestion pipelines for a variety of data software products and enhancement. ponent and system level diagnostics from enterprise systems, routers, software/firmware using Python pro- Systems Engineer #29195: Design, de- switches, and firewalls. gramming language. velop and test software modules to Software Engineer #18288: Analyze, support security and networking fea- Software Engineer Staff #6201: Design, design, program, debug and modify tures in Company products. Telecom- develop, troubleshoot and debug de- novel operating system techniques muting allowed. Travel required. sign specifications for network secu- and algorithms for On-Device man- rity products in physical and virtual Software Engineer Sr. Staff #8050: agement software products. environments that are being imple- Work with product line management Sr. Strategic Delivery Engineer #28167: mented in Data Center, Mobile LTE, and customers to analyze, design, and Design, develop and implement test- Enterprise and Cloud deployments. modify device and network manage- ing efforts which could include au- ability requirements and help imple- Software Engineer #37060: Analyze, tomated testing, regression testing, ment solutions. design, develop, program, debug scale testing, migration testing, certifi- and modify test management and Test Engineer Staff #15855: Using cation testing and end to end solution workflow related software applica- product definitions, scaling targets, testing for Network security products. tions for JUNOS developers and test and customer use scenarios, design, Software Engineer #29835: Develop communities. develop and implement product per- and maintain virtual machine creation formance and scaling tests for high QA Engineer #39696: Work within a and control software. Debug issues performance Ethernet switching security system testing team to debug found within the virtual cloud envi- platforms. and test complex multi-service net- ronment. Implement new OpenStack working and network security prod- Test Engineer #26482: Provide sys- and Contrail based functionality. ucts for Data Center and cloud. tem-level testing for routing protocols Software Engineer #39316: Design and

62 ComputingEdge April 2017 CAREER OPPORTUNITIES

COMPUTER PROGRAMMER, MEM- solution & maint. of Hyperion Financial PHIS, TENNESSEE: Limited domestic Mgmt. (HFM) appls. Execute coding, travel and/or occasional relocation to testing & maint. activities. Provide crit- client sites nationwide to write code, ical appl. design recommend. to client forms, and script using Java, JSP, Serv- which aligns with its long & short term IT develop applications and web ser- lets, JDBC, PL/SQL, HTML. Debug, trou- & business strategies & build solutions vices for JUNOS developers and test bleshoot existing code. Reply to: SVS as per external SEC reporting & mgmt. teams. Research software technolo- Technologies Limited, 8700 Trail Lake reporting requirements. Resp. for de- gies to optimize existing infrastruc- Drive, #228, Memphis, TN 38125 veloping solutions for Hyperion FDMEE, ture or software design. Hyperion Acct. Reconciliation &Hype- rion Planning appls. Advise clients on Software Engineer #7391: Design, REPORTS DEVELOPER: design & develop data models using Cognos; ana- financial best practices & drive day-to- develop, troubleshoot and debug day execution of client engagements functional, design and architectural lyze & troubleshoot Cognos reporting. BS+5 yrs. or MS+3. marcela.niemeyer@ related to EPM project. Create strategy, specifications for next generation laureate.net w/ Job # 20501BR in subj blueprint, & process & data optimiza- interfaces and long haul optical line. Job in McLean, Virginia. Laureate tion for clients for Oracle EPM products. products. Education, Inc. EOE. Requires: M.S. degree in Finance, Bus. Test Engineer #39241: Design, devel- Adm., or related field & 3 yrs. exp. in job offered or 3 yrs. exp. as a Consultant. op and implement testing for com- MANAGER. Job location: Miami, FL & Will accept B.S. (or foreign equiv.) & 5 plex multi-service networking and any other unanticipated locations in U.S. yrs. exp. in the finance in lieu of M.S. & network security products. Travel Required. Duties: Resp. for ad- 3 yrs. exp. Concurrent exp. must incl.: vising & implement. strategic solutions Product Marketing Manager #29873: 3 yrs. exp. executing coding, testing & for Finance & Enterprise Performance Create and maintain marketing as- maint. activities. Send resume (no calls) Mgmt. (EPM) processes incl. Financial sets for Company product lines and to: Michelle Ramirez, The Hackett Group, Close & Reporting, Acct. Reconcilia- solutions across all media types, to Inc., 1001 Brickell Bay Dr., Suite 3000, tion, Planning, Budgeting, Forecasting & establish, enhance, and distinguish Miami, FL 33131. Data Integration using Oracle EPM Prod- product within industry. ucts. Resp. for developing end-to-end Sustaining Engineer #51117: Diagnose, troubleshoot, repair and debug complex networking issues in the Data Center and Switching areas. Software Engineer #36797: Design, OCEAN PREDICTION develop, troubleshoot, debug and modify large scale networking POSTDOCTORAL POSITIONS software. Naval Research Laboratory, Juniper Networks is recruiting for our Westford, MA office: Stennis Space Center, MS Technical Support Engineer #37316: The Naval Research Laboratory is seek- Provide technical phone support to ing postdoctoral researchers to push for- customers and assistance to other support engineers to resolve prod- ward the frontiers of ocean forecasting. cover a wide scope uct issues. of physics including surface waves, thermohaline circula- Juniper Networks is recruiting for tion, nearshore circulation, and ocean/atmosphere coupling our Herndon, VA office: from global to nearshore scales. This challenging work Systems Engineer #33983: Provide includes processing and analysis of satellite and in water onsite technical guidance to the observations, construction of numerical model systems on customers on Juniper’s routing, high performance computing and assimilation for predicting switching, security solutions and services. Understand customers’ the ocean environment. For a quick overview of some of the environment and position Juniper’s research projects within the NRL oceanography division at solutions within the customers’ in- Stennis Space Center, visit the web site: http://www7320. frastructure. Must be willing to travel nrlssc.navy.mil/projects.php to client sites locally. Applicants must be a US citizen or permanent resident at Mail single-sided resume with job code # to time of application. Applications will be accepted until po- Juniper Networks sitions are filled. Please e-mail a resume and description of Attn: MS A.4.435 research interests: 1133 Innovation Way Sunnyvale, CA 94089 Gregg Jacobs: [email protected]

www.computer.org/computingedge 63 CAREER OPPORTUNITIES

INFOSYS LIMITED is in need of indi- teams with customer context, com- lead IT change management tracks for viduals to work full-time in Plano, Texas petitor and industry context, details medium to large engagements. Will and various and unanticipated locations on pain-points and client introduc- lead the engagement effort at differ- throughout the U.S. Must be willing tions required for opening diverse ent stages of IT projects from business to work anywhere in the U.S. as all job service offerings in own account(s). process consulting, and problem defini- opportunities may involve relocation to Travel required. (Req ID: 20548BR). tion to solution design and deployment. various and unanticipated client site lo- Industry Principal(s) - U.S. needed to Travel required. (Req ID: 20549BR). cations; any travel and/or relocation to be paid by employer pursuant to internal policy. We have multiple openings for each job opportunity, and are an Equal Opportunity Employer M/F/D/V. Please apply to Infosys Limited online at: https://www.infosys.com/careers/ job-opportunities/pages/index.aspx. Select the box labeled “Technical” which will bring you to the site with the “Experienced Professionals” link in the right column. Click on the “Experienced Professionals” link. Once a user account has been created, please follow the link for ‘Search Openings’ and click on ‘Advanced Search.’ Enter the Req ID listed below in the “Auto Req ID” box. NEXT ISSUE Engagement Manager(s) needed to navigate IT business account to SOFTWARE identify different kinds of deals; form and lead pursuit teams; help pursuit

TECHNOLOGY Oracle America, Inc. has openings for the following positions (all levels/types) in San Mateo County, including Redwood Shores, CA and San Bruno, CA; Alameda County, including Pleasanton, CA; San Francisco, CA; Santa Clara County, including Santa Clara and San Jose, CA; and other locations in the San Francisco Bay Area. Some positions may allow for telecommuting.

Hardware Developers (HWD317): Evaluate reliability of materials, properties and techniques used in production; plan, design and develop electronic parts, components, integrated circuitry, mechanical systems, equipment and packaging, optical systems and/or DSP systems. Product Managers (PM317): Participate in all software and/or hardware product development life cycle activities. Move software products through the software product development cycle from design and development to implementation, testing, and/or marketing. Software Developers (SWD317): Design, develop, troubleshoot and/or test/QA software. Applications Developers (APD317): Analyze, design, develop, troubleshoot and debug software programs for commercial or end user applications. Write code, complete programming and perform testing and debugging of applications. Programmer Analysts (PA317): Analyze user requirements to develop, implement, and/or support Oracle’s global infrastructure. Technical Analysts-Support (TAS317): Deliver solutions to the Oracle customer base while serving as an advocate for customer needs. Offer strategic technical support to assure the highest level of customer satisfaction. Consultants (TCONS317): Analyze requirements and deliver functional and technical solutions. Implement products and technologies to meet post-sale customer needs. Travel to various unanticipated sites throughout the U.S. required. Sales Consultants (TSC317): Provide presales technical/functional support to prospective customers. Design, validate and present Oracle’s software solutions to include product concepts and future direction. Travel to various unanticipated sites throughout the U.S. required. Software Developers (TSWD317): Design, develop, troubleshoot and/or test/QA software. Travel to various unanticipated sites throughout the U.S. required. Applications Developers (TAPD317): Analyze, design, develop, troubleshoot and debug software programs for commercial or end user applications. Write code, complete programming and perform testing and debugging of applications. Travel to various unanticipated sites throughout the U.S. required.

Submit resume to [email protected]. Must include job#. Oracle supports workforce diversity.

64 ComputingEdge April 2017 CLASSIFIED LINE AD SUBMISSION DETAILS: Rates are $425.00 per column inch ($640 minimum). Eight lines per column inch and average five typeset words per line. Send copy Applications are invited for:- at least one month prior to publication date to: Debbie Sims, Faculty of Engineering Classified Advertising, Computing Edge Magazine, 10662 Los Professors / Associate Professors / Assistant Professors Vaqueros Circle, Los Alamitos, CA 90720; (714) 816-2138; fax (Ref. 1700004N) The Faculty of Engineering is seeking several faculty posts at Professor / Associate (714) 821-4010. Email: [email protected]. Professor / Assistant Professor levels with prospect for substantiation. The professors will play a signifi cant role in the Cyber Security Center, which will be established by In order to conform to the Age Discrimination in Employment the Faculty of Engineering. Act and to discourage age discrimination, Computing Edge Cyber security is identifi ed as one of the Faculty’s strategic research areas, to be developed by both the Department of Computer Science & Engineering and Department of may reject any advertisement containing any of these phras- Information Engineering. Talented candidates are sought to complement existing efforts and create new synergies. Candidates in the following areas are encouraged to apply: es or similar ones: “…recent college grads…,” “…1–4 years - cryptography and computational theory in security maximum experience…,” “…up to 5 years experience,” or “…10 - network, system and software security - data security and privacy years maximum experience.” Computing Edge reserves the - computer forensic right to append to any advertisement without specific notice - hardware and IoT security Applicants should have a relevant PhD degree and a good scholarly record demonstrating to the advertiser. Experience ranges are suggested minimum potential for teaching and research excellence. requirements, not maximums. Computing Edge assumes that Appointments will normally be made on contract basis for up to three years initially commencing August 2017, which, subject to performance and mutual agreement, may since advertisers have been notified of this policy in advance, lead to longer-term appointment or substantiation later. The exact start date can be they agree that any experience requirements, whether stat- worked out with the successful applicants. Applications will be accepted until the posts are fi lled. ed as ranges or otherwise, will be construed by the reader as Application Procedure minimum requirements only. Computing Edge encourages em- Applicants please upload the full resume with a cover letter, copies of academic credentials, publication list with abstracts of selected published papers, a research ployers to offer salaries that are competitive, but occasionally plan, a teaching statement, together with names and e-mails addresses of three to fi ve a salary may be offered that is significantly below currently ac- referees to whom the applicant’s consent has been given for their providing reference (unless otherwise specifi ed). ceptable levels. In such cases the reader may wish to inquire of The University only accepts and considers applications submitted online the employer whether extenuating circumstances apply. for the posts above. For more information and to apply online, please visit http://career.cuhk.edu.hk.

TECHNOLOGY Intuit Inc. has openings for the following positions in Mountain View, California: Staff Application Operations Engineers (Job code: I-292): Apply a full understanding of the business, the customer, and the solutions that a business offers to effectively design, develop, and implement operational capabilities, tools and processes. Senior Business Data Analysts (Job code: I-2311): Write SQL queries to pull sales data using Site Catalyst. Technical Data Analysts (Job code: I-2422): Partner with marketing, finance, analytics, and cross-functional teams to interpret large volumes of data, address key business questions, from hypothesis to execution.

Positions in Woodland Hills, California: Software Engineers (Job code: I-3134): Apply software development practices to design, implement, and support individual software projects.

To apply, submit resume to Intuit Inc., Attn: Olivia Sawyer, J203-6, 2800 E. Commerce Center Place, Tucson, AZ 85706. You must include the job code on your resume/cover letter. Intuit supports workforce diversity.

www.computer.org/computingedge 65 CAREER OPPORTUNITIES

Apple Inc. has the following job opportunities in Cupertino, CA: Software Engineer Applications Software Quality Assurance Engi- big data, & artificial intelligence w/ (Req# 9KSNDX) Dev the next gen neer (Req# A9CU96) Build app test focus on Span user exp. geolocation system. Optimize & cases (auto & manual) & exec req’d Software Engineer Applications improve processing pipeline. tests. (Req# A7T2TW) Des & dev SW for Software Development Engineer Software Development Engineer iCloud reporting sys. (Req# 9U4RR8) (Lang Req – Ger- (Req# 9Q5MF8) Des & dev SW Systems Design Engineer (Req# man) Cond sw qual test to ensure testing for camera, video, audio, & 9CYVYJ2ND) Des and dev int SW the qual of groundbreaking tech graphics of Apple prods. tools and sys for the Wireless Des for large scale sys, spoken lang, big Software Engineer Applications dept. data, and AI with a focus on the (Req# ACQ24M) Dsgn, dev, & German user exp. Mechanical Quality Engineer support Apple’s realtime cmmuni- (Req# 9T65WU) Support new Ap- Software Engineer Applications cations infrstrctre. ple product launches from a mech, (Req# A6F33Z) Des, build, and Software Development Engineer tooling, eng & product dev level. sup new, critical infra systems and (Req# AAZPJ3) Perf diff types of Travel req 30%. frameworks. test ranging from fnctnl, feature, Product Design Engineer (Req# Software QA Engineering Lead regression & auto test. 9TATXU) Dev & implement high (Req# 9X5SQN) Lead Web QA Software Engineer Applications precision factory & manufacturing Team. Assist & guide in dev javas- (Req# AHYU5W) Validate sys automation processes in high vol- cript-based web automation. capacity to ensure suff resource ume environment for Apple prod- Software Engineer Applications availability. ucts. Travel required 35%. (Req# 9WNPT9) Des & dev SW for Software Development Engineer Product Quality Engineer (Req# telemetry systs. (Req# A8XSM7) Dsgn & dvlp re- 99N36Z) Dev & impl new tech to Software Engineer Applications al-time comm SW for FaceTime. enhance production quality, capac- (Req# A9GTLG) Des & drive new ity & efficiency. Travel req 30%. Software Development Manager prods/pltfrms frm concept to im- (Req# 9DTNVH) Des & lead dev Software Development Engineer plem in the Architecture tm. of Siri SW & languags. (Req# 9UJVV5) Des, dev, debug & Software Engineer Applications test Apple Maps SW on iOS & OSX. Software Engineer Applications (Req# 9XE23J) Res, des, dev, im- (Req# 9PNSHN) Des & implemnt Software Quality Assurance Engi- plmnt, & debug iOS apps. Travel SW to aid in the anlysis & correctn neer (Req# A7449N) Drive SW qual req’d 10%. of client-facng prdcts, usng machine & integration to improve cust exp. Software Engineer Applications learng technques, on a globl scale. Software Engineer Systems (Req# (Req# 9F8TLF) Research, des, dev, Software Development Engineer 9TU28B) Dev SW to automate en- & productize machine learning (Req# A3FW45) Des, dev, debug & gineering data collection in the RF mdls for iTunes & App Store rec- port bootloader & driver SW for Hardware organizatn. ommender sys for personalization. mobile realtime OS. Software Development Manager Software Development Engineer Software Development Engineer (Req# 9E5QEN) Mange the des of (Req# A2GTX4) Dsgn & implmnt (Req# A7A24T) Dsgn, dvlp & im- algorthms tht cntrl a cmplx mchncl/ features in iAd Workbench News plmnt search relevnce & quality sys elctrcl sys. Web App & API. Software Development Engineer Development Engineer (Req# Software Engineer Applications (Req# A3RW5F) Dvlp the SW build A8R2LR) Dsgn, implmnt & main- (Req# 9XJVXU) Des and dev large sys. Set up & supprt CI & dplymnt tain highly scure & redundant nt- scale distributed sys to sup workflow tools. wrk envirnmnts. engines that process and compute large amt of data in var domains. Software Development Engineer Software QA Engineer (Req# (Req# 9QEP8Z) (Lang Req – Span) A5EVZJ) Dsgn, dev, excute & ana- Software Engineer Applications Conduct SW Quality Testing to en- lyze low lvl automated tsts fr mbile (Req# 9WJ2EP) Develop REST Web sure the quality of groundbreaking devcs. services for third party interactions. tech for lrg scale sys, spoken lang,

66 ComputingEdge April 2017 CAREER OPPORTUNITIES

ASIC Design Engineer (Req# AF- Localization Engineer (Technical Hardware Development Engineer H2UQ) Archtct, dsgn & vrfy Pwr Translator) (Req# 9FM2U6) Trans- (Req# 9LTQ3L) Dsng HW for Ap- Mangmnt ICs. late Apple SW prdcts & rel mate- ple’s Input Devices group. Travel rials from English to Simplified Software Engineer Applications Req 20%. Chinese. (Req# A6D8PM) Dsgn & dvlp Software Engineer Applications mbile & web apps w automated Software Development Engineer (Req# 9JG29L) Des & dev SW for tsts. (Req# 9VSSMZ) Rspnsble for the Apple News ecosys & feats pertain- wireless quality on Apple macOS, Software Engineer Systems (Req# ing to content mgmnt & authoring. iOS, WatchOS, & TVOS devices. 9T8VNY) Archtct, spcfy, dsgn, Hardware Development Engineer dvlp, & launch Apple’s touch tech Hardware Development Engineer (Req# 9SZ2Z2) Dev, prototype & prdct chrctriztion & prdction in- (Req# A7M53A) Analyze, debug, implmnt cutting-edge color imag- strmntation SW. Travel Req’d 25% & imprv perf for new comp sys un- ing sol’s & tech for Apple prods. der dev. Systems Design Engineer (Req# Software Engineer Applications 9U32VA) Dev & optimize RF au- Engineering Project Lead (Req# (Req# 9W9SUA) Eval quality of al- tomation sys used on Apple’s new- A4YU2G) Plan, coordnte, & reprt gorithmic results in Apple Maps. est prods incl iPhones, iPods, iPads the status of SW eng prjcts. Machine Learning Manager (Req# & other wireless tech. Travel req’d: Software Development Engineer 9USPXF) Dsgn & execute a De- 20%. (Req# 9FJTY5) Des, impl & debug cision Sci prgrm fcsd on IDing, Software Quality Assurance Engi- location frmwrk SW, APIs & func- trgtng, & seizing fncl opportunities neer (Req# 9TA3Q6) Test audio & tionality on iOS, macOS, watchOS for AppleCare. vid tech w manual & automation & tvOS. tst methods. Software Development Engineer Software Development Engineer (Req# A7245T) Dsgn & implmnt Apple Inc. has (Req# 9Y6SJE) Des & dev SW and features for contnt mngmnt app for the following job test cases for testing map SW. lrge nmbr of pblishrs. opportunities in Software Quality Assurance Engi- Research Scientist/Engineer (Req# neer (Req# A663S5) Des test plans A6SN4E) Anlyze data fr Apple’s Austin, TX: to test wirless SW & HW featrs. SW & srvics. Write code to anlyz & visualze data. ASIC Design Engineer (Req# 9FT- Software Development Engi- JLV) Deliver low-power analysis, neer (Req# A393BM) Perform sys Software Development Engineer optimization, estimation, & mi- bring-up of new HW platforms. (Req# 9QJ38L) Dev & main- cro-archtcture power modeling for tain test plans. Carry out testing Hardware Development Engineer Apple’s GPU. across a variety of netwrk topolo- (Req# 9KT2B8) Dsgn antennas gies to ensure the hghst vid & au- ASIC Design Engineer (Req# for mobile communication devices. dio quality. 9XJVYN) Dsgn, simulate & verfy Travel req. 15%. analog crcuits usng Cadence tools. Firmware Engineer (Req# 9LUTLF) Software Development Engineer Des & dev firmware for wireless (Req# 9WY3JX) Qualify Wi-Fi fea- audio prdcts. ture fnctnalty & prfrmnce acrss various Apl pltfrms. Software Development Engineer (Req# A9VQE8) Analyze reqs, des Refer to Req# & mail Hardware Development Engineer & implement new user-facing web (Req# A8SVW4) Dev motion & be- resume to Apple Inc., apps hvr predictn algs & corrspndng SW ATTN: D.W. in C++/MATLAB. Software Development Engineer 1 Infinite Loop 104-1GM, (Req# 9YNM8P) Des & dvlp highly Development Engineer (Req# Cupertino, CA 95014. scalable distributed sys to process 9R2NK7) Detct & rspond to in- Apple is an EOE/AA m/f/ large scale data. formtn secrty threats. disability/vets.

www.computer.org/computingedge 67 TECHNOLOGY LinkedIn Corp. LinkedIn Corp. has openings in our Mtn View, CA location for: Software Engineer (All Levels/Types) (SWE0317SV) Design, develop & integrate cutting-edge software technologies; Senior Software Engineer in Test (Test Engineer) (6597.1766) Design & develop advanced test suites using object-oriented methodologies; Manager, Software Engineering (6597.722) Act as thought-leader for team and others, proposing and building investments in automation frameworks and tools to drive improvements and efficiencies across the larger test engineering organization; Performance Engineer (6597.1655) Design, develop & integrate cutting-edge software technologies; User Experience Designer (6597.1193) Develop original visual designs against specific communication priorities, utilizing tools such as Adobe Photo- shop, Adobe Illustrator, Apple Keynote, Microsoft PowerPoint, Microsoft Word & online media collections like Getty Images & iStockPhoto; Sr. Engineer, Site Reliability (6597.1050) Serve as a primary point responsible for the overall health, performance & capacity of one or more Internet-facing service; Senior Product Manager (6597.1896) Analyze the competitive environment, customers & product metrics to determine the right set of features to drive engagement & usage on LinkedIn; Engineer, Service Operations (6597.872) Coordinate short- & long-term initiatives with LinkedIn DevOps teams, as well as partners (CDN, DNS, monitoring/measurement) – prioritizing & driving project closure on all sides; Senior User Experience Designer (6597.1223) Create holistic design solutions that address business, brand, and user requirements; Associate Web Developer (6597.968) Design, develop & integrate cutting-edge software technologies; Product Manager, ProFinder (6597.1927) Develop a comprehensive product & marketing roadmap to deliver business goals; Manager, Software Engi- neering (6597.115) Manage the performance & career development of a small team of engineers, & own significant parts of LinkedIn products that require design, architecture, & coding; Quality Assurance Manager, Sales Systems (6597.1779) Develop a QA program strategy which includes planning, evaluating, designing, architecting, implementing & maintaining test frameworks.

LinkedIn Corp. has openings in our San Francisco, CA location for: Software Engineer (All Levels/Types) (SWE0317SF) Design, develop & integrate cutting-edge software technologies.

LinkedIn Corp. has openings in our New York, NY location for: Software Engineer (All Levels/Types) (SWE0317NY) Design, develop & integrate cutting-edge software technologies; Director, Software Engineering (6597.255) Architect, design, develop, & support the most visible Internet-scale products & infrastructure at LinkedIn.

Please email resume to: [email protected]. Must ref. job code above when applying.

68 ComputingEdge April 2017 CAREER OPPORTUNITIES

CONFERENCES in the Palm of Your Hand

IEEE Computer Society’s Conference Publishing Services (CPS) is now offering conference program mobile apps! Let your attendees have their conference schedule, conference information, and paper listings in the palm of their hands.

The conference program mobile app works for Android devices, iPhone, iPad, and the Kindle Fire.

For more information please contact [email protected]

TECHNICAL TECHNICAL TECHNOLOGY Oracle America, Inc. Oracle America, Inc. has openings for Oracle America, Inc. has openings for TECHNICAL has openings for PRODUCT ANALYST- TECHNICAL SUPPORT SUPPORT ANALYST MANAGER positions in Lehi, Utah. positions in positions in Orlando, FL. Colorado Springs, CO. Job duties include: deliver solutions to the Oracle customer base while serving as an Job duties include: Deliver solutions Job duties include: Contribute to the advocate for customer needs; offer to the Oracle customer base while development of customer support strategic technical support to assure the serving as an advocate for customer management model and consistent highest level of customer satisfaction. needs. business practices including team readiness Travel to various unanticipated sites to support new product releases or throughout the United States required. Apply by e-mailing resume to functionality. Apply by e-mailing resume to [email protected], Apply by e-mailing resume [email protected], referencing 385.19786. to [email protected], referencing 385.19904. Oracle supports workforce diversity. referencing 385.19495. Oracle supports Oracle supports workforce diversity. workforce diversity. www.computer.org/computingedge 69 TECHNICAL Oracle America, Inc. has openings for TECHNICAL ANALYSTS- SUPPORT positions in Lehi, Utah.

Job duties include: Deliver post-sales support and solutions to the Oracle customer base while serving as an advocate for customer needs.

Apply by e-mailing resume to [email protected], referencing 385.19669. Oracle supports workforce diversity.

TECHNOLOGY Oracle America, Inc. has openings for

HARDWARE Are Enemy Hackers Slipping through Your Team’s Defenses? DEVELOPERS positions in Burlington, MA. Protect Your Organization from Hackers by Thinking Like Them Job duties include: Evaluate reliability of materials, properties and techniques used Take Our E-Learning Courses in production; plan, design and develop in the Art of Hacking electronic parts, components, integrated circuitry, mechanical systems, equipment You and your staff can take these courses where you are and packaging, optical systems and/or DSP and at your own pace, getting hands-on, real-world training systems. that you can put to work immediately. www.computer.org/artofhacking Apply by e-mailing resume to [email protected], referencing 385.14655. Oracle supports workforce diversity.

70 ComputingEdge April 2017 CAREER OPPORTUNITIES Recognizing Excellence in High-Performance Computing Nominations are Solicited for the SEYMOUR CRAY SIDNEY FERNBACH & KEN KENNEDY AWARDS

SEYMOUR CRAY COMPUTER ENGINEERING AWARD Established in late 1997 in memory of Seymour Cray, the Seymour Cray Award is awarded to recognize innovative contributions to high-performance computing systems that best exemplify the creative spirit demonstrated by Seymour Cray. The award consists of a crystal memento and honorarium of US$10,000. This award requires 3 endorsements. Sponsored by:

SIDNEY FERNBACH MEMORIAL AWARD Established in 1992 by the Board of Governors of the IEEE Computer Soci- ety, this award honors the memory of the late Dr. Sidney Fernbach, one of the pioneers on the development and application of high-performance computers for the solution of large computational problems. The award, which consists of a certificate and a US$2,000 honorarium, is presented annually to an individual for “an outstanding contribution in the application of high-performance computers using innovative approaches.” This award requires 3 endorsements. Sponsored by:

ACM/IEEE-CS KEN KENNEDY AWARD This award was established in memory of Ken Kennedy, the founder of Rice University’s nationally ranked computer science program and one of the world’s foremost experts on high-performance computing. A certificate and US$5,000 honorarium are awarded jointly by the ACM and the IEEE Computer Society for outstanding contributions to programmability or productivity in high- performance computing together with significant community service or mentoring contributions. This award requires 2 endorsements.

Cosponsored by:

Deadline: 1 July 2017 All nomination details available at awards.computer.org www.computer.org/computingedge 71 NEW PREFERRED PLUS MEMBERSHIP

OPTIONS TRAINING & DEVELOPMENT FOR A BETTER FIT. RESEARCH

BASIC

STUDENT

And a better match for your career goals. IEEE Computer Society lets you choose your membership — and the benefits it provides — to fit your specific career needs. With four professional membership categories and one student package, you can select the precise industry resources, offered exclusively through the Computer Society, that will help you achieve your goals.

Learn more at www.computer.org/membership. Achieve your career goals with the fit that’s right for you. Explore your options below.

Training & Preferred Plus Research Basic Student Select your Development $60 $126 $55 $115 $55 $115 $40 $99 $8 membership IEEE Affiliate IEEE Affiliate IEEE Affiliate IEEE Affiliate Does not include Member Member Member Member Member Member Member Member IEEE membership

Computer magazine (12 digital issues)*

ComputingEdge magazine (12 issues)

Members-only discounts on conferences and events

Members-only webinars

Unlimited access to Computing Now, computer.org, and the new mobile-ready myCS

Local chapter membership

Skillsoft’s Skillchoice™ Complete with 67,000+ books, videos, courses, practice exams and mentorship resources Books24x7 on-demand access to 15,000 technical and business resources

Two complimentary Computer Society magazine subscriptions

myComputer mobile app 30 tokens 30 tokens 30 tokens

12 FREE 12 FREE Computer Society Digital Library Member pricing Member pricing Included downloads downloads

3 FREE 3 FREE Training webinars Member pricing Member pricing Member pricing webinars webinars

Priority registration to Computer Society events

Right to vote and hold office

One-time 20% Computer Society online store discount

* Print publications are available for an additional fee. See catalog for details.

www.computer.org/membership IEEE Computer Society 2017 Call for MAJOR AWARD NOMINATIONS Help Recognize Computing’s Most Prestigious

IEEE Computer Society awards recognize outstanding achievements and highlight significant contributors in the teaching and R&D computing communities. All members of the profession are invited to nominate individuals who they consider most eligible to receive international recognition of an appropriate society award.

Computer Entrepreneur Award Computer Pioneer Award Sterling Silver Goblet Silver Medal Vision and leadership resulting in the growth of Pioneering concepts and development of the some segment of the computer industry. computer field.

Technical Achievement Award W. Wallace McDowell Award Certificate/$2,000 Certificate/$2,000 Contributions to computer science or computer Recent theoretical, design, educational, technology. practical, or other tangible innovative contributions. Harry H. Goode Memorial Award Bronze Medal/$2,000 Taylor L. Booth Award Information sciences, including seminal ideas, Bronze Medal/$5,000 algorithms, computing directions, and concepts. Contributions to computer science and engineering education. Hans Karlsson Award Plaque/$2,000 Computer Science & Engineering Team leadership and achievement through Undergraduate Teaching Award collaboration in computing standards. Plaque/$2,000 Recognizes outstanding contributions to Richard E. Merwin Award for undergraduate education. Distinguished Service Bronze Medal/$5,000 IEEE-CS/Software Engineering Institute Outstanding volunteer service to the profession Watts S. Humphrey Software Process at large, including service to the IEEE Computer Achievement Award Society. (Joint award by CS/SEI) Plaque/$1,500 Harlan D. Mills Award Software professionals or teams responsible for Plaque/$3,000 an improvement to their organization’s ability to Contributions to the practice of software create and evolve software-dependent systems. engineering through the application of sound theory.

Deadline: 15 October 2017 Nomination Site: awards.computer.org For more information visit: www.computer.org/awards