January 2017

Contents | Zoom in | Zoom out For navigation instructions please click here Search Issue | Next Page

THE WORLD’S NEWSSTAND®

______

Digital Object Identifier 10.1109/MSP.2016.2636083

THE WORLD’S NEWSSTAND® Contents Volume 34 | Number 1 | January 2017

SPECIAL SECTION COLUMNS

SIGNAL PROCESSING EDUCATION 6 Society News VIA HANDS-ON AND 2017 Class of Distinguished Lecturers 9 Special Reports DESIGN PROJECTS A Networking Revolution Powered by Signal Processing John Edwards FROM THE GUEST EDITORS 13 48 Perspectives Hana Godrich, Arye Nehorai, Small Data, Mid Data, and Big Data Versus Ali Tajer, Maria Sabrina Greco, Algebra, Analysis, and Topology and Changshui Zhang Xiang-Gen Xia 82 SP Education 16 SIGNAL PROCESSING PROJECTS Students’ Design Project Series: AT TECHNISCHE UNIVERSITÄT Sharing Experiences DARMSTADT Hana Godrich Tim Schäck, Michael Muma, 89 Tips & Tricks and Abdelhak M. Zoubir ON THE COVER An Accurate and Stable Sliding DFT Computed by a Modified CIC Filter HANDS-ON LEARNING Integrating more hands-on experiences into formal engi- Denis A. Gudovskiy and Lichung Chu 31 neering education is mainstream, and significant efforts THROUGH RACING are being made in this direction. An insight into the imple- 94 Lecture Notes Qing Zhuo, Yanpin Ren, mentation challenges of design projects and experimental Compressive Privacy: From Information/ platforms from students in their freshmen through senior Yongheng Jiang, years, and solutions adopted to address them are offered Estimation Theory to Machine Learning and Changshui Zhang in this issue of IEEE Signal Processing Magazine through S.Y. Kung a series of article contributions from around the world. 104 Applications Corner TEACHING THE PRINCIPLES COVER IMAGE: ©ISTOCKPHOTO.COM/RAWPIXEL Discovering New Worlds 40 OF MASSIVE MIMO Muhammad Salman Khan, Erik G. Larsson, Danyo Danev, James Stewart Jenkins, and Nestor Becerra Yoma Mikael Olofsson, and Simon Sörman FEATURES 110 Book Digest 116 In the Spotlight 52 COMPRESSIVE VIDEO SENSING Richard G. Baraniuk, 0 0.08 Tom Goldstein, Aswin C. Sankaranarayanan, Christoph Studer, Ashok Veeraraghavan, and Michael B. Wakin

(a) (b) (c) 67 THE TECHNOLOGY BEHIND PG. 116 PERSONAL DIGITAL ASSISTANTS PG. 82

ADOBE STOCK Ruhi Sarikaya

IEEE SIGNAL PROCESSING MAGAZINE (ISSN 1053-5888) (ISPREG) is published bimonthly by the Institute of Electrical and Electronics Engineers, Inc., 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA (+1 212 419 7900). Responsibility for the contents rests upon the authors and not the IEEE, the Society, or its members. Annual member subscriptions included in Society fee. Nonmember subscriptions available upon request. Individual copies: IEEE Members US$20.00 (first copy only), nonmembers US$241.00 per copy. Copyright and Reprint Permissions: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright Law for private use of patrons: 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without fee. Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For all other copying, reprint, or republication permission, write to IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Copyright © 2017 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals postage paid at New York, NY, and at additional mailing offices. Postmaster: Send address changes to IEEE Signal Processing Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188 Printed in the U.S.A.

Digital Object Identifier 10.1109/MSP.2016.2636098

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 1

THE WORLD’S NEWSSTAND®

DEPARTMENTS IEEE Signal Processing Magazine

3 From the Editor EDITOR-IN-CHIEF Aleksandra Mojsilovic— Signal Processing: The Expected Min Wu—University of Maryland, College Park IBM T.J. Watson Research Center and the Unexpected U.S.A. Fatih Porikli—MERL Min Wu AREA EDITORS Shantanu Rane—PARC, U.S.A. Saeid Sanei—University of Surrey, U.K. Feature Articles Roberto Togneri—The University of 4 President’s Message Shuguang Robert Cui—Texas A&M University, Volunteerism Makes a Positive Difference U.S.A. Western Australia Alessandro Vinciarelli—IDIAP–EPFL Rabab Ward Special Issues Douglas O’Shaughnessy—INRS, Canada Azadeh Vosoughi—University of Central Florida Stefan Winkler—UIUC/ADSC, Singapore 113 Dates Ahead Columns and Forum Kenneth Lam—Hong Kong Polytechnic University, ASSOCIATE EDITORS—e-NEWSLETTER Hong Kong SAR of China Csaba Benedek—Hungarian Academy e-Newsletter of Sciences, Hungary Christian Debes— TU Darmstadt and Paolo Braca—NATO Science and Technology AGT International, Germany Organization, Italy Social Media and Outreach Quan Ding—University of California, Andres Kwasinski—Rochester Institute San Francisco, U.S.A. of Technology, U.S.A. Pierluigi Failla—Compass Inc, New York, EDITORIAL BOARD U.S.A. Mrityunjoy Chakraborty – Indian Institute of Marco Guerriero—General Electric Research, Technology, Kharagpur, India U.S.A. George Chrisikos – Qualcomm, Inc., Yang Li—Harbin Institute of Technology, China U.S.A. Yuhong Liu—Penn State University at Altoona, Patrick Flandrin—ENS Lyon, France U.S.A. Mounir Ghogho—University of Leeds, Andreas Merentitis—University of Athens, Greece U. K. Michael Muma—TU Darmstadt, Germany Lina Karam—Arizona State University, U.S.A. Xiaorong Zhang—San Francisco State University, Hamid Krim—North Carolina State University, U.S.A. U.S.A. ASSOCIATE EDITOR—SOCIAL MEDIA/OUTREACH Sven Loncˇaric´—University of Zagreb, Croatia Guijin Wang—Tsinghua University, China Brian Lovell—University of Queensland, Australia Jian Lu—Qihoo 360, China IEEE SIGNAL PROCESSING SOCIETY Henrique (Rico) Malvar—Microsoft Research, Rabab Ward—President U.S.A. Ali Sayed—President-Elect Yi Ma—ShanghaiTech University, China Carlo S. Regazzoni—Vice President, Stephen McLaughlin—Heriot-Watt University, Conferences Scotland Nikos D. Sidiropoulos—Vice President, Athina Petropulu—Rutgers University, Membership U.S.A. Thrasyvoulos (Thrasos) N. Pappas— Peter Ramadge—Princeton University, Vice President, Publications U.S.A. Walter Kellerman—Vice President, Shigeki Sagayama—Meiji University, Japan Technical Directions PG. 113 Erchin Serpedin—Texas A&M University,

IMAGE LICENSED BY INGRAM PUBLISHING U.S.A. IEEE SIGNAL PROCESSING SOCIETY STAFF ICASSP 2017 will be held in New Orleans, Louisiana, 5–9 Shihab Shamma—University of Maryland, Denise Hurley—Senior Manager of Conferences March. U.S.A. and Publications Hing Cheung So—City University of Hong Kong, Rebecca Wollman—Publications Administrator Hong Kong Isabel Trancoso—INESC-ID/Instituto Superior IEEE PERIODICALS MAGAZINES DEPARTMENT Técnico, Portugal Jessica Barragué, Managing Editor Pramod K. Varshney—Syracuse University, Geraldine Krolin-Taylor, Senior Managing Editor U.S.A. Mark David, Senior Manager Z. Jane Wang—The University of British Advertising and Business Development Columbia, Canada Felicia Spagnoli, Advertising Production Manager Gregory Wornell—Massachusetts Institute Janet Dudar, Senior Art Director of Technology, U.S.A. Gail A. Schnitzer, Mark Morrissey, Dapeng Wu—University of Florida, U.S.A. Associate Art Directors ASSOCIATE EDITORS—COLUMNS AND FORUM Theresa L. Smith, Production Coordinator Ivan Bajic—Simon Fraser University, Canada Dawn M. Melley, Editorial Director Rodrigo Capobianco Guido— Peter M. Tuohy, Production Director São Paulo State University, Brazil Fran Zappulla, Staff Director, Ching-Te Chiu—National Tsing Hua University, Publishing Operations Taiwan Michael Gormish—Ricoh Innovations, Inc. Xiaodong He—Microsoft Research Danilo Mandic—Imperial College, U.K. Digital Object Identifier 10.1109/MSP.2016.2636099

Promoting Sustainable Forestry _____ SFI-01681 SCOPE: IEEE Signal Processing Magazine publishes tutorial-style articles on signal processing research and IEEE prohibits discrimination, harassment, and bullying. applications as well as columns and forums on issues of interest. Its coverage ranges from fundamental principles For more information, visit to practical implementation, reflecting the multidimensional facets of interests and concerns of the community. Its http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html. mission is to bring up-to-date, emerging and active technical developments, issues, and events to the research, educational, and professional communities. It is also the main Society communication platform addressing important issues concerning all members.

2 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

FROM THE EDITOR

Min Wu | Editor-in-Chief | [email protected]______

Signal Processing: The Expected and the Unexpected

t will be the start of another new year Right after ICIP, I briefly stopped in enhancement in steel manufacturing and I when you receive this issue of IEEE the San Francisco Bay area, where I gave the signaling in China’s high-speed train Signal Processing Magazine (SPM). a keynote speech at a North American systems. As it turned out, many chal- Happy 2017 to all our readers, editors, and alumni forum of my college alma mater, lenging issues addressed by the keynote reviewers! Tsinghua University, in Beijing, China. speech have benefited from signal pro- Not long before I began writing this Different from ICIP, I did not expect cessing theories and techniques. Two editorial, the 2016 edition of the IEEE this forum to be a venue to see so much panel discussions on the recent hype of International Conference on Image Pro- signal processing other than the talk on artificial intelligence and the Internet of cessing (ICIP) was successfully held in microsignals for media security that I Things also touched on such issues as Phoenix, Arizona. In addition to the rich would be giving. I did my undergraduate sensing, denoising, and statistical learn- and timely technical sessions that ICIP is study in the Department of Automation ing from signals and data. In addition, well known for, the ICIP 2016 team—led at Tsinghua University. “Automation” as several alumni who are successful in ven- by General Chair Prof. Lina Karam, who an engineering major covers a combina- ture capital investment highlighted the is also serving on SPM’s senior edito- tion of control and robotics, electronic important roles of data and data analytics rial board, and Industrial Program Chair sensing and diagnosis, signal process- that they saw in developing sustainable Dr. Haohong Wang—spearheaded the ing, and pattern recognition; within the new businesses. first Visual Innovation Award. Going department, different specialty direc- Most speakers at the alumni forum over the finalists’ roster, you may very tions were rather compartmentalized would not consider themselves to be pro- well find yourself having been a user of historically. Perhaps it was due to the fessionals in signal processing, and not some of these technologies: the YouTube difficulty to find an exact matching many have read our magazine. It remind- video streaming service, the Lytro light- department in North American uni- ed me of “Signal Processing Inside,” a field camera, the Intel RealSense camera versities that college alumni from the notion coined in SPM’s September 2004 technology, the CUDA high-performance department went in different ways when editorial by then Editor-in-Chief Prof. computing by NVIDIA, the Netflix movie pursuing their graduate studies in North K.J. Ray Liu, and the blurred boundaries streaming service, the Oculus virtual real- America. Among them, you will find between disciplines discussed in my Sep- ity technology, and the Microsoft Kinect. experts on securing sensors and sensor tember 2016 editorial. Inspired by those For many signal processing profes- network, on supply chain management expected and unexpected venues where sionals, including those who regularly behind some of the most wanted con- signal processing shines, I am working attend ICIP—a flagship conference of sumer products, on designing the next- with our magazine editors to develop leads the IEEE Signal Processing Society generation mass spectrometry analyzer, on informative articles for our readers in (SPS)—it might almost have been taken and on international finance and policy the coming months. We welcome your for granted that signal processing plays making, just to name a few. suggestions on topics that you would like a key role behind these visual innova- Yet through this stimulating day-long to read about. tions. Whether it is image formation, event, I learned a great deal about many Best wishes to you all for a prosperous sensing, compression, or communica- broad applications of signal processing. new year ahead—another year filled with tions, signal processing provides the For example, a keynote speech given exciting signal processing! underlying technical foundation. before mine provided an overview on designing and analyzing sensing signals for fault-tolerant operations in such com- Digital Object Identifier 10.1109/MSP.2016.2632218 Date of publication: 11 January 2017 plex systems as the quality control and SP

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 3

THE WORLD’S NEWSSTAND®

PRESIDENT’S MESSAGE

Rabab Ward | SPS President | [email protected]______

Volunteerism Makes a Positive Difference

appy New Year! The onset of a new Dr. Nikos Sidiropoulos to his new posi- schedules. Over the past couple of years, Hyear is an exciting time filled with tion at SPS vice president, Membership. the SPS has expanded its volunteer renewal, opportunity, and possibility. This past year brought about a lot of roles to encourage diverse involvement As we close the book on 2016, it’s only exciting changes—in June, we launched opportunities. You can choose a posi- natural to reflect on our progress, cele- the new SPS website, accompanied tion that best fits your lifestyle whether brate our successes, and learn from our by the SPS Resource Center; http://rc you’re a student, young professional, in failures. But the beginning of a new year .signalprocessingsociety.org. With these, the middle of your career, or your career is also empowering—it reignites your a new era began for the Society, looking is winding down —or even if you are fire to try something new, to set goals, to toward and acting on a rapidly changing retired but want to stay active. be open to change and embrace new future. Our organization is changing. The SPS relies on its dedicated vol- challenges. The IEEE Signal Processing Our Society is changing. And, most no- unteer base—more than 1,000 members Society (SPS) hopes to invoke these tably, our membership is changing, and strong—to develop and manage Society principles and nimbleness into 2017 and we have to find new and effective ways activities, products, and services. Deci- for many years to come. We hope that to keep members of all ages, career stag- sion makers who sit on our boards and you’ll continue to work with us through es, professions, and fields engaged and committees play an integral role in the this exciting time! involved with the SPS and its activities. Society and its operations. These high- I would be remiss to not mention Volunteerism is a great way to encour- level roles cover a wide array of Society that 2017 marks the expiration of a age early involvement, building loy- needs in the areas of conferences, pub- valued member of the SPS Executive alty across a diverse member body with lications, membership, education, and Committee’s term. In 2013, the SPS Ex- varying interests, availability, needs, and more. Sitting on boards and committees, ecutive Committee divided the role of expertise. Volunteerism has evolved in while time-consuming, can be incred- vice president, Awards and Membership, itself, and SPS is working toward build- ibly rewarding and prestigious. Many establishing a newly individualized ing a strong volunteer base, with roles to high-ranking board members move on role of vice president, Membership, suit the growing needs for the Society to become decision makers in broader- to better examine, understand, and and its members alike. scale positions within the IEEE. serve SPS’s growing and diverse The nature of “volunteering” used Even among publications, conferenc- membership. Dr. Kostas Plataniotis to be immersive and intimidating, with es, education, and membership, there was chosen as to fill this position. excessive time commitments that made are a multitude of opportunities of vary- His vision and innovation were es- volunteerism seem like too much to jug- ing levels of involvement. You can be a sential to not only proving the role’s gle among other activities. Now, with reviewer or editor of one of our Society necessity but to driving its purpose the help of technology and the evolu- publications, form a committee to pro- and setting the course for future mem- tion of workplaces, a volunteer has new pose and host a conference or a meeting ber services and activities. We thank flexibility and freedom to choose his/ in a desired area, or propose a seasonal him for his effective and incredible her level of involvement, from demand- school workshop near you. Want to get service and are excited to welcome ing high-profile leadership roles to involved locally? Form an SPS Chapter, “microvolunteering” opportunities that or attend an event of an existing local spark interest and action without con- Chapter to strengthen connections with Digital Object Identifier 10.1109/MSP.2016.2628418 Date of publication: 11 January 2017 suming as much from our already busy other signal processing professionals

4 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

near you. Why not host a networking to get more involved but don’t really make connections within the field, and event or technical talk in conjunction know where you can step in. The SPS expand your career options. These are just with a local Chapter and involve local is always looking for volunteers to help a sampling of the many ways SPS mem- industry partners? with our ongoing visibility efforts. This bers can have a hand in Society activities The SPS has several technical com- can entail something as simple as send- and another way the SPS strives to help mittees (TCs) and special interest ing out a quick Tweet to promote the its members reach their goals—whether groups (SIGs) that help steer the techni- SPS or share signal processing news, or it the goals are only the new year or beyond. cal direction of the Society, contributing can be as involved as writing a post for our The SPS wishes you a happy, healthy, their expertise in regards to SPS confer- new SPS blog; http://signalprocessing and prosperous new year. For full infor- ences, awards, publications, and educa- society.org/publications-resources/blog. mation about volunteering with the tional activities. Anyone can become an We even have a group of volunteers on SPS, visit our website at http://signal affiliate member of a TC or SIG—both call when signal processing sources are processingsociety.org. If you have ques- are a great way to not only get involved needed for external media stories. Social tions or need guidance, please feel free with important Society activities but to media ambassadorship and blog contri- to contact me or our SPS Membership build relationships with other SPS mem- butions are both great ways for younger and Content Administrator Jessica Perry bers who share similar technical inter- members to get involved early without yet at [email protected].______ests to you. committing to more serious roles. Many of us don’t have time to dedi- Volunteering for the SPS in any cate to planning events, serving on capacity—whether for a couple of hours boards, or reviewing papers. Maybe a month to several days a year—is a great you’re new to the Society and want way to get involved, build your resume, SP

,(((-RXUQDORI6HOHFWHG7RSLFVLQ6LJQDO3URFHVVLQJ -6763

5HFHQW6SHFLDO,VVXHV EŽǀĞŵďĞƌϮϬϭϲʹǆƉůŽŝƚŝŶŐ/ŶƚĞƌĨĞƌĞŶĐĞĨŽƌŶĞƌŐǇͲĨĨŝĐŝĞŶƚĂŶĚ^ĞĐƵƌĞŽŵŵƵŶŝĐĂƚŝŽŶƐ KĐƚŽďĞƌϮϬϭϲʹĚǀĂŶĐĞĚ^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐĨŽƌƌĂŝŶEĞƚǁŽƌŬƐ ^ĞƉƚĞŵďĞƌϮϬϭϲʹ&ŝŶĂŶĐŝĂů^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐĂŶĚDĂĐŚŝŶĞ>ĞĂƌŶŝŶŐĨŽƌůĞĐƚƌŽŶŝĐdƌĂĚŝŶŐ ƵŐƵƐƚϮϬϭϲʹWĞƌƐŽŶͲĞŶƚĞƌĞĚ^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐĨŽƌ,ĞĂůƚŚdĞĐŚŶŽůŽŐŝĞƐ :ƵŶĞϮϬϭϲʹ^ƚƌƵĐƚƵƌĞĚDĂƚƌŝĐĞƐŝŶ^ŝŐŶĂůĂŶĚĂƚĂWƌŽĐĞƐƐŝŶŐ ƉƌŝůϮϬϭϲʹ^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐĨŽƌDŝůůŝŵĞƚĞƌtĂǀĞŽŵŵƵŶŝĐĂƚŝŽŶƐ DĂƌĐŚϮϬϭϲʹ^ƚŽĐŚĂƐƚŝĐ^ŝŵƵůĂƚŝŽŶĂŶĚKƉƚŝŵŝǌĂƚŝŽŶŝŶ^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐ &ĞďƌƵĂƌǇϮϬϭϲʹĚǀĂŶĐĞĚ^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐŝŶDŝĐƌŽƐĐŽƉǇĂŶĚĞůů/ŵĂŐŝŶŐ ^ĐĂŶƚŚĞYZĐŽĚĞĂŶĚǀŝƐŝƚ/yƉůŽƌĞĨŽƌŵŽƌĞŝŶĨŽƌŵĂƚŝŽŶĂďŽƵƚƚŚĞƐĞƐƉĞĐŝĂůŝƐƐƵĞƐ͘

Digital Object Identifier 10.1109/MSP.2016.2581299

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 5

THE WORLD’S NEWSSTAND®

SOCIETY NEWS

2017 Class of Distinguished Lecturers

he IEEE Signal Processing Society’s (1993) from the University of Iowa, (2014), and a National Science Founda- T(SPS’s) Distinguished Lecturer Pro- where he received the John Briggs tion (NSF) CAREER Award. The work gram provides the means for Chapters Memorial Award for the top undergrad- he supervised won student best paper to have access to well-known educators uate across all colleges. He obtained the awards at the IEEE Data Compression and authors in the fields of signal process- M.S. degree (1995) Conference in 2006 ing to lecture at Chapter meetings. While and Ph.D. degree and 2011 and the many IEEE Societies have similar pro- (1998) in electrical Chapters interested in IEEE Sensor Array grams, the SPS provides financial support engineering from the arranging lectures by the and Multichannel for the Chapters to take advantage of this University of Califor- Distinguished Lecturers Signal Processing service. Chapters interested in arranging nia, Berkeley, where Workshop in 2012 lectures by the Distinguished Lecturers can he received the Elia- can obtain information as well as five MIT obtain information from the Society’s web hu Jury Award for by sending an e-mail to thesis awards. He is

page (http://signalprocessingsociety.org/ outstanding achieve- [email protected].______a coauthor of Foun-

professional-development/distinguished-______ment in systems, dations of Signal lecturer-program)______or by sending an e-mail communications, Processing (Cam-

to [email protected].______control, or signal processing. bridge University Press, 2014). Candidates for the Distinguished Dr. Goyal was a member of techni- Dr. Goyal served on the IEEE Image Lecturer Program are solicited from the cal staff in the Mathematics of Commu- and Multidimensional Signal Processing Society technical committees, editorial nications Research Department, Bell Technical Committee (2003–2009); IEEE boards, Chapters, and other boards and Laboratories, Lucent Technologies, Image, Video, and Multidimensional Sig- committees by the Awards Board. The (1998–2001) and a senior research nal Processing Technical Committee Awards Board vets the nominations, and engineer for Digital Fountain, Inc. (2014); and the steering committee of the Board of Governors approves the (2001–2003). He was the Esther and IEEE Transactions on Multimedia final selection. Distinguished Lecturers Harold E. Edgerton Associate Professor (2013). He has served as editorial board are appointed for a term of two calendar of Electrical Engineering, Massachu- member, Foundations and Trends and years. Distinguished Lecturers named setts Institute of Technology (2004– Signal Processing (2006–present); sci- for 2017 are as follows. 2013), adviser, 3dim Tech, Inc. (winner entific advisory board of the Banff of the 2013 MIT $100K Entrepreneur- International Research Station for Math- Vivek K. Goyal ship Competition Launch Contest ematical Innovation and Discovery Vivek K. Goyal Grand Prize and 2013 MassChallenge (2011–present); the IEEE SPS Compu- obtained his B.S. Accelerator Gold), and was subsequent- tational Imaging SIG (2015–present); degree in mathemat- ly with Nest, an Alphabet company the IEEE Standing Committee on ics (1993) and his (2014–2016). He is now with the Industry DSP Technology (2016–pres- B.S.E. degree in elec- Department of Electrical and Computer ent); technical program cochair, Inter- trical engineering Engineering of Boston University. national Conference on Sampling He is an IEEE Fellow and was award- Theory and Application (2015); and ed the IEEE SPS Magazine Award conference cochair, SPIE Wavelets and Digital Object Identifier 10.1109/MSP.2016.2622958 Date of publication: 11 January 2017 (2002), the IEEE SPS Best Paper Award Sparsity conference series (2006–2016).

6 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Dr. Goyal’s research interests include associate editor, IEEE Transactions on Signal and Image Processing. From computational imaging, human percep- Circuits and Systems for Video Technol- 1996 to 1998, he had a joint appoint- tion, decision making, sampling, quan- ogy (2004–2006); associate editor, ment as director of research at the Insti- tization, and source coding theory. His IEEE Transactions on Signal Process- tute of Language and Speech lecture topics include first-photon imaging (2007–2009); associate editor, IEEE Processing in Athens. Since 1999, he ing and other extreme optical imaging, Journal on Selected Topics in Signal has been working as a professor at the social learning in decision-making Processing (2013–2015); member, NTUA School of ECE, where he is cur- groups, teaching signal processing IEEE Image and Multidimensional Sig- rently the director of the Intelligent with geometry, and the optimistic nal Processing Technical Committee Robotics and Automation Lab. He has Bayesian: replica method analysis of (2001-2006); member, IEEE Multime- held visiting scientist positions at the compressed sensing. dia Signal Processing Technical Com- Massachusetts Institute of Technology mittee (2005–2008); member, IEEE in the fall of 2012 and at the University Christine Guillemot Image, Video, and Multidimensional of Pennsylvania in the fall of 2016. Christine Guillemot Signal Processing Technical Committee Prof. Maragos served as associate holds a Ph.D. degree (2013–present); senior area editor, editor, IEEE Transactions on Acoustics, from Ecole Nationale IEEE Transactions on Image Process- Speech, and Signal Processing (1989– Superieure des Tele- ing (2016–2017); and steering commit- 1990); and IEEE Transactions on Pat- communications Paris. tee member, IEEE Transactions on tern Analysis and Machine Intelligence; She was with FRANCE Multimedia (2016). general chair, IEEE International Con- TELECOM, where she Over the past 20 years, Dr. Guille- ference on Visual Communications and was involved in various projects in the mot’s research has focused on numerous Image Processing (1992); general chair, area of coding for TV, high-definition aspects of image and video processing: International Symposium on Mathe- TV, and multimedia (November 1985 to modeling, representation, compression, matical Morphology and Its Applica- October 1997) and she worked at Bell- and communication. Her contributions tions to Image/Signal Processing core, New Jersey, as a visiting scientist concern algorithms for image and video (1996); general chair, MMSP (2007); (January 1990 to mid-1991). Since analysis, representation, coding, commu- program chair, European Conference November 1997, she has been the direc- nication, and for inverse problems such on Computer Vision (2010); ECCV tor of research at INRIA, as the head of as superresolution, inpainting, and resto- Workshop on Sign, Gesture, and Activi- a research team dedicated to the design ration. Her lecture topics include sparsity ty (2010); Dagstuhl Symposia on Shape of algorithms for the image and video and dimensionality reduction in image (2011 and 2014); Intelligent Robots processing chain, with a focus on anal- compression and superresolution; multi- and Systems Workshop on Cognitive ysis, representation, compression, and view and light fields processing: from Mobility Assistance Robots (2015); editing, including for emerging modali- analysis, representation, compression to general chair, EUSIPCO (2017); mem- ties such as high dynamic range imag- rendering; and from image to video and ber, SPS Digital Signal Processing ing and light fields. multiview inpainting. Technical Committee (1992–1998); Dr. Guillemot has coauthored nine IEEE SPS Image and Multidimensional book chapters, 65 publications in peer- Petros Maragos Signal Processing Technical Commit- reviewed international journals (IEEE Petros Maragos tee (1995–1999); IEEE SPS Multime- Transactions on Signal Processing, received the M.Eng. dia Signal Processing Technical IEEE Transactions on Image Process- diploma in electrical Committee (2009–2012); and member, ing, IEEE Transactions on Information engineering from the Greek National Council for Research Theory, and IEEE Transactions on Cir- National Technical and Technology. cuits and Systems for Video Technology), University of Athens He is the recipient or corecipient of 162 publications in international con- (NTUA) in 1980 and several awards for his academic work, ferences [IEEE International Conference the M.Sc. and Ph.D. degrees from including: U.S. NSF Presidential Young on Acoustics, Speech, and Signal Pro- Georgia Tech, Atlanta, in 1982 and Investigator Award (1987–1992); IEEE cessing (ICASSP), IEEE International 1985. In 1985, he joined the faculty of SPS Young Author Best Paper Award Conference on Image Processing, IEEE the Division of Applied Sciences at (1988), IEEE SPS Best Paper Award International Workshop on Multimedia Harvard University, Cambridge, Massa- (1994), IEEE W.R.G. Baker Prize Signal Processing (MMSP), and Euro- chusetts, where he worked for eight Award for the most outstanding original pean Signal Processing Conference years as professor of electrical engi- paper (1995), Pattern Recognition Soci- (EUSIPCO)], and 24 granted patents. neering, affiliated with the Harvard ety’s Honorable Mention Best Paper Dr. Guillemot is an IEEE Fellow. Robotics Lab. In 1993, he joined the Award (1996), and Best Paper Award, She has served as associate editor, faculty of the School of Electrical and Conference on Computer Vision and IEEE Transactions on Image Pro- Computer Engineering (ECE) at Geor- Pattern Recognition-2011 Workshop on cessing (2000–2003 and 2014–2016); gia Tech, affiliated with its Center for Gesture Recognition.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 7

THE WORLD’S NEWSSTAND®

Prof. Maragos was elected IEEE Dr. Petropulu is an IEEE Fellow He was an associate editor of IEEE Fellow for his research contributions in (2008) and the recipient of the 1995 Presi- Transactions on Signal Processing 1995 and received the 2007 EURASIP dential Faculty Fellow Award given by (1999–2001, 2008–2009, 2015–pres- Technical Achievements Award for NSF and the White House. She has served ent), EURASIP Signal Processing, and contributions to nonlinear signal pro- as editor-in-chief, IEEE Transactions on IEEE Signal Processing Letters (2006– cessing, systems theory, image, and Signal Processing (2009–2011); IEEE 2007). He has been a guest editor of sev- speech processing. In 2010, he was SPS vice president, conferences (2006– eral journals including IEEE Journal of elected fellow of EURASIP for his 2008); member-at-large, IEEE SPS Board Selected Topics for Signal Processing, research contributions. He has been of Governors (2004–2005); general chair, IEEE Journal on Selected Areas in Com- elected IEEE SPS Distinguished Lec- ICASSP 2005; recipient, IEEE Signal munications, and IEEE Signal Process- turer for 2017–2018. Processing Magazine Best Paper Award ing Magazine; lead guest editor, Prof. Maragos’ research and teach- (2005); recipient, IEEE SPS Meritorious International Journal of Robotics ing interests include signal processing, Service Award (2012); member, IEEE Research special issue on networked systems theory, machine learning, SPS Fellow Reference Committee (2012– robotics; general cochair, IEEE Global image processing and computer vision, 2014); and was selected as an IEEE Conference on Signal and Information audio and speech/language processing, Distinguished Lecturer for the SPS Processing (GlobalSIP 2016); member, cognitive systems, and robotics. In the (2017–2018). SPS Sensor Array and Multichannel aforementioned areas he has published Dr. Petropulu’s research interests Technical Committee (2006–2011 and numerous papers, book chapters, and span the area of statistical signal pro- 2015–present); member, Signal Process- has also coedited three Springer cessing, wireless communications, sig- ing for Communications Technical Com- research books, one on multimodal pro- nal processing in networking, physical mittee (1999–2005); and cochair, IEEE cessing and two on shape analysis. layer security, and radar signal process- Robotics and Automation Society Tech- Prof. Maragos’ lecture topics in- ing. Her research has been funded by nical Committee on Networked Robotics. clude multimodal spatiotemporal signal various government industry sponsors Dr. Sadler received the IEEE SPS processing and audio-visual perception, including the NSF, the Office of Naval Best Paper Award in 2006 and 2010, sev- nonlinear signal processing and dynam- research, the U.S. Army, the National eral ARL awards, three Army R&D ical systems on lattices, morphological Institutes of Health, the Whitaker Foun- Achievement awards, as well as the Out- and variational methods in image analy- dation, and Lockheed Martin. Her lec- standing Invention of the Year Award sis and computer vision, graph-based ture topics include sparse sensing-based from the University of Maryland in 2008. methods for clustering and segmenta- multiple-input, multiple-output radars; Dr. Sadler is a Fellow of the IEEE and tion, and multimodal gesture and the coexistence of radar and communi- ARL, and he has lectured at the Johns spoken command recognition in human- cation systems; cooperative approaches Hopkins University Whiting School of robot interaction. for physical layer security; cooperative Engineering for 14 years. approaches for improving the perfor- Dr. Sadler’s research interests span Athina P. Petropulu mance of wireless networks; mobile intelligent systems, with an emphasis Athina P. Petropulu beamforming; and localization of brain on distributed collaborative operation, received her under- activations based on electroencephalo- including multiagent autonomy, cogni- graduate degree from gram recordings and sparse signal tive networking, distributed sensing and the National Techni- recovery theory. signal processing, and mixed-signal cir- cal University of cuit architectures for low power sensing Athens, Greece, in Brian M. Sadler and cognition. His recent work focuses 1986, and the M.Sc. Brian M. Sadler is the on collaborative physical agents in and Ph.D. degrees from Northeastern U.S. Army senior stressful and complex environments; University, Boston, Massachusetts, in research scientist for “20-questions” strategies for machines 1988 and 1991, respectively, all in elec- Intelligent Systems, at to query humans; and the combination trical and computer engineering (ECE). the Army Research of distributed computation, control, and Since 2010, she has been a professor in Laboratory (ARL) cognitive networking. His lecture top- the ECE Department at Rutgers Univer- in Maryland. He ics include distributed collaborative in- sity, New Brunswick, New Jersey, having received his undergraduate and master’s telligent systems, human-autonomy served as chair of the department during education in electrical engineering from querying and interaction, and auton- 2010–2016. Before that she was a mem- the University of Maryland in 1984 and omous networking. ber of faculty at Drexel University, Phila- the Ph.D. degree in electrical engineering delphia, Pennsylvania. from the University of Virginia in 1993. SP

8 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

SPECIAL REPORTS

John Edwards

A Networking Revolution Powered by Signal Processing

ata networks are extending their which a single unit of light—a lone pho- signals that are represented by continu- Dreach into virtually every corner of ton—could not even be measured without ous random variables. “Thus, in order life. The rapidly emerging Internet being destroyed. Therefore, an efficient to characterize our quantum signals, of Things (IoT), for instance, promises quantum light source would allow com- we borrow a variety of techniques from to bring connectivity to just about every- pletely secure communication. classical signal processing,” he notes. thing. Technology analysis firm Gartner A research team led by Jelena “Some examples … are Fourier analy- (http://www.gartner.com/newsroom/ Vuckovic, a Stanford professor of elec- sis to analyze the spectral content of our id/3165317)______predicts that by 2020 there trical engineering, has spent the past signals and frequency filtering to exam- will be over 20 billion connected devices. several years working toward the devel- ine specific spectral content.” Signal processing is helping the IoT opment of nanoscale lasers and quan- The biggest challenge the researchers and other network technologies to operate tum technologies that might someday have faced so far is dealing with the fact faster, more efficiently, and very reliably. enable conventional computers to com- that quantum light is far weaker than the Advanced research also promises to open municate faster and more securely using rest of the light emitted by a modified new opportunities in key areas, such as light instead of electricity. Vuckovic laser, making it difficult to detect. Ad- highly secure communication and various and her team, including Kevin Fischer, dressing this obstacle, the team devel- types of wireless networks. a doctoral candidate and lead author of oped a method to filter out the unwanted a paper describing the project, believe light, enabling the quantum signal to be Seeking quantum communication that a modified nanoscale laser can be read much better. “Some of the light At Stanford University, researchers are used to efficiently generate quantum coming back from the modified laser is examining quantum communication as a light for fully protected quantum com- like noise, preventing us from seeing the potential way to quickly and reliably secure munication. “Quantum networks have quantum light,” Fischer says. “We can- Internet traffic. Yet before this goal can be the potential for secure end-to-end com- celed it out to reveal and emphasize the reached, they will have to overcome sev- munication wherein the information quantum signal hidden beneath.” eral major technical challenges, including channel is secured by the laws of quan- To deal with the noise issue, the re- developing devices that can actually send tum physics,” Fischer says. searchers turned to self-homodyning— and receive quantum data. An important “Our quantum light source produces an interferometric technique that was step in creating such devices is to develop single photons, one at a time, on originally invented as a method for de- a quantum light source that might some- demand,” Fischer continues. “Our tech- tecting radio-frequency signals, mixing day serve as the basis for secure quantum nology also poses the potential for two the signal in question with a strong lo- data transfers. or three photon sources as well.” Such cal oscillator. “We used an optical ana- Ordinary lasers can’t be used for secure light sources will be critical for future log of this technique to isolate quantum communication since they emit a “classi- quantum networking and computation as opposed to classical signals. By care- cal” light that would enable unauthorized applications. “They serve as the signal fully adjusting how the canceling light parties to extract data without detection. that goes into the input of any quantum and the classical light overlap, the un- A secure quantum network, on the other processor,” Fischer notes. wanted light is canceled and the once- hand, would be based on quantum light in Optical signal processing is handled hidden quantum light is revealed,” he by the optical elements themselves says (Figure 1).

Digital Object Identifier 10.1109/MSP.2016.2616376 at the speed of light. Quantum light, Self-homodyning and interferomet- Date of publication: 11 January 2017 Fischer says, is effectively the study of ric techniques generally require precise

THE WORLD’S NEWSSTAND®

Building such devices with the required low tolerances challenges even the most advanced fabrication techniques. “Our interferometer’s largest critical dimension is microns and smallest critical dimension is nanometers,” Fischer says. Creating a practical and cost-effective approach to integrating optical light devices into standard complementary metal–oxide–semiconductor (CMO) fabrication processes is yet another challenge facing the researchers. “Therefore, we’re also investigating CMOS-compat-

YOUSIF KELAITA/STANFORD UNIVERSITY KELAITA/STANFORD YOUSIF ible material platforms that can support FIGURE 1. An enlarged artist’s rendering showing a gallium arsenide chip. The pink vector (at the our technology,” Fischer says. bottom) depicts “classical” or laser light entering the chip. The blue structure in the center is indium As the researchers turn their atten- arsenide. This material acts like a special filter that allows classical light to pass through while also tion toward developing a functional pro- generating quantum light (shown in blue) that provides a secure way to transmit data. totype, commercial applications exist as only a distant possibility. “Not yet,” phase-locking of the signal and oscil- The biggest technical challenge still Fischer says. “We first need to demon- lator fields, which is challenging to facing the researchers is scaling the strate that our device works in a wave- achieve with light compared to radio fre- optical light devices down to a size that guide-based system.” quency fields. “This is challenging with will allow for integration into quantum light, because the wavelength is much networks. “We are working toward Simpler sensor networks smaller—on the order of microns as op- demonstrating adapting this technique Ioannis Schizas, an assistant professor posed to meters—which then necessar- in an on-chip quantum network, where of electrical engineering at the Univer- ily requires greater precision,” Fischer light propagates down a waveguide as sity of Texas at Arlington, is developing says. “Therefore, our advance was to opposed to through free-space,” Fisch- a sensing environment that would use find a device structure that generated er comments. multiple simple devices to collect and both the local oscillator field and the Using state-of-the-art nanofabrication process data that currently requires the quantum signal, which were then inher- technology, the team is currently engi- power of a supercomputer (Figure 2). ently aligned to one another.” neering its first quantum light devices. “Sensors provide huge amounts of data, but using and applying the data they collect requires a very powerful computer,” Schizas says. “I hope to eliminate that

1 need through simplicity of design.” 10 13 As he creates the new sensing envi- 0.9 1 2 14 11 16 ronment, Schizas is using several dif- 10 15 0.8 6 3 12 ferent types of commonly available 4 17 18 20 sensors to collaborate with each other

0.7 8 19 and gather various types of data that

0.6 can be either sorted or ignored. He hopes to eliminate the need for super- FC 0.5 computing by using optimization techniques to determine the best placement

0.4 25 5 of sensors, including thermometers, 9 23 0.3 28 21 22 accelerometers, pressure sensors, and 29 30 acoustic sensors equipped with digital 0.2 27 7 26 signal processors (DSPs) and wireless 24

0.1 communications support. Schizas says his research relies on the development 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 of novel signal processing techniques. “It is fair to say this is a signal process- FIGURE 2. A network of sensors observing a field. Ioannis Schizas, an assistant professor of electrical engineering at the University of Texas at Arlington, is developing a sensing environment that would use ing research project,” he states. multiple simple devices to collect and process data that currently requires the power of a supercomputer. Schizas says his research is currently (Photo courtesy of the University of Texas at Arlington.) focused on the development of general

10 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

algorithms with learning capabilities involving environmental monitoring (MIMO) technology that can coordinate that can identify different informative and the processing of ecological and several Wi-Fi routers at once, enabling portions within various types of sen- climatic data, the project will introduce the devices to triangulate data faster and sor data that may adhere to different beneficial data mining solutions to deal more consistently. The new approach, data models. “Heterogeneous sensing with the heterogeneous and high volume joint multiuser beamforming (JMB), systems consist of sensors with differ- data,” he notes. enables independent access points (APs) ent types of sensing and communication “Most, if not all, of the challenges to beamform their signals and com- capabilities,” he explains. “The main encountered so far relate to signal pro- municate with their clients on the same challenge is that the often large amount cessing issues,” Schizas says. A current channel as if the APs were a single large of acquired raw sensed data doesn’t pro- concern is finding a way to deal with MIMO transmitter. vide any clue of what lies beneath the sensor data that contains information In conventional wireless networks, sensed field.” for multiple objects of interest. “This multiple nearby transmitters cannot trans- One of the project’s major goals is gives rise to overlapping information mit simultaneously on the same frequency, to cluster data into groups containing clusters that are well known to chal- since the signals would collide and be- specific information about different lenge all existing clustering techniques,” come unreadable. MegaMIMO, however, sources or entities of interest. “Distrib- Schizas says. is designed to enable multiple independent uted processing techniques that do not Schizas is satisfied with the progress transmitters to transmit to multiple receiv- require a central processing center are made to date. “So far, we have developed ers at the same time and on the same fre- also being developed,” Schizas says. a novel combination of CCA with princi- quency and still allow receivers to decode Schizas is using canonical correla- pal component analysis to identify sensor their signals. tion analysis (CCA) to reveal correlated data that contains information about mul- “Of course, the signals collide, which data that contains similar information tiple sources and determine in that way is unavoidable,” says Hariharan Rahul, content. “Further, norm-one regulariza- the overlapping information clusters,” a former research team member and tion is combined with CCA to identify he says. Yet the current approach works currently a visiting researcher with the the specific entries that have similar only for linear models, not for nonlinear project. “But MegaMIMO access points information content and perform clus- data models. “Our goal is to generalize modify the transmitted signals so that tering,” he says. The proposed frame- our framework to address the nonlin- at each receiver, after collision, only the work is solved using a block coordinate ear case,” Schizas says. “Further, the desired signal to that receiver survives.” descent approach; principal component presence of nonstationary and time- MegaMIMO promises a several-fold analysis is employed to determine the varying statistics is another challenge that increase in wireless network throughput number of underlying sources/objects we are currently trying to address, relying compared to existing wireless networks, of interest. “Further, moving averag- on online and adaptive learning.” says Rahul. “Further, MegaMIMO can ing and least-mean squares filtering are Schizas notes that the project is do this simply by replacing the access employed to perform denoising and sig- still relatively new and that much work points, and without requiring any hard- nal reconstruction,” he notes. still remains to be done. “There is no ware or software modifications to end The project promises to open new commercial interest yet, but as we user devices,” he notes. ways for data-driven data clustering improve upon computational complex- The key enabling technology behind where there is no need to rely on avail- ity and generalize the applicability of JMB is a new low-overhead technique able statistical data models. Learn- the proposed algorithms we expect to for synchronizing the phase of multiple ing algorithms are being developed permeate benefits in existing sensing transmitters in a distributed manner. that solely rely on the available data to systems and raise commercial interest,” The design allows a wireless LAN to perform information clustering. “The he says. scale its throughput by continually add- proposed framework is pretty flexible, ing more APs on the same channel. and it is expected to create benefits in Collision-free Wi-Fi The researchers recently tested JMB many areas, including target tracking, Researchers in the Massachusetts with both software radio clients and machine learning, and image process- Instute of Technology’s (MIT’s) Com- off-the-shelf 802.11n cards in a deploying,” Schizas says. puter Science and Artificial Intelligence ment that simulated a densely congested Schizas hopes that the project will Lab (CSAIL) have developed a wireless conference room (Figure 3). Results from eventually lead to self-organizing sen- technology that promises to triple the the ten-access point software-radio test sor networks incorporating a variety speed of data transfers while also dou- bed showed a linear increase in network of positive attributes, including low bling signal range. throughput with a median gain of 8.1 to energy demands, robust architectures, The researchers, led by Dina Katabi, 9.4×. The results also showed that JMB can and prolonged life expectancies, dean MIT professor of electrical engi- provide throughput gains with standard, un- ployed in fields such as health care, de- neering and computer science, recent- modified 802.11n cards. fense, and structural and other types of ly demonstrated MegaMIMO 2.0, a MegaMIMO uses a variety of signal monitoring. “Especially in applications new multiple-input, multiple-output processing algorithms and techniques,

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 11

THE WORLD’S NEWSSTAND®

of the intended effect,” Katabi explains. MegaMIMO uses signal processing to enable access points to process signals in a synchronized manner, providing a light- weight, distributed approach that requires only minimal changes to the existing Wi-Fi wireless processing pipeline. In real-world applications, MegaMIMO promises to dramatically improve throughput in the dense wireless networks—both Wi-Fi and cellular—commonly deployed in large, public places, such as sports sta- diums, convention centers, hotels, airports, and shopping malls. The researchers are now focused on scaling up the prototype system into MIT FIGURE 3. MegaMIMO 2.0 research team members doctoral student Ezzeldin Hamed, visiting re- larger deployments consisting of scores of searcher Hariharan Rahul and Prof. Dina Katab with a prototype of their technology, which promises Wi-Fi access points. “We have had inter- to transfer wireless data over three times faster than existing systems. est from a variety of players in the wireless space, as well as end users that are including various properties of orthogo- “The Wi-Fi transmitters have to be syn- currently facing challenges with dense nal frequency-division multiplexing sig- chronized in time very tightly,” Katabi wireless scenarios,” Katabi says. nals, Fourier transforms and translation says. Such synchronization must occur between time and frequency domains, as on a nanosecond scale. Also, since Wi-Fi Author well as the efficient application of linear signals are waves, two waves can combine John Edwards (jedwards@john______time-invariant filters to signals. to cancel each other out or, on the other edwardsmedia.com) is a technology writ- As they developed the technology, the hand, enforce each other. “If you are not er based in the Phoenix, Arizona, area. researchers faced a challenge in coordi- careful about phase synchronization, the nating time and phase synchronization. wave can combine to create the opposite SP

6LJQ8SRU5HQHZ

DĞŵďĞƌƐŚŝƉŝŶƚŚĞ/^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐ^ŽĐŝĞƚǇ;^W^Ϳ͕ƚŚĞ/͛ƐĨŝƌƐƚƐŽĐŝĞƚǇ͕ĐĂŶŚĞůƉ ǇŽƵůĂǇƚŚĞĨŽƵŶĚĂƚŝŽŶĨŽƌŵĂŶǇǇĞĂƌƐŽĨƐƵĐĐĞƐƐĂŚĞĂĚ͗ ͻKEEdǁŝƚŚŵŽƌĞƚŚĂŶϭϵ͕ϬϬϬƐŝŐŶĂůƉƌŽĐĞƐƐŝŶŐƉƌŽĨĞƐƐŝŽŶĂůƐƚŚƌŽƵŐŚ^W^ĐŽŶĨĞƌĞŶĐĞƐ͕ ĂŶĚůŽĐĂůĞǀĞŶƚƐŚŽƐƚĞĚďǇŵŽƌĞƚŚĂŶϭϳϬ^W^ŚĂƉƚĞƌƐǁŽƌůĚǁŝĚĞ͘ ͻ^sǁŝƚŚŵĞŵďĞƌĚŝƐĐŽƵŶƚƐŽŶĐŽŶĨĞƌĞŶĐĞƐĂŶĚƉƵďůŝĐĂƚŝŽŶƐ͕ĂŶĚ ĂĐĐĞƐƐƚŽƚƌĂǀĞůŐƌĂŶƚƐ͕^ŝŐWŽƌƚƌĞƉŽƐŝƚŽƌǇ͕ĂŶĚ^W^ZĞƐŽƵƌĐĞĞŶƚĞƌ͘ ͻsEǁŝƚŚǁŽƌůĚͲĐůĂƐƐĞĚƵĐĂƚŝŽŶĂůƌĞƐŽƵƌĐĞƐ͕ĂǁĂƌĚƐĂŶĚ ƌĞĐŽŐŶŝƚŝŽŶƐ͕ĂŶĚƐŽĐŝĞƚǇͲǁŝĚĞǀŽůƵŶƚĞĞƌŽƉƉŽƌƚƵŶŝƚŝĞƐŝŶ ƉƵďůŝĐĂƚŝŽŶƐ͕ĐŽŶĨĞƌĞŶĐĞƐ͕ŵĞŵďĞƌƐŚŝƉ͕ĂŶĚŵŽƌĞ͘ >ĞĂƌŶŵŽƌĞĂďŽƵƚŵĞŵďĞƌƐŚŝƉŽƉƚŝŽŶƐ;ŝŶĐůƵĚŝŶŐĐŚŽŝĐĞƐŽĨĞůĞĐƚƌŽŶŝĐĂĐĐĞƐƐĂŶĚƉƌŝŶƚ ŽƉƚŝŽŶŽĨƚŚĞ/^ŝŐŶĂůWƌŽĐĞƐƐŝŶŐDĂŐĂǌŝŶĞ͕^W^ŝŐŝƚĂů>ŝďƌĂƌǇ͕ĂŶĚŵŽƌĞͿ͗ ŚƚƚƉ͗ͬͬƐŝŐŶĂůƉƌŽĐĞƐƐŝŶŐƐŽĐŝĞƚǇ͘ŽƌŐͬŐĞƚͲŝŶǀŽůǀĞĚͬŵĞŵďĞƌƐŚŝƉ

Digital Object Identifier 10.1109/MSP.2016.2647178

12 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

FROM THE GUEST EDITORS

Hana Godrich, Arye Nehorai, Ali Tajer, Maria Sabrina Greco, and Changshui Zhang

Special Article Series on Signal Processing Education via Hands-On and Design Projects

s professionals in the signal and teaching of the fundamental concept. At is mainstream, and significant efforts are A information processing field, we the same time, they are required to develop being made in this direction. An insight build on the tools and theories we teamworking skills, research and develop- into the implementation challenges of learned in our undergraduate studies, ment experience, and innovative thinking, design projects and experimental plat- adding knowledge and skills over the which is achievable through full-scale forms from students in their freshmen years. For many of us, it has been a while engineering design projects. For the latter, through senior years and solutions adopt- since our undergraduate studies, yet we students need to collaborate on more com- ed to address them are offered in this can probably recall some of these “Ah- plex engineering problems that integrate a issue of IEEE Signal Processing Maga- ha” moments when a full meaning of larger set of tools and disciplines to solve zine (SPM) through a series of article con- some fundamental theory sank in. As and work under realistic constraints. The tributions from around the world. educators and mentors, we seize on these diversity in engineering applications that Schäck, Muma, and Zoubir’s article, moments and our own experiences to utilize signal and information processing “Signal Processing Projects at Technische implement new successful teaching opens many possibilities when it comes Universität Darmstadt,” details year-by- methods that advance effective learning to the choice of experiments and projects year practices implemented throughout and skill development. that will keep students engaged in learn- undergraduate and graduate studies to In an era of unprecedented technology ing. The implementation of such practices support students’ hands-on experience. refresh rate, the challenge of providing a becomes feasible with the availability of The curriculum builds up theoretical high-quality engineering experience is affordable hardware platforms that incor- knowledge alongside laboratories and compounded by the required theoreti- porate significant on-board computation engineering projects that advance profes- cal background, the increasing multidis- capabilities alongside access to sensors, sional proficiency. Interdisciplinary as- ciplinary applications, and a growing actuators, and open-source software tools. pects, laboratories infrastructure, and the demand from the industry for engineers There are few opportunities to share role of competitions in this process are with “know-how” skills. Schools need to progresses and innovation made through discussed. This overview offers the read- constantly assess and restructure the man- undergraduate engineering design proj- er an insight into use practices, detailing ner in which they prepare students to meet ects. An overview of the state-of-the-art their advantages and challenges. these growing demands from the engi- methods used in providing students with Focusing on engineering projects neering workforce. Educators are faced practical engineering education is of and competitions, Zhuo, Ren, Jiang, with the greater challenge of preparing high interest to educators, researchers, and Zhang’s, article, “Hands-On Learn- their undergraduate students to deal with and professionals. A discussion on what ing Through Racing,” on the National real-life engineering problems as early is done around the world to advance stu- Collegiate Intelligent Model Car Com- as possible in their education while not dents’ hands-on experience will provide petition in China, introduces an educa- compromising on the required theoreti- valuable tools and practices to educators tion-through-challenge approach. In an cal knowledge base. Engineering pro- and will offer professionals in the indus- annual competition, participating teams grams need to find an effective way to try with a clearer image of the efforts need to design and build cars that will be incorporate application aspects into the made to increase engineering skills dur- racing against other teams. The stu- ing undergraduate studies. dents learn a multitude of engineering Integrating more hands-on experi- skills while developing teamwork capa- Digital Object Identifier 10.1109/MSP.2016.2620199 Date of publication: 11 January 2017 ences into formal engineering education bilities and collaboration skills. The article

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 13

THE WORLD’S NEWSSTAND®

offers extensive details on the structure of an opportunity to work in collaboration We hope that the introduction of this these competitions and the skill sets devel- with others on more complex tasks, train- series of articles dedicated to signal and oped through it, enabling the adoption of ing students to learn teamwork skills and information processing in engineering this competition-based approach by others. project management. These collabora- projects will promote communication and A focus on communications-related tions frequently entail a multidisciplinary discussion on undergraduate studies, capa- practices is given in the article, “Teach- effort. Signal and information processing bilities development, and increase interest ing the Principles of Massive MIMO” by plays an important role in many of these and involvement from the engineering Larsson, Danev, Olofsson, and Sörman. engineering projects. community. We look forward to bringing This contribution details the development While there are some channels in you the next set of informative articles in of a course targeting students’ exposure which students can share and publish upcoming issues of the magazine. to cutting-edge technology and emerging their engineering projects, there is a need concepts. The course is designed around for a more focused review on engineering Guest editors

building system-level understanding and projects that offer great opportunities for the Hana Godrich (g__o expanding the classical curriculum to inte- implementation of signal and information [email protected])______grate a project-like approach. A student processing techniques. With the rapid received the B.Sc. de- perspective is given throughout the article advancement in technology and plat- gree in electrical engi- along with lessons learned. It demonstrates forms available for project development, neering from the the students’ experience and how students’ there is high value is sharing the knowledge Technion Israel Insti- feedback has been used to further develop and results stemming from these efforts to tute of Technology, Haifa, in 1987; the the course impact. advance the general community. An over- M.Sc. degree in electrical engineering view of practical educational tools, appli- from Ben-Gurion University, Beer- Complementing this issue cation challenges, and keys to successful Sheva, Israel, in 1993; and the Ph.D. Complementing these three feature arti- implementation of these programs is of high degrees in electrical engineering from cles are two articles published in SPM’s interest to both academia and the industry. the New Jersey Institute of Technology, “SP Education” columns in the July To address this need, as part of Newark, in 2010. She is the undergrad- and November 2016 issues, which this article series, SPM has opened a uate program director in the Electrical paved the way for us to introduce the SigPort-based submission and archival and Computer Engineering Department readers to this effort of sharing best platform for sharing students’ projects con- at Rutgers University, Piscataway, New practices on hands-on training in signal tributions. This issue’s “SP Education” col- Jersey. Her research interests include processing. In SPM’s July issue, Simoni umn is the first to detail these highlighted statistical signal processing with appli- and Aburdene [1] shared their eight- projects. Through the SigPort repository, cation to wireless sensor networks, year experience and lessons learned in a number of undergraduate students and communication, smart grid, and radar developing application-oriented activi- their advisors shared information on rel- systems. She is a Senior Member of ties to help students better understand evant engineering projects. Overall, the the IEEE. signal processing theory and connect submitted projects had more than 400

the theory to real-world applications. downloads within a two-month period, Arye Nehorai (nehorai@_____

In the November 2016 “SP Educa- showcasing the keen interest in the com- ese.wustl.edu)______received tion” column, Richter and Nehorai intro- munity for such information. Contribu- the B.Sc. and M.Sc. de- duced the incorporation of undergraduate tions from around the world cover diverse grees from the Tech- research projects as a key component in fields and projects reflecting signal and nion Israel Institute of the Electrical and Systems Engineering information processing opportunities and Technology, Haifa, and program at Washington University, St. applications range. the Ph.D. degree from Stanford Universi- Louis, Missouri [2]. Thanks to the active It is encouraging to learn that SPM and ty, California. He is the Eugene and Mar- involvement of signal processing faculty its monthly eNewsletter will be working tha Lohman Professor of Electrical members, many of the successful proj- with the IEEE Signal Processing Society’s Engineering in the Preston M. Green De- ects were related to signal processing, Education Committee and SigPort Com- partment of Electrical and Systems Engi- and these experiences substantially mittee to continue accepting student proj- neering at Washington University in St. boosted the undergraduate enrollment ect submissions and theses in the broad Louis. He was editor-in-chief of IEEE and retention rate and attracted students areas of signal and information processing Transactions on Signal Processing from to pursue a career in engineering. to archive through the SigPort platform. 2000 to 2002. From 2003 to 2005, he was Undergraduate engineering design Summaries of the projects selected from the vice president, Publications, of the projects, commonly introduced in a stu- these submissions will be periodically IEEE Signal Processing Society (SPS), dents’ junior and senior years, allow highlighted in Inside Signal Processing the chair of the Publications Board, and a them to work on real-life problems while eNewsletter; and, if space allows, some of member of the SPS’s Executive Commit- applying their acquired knowledge and these projects may be showcased in SPM’s tee. He received the 2006 IEEE SPS Tech- creativity. Some of these projects provide “SP Education” column. nical Achievement Award and the 2010

14 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

IEEE SPS Meritorious Service Award. He with applications to smart grid opera- Changshui Zhang was elected Distinguished Lecturer of the tions). He received the NSF CAREER ([email protected]______

IEEE SPS (2004–2005). He received sev- Award in 2016. He is a Senior Member .cn)__ received the B.S. eral best paper awards in IEEE journals of the IEEE. degree in mathematics and conferences. He is a Fellow of the from Peking Universi- IEEE, the Royal Statistical Society, and Maria Sabrina Greco ty, Beijing, China, in

American Association for the Advance- (m.greco@iet.______unipi.it) 1986 and the Ph.D. degree from the ment of Science. graduated with a Department of Automation, Tsinghua degree in electronic University, Beijing, in 1992, where he is

Ali Tajer (tajer@ecse______engineering in 1993 currently a professor. He has authored

.rpi.edu)_____ received the and received the Ph.D. more than 200 papers published in jour- M.A. degree in statis- degree in telecommunication engineering nals and conferences. His current tics and Ph.D. degree in 1998, from the University of Pisa, research interests include machine learn- in electrical engineer- Italy, where she has been an associate ing, artificial intelligence, image processing from Columbia professor since December 2011. She is ing, pattern recognition, and evolutionary University, New York, in 2007 and 2010, an associate editor of IET Proceedings– computation. He is also an associate edi- respectively. He is currently an assistant Sonar, Radar, and Navigation, editor-in- tor of Pattern Recognition. professor of electrical, computer, and chief of IEEE Aerospace and Electronic systems engineering at Rensselaer Poly- Systems Magazine, member of the edito- References technic Institute, New York. His research rial board of the Springer Journal of [1] M. Simoni and M. Aburdene “Lessons interests include mathematical statistics Advances in Signal Processing, and learned from implementing application-orient- and network information theory with senior editorial board member of IEEE ed hands-on activities for continuous-time sig- applications in wireless communications Journal on Selected Topics of Signal Pro- nal processing courses,” IEEE Signal Process. and power grids. He serves as an editor cessing. Her general interests are in the Mag., vol. 33, no. 4, pp. 84–89, July 2016. for IEEE Transactions on Communica- areas of statistical signal processing, esti- [2] E. Richter and A. Arye Nehorai, “Enrich- tions and IEEE Transactions on Smart mation, and detection theory. She coau- ing the undergraduate program with research Grid and as the guest editor-in-chief for thored many book chapters and more projects,” IEEE Signal Process. Mag., vol. 33, IEEE Transactions on Smart Grid (spe- than 170 journal and conference papers. no. 6, pp. 123–127, Nov. 2016. cial issue on theory of complex systems She is a Fellow of the IEEE. SP

'R\RXNQRZ"ŽŐŝŶŽŶǁǁǁ͘ƐŝŐƉŽƌƚ͘ŽƌŐƵƐŝŶŐ/ǁĞďĂĐĐŽƵŶƚĐƌĞĚĞŶƚŝĂůƐ͘ 'ŽƚŽ͞ƐƵďŵŝƚǇŽƵƌǁŽƌŬ͟ŽŶƚŚĞƚŽƉŵĞŶƵĂŶĚƵƐĞƉƌŽŵŽƚŝŽŶ ĐŽĚĞǇŽƵϭϰϮϬϬĨŽƌĨƌĞĞƵƉůŽĂĚ͘ ͻĞǇŽŶĚƐůŝĚĞƐĂŶĚƉŽƐƚĞƌƐ͗^ŝŐWŽƌƚǁĞůĐŽŵĞƐƌĞƐĞĂƌĐŚĚƌĂĨƚƐ͕ǁŚŝƚĞ ƉĂƉĞƌƐ͕ƚŚĞƐĞƐ͕ƐůŝĚĞƐ͕ƉŽƐƚĞƌƐ͕ůĞĐƚƵƌĞŶŽƚĞƐ͕ĚĂƚĂƐĞƚĚĞƐĐƌŝƉƚŝŽŶƐ͕ƉƌŽĚƵĐƚ

ďƌŝĞĨ͕ĂŶĚŵŽƌĞ͘^ĞŶĚƋƵĞƐƚŝŽŶƐŽƌĐŽŵŵĞŶƚƐƚŚƌŽƵŐŚǁǁǁ͘Ɛŝ______ŐƉŽƌƚ͘ŽƌŐͬĐŽŶƚĂĐƚ͘ Digital Object Identifier 10.1109/MSP.2016.2643718 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 15

THE WORLD’S NEWSSTAND®

SIGNAL PROCESSING EDUCATION VIA HANDS-ON AND DESIGN PROJECTS

Tim Schäck, Michael Muma, and Abdelhak M. Zoubir

Signal Processing Projects at Technische Universität Darmstadt How to engage undergraduate students in signal processing practice

his article is meant to share our experience on integrating signal processing hands-on opportunities into formal T engineering education at Technische Universität (TU) Darmstadt, Germany. As many universities face the challenge of how best to provide hands-on experience to undergraduate students, we hope to inspire our colleagues and perhaps trigger new hands-on projects by sharing our insights. At TU Darmstadt, we believe that it is essential to provide undergraduate students with hands-on signal processing opportunities right from the starting point of their studies through graduation.

Introduction Hands-on education in signal processing has a long-standing tradition (e.g., [1]–[5]), and its importance, given the complexity of today’s engineering problems, is undisputed. At TU, we hope that we can—in one way or another— inspire some of our colleagues who are involved in educating the next generation of signal processing researchers and practitioners. We will briefly explain the format of the projects and highlight some important challenges in the implementation as well as successful strategies and pitfalls that we encountered. The time line of the curriculum, as shown in Figure 1, serves as a structure to present material in MAIN IMAGE ©ISTOCKPHOTO.COM/SKYNESHER; SCREEN IMAGE ©ISTOCKPHOTO.COM/ALENGO an ordered fashion. However, all sections can be read independently. We also illustrate how we utilize student competitions, such as the IEEE Signal Processing Cup, to stimulate innovation and collaboration between graduate and undergraduate students. Special attention is given to the many possibilities that collaborations with industry partners offer for students. The involvement of students in interdisciplinary research, which has a long-standing tradition at TU Darmstadt, is illustrated by the example of a cooperation between the signal processing group and the psychology group. Laboratories are central to our hands-on education for fresh- Digital Object Identifier 10.1109/MSP.2016.2619218 Date of publication: 11 January 2017 men to senior-year students. By promoting and extending our

THE WORLD’S NEWSSTAND®

labs, we increase the exposure to state-of-the-art research Signal processing within the curriculum of electrical and advanced equipment to our undergraduate students. For engineering and information technology at this reason, a separate section is devoted to presenting our TU Darmstadt diverse signal processing laboratories and the opportunities In Germany, the format of undergraduate education in they offer our students. Some of the laboratories are purely electrical engineering and information technology (ETiT) educative, while others additionally pro- was traditionally the diploma degree with vide the advanced students with hands- At TU Darmstadt, we a duration of five years. More recently, on research opportunities. The real-data believe that it is essential due to changes relating to the Bologna experiments include fascinating topics, Declaration, a more internationally such as real-time audio signal process- to provide undergraduate acknowledged system has been installed. ing, multiantenna receive beamforming, students with hands- It includes a three-year bachelor’s degree biomedical signal processing, geolocation, on signal processing followed by a two-year master’s degree and tracking, to mention a few. Our new- opportunities right from as new course structures. In this section, est laboratory considers the cutting-edge the starting point of their we describe how hands-on signal pro- research topic of bioinspired communi- studies through graduation. cessing projects are integrated into the cation. Here, students can generate their curriculum of ETiT at TU Darmstadt. At own data by performing single-cell exper- the end of each project, we list the posi- iments involving fluorescence microscopy and microfluid- tive aspects with 5, pitfalls with K, and additional hints ics. Based on their own data sets, the students can develop with 9. advanced signal and image-processing algorithms, e.g., for segmentation and tracking of single cells. We conclude First year with some practical general remarks on some fundamental At TU Darmstadt, we believe that it is essential to provide aspects that we have found to be important for successful undergraduate students with hands-on opportunities right design projects. Short interviews as well as photos and fig- from the starting point of their studies. During this phase, ures are used to make the article an enjoyable and informa- freshmen are especially motivated and highly curious. The tive read after a hard day inside signal processing. lack of fundamental knowledge in engineering and science is

Technical Complexity of Problems, Knowledge, Skills, and Independence of Students Master’s Project Advanced Signal Processing Seminars DSP Practical Bachelor’s Project Audio Pro-/Project Seminar Bio

RBL DSP Labs SPG CSS Engineering Practicals

Engineering Introductory Project

Education Research

Year 1 Year 2 Year 3 Year 4 Year 5

FIGURE 1. An overview of the hands-on activities in signal processing within the curriculum of electrical engineering and information technology at TU Darm- stadt. We offer a variety of Digital Signal Processing (DSP) Labs: the Communication and Sensor Systems (CSS) Lab, the Receive Beamforming Lab (RBL), the Bioinspired Communication Systems (Bio) Lab, the Advanced Real-Time Audio Processing (Audio) Lab, and the Signal Processing Group (SPG) Lab.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 17

THE WORLD’S NEWSSTAND®

often compensated by common sense coupled with creativity. to a cookie-baking machine (2010), to contributing to future Taking all of this into account, the Department of Electrical living (2013). Engineering and Information Technology created an intro- During the project, two RAs serve as advisers for each ductory project for freshmen in 2007. group. On one hand, the soft skills adviser, i.e., an RA from the Department of Humanities, assists in Engineering Introductory Project creating an encouraging group dynamic In the Engineering Introductory Project, Through the and helps the team in reflecting their interdisciplinary groups of approximately interdisciplinary exchange, teamwork and interactions. On the other ten students work together on a technical students are given the hand, the technical adviser, i.e., an RA of solution to a timely, practical, complex, and opportunity not only to the Department of Electrical Engineer- socially relevant problem (Figure 2). The improve their technical ing and Information Technology, answers project takes place when students are only questions regarding the technical aspects one to two months into their first semester, skills but also to develop and encourages the group to use engineer- and it is a welcome contrast to the funda- soft skills, such as ing tools. Members of the SPG participate mental coursework that is usually offered in teamwork and as technical advisers to guide and moti- this early phase of the curriculum. The over- self-organization. vate the freshmen with a special focus on all number of participating students is signal processing. approximately 500. Through the interdisciplinary exchange, The focus of the Engineering Introductory Project does students are given the opportunity not only to improve their not rely on technical details. Rather, the first insights into technical skills but also to develop soft skills, such as team- today’s engineering work in an interdisciplinary environ- work and self-organization. In this way, the students get an ment is provided. The project also gives the opportunity to impression of what awaits them later as professional engineers. make friends with classmates and to establish a first contact Also, students begin to network at an early stage, even between with the research associates (RAs) who serve as supervi- different disciplines. sors. RAs in Germany are appointed to assist professors with This project not only fosters didactic and technical learn- teaching, research projects, and, at the same time, to pursue a ing in a team with other students of different interests but the doctoral degree (Dr.-Ing.). This path excludes formal classes, freshmen also receive expert guidance from the professors of thus, the term research associates. The peers in the group ETiT in dedicated consultation hours. Here, they are given learn together and make practical experiences, which creates the opportunity to discuss their ideas and ask questions relat- a pleasant and inspiring environment, an inevitable require- ed to their project with a specialist in this particular area. For ment for creativity. example, Prof. Abdelhak Zoubir offers a consultation hour The topic of the Engineering Introductory Project is not in which freshmen ask questions about challenges related to announced before the start of the project. Each team has one the field of signal processing that they have identified within week to jointly develop an innovative solution. The team’s their project. final results are formally presented in front of a jury that is One challenge in implementing this design project is the composed of professors. Starting in 2007, the topics of the choice of an appealing and trendsetting topic. Its technical Engineering Introductory Projects were as diverse as devel- complexity must be adjusted to the students’ knowledge and oping a power supply package for outdoor holidays (2008), the given time frame. To test and evaluate possible solutions and to discover potential pitfalls throughout the project, the soft skills and technical advisers simulate the Engineer- ing Introductory Project task beforehand within a period of three days. Their experience flows back into the project description and enhances the quality of the design project. ■ 5 Freshmen practice working independently in interdisciplinary teams ■ 5 a first exposure to signal processing problems ■ K limited prior knowledge is assumed ■ 9 the choice of topic is essential.

Second year Before students in electrical engineering can be exposed to real-world problems in signal processing, they have to study the fundamentals of signal processing. In the second year, FIGURE 2. An interdisciplinary group of freshmen working together on their our students learn the basic concepts of signal processing Engineering Introductory Project. (Photo courtesy of Paul Glogowski/TU by taking the course, “Deterministic Signals and Systems” Darmstadt.) and “Fundamentals of Signal Processing.”

18 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Fundamentals of the signal processing unit nal processing: 1) the localization of acoustical sources, 2) The course, “Deterministic Signals and Systems,” teaches digital modulation, 3) multiple-input, multiple-output undergraduate students the principles of deterministic signals communication, 4) software-defined radio, 5) parasitic effects and system theory. It starts with the Fourier series and the Fou- in passive radio-frequency (RF) devices, 6) polarization of rier transform, treats linear time-invariant systems and convo- light, 7) RF field-effect transistor amplifier, and 8) the fields lution, and gives an overview of other signal transformations, and impedance of antennas. The students are guided to such as the Laplace transform, the z-transform, and the dis- acquaint themselves with each topic and are required to write crete-time Fourier transform. Students apply these transforma- reports about the conducted experiments. tions to solve tasks related to physical problems that are To illustrate the hands-on signal processing opportunities modeled by linear differential equations. offered to the students, consider the localization of an acous- Subsequently, the course, “Fundamentals of Signal Process- tical sources experiment. Here, students are given the oppor- ing,” covers basic concepts in signal processing, such as random tunity to localize acoustical sources in our laboratory. To this variables, stochastic processes, random signals and linear timeend, the third-year students estimate the time-differences invariant systems, optimal linear systems such as the matched of arrival and angles of arrival (AOA) using correlation and filter and the Wiener filter, and the method of least-squares. generalized correlation functions. For the final audio source Practical experiments with real-world data are shown in the localization, the students fuse multiple AOA measurements lecture to help the students grasp the basic ideas in an intuitive from distributed microphone arrays, as shown in Figure 3. The fashion. The aim of the lecture is, furthermore, to serve as an eight microphones are divided into pairs that are mounted on introductory course for more advanced lectures in DSP, adaptive the four walls of the laboratory (see Figure 4). The positions filtering, communications, and control theory. of the microphone arrays are given by p1, …, p 4, whereas the As for many other fundamental courses in ETiT at TU sound source is located at some unknown position in the center Darmstadt, the lectures are complemented by tutorials that of the room. The measurements are recorded with eight stan- are held by selected undergraduate teaching assistants (UTAs) dard Behringer B-5 condenser microphones and a Behringer of higher semesters. The advantage of having students do the Ultragain Pro-8 digital device, which is an eight-channel teaching instead of RAs or professors is that higher-semester analog/digital and digital/analog converter. All calculations students are aware of the difficulties from their own recent are performed in MATLAB. experience and can adequately support the more junior stu- In this laboratory, third-year students experience their first dents. By recruiting good students as UTAs, signal process- hands-on experiment. They can apply the freshly learned fun- ing can also be effectively advertised to more junior students. damentals of signal processing and have to submit their results The UTAs who run the tutorials strengthen their knowledge in a written report. This lab also fosters the ability to work in in signal processing and integrate more into our research teams. There exist some pitfalls regarding teamwork for hands- group. They usually conduct their bachelor’s or master’s the- on signal processing. First, it might occur that teams do not sis project with us and participate in other research projects distribute the work load evenly between the team members. or competitions. Second, the team members sometimes split up the work such ■ 5 Practical experiments with real-world data in the lec- that only some members run the hands-on experiments while tures help students grasp basic ideas in an intuitive way others write the report. In such cases, the laboratory adviser ■ K the involvement of UTAs as a means of integration needs to remind the students to participate in the hands-on ■ 9 the basic knowledge of signal processing is still missing experiments at every stage. at this point. ■ 5 The first hands-on experience in a DSP laboratory ■ 5 the fundamentals of signal processing are practiced Third year using real experiments In the third year, we offer a variety of hands-on opportuni- ■ 5 fosters the ability to work in teams ties ranging from practical signal processing experiments in ■ K sometimes weaker team members are less active during laboratories to small-scale research projects (Proseminar/ the experiments Project Seminar) and the bachelor’s thesis project. Students ■ 9 workload should be evenly distributed among team also can find their first exposure to interdisciplinary members. research projects in the Forum for Interdisciplinary Research As preparation for larger undergraduate research proj- Project (see “The Forum for Interdisciplinary Research ects, such as the bachelor’s thesis project, TU Darmstadt has Project”). In this section, we briefly explain the format of introduced the Proseminar and Project Seminar. These are the third-year projects, highlight the most important chal- described in the following sections. lenges in their implementation, and discuss our own successes and pitfalls. Proseminar Scientific work always starts with understanding the state of the Communication and sensor systems laboratory art. The students are expected to be informed about the research This practical consists of eight fundamental hands-on experi- that has already been conducted in the field of interest, first, ments from the field of communication engineering and sig- before reimplementing successful methods or even examining

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 19

THE WORLD’S NEWSSTAND®

The Forum for Interdisciplinary Research Project

Clearly, the most obvious way to provide students with processing psychophysiological data is challenging. Often hands-on experience in signal processing is to let them motion artifacts affect physiological time series. However, solve real-world problems. Interdisciplinarity naturally the data also offers the hands-on possibility to study many comes into play when dealing with hands-on research fundamental concepts, such as filter design, time-frequency projects that solve real-world problems, and it requires dif- and wavelet analysis, adaptive filtering, robust statistics for ferent types of expertise to be combined. TU Darmstadt dependent data, parameter and signal estimation, detec- has a long-standing culture of cooperation across depart- tion, and classification. ment boundaries. In this way, we supplement the classical One of the signal processing projects, which we conduct- department structure through flexible cooperation forms for ed during a master’s thesis project within the FiF, focused research and teaching. The Forum for Interdisciplinary on motion artifact removal in electrocardiographic (ECG) Research (FiF) builds on the successful interdisciplinary signals. The developed method had to provide satisfactory work at TU Darmstadt. The FiF was founded as a result of results for a large range of data and also be computation- a senate decision in December 2008. ally efficient. Further, it had to be programmed in a way For example, in 2012, FiF funded a research coopera- such that psychology students who did not have any signal tion between Prof. Abdelhak M. Zoubir (signal processing) processing background would be able to use it. All of and Prof. Augustin Kelava (psychology). The joint research these requirements together provided a realistic hands-on topic was to investigate the synchronization of physiologi- framework to conduct a master’s thesis project, which cal signals in emotional situations. Emotion-eliciting situa- resulted in a publication at the European Signal Processing tions are accompanied by changes of multiple variables Conference 2012 [7]. associated with subjective, physiological, and behavioral We observed that the interdisciplinarity of the FiF project, responses. The quantification of the overall simultaneous as well as the feeling of being able to solve real problems, synchrony of psychophysiological reactions plays a major created a unique team spirit among the students. Even role in emotion theories and has received increased atten- today, signal processing students use the databases that tion in recent years. From a psychometric perspective, the were established to develop and evaluate new methods. reactions represent multivariate nonstationary intraindividu- Also, psychologists use the signal processing methods to al time series. study the synchronization of psychophysiological signals in Undergraduate signal processing and psychology stu- the body in many different emotion-eliciting situations dents, supervised by research associates (RAs) and profes- (Figure S1). New challenges and opportunities arise from sors, cooperated to robustly determine the synchrony of the possibility to integrate wireless body-worn sensors into psychophysiological reactions. Within the FiF project, the psychological experiments. This allows the psychologists bachelor’s and master’s thesis projects were undertaken in to undertake more realistic experiments and provides the both departments. The cooperation quickly revealed that signal processing students with new and even more challenging data sets. t 5 Psychophysiological data is challenging and requires the study of fundamental concepts t 5 databases, once established, can be reused by other signal processing students t 5 the interdisciplinary nature of the project requires explaining fundamental concepts without using equations t 5 excellent results could be obtained, and publications as well as bachelor’s and master’s thesis projects were produced t K it takes a long time until students from both research fields speak the same language t K student projects are of a short duration and tend to conclude when a student has reached his/her peak in productivity FIGURE S1. An undergraduate student at TU Darmstadt records physiologi- t 9 documentation of the methods and datasets is essen- cal signals while watching an emotion-inducing video clip. tial in interdisciplinary research.

20 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

new approaches. This literature survey is the subject of the Proseminar that, as a first step toward scientific work, lays the 6 5 foundation for performing hands-on research projects later on. It takes about four weeks to accomplish the Proseminar. p3 = (2.70, 5.27) During the Proseminar, students read books, papers, or complementary work on a given subject in signal processing. Source Location p∗ The topics are not limited to fundamentals but may also cover 7 Unknown 4 very recent papers and approaches. To finalize the Prosemi- 8 3 p2 = (5.19, 3.21) nar, the third-year students summarize their results in a written p4 = (0.37, 3.54) report that is checked by the supervising RA. Through the intensive literature survey, the students learn y to understand, analyze, and summarize scientific papers. p1 = (2.69, 0.19) However, the papers must be carefully selected. They are usu- x ally aligned with the doctoral research of the supervising RA. 12 Another challenge is to define a project such that it can be completed within four weeks while also considering the students’ FIGURE 3. The setup of the Localization of Acoustic Sources experiment prior knowledge. that is part of the Communication and Sensor Systems Laboratory offered to third-year students. Project Seminar After having reviewed the literature in the Proseminar, students investigate and solve a specific signal processing problem by reproducing an already existing approach. They typically reimplement an algorithm from a published paper and try to reproduce its results. During the project, students are allowed to suggest their own modifications and extend the methods to improve the results. Such creative contributions yield a bonus when it comes to grading. However, their own contributions are not required and are primarily the aim of the subsequent bachelor’s thesis project. The reproducibil- ity of the publications is essential, and the paper content and FIGURE 4. A microphone pair of the Localization of Acoustic Sources quality must be checked by the supervising RA beforehand. experiment that is attached to one of the walls of the Signal Processing In addition to the reimplementation, students search for and Laboratory. analyze scientific reference publications and, in the end, summarize the obtained results and their conclusions in a written Bachelor’s thesis project report. The outcomes of the Project Seminar are defended in The third year concludes with the bachelor’s thesis project, front of the research group and students in an oral presentation. which usually builds upon the Proseminar and Project Semi- The duration of the Project Seminar is about two months. nar. The bachelor’s thesis project is designed to last three For the students, the Project Seminar is one of the first months. The length of the bachelor thesis is typically 40–80 opportunities to develop skills in MATLAB, Latex, and Bib- pages. Approximately 100 students in ETiT finish their bache- TeX, which are necessary tools for scientific work in signal lor thesis each year. Similar to the Project Seminar, the stu- processing and the basis for subsequent hands-on projects. The dents have to give a 20-minute presentation and defend their aim of the Project Seminar is to practice applying methods work in front of an audience. The bachelor’s thesis project of signal processing to practical problems and to gain some offers the possibility for students to be creative and to develop knowledge in a particular research area, which can be built new ideas and algorithms. Students can either explore new upon in later projects. As for the Proseminar, the challenge is ways of solving a specific problem, compare different meth- to define the project such that it can be completed on time. ods and their performance, or improve an existing approach Often, the workload for the RA who supervises the projects is by extending or enhancing particular aspects. In general, the high, as regular meetings with students are essential to ensure bachelor’s thesis project is based on research questions pro- a high quality of work. Also, the corrections of the report often vided by the RA. In our group, we offer hands-on experi- include tedious linguistic corrections, since the students are ments using, e.g., biomedical, audio, or ultrasound data, which not yet acquainted with scientific English. can be acquired by the students in our signal processing lab. ■ 5 Develop skills in MATLAB, Latex, and BibTeX Outstanding bachelor’s thesis projects can lead to publications ■ 5 apply signal processing methods to practical problems and visits to conferences such as the IEEE Workshop on Sta- ■ 5 deepen knowledge in a particular field tistical Signal Processing, the European Signal Processing ■ K the workload for the RA who supervises the projects is Conference (EUSIPCO), or the International Conference on high in relation to the outcome. Acoustics, Speech, and Signal Processing (ICASSP).

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 21

THE WORLD’S NEWSSTAND®

One of our undergraduate students who was published at ■ 5 students can build upon the work they did in the ICASSP’14 was Jack Dagdagan (see Figure 5). In his paper Proseminar and Project Seminar [6], he developed a robust method for testing stationarity in the ■ 9 acquiring real data and working with it must be well presence of outliers. Jack recollects, “In my bachelor’s thesis planned, otherwise it would exceed the three-months nomi- project, I evaluated my algorithm first with simulated data nal length and assumed a certain outlier model. I was not sure whether ■ 9 all of the aforementioned projects, i.e., Proseminar, the method would still perform reasonably well when using Project Seminar, and the bachelor’s thesis project, are grad- real data. So I was very excited when I started recording real ed after the seminar by the professor. data. I noticed that the computational complexity was much higher with real data, since the simulated data had a sample Fourth and fifth year size of only 1,024 and I recorded some seconds of audio with The following programs and activities are part of the mas- a 48,000-Hz sample rate. I realized that the outliers were ter’s program of ETiT, which forms the final two-year stage completely different to the outliers in the simulations. But of the undergraduate education at TU Darmstadt. I was very happy that the performance of my method was still very good. If you develop an algorithm DSP Practical and evaluate with simulations, you could In their fourth and fifth Fourth-year students can attend the DSP optimize by tweaking your parameters such Practical, either in parallel to or after the that your algorithm fits the model perfect- years, signal processing course “Digital Signal Processing.” It ly. However, if your model is very special, students at TU Darmstadt offers the chance to further familiarize your method won’t work in reality. But if are ready to tackle some onself with signal processing programs, you test your algorithm on real data and still more challenging and such as MATLAB, and put theory into achieve good results, it shows that either realistic problems. practice. Students participating in this lab your model was very diverse or that your are able to apply the concepts from the algorithm runs very well independently of lectures. This covers mainly the design of the model you are using.” finite impulse response and infinite impulse response filters Another lesson that Jack learned from his hands-on expe- as well as parametric and nonparametric spectrum estima- rience is the difference between theory and practice. “In tion; examples of the latter are shown in Figure 6. Real-world theory, you learn the definitions of stationarity, such as wide- signals, such as speech and audio signals, touch-tone tele- sense stationarity (WSS),” he says. “However, in practice, you phone dialing signals, temperature recordings, or biomedical realize that you can never have perfect WSS. Thus, you need measurements, are either provided to or recorded by the stu- to set a threshold above which you determine the signal not dents. UTAs help to supervise the undergraduate students to be stationary anymore. Before my hands-on experience in during the experiments. For example, the biomedical experi- this bachelor’s thesis project, I would not have thought of sta- ment, where students record each others electrocardiogram tionarity in this way.” (ECG) signal and perform spectrum estimation, was designed ■ 5 Outstanding bachelor’s thesis projects that use real- with the help of a UTA. The experiments are conducted in the world data can lead to publications and conference visits SPG Lab, which is described in the “Laboratories” section. The DSP Practical is composed of eight practicals and two real-data acquisition sessions. Approximately ten groups of two to three students work together to solve signal processing tasks. As an introductory part for every experiment, students receive handouts with the underlying theory and some prepa- ratory questions. The students’ understanding of the theory is checked by the supervisor at the beginning of each experiment. In this way, we ensure the students’ adequate preparation for the practicals. Furthermore, for every experiment, each group writes a report in which they wrap-up their results, answer all questions, and include plots and code from the experiment. At the end of the semester, a final exam is held. ■ 5 Students are able to apply concepts learned in the DSP course using real-world data ■ 5 students can further familiarize themselves with signal processing tools, such as MATLAB ■ 5 students can acquire their own measurements ■ K FIGURE 5. Jack Dagdagan, an undergraduate student at TU Darmstadt, the time slot per experiment is tight presenting the results of his bachelor’s thesis project at ICASSP’14 in ■ K tasks are explicitly predefined, and the time for trial and Florence, Italy. error is very limited.

22 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Advanced seminars in signal processing In their fourth and fifth years, signal processing students at TU Darmstadt are ready to tackle some more challenging and realistic problems. At this time, they have acquired sufficient fundamental knowledge, and they are also used for reading and reproducing results from papers. The SPG offers an advanced seminar in which master’s students tackle small- scale real-world problems to prepare students for their mas-

ter’s thesis project, as well as to provide the opportunity to get Parametric Power to know the RAs and practice working in teams. The seminar, Spectral Density Estimate “Advanced Topics in Statistical Signal Processing,” is aimed at students who have an interest in signal processing and a 0 10 20 30 40 50 desire to extend their knowledge of signal processing in prep- Frequency (Hertz) aration for future project work, e.g., their master’s thesis proj- (a) ect and their working careers. The seminar consists of a short series of lectures (i.e., four to five), followed by student group projects (six to eight weeks), a presentation of the achieved results, and a final exam. Usually, up to 20 students participate in the seminars. The topics of the lectures and the student projects are different every year. The RAs are free to propose students’ projects, and the students make their choice based on their own interest. In this way, both the students and their supervisors are highly motivated. Nonparametric Power

Students often use the SPG Lab to investigate topics such as Spectral Density Estimate direction-of-arrival estimation or localization of sound sources in impulsive noise environments. In many cases, students 0 1020304050 become creative in the experiments. For example, one group Frequency (Hertz) used their mobile phones to play sound files (both signal and (b) noise) in combination with a miniature train to create audio sources that moved on a fixed trajectory. FIGURE 6. Examples of (a) a parametric power spectral density estimate ■ 5 Deepen the knowledge in signal processing and (b) a nonparametric power spectral density estimate. Both estimates ■ 5 preparation for final year master’s thesis project were computed during the DSP Practical with data collected by the undergraduate students. ■ 5 groups work on a common research topic and present results ■ 5 contact with RAs is intensified latter include, e.g., computational efficiency of the algorithm, ■ K time is often too short for the students to gain a deep communicational load between sensors, memory restrictions, insight. or real-time requirements. In our experience, the students enjoy incorporating such realistic requirements into their Final year master’s thesis project algorithm design. The four-semester master’s program in ETiT consists of com- It is not uncommon that the projects are carried out in pulsory core courses, compulsory elective courses, and elec- cooperation with a selected industry partner or a research cen- tive courses as well as the master’s thesis project. In their ter. In case of such cooperation, it must be emphasized that master’s thesis project, the students work independently for a we take special care to make sure that the master’s students duration of six months on a scientific project under the super- perform real-world signal processing research tasks under the vision of one of the RAs. The research topics are larger and supervision of a qualified supervisor. For this reason, coop- more complex compared to the bachelor’s thesis project. erations for master’s thesis projects are only possible with For many students, the master’s thesis project offers the trusted institutions with which a solid research partnership possibility to conduct research on real-world data. From our has been established. experience, the best results are obtained when the students are At TU Darmstadt, the students have the unique possibility involved in collecting their own data. In this way, they acquire to choose between a broad range of hands-on topics at national hands-on contextual information and can better understand and international locations. Examples of past SPG master’s the data, e.g., in terms of the signal quality, the measurement thesis projects of Prof. Zoubir in cooperation with research or principle or the assumptions on the noise distributions. Fur- industry partners are as diverse as ther, when the master’s thesis project solves a real-world prob- ■ signal processing for photometric glucose measurement in lem, the students take more care that the developed algorithms hand-held devices at a German pharmaceuticals and diag- are designed in accordance with practical requirements. The nostics company

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 23

THE WORLD’S NEWSSTAND®

■ room shape estimation in through-the-wall radar imaging in Technology.” A schematic sketch of the relation between the cooperation with the Center for Advanced Communications, eye’s wavefront aberration and the ECG, the BP, and respira- Villanova University, United States tion signals is shown in Figure 7. ■ directional integration of wirelessly transmitted audio sig- “Prof. Iskander has his own teaching philosophy [9], and, nals into hearing aid processing, and many more. after the master’s thesis project, it was clear to me that I Clearly, each of these applications has its own challenges. wanted to continue with signal processing research. Work- “When I was a master’s student at TU ing with real data in a team consisting of Darmstadt,” remembers Dr.-Ing. Michael For the development engineers and eye researchers allowed me Muma, now a postdoctoral researcher with of DSP algorithms that to grasp the importance of signal process- the SPG, “I was walking through the cor- show good and robust ing. Today’s measurement devices and ridor and saw a notice that immediately data analysis are too complex to be han- caught my attention. The notice literally performance dled without a rigorous understanding of opened my eyes to an exciting application in real product signals and systems theory. At the SPG, of signal processing: the research of human applications, real-time it is important that students are given the vision. Doing my master’s thesis project tests with typical signals opportunity to work hands-on from the in the Contact Lens and Visual Optics in realistic environments start. I am very grateful that I could do my Laboratory (CLVOL), which is part of the are essential. master’s thesis project in such an inspir- School of Optometry and Vision Science ing environment full of high-tech custom at Queensland University of Technology, equipment,” says Dr. Muma. Brisbane, Australia, was an amazing experience. The CLVOL ■ 5 Students are ready to undertake larger projects and to has a sophisticated range of unique measurement and analysis work independently techniques. These include methods to investigate the shape of ■ 5 cooperation with selected industry and research partners the cornea, the optical characteristics of the eye, visual per- provides hands-on experiences formance of contact lenses, and the biometric properties of ■ 5 outstanding projects that use real-world data can lead to the eye. After all the coursework, I really wanted to test my publications and conference visits skills in a practical environment, and I also wanted to see how ■ 5 recruitment of RAs research was organized outside the university. While I was at ■ K master’s students often stop at the height of their the CLVOL, I took all the measurements myself. This involved productivity. the synchronous measurement of the eye’s wavefront aberrations, cardiac function, blood pulse and respiration signals. We Laboratories were among the first ones to analyze the role of cardiopulmo- Central to our hands-on education, from freshmen to senior- nary signals in the dynamics of wavefront aberrations [8]. My year students, are the signal processing laboratories at TU master’s thesis project was under the supervision of Dr. Robert Darmstadt. This section presents our diverse signal processing Iskander, who is now a professor at Wroclaw University of laboratories and the opportunities they offer for our students.

1 ) t ( 0 2 Z 0 0123456 1 ) t Re( 0 0123456 1 ) t BP( 0 0123456 1 ) t

ECG( 0 0123456 Time (Seconds)

FIGURE 7. An example of a hands-on master’s thesis project undertaken in cooperation with the CLVOL at Queensland University of Technology, Brisbane, 0 Australia. The picture shows a schematic sketch of the defocus component of an eye’s wavefront aberration Zt2 ( ), the respiration signal Re(t), the blood pulse BP(t), and the ECG(t).

24 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

As detailed next, some of the laboratories are mainly educa- Those real-time tests show a variety of natural setups that tive in nature, while others additionally provide the advanced cannot be covered by data simulations. Prof. Henning Puder’s students with hands-on research opportunities and expose research group, Adaptive Systems for Speech and Audio Sig- them to state-of-the art research and advanced equipment. nal Processing, provides such a system for the development of audio processing algorithms in students’ projects. The core The SPG Lab component is a real-time DSP system, Speedgoat [10], with In our group, we offer hands-on experience to the undergradu- 12 analog audio input and eight output signals (see Figure 8). ate students in the SPG Lab, consisting of a biomedical sensor The signals are processed with low latency (<1 millisecond). lab, a basic audio signal processing lab, and a synthetic aper- Algorithms can be implemented in high-level programming ture sonar lab. It is mainly funded by the so-called resources languages such as MATLAB/Simulink. To this end, a com- for quality assurance of study and teaching (QSL)—essential- piler converts the Simulink code to C-code, which can be ly, a fund for enhancing hands-on experiences—in teaching. run natively on the Speedgoat system. In the biomedical sensor lab, a variety of sensors offer the The development of algorithms for hearing devices, opportunity to acquire own measurements, such as ECG, pho- such as hearing aids or hearing-aid glasses with a focus on toplethysmogram (PPG), and blood pressure. The data is feedback cancellation and beamforming, is one research recorded using ADInstrument devices that are originally focus of Prof. Puder’s Audio Signal Processing Group. For designed for research and teaching at universities. beamforming, the two microphones in each of the left- and ■ 5 Students can use the SPG Lab for their bachelor’s or right-worn hearing aids are combined to a four-microphone master’s thesis projects beamformer system; whereas, in hearing-aid glasses, several ■ 5 students can collect their measurements microphones can be integrated in the ear pieces. Here, even ■ 5 undergraduate students can jointly carry out research narrower beams can be realized due to larger microphone with RAs distances and a higher number of microphones. ■ 5 even patented technologies have been developed in this lab Such hearing systems need to be evaluated under realis- ■ 9 keeping the lab up-to-date and providing GUIs and help tic conditions, i.e., worn on the head and not in a free field. to students is time consuming. We use the Knowles Electronics Manikin for Acoustic Research (KEMAR) [11] as a well-established head model. Advanced real-time Audio Processing Lab The KEMAR is shown in Figure 9. This allows us to model For the development of DSP algorithms that show good and the head with respect to head shading as well as to model the robust performance in real product applications, real-time tests ear channel, which is necessary for realistic feedback tests of with typical signals in realistic environments are essential. hearing-aid devices.

FIGURE 8. The real-time Speedgoat system (left) and KEMAR (right) at the Audio Processing Lab.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 25

THE WORLD’S NEWSSTAND®

Several loudspeakers positioned at different AOA serve The Receive Beamforming Lab as sound sources. The real-time system is built in a typi- Prof. Marius Pesavento’s Communication Systems Group cal office room, showing a rather high reverberation time. offers a student experiment in the field of multiantenna Acoustic curtains, which can be opened and closed, allow receive beamforming. The experiment is based on the WARP tests of the systems within different reverberant acoustic v3 Kit by Mango Communications [12] (see Figure 12), which environments. includes an easy access MATLAB interface. The main idea of the experiment is to give students insight into the application The Bioinspired Communication Systems Lab of receive beamforming as part of a complete transmitter- The Bioinspired Communication Systems Lab of Prof. receiver chain, starting from the antennas that were designed Heinz Koeppl conducts research in statistical signal pro- at TU Darmstadt specifically for the experiment and ending cessing and machine learning in the context of biomolecu- with digital baseband signal processing algorithms imple- lar systems. Due to the availability of wet-lab facilities in mented in MATLAB. A main challenge in the design of the the group (see Figure 10), students can generate their own experiment was to find the best tradeoff between performance data by performing single-cell experiments involving fluo- and complexity on the one hand, and comprehensibility of the rescence microscopy and microfluidics, which is shown in exercise on the other. Figure 11. The microfluidic chips used for student projects In the student experiment, an antenna array is used that are further optimized and fabricated in-house. The hands- consists of eight patch antennas designed for the 2.4-GHz on work also involves the processing of raw imaging data industrial, scientific, and medical (ISM) band on the receiver side to obtain accurate segmentation and temporal tracking of (see Figure 13). The antenna array is connected to two WARP v3 single cells. boards with four RF ports per board. On the transmitter side,

FIGURE 9. A hearing-aid dummy worn on the ear model of the KEMAR. The dummy is connected via cables to the real-time system.

FIGURE 11. Single-cell recording and tracking in a swarming assay of bacteria Bacillus subtilis taken at the Bioinspired Communication Systems Lab.

FIGURE 10. The wet-lab facilities at the Bioinspired Communication Systems Lab. FIGURE 12. The WARP v3 test bed by Mango Communications.

26 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

two independent and freely movable patch antennas, also designed for the 2.4-GHz ISM band, are employed. Both patch antennas are connected to a third WARP v3 board and can be operated independently to model two independent transmitters. For simplicity, all boards are synchronized in RF and sampling frequency by external cables and the boards are connected to a MATLAB server via Ethernet, which is used for offline baseband processing. The goal of the experiment is to provide user data separation by means of receive beamforming. Two different concepts are implemented and tested by the students. In the first approach, receive beamformers are designed based on channel state information acquired from pilots; while, in the second approach, a line-of-sight model is employed to model the FIGURE 13. The antenna array consisting of eight patch antennas designed channels parametrically. For ease of implementation, a simple for the 2.4-GHz ISM band. MATLAB interface is provided that students can use to perform all the required signal processing, i.e., pulse-shaping, creativity skyrockets. When there is a successful competition sampling, timing synchronization, and channel estimation. As outcome, students gain additional benefits by having the pos- a result, students can directly focus on the sibility of visiting a conference, receiving a beamforming implementation. From our experience, prize, or gaining prestige. Next, two examples A particular challenge in carrying students who took part of successful participation in students’ com- out student experiments on the wireless petitions are given. test bed is to visualize and evaluate the in competitions show effects of different procedures and meth- not only higher technical Case study competition ods. Therefore, the experiment is divided understanding but by Rohde & Schwarz into different tasks. For example, display- also higher motivation Together with the German Association for ing the quadrature phase-shift keying and enthusiasm. Electrical, Electronic, & Information Tech- (QPSK) signal constellations before and nologies (VDE), Rohde & Schwarz organiz- after demodulation or displaying a spa- es an international case study competition tial power spectrum to estimate the user locations. During the for undergraduate students [13] in the field of mobile com- experiment, students can directly see, e.g., the effects of the munications. Its aim is to offer students the opportunity to antenna orientation on the quality of the demodulated QPSK expand their scientific knowledge and solve real-life techni- or the variation of the spatial spectrum as the user locations cal problems. It is the organizer’s intention that students not are changed. only deploy specific theoretical knowledge but also enhance In summary, during the course and in the course evalua- their teamwork and creativity skills. The first round of the tion, very good feedback was received from the students. The competition takes place at universities, where participants implementation of a complete transmitter-receiver chain helps work on a technical problem in the area of mobile commu- students to better understand and link the individual opera- nications. A jury consisting of one professor and several tions in wireless communications while the experiments help company employees decides on the best solution. The win- to visualize the effects of different operations. ning team members are then invited to the finals at the company’s headquarters in Munich. In the final competi- Competitions tion, teams from different universities compete against each The SPG seeks to participate in student competitions, as we other in finding the best solution to a complex problem. believe this is one of the best ways to provide undergraduate The winning team receives a prize as well as €2,000 for students with the opportunity to have hands-on experience their university. and to put their signal processing knowledge into practice in a In 2012, the student team, Shannon’s Hounds, of the SPG real-world project. Furthermore, students again learn to coop- took part in the case study competition whose theme was erate within a team and develop their interest in signal pro- “Engineer the future! The future of mobile communications cessing research. From our experience, the students who took is on you” (see Figure 14). In the competition in which 220 part in competitions show not only higher technical under- students from Germany and Singapore participated, students standing but also higher motivation and enthusiasm. They are had to solve complex tasks concerning the ISO/OSI-layers of inspired by their hands-on experience and their voluntary and the LTE cellular network. At the finals in Munich, modern ungraded achievements, which can also lead to better overall measurement equipment provided by Rohde & Schwarz had performance in their studies. to be used to find solutions. “We were excited to work with One important aspect to mention again is teamwork. If real modern measurement devices and have hands-on experi- the team works harmoniously and everyone enjoys their time, ence at the finals in Munich,” said Mark Ryan Balthasar from

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 27

THE WORLD’S NEWSSTAND®

FIGURE 14. Thomas Rösner (far left) and Stefan Diebenbusch (far right) from Rohde & Schwarz stand with team Shannon’s Hounds members: (from left) Lisa Hesse, Patricia Binder, Mark Balthasar, Fabio Nikolay, and Cevin Sehrt at the ceremony in Munich, 2012. (Photo courtesy of Rohde & Schwarz.)

the team representing TU Darmstadt. Shannons’s Hounds per- the heart rate using PPG signals recorded from subjects’ wrists formed outstandingly well and won the competition against during physical exercise. See [15] for more information on the ten teams from Germany and Singapore. Each team member competition. RAs Michael Muma and Tim Schäck recruited received an Apple iPad and EUR€2,000 for the university. The seven students with interest and motivation in signal process- team decided that the prize money would be spent on hands-on ing in August 2014. In total, approximately 270 undergraduate experiments for teaching purposes. Two out of the five mem- students split among 66 teams registered for the competition. bers of the team are now RAs with the SPG, and one is with the “For the next months, we arranged regular meetings where Communications Engineering Group. we discussed and developed different approaches,” remembers Tim Schäck. “We built a biomedical laboratory with our own The IEEE Signal Processing Cup PPG sensors to be able to take measurements and collect addi- The IEEE Signal Processing Cup was initiated in 2013 by tional data. For this, we employed a student with the necessary the IEEE Signal Processing Society to increase students’ hardware skills as an undergraduate research assistant whose interest in signal processing and to raise their awareness of its task was to develop a framework for the collection of measure- applications in real life [14]. Undergraduate students are pro- ments. Hence, the team members also had the option to gain vided with an opportunity to form teams and work together hands-on experience in our lab, which was much help to the to solve a challenging and interesting real-world problem students in the competition. using signal processing techniques and methods. In 2014, “After submitting our algorithm and results in February approximately 100 undergraduate students from all over the 2015, we were more than happy to find out one month later world took part in 25 different teams. The theme for the first that our team was among the best three teams and that we were competition was “Image Restoration/Super-Resolution for invited to take part in the final competition at ICASSP 2015 Single Particle Analysis.” in Brisbane, Australia. Fortunately, we managed to get five After the release of the new competition theme for 2015, students to fly to Brisbane and present the work at the finals. the SPG decided to take part in the second edition of this pres- As if this was not enough excitement for the students, their tigious competition with its student team, Signal Processing presentation convinced the jury of the new method and Signal Crew Darmstadt. The task of the competition was to estimate Processing Crew Darmstadt won the IEEE Signal Processing

28 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Cup 2015 against tough competitors from Bangladesh Univer- Interculturality sity of Engineering and Technology and Soongsil University in Special emphasis should be given to integrating students from South Korea,” says Schäck. Figure 15 shows the members of other cultures. In [17], Prof. Zoubir describes challenges in the team at the ceremony in Brisbane. The competition topic having intercultural groups in research. For example, indepen- and results have been published in [14]. dence in research has a high priority at TU Darmstadt, but “After a slow start, we managed to sit together as a team some researchers are not used to such kinds of freedom. Thus, and were able to formulate subproblems, which were assigned misunderstandings can occur among the team members. Sim- among the team members. We discussed several approaches ilarly, in engineering design projects, students often work in and often ideas that seemed right in the intercultural teams in which not everyone beginning were dismissed or modified shares the same work attitudes or values. to produce even better results,” reports Integrating students into Here, honest and direct communication is team member Maximilian Huettenrauch. social activities of the very important. “I think the SP Cup can be very helpful research group helps to in the sense that one can work on close- recruit good students. Gender equality to-real-world problems. The problem was Gender equality is always a central topic not as contrived as university tasks often for the success of our design projects. are, and the data was collected from real experiments. It also Thankfully, we are supported by gender equality representa- showed that, often, it is not the most complex and sophisti- tives who act on behalf of all female students and staff mem- cated concepts that lead to good results, but rather starting bers in the department in all matters relating to research or out with a basic idea and adding bits and pieces to this ini- teaching and also provide other services. Currently, at the tial idea,” Huettenrauch continues. Additional feedback from SPG, more than half of our RAs are female, which helps participating students and supervisors have been published attract female undergraduate students to signal processing in [15]. The main part of our prize-winning algorithm [16] hands-on projects. was published at EUSIPCO 2015 in Nice, France, by the two supervisors and one of the undergraduate students, who also Social activities continued working on heart rate estimation in his master’s Integrating students into social activities of the research group thesis project. helps to recruit good students. These activities may include events such as an end-of-year party and visits to collaboration Practical remarks for successful design projects partners in research or industry. During such visits, under- We conclude this article by briefly sharing our experience on graduate students see signal processing in action and often the some fundamental aspects that we have found to be important participants ask for hands-on topics and wish to perform for the success of signal processing design projects. research projects within our group.

FIGURE 15. Members of Signal Processing Crew Darmstadt at the final presentation at ICASSP 2015 in Brisbane, Australia.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 29

THE WORLD’S NEWSSTAND®

Mentorship Abdelhak M. Zoubir ([email protected])______From the beginning of their academic studies, our students are received his Dr.-Ing. degree from Ruhr-Universität Bochum, mentored by the professors who provide guidance throughout Germany, in 1992. He was with Queensland University of the entire duration of study and offer one-on-one meetings. Technology, Australia, from 1992 to 1998. He then joined the These meetings also offer the possibility to inform the inter- Curtin University of Technology, Australia, as a professor of ested students about the hands-on projects that we offer. telecommunications and was interim head of the School of Electrical and Computer Engineering from 2001 to 2003. Evaluation Since 2003, he has been a professor of signal processing at We constantly try to improve our courses, labs, and seminars. Technische Universität Darmstadt, Germany. He is an IEEE For this to happen, we run evaluations by means of detailed Distinguished Lecturer (class of 2010–2011), past chair of the questionnaires. Some of the questions explicitly concern Signal Processing Theory and Methods Technical Committee hands-on experiences. In this way, we receive and are able to of the IEEE Signal Processing Society, and the editor-in-chief take into account feedback from the students on how to of IEEE Signal Processing Magazine (2012–2014). His increase the number and quality of hands-on projects. research interest lies in statistical methods for signal processing applied to telecommunications, radar, sonar, car engine Acknowledgments monitoring, and biomedicine. He has published more than 400 We would like to thank Christian Steffens, Prof. Henning journal and conference papers in these areas. He is a Fellow of Puder, Prof. Heinz Koeppl, Prof. Marius Pesavento, Ann- the IEEE. Kathrin Seifert, Dr. Michael Fauß, and Mark Ryan Balthasar from the Institute of Telecommunications for their References efforts in helping us give this article a broadband overview [1] G. C. Orsak and D. M. Etter, “Collaborative SP education using the internet and of the hands-on activities in signal processing at TU Darm- MATLAB,” IEEE Signal Process. Mag., vol. 12, no. 6, pp. 23–32, 1995. stadt. The work of Dr. Muma was supported by the project [2] D. M. Etter, G. C. Orsak, and D. H. Johnson, “Distance teaming experiments in undergraduate DSP,” in Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, HANDiCAMS, which acknowledges the financial support 1996, vol. 2, pp. 1109–1112. of the Future and Emerging Technologies (FET) pro- [3] J. H. McClellan, R. W. Shafer, and M. A. Yoder, DSP First: A Multimedia gramme within the Seventh Framework Programme for Approach. Englewood Cliffs, NJ: Prentice Hall, 1997. Research of the European Commission, under FET-Open [4] J. E. Greenberg, B. Delgutte, and M. L. Gray, “Hands-on learning in biomedical signal processing,” IEEE Eng. Med. Biol. Mag., vol. 22, no. 4, pp. 71–79, 2003. grant number 323944. [5] H. T. Wu, P. C. Hsu, C. Y. Lee, H. J. Wang, and C. K. Sun, “The impact of sup- plementary hands-on practice on learning in introductory computer science course Authors for freshmen,” Comput. Educ., vol. 70, pp. 1–8, Jan. 2014. [6] J. Dagdagan, M. Muma, and A. M. Zoubir, “Robust testing for stationarity in the Tim Schäck ([email protected])______received the presence of outliers,” in Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, Dipl.-Ing. degree in electrical engineering and information 2014, pp. 3464–3468. technology from Technische Universität Darmstadt, Germany, [7] F. Strasser, M. Muma, and A. Zoubir, “Motion artifact removal in ECG signals using multi\resolution thresholding,” in Proc. 20th European Signal Processing in 2013, where he is currently working toward his Ph.D. Conf., 2012, pp. 899–903. degree in the Signal Processing Group, Institute of [8] M. Muma, D. R. Iskander, and M. J. Collins, “The role of cardiopulmonary sig- Telecommunications. His research focuses on biomedical signals in the dynamics of the eye’s wavefront aberrations,” IEEE Trans. Biomed. Eng., vol. 57, no. 2, pp. 373–383, Feb 2010. nal processing. He was the supervisor of the Technische [9] D. R. Iskander. (2016, Mar. 9). Teaching philosophy [Online]. Available: http://___ Universität Darmstadt student team that won the international ______dri.pwr.edu.pl/teaching-philosophy IEEE Signal Processing Cup 2015 with the competition topic [10] Speedgoat GmbH. (2016, Mar. 9). Speedgoat: x86/FPGA real-time hardware “Heart Rate Monitoring During Physical Exercise Using for Simulink [Online]. Available: https://www.speedgoat.ch/______[11] G.R.A.S. Sound & Vibration A/S. (2016, Mar. 9). KEMAR: Rejuvenated. Wrist-Type Photoplethysmographic Signals.” [Online]. Available: http://kemar.us Michael Muma ([email protected])______received the [12] I. Mango Communications. (2016, Mar. 9). WARP v3 Kit. [Online]. Available: Dipl.-Ing. degree in 2009 and the Ph.D. (Dr.-Ing.) degree http://mangocomm.com/products/kits/warp-v3-kit [13] Rohde & Schwarz. (2016, Mar. 10). R&S Engineering Competition 2015. (summa cum laude) in electrical engineering and information [Online]. Available: http://karriere.rohde-schwarz.de/en/career/fallstudienwettbewerb/ technology in 2014, both from Technische Universität start/__ Darmstadt, Germany, where he is currently a postdoctoral fel- [14] K. M. Lam, C. O. S. Sorzano, Z. Zhang, and P. Campisi, “Undergraduate students compete in the IEEE Signal Processing Cup: Part 1,” IEEE Signal Process. low in the Signal Processing Group. His research is on robust Mag., vol. 32, no. 4, pp. 123–125, July 2015. statistics for signal processing; biomedical signal processing; [15] Z. Zhang, “Undergraduate students compete in the IEEE Signal Processing and robust distributed detection, classification, and estimation Cup: Part 3,” IEEE Signal Process. Mag., vol. 32, no. 6, pp. 113–116, Nov. 2015. for wireless sensor networks. He was the supervisor of the [16] T. Schäck, C. Sledz, M. Muma, and A. M. Zoubir, “A new method for heart rate monitoring during physical exercise using photoplethysmographic signals,” in Technische Universität Darmstadt student team that won the Proc. 23th European Signal Process Conf., Aug. 2015, pp. 2666–2670. international IEEE Signal Processing Cup 2015. He coorga- [17] E. Keller, “Geduld und Moderationskompetenz–Fingerspitzengefühl und tradi- tionelle Kniffe wirken in multinationalen Forschungsgruppen,” Hoch3. Die Zeitung nized the 2016 Joint IEEE Signal Processing Society and der TU Darmstadt, vol. 2015, no. 1, Feb. 2015. EURASIP Summer School on Robust Signal Processing. SP

30 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

SIGNAL PROCESSING EDUCATION VIA HANDS-ONAND DESIGN PROJECTS

Qing Zhuo, Yanpin Ren, Yongheng Jiang, and Changshui Zhang

Hands-On Learning Through Racing Signal processing and engineering education through the China National Collegiate Intelligent Model Car Competition

he Intelligent Model Car Competition (IMCC) of China is an annual collegiate contest where student T teams design, build, and race a model car around a track, and the fastest car that completes the track without failure wins [1]. The IMCC is in collaboration with the global NXP Cup Challenge, which was formerly known as the Freescale Cup Challenge until the acquisition of Freescale Semiconductor Inc. by NXP Semiconductors [2]. Creating this smart, autonomous car requires students to develop the hardware and software of motor control to propel and steer their model cars. It provides a collaborative, competitive, and hands-on way for students to learn about and make a synergistic use of theories and techniques from undergraduate engineering studies, such as sensing and control, circuit design and implementation, and embedded system and software programming. The first competition, formerly known as the Smart Car Race, began in 2003 in South Korea with 80 student teams. Since then, the NXP Cup has expanded to China, India, Malay- sia, Latin America, North America, and Europe, engaging hundreds of schools and tens of thousands of students a year [2], [3]. China started its nationwide college-level smart car race in 2006. It has undergone a rapid growth since then and has just celebrated its tenth anniversary. The challenges MAIN IMAGE ©ISTOCKPHOTO.COM/SKYNESHER; SCREEN IMAGE ©ISTOCKPHOTO.COM/ALENGO and ingenuity posed by this competition has attracted an increasing number of students year by year. As shown in Figure 1, participation has risen from about 112 teams of 57 colleges in 2006 to over 2,000 teams of more than 400 colleges in recent years. For the past five years, more than 30,000 students have attended the contest annually; and so far this decade-long race has engaged more than 150,000 students in total, providing them a valuable hands-on educational experience of engineering. As members of the organizing committee of the IMCC, in this article we provide an overview and highlights of the competition tasks and rules, the role that signal processing plays, and Digital Object Identifier 10.1109/MSP.2016.2619985 Date of publication: 11 January 2017 the curricula that is built on the competition.

THE WORLD’S NEWSSTAND®

article. It has attracted an overwhelming number of students over the past 3,000 442 450 decade, and its scale is now the largest 421 in the world. What initially began as one 383

357 competition category has now expanded

2,200 to six, and the competition tasks have 2,043

2,000 287 1,913 300 been diversified. Each category has chal- 1,744 lenges that are suitable for students at a 230 different stage in their college study, so 190 1,232 that students ranging from freshmen to

Number of Teams seniors can all participate. Along with

1,000 130 150

720 the IMCC, a large number of microcon- 551

Number of Participating Colleges troller unit (MCU) teaching labs, text- 57

242 books, and innovation training centers 112 0 0 have been developed in many universi- 2006 2007 2008 2009 2010 2011 2012 2013 2014 ties. The development of such educational material and infrastructure have FIGURE 1. The number of participating teams and colleges of the IMCC from 2006 to 2014. enabled and expanded hands-on training for engineering students nationwide.

Motivation of a nationwide engineering competition Tasks and rules of the model car competition The launch of the IMCC was supported by the Ministry of Edu- All racing teams use a standard kit of model car designated cation of China and its Committee of Education Instruction of by the organizing committee. Team members are required to Automation Specialty. A main motivation to launching the design and develop their own hardware and software for their competition was that the engineering curri- cars [4], [5]. As mentioned previously, each cula at the college level were too theoretical China started its finished model car must be capable of self- and generally too slow to catch up to the fast navigating along a challenging racetrack as pace of the contemporary technological nationwide college-level fast as it can. The teams will be ranked development. As a result, students tended to smart car race in 2006. according to the time taken by the model focus more on test-oriented skills, and did It has undergone a rapid car to complete one round of the racetrack. not pay enough attention nor had enough growth since then and Only undergraduate students are permit- hands-on opportunities to solve real-world has just celebrated its ted to participate in this nationwide com- engineering problems in a team setting; they tenth anniversary. petition in China. Each team is allowed to would lack curiosity and interest and were have up to three students and no more than not sufficiently motivated to learn and inno- two faculty advisors. Typically, as shown in vate. These problems are not unique in China, as the higher edu- Figure 2, an annual competition lasts ten months as an extra- cation in engineering in many other countries around the world curricular activity, from launching in the previous November to have faced similar challenges. the division competition in July, and to the final race in August. The NXP-sponsored model car competition helps address this Early rounds of the competition are carried out in eight problem and bring hands-on engineering education to many col- racing divisions covering different geographic areas in China. lege campuses around the world. The IMCC in China has sev- The top teams from each race division are qualified for the eral notable characteristics, including the competition setup, the final race. During the final race, speed-based race sessions are rules, and the evaluation criteria that will be discussed later in this held in which the time for each finalist car to complete one

Team Enrollment Eight Sub Division Contest

Prev. November 1st Late March November- July of Previous August Year Early July

Launch the Car Development National Finals Race Rules

FIGURE 2. The time line of the annual IMCC.

32 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

round of the racetrack is used to rank the teams; in addition, an acquired by such sensors are typically binary or one-dimen- open-ended competition is held concurrently, with innovation sional. Thus, the algorithms for signal processing and control themes related to future industrial intelligent cars to encourage decisions can be relatively simple compared to the other cate- students to develop creative ideas and implementations. gories. This category is suitable for younger students with primary engineering knowledge. Basic elements of the racetrack With the exception of a beacon-based sensing category to Camera sensing be discussed later in this section, the racetracks are com- Planar-array CCD or complementary metal–oxide–semi- posed of several kinds of elements: straight sections, curved conductor (CMOS) cameras are used to pilot the car in this sections, crossroads, hills, and roadblocks category. A model car equipped with a (see Figure 3). The characteristics of the camera is shown in Figure 6. Two-dimen- racetrack and its elements are given in the What initially began as sional image acquisition and processing rules released at the launch of the compe- one competition category requires more computing capabilities tition. A detailed graph of the racetrack has now expanded to six, than for the other sensing categories. Stu- is revealed to the teams onsite right and the competition tasks dent teams in this category often equip before the start of the competition. The have been diversified. their model cars with a high-perfor- sensing and control algorithms embed- mance 32-bit MCU with larger random ded in the race cars are expected to work access memory (RAM) storage and high- with all these elements and different combinations of them. er million instructions/second in computing power. Using image processing and computer vision algorithms, it Competition categories is possible for a model car to deal with a more complex track To enable self-navigation of model cars, different kinds of layout, predict the direction of the road, and plan for its motion sensors are explored to capture position signals for further on the racetrack. As a camera can capture images farther ahead processing. Based on the sensor types and race tasks, the of the racetrack, camera cars are usually the fastest among all competition is divided into several categories that have dif- of the cars in the competition categories. ferent levels of technical challenge, as illustrated in Figure 4. The basic categories only require some elementa- ry knowledge of signal processing, control, and circuits, thus allowing younger undergraduate students to participate; on the other hand, the advanced and creative catego- Level of Challenge ries may use the technical knowledge and skills from High Chasing students’ design training or capstone projects. The wide Two Wheel Beacon Energy Saving variety of categories gives students an opportunity to par- Mediu ticipate in several competitions during their college career, Camera Photoelectric as they grow in knowledge, experience, and maturity. In Synergistic Lo what follows, we briefly review the characteristics of each Metal Engineering Skills competition category. Racetrack Required Basic Advanced Innovative Photoelectric sensing In this competition category, a model car can be equipped with photoelectric sensors, such as an infrared (IR) light- FIGURE 4. The different categories in the competition. emitting diode sensor and linear charge-coupled device (CCD) sensors, to detect the racetrack. Figure 5 shows a model car equipped with photoelectric sensors. The signals MCU Board

Steering Servo

IR LED Sensor

FIGURE 3. The different elements in a racetrack. FIGURE 5. A model car with photoelectric sensors.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 33

THE WORLD’S NEWSSTAND®

also be located low to deliberately lower the center of gravity of Camera the car. As shown in Figure 7, these arrangements can increase Sensor the stability and the racing speed of the car. Compared to the other competition categories, the signal Car Chasis processing and control methods applied in this category are generally more sophisticated for the race car to maintain its Road Edge balance while moving smoothly on the racetrack. The Kal- man filter algorithm is often used by participating teams to Race Track calculate the angle of the car position, and a double closed-loop speed control is implemented to drive the wheels.

Metal racetrack For the metal racetrack competition category, the racetrack is FIGURE 6. A model car with a CMOS camera. laid with two parallel strips of aluminum foil, and a direct current (dc) power of 12 V 5 A is applied to the two alumi- Two-wheel car num strips. A model car participating in this competition cat- There are two specific characteristics and challenges in this egory is required to guide itself by detecting the metal foils, competition category. First, the model car only has a total of and is also allowed to pick up electricity through the alumi- two wheels, left and right. To propel the car to move upright, num foils to drive its motor. The racetrack components are sensors such as the gyroscope and accelerometer need to be shown in Figure 8. The main competition goal of this catego- employed, and the signals from these sensors must be ry is to design a highly power-efficient intelligent model car. acquired and processed properly in real time. Second, instead The judging criteria is not only by the speed of the car, but of putting visible black edge lines on the racetrack as in the also by the total energy consumed during the competition. other competition categories, one enameled wire with a diam- To sense the guiding metal strips, several coils may be eter of 0.5 mm is laid along the center of the racetrack to mounted in front of the race car, and a high-frequency ac signal guide the movement of the model car. Alternating current (ac) from an oscillator circuit may be applied on the coils. The alter- (100 mA, 20 kHz) flows along the wire, which generates an nating electromagnetic field generated by the coils will induce oscillating magnetic field on the racetrack. With such a race- the eddy current on the surface of the metal strips. In turn, the track design, one way to guide the model car is to use two eddy current will change the amplitude and frequency of the inductor coils to sense the varying magnetic field. ac in the coils. The specific variation depends on the related To sense the racetrack, detection coils can be mounted to distance between the detecting coils and the metal strips. By the car on a well-designed extending bracket. The battery can using the amplitude demodulation or frequency demodulation, the car can detect its deviation from the guiding strips. Such sensing and signal processing methods may be implemented by circuit or by software. Sensor Control Board Battery Mounting Bracket Two-car chasing Sensors used in this competition category are similar to the one for camera sensing. Here, each team is required to design two model cars to run one after another on the racetrack. Figure 9 shows two students preparing their chasing cars on Two-Wheel Chassis the racetrack. The final score (T) contains two portions: the total time FIGURE 7. A two-wheel car with a long sensor bracket. (T1) for the two cars to complete a round, and the lag (T2) between them as they cross the finish- ing line. The formula for calculating the total score is TT=+15 T2. Track Side To perform well in this competition, 50 mm the two race cars should coordinate Gap 5 mm well by wireless communication. They must avoid colliding with each other 50 mm 50 mm while at the same time avoid being too dc Aluminum Power Adhesive far apart. This is the most challenging Track Side Tape among all the competition categories. Detecting the distance between the FIGURE 8. A racetrack option with a metal strip. two cars is crucial in this competition.

34 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Usually, an ultrasonic signal and an IR signal are sent back simultaneously from the leading car. The second car can determine the distance by detecting the time-lag between the receiving ultrasonic and IR signals.

Beacon-based racing Unlike the other competition categories, there is no visible racetrack in this competition. Several beacons are distributed on the competition field and can be turned on by a referee system in a random sequence. Model cars should move to approach the lighting beacons. As long as a car moves inside the detection region of a beacon, the referee system will turn on the light of the next beacon. The arrangement of the beacon group field is illustrated in Figure 10. The challenges of this competition include detecting targets that may be located relatively farther away from the race car, FIGURE 9. avoiding collision onto the beacon during the navigation, and Students preparing two chasing cars on the racetrack. planning a motion path for the race car. Many participating teams have used a camera to search for the beacon. To differentiate from the surrounding light sources, the beacon flashes at about 10 Hz. This flash pattern is used Beacon B3 to locate the beacon in the middle of ambient light. Participants B4 have shown that target detection based on the frame difference Field Border image is a robust approach to locate the beacon in most types of environments. Judgement Computer B2 dc B1 Forward-looking innovation category B5 To foster creative thinking, the IMCC also has an open-ended Competition Field category inviting teams to contribute innovative ideas and Bus Interface designs. Different themes are set up for each year. The most Module recent competition’s theme, for instance, was innovative FIGURE 10. designs on energy saving and a future smart city. Teams are The beacon group competition field. required to submit a detailed technical report as well as a video of their work. The top teams are invited to the final race each August to showcase their work. Figure 11 shows a winning entry of a recent competition, where the student team designed and prototyped a parking facility in a futuristic community.

Signal processing techniques used in the competition There are many different technologies used in a model car to compete in the respective category [6], and signal processing is one of the key components. Students receive hands-on training and strengthen their understanding of signal processing through the competition.

Signal sensing and sampling As the model car is controlled by the MCU, almost all sig- FIGURE 11. An exhibit of the creative design category of the IMCC. nals coming from the sensors would be sampled and convert- ed into digital. Some of these signals are already digitized by the sensor module themselves so that they can be passed As students learned from fundamental signal processing directly from the sensor to the data transfer port of the MCU. courses, the sampling frequency need to be properly deter- Some other sensing signals are obtained in an analog form, mined according to the property of the signal. For example, for example, the photoelectric sensor signal and induction the signal that comes from the induction coils in the two-wheel coils signals. These signals need to be sampled after condi- competition has an alternate voltage with a fixed frequency of tioning circuits by an analog-to-digital converter (ADC) 20 kHz. It is a narrow-band signal, and thus with proper pro- module on the MCU. cessing, it could be sampled at a frequency that is far lower

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 35

THE WORLD’S NEWSSTAND®

analog filters to condition the sensing signals, and digital filters implemented by embedded software for linear, non- 16.5 16.5 11.5 10 10 10 10 10 10 10 11.5 16.5 16.5 linear, or adaptive signal processing. The main source of interference on board comes from the motor driver circuit with high peak currents up to 20 A. The noise sparks travel through FIGURE 12. The layout of the photoelectric sensor placement. the power supply line into the control circuitry, causing erratic behaviors. Most of the noise can be reduced by than the nominal Nyquist frequency. Based on these sampled using analog filters in the sensing and power circuits. However, data, the amplitude of the signals can be determined. some noise is still left in the sensing signal and is sampled into Another example of sampling is to detect the center of the MCU. This remaining noise can be dealt with by digital the racetrack. In the photoelectric sensing competition, the signal processing. sensors sample the light intensity of the racetrack surface Other noise sources may also be seen in each specific com- spatially at several isolate spots. As the petition setup. For example, we previously number of the sensors is confined to no Two-car chasing is mentioned that in the two-wheel competi- more than 16 by the competition rules, the the most challenging tion, a model car must keep an upright pose arrangement of the sensors must be well among all the competition while running along the racetrack. An iner- designed to increase sampling range and tial measurement unit (IMU) is mounted on accuracy. Figure 12 shows a common lay- categories. the chassis to measure the inclination of the out of photoelectric sensors in the competi- model car. The IMU outputs two signals: tion, whereby the sensors are arranged in one line in front of the gyroscope signal that gives the angular speed, and the the car with different spacing between them, and the space in accelerometer signal that provides information on the “down” the middle is smaller than the outer ones. This kind of non- direction. Although the dip angle of the car can be calculated uniform sampling was shown to perform better than the uni- according to the accelerometer signal alone, the movement of form spatial sampling in the competition. the car produces much noise mixed with the angular position. To address this problem, student teams have employed Kal- Denoising and parameter estimation man filters when computing the angular position (shown in An important task of the competition is to take the sensing Figure 13), combining the gyroscope and accelerometer sig- signals that are often noisy or distorted and process them to nals in a denoised fashion. Some student teams have simplified extract important parameters or other useful information and the filtering algorithm to allow for more efficient estimation of pass it on to the control and decision mechanism to race the the attitude angle of a two-wheel race car. In addition, one of model car. the teams in the two-wheel competition carefully studied the Student teams have employed a variety of filters in the noise levels in the IMU data when the pulse-width modulation model car control systems to deal with the interference coming (PWM) voltage with different duty ratio is applied to drive the from external sources or onboard circuits. They have used both motors. They incorporated this into a Kalman filter implementation to further reduce the noise when estimating the pose of the model car.

Image processing and computer vision Thanks to the power of visual sensing, student teams prefer using cameras to guide their model cars in the competition categories of camera sensing, chasing, and beacon-based rac- 1 ing. By using a camera to continuously capture images of the racetrack, a wide range of information about the car’s current 2 state and potential future situation can be inferred. Detection range is a key factor affecting the achievable speed of a self-navigating race car. Because there exists a 3 constant delay when a servo changes the direction of the front wheels of a race car, detection in advance can compensate this delay. The sooner the detection of the curve on the racetrack FIGURE 13. Using a Kalman filter to determine the angular position of a car. is, the higher speed the race car can run without going out of Curve 1: accelerometer signal; curve 2: gyroscope signal; and curve 3: the the bounds of the racetrack. Image processing from camera Kalman filter output signal. data also enables the extraction of more detailed information

36 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

of the racetrack. For example, detecting the obstacles on the Curricula development based on the competition racetrack, including hills and roadblocks, is generally easier An important goal of the IMCC is to integrate the learning of and more accurate by a camera than by other types of sensors. multidisciplinary knowledge and the training of comprehensive Despite the aforementioned advantages, abilities important to engineering profession- there are challenges in performing image als through the process of building and racing processing on a low-cost embedded plat- Detection range is a model cars. As mentioned earlier, the respec- form. The main limitation comes from the key factor affecting the tive expertise involved include analog/digital shortage of RAM memory capacity and the achievable speed of a electronics, embedded system, electronic limited computing power of the MCU. To self-navigating race car. design automation, control engineering, signal strike a good tradeoff between the process- processing, pattern recognition, and more. ing speed and accuracy, it is desirable for Figure 16 illustrates the overall process for the MCU to subsample the image into a lower resolution to students to attend the competition, together with the respective work with, and such low-resolution low-quality image poses a disciplines supporting the different tasks in the competition. challenge for student teams to reliably detect the racetrack and other elements on the track. Figure 14 shows an example scene of a model car on the racetrack. Inset (a) is the binary image captured in the MCU’s RAM. Inset (b) shows the center line of the racetrack extracted from the image above by the image processing algorithm. The image captured by the camera on board of a race car also has a large distortion caused by the low viewing angle of the camera. To obtain more accurate information of the road, a perspective transformation is applied upon the image to correct such distortion. Figure 15 shows the effect of perspective transformation. Figure 15(a) reveals the original grayscale image; (b) shows the center line of the racetrack extracted from the gray image; and (c) gives the correct center line after the perspective transformation. In summary, signal processing techniques are applied extensively on various aspects of the race cars. This compe- (a) (b) tition allows students to practice their knowledge of signal processing in task-oriented training scenarios, which com- FIGURE 14. A model car with a camera mounted on the top of the pole in plements the traditional classroom learning to better prepare the middle of the car. The inset (a) shows the binary image captured in students for future real-world R&D. the RAM; (b) shows the center line of the racetrack.

2,500 250

2,000 200

1,500 150

1,000 100

50 500

0 0 0 100 200 300 –1,000 –5000 500 1,000 (a) (b) (c)

FIGURE 15. (a) The origin image of the S-shape curved racetrack. (b) The center line of the racetrack. (c) The center line after the perspective transformation.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 37

THE WORLD’S NEWSSTAND®

After a decade of annual intelligent car competitions in independently, make design decisions, and implement China, a total of 176 practical training centers have been the designs. When students encounter problems in hard- set up in 132 colleges across the country, and 115 courses ware or software design and testing, mentors can guide have been developed or redesigned in Chinese universities. them to troubleshoot the problems. They can also advise More than 36 textbooks have been students to evaluate the possible out- published with different focuses and More than 150,000 comes of a design decision. With help scopes, providing guidance on the core students have from these mentors, students can learn knowledge and skills to students who are attended the competition how to balance analysis and experi- interested in participating in the race car mentation, which helps enhance their competition, or simply for a fun hands- and flourished because theoretical understanding and practical on extracurricular practice. In addition, of it. execution capabilities. Also, mentors a large number of technical reports on usually discuss and summarize with the successful approaches in the previous competitions their students the experiences and lessons learned after have been archived, from which new students can gain each competition. insights on the essential functional blocks and the associated circuitries. Closing remarks Furthermore, an excellent group of faculty members The IMCC in China has received many awards and recog- and instructors nationwide have served as mentors to nitions in recent years. By providing college students and guide student teams. They provide guidance and technical programs with an engaging way to learn/teach, this annual resources to students, while leaving students room to think competition has built a strong reputation and fostered

Electric, Computer, Mechanics Electrical Instruments Control

Signal Processing Chasis Sensor Interface: Pattern Recognition Adjusting, Motor/Servo Photocell Camera, Intelligent Control Steer Driver Gyro Optical Software Engineering Linkage Encoder Communication

Cross-Field Multidisciplinary

Movement Path Track Adaptive Quality Planning Recognition Control

Conceiving Designing Operating Implementing

System Preliminary Make Make Design Contest Plan Hardware Assembly Enrollment July Technical Theory Exchange Analysis Purchase Develop Finals Devices Software Debug August Simulation

FIGURE 16. An illustration of multidisciplinary involvement and synergy for the IMCC race car design and implementation.

38 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

interactions and collaborations with The IMCC helps students Automation at Tsinghua since 1997. His many organizations. The industrial spon- strengthen communication research interests include signal processing sor, Freescale Semiconductor, which and teamwork skills and and pattern recognition, embedded computer merged with NXP, has enthusiastically systems, and power electronics. His educa- supported the competition since 2006. inspires them to pursue tional awards and recognitions received from More than 150,000 students have attend- engineering careers and Tsinghua University include Ten Most ed the competition and flourished be- become future innovators. Favorite Teachers (2010) and Outstanding cause of it. An overwhelming amount of Mentor (2012). He has been a key member positive feedback from student participants has been on the Intelligent Model Car Competition Organizing Committee received. The IMCC has provided students with an oppor- for ten years. tunity to learn multiple engineering disciplines, most nota- Yanpin Ren ([email protected])______received her B.S. bly electrical and computer engineering and mechanical and Ph.D. degrees in control theory and control engineering engineering, and put them into a synergistic use. The from Tsinghua University, Beijing, China, in 1995 and 1999, IMCC helps students strengthen communication and team- respectively. She worked at Lenovo from 1999 to 2006 with work skills and inspires them to pursue engineering careers an R&D focus on smart homes. She is currently a senior and become future innovators. engineer in the Department of Automation at Tsinghua The competition has successfully met the educational University. She is heavily involved in the design and teaching goals originally set by the administrators and sponsors. To of electronics laboratory courses. Her research interests ensure its continued success, we are working on addressing include digital and analog circuit design, embedded systems, several newly emerging issues. For example, several competi- and virtual instruments. tion categories that use optic sensors to detect the road have Yongheng Jiang ([email protected])______received his seen a nontrivial amount of requirements for the competition B.S. and Ph.D. degrees in automation from Tsinghua University, conditions for the model cars to work well, and this added Beijing, China, in 1998 and 2003, respectively, where he is cur- an extra burden to the onsite event organization. Also, as the rently a faculty member in the Department of Automation. His number of teams increases significantly, the cost of organiz- research interests include process scheduling, trajectory plan- ing the race also increases. To overcome these difficulties, ning, and optimization technology. the competition tasks and rules need to be revised periodi- Changshui Zhang ([email protected])______received cally. Another new trend with the help of the Internet is the his B.S. degree in mathematics from Peking University, possibility to build some standard competition platforms, Beijing, China, in 1986, and his M.S. and Ph.D. degrees in which allow students to download their software into their control science and engineering from Tsinghua University, model car remotely, facilitating their participation of the pri- Beijing, in 1989 and 1992, respectively. He joined the mary competition. faculty of the Department of Automation, Tsinghua The contents and formats of the IMCC should adapt with University, in 1992, where he is currently a professor and new technological advancement. The state-of-the-art tech- the director of the Institute of Information Processing. He nologies and the advanced educational concepts rejuvenate the has authored more than 200 papers. His research interests competition, supporting engineering students to develop their include pattern recognition and machine learning. He is an interests and embark on their professional careers. associate editor of the journal Pattern Recognition and served as the associate chair of education in his department Acknowledgments and university. We would like to thank our colleagues on the IMCC committee, including Jingchun Wang, Kaisheng Huang, and Ming References [1] Q. Zhuo, J. Wang, K. Huang, Y. Jiang, M. Zeng, G. Shen, and X. Wang, “A ten Zeng, for their collegial efforts during the organization of the years review of the national student intelligent car race,” in Collected Works of competition and the suggestions in conceiving the competi- Decade Development of Collegiate Intelligent Car Racing in China. China Univ. tion setup and rules. This work was supported by the National Teaching, 2012 (in Chinese). [2] NXP/Freescale Cup model car race overview [Online]. Available: https://___ Experimental Teaching Demonstration Center Program, community.nxp.com/docs/DOC-1011 National Special Fund for Improving Basic Education Condi- [3] The 2016 NXP Cup rules used for competitions in Americas, Malaysia, and tions and NSFC the National Science Foundation of China Taiwan [Online]. Available: https://community.freescale.com/docs/DOC-1284 (NSFC grant 91420203). [4] G. Z Shen. (2015). Illustration of complex engineering problem: Intelligent mode car competition [Online]. Available: http://www.smartcar.au.tsinghua.edu.cn/ column/pxxz______Authors [5] Competition rules of the 11th Intelligent Car Competition in China, 2015 [Online]. Available: http://www.smartcar.au.tsinghua.edu.cn/ Qing Zhuo ([email protected])______received his B.S. [6] J. Sun, W. Huang, and S. Niu. (2009). Technical report documents of the teams and Ph.D. degrees in control theory and control engineering from participated on the final competition in China [Online]. Available: http://www Tsinghua University, Beijing, China, in 1993 and 1997, respec- ______.smartcar.au.tsinghua.edu.cn/ tively. He has been serving on the faculty of the Department of SP

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 39

THE WORLD’S NEWSSTAND®

SIGNAL PROCESSING EDUCATION VIA HANDS-ON AND DESIGN PROJECTS

Erik G. Larsson, Danyo Danev, Mikael Olofsson, and Simon Sörman

Teaching the Principles of Massive MIMO Exploring reciprocity-based multiuser MIMO beamforming using acoustic waves

assive multiple-input, multiple-output (MIMO) is currently the most compelling wireless physical layer tech- Mnology and a key component of fifth-generation (5G) systems. The understanding of its core principles has emerged during the last five years, and material is becoming available that is rigorously refined to focus on timeless fundamentals [1], facilitating the instruction of the topic to both master- and doctoral-level students [2]. Meaningful laboratory work that exposes the operational principles of massive MIMO is more difficult to accomplish. At Linköping University, Swe- den, this was achieved through a project course, based on the conceive-design-implement-operate (CDIO) concept [3], and through the creation of a specially designed experimental setup using acoustic signals. The course was developed with the following three objectives in mind: ■ Exposure of students to emerging concepts and to the technology of the future, not of the past. This target was inherently met via the focus on massive MIMO. ■ Promotion of a systems view of thinking, requiring the synthesis of knowledge acquired through the classical curriculum. This goal was achieved through the development of a unique acoustic, reciprocity-based massive MIMO laboratory setup (Figure 1), with students taking the lead both in its development and evaluation and, subsequently, its refinement. MAIN IMAGE ©ISTOCKPHOTO.COM/SKYNESHER; SCREEN IMAGE ©ISTOCKPHOTO.COM/ALENGO ■ Fosterage of genuine teamwork, preparing students for a dynamic work environment. This was efficiently facilitated through the adoption of the CDIO project concept, which integrates technical work with the instruction of project management and entrepreneurial skills.

Massive MIMO: The scalable 5G wireless access technology Massive MIMO exploits the use of large antenna arrays at wireless base stations to simultaneously serve a large number of autonomous terminals through spatial multiplexing. The multiplexing takes the form of beamforming, also known as Digital Object Identifier 10.1109/MSP.2016.2618914 Date of publication: 11 January 2017 multiuser precoding, effectively creating transmitted signals

THE WORLD’S NEWSSTAND®

that add up constructively on the spots where the terminals are terminal is deterministic and frequency- (subcarrier-) inde- located and destructively almost everywhere else (Figure 2). pendent, which greatly simplifies resource allocation As the ultimately most useful and scalable form of multiuser problems, and facilitates simple closed-form solutions for MIMO, massive MIMO departs from conventional MIMO power control. technology in several ways: ■ Different base stations do not cooperate, other than for ■ The acquisition of accurate instantaneous channel state “slow” tasks such as power control and pilot sequence information (CSI) at the base station is facilitated through assignment. Macrodiversity against shadow fading is time-division duplexing operation and the transmission of accomplished by appropriate terminal-to-base station asso- pilot waveforms by the terminals. On uplink, terminals ciation; the large number of spatial degrees of freedom transmit pilots and payload; subsequently on downlink, guarantees that every base station has room to accommo- the base station beamforms to the terminals. Reciprocity date additional terminals with very high probability. of uplink-downlink propagation is essential to the use ■ By virtue of the large array gain and the ability to null out of uplink CSI for downlink precoding, and it is achieved interference, max-min fairness power control is feasible and in practice through calibration of the can be exploited to yield uniform quality of radio-frequency (RF) chains [4]. The Massive multiple-input, service within each cell. reliance on reciprocity permits accurate multiple-output (MIMO) channel training even in highway-speed Communications laboratory exercises: mobility scenarios. is currently the most RF versus acoustic ■ Extraordinary spectral efficiencies are compelling wireless Laboratory work that uses communications achieved [5], although no attempt is made physical layer technology, over RF in general requires substantial to operate at the Shannon limit. The base and a key component equipment investments: the availability of a station uses linear signal processing, and of fifth-generation sufficiently interference-free environment the terminal uses almost no signal pro- (5G) systems. (alternatively spectrum licenses for experi- cessing at all. CSI is acquired only at the mental operation); the use of high sampling base station, obviating the need for rates, which, in turn, generates large quanti- downlink pilots. This renders massive MIMO entirely scal- ties of baseband data; and the access to high-performance able with respect to the number of base station antennas. measurement equipment for calibration and debugging pur- ■ Each terminal is assigned the full bandwidth. Thanks poses. Owing to precise requirements on the phase reference to channel hardening, the effective channel gain for each distribution, these difficulties are significantly accentuated

FIGURE 1. The massive MIMO laboratory setup at Linköping University. (Photo courtesy of Mikael Olofsson.)

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 41

THE WORLD’S NEWSSTAND®

when going from single-antenna to multiple-antenna setups. receive directions, is referred to as a transceiver device in Consequently, experimental activities on massive MIMO are the following. concentrated to research-grade test beds that require the ■ Fourteen transceiver devices were used to form a base investment of many person-years to build, and permanent station array and two were used to form two independent engineering staff to maintain—thus inherently unsuitable for terminals. All devices were connected to a personal com- instructional purposes. Worldwide, only few such test beds, puter equipped with multichannel digital-to-analog (D/A) operational and under construction, are known [5]–[10]. and analog-to-digital (A/D) cards. At a 6.25-kHz sam- The basic physics of wave propagation is substantially pling rate per channel, the aggregate baseband data rate the same for electromagnetic wave propagation and for from all channels combined equaled 1,200 kilobits/second, acoustic wave propagation, disregard- facilitating real-time processing in a ing polarization aspects. Consequently, MATLAB environment. wireless communications course labora- Spoofing is accomplished For further technical details, see tory work may use sound waves instead using fake biometric “Detailed Description of the Transceiver of RF, which for point-to-point commu- samples expressly Device Design.” A live demonstration may nications requires only a loudspeaker synthesized or be found at [11]. as a transmitter and a microphone as a receiver. The wavelength of sound in the manipulated to Promoting a systems view of thinking audible regime is comparable to the wave- provoke artificially high University education traditionally is rather length of radio in the gigahertz regime; comparator scores. modularized, consisting of courses with a hence, the channel coherence time for well-defined but often very specific scope. acoustic communication indoors is suf- In contrast, practicing engineers must be ficiently long to easily permit real-time experimentation able to solve problems by synthesis of knowledge received over a time-invariant channel. Additionally, many phe- both in formal training and courses, and acquired from experi- nomena, such as small-scale fading, can be observed with ence. The traditional curriculum in communications is no an acoustic setup as well. The low bandwidth of audible exception: while much time in class is spent on proving capaci- sound results in low sampling rates, modest requirements on ty theorems and computing error probabilities, most laboratory clock synchronization, and manageable amounts of base- time (and most engineering efforts) tends to be spent on “mak- band data. These features are routinely exploited in many ing it work”: solving practical problems, and integrating com- university course laboratory exercises. ponents together into systems. Formal training and knowledge of textbook material is indispensable, but real problem solving Teaching MIMO and massive MIMO principles requires both trial-and-error, and the sourcing of information in the lab from books, colleagues, and the Internet. Laboratory work on point-to-point wireless communications The experimental setup and its development trained students is straightforward. In contrast, serious experimentation with in every aspect of communications, ranging from the under- massive MIMO concepts is a less obvious task, owing to the standing of wave propagation, fundamental massive MIMO need for many simultaneously operating transceivers and theory, hands-on design and soldering of electronics, software the uplink-downlink reciprocity requirement. The experi- programming, and algorithm design. This substantially took mental setup developed in our course addressed this diffi- the students’ understanding of a “system” from the input-output culty as follows: box typically taught in signals-and-systems courses, to a “sys- ■ A loudspeaker element natively functions as a sound trans- tem” meaning a large set of connected, diverse components. mitter when fed with an amplified input signal. It was Answers were not given to the students but had to be sought experimentally revealed that oppositely, it also can func- through consultation with faculty and senior graduate students. tion well as a receiver (microphone), by connecting it to a In fact, as the project was previously untested, in many cases high-input-impedance amplifier and measuring the result- instructors did not know the answers to all of the questions. A ing signal. minor risk was taken that the course would not work out, in ■ A set of off-the-shelf loudspeaker elements was acquired which case a backup plan was to only perform certain measure- and redesigned as follows: ments, so that reporting of a minimum level of requirements for 1) the built-in amplifier chain was retained, yielding the the course could be adequately completed. transmit branch. 2) an amplifier was installed to use the loudspeaker ele- Providing true teamwork experience ment as microphone, yielding the receive branch. through a CDIO project 3) a relay that galvanically isolates the transmit and receive The CDIO concept [3] was developed by the Massachusetts branches was installed. Institute of Technology in the United States and Chalmers The required additional electronics was fitted inside of the Institute of Technology, Linköping University, and Royal loudspeaker element casing (Figure 1). The rebuilt loudspeaker Institute of Technology, all in Sweden, in the late 1990s and element, now functioning reciprocally in both the transmit and the early 2000s. The initial efforts were supported by the

42 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Knut and Alice Wallenberg Foundation. The CDIO framework As one of the founders of the CDIO Since then more than 120 schools and uni- focuses on the teaching of concept, Linköping University has adopted versities worldwide have joined the CDIO engineering fundamentals and implemented the idea from its incep- initiative. One of the reasons for the prolif- tion. All engineering master’s programs eration and success of the concept is that it from the perspective include a mandatory CDIO course in the profoundly facilitates teamwork training of real-world systems curriculum. In collaboration with several for students. and products. global companies that have local offices in The CDIO framework focuses on the Linköping, notably Ericsson and SAAB, teaching of engineering fundamentals from a specific project model, Linköping Inter- the perspective of real-world systems and products. Through active Project Steering (LIPS) [12], has been developed to constant input from students, faculty, and engineers, the con- support the project management process. The LIPS model cept has evolved over the years. It currently represents the introduces a set of documents that the students are required state of the art in project course organization in engineering to write during the project, as well as a set of milestones and schools, preparing the students for the challenges of profes- tollgates that underpin the planning of the project work. In sional life in industry. the beginning of the course, students usually express doubts

Frequency

Channel Coherence Interval

UL DataUL Pilots DL Data

Time

FIGURE 2. Massive MIMO.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 43

THE WORLD’S NEWSSTAND®

Detailed Description of the Transceiver Device Design

The setup consists of a general-purpose computer, digital-to- form a differential-in and differential-out amplifier with a lin- analog (D/A) and analog-to-digital (A/D) converter cards, ear voltage gain of 200 (46 dB). modified active loudspeakers, and signal distribution hard- The signal is then sent via a cable to a distribution box ware. An overview of this setup is given in Figure S1. where signals from all speaker pairs are further amplified Sixteen channels are used from each of the A/D- and D/A- before they are sent to the A/D-converter card. The signals cards (DA12-16 [S1], [S2] and AD12-64 [S3], [S4] from travel differentially from the speakers to the distribution box Contec), one A/D- and D/A-channel per speaker. The to counteract capacitively coupled noise. The last three speakers are used both as speakers and as microphones. OP-amps in the detection signal chain were placed on the We used small active off-the-shelf speakers (Maxxtro collection board in this distribution box together with the cor- MX-US-08 [S5]), where the original power amplifiers are responding circuits for the other seven speaker pairs. The first used as such. two of those OP-amps are used as voltage followers, and The speakers come in pairs, where one serves as the mas- their main task is to remove inductively coupled noise by ter and contains all the electronics, while the other serves as breaking the closed circuit that otherwise would have the slave and contains only a speaker element. The master formed. They are also accompanied by Resistor-Capacitor- speaker was modified to facilitate the selection between two links that form first-order high-pass filters to remove any offset modes: either both the master and the slave operate as loud- voltage introduced by the OP-amps in the speaker. These are speakers, or they operate as microphones. This modification the two extra OP-amps compared to a standard instrumenta- is illustrated in Figure S2. On the detection board, relays are tion amplifier. The fifth and final OP-amp in this signal chain used to switch between two operation modes: 1) using the is a differential-in single-ended-out amplifier that provides an speaker elements of a loudspeaker pair as transmitters, by additional linear voltage gain of 10 (20 dB). connecting them to the outputs of the power amplifiers, or 2) This setup poses special demands on the OP-amps of the using the speaker elements as microphones, by connecting detection board in the master speaker box. We amplify weak them to the inputs of the receive chain. This way, the hard- signals, which means that we need low-noise amplifiers on ware supports half-duplex communication. the detection board. Also, the fairly large initial voltage gain The receive chain for a single speaker is essentially a tradi- (200) demands that the input offset voltage of these first tional instrumentation amplifier that usually is built around OP-amps is low. Finally, these OP-amps must be able to oper- three OP-amps and seven resistors to determine the voltage ate at supply voltage 5 V, since that is what is available in gain, as illustrated in Figure S3. In our implementation the master speaker box. We use MCP6024 I/P [S6], which (Figure S2), we use five OP-amps. The first two OP-amps is a quad low-noise OP-amp that accepts supply voltage were placed on the circuit board in the master speaker and from 2.5 V to 5.5 V and delivers full output swing. Its input offset voltage is in the range ±500 μV. With voltage gain 200, this limits the output offset voltage to be in the range Computer Master Slave ±100 mV. The gain-bandwidth product of this OP-amp is Speakers Speakers 10 MHz, so with voltage gain 200, we end up with a bandwidth of 50 kHz. D/A Card The sampling frequency is determined by the A/D- and D/A-cards. They operate at a maximum of 100 ksam- ples/second, which is distributed between the used chan- A/D nels. Thus, with 16 channels, the sampling frequency Distribution Box Distribution Card used is 6,250 samples/second. The bandwidth 50 kHz

Eight Speaker Pairs Eight Speaker of the analog part of the detection signal chain is therefore well sufficient. While our experience from the particular choice of FIGURE S1. An overview of the setup. hardware has been positive, future generations of the

about the necessity of the documentation and the extent of the get to know each other and decide on the different roles that planning required, but most of them usually appreciate the they will have in the project group. In addition to the crucial LIPS model once the course is finished. role of project manager, the group members usually assign a The project team normally consists of four to seven stu- person responsible for documentation, quality, testing, design, dents. Generally, these students have not previously worked and customer contact, respectively. The project task is deter- together, and an immediate task at the start of the project is to mined in detail by the customer through the definition of a

44 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

D/A Distribution Box Master Speaker Box Slave Card Speaker Connection Board Original Amplifier Board (Details Omitted) Box 2 ch + + A1 A1 – – GND

A/D Collection Board Detection C C 0V Card Board + R3 R2 + 5V A2 + R R + – R3 1 1 – A3 A3 A C R – R – 2 + 3 3 + A2 R1 R3 R1 R3 – R3 2 ch R3 R2 R3 – R3 – A3 A ++3 Similarily to Above

GND

Digital Relay Control Board (Details Omitted) Out

FIGURE S2. A schematic of the hardware, illustrating the circuitry for one speaker pair. A1 is the original power amplifier for one speaker. A2 is 1⁄4 of a TL074 [S7] quad OP-amp. A3 is 1⁄4 of a MCP6024 [S6] quad low-noise OP-amp. Discrete component values: R1 = 1 kΩ, R2 = 10 kΩ, R3 = 100 kΩ, C = 1 μF.

References [S1] Contec. Digital to analog output board for PCI, 16ch type-DA12- + 16(PCI), Datasheet, Ver. 2.13 [Online]. Available: http://www2.contec .co.jp/prod_data/da1216pci/da1216pci_e.pdf

– ______– [S2] Contec. (2013, Dec.). Digital to analog output board for PCI, DA12- Input 16(PCI)-DA12-8(PCI)-DA12-4(PCI), User’s Guide [Online]. Available:

Output + Voltage http://www2.contec.co.jp/dl_data/LZJ367/LZJ367_131203.pdf

Voltage – [S3] Contec. Multi-channel analog input board for PCI-AD12-64(PCI), + Datasheet, Ver. 3.13 [Online]. Available: http://www2.contec.co.jp/ prod_data/ad1264pci/ad1264pci_e.pdf______[S4] Contec. (2013, Dec.). 64/16 Channel analog to digital input board for PCI-AD12-64(PCI)-AD12-16(PCI), User’s Guide [Online]. Available: h___ttp:// FIGURE S3. A traditional instrumentation amplifier based on three www2.contec.co.jp/dl_data/LZJ371/LZJ371_131202.pdf______OP-amps and seven resistors. [S5] Maxxtro. Maxxtro mini speaker MX-US-08, Datasheet [Online]. Available: http://www.elfa.spb.ru/uploads/tdpdf/kt856563.pdf [S6] Microchip. (2009). MCP6021/1R/2/3/4, Datasheet [Online]. experimental setup may use other components, in particu- Available: ______http://ww1.microchip.com/downloads/en/ DeviceDoc/21685d.pdf______lar, eliminating the master–slave configuration inherited [S7] Texas Instruments. (2015, June). TL07xx low-noise JFET-input operation- from the choice of the Maxxtro loudspeaker pairs, for al amplifiers, Datasheet [Online]. Available: http://www.ti.com/lit/ds/ which there is no natural need in our application. symlink/tl074.pdf______

set of requirements that the final product should fulfill. These in the form of a supervisor (normally a senior Ph.D. student tasks are usually renegotiated with the project group until a or teaching assistant) as well as topic expertise (from other final set of requirements is agreed upon, a few weeks after junior faculty members and Ph.D. students). Figure 3 depicts the start of the project. The role of the customer is taken by the various roles and the interactions, with solid lines indicat- a faculty member, and the course director can also assume ing formal contacts (documents, official meetings) and dashed this role. The university also provides support to the students lines representing informal contacts (advice, meetings). Since

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 45

THE WORLD’S NEWSSTAND®

The course project was conducted Project Group of Students Faculty Members according to a project model, which rather precisely defines rules and sup- Project Manager Course Director port for the students’ work process. This helps the students organize their work in an efficient manner and make a care- Project Members Customer fully considered system design before implementing it. The project thus gives Customer Contact Responsible the students the opportunity to work in Reference Group a fashion that is similar to how projects Documentation Leader (Junior Faculty Members are conducted in industry. and Ph.D. Students) In this course, we let the students Test Manager work independently, without offering Supervisor Quality Assurance Manager any initial pointers on how to solve the problem at hand. Instead, we give encouragement to invent and research Main Designer Topic Experts potential solutions based on knowledge from previous courses, and provide Formal Contact (Official Meetings, Documentation) access to expert supervisors who can Informal Contacts (Meetings, Advice) answer questions and discuss ideas. Our general experience is that students FIGURE 3. The roles of the students and faculty members in our implementation of the CDIO project are very successful in forming a work- course concept. ing project group, and they enjoy the opportunity to design a system mostly the project group members work together continuously, the on their own, learning by acquiring hands-on experience. everyday informal contacts between the group members are not shown in the figure. Lessons learned and future directions Our general experience is that all students involved in these The experimental acoustic massive MIMO setup that we have projects have been enthusiastic and have contributed accord- developed is inexpensive and scalable, and the learning curve ingly to their tasks. The group dynamic has varied during the for the equipment has been appropriate for the course. The years and has been somewhat dependent project has been well-received by students in on the interests and ambition levels of the The purpose of our course, that they can apply their knowledge of com- participants. Many original ideas from stu- in general, is to provide a munications theory to build a complete work- dents have been implemented. Usually, the genuine teamwork learning ing system from scratch. They appreciate the desired results have been achieved. Since structure of the project in that they get to try the time scope for the course is strictly experience in the field of out their own ideas in solving a complex prob- limited to one semester, there is no room telecommunications. lem defined on a relatively high level. They for delays. This sometimes has resulted in also appreciate the access to expertise when products that do not reach full functionality. However, in those help is needed. The students have expressed that they enjoy that cases the well-documented work has helped subsequent proj- the course gives them a basic understanding of the concept of ect groups to continue the development and further extend the massive MIMO, as an early insight into the main cutting-edge capabilities of the product. technology component of next-generation wireless networks. The purpose of our course, in general, is to provide a Some of the students’ feedback is as follows: genuine teamwork learning experience in the field of tele- ■ We could have been more thorough in the research communications. Building an audio test bed for demonstra- of equipment that was chosen for the course, since there tion of the massive MIMO concept specifically was a vehicle were some compatibility issues (drivers for the A/D and for bringing the research frontiers to the students and has D/A cards). ignited their interest for the field, as well as prepared them ■ The system should be expanded with more units to allow for an industrial career. The combination of hardware and for a larger scale test system. Sixteen units is borderline software development is a unique aspect of our course. The “massive,” and with 16 units it is difficult to get good challenges of handling hardware imperfections has stimu- results with four units as terminals and the remaining 12 lated the creative skills of the involved students. Combining constituting the massive MIMO array. This certainly makes all of the components and implementing the desired algo- for good future projects in the coming years. rithms required solid teamwork and devotion to the task. The ■ Many students would like to see how the different parts of a product created so far has shown useful capabilities and is communication system go together in real-world systems, also scalable. given more in detail in earlier courses, to be better prepared

46 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

for this kind of task. As in most universities, the majority of obtained the Docent title in data transmission. His research the regular coursework focuses on “theory,” more than prac- interests are within the fields of coding, information, and tical implementation aspects. communication theory. He has authored or coauthored two According to the students, the most book chapters, 16 journal papers, and important experience from the project was According to the students, more than 30 conference papers on these to create an actual, complete communication topics. Since 2012, he has been a board system, with real hardware. This enabled the most important member of the IEEE Sweden Vehicular the students to use much of their knowledge experience from the Technology/Communications/Information from courses, while also encountering the project was to create Theory joint Chapter. problems that are always present with physi- an actual, complete Mikael Olofsson (mikael.olofsson@______cal systems and real hardware. Solving these communication system, liu.se)___ received his M.Sc. degree in electri- kinds of problems is an excellent way for with real hardware. cal engineering in 1992 and his Ph.D. the students to enhance their general skills degree in data transmission in 2002, both in engineering. At the same time, they are from Linköping University, Sweden. Since introduced to the next-generation communication technology. 2000, he has been active as a teacher at Linköping Another aspect that the students benefit from during the University, first as a lecturer and, since 2002, as an associate project course is the opportunity to work in groups with professor. He teaches electronics and communication and, in mixed nationalities. Since the course is on the master level, his role as a teacher, has produced a number of tutorials for typically exchange students from different countries take the his courses, of which one is a published textbook. course, in addition to Swedish engineering students. Thus, Simon Sörman ([email protected])______received the students are able to work in a somewhat international his M.Sc. degree in 2016 in electrical engineering from environment, which is a good preparation for future employ- Linköping University, Sweden. During his studies, he partici- ment as engineers. pated in the course described in this article. He is currently a Taken together, the students appreciated the opportunity researcher on wireless access networks at Ericsson Research, to work practically with a physical system that implements Linköping, and is working toward 5G development. the basic principles of a cutting-edge technology, massive MIMO. They also enjoy working in a very structured manner, References [1] T. L. Marzetta, E. G. Larsson, H. Yang, and H. Q. Ngo, Fundamentals of following a project model which sets up a well-defined, logi- Massive MIMO. Cambridge, U.K.: Cambridge Univ. Press, 2016. cal framework for their work. This project is still relatively [2] E. G. Larsson, H. Q. Ngo, T. L. Marzetta, and H. Yang, Homework Problems to new, and leaves continued opportunities for improvements in go with Fundamentals of Massive MIMO [1], including solution manual. [Online]. the education and preparation of students for future engineer- Available: www.cambridge.org/Marzetta [3] The conceive-design-implement-operate (CDIO) [Online]. Available: http://___ ing work. ____cdio.org/ [4] J. Vieira, F. Rusek, and F. Tufvesson, “Reciprocity calibration methods for mas- Authors sive MIMO based on antenna coupling,” in Proc. IEEE Global Communications Conf., Austin, TX, 2014, pp. 3708–3712. Erik G. Larsson ([email protected])______received his Ph.D. [5] A. Nordrum. (2016, May). “5G researchers set new world record for spectrum degree from Uppsala University, Sweden, in 2002. He is a pro- efficiency,” IEEE Spectr., [Online]. Available: http://spectrum.ieee.org/techtalk/ fessor at Linköping University, Sweden. His main professional ______telecom/wireless/5g-researchers-achieve-new-spectrum-efficiency-record [6]J.Vieira,S.Malkowsky,K.Nieman,Z.Miers,N.Kundargi,L.Liu,I.Wong, interests are within signal processing, communication theory, V. Öwall, O. Edfors, and F. Tufvesson, “A flexible 100-antenna testbed for applied information theory, wireless systems, and 5G. He was massive MIMO,” in Proc. IEEE GLOBECOM Workshops, Austin, TX, 2014, pp. the chair of the IEEE Signal Processing Society Signal 287–293. [7] TELEMIC Group. (2016). Massive MIMO 5G USRP testbed—KU Leuven Processing for Communications and Networking technical [Online].Available: http://www.esat.kuleuven.be/telemic/research/NetworkedSystems/ committee in 2015 and 2016, chair of the steering committee ______infrastructure/massive-mimo-5g of IEEE Wireless Communications Letters in 2014 and 2015, [8] P. Harris, S. Zang, A. Nix, M. Beach, S. Armour, and A. Doufexi, “A distributed massive MIMO testbed to assess real-world performance and feasibility,” in Proc. and he organized the Asilomar Conference on Signals, IEEE 81st Vehicular Technology Conf., Glasgow, Scotland, 2015, pp. 1–2. Systems, and Computers (general chair 2015, technical chair [9] X.Yang,W. J.Lu,N.Wang,K.Nieman,S.Jin,H.Zhu,X.Mu,I.Wong,Y. 2012). He received the IEEE Signal Processing Magazine Best Huang, and X. You, “Design and implementation of a TDD-Based 128-antenna massive MIMO prototyping system,” arXiv: 1608.07362 [Online]. Available: https://___ Column Award twice, in 2012 and 2014, and the IEEE arxiv.org/abs/1608.07362 ComSoc Stephen O. Rice Prize in Communications Theory in [10] N. Choubey and A. Y. Panah. (2016). Introducing Facebook’s new terrestrial connectivity systems—Terragraph and Project ARIES [Online]. Available: https://___ 2015. He is a Fellow of the IEEE. ______code.facebook.com/posts/1072680049445290/introducing-facebook-s- Danyo Danev ([email protected])______received his M.Sc. new-terrestrial-connectivity-systems-terragraph-and-project-aries______degree in mathematics from Sofia University, Bulgaria, in [11] Acoustic massive MIMO test bed (2016). [Online]. Available: ______https://www .youtube.com/watch?v=i1OLEEkiobA 1996 and his Ph.D. degree in electrical engineering from [12] T. Svensson and C. Krysander, Project Model LIPS. Lund, Sweden: Student lit- Linköping University, Sweden, in 2001, where he is current- teratur, 2011. ly an associate professor teaching a number of communication engineering and mathematics courses. In 2005, he SP

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 47

THE WORLD’S NEWSSTAND®

PERSPECTIVES

Xiang-Gen Xia

Small Data, Mid Data, and Big Data Versus Algebra, Analysis, and Topology

ig data has become a hot topic dur- eat will be no more than 900 apples per algebra and abstract algebra; analysis, Bing the last few years. But at times, day. You might want to ask why it is not e.g., real analysis and functional analysis; its meaning has been quite confus- exactly the sum, i.e., 900, of the 400 and and topology and geometry, e.g., alge- ing. I hope that through sharing my 500 apples. The reason is that these two braic topology and differential geometry. thoughts in this article, we can have a families may have some members in In my opinion, all of these subjects are better understanding of what big data is. common and some of them from one about counting and calculation, which is, Whenever you see data, you may family may be married to another in the of course, all that mathematics is about. think that it is related to numbers and other family. In this case, the total count In algebra, you can count exactly. In counting. In fact, today, data is more may not be accurate, but you can have an analysis, you may not be able to count general than numbers. However, when accurate upper bound. Is this small data exactly but roughly or just estimate. You data are input in computers, they be- or something else? I would like to think might want to ask, where are probability come bits and/or numbers. of it as mid data. and statistics? They belong to analysis So, what is big data? When was Next, it comes to the number of apples since they belong to measure theory, it started? Where will it lead? These consumed in the world. How many which belongs to real analysis. In topol- questions may have different answers apples do the people ogy, you are not able to different readers. To me, trained in on the earth eat per In mathematics, there are to count the whole mathematics as a signal processing pro- day? To find out, one mainly three subjects: thing, but one still fessor in an electrical engineering de- might say, let us make algebra, e.g., high school wants to count. In partment, data is quite natural, and so in a table of the numbers this case, what can this article, I provide my answers to the of apples eaten per day algebra and abstract be done? You can aforementioned questions. for every country. It is algebra; analysis, think of the whole First of all, what is big data? Unfor- approximately 300 e.g., real analysis and thing as consisting tunately, there is no precise mathemati- million for the United functional analysis; and of several pieces and cal definition for this concept. Big data States, 300 million topology and geometry, then just count for the or small data is relative. To see what big for Japan, etc. Oops, e.g., algebraic topology number of pieces. The data is, let us first look at what small how many apples do real question is: what data is. Each person in my family, which the people eat in North and differential geometry. is a piece, and what is consists of four people, eats two apples Korea per day? Unfor- topology and geom- per day. Therefore, my family eats eight tunately, there is no trustworthy data avail- etry about? It is a kind of index that you apples per day. This is small data and is able. So, what do we do? Can we still count may get in the limiting case. If I am asked accurate. What is next? For example, my the numbers of apples consumed per day to make an analogy between mathematics whole family, including all relatives, eats for the whole world? No, but we may use and data classification, I would say that 400 apples per day. My neighbor’s whole some colors to mark the levels of the num- algebra corresponds to small data, analy- family, including all of their relatives, bers for all of the countries on a map. In this sis corresponds to mid data, and topology/ eats 500 apples per day. Then, the total case, I would consider it big data, i.e., it is so geometry corresponds to big data. number of apples these two families big that no one can even estimate its volume but can only get some high-level indices. Small data and algebra

Digital Object Identifier 10.1109/MSP.2016.2607319 In mathematics, there are mainly As discussed previously, mathematics Date of publication: 11 January 2017 three subjects: algebra, e.g., high school is about counting and calculation. In

THE WORLD’S NEWSSTAND®

fact, calculation is a type of counting. In the real multiplication, and stands of a norm space, i.e., if there is an opera- many calculations, finding the solutions for the norm used in the domain. In tion on a set that satisfies (3) for any of equations is always one of the most other words, the norm of the product of two elements x and y in the set, this set important tasks. Among finding the any two elements is equal to the product with some additional scaling property solutions of equations, finding the roots of the norms of the two elements. This is called a norm space. It is the key for of polynomials is probably the most is clear when x and y are two complex functional analysis or analysis, includ- important. The fundamental theorem numbers but is less obvious for other ing measure theory and/or probability of algebra tells us that any nonconstant cases. A general design satisfying (1), theory and statistics. In this case, in (3), single-variable polynomial has at least as generalizations of complex numbers, the dot sign is the addition +, and (3) one complex root, which means that any quaternionic numbers, and octonionic is correspondingly called the triangular single-variable polynomial equation can numbers, is called compositions of qua- inequality. In my opinion, the differ- be solved with possibly complex num- dratic forms [1]. A 6knp,,@ Hermitian ence between algebra and analysis is the bers as solutions/roots. We know that composition formula is difference between the norm equality roots of a polynomial of a degree lower and the norm inequality shown in (1) than five have closed forms in terms 2222and (3), respectively. It is the same as the ^^xxyy1 ++ggknhh1 ++ of the coefficients of the polynomial. 22 second apple example mentioned previ- =++zzg p (2) However, for a polynomial of a degree 1 ously, where of five or higher, its roots may not have closed forms in terms of its coefficients, where | | stands for the absolute value, {}400 apples in one family which was first mathematically proven Xx= ^h1 f xk Yy= ^h1 f yn ,, and , , , {}500 apples in another family by Galois and is, therefore, called the are systems of indeterminates, and # {}400 apples in one family Galois theory. To do so, Galois invented zzXYii= (,) is a bilinear form of X and + the concepts of group, ring, and field, Y. As an example, let knp===2 and {}500 apples in another family which led to modern mathematics. The z1=- xy 11 xy 22,. z 2 =+ xy 12 xy 21 =+=400 500 900, smallest field is the binary field {0, 1}, This corresponds to the following case. and the largest is the complex field C The product of the absolute values of where the dot sign in (3) corresponds to that is the set of all complex numbers. two complex numbers is equal to the the union of two sets and the real addi- The reason why C is the largest field is absolute value of the product of the two tion, respectively. I feel that this corre- because every polynomial equation over complex numbers, i.e., if xx=+12j x sponds to mid data. the complex field can be solved already and yy=+12j y for real-valued Another observation about the above by the fundamental theorem of alge- xxyy1212,,, and zz=+12j z = xy , norm inequality is that the dot opera- bra. There are many kinds of subfields then zxyxy== . More designs tion in (3) for two elements x and y can and extended fields, such as algebraic on the compositions of quadratic forms be thought of as a general operation as number fields, by including, e.g., some can be found in [2], which has found we have seen above for different cases roots of unity, i.e., exp(),- 2r j/m for applications as space-time coding in of the dot sign. The norm inequality (3) some positive integer m, in the middle wireless communications with multiple becomes the triangular inequality when of {0, 1} and C. After the complex field, transmit antennas. the dot is +, as mentioned previously. mathematicians generalized C to qua- With this in mind, I would say that When the dot is a true product of two ternionic numbers that form, in fact, a algebra is with the norm identity, where elements, such as the matrix multiplica- domain as well as octonionic numbers. you are able to count precisely (the same tion of two matrices, the inequality (3) For example, a quaternionic number as the first apple example mentioned pre- is the conventional norm inequality. The can be equivalently written as viously), where 24824$$== and norm inequality (3) becomes the Cau- 500+= 400 500 + 400 , when the chy–Schwarz inequality when the dot is x y cm)), dot sign in (1) is the real multiplication the inner product - y x and the real addition, respectively. This, b where x and y are two complex num- in my opinion, corresponds to small data. # ftgtdt() () bers. With these generalizations, mathe- a b 12//b 12 maticians found that the most important Mid data and analysis # 88##f() t2 dtBB g () t2 dt , a a property from all of these structures is In most cases, the norm identity (1) does (4) the norm identity not hold. Instead, it is the following inequality: where the equality holds if, and only xy::= x y (1) if, functions ft ( ) and gt ( ) are lin- xy::# x y (3) early dependent, i.e., ft()= cgt () or for any two elements x and y in the gt()= cft () for some constant c . From domain of interest, where the dot stands for any two elements x and y in a set this observation, almost all inequalities for the multiplication in the domain or called space. This leads to the concept can be derived from the norm inequality

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 49

THE WORLD’S NEWSSTAND®

surface, then the surface has genus 0. For the torus shown in Figure 1(b), it is impossible to do so because, if one picks up a simple loop around the hole, this loop cannot be continuously contracted to any point on the surface. However, if the torus is cut in the middle with one cut, as shown in Figure 1(b) [note that there are two cuts total shown in Fig- ure 1(b)], then it is not possible to have a loop around any hole; thus, any simple loop can be continuously contracted on the surface to a point. In this case, the torus has genus 1, i.e., one and only one (a) (b) (c) cut is used/needed to do so. As shown in Figure 1, genus is a topologically FIGURE 1. The genus of an object; (a) genus 0, (b) genus 1, and (c) genus 2. invariant variable in the sense that two shapes may look totally different, but they have the same genus, where

(3). Many fundamental results are http://cir.institute/collective-______the objects in the first row have zero, derived by the Cauchy–Schwarz intelligence)______or large groups of birds one, and two holes, and are topologi- inequality (4), i.e., the norm inequal- flying in the sky (see http://becausebirds cally equivalent to those in the second

ity. For example, the Cauchy–Schwarz .com/2014/07/29/how-do-bird-flocks-______row, respectively. inequality leads to the conclusion that work),___ he or she may not be able to A possible application of the afore- the optimal linear time-invariant fil- count exactly or estimate approxi- mentioned concept of genus in topology ter to maximize the output signal-to- mately how many fish or birds are would be in the current investigations of noise ratio is, and only is, the filter that there. One may just big data representa- matches to the signal, i.e., the matched count how many What is big data? There is tion that plays an filter. It has been extensively used in disconnected groups no precise mathematical important role in big radar and communications. Another of fish. If a person definition for this concept. data analysis. One application of the Cauchy–Schwarz treats each group as efficient way to repre- inequality is the proof of the Heisen- a visible hole of the Big data or small data sent big data is to use berg uncertainty principle (HUP). It ocean, it is the con- is relative. a proper tensor [5]. says that the product of the time width cept of genus, i.e., When big data is too and the bandwidth is lower bounded one of the key concepts in topology, big and its tensor representation is prop- by one half, and the lower bound is where the number of holes (or fish erly used, it may be treated as a multidi- reached if, and only if, the signal is groups in this case) in an object (i.e., the mensional massive object. In this case, its Gaussian, i.e., abtexp()- 2 for some ocean) is the genus of the object. More topological properties, such as genus, may constant a and some positive constant precisely, the genus of a connected, ori- become simple but is an important feature. b. As a simple consequence of the entable surface is an integer represent- As we have discussed previously, HUP, one is not able to design a signal ing the maximum number of cuttings when an object is too complicated or that has an infinitely small time width along nonintersecting, closed simple curves too massive, the indices and/or the and infinitely small bandwidth simul- without rendering the resultant mani- topologically invariant variables such taneously. Otherwise, a person would fold disconnected [4]. In the aforemen- as the genus, i.e., the number of holes be able to design as many orthogonal tioned definition, cutting is understood as and/or disconnected pieces, come to signals as possible in any finitely lim- the conventional cutting by a knife. Some the picture. These topologically invari- ited area of time and frequency, i.e., it simple examples are shown in Figure 1. ant variables may be obtained by tak- would have infinite capacity for com- Another simple, but more mathemati- ing a limit when some parameters go to munications over any finite bandwidth cal, way to understand it is as follows. infinity, which may smooth out all the channel. One can see that both results If any loop (i.e., a simple closed curve) uncertainties or unknowns caused by have played key roles in science and on a surface (a solid object, such as a the massiveness and make the calcu- engineering in recent history. solid ball), such as the sphere shown lations possible. In other words, tak- in Figure 1(a), can be continuously (on ing a limit may simplify the calculation. Big data and topology the surface or inside the solid object) One simple example is the calculation of When a person sees several large contracted/tightened (also called con- the integration of a Gaussian function. groups of fish moving in the ocean (see tinuously transformed) to a point on the For any finite real values a and b and

50 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

b 2 a positive constant a , # edt-at does when SNR is large enough. Zheng and birds fly in the sky, and/or a massive a not have a simple closed form while Tse [3] obtained the following well- number of people on the ground travel 3 2 # edt- at does. Another example is known DMT: around the world. Today, massive bits - 3 the diversity and multiplexing tradeoff are transmitted through both wired and (DMT) obtained by Zheng and Tse dr( )(=- m r )(), n - r wireless channels called the Internet. [3] for multiple-input, multiple-output The key is how to get some indices, (MIMO) antenna systems in wireless where m and n are the numbers of trans- trends, or patterns from these massive communications, which becomes a nec- mit and receive antennas, respectively. data and/or how to find a needle in the essary parameter in One can see that both ocean. What will big data lead to tomor- designing a MIMO I consider that small data r and dr ( ) are sort row? Or, how deep can we go toward wireless communi- corresponds to algebra, of indices, and they infinity tomorrow? Or, how fast will a cation system. Let mid data corresponds to are only meaningful computer be tomorrow? R be the transmis- analysis, and big data when SNR approach- sion rate in bits/sec- es infinity, i.e., in a Author corresponds to topology ond/hertz. Let r be massive transmission Xiang-Gen Xia (xianggen@gmail.com)______the normalized rate in mathematics. rate case or big data is the Charles Black Evans Professor in r = R/(log SNR ), case. This is the case the Department of Electrical and where SNR stands for signal-to-noise when it is impossible to count one ele- Computer Engineering, University of ratio and is the channel SNR. When ment by one element for a massive data, Delaware, Newark. His main research SNR is huge, one may expect that R is and one needs to sort out its index, such as interests include wireless communica- huge as well by Shannon’s channel capac- exponentials and/or genus, in some way tions and radar signal processing. He is a ity formula that is about log(), SNR i.e., to describe and/or extract features from Fellow of the IEEE. massive data (or big data) can be trans- the massive/big data. I think this belongs mitted through the channel. In this case, to topology in mathematics. Thus, in my References R [1] D. B. Shapiro, Compositions of Quadratic counting may be not possible, while opinion, topology in mathematics corre- Forms. New York: De Gruyter, 2000. counting r becomes more reasonable, sponds to big data, where it is impossible [2] K. Lu, S. Fu, and X.-G. Xia, “Closed-form where r is called the multiplexing gain. or not necessary to count one element by designs of complex orthogonal space-time block codes of rate (k+1)/(2k) for 2k-1 or 2k transmit anten- Let Pe be the error probability at the one element. nas,” IEEE Trans. Inform. Theory, vol. 51, pp. 4340– receiver of a MIMO modulation scheme 4347, Oct. 2005. with transmission rate R. Let Summary and discussion [3] L. Zheng and D. N. C. Tse, “Diversity and multiplexing: A fundamental tradeoff in multiple antenna In summary, I consider that small data channels,” IEEE Trans. Inform. Theory, vol. 49, log()Pe dr()=- lim . (5) corresponds to algebra, mid data corre- pp. 1073–1096, May 2003. SNR " 3 log()SNR sponds to analysis, and big data corre- [4] Wikipedia. Genus. (2016). [Online]. Available: https://en.wikipedia.org/wiki/Genus_(mathematics) dr Then, () is the index of the negative sponds to topology in mathematics. Was [5] A. Cichocki. (2014). Era of big data processing: A exponential of the error probability Pe big data started when it was named? new approach via tensor networks and tensor decom- diversity gain positions. arXiv: 1403.2048v4 (cs.ET) [Online]. and called the . Of course not. Big data has existed for Available: http://arxiv.org/abs/1403.2048v4 a long time, as massive groups of fish -dr() Pe . SNR , (6) move in the ocean, massive groups of SP

ERRATA

eference [8] in the “SP Education” We apologize for any confusion this may Signal Process. Mag., vol. 33, no. 6, pp. 123–127, column of the November 2016 issue have caused. The corrected reference is Nov. 2016. R [2] BCI hand control download [Online]. Available: of IEEE Signal Processing Maga- shown in [2]. http://classes.engineering.wustl.edu/ese497/BigFiles/ zine [1] was published missing a URL. IpsiHand_v5.wmv______References Digital Object Identifier 10.1109/MSP.2016.2636258 [1] E. Richter and A. Nehorai, “Enriching the under- Date of publication: 11 January 2017 graduate program with research projects,” IEEE SP

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 51

THE WORLD’S NEWSSTAND®

Richard G. Baraniuk, Tom Goldstein, Aswin C. Sankaranarayanan, Christoph Studer, Ashok Veeraraghavan, and Michael B. Wakin Compressive Video Sensing Algorithms, architectures, and applications

he design of conventional sensors is based primarily on the 10 million measurements of the scene. Yet, almost immediately Shannon–Nyquist sampling theorem, which states that a after acquisition, redundancies in the image are exploited to T signal of bandwidth W Hz is fully determined by its dis- compress the acquired data significantly, often at compression crete time samples provided the sampling rate exceeds 2 W ratios of 100:1 for visualization and even higher for detection samples per second. For discrete time signals, the Shannon– and classification tasks. This example suggests immense wast- Nyquist theorem has a very simple interpretation: the number of age in the overall design of conventional cameras. data samples must be at least as large as the dimensionality of Compressive sensing (CS) (see “CS 101” and [6], [14], [16], the signal being sampled and recovered. This important result and [24]) is a powerful sensing paradigm that seeks to allevi- enables signal processing in the discrete time domain without ate the daunting sampling rate requirements imposed by the any loss of information. However, in an increasing number of Shannon–Nyquist principle. CS exploits the inherent structure applications, the Shannon–Nyquist sampling theorem dictates (or redundancy) within the acquired signal to enable sampling an unnecessary and often prohibitively high sampling rate (see and reconstruction at sub-Nyquist rates. The signal structure “What Is the Nyquist Rate of a Video Signal?”). As a motivating most commonly associated with CS is that of sparsity in a example, the high resolution of the image sensor hardware in transform basis. This is the same structure exploited by image modern cameras reflects the large amount of data sensed to cap- compression algorithms, which transform images into a basis ture an image. A 10-megapixel camera, in effect, takes [e.g., using a wavelet or discrete cosine transform (DCT)] where they are (approximately) sparse. In a typical scenario, a CS still-image camera takes a small number of coded, linear Digital Object Identifier 10.1109/MSP.2016.2602099 Date of publication: 11 January 2017 measurements of the scene—far fewer measurements than the

THE WORLD’S NEWSSTAND®

number of pixels being reconstructed. Given these measure- is, we can exploit the rich temporal redundancies in a video to ments, an image is recovered by searching for the image that reconstruct frames from far fewer measurements than is pos- is sparsest in some transform basis (wavelets, DCT, or other) sible with still images. Yet the demands of video CS in terms while being consistent with the measurements. of the complexity of imaging architectures, signal models, and In essence, CS provides a framework to sense signals with reconstruction algorithms are significantly greater than those of far fewer measurements than their ambient dimensionality (i.e., compressive still-frame imaging. Nyquist rate), which translates to practical benefits including There are three major reasons that the design and imple- decreased sensor cost, bandwidth, and time of acquisition. These mentation of CS video systems are significantly more difficult benefits are most compelling for imaging modalities where sens- than those of CS still-imaging systems. The first challenge is ing is expensive; examples include imaging in the nonvisible the gap between compression and CS. State-of-the-art video spectrum (where sensors are costly), imaging at high spatial and models rely on two powerful ideas: first, motion fields enable temporal resolutions (where the high bandwidth of sensed data the accurate prediction of image frames by propagating inten- requires costly electronics), and medical imaging (where the time sities across frames; second, motion fields are inherently more of acquisition translates to costs or where existing equipment is compressible than the video itself. This observation has led too slow to acquire certain dynamic events). In this context, archi- to today’s state-of-the-art video compression algorithms (not tectures like the single-pixel camera (SPC) [27] provide a prom- to be confused with CS of videos) that exploit motion infor- ising proof of concept that still images can be acquired using a mation in one of many ways, including block-based motion small number of coded measurements with inexpensive sensors. estimation (MPEG-1), per-pixel optical flow (H.265), and There are numerous applications where it is desirable to wavelet lifting (LIMAT). Motion fields enable models that extend the CS imaging framework beyond still images to incor- can be tuned to the specific video that is being sensed/pro- porate video. After all, motion is ubiquitous in the real world, cessed. This is a powerful premise that typically provides an and capturing the dynamics of a scene requires us to go beyond order of magnitude improvement in video compression over static images. A hidden benefit of video is that it offers tremen- image compression. dous opportunities for more dramatic undersampling (the ratio The use of motion fields for video CS raises an important of signal dimensionality to measurement dimensionality). That challenge. Unlike the standard video compression problem,

What Is the Nyquist Rate of a Video Signal?

Conventional videos, sampled at 24–60 frames/second (fps), may, in fact, be highly undersampled in time— Temporal Frequency objects in the scene can move multiple pixels between Spectral Support adjacent frames. Some compressive sensing (CS) architec- of Analog Video tures, however, measure a video at a much higher temporal rate. For example, the single-pixel camera (SPC) may Spatial Frequency take tens of thousands of serial measurements per second. Temporal In such cases, the scene may change very little between Bandwidth of Sampled adjacent measurements. This raises some interesting ques- Video tions: what is the Nyquist rate of a video signal, and how Spatial does it compare to CS measurement rates? Resolution One can gain insight into these questions by considering of Optics the three-dimensional analog video signal that arrives at a FIGURE S1. The limited spatial resolution of an imaging system may camera lens; both conventional and CS imaging systems also limit its temporal bandwidth. can be viewed as blurring this signal spatially (due to the optics and the pixelated sensors) and sampling or measuring both the camera optics and the pixel sensors, when the spa- it digitally. If a video consists of moving objects with sharp tial bandwidth of the video is limited, so too is its temporal edges, then the analog video will actually have infinite band- bandwidth, as illustrated by the black rectangle in the figure. width in both the spatial and temporal dimensions. However, This suggests that the video sensed by architectures such as it can be argued that the support of the video’s spectrum will the SPC may in fact have a finite temporal bandwidth, and tend to be localized into a certain bowtie shape, as shown this fact can be used to reduce the computational complexity in blue in Figure S1. The salient feature of this shape is that of sensing and reconstructing the video. In particular, it is not high temporal frequencies coincide only with high spatial necessary to reconstruct at a rate of thousands of fps. frequencies. Thus, because of the limited spatial resolution of Additional details are provided in [62].

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 53

THE WORLD’S NEWSSTAND®

CS 101

Compressive sensing (CS) exploits the fact that a small and By sparse, we mean that only KN% of the entries in s are carefully selected set of nonadaptive linear measurements of a nonzero, or that there exists a sparsifying transform W such compressible signal, image, or video carries enough informa- that most of the coefficients of a:= Ws are zero. By compress- tion for reconstruction and processing [16], [24]; for a tutorial ible, we mean that s or a is approximately sparse. Let -1 treatment see [6], [14]. W :[= }}12 ,, f , }N ] represent the inverse of the N # N The traditional digital data acquisition approach uniformly basis matrix; then, s = W-1 a and ys==UUW-1 a. samples the three-dimensional analog signal corresponding Typically in CS, the sparse signal s or its sparse coeffi- to the time variations of a scene; the resulting samples cients a is recovered by solving an optimization problem Vxyt[,,] in space (xy , ) and time (t) are sufficient to perfectly of the form (1), where f measures the fidelity of the recov- - UW-1 a 2 recover a bandlimited approximation to the scene at the ery (e.g., using the squared error y 2 ) and g is Nyquist rate. Let the abstract vector s represent the Nyquist- a regularization penalty (e.g., the ,1 -norm a 1, which rate samples of the scene Vxyt[,,]; see “What Is the Nyquist promotes sparsity of a). In these cases, the resulting prob- Rate of a Video Signal?” for a discussion of the Nyquist rate lem is convex, which guarantees a single global minimizer of a time-varying scene. Because the number of samples that can be found using a range of algorithms. required for real-world scenes, N, is often very large, for While the design of the sensing matrix U is beyond the example, in the billions for today’s consumer digital video scope of this review, typical CS approaches employ a ran- cameras, the raw image data is typically reduced via data dom matrix. For example, we can draw the entries of U as compression methods that typically rely on transform coding. independent and identically distributed !1 random vari- As an alternative, CS bypasses the Nyquist sampling ables from a uniform Bernoulli distribution [8]. Then, the process and directly acquires a compressed signal measurements y are merely M different sign-permuted linear representation using M 1 N linear measurements combinations of the elements of s. Other choices for U M between s and a collection of linear codes {z [m ]} m = 1 as exist in the literature, such as randomly subsampled Fourier in ym[]=GH s ,[]z m . Stacking the measurements ym [ ] into or Hadamard bases. In this case, multiplication by U can be the M-dimensional vector y and the transpose of the accomplished using fast transform algorithms, which enables codes z[]m T as rows into an M # N sensing matrix U , faster reconstruction than is possible with random matrices. we can write ys= U . It is important to emphasize that CS is not a panacea for The transformation from s to y is a dimensionality reduction all the world’s sampling problems [7]. In particular, to and does not, in general, preserve information. In particular, apply the concept profitably, it is critical that the signal s because M 1 N, there are infinitely many vectors sl that sat- possess a lower inherent dimensionality than its ambient isfyys= U l . The magic of CS is that U can be designed dimensionality (e.g., sparse structure) and that the degree such that sparse or compressible signals s can be recovered of undersampling N/M be balanced with respect to the exactly or approximately from the measurements y. signal’s signal-to-noise ratio [22].

where the frames of the video are explicitly available to per- imaging architectures that can deliver very high measurement form motion estimation, in CS, we have access only to coded rates or reconstruct at different resolutions depending on the and undersampled measurements of the video. We are thus available data. The third challenge is computational complex- faced with a chicken-or-egg problem. Given high-quality video ity. Even moderate resolution videos result in high bandwidth frames, we could precisely estimate the motion fields; but streaming measurements. Typical CS video recovery algorithms we need precise motion estimates in the first place to obtain are highly nonlinear and often involve expensive iterative opti- high-quality video frames. The second challenge is the laws of mization routines. Fast (or even real-time) reconstruction of CS causality and imaging architectures. Time waits for no one. A video is challenging because it requires a data measurement sys- distinguishing property of the video sensing problem over still tem, fast iterative algorithms, and high-performance hardware imaging is the fundamental difference between space and time. jointly designed to enable sufficiently high throughout. The ephemeral nature of time poses significant limitations on The goal of this article is to overview the current approach- the measurement process—clearly, we cannot obtain additional es to video CS and demonstrate that significant gains can be measurements of an event after it has occurred. As a conse- obtained using carefully designed CS video architectures and quence, it is entirely possible that a compressive camera does algorithms. However, these gains can only be realized when not capture a sufficient number of measurements to recover the there is cohesive progress across three distinct fields: video frames of the video. Overcoming this challenge requires both models, compressive video sensing architectures, and video an understanding of the spatial-temporal resolution tradeoffs reconstruction algorithms. This article reviews progress that associated with video CS and development of novel compressive has been made in advancing and bringing these fields together.

54 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

We discuss some of the landmark results in video CS and high- ■ Spectral and angular multiplexing cameras (SAMCs): light their key properties and the rich interplay among models, SAMCs boost resolution in the spectral domain, which architectures, and algorithms that enable them. We also lay out can be useful for hyperspectral and light-field video sens- a research agenda to attack the key open research problems and ing. As with TMCs, the bottleneck of these architectures is practical challenges to be resolved in video CS. also the measurement rate constraint imposed by the ADC. Each of these flavors of a CS system aims to break the Video sensing systems Nyquist barrier to obtain either higher spatial, temporal, or In this section, we discuss the current compressive imaging spectral resolution. In the following sections we discuss the architectures that have been proposed for CS video. The archi- key design considerations and existing implementations of tectures can be broken down into three categories (see Table 1). these three camera types. ■ Spatial multiplexing cameras (SMCs): SMCs optically superresolve a low-resolution sensor to boost spatial resolu- SMCs tion. SMCs are invaluable in regimes where high-resolution SMCs apply CS multiplexing in space to boost the spatial sensors are unavailable, as in terahertz/millimeter-wave and resolution of images and videos obtained from sensor arrays magnetic resonance imaging (MRI), or extremely costly, with low spatial resolution. The use of a low-resolution sen- as in short or medium wavelength infrared (SWIR and sor enables SMCs to operate at wavelengths where corre- MWIR) sensing. sponding full-frame sensors are too expensive, such as at ■ Temporal multiplexing cameras (TMCs): TMCs optically SWIR, MWIR, terahertz, and millimeter wavelengths. SMCs superresolve a low-frame-rate camera to boost temporal employ a spatial light modulator, such as a digital micro- resolution. TMCs are mainly used to overcome the limita- mirror device (DMD) or liquid crystal on silicon (LCoS), to tions imposed on the measurement rate by the analog-to- optically compute a series of coded inner products with the digital converter (ADC) and are optimized to produce a rasterized scene s; these linear inner products determine high-frame-rate video at high spatial resolution with low- the rows of the sensing matrix U (recall the notation from frame-rate sensors. “CS 101”). It is worth mentioning that the SMC approach

Table 1. The key architectures for CS video and their properties.

Type Name Application Modulator Best-known capabilities Limitations SMC SPC Infrared DMD Spatial resolution 128 × 128 Operational speed of DMD imaging Time resolution 64 fps Result [27] LiSens/FPA-CS Infrared DMD Spatial resolution 1,024 × 768 Need for precise optical imaging Time resolution 10 fps alignment/calibration Result [19], [78] TMCs Coded strobing High-speed Mechanical/ Spatial resolution (sensor) Periodic scenes imaging ferroelectric shutter Time resolution 2,000 fps Result [75] Flutter shutter High-speed Mechanical/ Spatial resolution (sensor) Locally linear motion imaging ferroelectric shutter Time resolution 4 × sensor fps Result [64]

P2C2 High-speed LCoS Spatial resolution (sensor) Dynamic range of sensor imaging Time resolution 16 × sensor fps Result [65]

Per-pixel shutter High-speed LCoS/electronic Spatial resolution (sensor) Light loss imaging shutter Time resolution 16 × sensor fps Result [39]

CACTI High-speed Translating mask Spatial resolution (sensor) Mechanical motion imaging Time resolution 100 × sensor fps Result [51]

Light-field video Dynamic LCoS, used as Time resolution sensor fps Loss of spatial resolution can refocusing programmable Result [71] be severe for high spectral/ coded aperture angular resolutions Hyperspectral CASSI Spectroscopy Static mask Time resolution sensor fps video Result [76]

fps: frames/second; FPA: focal plane array; P2C2: programmable pixel compressive camera; CACTI: coded aperture compressive temporal imaging; CASSI: coded aperture snapshot spectral imaging.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 55

THE WORLD’S NEWSSTAND®

is equally applicable to modalities outside Fast (or even real-time) images onto a single sensor, the spatial res- the scope of this article, such as MRI [52], reconstruction of CS olution is limited by the density of mirrors where the physics of image formation pro- video is challenging on the DMD. duces a measurement system that can be because it requires a data Since the proposal of the original SPC in interpreted as subsampling the Fourier [27], numerous authors have developed alter- transform of the sensed image. measurement system, native SPC architectures that do not require a fast iterative algorithms, DMD for spatial light modulation. In [41], a SPC and high-performance liquid-crystal display panel is used for spatial The SPC [27] acquires images using only a hardware jointly designed light modulation; the use of a transmissive single sensor element (i.e., a single pixel) to enable sufficiently light modulator enables a lensless architec- and taking significantly fewer multiplexed high throughout. ture. Sen and Darabi [70] use a camera-pro- measurements than the number of scene jector system to construct an SPC exploiting pixels. In the SPC, light from the scene is a concept referred to as dual photography focused onto a programmable DMD, which directs light from [69]; the hallmark of this system is its use of active and coded a subset of activated micromirrors onto the single photodetec- illumination that can be beneficial in certain applications, par- tor. The programmable nature of the DMD’s micromirror ori- ticularly microscopy. entation enables one to direct light either toward or away from the photodetector. As a consequence, the voltage measured at Beyond SPCs—Multipixel detectors the photodetector corresponds to an inner product of the As mentioned previously, the measurement rate of an SPC image focused on the DMD and the micromirrors directed is limited by the pattern rate of its DMD, which is typically toward the sensor (see Figure 1). Specifically, at time t, if the in the tens of kilohertz. This measurement rate can be insuf- DMD pattern is represented by z [t ] and the time-varying ficient for scenes with very high spatial and temporal reso- scene by Vxyt[,,] (where x and y are the two spatial dimen- lutions. This issue can be combatted using an SMC with sions and t is the temporal dimension), then the photodetector F sensor pixels (photodetectors), each capturing light from measures a scalar value yt [ ]=+GHz [ t ], V [·,·, t ] et [ ], where a nonoverlapping region of the DMD. The measurement GH·,· denotes the inner product between the vectors and et [ ] rate of the SMC increases linearly with the number of pho- accounts for the measurement noise. If the scene is static, that todetectors. Taking into account that the maximum mea- is, Vxyt[,,]= V0 [,], xy then the measurement vectors can be surement rate is capped by the sampling rate of the ADC, stacked as columns into a measurement matrix, with we can write the measurement rate for an SMC with F T U = [,,zz12f , zM ]. The SPC leverages the relatively high photodetectors as pattern rate of the DMD, which is defined as the number of min"F # RRDMD,, ADC, unique micromirror configurations that can be obtained in unit time. This pattern rate, typically 10–20 kHz for commer- where RDMD is the pattern rate of the DMD and RADC is the cially available devices, defines the measurement bandwidth sampling rate of the ADC. Hence, the smallest number of pho- (i.e., the number of measurements per second) and is one of todetectors for which the measurement rate is maximized is the key factors that defines the achievable spatial and tempo-

ral resolutions. Because SPCs rely on the DMD to modulate (minimum number of sensor pixels)Fmin= RR ADC / DMD.

Photodetector A/D Main Lens Relay Lens ,

Image Binary DMD Focused on Pattern on the DMD the DMD

FIGURE 1. The operation principle of the SPC. Each measurement corresponds to an inner product between the binary mirror–mirror orientation pattern on the DMD and the scene to be acquired. (Figure courtesy of [67].)

56 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

In essence, at FF= min we can obtain the measurement rate of recently, there have also emerged multipixel multiplexing-based a full-frame sensor but using a device with potentially a frac- cameras that completely get rid of the lens and replace the lens tion of the number of photodetectors. This can be invaluable with a mask and computational reconstruction algorithms [2]. for sensing in many wavebands, for example, SWIR. As a case study, consider an SMC with a DMD pat- TMCs tern rate RDMD = 10 kHz and an ADC with a sampling rate TMCs apply CS multiplexing in time to boost the temporal res- RADC = 10 MHz. Then, for a sensor with Fmin = 1 , 000 pixels, olution of videos obtained from sensor arrays with low tempo- we can acquire 10 million measurements per second. An SPC, in ral resolution. Again, let Vxyt[,,] be a three-dimensional (3-D) comparison, would acquire only 10,000 measurements per sec- signal representing a time-varying scene. Due to the assumed ond. Consequently, multipixel SMCs can acquire videos at sig- low frame rate of the sensor, we obtain a scene measurement nificantly higher spatial and temporal resolutions than an SPC. once every T seconds, where T is too large. If the SLM has an There have been many multipixel extensions to the SPC operational speed of one pattern every TSLM seconds, then each concept. The simplest approach [46] maps the DMD to a measurement of a TMC takes the form of a coded image: low-resolution sensor array, as opposed to a single photodetector, such that each pixel on the sensor observes a nonoverlapping C-1 yxyt[,,0 ]=+/ z [,,] xyj Vxyt [,,0SLMjT ], patch or a block of micromirrors on the DMD. SMCs based on j = 0 this design have been proposed for sensing in the visible [78], SWIR [19], and MWIR [54]. Figure 2 shows an example of where z[,,]xyj is the attenuation pattern on the SLM at spatial the increased measurement rates offered by the LiSens camera location (,xy ) and time jTSLM. Here, each coded image mea- [78], which uses a linear array of 1,024 photodetectors. More sured by the TMC multiplexes C frames of the high-speed

108 SPC Relay Lens RADC 107 Objective Lens DMD 106

D M D R 105 ∗ Measurement Rate F

Fmin Line 104 Sensor Cylindrical Lens 100 102 104 106 Number of Pixels F (a) (b)

LiSens SPC 768 × 1,024 64 × 64 128 × 128 256 × 256 0.88 s Capture Duration 0.11 s

(c)

FIGURE 2. The multipixel SMCs support significantly higher sensing rates than an SPC. (a) The measurement rate as a function of the number of sensor pixels. An optimized SMC with F min pixels delivers the highest possible measurement rate. (b) Lab prototypes of the SPC and LiSens cameras, each placed on the one arm of a single DMD. The measurement rate of the LiSens camera is nearly 1 MHz, while that of the SPC is 20 kHz. (c) Comparisons between LiSens, which uses 1,024 sensor pixels, and an SPC for a static scene. Each row corresponds to a different capture duration, defined as the total amount of time that the cameras have for acquiring compressive measurements. The larger measurement rate of the LiSens camera enables it to sense scenes with very high spatial resolution even for small capture durations. (Photos courtesy of [78].)

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 57

THE WORLD’S NEWSSTAND®

video, and hence, we obtain one coded SMCs apply CS examples include hyperspectral CS video image every CTSLM seconds. Our goal is to multiplexing in space cameras that sense spatial, spectral, and recover the frames of the high-speed video to boost the spatial temporal variations of light in a scene and VxykT[,,SLM ] from a single or a sequence light-field video cameras that sense spatial, of coded images/measurements. resolution of images angular, and temporal variations. In both and videos obtained from cases, imaging at high resolution across all Global shutters sensor arrays with modalities simultaneously requires that we The simplest instance of a TMC uses a glob- low spatial resolution. handle both high measurement rates (this is al shutter together with a conventional cam- typically limited by the ADC sampling rate) era. In a global shutter, the SLM code and low light levels (due to scene light UU[,,]xyjj= [] is spatially invariant, which can be imple- being resolved into various modalities). CS techniques, more mented by using a programmable shutter or by using the image specifically, signal models, can address both bottlenecks. sensor’s built-in electronic shutter. Veeraraghavan et al. [75] Examples of compressive cameras include the coded aperture showed that periodic scenes can be imaged at very high tempo- snapshot spectral imaging architecture [76] and compressive ral resolutions using a global shutter [64]. This idea has been hyperspectral imaging using spectrometers [50] for spectral extended to nonperiodic scenes in [40], where a union-of-sub- multiplexing and the work of Marwah et al. [58] and Tambe space model was used to temporally superresolve the captured et al. [71] for angular multiplexing. scene. However, global shutters are fundamentally limited to providing only spatially invariant coding of the video; this can Models for video structure be insufficient to provide a rich-enough encoding of a high- Recovering a video from compressive linear measurements speed video. Hence, in spite of their simplicity, global shutters requires one to extract the video signal s from the measure- fail for scenes with complex motion patterns. ments ys= U (recall “CS 101”). Here, s might represent a certain block of pixels, an entire video frame, or an ensemble Per-pixel shutters of frames, depending on the sensing architecture and the spe- Reddy et al. [65] proposed the programmable pixel compressive cific recovery algorithm employed. All of these are functions camera (P2C2), which extends the global shutter idea with per- of the underlying time-varying scene Vxyt[,,]. Because the pixel shuttering. Here, each pixel has its own unique code that is number of measurements M is less than the video signal’s typically binary valued and pseudorandom. The P2C2 architec- ambient dimensionality N, infinitely many vectors sl may sat- ture uses an LCoS SLM placed optically at the sensor plane and isfyys= U l . Hence, to recover s from y, a model that cap- carefully aligned to a high-resolution two-dimensional (2-D) tures the scene structure (or a priori information) of s with a sensor. The P2C2 prototype achieves 16 # temporal superreso- small number of degrees of freedom is required; the model can lution, even for complex motion patterns. Hitomi et al. [39] then be included in the recovery algorithm. This section sur- extended the P2C2 camera using a per-pixel coding that is more veys several popular models for characterizing low-dimension- amenable to implementation in modern image sensors with per- al structure in videos. pixel electronic shutters. Here, U[,,]xyjjj=-d [0 (,)]xy ; that is, each pixel observes the in tensity at one of the subframes of Single-frame structure the high-speed video, and the selection of this subframe varies The structure of a single video frame can be characterized spatially. Llull et al. [51] and Koller et al. [47] proposed a TMC using standard models for conventional 2-D images. Natural that achieves temporal multiplexing via a translating mask in images have been shown to exhibit sparse representations in the sensor plane. This approach avoids the hardware complexity the 2-D DCT, 2-D wavelet, and curvelet domains [15], [56]. involved with DMD and LCoS SLMs and enjoys other benefits, Images have also been shown to have sparse gradients. The including low operational power consumption at the cost of total variation (TV) seminorm promotes such gradient spar- having a mechanical component (the translating mask). sity simply by minimizing the ,1 norm of an image’s 2-D gradient [52]. To fully exploit the structure in a 3-D video, Additional TMC designs one needs to characterize the spatial and temporal dimen- Gu et al. [36] used the rolling shutter of a complementary metal– sions simultaneously, rather than reconstructing each frame oxide–semiconductor (CMOS) sensor to enable higher temporal independently and only accounting for spatial structure. resolution. The key idea is to stagger the exposures of each row Hence, the spatial 2-D regularizers described previously randomly and use image/video models to recover a high-frame- often appear as building blocks of more sophisticated 3-D rate video. Harmany et al. [37] extended coded aperture systems video models. by incorporating a global shutter; the resulting TMC provides immense flexibility in the choice of the measurement matrix U. Sparse innovation models One of the simplest possible models accounting for multi- SAMCs frame structure assumes that a video can be reduced into a SAMCs apply CS multiplexing to sense variations of light in a static and a dynamic component. This model—while restric- scene beyond the spatial and temporal dimensions. Two specific tive—is applicable, for example, in surveillance applications,

58 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

where a scene is observed from a distant One of the simplest successfully employed for CS video recon- static camera. We can decompose each possible models struction in [39]. frame of such a video into a static back- accounting for multiframe ground frame and a number of small structure assumes that Linear dynamical systems (sparse) foreground objects that may Linear dynamical systems (LDSs) model change location from frame to frame. A a video can be reduced the dynamics in a video using linear sub- natural way of modeling such structure is into a static and a space models. Such models have been to assume that the differences between dynamic component. used extensively in the context of activity consecutive frames have a sparse represen- analysis and dynamic textures. Video CS tation in some transform basis. That is, for using LDS reduces to the estimation of the two consecutive video frames Vxyt[,,1 ] and Vxyt [ , ,2 ], one LDS parameters, including the observation matrix and the may assume that the difference frame Vxyt[,,21 ]- Vxyt [,, ] state transition matrix, from compressive measurements. has a sparse representation in a basis such as a 2-D wavelet Approaches for parameter estimation have included recur- basis. Such models have been explored in detail in the con- sive [73] as well as batch methods [66]. Furthermore, [66] text of CS [17], [57], [74] and can be viewed as special cases demonstrates the use of the recovered LDS parameters for of the more advanced motion-compensation techniques activity classification. described below. Motion compensation Low-rank matrix models While regularizers such as 3-D wavelets and 3-D TV minimi- An alternative approach to scene modeling involves reorganiz- zation can be used for CS video reconstruction, it is worth not- ing a 3-D video signal into a 2-D matrix, where each column ing that conventional video compression algorithms (such as of the matrix contains a rasterized ordering of the pixels of one H.264) do not employ such simple techniques. Rather, video frame. A variety of popular concise models for matrix because objects in a video may move (or translate) several structure can then be interpreted as models for video structure. pixels between adjacent frames, it is typical to employ block- One of the most prominent models asserts that the matrix is based motion compensation and prediction, where each video low rank; this is equivalent to assuming that the columns of the frame is partitioned into blocks, the location of each block is data matrix live in a common, low-dimensional subspace. In predicted in the next frame, and only the residual of this pre- the context of video modeling, a seminal result by Basri and diction is encoded. Jacobs [9] showed that collections of images of a Lambertian Some CS video architectures may require reconstructions object under varying lighting often cluster close to a nine- of video sequences with high temporal frame rates. In these dimensional subspace. This property can be useful for model- cases, there may be relatively little object motion between ing videos of stationary scenes where the illumination consecutive frames. Consequently, motion compensation may conditions change over time. not be required, and techniques such as 3-D TV may result in To account for both variations in background illumination high-quality scene recovery. and for sparse foreground objects that move with time, one can In other cases, however, it may be necessary to predict and extend the low-rank matrix model to a low-rank-plus-sparse compensate for the motion of objects between consecutive model [79], [80]. A sparse matrix, added to the original low- frames. This presents an interesting chicken-or-egg problem: rank matrix, accounts for sparse foreground innovations, such motion compensation can help in reconstructing a video, but as small moving objects. Again, such models are particularly the motion predictions themselves cannot be made until (at suitable for surveillance applications. least part of) the video is reconstructed. One iterative, multiscale technique has been proposed [62] that alternates between TV minimization and sparse dictionaries motion estimation and video reconstruction: the recovered Sparsifying transforms such as wavelets, curvelets, and the video at coarse scales (low spatial resolution) is used to esti- DCT have natural extensions to 3-D [56], [77], [82] and can be mate motion, which is then used to boost the recovery at finer employed for jointly reconstructing an ensemble of video scales (high spatial resolution). Given the estimated motion frames. TV minimization can also be extended to 3-D [35], [49]; vectors, a motion-compensated 3-D wavelet transform can be minimizing the 3-D TV seminorm of a video promotes frames defined using the LIMAT technique [68]. Another approach with sparse gradients across spatial and temporal dimensions. initially reconstructs frames individually, estimates the motion It is also possible to learn specialized (possibly overcom- between the frames, and then attempts to reconstruct any resid- plete) bases that enable sparse representations of patches, ual not accounted for by the motion prediction [30]; see also frames, and videos from training data. A variety of so-called [45] for a related technique. The logistics of block-based video dictionary learning algorithms have been proposed that learn sensing and reconstruction are discussed in detail in [30]. sparsifying frames W (see, e.g., [1] and “CS 101”). Dictionary learning algorithms can be used not only to generate diction- Optical flow aries that sparsify images but also to sparsify videos in both A more general approach to motion compensation involves the the spatial and temporal dimensions. This approach has been optical flow field. Given two frames of a video, Vxyt [ , ,1 ] and

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 59

THE WORLD’S NEWSSTAND®

Vxyt[,,2 ], optical flow refers to the flow field {uxy ( , ), vxy ( , )} greedy. Note that there are algorithms that do not fit well into such that Vx[++ uxy (,), y vxy (,),] t1 =Vxyt[,,2 ]. Optical flow these categories (such as iterative hard thresholding [12], which enables one to represent the frames of a video using a small has features of both); a discussion of such methods is beyond collection of key frames plus optical flow fields that synthesize the scope of this article. (extrapolate) the video from the key frames. Optical flow fields are often significantly more compressible than images. Such an Variational methods approach is closely related to the block-based motion compen- Variational methods for CS video recovery perform scene sation models described earlier but is distinguished by its reconstruction by solving optimization problems using itera- explicit attempt to model motion on a per-pixel basis. tive algorithms. Most variational methods suitable for high- A key challenge in the use of optical flow models for video dimensional problems can be classified into two categories, CS is—once again—that, in the context of sensing, we do constrained and unconstrained, as detailed next. not have access to the flow fields nor do we have access to high-quality images from which to estimate the flow fields. Constrained problems Reddy et al. [65] resolve this chicken-or-egg problem by first The first category solves constrained problems of the form recovering a video with simple image-based priors, estimating the optical flow field on the initial reconstruction, and sfsygzzst =+argmin (|)()UWsubject to =. (1) sz, subsequently recovering the video again while simultaneously enforcing the brightness constancy constraints derived using Here, the function f models the video acquisition process optical flow. They show that a 30-frames/second (fps) sensor (optics, modulation, and sampling), and g is a regularizer that can be superresolved to a 240–480-fps sensor by temporal promotes sparsity under the transformation defined by W. modulation using an LCoS device. In the context of SMCs, For example, basic frame-by-frame recovery with 2-D wavelet Sankaranarayanan et al. [67] use a specialized dual-scale sens- sparsity can be formulated as an unconstrained problem with UU=- 2 = c ing (DSS) matrix that provides robust and computationally fsy() y s2 and gz() z 1 , where s contains a inexpensive initial scene estimates at a lower spatial resolution. vectorized image frame, U is the sensing matrix, W is a 2-D This enables this approach to robustly estimate optical flow wavelet transform, and c 2 0 is a regularization parameter. fields on a low-resolution video. Optical flow-based video CS Under a TV scene model, the matrix W is a discrete gradient has also been applied for the dynamic MRI problem, where operator that computes differences between adjacent pixels. carefully selected Fourier measurements provide robust initial 3-D TV video recovery can be achieved by stacking multiple scene estimates [3]. The concept of DSS sensing matrices has vectorized video frames into s and defining W to be the 3-D been improved recently by the sum-to-one (STOne) transform discrete gradient across both spatial dimensions and time. [35], which enables the fast recovery of low-resolution scene Optical flow constraints can be included by forming a sparse estimates at multiple resolutions. matrix W that differences pixels in one frame with pixels that lie along its flow trajectory in other frames. Video recovery techniques It can be shown that the solution to (1) corresponds to a While the mathematical formulations of video CS recovery saddle point of the so-called augmented Lagrangian function problems resemble other canonical sparse recovery problems, b three important factors set video recovery apart from other L(,,szm )=++-- f (UW s y ) gz () z s m 2 , (2) 2 2 types of sparse coding. First, video recovery problems are extremely large and have high memory requirements. where m is a vector of Lagrange multipliers. Constrained Methods for high-resolution video recovery must scale to hun- problems of the form (1) for CS video can be solved efficient- dreds of millions of unknowns. Second, sparse representations ly using the alternating direction method of multipliers of videos with complex structures may contain tens of thou- (ADMM) [13], [28], [31] or the primal-dual hybrid gradient sands (or more) of nonzero entries. Consequently, algorithm (PDHG) method [18], [29]. The ADMM and PDHG methods implementations that require large dense matrix systems are alternate between minimization steps for s and z and maxi- intractable, and methods must exploit fast transforms. Third, mization steps for m until convergence is reached. Such meth- high-quality video recovery often involves noninvertible spar- ods have the key advantage that they enable the inclusion of sity transforms, and so reconstruction methods that handle powerful, noninvertible video models such as 3-D TV or opti- cosparsity models are desirable. Some recovery problems cal flow. This advantage, however, comes at the cost of higher require more sophisticated (or unstructured) models, such as memory requirements and somewhat more complicated itera- optical flow constraints, that cannot be handled efficiently by tions. To improve the convergence rates of solvers for con- simple algorithms. All of these factors impact algorithm per- strained problems, accelerated algorithm variants have been formance on different reconstruction applications. developed [18], [32], [33]. This section overviews the range of existing recovery techniques and investigates the tradeoffs between reconstruction Unconstrained problems quality and computational complexity. For simplicity, we focus If the sparsity transform W is invertible, then the constraint in on two categories of reconstruction methods, variational and (1) can be removed by replacing the vector s with W-1 z . This

60 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

leads to the second category of recovery methods that solve sampling matching pursuit (CoSaMP) and subspace pursuit unconstrained problems of the following simpler form: [21], [60], constantly maintain a full support set of K nonzero entries but add strong and replace weak entries in an iterative t zfzygzt =+argmin ()().U (3) fashion. Parallel greedy pursuit algorithms have the advantage z that they can enforce structured models on the support set, t - Here, the matrix UUW= 1 and z contains the representation of such as a wavelet tree structure [5]. a single frame or the entire video in the sparsity transform domain. The main drawbacks of greedy algorithms, however, are For example, in the case of wavelet sparsity, solving (3) recovers that 1) they are typically unable to handle noninvertible spar- the video’s wavelet coefficients; the final video is obtained by sity transforms used for video reconstruction such as TV, opti- applying the inverse wavelet transform to the solution. cal flow, or overcomplete wavelet frames; 2) accurate solutions Unconstrained problems of the form (3) can be solved are guaranteed only when the measurement operator satisfies efficiently using forward–backward splitting (FBS) [20], stringent conditions (such as the restricted isometry property or fast iterative shrinkage/thresholding (FISTA) [10], fast adap- similar incoherence conditions [60], [72]); and 3) they require tive shrinkage/thresholding algorithm (FASTA) [34], sparse solving large linear systems on every iteration. For small num- reconstruction by separable approximation (SpaRSA) [81], or bers of unknowns (<10,000), the factorization of these systems approximate message passing (AMP) [25], [55]. FBS is the can be explicitly represented and updated cheaply using rank- most basic variant for solving unconstrained problems and one updates. For the large video CS problems considered here, performs the following two steps for the iterations k = 12,,f iterative (conjugate gradient) methods are recommended. These until reaching convergence: methods require only matrix multiplications (which can exploit fast transforms) and have lower memory requirements because + tt zztkkk1 =+x UU* d fzy(|) k and (4) they do not require the storage of large and dense matrices. kk++111 t 2 zgzzz=+-argmin ()2 , (5) z 2 Reconstruction quality versus computational complexity There are many choices to make when building a compressive where {}xk is some step size sequence. FBS finds a global video pipeline, including measurement operators, video mod- minimum of the objective function (3) by alternating between els, and reconstruction algorithms. Most reconstruction algo- the explicit gradient-descent step (4) in the function f and the rithms are restricted as to what measurement operators and proximal (or implicit gradient) step (5) in the function g. sparsity models they can support. To achieve the best perfor- The key operations of the gradient step (4) are matrix–vector mance, the reconstruction algorithms, video models, and data t t multiplications with U and U* . These multiplications can be acquisition pipelines must be designed jointly; this implies t carried out efficiently when U is a composition of fast trans- that there are tradeoffs to be made among reconstruction forms, such as subsampled Hadamard/Fourier matrices and speed, algorithm simplicity, and video quality. wavelet or DCT operators. When g is a simple sparsity-pro- The classical approach to CS video recovery is to search for moting regularizer, such as the ,1 norm, the proximal step (5) the video that is compatible with the observed measurements is easy to compute in closed form using wavelet shrinkage. while being as sparse as possible in the wavelet domain. When The computational complexity of FBS can be reduced signifi- an invertible wavelet transform is used, the reconstruction cantly using adaptive step-size rules for selecting {}xk , accel- problem can be transformed into an unconstrained problem of eration schemes, restart rules, momentum (or memory) terms, the form (3), which can be solved efficiently using variational and so forth, as is the case for FISTA, FASTA, SpaRSA, and methods such as FBS. If we further assume that the wavelet AMP. See the review article [34] for more details. transform is orthogonal, then we can use off-the-shelf greedy pursuit algorithms, such as CoSaMP. Unfortunately, while Greedy pursuit algorithms unconstrained optimization is simple to implement and highly Greedy pursuit algorithms are generally used for uncon- efficient, wavelet-based scene priors generally result in lower strained problems and iteratively construct a sparse set of reconstruction quality than noninvertible/redundant sparsity nonzero transform coefficients. Each iteration begins by models like TV. For this reason, we are often interested in con- identifying a candidate sparsity pattern for the unknown vec- strained solvers that interface with TV-based video models and tor z. Then, a least-squares problem is solved to minimize optical flow constraints. Ut - 2 zy2 , where z is constrained to have the prescribed To examine the associated performance/complexity trad- sparsity pattern. eoffs, we compare a variety of reconstruction methods using Existing greedy pursuit algorithms can be classified into the same measurement operator. A stream of 65,536 STOne sequential greedy pursuit algorithms and parallel greedy pur- measurements [35] was acquired from a 256 × 256 pixel suit algorithms. Sequential methods include orthogonal match- video having 16 frames. Videos were reconstructed separately ing pursuit (OMP), regularized OMP (ROMP), and stagewise using various models and solvers that were implemented in OMP (StOMP) [26], [61], [72]. These methods successively MATLAB. We consider unconstrained recovery using CoS- add more and more indices to the support set until a maximum aMP and FBS, which are restricted to using invertible regular- sparsity K is reached. Parallel methods, such as compressive izers. In the wavelet case, we consider 1) 2-D frame-by-frame

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 61

THE WORLD’S NEWSSTAND®

2-D Wavelet (CoSaMP) 2-D Wavelet (FBS) 3-D DCT (FBS) Original Video 752 Seconds 45 Seconds 332 Seconds

Adjoint 3-D Wavelet (FBS) Optical Flow (SPGL1) 3-D TV (PDHG) 0.001 Second 134 Seconds 415 Seconds 29 Seconds

FIGURE 3. A CS video recovery comparison with different video models. For each model, we recover a 16-frame video with 256 # 256 pixel resolution from 216 STOne transform measurements, corresponding to a 16:1 compression ratio. Sparsity models include 2-D (across space) and 3-D (across space and time) wavelet sparsity using the Haar wavelet, the 3-D DCT, optical flow constraints, and 3-D TV. For each experiment, we also provide the total runtime for recovering 16 frames.

recovery that does not exploit correlations across time, and efficacy of exploiting correlations across time. The key advantage 2) 3-D wavelet recovery that performs a 3-D wavelet transform of 2-D models is that they enable parallel frame-by-frame recon- across space and time. We also consider sparsity under the struction, for example, by dispatching different recovery prob- 3-D DCT, which is invertible and enjoys extensive use in lems on separate CPU cores. Finally, we see that for these types of image and video compression. We furthermore consider solv- large-scale reconstruction problems, variational methods require ers for constrained problems that handle more sophisticated substantially lower runtimes than greedy pursuit algorithms. The sparsity models. In particular, we compare 3-D TV models CoSaMP result in Figure 3 is for frame-by-frame reconstructions with PDHG and optical flow constraints with ADMM (as in with a sparsity level of K = 256 nonzero wavelet coefficients CS-MUVI [67]). As a baseline, we perform CS video recovery per image. CoSaMP’s runtime increases dramatically for larger without scene priors by simply computing UT y, the product K or when 3-D regularizers are used. This is because each itera- of the adjoint of the measurement operator with the vector tion requires the solution to a large least-squares problem using of measurements. Because the measurement operator is a multiple iterative (conjugate gradient) steps. Hence, such greedy subsampled orthogonal matrix, this corresponds to a least- pursuit algorithms turn out to be efficient only for highly sparse squares recovery using the pseudoinverse. All experiments signals and not for general CS video problems. are carried out on an off-the-shelf laptop with 16 GB memory and a 2.6 GHz i5 central processing unit (CPU) with two Perspectives and open research questions physical cores (no parallelism was used for reconstruction). The video CS problem has spawned a growing body of Sample frames from our experiments together with the research that spans signal representations and models, com- required runtime are shown in Figure 3. We observe that TV putational sensing architectures, and efficient optimization regularization and optical flow models dramatically outperform techniques. This has led to a vibrant ecosystem of methodol- wavelet-based recovery in terms of video quality. Furthermore, ogies that have transitioned the theoretical ideas of CS into 3-D models lead to significantly improved image quality with concrete application-specific concepts. We conclude by fewer artifacts than 2-D models, despite the fact that both recon- highlighting some of the important open questions and structions see the same amount of data. This demonstrates the future research directions.

62 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Real-time CS video recovery with today’s hardware High-quality CS video recovery requires complex algorithms Toward More Complex that include powerful video models. While offline video Algorithms and Video Models recovery is always feasible, reconstruction using more sophis- 1 G 1 E FLOPs ticated scene models (e.g., using optical flow) can easily take 256 × 256 1,920 × 1,080 several seconds to minutes even for only a few low-resolution at 12 fps at 25 fps frames. As a consequence, applications that necessitate real- time video recovery face extreme implementation challenges. From our experiments in Figure 3, we see that even the fastest 1 M 1 G FLO A algorithms with basic video models are more than 20 # to chievable FLOP/Pixel Throughput 1 # Ps P 200 below real time when executed in MATLAB on 1 T FLOPs F L off-the-shelf CPUs. O P Quite surprisingly, when counting the number of floating- 1 K s point operations (FLOPs) required for the main transforms of Toward Higher Resolution these methods, we observe that real-time CS video recovery 1 K 1 M 1 G 1 T with variational methods is within reach of existing hardware. Pixel/Second In fact, variational-based scene recovery of a 256 × 256 pixel Greedy Algorithms scene at 12 fps requires only about 20 GFLOPs, which is well Optical Flow Models below that of programmable processing hardware, such as Variational Methods CPUs, graphics-processing units (GPUs), and field-programmable gate arrays (FPGAs) that achieve peak throughputs FIGURE 4. The complexity (in FLOPs per pixel) versus resolution (in pixels of a few TFLOPs. Similarly, existing application specific per second) for greedy algorithms, variational methods, and optical-flow integrated circuit (ASIC) designs that target CS recovery models for the video scene in Figure 3. Variational methods (including problems [11], [53] are able to solve variational problems 3-D TV and 3-D/2-D wavelets) require the lowest complexity and enable with more than 200 GOPS (the computations are typically real-time CS video recovery with existing hardware (the diagonal dotted carried out with fixed-point arithmetic instead of floating line shows the FLOPs limit of current reprogrammable hardware). Optical flow models exceed the capabilities of current hardware and require the point) using low silicon area and low power when imple- development of more efficient computational methods and faster process- mented in modern CMOS technology nodes. In Figure 4, ing architectures. we compare the complexity versus the resolution of various CS video recovery methods. One can observe that even higher resolutions like 1080p HD are feasible in real time Third, as previously discussed at length, CS reconstruction with computationally efficient algorithms. Nevertheless, no algorithms have high computational complexity, and hence real-time CS video recovery implementation has been pro- avoiding a reconstruction step in the overall processing pipe- posed in the open literature, which can mainly be attributed line can be beneficial. to the lack of highly optimized and massively parallel CS There has been some limited work on inference from video recovery pipelines for programmable hardware (CPUs, linear compressive measurements. Davenport et al. [23] GPUs, or FPGAs) as well as dedicated integrated circuits perform compressive classification and detection by using a (ASICs). This is definitely a fruitful area for future work. matched filter in the compressive domain. Their key observation is that random projections preserve distances as well Compressive inference rather than recovery as inner products between sparse vectors; thus, inference The main results of CS are directed toward providing novel tasks like hypothesis testing and certain filtering opera- sampling theorems that determine the feasibility of signal tions can be performed directly in the compressive domain. reconstruction from an underdetermined set of linear measure- Hegde et al. [38] show that manifold learning (or nonlinear ments. However, reconstruction is often not the eventual goal dimensionality reduction) can be performed just as well on in most applications, which range from detection and classifi- the compressive measurements as on the original data, pro- cation to tracking and parameter estimation. While these tasks vided the data arises from a manifold with certain smooth- can all be performed postreconstruction (on the output of a ness properties. Sankaranarayanan et al. [66] demonstrate reconstruction procedure), there are important benefits to be that for time-varying systems well approximated as linear gained by performing them directly on the compressive mea- dynamical systems, the parameters of the dynamical sys- surements. First, tasks like detection, classification, and track- tem can be directly estimated given compressive measure- ing are inherently simpler than reconstruction—hence, there is ments. Recently, Kulkarni and Turaga [44] proposed a novel hope that we can perform them with fewer measurements. method based on recurrence textures for action recognition Second, CS reconstruction is intrinsically tied to the signal from compressive cameras especially for self-similar fea- models used for the unknown signal, and these signal models ture sequences [43]. Apart from these early attempts, there prioritize features that deal with visual perception, which often is very little in the literature exploring high-level inference is not the most relevant for the subsequent processing tasks. from compressive imagers.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 63

THE WORLD’S NEWSSTAND®

A major hurdle to successful compressive inference in the Authors video context is the mismatch between part-based models, used Richard G. Baraniuk ([email protected])______received his B.S. in computer vision, and global random embeddings, the corner- degree from the University of Manitoba, Canada, in 1987; stone of the CS theory. Part-based models have had remarkable his M.S. degree from the University of Wisconsin-Madison success over the past decade in object detection and classifica- in 1988; and his Ph.D. degree from the University of tion problems. The key enabler of part-based inference is a local Illinois at Urbana-Champaign in 1992. He is the Victor E. feature description that helps isolate objects from background Cameron Professor of Electrical and Computer Engineering clutter and provides robustness against object variations. How- at Rice University, Houston, Texas, and the founder and

ever, the conventional CS measurements are dense random director of OpenStax (openstax.org).______His research interests projections that are not conducive to local feature extraction include new theory, algorithms, and hardware for sensing, without reconstructing the signal first. Hence, there is an urgent signal processing, and machine learning. He is a Fellow of need for CS measurement operator designs that enable local the IEEE and the American Association for the feature extraction. Advancement of Science and has received national young investigator awards from the National Science Foundation From measurements to bits—Toward nonlinear and the Office of Naval Research; the Rosenbaum sensing architectures Fellowship from the Isaac Newton Institute of Cambridge One of the important distinctions between video CS and video University, United Kingdom; the Electrical and Computer compression is the nature of representing the compressed data. Engineering Young Alumni Achievement Award from the Compression aims to reduce the number of bits used to repre- University of Illinois at Urbana–Champaign; the IEEE sent the video. In contrast, CS measurements are typically rep- Signal Processing Society Best Paper, Best Column, resented in terms of real values with infinite (or arbitrarily Education, and Technical Achievement Awards; and the large) precision; here, the number of actual measurements is IEEE James H. Mulligan, Jr. Medal. the criterion to reduce/optimize. The focus on reducing the Tom Goldstein ([email protected])______received his B.A. number of measurements is often misplaced in many sensing degree in mathematics from Washington University, St. scenarios; for example, in high-speed video CS, the bottleneck Louis, Missouri, in 2006 and his Ph.D. degree in mathematics is solely due to the operating speed of the ADC, whose perfor- from the University of California, Los Angeles, in 2010. He mance is measured in the number of bits acquired per second. has been a visiting research scientist with Stanford Hence, compressively sensing while respecting the bottlenecks University, California, and Rice University, Houston, Texas. imposed by the ADC sampling frequency requires us to con- He is currently an assistant professor of computer science at sider measurements in terms of bits. While there has been the University of Maryland, College Park. His research inter- some effort in the area of 1-bit CS [4], [42], [63] and the trad- ests include numerical optimization, distributed computing, eoff between measurement bits and measurement rate [48], image processing, and machine learning.

this aspect is still largely unexplored in literature. In particular, Aswin C. Sankaranarayanan ([email protected])______there is a need for new kinds of nonlinear sensing architectures received his B.S. degree in electrical engineering from the that optimize system performance in the context of the practi- Indian Institute of Technology, Madras, in 2003 and his Ph.D. cal realities of sensing (quantization, saturation, etc.). Some degree from the University of Maryland, College Park, where initial progress in this direction for CS has been made in [59], he was awarded the distinguished dissertation fellowship by but the area remains wide open for research. the Department of Electrical and Computer Engineering (ECE) in 2009. He was a postdoctoral researcher in the Digital Signal Acknowledgments Processing group at Rice University, Houston, Texas. He is We thank David Robert Jones for his invaluable suggestions currently an assistant professor in the ECE Department at and Doug Jones for the JAM. Richard G. Baraniuk was sup- Carnegie Mellon University, Pittsburgh, Pennsylvania. His ported by National Science Foundation (NSF) grants CCF- research encompasses problems in compressive sensing and 1527501 and CCF-1502875, Defense Advanced Research computational imaging. He has received best paper awards at Projects Agency (DARPA) Revolutionary Enhancement of the Computer Vision and Pattern Recognition Workshops on Visibility by Exploiting Active Light-fields grant HR0011- Computational Cameras and Displays (2015) and Analysis and 16-C-0028, and Office of Naval Research (ONR) grant Modeling of Faces and Gestures (2010). N00014-15-1-2735. Tom Goldstein was supported by NSF Christoph Studer ([email protected])______received his M.S. grant CCF-1535902 and ONR grant N00014-15-1-2676. Aswin and Ph.D. degrees from ETH Zurich, Switzerland, in 2005 and C. Sankaranarayanan was supported by NSF grant IIS-1618823 2009, respectively. He has been a postdoctoral student and and Army Research Office grant W911NF-16-1-0441. research scientist at ETH Zurich and Rice University, Houston, Christoph Studer was supported in part by Xilinx Inc. and by Texas, and is currently an assistant professor in the School of NSF grants ECCS-1408006 and CCF-1535897. Ashok Electrical and Computer Engineering, Cornell University, Veeraraghavan was supported by NSF grant CCF-1527501. Ithaca, New York. His research interests are at the intersection Michael B. Wakin was supported by NSF CAREER grant CCF- of digital VLSI circuit and system design, signal and image 1149225 and grant CCF-1409258. processing, and wireless communication.

64 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Ashok Veeraraghavan ([email protected])______received his [9] R. Basri and D. W. Jacobs, “Lambertian reflectance and linear subspaces,” IEEE B.S. degree in electrical engineering from the Indian Institute Trans. Pattern Anal. Mach. Intell., vol. 25, no. 2, pp. 218–233, 2003. [10] A. Beck and M. Teboulle, “Fast gradient-based algorithms for constrained total of Technology, Madras, in 2002 and his M.S. and Ph.D. variation image denoising and deblurring problems,” IEEE Trans. Image Process., degrees from the Department of Electrical and Computer vol. 18, no. 11, pp. 2419–2434, 2009. Engineering, University of Maryland, College Park, in 2004 [11] D. E. Bellasi, L. Bettini, C. Benkeser, T. Burger, Q. Huang, and C. Studer, “VLSI design of a monolithic compressive-sensing wideband analog-to-informa- and 2008, respectively. He is an assistant professor of electri- tion converter,” IEEE Trans. Emerg. Sel. Topics Circuits Syst., vol. 3, no. 4, pp. cal and computer engineering at Rice University, Houston, 552–565, 2013. Texas, where he directs the Computational Imaging and [12] T. Blumensath and M. E. Davies, “Iterative hard thresholding for compressed Vision Lab. His research interests are broadly in the areas of sensing,” Appl. Comput. Harmon. Anal., vol. 27, no. 3, pp. 265–274, 2009. [13] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimiza- computational imaging, computer vision, and robotics. tion and statistical learning via the alternating direction method of multipliers,” Before joining Rice University, he spent three years as a Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1–122, 2011. research scientist at Mitsubishi Electric Research Labs in [14] E. Candès and M. Wakin, “An introduction to compressive sampling,” IEEE Cambridge, Massachusetts. His work has received numerous Signal Process. Mag., vol. 25, no. 2, pp. 21–30, Mar. 2008. [15] E. J. Candès and D. L. Donoho, “New tight frames of curvelets and optimal awards, including the doctoral dissertation award from the representations of objects with piecewise c2 singularities,” Commun. Pure Appl. Department of Electrical and Computer Engineering at the Math., vol. 57, no. 2, pp. 219–266, 2004. University of Maryland, the Hershel M. Rich Invention [16] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Award from Rice University, and the Best Poster Runner-Up Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006. Award from the 2014 International Conference on [17] V. Cevher, A. C. Sankaranarayanan, M. F. Duarte, D. Reddy, R. G. Baraniuk, Computational Photography. and R. Chellappa, “Compressive sensing for background subtraction,” in Proc. European Conf. Computer Vision, Marseille, France, 2008, pp. 155–168. Michael B. Wakin ([email protected])______received his [18] A. Chambolle and T. Pock, “A first-order primal-dual algorithm for convex B.S. degree in electrical engineering and his B.A. degree in problems with applications to imaging,” J. Mathematical Imaging Vision, vol. 40, mathematics in 2000, his M.S. degree in electrical engineer- no. 1, pp. 120–145, 2011. ing in 2002, and his Ph.D. degree in electrical engineering [19] H. Chen, S. Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. in 2007 from Rice University, Houston, Texas. He is the IEEE Conf. Computer Vision and Pattern Recognition, Boston, MA, 2015, Ben L. Fryrear associate professor in the Department of pp. 2358–2366. Electrical Engineering and Computer Science at the [20] P. L. Combettes and J.-C. Pesquet, “Proximal splitting methods in signal processing,” in Fixed-Point Algorithms for Inverse Problems in Science and Colorado School of Mines (CSM), Golden. He was a Engineering. New York: Springer-Verlag, 2011, pp. 185–212. National Science Foundation (NSF) mathematical sciences [21] W. Dai and O. Milenkovic, “Subspace pursuit for compressive sensing recon- postdoctoral research fellow at the California Institute of struction,” IEEE Trans. Inf. Theory, vol. 55, no. 5, pp. 2230–2249, 2009. Technology, Pasadena, in 2006–2007 and an assistant pro- [22] M. Davenport, J. Laska, J. R. Treichler, and R. G. Baraniuk, “The pros and cons of compressive sensing for wideband signal acquisition: Noise folding versus fessor at the University of Michigan, Ann Arbor, in 2007– dynamic range,” IEEE Trans. Signal Process., vol. 60, no. 9, pp. 4628–4642, 2008. His research interests include sparse, geometric, and 2012. manifold-based models for signal processing and compres- [23] M. A. Davenport, P. T. Boufounos, M. B. Wakin, and R. G. Baraniuk, “Signal processing with compressive measurements,” IEEE J. Sel. Topics Signal Process., sive sensing. He has received the NSF CAREER Award, the vol. 4, no. 2, pp. 445–460, 2010. Defense Advanced Research Projects Agency Young Faculty [24] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no. 4, Award, and the CSM Excellence in Research Award for his pp. 1289–1306, Apr. 2006. research as a junior faculty member. [25] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” Proc. Natl. Acad. Sci. U.S.A., vol. 106, no. 45, pp. 18914– 18919, 2009. References [26] D. L. Donoho, Y. Tsaig, I. Drori, and J.-L. Starck, “Sparse solution of underde- [1] M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing termined systems of linear equations by stagewise orthogonal matching pursuit,” overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process., IEEE Trans. Inf. Theory, vol. 58, no. 2, pp. 1094–1121, 2012. vol. 54, no. 11, pp. 4311–4322, 2006. [27] M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, [2] M. S. Asif, A. Ayremlou, A. Sankaranarayanan, A. Veeraraghavan, and R. and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Baraniuk, “Flatcam: Thin, lensless cameras using coded aperture and computation,” Process. Mag., vol. 25, no. 2, pp. 83–91, Mar. 2008. IEEE Trans. Comput. Imag., to be published. DOI: 10.1109/TCI.2016.2593662 [28] J. Eckstein and D. P. Bertsekas, “On the Douglas–Rachford splitting method [3] M. S. Asif, L. Hamilton, M. Brummer, and J. Romberg, “Motion-adaptive spa- and the proximal point algorithm for maximal monotone operators,” Mathematical tio-temporal regularization for accelerated dynamic MRI,” Magn. Reson. Medicine, Programming, vol. 55, nos. 1–3, pp. 293–318, 1992. vol. 70, no. 3, pp. 800–812, 2013. [29] E. Esser, X. Zhang, and T. F. Chan, “A general framework for a class of first [4] R. Baraniuk, S. Foucart, D. Needell, Y. Plan, and M. Wootters, “Exponential order primal-dual algorithms for convex optimization in imaging science,” SIAM J. decay of reconstruction error from binary measurements of sparse signals,” arXiv Imag. Sci., vol. 3, no. 4, pp. 1015–1046, 2010. preprint arXiv:1407.8246, 2014. [30] J. E. Fowler, S. Mun, E. W. Tramel, M. R. Gupta, Y. Chen, T. Wiegand, and H. [5] R. G. Baraniuk, “Optimal tree approximation with wavelets,” Proc. SPIE, vol. Schwarz, “Block-based compressed sensing of images and video,” Found. Trends 3813, pp. 196–207, 1999. Signal Processing, vol. 4, no. 4, pp. 297–416, 2010. [6] R. G. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag., vol. 24, [31] R. Glowinski and P. Le Tallec, Augmented Lagrangian and Operator-Splitting no. 4, pp. 118–121, 2007. Methods in Nonlinear Mechanics. Philadelphia, PA: SIAM, 1989. [7] R. G. Baraniuk. (2015). Compressive nonsensing, Norbert Weiner Lecture, Univ. [32] T. Goldstein, M. Li, and X. Yuan, “Adaptive primal-dual splitting methods for of Maryland. [Online]. Available: http://www.norbertwiener.umd.edu/______statistical learning and image processing,” in Advances in Neural Information FFT/2015/15-TAs/baraniuk.html______Processing Systems 28, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, Eds. Red Hook, NY: Curran Associates, Inc., 2015, pp. 2080–2088. [8] R. G. Baraniuk, M. Davenport, R. DeVore, and M. Wakin. “A simple proof of the restricted isometry property for random matrices,” Constr. Approx., vol. 28, no. [33] T. Goldstein, B. O’Donoghue, S. Setzer, and R. Baraniuk, “Fast alternating direc- 3, pp. 253–263, Dec. 2008. tion optimization methods,” SIAM J. Imag. Sci., vol. 7, no. 3, pp. 1588–1623, 2014.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 65

THE WORLD’S NEWSSTAND®

[34] T. Goldstein, C. Studer, and R. Baraniuk, “A field guide to forward–backward [58] K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field splitting with a fasta implementation,” arXiv preprint arXiv:1411.3406, 2014. photography using overcomplete dictionaries and optimized projections,” ACM Trans. Graph., vol. 32, no. 4, p. 46, 2013. [35] T. Goldstein, L. Xu, K. F. Kelly, and R. G. Baraniuk, “The STOne transform: Multi-resolution image enhancement and real-time compressive video,” arXiv pre- [59] A. Mousavi, A. Patel, and R. G. Baraniuk, “A deep learning approach to struc- print arXiv:1311.3405, 2013. tured signal recovery,” in Proc. 53rd Annu. Allerton Conf. Communication, Control, and Computing, Monticello, IL, 2015. [36] J. Gu, Y. Hitomi, T. Mitsunaga, and S. Nayar, “Coded rolling shutter photography: Flexible space-time sampling,” in Proc. 2010 IEEE Int. Conf. Computational [60] D. Needell and J. A. Tropp, “CoSaMP: Iterative signal recovery from incom- Photography, Cambridge, MA, pp. 1–8. plete and inaccurate samples,”. Appl. Comput. Harmon. Anal., vol. 26, no. 3, pp. 301–321, Aug. 2009. [37] Z. T. Harmany, R. F. Marcia, and R. M. Willett, “Compressive coded aperture keyed exposure imaging with optical flow reconstruction,” arXiv preprint [61] D. Needell and R. Vershynin, “Uniform uncertainty principle and signal recov- arXiv:1306.6281, 2013. ery via regularized orthogonal matching pursuit,” Found. Comp. Math., vol. 9, no. 3, pp. 317–334, 2009. [38] C. Hegde, M. Wakin, and R. Baraniuk, “Random projections for manifold learning,” in Proc. 22nd Annu. Conf. Neural Information Processing Systems, [62] J. Y. Park and M. B. Wakin, “Multiscale algorithm for reconstructing videos Vancouver, Canada, 2008, pp. 641–648. from streaming compressive measurements,” J. Electronic Imaging, vol. 22, no. 2, p. 021001, 2013. [39] Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proc. [63] R. Saab, R. Wang, and O. Yilmaz, “From compressed sensing to compressed 2011 Int. Conf. Computer Vision, Barcelona, Spain, pp. 287–294. bit-streams: Practical encoders, tractable decoders,” arXiv preprint arXiv:1604.00700, 2016. [40] J. Holloway, A. C. Sankaranarayanan, A. Veeraraghavan, and S. Tambe, “Flutter shutter video camera for compressive sensing of videos,” in Proc. 2012 [64] R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion IEEE Int. Conf. Computational Photography, Seattle, WA, pp. 1–9. deblurring using fluttered shutter,” ACM Trans. Graph., vol. 25, no. 3, pp. 795–804, 2006. [41] G. Huang, H. Jiang, K. Matthews, and P. Wilford, “Lenless imaging by compressive sensing,” in Proc. Int. Conf. Image Processing, Melbourne, Australia, [65] D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2C2: Programmable pixel 2013, pp. 2101–2105. compressive camera for high speed imaging,” in Proc. 2011 IEEE Conf. Computer Vision and Pattern Recognition, Colorado Springs, CO, pp. 329–336. [42] L. Jacques, J. N. Laska, P. T. Boufounos, and R. G. Baraniuk, “Robust 1-bit compressive sensing via binary stable embeddings of sparse vectors,” IEEE Trans. [66] A. C. Sankaranarayanan, P. K. Turaga, R. Chellappa, and R. G. Baraniuk, Inf. Theory, vol. 59, no. 4, pp. 2082–2102, 2013. “Compressive acquisition of linear dynamical systems,” SIAM J. Imag. Sci., vol 6, no. 4, pp. 2109–2133, 2013. [43] I. Junejo, E. Dexter, I. Laptev, and P. Perez, “View-independent action recognition from temporal self-similarities,” IEEE Trans. Pattern Anal. Mach. Intell., vol. [67] A. C. Sankaranarayanan, L. Xu, C. Studer, Y. Li, K. F. Kelly, and R. G. 33, no. 1, pp. 172 –185, 2011. Baraniuk, “Video compressive sensing for spatial multiplexing cameras using motion-flow models,” SIAM J. Imag. Sci., vol. 8, no. 3, pp. 1489–1518, 2015. [44] K. Kulkarni and P. Turaga, “Recurrence textures for activity recognition using compressive cameras,” in Proc. 2012 19th IEEE Int. Conf. Image Processing, Lake [68] A. Secker and D. Taubman, “Lifting-based invertible motion adaptive trans- Buena Vista, FL, pp. 1417–1420. form (LIMAT) framework for highly scalable video compression,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1530–1542, 2003. [45] L.-W. Kang and C.-S. Lu, “Distributed compressive video sensing,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, Taiwan, China, 2009, [69] P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. pp. 1169–1172. Lensch, “Dual photography,” ACM Trans. Graph., vol. 24, no. 3, pp. 745–755, 2005. [46] K. Kelly, R. Baraniuk, L. McMackin, R. Bridge, S. Chatterjee, and T. Weston, [70] P. Sen and S. Darabi, “Compressive dual photography,” Comput. Graphics “Decreasing image acquisition time for compressive imaging devices,” U.S. Patent Forum, vol. 28, no. 2, pp. 609–618, 2009. 8,860,835, Oct. 14, 2014. [71] S. Tambe, A. Veeraraghavan, and A. Agrawal, “Towards motion aware light [47] R. Koller, L. Schmid, N. Matsuda, T. Niederberger, L. Spinoulas, O. field video for dynamic scenes,” in Proc. 2013 Int. Conf. Computer Vision, Sydney, Cossairt, G. Schuster, and A. K. Katsaggelos, “High spatio-temporal resolution Australia, pp. 1009–1016. video with compressed sensing,” Opt. Express, vol. 23, no. 12, pp. 15992–16007, [72] J. Tropp et al., “Greed is good: Algorithmic results for sparse approximation,” 2015. IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231–2242, 2004. [48] J. Laska and R. G. Baraniuk, “Regime change: Bit-depth versus measurement- [73] N. Vaswani, “Kalman filtered compressed sensing,” in Proc. 15th IEEE Int. rate in compressive sensing,” IEEE Trans. Signal Process., vol. 60, no. 7, pp. 3496– Conf. Image Processing, San Diego, CA, 2008, pp. 893–896. 3505, 2012. [74] N. Vaswani and W. Lu, “Modified-cs: Modifying compressive sensing for [49] Y. Le Montagner, E. Angelini, and J.-C. Olivo-Marin, “Video reconstruction problems with partially known support,” IEEE Trans. Signal Process., vol. 58, no. using compressed sensing measurements and 3d total variation regularization for 9, pp. 4595–4607, 2010. bio-imaging applications,” in Proc. 2012 19th IEEE Int. Conf. Image Processing, Lake Buena Vista, FL, pp. 917–920. [75] A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic events,” IEEE Trans. Pattern Anal. [50] C. Li, T. Sun, K. F. Kelly, and Y. Zhang, “A compressive sensing and unmixing Mach. Intell., 33(4):671–686, Apr. 2011. scheme for hyperspectral data processing,” IEEE Trans. Image Process., vol. 21, no. 3, pp. 1200–1210, 2012. [76] A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt., vol. 47, no. 10, pp. [51] P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. 44–51, 2008. Brady, “Coded aperture compressive temporal imaging,” Opt. exp., vol. 21, no. 9, pp. 10526–10545, 2013. [77] M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. G. Baraniuk, “Compressive imaging for video representation and coding,” in [52] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed sensing Proc. Picture Coding Symposium, Beijing, China, 2006. MRI,” IEEE Signal Process. Mag., vol. 25, no. 2, pp. 72–82, 2008. [78] J. Wang, M. Gupta, and A. C. Sankaranarayanan, “LiSens—a scalable archi- [53] P. Maechler, C. Studer, D. E. Bellasi, A. Maleki, A. Burg, N. Felber, H. tecture for video compressive sensing,” in Proc. 2015 IEEE Int. Conf. Kaeslin, and R. G. Baraniuk, “VLSI design of approximate message passing for Computational Photography, Houston, TX, pp. 1–9. signal restoration and compressive sensing,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 2, no. 3, pp. 579–590, 2012. [79] A. E. Waters, A. C. Sankaranarayanan, and R. G. Baraniuk, “SpaRCS: Recovering low-rank and sparse matrices from compressive measurements,” in Proc. [54] A. Mahalanobis, R. Shilling, R. Murphy, an d R. Muise, “Recent results of Neural Information Processing Systems, Granada, Spain, 2011, pp. 1089–1097. medium wave infrared compressive sensing,” Appl. Opt., vol. 53, no. 34, pp. 8060– 8070, 2014. [80] J. Wright, A. Ganesh, K. Min, and Y. Ma, “Compressive principal component pursuit,” Information and Inference, vol. 2, no. 1, pp. 32–68, 2013. [55] M. A. Maleki, “Approximate message passing algorithms for compressed sensing,” Ph.D. dissertation, Stanford Univ., Stanford, CA, 2010. [81] S. J. Wright, R. D. Nowak, and M. A. Figueiredo, “Sparse reconstruction by separable approximation,” IEEE Trans. Signal Process., vol. 57, no. 7, [56] S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way, 3rd ed. New pp. 2479–2493, 2009. York: Academic Press, 2008. [82] L. Ying, L. Demanet, and E. Candes, “3D discrete curvelet transform,” Proc. [57] R. Marcia and R. M. Willett, “Compressive coded aperture video reconstruc- SPIE, vol. 5914, p. 591413, 2005. tion,” in Proc. 2008 16th European Signal Processing Conf., Lausanne, Switzerland, pp. 1–5. SP

66 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Ruhi Sarikaya The Technology Behind Personal Digital Assistants An overview of the system architecture and key components

BACKGROUND IMAGE ©ISTOCKPHOTO.COM/ TRAFFIC_ANALYZER; PDA IMAGE ©ISTOCKPHOTO.COM/HUDIEMM

e have long envisioned that one day computers will and structured data served by applications and content provid- understand natural language and anticipate what we ers has emerged. These advances, along with increased com- Wneed, when and where we need it, and proactively putational power, have broadened the application of natural complete tasks on our behalf. As computers get small- LU to a wide spectrum of everyday tasks that are central to a er and more pervasive, how humans interact with them is user’s productivity. We believe that as computers become becoming a crucial issue. Despite numerous attempts over the smaller and more ubiquitous [e.g., wearables and Internet of past 30 years to make language understanding (LU) an effec- Things (IoT)], and the number of applications increases, both tive and robust natural user interface for computer interaction, system-initiated and user-initiated task completion across vari- success has been limited and scoped to applications that were ous applications and web services will become indispensable not particularly central to everyday use. However, speech rec- for personal life management and work productivity. In this ognition and machine learning have continued to be refined, article, we give an overview of personal digital assistants (PDAs); describe the system architecture, key components, and technology behind them; and discuss their future Digital Object Identifier 10.1109/MSP.2016.2617341 Date of publication: 11 January 2017 potential to fully redefine human–computer interaction.

THE WORLD’S NEWSSTAND®

Introduction PDAs We are living in the mobile Internet computing cycle. During the past decade, mobile devices have experienced unprece- What is a PDA? dented growth. According to Statista [65], there are currently A PDA is a metalayer of intelligence that sits on top of other more than 4.6 billion mobile phone users in the world, and services and applications and performs actions using these ser- the number is expected to grow even more moving forward. vices and applications to fulfill the user’s intent. A user’s intent With this phenomenal increase in volume came technical could be explicit, where the user commands the system to per- sophistication and improved capabilities of mobile devices form an action, or it could be inferred, where the agent notifies (both on the hardware and software sides), particularly or makes suggestions upon evaluation of one or more trigger- around applications and web services where users can coming conditions it has been tracking. PDAs make use of some plete a wide array of tasks. As the need and expectation to do core set of technologies, such as machine learning, speech rec- more grew, despite improvements, a limited natural user ognition, LU, question answering (QA), dialog management interface has remained as one of the major bottlenecks in (DM), language generation (LG), text-to-speech (TTS) synthe- interacting with these devices. PDAs (also known as virtual sis, data mining, analytics, inference, and personalization. assistants) precisely target this problem and have the promise of enhancing a user’s productivity by either proactively pro- Why do we need PDAs? viding the information the user needs in the right context PDAs are built to help the user get things done (e.g., setting up (i.e., time and place) or reactively answering a user’s ques- an alarm/reminder/meeting, taking notes, creating lists) and pro- tions and completing tasks through natural language. Tasks vide easy access to personal/external structured data, web ser- can be related to device functionality, vices, and applications (e.g., finding the applications, or web services. user’s documents, locating a place, making Research on PDA technology, how- PDAs have the promise reservations, playing music). They also ever, started much earlier than the of enhancing a user’s assist the user in his or her daily schedule emergence of mobile devices. Over the productivity by either and routine by serving notifications and last 20 years, researchers have investi- proactively providing alerts based on contextual information, gated personalized virtual assistant agents the information the such as time, user location, and feeds/infor- targeting specific domains, including user needs in the mation produced by various web services, tourism, elder care, device control, and given the user’s interests (e.g., commute home and office applications [1]–[5]. right context or alerts to/from work, meeting reminders, However, attempts at bringing them to reactively answering a concert suggestions). Collectively, these market earlier have failed because of user’s questions and functionalities are expected to make the their limited utility. completing tasks through user more productive in managing his or Over the past five years, there has natural language. her work and personal life. been tremendous investment in PDA For example, airline travel is a com- technology by both small and big tech- monly supported scenario by most PDAs. nology companies. Siri [17], [66], Google Now [67], Cor- If the user has booked a flight and received a confirmation tana [68], and Alexa [69] are the major personal assistants e-mail along with an itinerary, the PDA scans the e-mail, in the market today, and they provide proactive and/or reac- extracts the flight information, and stores it on the service. On the tive assistance to the user. Proactive assistance refers to the day of travel, the PDA computes the user’s current location using agent taking an action to assist the user without the user’s the global positioning system (GPS) on the device, checks the explicit request. Reactive assistance refers to the agent traffic conditions to the airport, and tells the user when to leave responding to the user’s voice or typed command to assist for the airport. It also checks the flight status and updates the him or her. The number of smartphone users using PDAs user if there is a delay, using a flight card as shown in Figure 1. increased from 30% in 2013 to 65% in 2015 [70], indicating Additionally, it provides weather forecasts for the destination as increased adoption. well as currency conversion rates. Typically, a user has to use PDAs have become a key capability in most smartphones. multiple applications to go through each of these steps to find out They are now also deployed in tablets, laptops, desktop PCs, the needed information that is listed on the cards for the travel. and headless devices (e.g., Amazon Echo), and some are also None of these atomic steps is significant in isolation, but stitch- even integrated into operating systems. These agents are ing them together can potentially mark a breakthrough in use- designed to be personal; they know their user’s profile, where- fulness to the user. This is the key promise of PDAs. abouts, schedules, and so forth. They can proactively start interactions with their user through notifications and system- What is personal about PDAs? initiated questions or reactively respond to user requests. User– PDAs are expected to be personal. Ideally, the PDA is expect- PDA interactions typically take place via natural language, ed to know who its user is, what its user does, its user’s inter- where the user speaks to the agent as if he or she were speaking ests, what its user needs, and when and where its user needs to a real human assistant. it. Despite numerous efforts over the past 20 years to make

68 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

human–computer interaction personal (particularly around web search), personalization has remained largely broken until recently, not only for PDAs but also for general human– computer interaction due to the four main gaps. 1) Data: The system had a limited amount of data to properly model the user and his or her interests. It understands the user based on the experience it provides and the feedback loop it uses. 2) Computing: The limited computing power and machine learning were not adequate for modeling the complexity of user behavior. 3) Interest: There has been conflict of interest between the user, the platform, and those who pay for the user’s attention. The system has not been necessarily prioritizing the user’s interests over these other actors. 4) Content/action: The system does not support the actions the user wants to perform or does not have the content to serve the user’s interests. (a) (b) During the past seven years, two primary changes have occurred that allowed PDAs to be personal: 1) the increased FIGURE 1. The proactive flight cards. (a) Summary and suggestions for the number of device sensors on mobile phones and 2) the quality trip. (b) Flight details for the first leg. and quantity of the user data and digital artifacts (on the device and/or in the cloud) coming from the web services and applica- opens up new solutions to already existing problems [6]. User tions the user accesses. As a result of these changes, it is now experiences that currently exist can also be enhanced by the possible to represent a user along four axes: available data. For example, using activity detection, the PDA 1) user profile: user’s name, age, gender, parental status, pro- can hold the incoming call or send a short message service fession, employer, home, work, people in his or her close (SMS) text message to the caller if the user is biking or it can circle (people graph), favorite places, files, documents, turn up the volume if the user is climbing stairs/walking. music, photos, and interests explicitly provided by the user The three main mobile platforms (Android, iPhone operat- 2) digital activity: digital artifacts (e.g., calendar, e-mail, ing system, Windows) support four broad categories of sensors social media activity, web searches) on applications and on mobile devices. web services 1) Motion: This sensor set includes accelerometers, gravity 3) space: physical location of the user sensors, gyroscopes, and rotational vector sensors. They 4) time: time at which a specific digital or physical activity measure acceleration and rotational forces along three axes. takes place. They measure movement and orientation of the device. These four dimensions, when considered together, blend the 2) Environmental: These sensors measure various environmental physical with the digital world and open up new possibilities for conditions, such as ambient air temperature and pressure, illu- powerful inferences and deep user understanding. Inevitably, mination, and humidity. This category includes barometers, managing and protecting privacy and security of the user data photometers, magnetometers, and thermometers. and information is a major concern, and what has been done 3) Position and location: These sensors measure the physical in that space is critical, but it is outside the scope of this article. position and location of a device. This category includes orientation sensors and magnetometers. The magnetometer Mobile device sensors can determine the rotation of the device relative to magnet- The computational power and capabilities of mobile phones ic north. It can also detect magnetic fields around the are increasing every year. The number of built-in sensors on device. GPS and Wi-Fi (not really a sensor in the tradition- smartphones (e.g., Samsung Galaxy) more than tripled during al sense) determine the location of the device. the past five years [71]. Smartphone sensors measure motion/ 4) Proximity: This sensor detects whether the phone is brought orientation, GPS coordinates, and many other user and envi- near the face during a phone call. This functionality disables ronmental conditions. For example, a device’s gravity sensor the touch screen, preventing inadvertent input to the phone provides data to infer complex user gestures and motions, from the user’s face and can also save battery power. such as shake, swing, or rotation. The rich high-precision data coming out of these sensors are made available through appli- System architecture cation programming interfaces (APIs) and are used in numer- The scenarios that the PDAs support can be divided into two ous applications and scenarios. The information is sensitive, main categories: 1) proactive and 2) reactive assistance. The as it is personal and contextual. Making it available opens up conceptual agent architecture designed to support these two new research areas like fitness and health applications or modes of assistance is shown in Figure 2. The system

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 69

THE WORLD’S NEWSSTAND®

is to provide assistance to automate Reactive Assistance Proactive Assistance tasks or further the user’s interests for things he or she cares about, all within context, without explicit user request [8]. “Book_ Taxi” User Experience “Restaurant Suggestions” To achieve that, the agent is designed to possess a set of attributes; it should be valuable in that it advances the user’s Data Reactive Assistance Proactive Assistance interests and tasks, while not interfering Back-End Databases, ASR, LU, Dialog, Inferences, User with the user’s own activities or atten- Services, and Client LG, TTS Modeling, Suggestions tion unless it has the user’s explicit Signals approval. It should be unimposing. The agent should be transparent in what it knows about the user. It should be antic- Device/Service End Points (Phone, PC, Xbox, Web Browser, Messaging Applications) ipatory and know the future needs of the user and bring opportunities to the sur- (a) (b) face. The agent should also continuously learn and refine its decisions from the FIGURE 2. The personal digital agent architecture for (a) reactive assistance and (b) anticipatory computing. feedback signals it receives regarding the actions it takes. These principles put the user at the center, and the agent’s architecture depicts proactive and reactive user experiences, actions are considered valuable only if they ultimately add value data, and service end points. Reactive assistance is shown in for the user. Proactive assistance operates on the proactivity Figure 2(a), where the user issues an explicit natural language continuum [31], which ranges from zero to full automation, command (e.g., “book me a taxi”) to the agent. The user request allowing for the following scenarios: is handled through a set of reactive assistance components, ■ do it yourself (no help from the agent) such as speech recognition, LU, and DM. The data coming ■ user tells the agent what to pay attention to (notifications/ from various back ends, and applications are served to the user alerts) according to the constraints specified in the natural language ■ agent infers user’s habits/patterns and makes suggestions query. The experience (reactive and/or proactive) can be served (inference/suggestions) in one or more of the different device or service end points. ■ agent makes decisions and takes actions (full autonomy on Proactive assistance [Figure 2(b)] involves anticipatory task decisions/executions). computing, where the personal digital agent does things in Most of the currently supported proactive scenarios are a contextual manner (i.e., at the right time and place) that it notifications/alerts and suggestions. Even though there is some expects is valuable to the user without an explicit user request. preliminary work, none of the agents in production supports Proactive assistance makes use of inference, user modeling, autonomous decision making and action taking on behalf of and ranking to power experiences. Back- the user without confirmation. end data, device, applications, and web Even though proactive The proactive agent system architecture services signals are leveraged for proactive and reactive parts of the is shown in Figure 3. Signals coming from inference and triggering. web services, device sensors, and the user’s Even though proactive and reactive parts current PDA architectures profile are processed, where processing of the current PDA architectures are built in are built in isolation, in includes parsing, enriching, and filtering isolation, in principle they can use a single principle they can use to merge device and service data. The next architecture to enable both types of experi- a single architecture step is aggregation, which joins the pro- ences. In fact, most proactive scenarios have to enable both types of cessed data streams through time and space reactive extensions and vice versa. For exam- experiences. (i.e., location) about the user’s whereabouts ple, if the user makes a restaurant reservation and actions/tasks done at specific times and (reactively), the agent may (proactively) sug- places. This step blends the physical and gest a movie after the dinner or may offer to book a cab to digital worlds and allows for powerful inferences that capture take the user to the restaurant. Data and context are shared repetitive behavior and events in both worlds. The signals are between the two assistance modes. Next, we focus on the pro- used to make inferences and train machine-learned models active system architecture and the components that power for modeling the user and his or her interests. The same set proactive scenarios. of signals is also used to set rules for notifications and alerts the user wants the agent to serve. The models and rule reci- Proactive assistance pes are deployed to a run time environment. Once proactive Proactive assistance is based on the theory of proactivity that scenarios are deployed in production, capturing and feeding describes user desires and a model of helpfulness [7]. The goal back user behavior signals regarding notifications, alerts, and

70 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Suggestion, Notifications, Offline Inference and Rule Recipe Authoring and Alerts Run Time

Personal Profile Data Device Sensor Signals Collect and Inference and Deploy and Aggregate User Web Services Signals Process Learn Publish

Rule Recipe Authoring

FIGURE 3. The proactive assistance system architecture. suggestions are essential for the proactive agent to learn and different yet related (through time or location) events. For adapt to the user. instance, if the user made a restaurant reservation in a metro- politan city downtown, the agent may suggest nearby parking Notifications and alerts places. Through inference, the agent can learn certain facts In the notifications/alerts category, the system allows the user about the user, by reasoning over the user’s whereabouts and to set rules to define the triggers for certain actions. If the trig- movement patterns through time and location. For example, gering condition evaluates to TRUE, the action is executed. the user’s home and work location could be inferred by join- The rules are defined over a set of signals. These signals are ing GPS data with time over several weeks. If the user is produced by an information channel that can be evaluated by spending most or all of his or her time between 9 a.m. and the proactive agent. These channels represent many types of 5 p.m. during weekdays at a specific location over several information, such as date/time and location, as well as con- weeks, that is likely to be the user’s work location. Likewise, stantly updated data feeds generated by various web services, the user’s commute hours between home and work could also which include weather, sports, news, finance, and entertain- be inferred from combining home and work location with the ment. For example, one can create rules to obtain an alert when GPS data during the likely morning and evening commute the Seattle Seahawks score a touchdown. A user can set a rule hours over several weeks. This inference is used to proactive- to be reminded of his or her mother’s birthday. It is also possi- ly show the traffic commute cards around the time the user ble to combine these signals to formulate more complex trig- typically commutes to/from work (or home). gering rules. For example, a specific flight departure time, a The key questions here are determining the type of sugges- user’s physical location, and a commute time to the airport can tion and when to do it, because there is an associated cost with all be used to trigger an alert that reminds the user that it is the suggestion (if the action, relevance, or timing is wrong). To time to leave for the airport. Once the trigger rule is set, the get around the cold start problem (if the agent does not have agent monitors the signals from the corresponding information access to the user’s past activity through a feedback loop or the channels to evaluate the rule. If the rule evaluates to TRUE, the user is accessing the PDA for the first time), the user is also agent takes an action. The actions are communicated to the given the ability to teach the agent his or her interests from a user in a target device-specific manner, which could be a pro- precompiled list of topics, including news, sports, finance, tech- active entity card, SMS, or even a phone call. This type of pro- nology, dining, and entertainment. The decision for taking a active agent programming falls under the if-then recipes [60], proactive action is driven by a machine-learned model, given the in which simple rules allow users to control many aspects of costs and benefits as constraints. The machine-learned model their digital life. combines a set of information in the user’s profile, demographic and content-based profiles, and online user behavior signals Inference and suggestions (such as click through, dwell time, and dismissal), along with In the suggestions category, the agent infers the user’s habits the user’s recent relevant activity (e.g., similar content searches), and routines by reasoning over his or her past behavior and which are captured in the history variable h in (1). This is used makes a personalized recommendation to the user with the to model whether a specific user u will like the specific sug- goal of furthering the user’s interest. For example, knowing gested entity e . Standard machine-learning techniques, such as that the user watched comedy movies featuring a particular maximum entropy models [35], gradient boosted decision trees actor in the past, the agent may suggest a new comedy movie [55], and deep learning techniques [48], are used to incorporate featuring that same actor in the future. The agent can also both user-specific online and offline signals to estimate the suggest new experiences based on the logical sequencing of probability that the user is expected to like the suggested entity

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 71

THE WORLD’S NEWSSTAND®

(i.e., content). Within the maximum entropy modeling framework, the probability is computed using

/ miifoeu(,,,)h Previous Turn Context i Poeuh = e (,,)/ m jjfoeu(l ,,,)h . (1) / j!J ol e

Here, o denotes outcome (e.g., like or dislike). HS HS HR + HR +

Policy Notice that the denominator includes a sum over all Cross Task Ranking and possible outcomes, ol, which is essentially a nor- malization factor for probabilities to sum to 1. The

Policy functions fi are usually referred to as feature functions, or simply features. These binary feature functions are given as == '1, if ooiiand qeu(,,)h 1 Per-Task Policy Per-Task Policy Per-Task fi (,,,) oeuh = , (2) 0, otherwise

where oi is the outcome associated with feature fi and qeui (,,)h is an indicator function on the user, suggestion, and history. The model parameters mi are learned on labeled data, which capture the user’s response (e.g., like or dislike) for the suggest-

Service Providers Dialogs Task-Oriented Chitchat QA Search Web Service Providers Dialogs Task-Oriented Chitchat QA Search Web ed content in the past. The specific features of the suggested entity (e) include the estimated value of suggestion type, value of suggestion instance, timing of the suggestion, cost of mistake, cost of interruption, and urgency (time sensitivity) of the suggestion. Learned thresholds on State Update the model outputs govern the number of suggestions of Selection Selection Flexible Item Flexible Flexible Item Flexible each type that can be displayed concurrently, the maximum frequency for suggestions of each type, and the permitted or prohibited modalities of each suggestion type. The thresholds for acting, asking, suggesting, or doing nothing are established from a range of default SCO SCO values according to user-stated advice and elicited initial preferences from the configuration wizard.

Reactive assistance Reactive assistance is traditionally known as the conversational understanding system. The conversational understanding system for PDAs spans a wide spectrum of domains, including goal-/task- Natural LU Natural LU

Input Parsing oriented dialogs [53], [16], chitchat, QA [37], and classical web search answers. The conversational understanding system also handles additional input modalities besides speech, such as typing and/or touch. Each of these domains is commonly referred to as an answer (A). Some of the domains involve Speech Speech First Turn: device functionality (e.g., alarm, SMS, calling, Recognition Recognition Second (and Subsequent) Turns: Second (and Subsequent)

Input Recognition note, reminder), while others may involve web services and applications (e.g., directions to a particular location, movie hours at a theater, factoids, stock prices, weather). Typed Query Typed Typed Query Typed Spoken QuerySpoken

Spoken QuerySpoken The reactive assistance system architecture is The reactive assistance architecture: conversational understanding system architecture. The reactive assistance architecture: shown in Figure 4. The user submits a request to the PDA to perform a task or seek information using one

FIGURE 4. FIGURE of the modalities, and the agent interprets the request

72 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

and generates a response. For voice queries, the first step is to Speech recognition recognize the spoken words [9], [10]. The LU component takes The speech recognition component maps the human speech rep- the speech transcription (or the text input if the user types) and resented in acoustic signals to a sequence of words represented in performs a semantic analysis to determine the underlying user text. Let X denote the acoustic observations in the form of fea- intent [11]–[14], [17]. The user’s intent could be related to infor- ture vector sequence and Q be the corresponding word sequence mation search, QA, chitchat, or task-oriented specialist dialogs. (i.e., query). The speech recognition decoder chooses the word t Because PDAs support multidomain and multiturn interactions, sequence, Q, with the maximum a posteriori probability accord- multiple alternate semantic analyses (typically at least one for ing to the fundamental equation of speech recognition [9]: each domain) are generated in parallel for late binding on the t QPXQPQ= argmax ()(), (5) user’s intent [15]. These semantic analyses are sent to the dialog Q state update component, which includes slot carryover (SCO) [38], flexible item selection from a list [39], knowledge fetch where PXQ() and PQ ( ) are the probabilities generated by from the service providers, and dialog hypothesis generation the acoustic and language models, respectively. Traditionally, [15], [16]. Note that in this framework, we consider chitchat, speech recognition systems are trained to optimize the lexical QA, and web search as an additional set of LU domains. All the form. However, displaying the grammatically and semantically dialog hypotheses are ranked by the hypothesis ranking (HR) correct version of the output (i.e., display form) has become an module. The top hypothesis is selected by the hypothesis selec- important requirement for PDAs, because it makes it easy for tion (HS) module by taking the provider responses (i.e., knowl- the user to infer whether the system correctly heard and recog- edge results) into account [18]. The top dialog hypothesis (along nized the spoken query. For example, the following two speech with the ranked dialog hypothesis distribution) is the input to recognition outputs are lexically equivalent: the dialog policy component, which determines the system ■ how is the traffic on u s one oh one (lexical form) response based on the scenario and business logic constraints. ■ how is the traffic on US 101 (display form). Typically, for voice input, the agent speaks the natural language However, proper tokenization in the second hypothesis pro- response via the TTS synthesis engine [19]. vides a valuable hint that the agent understood what the user The reactive assistance behavior is governed by (3). The goal meant. Typically, the tokenization is applied as a separate post- of the reactive agent is to provide the best system response Rt processing module after running the speech recognition decoder. to a given user query, Q . The system response, R , consists of a In recent years, advances in deep learning and its applica- dialog act, which includes system action (e.g., information to be tion to speech recognition have dramatically improved state- displayed, question to be asked, or action to be executed), natural of-the-art speech recognition accuracy [9], [10], [20], [21]. language prompt, and a card in which the response is displayed Deep learning allows computational models that are composed of multiple processing layers to learn representations of data t = " f r , R argmax P(,,,,),R QBAA1 BN B (3) with multiple levels of abstraction. These advances played a R key role in the adoption of PDAs by a large number of users where BA1 denotes the current belief about the dialog state of making it a mainstream product. the answer A1 (e.g., weather, alarm, places, reminder, sports, etc.) after processing query Q , and Br shows the system’s LU belief about the state of the interaction across all answers for The problem of LU for PDAs is a multidomain, multiturn, the current session. In practice, it is hard to solve (3). Instead, a contextual query understanding [17], [22]–[25], subject to the suboptimal solution can be achieved with the assumption that, constraints of the back-end data sources and the applications given the query Q and the beliefs for the dialog states of the in terms of the filters they support and actions they execute. individual answers A1 through AN , the per answer response is These constraints are represented in a schema. In practice, conditionally independent while LU semantically parses and analyzes the query, it does not do so according to a natural LU theory [26]; rather, pars- t rr R = argmax{P (R QB ,AA1 , B ),f , P (R QB ,N , B )}. (4) ing and analysis are done according to the specific user ! RR1,..., RN experience and scenarios to be supported. This is where the r Here, P(,,)R QBAi B denotes the probability the system semantic schema comes into play, as it captures the con- assigns to response (R) generated by answer Ai , given the straints of the back-end knowledge sources and service APIs, answer’s belief about its dialog state and the system’s belief, while allowing free form of natural language expression to Br . This formulation allows the individual answers to manage represent different user intents in an unambiguous manner. their own dialog state and generate their own responses in par- There are two main approaches to LU: rule based and allel. Therefore, it is possible to scale to many domains and machine learned [11], [32]. The rule-based approach is about answers without substantially increasing the overall system hand authoring a set of rules to semantically parse the query response latency. The HR component operates as a metalayer, [27]. It can also be used for addressing the errors and dis- arbitrating between different answer responses, given its belief fluencies introduced by a speech recognizer [28], [49]. (i.e., Br ) about the state of the overall interaction [15]. Next, we State-of-the-art systems use machine-learned models for LU will briefly describe the key components in Figure 4. [12], [17], [23], [24]. In a commonly used LU architecture,

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 73

THE WORLD’S NEWSSTAND®

a query is first classified into one of the supported domains generated by the domain, intent, and slot models to represent (or a catch-all domain such as web). Typically, support vector the complete semantic understanding of the query. machines, boosted decision trees, maximum entropy models Some of the key PDA experiences that are handled are [35], or neural networks [13], [33], [34] are used for model- multiturn in nature. Without using contextual information, the ing. These classifiers are trained in either binary or multi- queries could be ambiguous and potentially interpreted differ- class mode depending on the design choices [12], [23], [24]. ently. For example: Under each domain, there are a domain-specific intent and ■ (turn 1) how is the weather in New York (weather) slot model. Intents and slots can be shared across different ■ (turn 2) what about the weekend (weather). domains. A domain can be considered as a collection of Here is another scenario, where we observe the exact same related intents, which do not have any conflict. For example, query in the second turn: a weather domain contains check_weather and get_weather_ ■ (turn 1) how is my schedule (calendar) stats intents. Intent detection uses the same machine-learning ■ (turn 2) what about the weekend (calendar). techniques listed previously, and it is framed as a multi- Interpreting the follow-up queries in isolation is difficult, as class classification problem. Slot tagging is considered as a they are ambiguous and require context for proper interpreta- sequence classification problem. Conditional random fields, tion. LU models are built in a contextual manner to solve this maximum entropy Markov models [35], and, more recently, problem [15], [22]–[24]. To handle multiturn interactions, in deep learning techniques [12], [24] are used for slot tagging. addition to basic domain, intent, and slot models, one needs to LU models are trained in a supervised fashion using labeled build context carryover models to help with state tracking [38] data and properly weighted lexicons [36]. When a user issues and on-screen selection models [39], as shown in Figure 4, for a query, domain and intent classifiers are run to determine selecting items from a list of results presented to the user in the domain and intent of the query, and the slot tagger tags follow-up turns. semantic slots. Tagged slots are resolved into canonical values, and in some cases multiple slots are combined into a QA parameter. Parameters are used either to fetch results from Because users are expecting PDAs to answer any question, the knowledge back end or to invoke an application API. open-domain QA [56] is another scenario that is handled by The goal of the LU component (for answer Ai) is to convert all PDAs to differing degrees. Examples of open-domain

each input query into a set of semantic frames, FAi , given the factoid QA include the following questions: context represented in the current beliefs about the dialog state ■ How old is Bill Gates?

BAi, that is, we seek ■ How tall is Mount Everest? ■ Who directed Avatar? t " t , FPFQBAAi = argmax (,)i . (6) F The answers to these questions are precise short phrases. ■ Bill Gates is 60 years old.

Semantic frame, FAi , for answer Ai encapsulates the ■ Mount Everest is 8,848 meters high. semantic meaning of a query with a tuple of domain, intent, ■ James Cameron directed the movie Avatar. and slot list: QA has had a long history [29] and has seen rapid advancement in the past decade, spurred by government-funded pro- GDOMAIN, INTENT, SLOTSH . grams that required system building, experimentation, and The slots are a list of key-value pairs: evaluation of systems [30]. Advancements in search engine technology, such as query formulation and query-document GSLOT_NAME, SLOT_VALUEH. analysis through click logs, have also contributed to innovation The data structure for semantic frame, shown in Figure 5, in QA [52]. is used to combine the different pieces of semantic analysis The system architecture for a typical QA system is shown in Figure 6. For a given query directed at a PDA, the QA system first classifies the query into one of the question types reminder (typically ten to 15). Note that the query may not be a ques- create_single_reminder tion. Even if it is a question, it may not be supported by the QA system. These categories are also included as part of the question classification step. The answer-candidate generation step processes the question, generates various alternate formulations of it, and queries the knowledge sources, which include a knowledge graph, web documents, Wikipe- dia, and search engines. The answer-candidate ranking step extracts a number of features for each question/answer pair and applies a ranking model to rank and assign confidences FIGURE 5. The semantic frame for the query: “remind me to call my mom to each answer candidate [56]. The overarching principles of at 9 a.m.” QA systems are massive parallelism, confidence estimation,

74 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

and integration of shallow and deep knowledge from many cover new domains, intents, or slots [41]–[43], [51] or improv- knowledge sources [37]. ing existing experiences [44], [45].

Knowledge back ends DM and policy A significant part of PDA scenarios is about accessing Many of the reactive scenarios enabled in PDAs require spoken knowledge and entities. For example, when a user asks for a DM to handle a wide range of tasks and domains. DM is at the factoid regarding a movie or director, LU models tag such bottom of the reactive stack, where all the information from slots (i.e., facets) as movie name, release date, actors, and upstream components is consolidated and the final decision about directors. Slots are used to build a query sent to these knowl- the system response is made and communicated to the user. edge bases to fetch the relevant entity and relationships for Much of the research on spoken dialog systems in academia which the user is looking. These knowledge bases store fac- has targeted single-domain applications [47], [50], where the tual information in the form of entities and their relation- problem of accurately tracking the user’s goal (e.g., finding res- ships, covering many domains (people, places, sports, taurants that satisfy a number of user constraints) has received business, etc.) [57]. The entities and their relationships are considerable attention in the literature [53]. The primary line organized using the World Wide Web Consortium Resource of research has been the statistical modeling of uncertainties Description Framework (RDF) [40]. An RDF semantic and ambiguities encountered by dialog managers due to speech knowledge base (also known as a semantic knowledge graph) recognition and LU errors along with ambiguity in natural lan- represents information using triples of the form subject-pred- guage expressions. Included among the most successful statis- icate-object, where in graph form, the predicate is an edge tical approaches are graphical models that are concerned with linking an entity (the subject) to its attributes or another relat- decision making with delayed rewards [53]. However, large- ed entity, as seen in Figure 7. A popular open-source RDF scale production systems such as PDAs pose a different set of semantic knowledge base is Freebase [46]. Other RDF problems. The large number of supported domains, integration semantic knowledge bases that are much larger in size are of task-oriented dialogs, QA, chitchat, and web answers and Facebook’s Open Graph, Google‘s Knowledge Graph, and managing conversation in a coherent way pose new challenges Microsoft’s Satori. Both Google’s Knowledge Graph and Microsoft’s Satori knowledge graph have over 1 Ranked billion entities and many more entity Question Question Answer-Candidate Answer-Candidate Answer relationships. They power entity-relat- Classification Generation Ranking ed results that are generated by Google’s and Microsoft Bing’s search engines. Recently, there has been a surge of interest in exploiting these Knowledge Search Wikipedia Web Documents knowledge sources, especially the Graph Engine RDF semantic knowledge bases, to Knowledge Resources reduce the manual work required for expanding conversational systems to FIGURE 6. The QA system architecture.

Director e1 e2

Name source_url Name born_in

James Avatar Kapuskasing http://www.imdb.com/title/tt0499549 Cameron

Type

Film/Film

FIGURE 7. An example of part of a semantic knowledge graph representing the relationships, described as RDF triples, between the entities James Cameron (e1) and the film Avatar (e2).

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 75

THE WORLD’S NEWSSTAND®

[25]. Mixing different modalities to complete the tasks is current turn identification, whether the user has prompted for another challenging area for PDAs. the same information before) and generates a natural and In PDAs, a dialog manager supports execution of a variable grammatical utterance to convey the system response. There number of goal-oriented tasks [16], [54]. Tasks are defined in are a number of factors that feed into the LG design, such as terms of information to be collected from the user, correspond- information presentation, presenting enough information (to ing LG prompts, and interfaces to resources (such as data hosted give a good overview of the state of the task) versus keeping in external services and applications) that will execute actions the utterances short and understandable, handling error states, on behalf of the user. As shown in Figure 4, at each turn, the and repeated tries [59]. dialog state is updated, taking into consideration the multiple There are three main approaches to LG: 1) template-based, 2) LU results across different turns. SCO [38] does contextual rule-based (linguistic), and 3) corpus-based approaches [59], [61]. carry over of slots from previous turns, using a combination of Most of the PDAs use the template-based approach, because com- rules and machine-learned models with lexical and structural paratively less effort is needed to develop and maintain the tem- features from the current and previous turn utterances. Flexible plates. The template-based LG module typically starts out from a item selection uses task-independent, machine-learned models semantic representation (e.g., semantic frame), generating “QFC [39] to handle disambiguation turns where the user is asked to in Redmond is open from 7:00 a.m. to 10:00 p.m.” in response to select between a number of possible items. The task updater the user query, “Is QFC in Redmond open today?” For example, module is responsible for applying both task- STOREHOURS: [PLACENAME(“QFC”), independent and task-specific dialog state It is generally difficult to LOCATION(“REDMOND”)] associates updates. Task-dependent processing is driven it directly with a template, such as [PLACE- by a set of configuration files, or task forms, empirically evaluate the NAME] in [LOCATION] is open from with each form encapsulating the definition quality of proactive and [TIMEBEGIN] to [TIMEEND], where of one task. Using the task forms, this mod- reactive user experiences the gaps represented by [PLACENAME], ule initiates new tasks, retrieves information for PDAs. [LOCATION], [TIMEBEGIN], and [TIME- from knowledge sources, and applies data END] are filled by looking up the relevant transformations (e.g., canonicalization). Data information in the dialog state. The TTS transformations and knowledge source lookups are performed engine consumes the LG output and synthesizes the text into using resolvers [16], [54]. Dialog policy execution is split into speech [19]. task-specific and global policy. The per-task policy consists of analyzing the state of each task currently in progress and sug- Metrics and measurement for PDAs gesting a dialog act to execute. The dialog acts include show It is generally difficult to empirically evaluate the quality of results, disambiguation, prompt for missing value, prompt for no proactive and reactive user experiences for PDAs. PDAs are results found, start over, go back, cancel, confirm, complete the complex systems with many components in the system stack, task, and so forth, in accordance with the ranked semantic frame spanning client and multiple cloud services, and it is hard to output. The output of the task updater module is a set of dialog separate any one component from the rest. Each component hypotheses representing alternative states or dialog actions for has its own functional and quality metrics. Metrics could be each task in progress. The dialog hypotheses are ranked using offline, measured with sampled data sets, or online, measured HR [15], [18], which generates a ranked order and score for each with actual user traffic of the live system. hypothesis. This acts as a pseudobelief distribution over the possible dialog/task states. HS policy selects a top hypothesis based Component metrics on contextual signals, such as the previous turn task, rank order, The following are the offline component metrics tracked individ- and scores as well as business logic. ually to improve the quality of each component. They are com- HR uses an implementation of LambdaMart [55] to rank puted using a set of ground truths generated by human judgments: hypotheses. Previously, various approaches have also been ■ speech recognition: word error rate (WER), sentence error presented for reranking for spoken LU [58] but have focused rate, slot WER, keyword WER, semantic WER (for non- on single-domain applications. search tasks) ■ LU: domain accuracy, intent accuracy, precision/recall, slot LG F1 measure, semantic frame accuracy Once the system response is determined by the dialog manag- ■ dialog: dialog state tracking accuracy—in a distribution er, it is communicated to the user in a natural way. If the (e.g., N-best list) of dialog state hypotheses, percent accu- query is a speech query, a spoken system response is returned. racy of the top-ranked hypothesis, selection accuracy, SCO If the query is typed, there is no spoken system response but accuracy rather a natural language system response that the user sees ■ LG: mean opinion score (MOS) of the LG quality, bilin- on the screen as a card. In either case, the natural LG compo- gual evaluation understudy score nent receives the dialog output in the form of a system ■ knowledge: knowledge relevance, coverage, precision/ response along with the dialog state, which encapsulates the recall state of the interaction between the user and system (e.g., ■ TTS synthesis: MOS, intelligibility, expressiveness

76 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

■ proactive suggestions/notifications: precision/recall. LU Accuracy (%) 100 In Figure 8, we show LU accuracy 90 80 for domain, intent, slot tagging (F1 70 measure), and semantic frame for some 60 50 of the reactive experiences supported in 40 30 Cortana for a sample training and test 20 10 data set [25], [45]. The domain, intent, 0 and slot accuracy is around 90% across Web Alarm domains. The average semantic frame Mystuff Places Overall Calendar Weather accuracy is 82%. Note that semantic On Device Reminder Entertainment frame combines domain, intent, and Communication slots; therefore, errors in these compo- Domain Intent Slot F1 Semantic Frame nents contribute to the semantic frame error rate. FIGURE 8. The LU domain, intent, slot, and semantic frame accuracy. In Figure 9, we show the impact of HR in picking the right hypothesis over the LU model confidences. HR improves in picking the correct Semantic Frame Accuracy (%) semantic frame by 2%. HR has the 100 90 full view of all the LU analyses 80 coming from different domains, and 70 60 so it can arbitrate between compet- 50 40 ing hypotheses. 30 20 10 End-to-end quality metrics 0

The fact that individual component Web Alarm Mystuff Places Overall accuracies are high may not mean that Calendar Weather On Device Reminder the PDA, as a product, has high accu- Entertainment Communication racy. There are several factors contributing to this. For example, speech LU HR recognition may not be accurate even if LU is accurate, and knowledge FIGURE 9. The semantic frame accuracy for LU versus HR. results may not be relevant. There may be operational service reliability issues, back-end availability, network communication issues, robustness of wake-up word detection, and so forth, which all contribute toward quality of the user experience with the product. Moreover, a user’s intent could be understood by the LU component, but the underlying application or service may not support that intent. Integrating different client and service components is a challenging software engineering problem, as it uncovers numerous scenario, design, service, and client shortcomings, which take the most time in improving the system. Therefore, end-to-end (E2E) product quality metrics are critical for the success of the product, as they correlate well with the actual user experience. They are also used for evaluating the contribution of the individual components to the overall product experience based on the analysis of how much each component contributes to the E2E error rates. Some of these metrics are as follows. ■ E2E accuracy is measured through human judgment of (a) (b) query–response pairs on a five-point scale, where the user is shown a screenshot containing the system response, FIGURE 10. The system responses for the query “Chinese food near my similar to the ones in Figure 10. Human judges assign a location.” (a) The correct result from the places domain and (b) the web score between 1 (terrible) and 5 (perfect) to each search result.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 77

THE WORLD’S NEWSSTAND®

query response pair. Success includes the ratings of 3 (okay), In Figure 11, we show the E2E query–response pair accu- 4 (pretty good), and 5 (perfect). racy. In the figure, E2E Success* denotes the accuracy after ■ Side-by-side (SBS) compares system A with system B, leaving out the use cases the scenario is not designed to handle where the two systems could be the same system (or com- in the first place. For example, the user wants to delete an alarm petitive systems) at different points in time differing in on the PDA, but the scenario is not supported by design. Instead, updates and improvements. SBS is an A/B test on a five- the PDA shows either an irrelevant web search result or invokes or seven-point scale, where human judges pick one system the alarm application. over the other based on the results shown to the judge. For In Figure 12, we show the distribution of user dissatisfaction example, in Figure 10, we show two PDA system respons- with regard to the sources of error across different components es for the query “Chinese food near my location.” In an of the system. The numbers are based on feedback of about SBS evaluation, the human judges compare the two 10,000 real users. LU along with unsupported system action responses as to whether the left/right response is better (e.g., user wants to delete an alarm but system does not sup- than the other on a scale of +3 to -3 , where 0 shows that port that action) are the biggest sources of user dissatisfaction. they are equal. SBS is a more sensitive metric compared to Frequent fallback to web search (i.e., text links in search results) E2E accuracy. when the system does not a have precise answer, a lack of LG ■ Online user/system behavior-based metric measures the (for scenarios where the user expects the system to talk), and user satisfaction and dissatisfaction with the PDA experi- speech recognition errors are the main buckets of user dissatis- ences. It is a model that uses a set of feedback signals from faction with PDAs. the user and the system that correlates with the quality of the user experience. The signals include actions executed, Technology and user experience challenges query reformulation, total elapsed time for task comple- While the functionality and types of tasks a PDA can perform tion, landing page dwell time, click through, start over, are quite diverse and users find great value in using them, cancel, click back rates, and latency. there are still a number of user experience and technical chal- It is quite challenging to evaluate PDAs, as they provide a lenges that have yet to be addressed properly. We categorize wide range of experiences, including voice commands, task these challenges into the following groups. completion, chitchat, QA, and web search. Therefore, success/ failure signals for online measurement could be quite different User experience challenges [62]. An instance where no click has occurred on the screen (i.e., User experience challenges include the challenges in the fol- abandonment) in one experience may mean user satisfaction (e.g., lowing list. showing a correct weather card), but it may mean dissatisfaction ■ Operation errors: There is a discrepancy between the in another domain (e.g., showing a list of restaurants and prompt- user’s mental model of the PDA’s functionality and scenario ing the user to select), where the user leaves satisfied in the for- coverage versus the actual PDA functionality. Users are mer but dissatisfied in the latter. There are also additional metrics often unaware of the total extent of the operations a PDA used for business and overall product success, including user can perform. Users may not understand how to use the count, daily/monthly average users, sessions per user, query vol- PDA application or what they need to say to get the result ume, unique query count, and number of proactive page views. they desire. Current user interfaces lack the ability to

Query–Response Accuracy 100 90 80 70 60 50 40 30 20 10 0

Web Alarm Places Calendar Weather Average On Device Reminder

Communication E2E Success E2E Success*

FIGURE 11. The E2E query–response accuracy. E2E Success* denotes the accuracy after leaving out the use cases that the scenario is not designed to handle in the first place.

78 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

provide sufficient information about how to use the system and intuitive sequence of operations to complete a task. User Dissatisfaction Distribution PDAs are also not able to adapt to different user profiles Fallback to Search and their way of operations. 12% 16% No LG ■ Lack of competence: PDAs are not at a level to reliably decide when to help the user, what to help the user with, Unsupported Action 14% and how to help him or her. This also creates a trust issue 24% between the user and the agent, and the user may not feel Back-End Failure 21% comfortable delegating a task to the agent. 13% LU Failure ■ Privacy and security: Privacy and security of the user’s data and profile are a concern for users. Questions such as ASR how much a PDA can/should know about its user and what the control mechanisms are remain open. FIGURE 12. The E2E overall user dissatisfaction distribution over different components. Technical challenges Technical challenges include capability should be highlighted in other devices and end ■ Experience scaling: It is critical to integrate third-party points. For example, Cortana started with the phone, but it applications and services into PDAs to scale both reactive is now made available in PCs and tablets and even on and proactive experiences that can be handled by PDAs. Xbox. This in turn creates another problem; not all experi- Building the right tools and infrastructure to easily enable ences make sense for a given device, or the same query such integration is an open problem. User feedback may be interpreted differently on different devices. For shows that unsupported experiences are one of the single example, users cannot send SMS from a PC or Xbox biggest sources of user dissatisfaction. For example, a using PDAs, but they can do it on a phone. The query “go user wants to be able book a taxi or calculate the mort- home” could mean “get driving directions to home” on the gage payments for a house using speech with his or her phone experience, but it may mean “go to shell” on the PDA, but these, and many more scenarios, may not be Xbox device. supported by the PDA. ■ Service challenges: PDAs use numerous services to enable ■ Speech recognition challenges: Despite all the recent prog- a given scenario. There are also software engineering and ress in speech recognition with the application of deep service challenges that impact the overall user experience. learning techniques, issues such as background noise, For example, if the latency for handling a user request is too speaker accent, Bluetooth, side speech, pocket dial, and high, it reflects negatively on the user experience. In fact, unintentional wake up remain to be addressed [9]. To truly instability may even stop users from using the scenario alto- fulfill its promise, a PDA should recognize all the personal gether. Likewise, users expect high reliability and availabili- words that the user cares about. This includes any name ty (e.g., > 99.9%) from the services handling the requests. (not just English), any place, and any thing (e.g., user’s All of these are requirements and constraints that influence contact list), essentially leading to an open-domain, unlim- the system design. ited vocabulary speech recognition problem. ■ LU challenges: Domain scaling to cover many more Moving forward domains, high-quality LU model development, and contin- Research on human work habits and task management [3], ual refinement with feedback loop data are the main chal- [63], [64] shows that people usually complete all their impor- lenges. Building reusable models across different tasks is tant tasks yet may fail to successfully complete tasks with soft an important problem to solve as well. deadlines or may forget less-critical details. In the short term, ■ DM: Heterogeneous knowledge back ends and application PDAs can provide great utility by becoming the digital mem- interfaces are a bottleneck for expanding the domains and ory that users can depend on for help with completion of tasks a PDA can cover. The APIs for different applications/ everyday tasks. In fact, it is these scenarios that are used most services for the actions they perform as well as data/knowl- by the users (e.g., reminders, meetings, and some proactive edge back ends are not standardized and require custom notifications and alerts). interface work and query building. Because time is a critical asset, improving personalized ■ Locale/market expansion: Building a proactive or reactive time management and utility through proactive and reac- experience, not for English but for other languages and tive task delegation and completion seems to be a plausible markets, is another open problem. This requires reusing or and desirable long-term goal for PDAs. In the future, it is building all the resources (e.g., data, content, and models) the scalable and seamless third-party integration that can and capabilities for new locales and markets. substantially increase the scenario and experience coverage ■ Different device/end points: Even though smartphones and determine whether PDAs will fulfill the promise of a were the initial target device for deploying PDAs, soon it true personal assistant that users can depend upon to manage became evident that the underlying intelligence and agent their personal and work life, effectively making them more

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 79

THE WORLD’S NEWSSTAND®

productive. The walls between applications may start to break [2]H.Chalupsky,Y.Gil,C.Knoblock,K.Lerman,J.Oh,D.Pynadath,T.Russ, and M. Tambe, “Electric elves: Agent technology for supporting human organiza- down if PDAs achieve app/service composition to complete tions,” AI Mag., vol. 23, no. 2, pp. 11–24, 2002. new tasks in a scalable way. [3] K. Myers, P. Berry, J. Blythe, K. Conley, M. Gervasio, D. McGuinness, D. PDAs will surface on many different devices and envi- Morley, A. Pfeffer, et al., “An intelligent personal assistant for task and time management,” AI Mag., vol. 28, no. 2, pp. 47–61, 2007. ronments. This will create new signal processing challenges, [4] SRI International. (2003–2009). CALO: Cognitive Assistant that Learns and such as accurate speaker separation and tracking in multi- Organizes. [Online]. Available: pal.sri.com speaker environments (e.g., home, car), robustness with [5] N. Yorke-Smith, S. Saadati, K. L. Myers, and D. N. Morley, “The design of a pro- respect to different device types, and robust speech recogni- active personal agent for task management,“ Int. J. Artif. Intell. Tools, vol. 21, no. 1, 2012. tion across age, gender, and accent. Advances in algorithms, [6]S.Sohrab,M.Zhang,C. J.Karr,S. M.Schueller,M. E.Corden,K. P.Kording, signal processing, and machine learning would be needed to and D. C. Mohr, “Mobile phone sensor correlates of depressive symptom severity in solve these problems. daily-life behavior: An exploratory study,” J. Med. Internet Res., vol. 17, no. 7, p. e175, 2015. On the industry front, the investment and competition in [7] S. Schiaffino and A. Amandi, “User-interface agent interaction: Personalization PDA technology will keep increasing over the next decade. issues,” Int. J. Human-Computer Studies, vol. 60, no. 1, pp. 129–148, 2004. It is seen by some that PDAs may set the balance of power [8] P. Maes, “Agents that reduce work and information overload,” J. ACM, vol. 37, in the next phase of the Internet, if it becomes the gateway no. 7, pp. 30–40, 1994. to applications and services with the proliferation of IoT [9] J. Li, L. Deng, R. Haeb-Umbach, and Y. Gong, Robust Automatic Speech Recognition, a Bridge to Practical Applications. New York: Academic, 2015. devices. It is too early to call it an inflection point for PDA [10] G. Saon, H. K. J. Kuo, S. Rennie, and M. Picheny, “The IBM 2015 English technology. It is likely that true natural language human– conversational telephone speech recognition system,” in Proc. Interspeech, computer interaction with gadgets may take another decade Dresden, Germany, 2015. to be second nature. [11] G. Tür and R. De Mori, Eds., Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. Hoboken, NJ: Wiley, 2011. [12] A. Deoras, R. Sarikaya, G. Tür, and D. Hakkani-Tür, “Joint decoding for Acknowledgments speech recognition and semantic tagging,” in Proc. Interspeech,2012, pp. 1067– I would like to thank the past and present members of the 1070. Language Understanding and Dialog Systems Group at [13] R. Sarikaya, G. Hinton, and A. Deoras, “Application of deep belief networks for natural language understanding,” IEEE Trans. Audio, Speech, Language Microsoft, who built the conversational understanding and Process., vol. 22, no. 4, pp. 778–784, 2014. DM capabilities of Cortana: Alex Rochette, Asli Celikyilmaz, [14]N.Gupta,G.Tür,D.Hakkani-Tür,S.Bangalore,G.Riccardi, andM.Gilbert, Beatriz Diaz Acosta, Chandra Akkiraju, Daniel Boies, Danko “The AT&T spoken language understanding system,” IEEE Trans. Audio, Speech, Language Process., vol. 14, no. 1, pp. 213–222, Jan. 2006. Panic, Derek Liu, Divya Jetley, Diamond Bishop, Elizabeth [15] J. P. Robichaud, P. Crook, P. Xu, O. Z. Khan, and R. Sarikaya, “Hypotheses Krawczyk, Gabrielle Knight, Hisami Suzuki, Jean-Phillipe ranking for robust domain classification and tracking in dialogue systems,” in Proc. Robichaud, John Nave, Khushboo Aggarwal, Kjel Larsen, Interpseech, Singapore, 2014, pp. 145–149. Logan Stromberg, Minwoo Jeong, Nikhil Ramesh, Omar Zia [16] P. A. Crook, A. Marin, V. Agarwal, K. Aggarwal, T. Anastasakos, R. Bikkula, D. Boies, A. Celikyilmaz, S. Chandramohan, Z. Feizollahi, R. Khan, Paul Crook, Puyang Xu, Rachel Morton, Ravi Bikkula, Holenstein, M. Jeong, O. Z. Khan, Y. B. Kim, E. Krawczyk, X. Liu, D. Panic, V. Roman Holenstein, Roy Tan, Steve Kofsky, Tasos Anastasa- Radostev, N. Ramesh, J. P. Robichaud, A. Rochette, L. Stromberg, and R. Sarikaya, “Task completion platform: A self-serve multi-domain goal oriented dia- kos, Vasiliy Radostev, Vipul Agarwal, Young-Bum Kim, and logue platform,” in Proc. North American Chapter of the Association for Zhaleh Feizollahi. Computational Linguistics: Human Language Technologies, San Diego, CA, 2016, pp. 47–51. [17] J. R. Bellegarda, “Spoken language understanding for natural interaction: The Author Siri experience,” in Natural Interaction with Robots, Knowbots and Smartphones. Ruhi Sarikaya ([email protected])______is the director of New York: Springer-Verlag, 2014, pp. 3–14. applied science at Amazon. He received his B.S. degree from [18] O. Z. Khan, J. P. Robichaud, P. Crook, and R. Sarikaya, “Hypotheses ranking and state tracking for a multi-domain dialog system using ASR results,” in Proc. Bilkent University, Ankara, Turkey, in 1995; his M.S. degree Interspeech, Dresden Germany, 2015. from Clemson University, South Carolina, in 1997; and his [19] A. J. Hunt and A. W. Black. “Unit selection in a concatenative speech synthesis Ph.D. degree from Duke University Durham, North Carolina, system using a large speech database,” in Proc. Int. Conf. Acoustics, Speech, and Signal Processing, 1996, vol. 1, pp. 373–376. in 2001, all in electrical and computer engineering. He was a [20] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, principal science manager at Microsoft from 2011 to 2016, V. Vanhoucke, et al., “Deep neural networks for acoustic modeling in speech recog- where he founded and managed the team that built language nition: The shared views of four research groups” IEEE Signal Process. Mag., understanding and dialog management capabilities of Cortana vol. 29, no. 6, pp. 82–97, 2012. [21] H. Sak, A. W. Senior, and F. Beaufays, “Long short-term memory recurrent and Xbox One. Before Microsoft, he was with IBM Research neural network architectures for large scale acoustic modeling,” in Proc. for ten years. Prior to joining IBM in 2001, he was a Interspeech, Singapore, 2014, pp. 338–342. researcher at the University of Colorado at Boulder for two [22] A. Bhargava, A. Celikyilmaz, D. H. Tur, and R. Sarikaya, “Easy contextual intent prediction and slot detection,” in Proc. Int. Conf. Acoustics, Speech, and years. He has authored more than 100 technical papers and Signal Processing, May 2013, pp. 8337–8341. has more than 60 issued/pending patents. He is a Senior [23] P. Xu and R. Sarikaya, “Contextual domain classification in spoken language Member of the IEEE. understanding systems using recurrent neural network,” in Proc. Int. Conf. Acoustics, Speech, and Signal Processing, 2014, pp. 136–140. [24] C. Liu, P. Xu, and R. Sarikaya, “Deep contextual language understanding in References spoken dialogue systems,” in Proc. Interspeech, 2015, pp. 120–124. [1]T.Mitchell,R.Caruana,D.Freitag,J.McDermott, and D.Zabowski, “Experience with a learning personal assistant,” Commun. ACM, vol. 37, no. 7, [25] R. Sarikaya, P. A. Crook, A. Marin, M. Jeong, J. P. Robichaud, A. pp. 80–91, 1994. Celikyilmaz, Y. B. Kim, A. Rochette, O. Z. Khan, X. Liu, D. Boies, T.

80 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Anastasakos, Z. Feizollahi, N. Ramesh, H. Suzuki, R. Holenstein, E. Krawczyk, [49] V. Zue, S. Seneff, J. R. Glass, J. Polifroni, C. Pao, T. J. Hazen, and L. and V. Radostev, “An overview of end-to-end language understanding and dialog Hetherington, “Jupiter: A telephone-based conversational interface for weather management for personal digital assistants,” in Proc. IEEE Spoken Language information,” IEEE Speech Audio Process., vol. 8, no. 1, pp. 85–96, 2000. Technology, San Diego, CA, 2016. [50] R. Pieraccini and J. Huerta, “Where do we go from here? Research and com- [26] T. Winograd, “Understanding natural language,” Cognitive Psych., vol. 3, no. 1, mercial spoken dialog systems,” in Proc. 6th SIGdial Workshop on Discourse and pp. 1–191, 1972. Dialogue,2005. [27] W. A. Woods, “Transition network grammars for natural language analysis,” [51] Y. B. Kim, K. Stratos, R. Sarikaya, and M. Jeong, “New transfer learning tech- Commun. ACM, vol. 13, no. 10, pp. 591–606, 1970. niques for disparate label sets,” in Proc. Association for Computational Linguistics, 2015, pp. 473–482. [28] W. Ward and S. Issar, “Recent improvements in the CMU spoken language understanding system,” in Proc. Workshop Human Language Technology, [52] J. Huang and E. N. Efthimiadis, “Analyzing and evaluating query reformula- Association for Computational Linguistics, 1994, pp. 213–216. tion strategies in web search logs,” in Proc. 18th ACM Conf. Information and Knowledge Management,2009, pp.77–86. [29] R. F. Simmons, “Natural language question-answering systems: 1969,” Commun. ACM, vol. 13, no. 1, pp. 15–30, 1970. [53] S. Young, M. Gasic, S. Keizer, F. Mairesse, J. Schatzmann, B. Thomson, and K. Yu, “The hidden information state model: A practical framework for POMDP- [30] T. Strzalkowski and S. Harabagiu, Eds., Advances in Open-Domain Question- based spoken dialogue management,” Comput. Speech Lang., vol. 24, no. 2, pp. Answering. Berlin, Germany: Springer-Verlag, 2006. 150–174, 2009. [31] C. L. Isbell and J. S. Pierce, “An IP continuum for adaptive interface design,” in [54] A. Marin, P. Crook, O. Z. Khan, V. Radostev, K. Aggarwal, and R. Sarikaya, Proc. Human–Computer Interaction Int. 2005, Las Vegas, NV. “Flexible, rapid authoring of goal-orientated, multi-turn dialogues using the task [32] E. Levin, R. Pieraccini, and W. Eckert, “A stochastic model of human-machine completion platform,” in Proc. Interspeech 2016, San Francisco, CA, 2016, pp. interaction for learning dialog strategies,” IEEE Speech Audio Process., vol. 8, no. 1, 1571–1572. pp. 11–23, 2000. [55] C. J. C. Burges, K. M. Svore, P. N. Bennett, A. Pastusiak, and Q. Wu, [33] Y. Shi, Y. C. Pan, M. Y. Hwang, K. Yao, H. Chen, Y. Zou, and B. Peng, “A “Learning to rank using an ensemble of lambda-gradient models,” J. Mach. Learn. factorization network based method for multi-lingual domain classification,” Res., vol. 14, pp. 25–35, 2011. in Proc. Int. Conf. Acoustics, Speech, and Signal Processing,2015, pp. [56] W. T. Yih, X. He, and C. Meek. “Semantic parsing for single-relation question 5276–5280. answering,” in Proc. Association for Computational Linguistics, 2014, pp. 643– [34] G. E. Hinton, S. Osindero, and T. Yee-Whye, “A fast learning algorithm for 648. deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, 2006. [57] M. Nickel, M, K. Murphy, V. Tresp, and E. Gabrilovich, “A review of relational [35] I. H. Witten and F. Eibe, Data Mining: Practical Machine Learning Tools and machine learning for knowledge graphs,” Proc. IEEE, vol. 104, no. 1, pp. 11–33, 2016. Techniques. San Mateo, CA: Morgan Kaufmann, 2005. [58] M. Dinarelli, A. Moschitti, and G. Riccardi, “Discriminative reranking for spo- [36] X. Liu and R. Sarikaya, “A model based approach to weight dictionary entities ken language understanding,” IEEE Trans. Audio, Speech, Language Process., vol. for spoken language understanding,” in Proc. IEEE Spoken Language Technology, 20, no. 2, pp. 526–539, Feb. 2012. Lake Tahoe, NV, 2014. [59] V. Rieser and O. Lemon, “Natural language generation as planning under [37] D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. Kalyanpur, A. uncertainty for spoken dialogue systems,” in Empirical Methods in Natural Lally, J. W. Murdock, et al., “Building Watson: An overview of the DeepQA proj- Language Generation. Berlin, Germany: Springer-Verlag, 2010, pp. 105–120. ect,” AI Mag., vol. 31, no. 3, pp. 59–79, 2010. [60] R. J. Kate, Y. W. Wong, and R. J. Mooney, “Learning to transform natural to [38]D.Boies,R.Sarikaya,A.Rochette,Z.Feizollahi, and N.Ramesh, formal languages,” in Proc. 12th Nat. Conf. Artificial Intelligence (AAAI-05), “Techniques for updating partial dialog state,” U.S. Patent Application Pittsburgh, PA, 2005, pp. 1062–1068. 20150095033, 2013. [61] A. H. Oh and A. I. Rudnicky, “Stochastic language generation for spoken dia- [39] A. Celikyilmaz, Z. Feizollahi, D. Hakkani-Tür, and R. Sarikaya, “Resolving logue systems,” in Proc. 2000 ANLP/NAACL Workshop on Conversational referring expressions in conversational dialogs for natural user interfaces,” in Proc. Systems, 2000, vol. 3, pp. 27–32. Empirical Methods on Natural Language Processing, 2014, pp. 2094–2104. [62] J. Jiang, A. H. Awadallah, R. Jones, U. Ozertem, I. Zitouni, R. G. Kulkarni, [40] N. Shadbolt, W. Hall, and T. Berners-Lee, “The semantic web revisited,” IEEE and O. Z. Khan, “Automatic online evaluation of intelligent assistants,” in Proc. Intell. Syst., vol. 21, no. 3, pp. 96–101, 2006. 24th Int. World Wide Web Conf., 2015, pp. 506–516. [41] R. Baeza-Yates and A. Tiberi, “Extracting semantic relations from query logs,” [63] M. Czerwinski, E. Horvitz, and S. Wilhite, “A diary study of task switching in Proc. ACM Special Interest Group on Knowledge Discovery and Data Mining, and interruptions,” in Proc. ACM Conf. Human Factors in Computing Systems, 2007, pp. 76–85. Vienna, Austria, 2004, pp. 175–182. [42] L. Heck, D. Hakkani-Tür, and G. Tür, “Leveraging knowledge graphs for web- [64] V. Bellotti, B. Dalal, N. Good, P. Flynn, D. G. Bobrow, and N. Ducheneaut, scale unsupervised semantic parsing,” in Proc. Interspeech, August 2013, pp. 1594– “What a to-do: Studies of task management towards the design of a personal task list 1598. manager,” in Proc. ACM Conf. Human Factors in Computing Systems, Vienna, Austria, 2004, pp. 735–742. [43] A. El-Kahky, X. Liu, R. Sarikaya, G. Tür, D. Hakkani-Tür, and L. Heck, “Extending domain coverage of language understanding systems via intent transfer [65] Statista. (2016). Number of mobile phone users worldwide from 2013 to 2019 between domains using knowledge graphs and search query click logs,” in Proc. (in billions). [Online]. Available: http://www.statista.com/statistics/274774/ IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Florence, Italy, 2014, forecast-of-mobile-phone-users-worldwide/______pp. 4067–4071. [66] Siri. Apple. [Online]. Available: http://www.apple.com/ios/siri/ [44] Y. Ma, P. Crook, and R. Sarikaya, “Statistical inference over knowledge graphs for [67] Google Now. Google. [Online]. Available: https://www.google.com/landing/ task-oriented spoken dialog systems,” in Proc. Int. Conf. Acoustics, Speech, and Signal ______now/ Processing, Brisbane, Australia, 2015, pp. 5346–5350. __ [68] Cortana. Microsoft. [Online]. Available: https://www.microsoft.com/en-us/ [45] Y.-B. Kim, M. Jeong, and R. Sarikaya, “Semi-supervised slot tagging with par- ______mobile/experiences/cortana/ tially labeled sequences from web search click logs,” in Proc. North American ______Chapter of the Association for Computational Linguistics, 2015, pp. 84–92. [69] Alexa. Amazon Developer. [Online]. Available: h______ttps://developer.amazon .com/public/solutions/alexa [46] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: A col- laboratively created graph database for structuring human knowledge,” in Proc. 2008 [70] M. Meeker. (2016, June 1). Internet trends 2016—Code conference. Kleiner ACM SIGMOD Int. Conf. Management of Data, New York, 2008, pp. 1247–1250. Perkins Caufield Byers. Menlo Park, CA. [Online]. Available: http://www.kpcb .com/blog/2016-internet-trends-report [47] S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue, “GALAXY-II: A reference architecture for conversational system development,” in Proc. Int. Conf. [71] Samsung. (2014, Apr. 23). 10 sensors of Galaxy S5: Heart rate, finger scanner Spoken Language Processing, 1998, vol. 98, pp. 931–934. and more. Samsung Newsroom. [Online]. Available: http://news.samsung.com/ global/10-sensors-of-galaxy-s5-heart-rate-finger-scanner-and-more [48] P. S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, “Learning deep ______structured semantic models for web search using clickthrough data,” in Proc. 22nd ACM Int. Conf. Information and Knowledge Management,2013, pp. 2333–2338. SP

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 81

THE WORLD’S NEWSSTAND®

SP EDUCATION

Hana Godrich

Students’ Design Project Series: Sharing Experiences

he fast pace of technology refresh dents in their senior year to a position interplatform computability, and open Toffers new opportunities in engineer- where it can be applied successfully. source software that support the devel- ing training and education. Under- Low-cost, multifeature computer plat- opment of diverse, signal and information graduate engineering programs are forms, being developed at an out- processing-based applications in the In- seeking the right balance between theory standing rate, alongside open-source ternet of things, wearables, three-dimen- and practice, as students are increasingly software, offer implementation capa- sional printing, autonomous vehicles, expected to have more knowledge of bilities like never before. The avail- drones, biomedical devices, and more. cutting-edge technologies along with the ability of high-performance, low-cost, Embedded systems such as smart- fundamental understanding of engineer- and small footprint single-board com- phones, smart watches, and health- ing concepts and modeling tools. puters and microcontrollers systems tracking wristbands, are part of our Instructors constantly look for opportu- presents versatile building blocks that daily lives and, at times, offer a more nar- nities to enrich the learning experience can be used by the students based on row application use and limited resourc- through demonstrations and hands-on their individual interests and applied es. They are commonly used in telecom/ experience to address a growing demand in multidisciplinary projects. As an datacom, automation, military, medi- from the industry for engineers with example, Raspberry Pi [1] was intro- cal, and automotive applications. Some “know‐how” skills. This can be accom- duced to the market in 2012 and has companies open their systems to the re- plished through traditional channels such since gone through three generations search and development of new applica- as instructional laboratories, independent with the latest being Raspberry Pi tions. Google released the source code study, and research activities or, alterna- 3B. In a mere 8.56 # 5.65-cm pack- for Google Glass [5], declaring it as an tively, through student design project age, a weight of 45 g, and a price tag open-to-hackers platform. Apple pro- programs. In the United States, many of US$35, Raspberry Pi 3B offers a vides access to its sensory system and universities and colleges have programs 1.2-GHz 64-bit Quad Core ARM Cor- enables turning an iPhone into a medi- for senior design projects in place, also tex-A53 central processing unit (CPU) cal diagnostic device through Apple referred to as capstone projects. These with 1 GB of memory, on-board net- Research Kit [6]. iRobot released the engineering projects are an assimilation work Ethernet, wireless and Bluetooth, Create 2 Programmable robot [7], an of knowledge and capabilities built over 17 GPIO general-purpose input/output open-sourced electronics prototyping the first three years in undergraduate (GPIO) and more. A smaller version of platform that enables integration of studies and, in many cases, are deeply it was introduced in May 2016 in the sensory systems and microcomputers. reliant on signal and information pro- form of Raspberry Pi Zero, offering a These are a few examples in a growing cessing expertise. 1-GHz CPU with 512 MB of memory trend with exciting possibilities for stu- A background in signal and infor- and 40 GPIO for US$5 in a 6.5 cm dents and developers by gaining access mation processing is fundamental to # 3 cm package and a weight of 9 g. to hardware and software tools that can many engineering applications, yet it Intel’s Edison computer module [2], a be manipulated for varies applications is gradually built through a sequence wide range of Arduino products [3], and in computer vision, robotics, machine of engineering courses that bring stu- Texas Instruments [4] introduce similar learning, cybersecurity, biomedical, and opportunities, to name a few. These pop- biometrics, to name a few. ular products are accompanied by large This progress in affordable and ac- Digital Object Identifier 10.1109/MSP.2016.2620157 Date of publication: 11 January 2017 selection of add-ons, built-in libraries, cessible hardware and software tools can

THE WORLD’S NEWSSTAND®

be instrumental in enhancing students’ One of these projects, “Graph Fre- effective means to introduce the rela- hands‐on experience during undergradu- quency Analysis of Brain Signals,” by tionship between data and information, ate studies while building solid theoreti- Leah Goldsberry, Weiyu Huang, and Dr. commonly achieved through the use of cal foundations. A new article series and Alejandro Ribeiro with the Department alternative representation of data, the contributions platform has been devel- of Electrical and Systems Engineering in project is designed to explore the re- oped by a team of guest editors for IEEE the University of Pennsylvania, Philadel- lationship between neuroscience and Signal Processing Magazine who wish phia (http://arxiv.org/pdf/1512.00037v2 graph Fourier transform (GFT). The no- to promote the sharing of experiences __.pdf), is evaluating the practice of graph tion of the GFT and graph filters is pre- and best practices with undergraduate signal processing methods in neurosci- sented to decompose a given subject’s studies, engineering projects, research, ence. Motivated by the need to identify brain signal into sections that represent and innovation in diverse signal and information processing applications. Stu- dents and advisers have been invited to Nodal contribute information and practices in Time Series engineering design projects in SigPort Time (https://sigport.org/events/spm-student-______Windows design-project-series#citation-ieee).______The r = 0.62 r = 0.35 r = 0.28 r = 0.31 Pairwise goal of this initiative is to start a dis- Coherence cussion on the role of experimental and Connectivity project-based practices in modern sig- Network nal and data processing education and Brain Network Averaged support fast-track progress by sharing Nodes Network “know-how” experience. The SigPort submissions give a first glimpse into the (a) potential of this idea. With more than Graph Signal Representation of Nodal Time Series 400 downloads within a time span of t just a few weeks, it seems that there is a need within the signal and information t processing community that should be Average Network t+1 further explored and advanced. The projects submitted as the first t+1 wave of response to the call for contri- (b) butions articulate the field’s diversity and the multitude of applications, meth- FIGURE 1. (a) A brain network representation using average functional coherence values. (b) A brain ods, and hardware and software tools graph signal using regional fMRI data for each time point t. See the original figure and report at explored by students and faculty. https://sigport.org/documents/graph-frequency-analysis-brain-signals.

0 0.08

(a) (b) (c)

FIGURE 2. An example distribution of decomposed signals across all brain regions for the first experiment. Average energy with respect to (a) xL, (b) xM, and (c) xH. A thresholding is applied.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 83

THE WORLD’S NEWSSTAND®

various modes of variability. The student addressed in a separate project, “Real- clear scope of work in terms of reliabil- may then autonomously explore differ- Time Control of Hand Prosthesis Using ity, portability, gestures support, inter- ent frequencies or temporal variability. EMG,” by Or Dicker, Aviv Peleg, Dr. face quality, and setup time along with Functional magnetic resonance imag- Tal Shnitzer, Dr. Oscar Lichtenstein, a low price target. The Intel Edison ing (fMRI)-based experiments studied and Dr. Yair Moshe with the Signal and board is used along with EMG sensors the response to visual cues over a train- Image Processing Lab, Andrew and (Myo armband) and a printed prostatic ing period, aiding in the modeling of Erna Viterbi Faculty of Electrical Engi- hand. The project includes data collec- average connectivity in brain networks. neering, Technion–Israel Institute of tion from the EMG sensors sampled at Filtering is then used to decompose the Technology, Haifa, Israel (https://_____ 200 Hz and communicated to the

graph signals to look for a correlation ______sigport.org/documents/real-time- microcomputer via Bluetooth. Data is between the decomposed signals and a ______control-hand-prosthesis-using-emg). processed to determine the required subject’s performance when learning a This project is motivated by the need to hand gesture via the implementation of task (see Figures 1 and 2). design a low-cost multifunctional alter- simple feature extraction at each time Hand prosthesis controls, based on native for high-cost prostheses for segment. The mean absolute value electromyography (EMG) signals, are below-the-elbow amputees. It sets a method was found to perform best for this case. Each time segment was classified to one of six gesture classes using K-nearest neighbors. The prototype was tested to demonstrate high classification BTBT PWM success rates and multiple gestures support with a low cost (evaluated at US$345). Figures 3–5 illustrate the design and functionalities. Another closely related project, “Micro Hand Gesture Recognition FIGURE 3. An overview of the project “Real-Time Control of Hand Prosthesis Using EMG System.” System Using Ultrasonic Active Sens- ing Method,” by Yu Sang, Quan Wang, Input Output and Dr. Yimin Liu with the Intelligent Sensing Lab, Department of Elec- tronic Engineering, Tsinghua Uni- versity, Beijing, China (https://www______.youtube.com/watch?v=8FgdiIb9WqY;

h______ttps://sigport.org/documents/micro- ______hand-gesture-recognition-system-using-

ultrasonic-active-sensing-method),______looks into the use of ultrasonic active sensing methods for microhand gesture recognition. The pulsed radar signal processing technique is used to obtain time-sequential range-Doppler features. Object distance and velocity FIGURE 4. The six implemented hand gestures. are measured through a single channel to reduce hardware complexity. A hidden Markov model approach is used to classify time-sequential range-Doppler features. A state transition mechanism significantly compresses the data and extract intrinsic signatures. A real-time prototype was developed and an average recognition accuracy of 90.5% for seven gestures was achieved. Related works include the WiSee gesture recognition system developed in 2013 by Patel et al. [8] to leverage wireless signals for home sensing and recognition under complex FIGURE 5. The constructed prosthetic hand. conditions and Google’s Soli project that

84 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

uses wearable and micro hand gestures to control smart devices [9]. Figures 6–8 illustrate the design and functionalities. A tutorial for a do-it-yourself Sky Im- ager is offered in “DIY Sky Imager for Weather Observation,” by Soumyabrata Dev (Nanyang Technological Univer- sity), Florian M. Savoy (University of Il- (a) Finger (b) Button On (c) Button Off linois at Urbana Champaign‘s Singapore and Advanced Digital Sciences Center), Dr. Yee Hui Lee (Nanyang Technologi- cal University), and Dr. Stefan Winkler (University of Illinois at Urbana Cham- paign‘s Singapore and Advanced Digital Sciences Center); for more information see https://github.com/FSavoy/DIY-sky- (d) Motion Up (e) Motion Down (f) Screw imager____ and https://sigport.org/documents/ FIGURE 6. Examples of micro hand gestures, named by (a) finger, (b) button on (BtnOn), (c) button ______diy-sky-imager-weather-observation. off (BtnOff), (d) motion up (MtnUp), (e) motion down (MtnDn), and (f) screw.

Frame 8 Reflection Intensity Contour Frame 16 Reflection Intensity Contour

0.08 0.08 Index Finger 0.07 0.07

0.06 State 1 0.06 State 4 0.05 Thumb 0.05 Range at m 0.04 Range at m 0.04 State-1 State 3 Noise State 3 0.03 State-1 0.03 State-1 0.02 Noise 0.02

–0.4 –0.3 –0.2 –0.1 0 0.1 0.2 0.3 0.4 –0.4 –0.3 –0.2 –0.1 0 0.1 0.2 0.3 0.4 Velocity at milliseconds Velocity at milliseconds

Frame 16 Reflection Intensity Contour Frame 25 Reflection Intensity Contour

0.08 0.08

0.07 0.07

0.06 0.06 State 4 0.05 0.05 State 9 Range at m 0.04 Range at m 0.04 State 3 0.03 0.03

0.02 0.02

–0.4 –0.3 –0.2 –0.1 0 0.1 0.2 0.3 0.4 –0.4 –0.3 –0.2 –0.1 0 0.1 0.2 0.3 0.4 Velocity at milliseconds Velocity at milliseconds

FIGURE 7. Range-Doppler feature frames sampled in the “button off” (see Figure 1) gesture. (a) Index finger and thumb separate at a 3-cm distance. Noises will be removed using time smoothing to increase robustness as the three marked “noise” objects in the first subfigure. (b) and (c) The index finger is moving down with acceleration while the thumb almost keeps static with a tiny velocity moving up. (d) Two fingers get touched. Note that the object’s trajectory will always be a curve in the range-Doppler plane. The symbolized states are labeled at the center of the detected objects.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 85

THE WORLD’S NEWSSTAND®

As shown in Figure 9, this image pro- erwise be a high-cost system. The proj- ate a segmented binary image. A simple cessing application is Raspberry Pi- ect involved a digital single-lens reflex algorithm is employed to calculate the based and incorporates off-the-shelf camera and image processing per- instantaneous cloud coverage. This is components to replace what would oth- formed by the microcomputer to gener- useful for regular monitoring of cloud formation over a region. Chien-Sheng Yang and Dr. Lav R. Varshney with the University of Il- A12 A23 linois at Urbana-Champaign present a more research-oriented project in Hidden State Z 1 Z2 Z3 “Self-Sustainable OFDM Transmis- A21 A 32 sions with Smooth Energy Delivery” Φ Φ Φ Φ 12 Φ 22 Φ 32 Φ (https://sigport.org/documents/self-______Multinominal HMM 11 21 23 33 sustainable-ofdm-transmissions-smooth-______

energy-delivery).______This project studies the question: “Is it possible to have Symbol Sequence S S S 1 2 3 small peak-to-average power ratio (PAPR) in the cyclic prefix of an OFDM Feature Extraction Symbolize Symbolize Symbolize signal, while maintaining self- sustain- State 8 ability?” A new system architecture, Range-Doppler Plane shown in Figure 10, is proposed that State 1 State 4 employs a frame-theoretic method that demonstrates significant improvement Object Track Object Track in PAPR of the cyclic prefix in self-sustainable OFDM. FIGURE 8. Feature extraction and classification workflow. Audio processing is at the core of the “Automatic Lyrics Display System for Live Music Performances” project by Karan Vombatkere, Bochen Li, and Dr. Zhiyao Duan with the University of Rochester, New York (https://sigport______

.org/documents/automatic-lyrics-dis-______play-system-live-music-performances).______The primary objective of the project is to design and implement a computational system that can follow live music performances (e.g., choruses) in real FIGURE 9. The design of the DIY Sky Imager. time and display pre-encoded lyrics for

E2 Lowest Cyclic Data Channel IDFT PAPR Prefix Expansion t Selection

EP Cyclic Prefix Energy Harvester

Cyclic Erasure Decoding DFT Prefix Reconstruction IDFT Equalization DFT Retrieval

FIGURE 10. A self-sustainable OFDM system with EPS-CP for PAPRCP reduction.

86 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Audio Live Music Live Audio Recording Live Audio Chroma Input Waveform Data Thread Chroma Vector Chroma Real-Time Dynamic Feature Time Extraction Warping Reference Audio Chroma Vector Reference Audio Waveform Data Reference Audio

Import Reference Alignment Instance Lyrics Data Lyrics Prealigned Display with Reference Lyrics in Real Output Lyrics Aligned with Live Music Audio Time

FIGURE 11. Lyrics display system flowchart.

the audience. As basic harmonic pro- on the NI compact reconfigurable tools. Some concentrate on developing gression of two recordings is similar, input-output setup. With a sampling devices that offer a solution for so- the chroma feature is used to represent frequency of 100 KHz per channel and cial or environmental needs. Others the audio data. It uses 12 bins to rep- on-board field-programmable gate ar- are focused on mathematical model- resent the relative energy of the audio ray unit programmed to minimize la- ing and research to advance existing in the 12 semitones of a musical octave, tency, the collected data was process technology. A tutorial project offered represents the harmonic content of the to estimate the source angle-of-arrival. step-by-step guidelines for building audio. A prerecorded version of a con- The sine sweep method was found to be an image acquisition and processing cert serves as reference for lyrics align- the most effective, where a sinusoid of platform. Many projects concentrated ment. This is performed through online temporally increasing frequency, both on embedded systems and all incorpo- dynamic time wrapping. See the design linearly and exponentially, is applied rated signal and information process- and interface shown in Figures 11 and via accurate speakers. ing methods commonly thought in 12, respectively. This first set of project highlights fundamental courses. A National Instruments (NI)-based manifest a diverse collection of un- The main objective of this effort is project is detailed in “Acoustic Detec- dergraduate engineering projects and to expand the knowledge base in the tion and Localization of Impulsive Events in Urban Environments” by Sabeeh Irfan Ahmad, Hassan Shah- baz, Hassam Noor, Dr. Momin Up- pal, and Dr. Muhammad Tahir with the Syed Babar Ali School of Science and Engineering, Lahore University of Management Sciences, Pakistan. The project focused on the detection of impulsive acoustic events and localizing the source in an urban environment. As illustrated in Figure 13, the system used an array of microphones that recorded the sounds and transmitted raw data to a central fusion center, based FIGURE 12. A graphic user interface.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 87

THE WORLD’S NEWSSTAND®

community and shorten the learning curve in the academia while encouraging a discussion in the community on education and engineering studies allowing for rethinking teaching method to a promote more disruptive and multidisciplinary engineering.

Author Hana Godrich ______

for Gunshot Identification for ([email protected] Statistical+Signal Processing

Source and Shoots it. ___.edu) received the B.Sc. degree in elec-

to the Rig, Whichto the Rig, the Tracks trical engineering from the Technion

Gunshot Localization is Performed, Israel Institute of Technology, Haifa, in Coordinates are Calculated and Sent 1987; the M.Sc. degree in electrical engineering from Ben-Gurion Uni - versity, Beer-Sheva, Israel, in 1993; and the Ph.D. degrees in electrical engineering from the New Jersey Institute of Technology, Newark, in 2010. She is the undergraduate program director in the Electrical and Computer Engineering department at Rutgers University, Source Vector Piscataway, New Jersey. Her research Compact Reconfigurable The National Instruments interests include statistical signal processing with application to wireless sen- Input-Output (cRIO) Acquires and the Incoming Signals and Stores A Computer Samples and Stores it for Offline Algorithmic Processing it for Processes the Signals, Locating the Processes the Signals, sor networks, communication, smart grid, and radar systems. She is a Senior Member of the IEEE.

References [1] RaspberryPi. [Online]. Available: https://www______.raspberrypi.org/

[2] RaspberryPi. [Online]. Available: https://github______.com/raspberrypi/documentation/blob/master/setup/ ______README.md [3] Arduino. [Online]. Available: ______https://www.arduino .cc/en/Guide/Windows;______[4] Texas Instruments, Inc. [Online]. Available: http://www.ti.com/ Sound

[5] Google.com. [Online]. Available: https://developers______.google.com/glass/ Microphone Array

Detects the Incoming [6] Apple.com. [Online]. Available: http://www .apple.com/researchkit/ [7] iRobot.com. [Online]. Available: http://www .irobot.com/About-iRobot/STEM/Create-2/Projects .aspx__ [8] Q. Pu, S. Gupta, S. Gollakota, and S. Patel, “Whole-home gesture recognition using wireless signals,” in Proc. 19th Annu. Int. Conf. Mobile Computing & Networking, ACM, 2013, pp. 27–38. [9] Google ATAP. (May 29, 2016). Project soli [Online]. Available: https://atap.google.com/ ____soli/2015 Sudden The test bed for data acquisition and offline processing. Sound Wave and Produces a A Gun Fires Toy SP FIGURE 13. FIGURE

88 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

TIPS & TRICKS

Denis A. Gudovskiy and Lichung Chu

An Accurate and Stable Sliding DFT Computed by a Modified CIC Filter

k he sliding discrete Fourier transform Introduction Xkn ()=+-WM ^hXkxxnnnM--1 () . T(SDFT) is a popular algorithm used In this article, we present a method (2) in nonparametric spectrum esti- to improve the SDFT algorithm. The mation when only a few frequency bins improved algorithm is called cascade Equation (2) can be implemented as a of an M-point discrete Fourier transform integrator-comb (CIC)-SDFT, which is filter with a comb stage followed by an (DFT) are of interest. Although the clas- based on extending the idea established integrator stage, as shown in Figure 1(a). sical SDFT algorithm described in [1] is in the mSDFT algorithm and mostly This conventional SDFT filter has a computationally efficient, its recursive in the context of spectrum estimation z-domain pole on the unit circle located k structure suffers from accumulation and application. Similarly to mSDFT, we at z = WM. Hence, it is only marginally rounding errors, which lead to poten- move the DFT bin of interest k to the stable in finite precision recursive cal- tial instabilities or inaccurate output. zero position to exclude complex coef- culations with accumulation, except at Duda [2] proposed a modulated SDFT ficient multiplication in the recursive points when poles z =!1 or z =!j . (mSDFT) algorithm, which has the prop- stage and avoid instabilities. In addi- Duda [2] proposed to shift the Xk ( ) erty of being guaranteed stable without tion, CIC-SDFT comprises a modi- DFT bin of interest to the k = 0 bin prior sacrificing accuracy, unlike previous fied CIC filter structure proposed by to calculating the comb stage in SDFT. approaches described in [1], [3], and Hogenauer [8]. Two goals are achieved Thus, the Xkn () calculation is simpli- [4]. However, all of these convention- using this approach. First, the accuracy fied to multiplication of input signal -km al SDFT methods presume DFT compu- of spectrum estimation is improved by xn by the modulation sequence W M , tation on a sample-by-sample basis. This using high-order CIC filters without followed by calculation of a new zero- is not computationally efficient when the computationally expensive windowing frequency Yn ()0 DFT bin expressed as DFT needs only to be computed every in the frequency domain as described in RR^h2 1 samples. To address such [1]. Second, the complexity of the SDFT Xknnn()== Y ()00 Y--1 () +- y nnM y , cases when R-times downsampling is can be further decreased by reducing (3) needed, Park et al. [5] proposed a hop- the DFT output rate, also achieved by -km ping SDFT (HDFT) algorithm. Recent- [5]–[7]. where yn = W M xn and ynM- = --km( M) ly, Wang et al. [6] presented a modulated W M xnM- . HDFT (mHDFT) algorithm, which com- SDFT and mSDFT The mSDFT structure is shown bines the HDFT algorithm with the The kth frequency bin of an M-point in Figure 1(b). Complex multiplica- mSDFT idea to maintain stability and DFT at time index n for input signal x tion in the integrator stage is unneces- 0 accuracy at the same time. In parallel, is defined by sary because W M = 1. Therefore, the Park [7] updated the HDFT algorithm mSDFT filter becomes guaranteed- with its guaranteed stable modification M- 1 stable and accurate at the same time. In -km called gSDFT, which exists only for cer- Xkn ()= / W M xqm+ , (1) addition, complex multiplication in the tain M and L relationships. m = 0 recursive stage might limit the clock rate of the digital circuit, which is avoided in where qnM=- +1, 01##kM- , this method by effectively moving it into and the complex exponential factor the feedforward part. = j2r/M Digital Object Identifier 10.1109/MSP.2016.2620198 WM e . A recursive equivalent of The drawback of the mSDFT algo- Date of publication: 11 January 2017 (1) is given by rithm compared to the conventional

THE WORLD’S NEWSSTAND®

- -ML M- 1 L = ()1 z = eo/ -m k Hz() -1 L z . WM ()1 - z m = 0 (5) It is known that the magnitude response Xn (k) xn of the CIC filter evaluated at ze= j~ can be written as sin(/)M~ 2 L H()~ = , (6) z –M z –1 sin(/)~ 2 xn–M Xn–1 (k) (a) where ~ is a normalized angular frequency and -r~r## radians/ x X (k) n yn n sample. In this case, the magnitude response H()~ of the CIC filter is the magnitude response W()~ of a window function wm() applied to a discrete- z –M z –1 –km WM yn–M Xn–1 (k) time Fourier transform (DTFT), when (b) a finite length DFT is being computed. Note that the computed CIC-SDFT FIGURE 1. SDFT filter structures: (a) conventional and (b) mSDFT. magnitude response should be normalized according to the CIC decimator gain, which is equal to ML. SDFT is the necessity to generate a mod- M-1 = / + -km Yynqm()0 . (4) ulating sequence W M instead of keep- m = 0 Window function k ing a fixed twiddle factor WM . Note that The described CIC-SDFT algorithm the output of the mSDFT in Figure 1(b) Equation (3) rewrites the moving aver- provides two salient features. First, -+km()1 has W M phase shift compared to age filter (4) in recursive form, which, it improves spectrum estimation per- DFT, which does not have an effect on indeed, is a first-order CIC filter with- formance using a naturally embed- magnitude spectrum estimation appli- out rate change. Using (3) and (4), we ded B-spline window function wm(), cations. The last two topics are not cov- can generalize the mSDFT idea and which is defined as self-convolution ered in this article since they are well apply the CIC filter theory [8] for DFT of L length-M rectangular functions. described in [2]. spectrum estimation. Equation (6) shows that, for L = 1, The general structure of CIC-SDFT CIC-SDFT provides an exact DFT spec- CIC-SDFT is depicted in Figure 2, which contains trum. However, the spectral leakage can complex multiplication of input signal be reduced by increasing the CIC filter -km Recursive structure xn by the modulating sequence WM , order L, which is equivalent to a higher- One can note that the mSDFT filter followed by a CIC decimation filter with order filter magnitude response W().~ structure depicted in Figure 1(b) looks R rate change. The CIC decimator con- For example, it can be shown that exactly like a modified first-order CIC tains an integrator section with L inte- WL = 1 ()~ corresponds to a rectangular filter with an additional complex multi- grator stages, a downsampler by R, and window of length M in the time domain, -km plication by WM . This is not a coin- a comb section with L comb stages. The WL = 2 ()~ corresponds to a triangular cidence, since the mSDFT calculates CIC filter part is equivalent to a cascade window of length (21M - ), and so on. the Yn ()0 (zero frequency) DFT bin of L moving average filters with transfer In general, windowing is an given by function expressed as expensive operation. Conventional time- domain windowing would compromise computational simplicity of the SDFT algorithm. Hence, frequency-domain ~ x y CIC X (k) convolution of adjacent DFT outputs n n I I C C n 0 L–1 R 0 L–1 with another window function was proposed in [1]. Practically, it is limited to only short window functions with W –km M preferably power-of-two coefficients because SDFT complexity grows M faster than a linear function of win- z –1 – z R dow length. For comparison, Figure 3 illustrates several DFT magnitude FIGURE 2. The recursive CIC-SDFT filter structure. responses for M = 32. First, it shows

90 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

three normalized magnitude responses of WL = 123,, ()~ window functions gen- 0 erated by CIC-SDFT with lengths M, WL = 1 (M) W (),21M - and (32M - ), respectively. –10 L = 2 (2M–1) WL = 3 (3M–2) The last plot depicts the magnitude WHanning(5) response of the optimized five-point –20 Hanning window described in [1], which has three nonzero coefficients –30 –1/4, 1/2, and –1/4. As can be seen, –40 CIC-SDFT realizes a powerful window function compared to the short –50 Hanning window. For example, an Magnitude (dB) interference that falls into the first sid- –60 elobe will be attenuated by 13, 26, and 39 dB for L = 123,,, respectively. –70

Estimator variance –80 Assuming that the input signal x is cor- 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Frequency (ω/π) rupted by additive white Gaussian noise (AWGN) with variance v2 = 1SNR/ , where SNR is the signal-to-noise ratio, FIGURE 3. The window function comparison. the CIC-SDFT variance can be written as Computational complexity can be accomplished by a number of t var{()}/Xkn = C SNR , (7) The second feature of CIC-SDFT is approaches. For example, a simple table- reduced computational complexity for based method can be used for some appli- where coefficient C depends on the R > 1 cases. When R = 1 and L = 1 , cations to generate the complex exponent -km convolution of L rectangular windows. CIC-SDFT performs the same num- WM or, in the general case, a generator Each rectangular window can be rep- ber of computations as mSDFT. When and a phase shift corrector described in resented as a length-M column vector R > 1, the digital circuit after the inte- [2] can be used. The phase shift correc- of all ones. Then, coefficient C can be grator section operates at a fs /R clock tor is not needed for magnitude spectrum expressed as rate and the memory size in the comb estimation. Note that a conventional section is decreased by a factor of R. SDFT in Table 1 is the method that can- LM()-+11 That is a significant reduction in com- not be guaranteed stable and accurate at C = 1 / 11))f ) 12 . 2L cm123444444 M m = putational complexity and considerably the same time due to recursive complex 0 L m (8) simplifies digital circuit implementa- multiplication. The gSDFT and mHDFT The closed-form expression of (8) can tion. Table 1 summarizes computation- computational workload was recalcu- be written in vector form as al complexity by comparing the number lated for the single-bin case. As can be of complex multiplications, complex seen, these algorithms are not beneficial = LL- 122 CM||Ab || / , (9) additions, and memory size needed to in this configuration unless a significant calculate a single-bin DFT. subset of M bins has to be calculated. On ;1E # where vector b = 0 , 1 is an M 1 In addition, it states whether a par- the other hand, CIC-SDFT is a beneficial column vector of all ones, 0 is an ticular method requires generation of method, when only one or a few bins of ()()LM--111# column vector of all the modulating sequence and the out- an M-point DFT need to be computed zeros, and A is a Toeplitz matrix with b as put phase shift correction or not, which with R downsampling. the first column and 111# ((LM-+ ) ) vector u = [100f 0 ] as the first row. = The case L 1 gives exactly the Table 1. Single-bin DFT computational complexity. variance of a moving average filter 2 Method Mul. Add. Mem. Mod. Seq. Ph. Cor. vL = 1 = 1SNR/(M ). The variance of a DFT MM – 1 0 No No second-order filter is decreased by a factor of 3/2 for any large M. The variance of the SDFT 1 2 M + 1 No No output periodogram for high SNR can be mSDFT 1 2 M + 1 Yes Yes approximated by gSDFT, mHDFT 1 2 M +-21R No No CIC-SDFT 1 Lcm1 + 1 Lcm1 + M Yes Yes t 2 R R var{()}Xkn . 2 C / SNR . (10)

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 91

THE WORLD’S NEWSSTAND®

a cascade of L sections, where each sec- ~ tion contains a comb stage followed by xn yn CIC X n (k) N0 NQ–1 R0 R L–1 an integrator stage. Then, the bit-width increases by only log2 (/)M R =- (PQ ) bits per section. The total bit-width at the –km WM output of partially nonrecursive structure is identical to (11). (1 + z –1)L 2 Figure 4 illustrates an alternative M – –1 z R z partially nonrecursive implementation of CIC-SDFT. First, it contains Q nonrecursive stages NN01,f Q- accord- FIGURE 4. The partially nonrecursive CIC-SDFT filter structure. ing to the aforementioned description. Second, nonrecursive stages are followed by a cascade of L recursive --12LLPQ--1 f Partially nonrecursive structure Hz2 ()=+ (11zz )f ( + ) sections RR01,.L - Due to the fact Although recursive CIC-SDFT reduces M/R- 1 L that bit-widths are increased on a -m computational complexity for R > 1, it = eo/ z . per-stage basis rather than all at once = experiences a bit-growth according to m 0 (14) at the input of the CIC-SDFT, the CIC filter theory. Assuming an input computational and circuit complexity bit-width Bin, the bit-width B used for From (13) and (14) it is clear that the decreases. Moreover, since downsam- all computations in the CIC-SDFT can CIC-SDFT transfer function H(z) can pling is performed as early as pos- be expressed as be split into two parts: Hz1 () computed sible, the number of add operations is in nonrecursive fashion and Hz2 () com- minimized as well. Another important BL=+^ log2in() Mh B . (11) puted in recursive fashion. advantage of the structure shown in The first nonrecursive part compris- Figure 4 is the ability to model CIC- Wider bit-widths have to be used as the es Q stages where each stage increases SDFT using floating-point arithmetic, number of stages L and DFT length M bit-width by L bits and downsamples since overflows in the integrator stages grows. This drawback can be solved by the output by two. For each stage, which are avoided. The latter property allows using nonrecursive structures derived calculates (),1 + z-1 L several imple- to implement a rounding operation from polynomial factoring and applying mentations are possible. For example, between algorithm stages. polyphase decomposition [9]. Then for it can be realized as a length-L cascade power-of-two DFT length M = 2P, the of ()1 + z-1 operations or direct expo- Summary transfer function of CIC-SDFT can be nentiation of the whole stage. The latter In this article, a novel SDFT algorithm written as can be expressed as a transfer function called CIC-SDFT was introduced. It HzN (), and for L = 4 generalizes a previous approach by M-1 L = eo/ -m incorporating a modified CIC filter Hz() z =+-14 m = 0 HzN () (1 z ) structure. Such a generalization adds --24 =+()()11zz--12LL + =+16zz + two new programmable parameters: -- --P-1 ++12 filter order and output rate. Filter # ()11++zz42LLf(). 41zz() =+ order is responsible for an embedded (12) HzNN12() Hz (), (15) window function, and therefore deter-

Assuming a power-of-two downsam- where HzN1 () and HzN2 ( ) are new mines the spectrum estimator variance pling factor R = 2Q , (12) can be rewrit- poly phase components. Note that each and interference rejection capabilities. ten as polyphase component may downsample Sidelobe level is proportional to the computations by two prior to perform- filter order as -13L dB for the embed- Hz() ing add operations. The nonrecursive ded window function. The closed-form =+--12LL +g + - 2Q-1 L 123()()()4444444411zz4 44444444 1 z4 part experiences bit growth of only expression for the variance of CIC- Hz1() L log2 (),R which means that the total SDFT estimator and its periodogram # ++--22QPLLg -1 bit-width at its output can be written as was provided. 123()()44444411zz 444444 Hz2() Programmability of the output rate

= HzHz12() (). (13) BLN =+log2in()R B . (16) allows one to decrease algorithm computational complexity when needed. Next, assuming that the Hz1 () output The second recursive part now imple- Specifically, the number of memory is followed by a downsampler by R, the ments an Lth-order moving-average filter cells and half of the add operations in transfer function Hz2 () in (13) can be of length M/R with the transfer function recursive CIC-SDFT are inversely pro- simplified to Hz2 (). Such a filter can be realized using portional to the downsampling factor R.

92 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Bit growth can be minimized using Lichung Chu (lichung.chu@olympus______Signals, Systems, and Computers, vol. 1, Nov. 1997, pp. 111–115. the presented partially nonrecursive ___.com) is a director of engineering with [5] C. Park and S. Ko, “The hopping discrete Fourier structure of CIC-SDFT, which is suit- Olympus Communication Technology transform,” IEEE Signal Process. Mag., vol. 31, no. able for digital circuit implementation of America, San Diego, California. 2, pp. 135–139,Mar.2014. and algorithm modeling using floating- [6] Q. Wang, X. Yan, and K. Qin, “High-precision, permanently stable, modulated hopping discrete point arithmetic. References Fourier transform,” IEEE Signal Process. Lett., vol. [1] E. Jacobsen and R. Lyons, “The sliding DFT,” 22, no. 6, pp. 748–751, June 2015. Acknowledgments IEEE Signal Process. Mag., vol. 20, no. 2, pp. [7] C.-S. Park, “Fast, accurate, and guaranteed stable 74–80, Mar. 2003. sliding discrete Fourier transform,” IEEE Signal We would like to thank Dennis R. Mor- [2] K. Duda, “Accurate, guaranteed stable, sliding dis- Process. Mag., vol. 32, no. 4, pp. 145–156, July 2015. gan for improving this manuscript. crete Fourier transform,” IEEE Signal Process. Mag., [8] E. Hogenauer, “An economical class of digital fil- vol. 27, no. 6, pp. 124–127, Nov. 2010. ters for decimation and interpolation,” IEEE Trans. [3] E. Jacobsen and R. Lyons, “An update to the slid- Acoust., Speech, Signal Process., vol. 29, no. 2, pp. Authors ing DFT,” IEEE Signal Process. Mag., vol. 21, no. 1, 155–162, Apr. 1981. pp. 110–111, Jan. 2004. Denis A. Gudovskiy (denis.gudovskiy@______[9] R. Losada and R. Lyons, “Reducing CIC filter [4] S. Douglas and J. Soh, “A numerically-stable slid- complexity,” IEEE Signal Process. Mag., vol. 23, no. us.panasonic.com)______is a design engineer ing-window estimator and its application to adaptive 4, pp. 124–126, July 2006. with Panasonic, Cupertino, California. filters,” in Proc. 31st Annual Asilomar Conf. on SP

ERRATA

he author order in the article “Smart incorrectly due to a production error. We Reference Driver Monitoring: When Signal apologize for any confusion this may [1] A. S.Aghaei,B.Donmez,C. C.Liu,D.He,G. T Liu, K. N. Plataniotis, H. Y. Winnie Chen, and Z. Processing Meets Human Factors” have caused. The correct order of the Sojoudi, “Smart driver monitoring: When signal pro- IEEE cessing meets human factors,” IEEE Signal Process. in the November 2016 issue of byline is as follows: Mag., vol. 33, no. 6, pp. 35–48, Nov. 2016. Signal Processing Magazine [1] printed Amirhossein S. Aghaei, Huei-Yen Winnie Chen, George Liu, Cheng Liu, Zohreh Sojoudi, Dengbo He, Birsen Don- Digital Object Identifier 10.1109/MSP.2016.2636256 Date of publication: 11 January 2017 mez, and Konstantinos N. Plataniotis. SP

______

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 93

THE WORLD’S NEWSSTAND®

LECTURE NOTES

S.Y. Kung

Compressive Privacy: From Information/Estimation Theory to Machine Learning

ost of our daily activities are now remain highly vulnerable to unauthor- case of a bomb explosion, images from Mmoving online in the big data era, ized leakages and hacker attacks. various mobile sources near the crime with more than 25 billion devices scene may be collected by authorities for already connected to the Internet, to pos- Data owner should have wide-scale forensic analysis. Ideally, the sibly over a trillion in a decade. Howev- control over data privacy uploaded images should provide critical er, big data also bears a connotation of Privacy-preserving (PP) tools have a and relevant information to help capture “big brother” when personal information broad spectrum of applications, cover- the targeted suspects while protecting (such as sales transactions) is being ubiq- ing numerous types of Internet data the full facial images of the innocent uitously collected, stored, and circulated (such as speech, image, location, and from being leaked to the public. around the Internet, often without the media/social/health data). They all Data are not just a collection of data owner’s knowledge. Consequently, a require a delicate balance between uti- words/numbers working in isolation, new paradigm known as online privacy lization and privacy. For example, in rather they encompass the global and or Internet privacy is becoming a major concern regarding the privacy of personal and sensitive data. Public Space: Cloud As depicted in Figure 1, Internet data live in two different worlds: 1) the Cloud private sphere, where data owners gen- Server erate and process decrypted data, and Intruder Trusted Authority 2) the public sphere, where the data in cloud servers are presumably encrypted Encrypted and therefore unaccessible by intruders. Adversary Data However, the data may be decrypted by the intended and trusted “authorities,” who will be provided with the right key for the data decryption. Following the cryptographic channel and adversarial Decrypted models, formally defined by Claude Data Shannon, we shall also name the data owner, intended user, and intruder as Alice, Bob, and Eve, respectively. Recall that the typical security protocol hinges Private Space: Clients upon Alice’s passing a decrypting key to Bob but not to Eve. Since the notion FIGURE 1. of an unbreakable key is questionable, In collaborative learning environments, individual data are uploaded to the cloud. From the privacy perspective, data in the private sphere versus the public sphere should be treated differentially, there is no wonder that Internet data which calls for a novel PP encoding paradigm, known as compressive privacy (CP). For privacy pro-

Digital Object Identifier 10.1109/MSP.2016.2616720 tection, the query data uploaded to the public sphere should be designed to retain the information Date of publication: 11 January 2017 useful for the intended application and should not be easily repurposed into malicious exploitation.

THE WORLD’S NEWSSTAND®

highly coordinated control of information. PP protocols allow the data owner Utility Space to control the fate of the data, instead of chancing with the protection prom- ised by the cloud severs. To this end, Bob (u) the Defense Advanced Research Proj- I(u;y) ects Agency (DARPA) has championed a major Brandeis Program to develop Query (y) Alice (x) novel communication protocols so that I(p;y) the uploaded data are useful only for the intended utility but not easily repurposed Feature Space into privacy intrusion. Eve (p)

Privacy communication paradigm Privacy Space Note that, as depicted in Figure 2, both FIGURE 2. Our study on privacy preservation involves joint optimization over three design spaces: Bob and Eve will receive the same data 1) the feature space (Alice), 2) the utility space (Bob), and 3) the privacy space (Eve). Alice (the (denoted by y), i.e., there is no key data owner) wants to convey certain information relevant to Bob (the intended user/utilizer) while preventing it from being eavesdropped by Eve (the intruder). In the CP paradigm, both Bob and Eve required. Ideally, in this paradigm, the will receive the same data (denoted by y), i.e., there is no key required. We propose a query encod- query y should be useful to (friendly) ing scheme which is 1) information preserving from the utility’s perspective but 2) information lossy Bob but useless to (malicious) Eve. from the perspective of privacy. Collectively, such a scheme is called CP. This article explores the More realistically, we would like to see utility-privacy tradeoff analysis via comparing the I (uy ; ) and I (py ; ). to it that the query may retain information as lossless as possible to Bob and, ries, however, the DP approach remains While formal courses on information at the same time, be as lossy as possible somewhat unwieldy for many real-world theory, estimation theory, and machine against Eve. applications. This prompts us to explore learning are highly recommended for In short, it should also be recognized other desensitizing methods for privacy advanced and serious researchers, we that the security protocols for data shar- preservation. shall nevertheless review the basic mate- ing are neither necessary nor sufficient rials for novice readers, hopefully mak- for data privacy protection. Therefore, it Compressive privacy ing the article somewhat self-contained. is worth installing both the security and The query, denoted by y , is represented privacy protocols for maximal protec- by yx= f( ,f ) where x denotes the fea- Information and estimation theory tion of Internet data. ture vector representing the original data Let the original data (owned by Alice) and f is an independent random noise. be represented by a vector space con- Differential privacy and Unlike DP, the CP approach allows the taining M-dimensional random vectors compressive privacy query to be tailor-designed according T to the known utility and privacy mod- x = [,xx12 ,f , xM ]. Differential privacy els. As depicted in Figure 2, we propose In the differential privacy (DP) theory a query encoding scheme that is 1) To convey information concerning x, [1], a desensitizing function K is information preserving from the utili- the design of PP query y must be based said to provide e-DP of the data if ty’s perspective but 2) information lossy on joint consideration of both the utility Pr[()K DSePr!# ]e [(K DSl ) ! ] for from the privacy’s perspective. Collec- maximization (for Bob) and the privacy all S ! Range(K ) and all data sets D tively, such a scheme is called CP. This protection (against Eve). Mathematically, and Dl differing by one entry. Note also article explores the tradeoff analysis ■ Utility function: The utility func- that the DP sensitization does not neces- between the utility mutual information tion is denoted by ux ( ). sarily require the utility function to be (between y and Bob) and its privacy ■ Privacy function: The privacy func- known in advance, and DP guarantees counterpart (between y and Eve). This tion (i.e., cost function) is denoted by that the distribution of the search result is in a sharp contrast to most machine- px(). should be indistinguishable (modulo a learning problems, where the design factor of ee) with or without the missing goal is exclusively focused only on the Prior work on non-Gaussian models entry. Two types of “differential-log-like- utility information. A natural formulation for the utility-pri- lihood” criteria, e-DP versus e-informa- vacy tradeoff analysis involves optimiz- tion privacy (a stronger privacy metric), Scope and prerequisite of the article ing I(;)uy [respectively, I (py ; )] while are analyzed and compared in the study This article explores the rich synergy setting a bound on I(;)py [respectively, of the so-called privacy funnel in [2]. between information theory, estimation I(;)].uy More specifically: Due to the absence of systematical meth- theory, and machine learning and, ulti- ■ Information bottleneck (IB) [4].In ods for the derivation of optimal que- mately, develops a PP methodology—CP. the IB scenarios, it is assumed that

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 95

THE WORLD’S NEWSSTAND®

#n = ! 0M T px() x , i.e., Eve has no additional U is the projection matrix Rxu = E[|]xxuu y role here. Two specific examples are characterizing the utility subspace. -1 -2 T -1 =+^hRx v ff as follows: ■ Privacy subspace. The cost (i.e., =-RR6v2 +TTRR@-1 – Alice wants to transmit as much privacy) function is represented by xxffffxx. (2) information as possible, relevant to pPx= T , where P ! 0M# o is the Bob, while consuming minimum matrix characterizing the privacy This leads to a postquery Gaussian dis- t resources on bandwidth/storage. subspace. (Here we shall simply tribution for x , denoted by N^hx ,Rxu . – Alice wants to transmit as much assume the subspace projection For multiquery cases, just change f " Mm# 2 m#m information as possible, relevant matrices U and P as given, leaving F ! R and v " / e ! R . to Bob, while preventing the origi- their learning/estimation strategies a nal data from being reconstructed later discussion.) Effect on the estimation error or leaked. of the original feature vector The objective is to design the query Gaussian distribution Conventionally, we would like to maxi- y, which maximizes the utility gain, with linear optimal query mally preserve the fidelity, i.e., to best prescribed by ux(), while keeping the The linear query is represented by reconstruct the original data. In this data bandwidth below a certain bound. case, natural formulation for the optimal ■ Privacy funnel (PF) [2]. In this sce- y =+fxT f, (1) query vector(s) is as follows: nario, it is assumed that ux()= x , argmaxfx {()} trace R u (3) i.e., Bob has no role here. Now Alice where we assume that f = 1 and f wants to transmit the original data, as is an independent random noise, with whose optimal solution lies exactly on 2 2 much as possible, while preventing variance vvf = , all without loss of the principal component analysis (PCA) leaking sensitive data to Eve. (Recall generality. eigen-subspace. that Eve may perform any adversarial The amount of information con- inference attack, based on the query tained in x (as well as u and p) can be Effect on the utility and y, to intrude the privacy prescribed quantitatively measured by its entropy privacy entropies bypx ( ).) Therefore, we must design It is obvious that the additional query a query y that can minimize the pri- Hd()xxxx/- # pp ()log () , knowledge can only reduce the entro- vacy leakage while assuring a guar- py of u and p . The utility and privacy anteed level of information on the where the integration is taken over the functions are linear functions of the state original data is being conveyed. M-dimensional vector space. Assume vector x, so they are Gaussian distrib- Most analysis regarding IB [4] and that x has a Gaussian distribution uted with the utility covariance matrices t t PF [2], especially those pertaining to N^hx0, Rx , where x0 and Rx are the (before and after the query) given as the convex optimization, may be natu- mean and covariance matrix of x . The TT rally extended to the general case when entropy and covariance matrix are RRux/ UU and RR uuu= UU x(4) ux()! x and px ( )! x , i.e., both Bob closely connected and Eve have their own specific goals. and the privacy covariance matrices as 1 M H()x =+log22^hRx log (2re ). 2 2 TT Gauss–Markov estimation theorem RRpx//PPand RR pxuu PP . (5) The assumption of Gaussian distribution of the data is vital to our development Effect of query on covariance matrix Example 1. The double income problem (DIP) t of the CP theory. More specifically, Assume that E [xx ] = 0 and E [(x - Here, a two-dimensional vector ttT T it allows us to make use of the results xxx00)(-= ) ]Rx . Before query, the x = []xx12 represents the two indi- t that 1) the amount of information can initial estimate of x is x0 , and the ini- vidual incomes of a couple. be quantified as a log function of its tial error-covariance-matrix (ECM) is ■ From the utility perspective, to ttT variance and 2) the difference of vari- E[(xxxx--=00 )( ) ] Rx. Now that assess the family’s total income, the ances (before-and-after-query) can be we are given the knowlege of the query, utility function should be set as T derived via the classic Gauss–Markov y =+fx f. According to the well-known uux==+()xx12 . estimation theorem [3]. As proven next, Gauss–Markov theorem; cf. [3, ch. ■ From the privacy perspective, the the Gaussian assumption leads us to a 15], the optimal estimation of x is query should not pry into the in come simple (eigenvalue-based) optimal solu- disparity between the couple. To pro- t = tion in closed form. xxyE[|] tect such privacy, the privacy function tt-2 -1 -2 TT-1 Now, as depicted in Figure 2, the =+xf0 vv^hRx + fffx[].y - 0 is set as ppx= ()=-xx12. design will be optimized over two com- Suppose that the initial covariance peting linear vector spaces: Let xxxu / t - denote the updated matrix of x is ■ Utility subspace. The utility function estimation error (given the query y) 8 - 6 Rx = ; E . is represented by uUx= T , where with its corresponding ECM as follows: - 6 10

96 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

T 2ou Note that U = [11 ] and P = douu/ 2^h-=-log , (6) o []11- T and the initial utility and pri- u p(x) 2 vacy variances are respectively vu = 6 which reflects the reduction of the 2 and vp = 30, cf. (4) and (5). entropy due to y, i.e., the difference Let us study two scalar queries: between H(|)uy and H (u ). yx=++08..12 06 xf and yxl = 06 . 1 x ++08..x2 f According to (16), y and Theorem 1. Utility-Driven Differential Information yl Maximization (DIM) deliver almost the same differen- FIGURE 3. . The Gaussian distribution of the tial information gain 27.. However, The query vector f in (1) that maximizes original privacy function is shown by the solid because the spouse’s income is initially the mutual information I(;)uy can be green curve with the initial variance, say, 2 more private than the husband’s (note derived via the following optimizer: vp = 30 as exemplified in our DIP case study. 2 l Two post-query and narrower distribution 10 8), y and y have different effects d = argmaxfu()f curves are: a) dotted red curve with the vari- on the variances of u and p . T 2 = u ff Ω ance reduced to vpu 29 . 3535 after a less in- In Figure 3, the distribution of pp , , argmaxf T 2 , (7) y fI[]Rx + v f trusive query , and b) dashed blue curve with ul 2 and p are respectively represented a further reduced variance of vpul = 25 . 7168 by the solid green, dotted red, and where after a more intrusive query yl . dashed blue curves. More specifically, -1 T Ω / RRx UUu Rx (8) yl is more intrusive than y, implying a higher weight placed on the spouse’s denotes the utility amplification matrix income would leak more on the dispar- (UAM). The optimal solution can be H(u⎢y) 2 H(u) ity. Indeed, we have vpu =29. 35 and computed from the principal eigen- 2 vpul =25., 72 cf. (5). On the other hand, vectors of the generalized eigenvalue I(u,y) 2 from the utility perspective, yl outper- decomposition: eig( Ω,Rx + v I ). forms y in producing a narrower util- H(y) 2 2 ity variance: vvul ==172..;1 u 196 Proof of theorem 1 cf. (4). We shall proceed with the proof of (a) 4 Theorem 1 for the scalar case, and then show how to extend the proof to the vec- H(u⎢y) Utility mutual information tor case. H(p) H(p⎢y) H(u) maximization ) I(u,y) ,y p In information theory, H (u ) denotes the Proof of the scalar case I( entropy of uuy ,I ( ; ) denotes the mutual Assume that U ! 0M#1 and u ! 0 , H(y) information between u and y while then the UAM [see (8)] has a simpler form: (b) H(|)uy denotes the conditional entropy -2 T Ω / vuxRRUU x. (9) of u given the new knowledge on the query y. The utility entropy before the Also, for the scalar case, FIGURE 4. (a) From the utility’s perspective, maximizing the mutual information I(;)uy is query is the same as minimizing the conditional entropy 2 T 2 T vovu ==UURRxuand u = UUxu . H uy y 1 n u (|), given a query . (b) For the optimal He()u =+log22^hRu log (2r ). 2 2 (10) utility-privacy tradeoff, we want to find a query y to yield larger I (uy ; ) and smaller I (py ; ). As illustrated in Figure 4(a), H (u ) = Via (6), This calls for an optimization metric called DIG:/ (uy ; )- I ( py ; ). IH(;)uy+ ( u | y ), thus maximizing the 2o vv2 - 2 vv2 - 2 mutual information is equivalent to d =-u =- uuuu = u u u o 2 2 minimize the conditional entropies, i.e., u vu vu Proof of the vector case (11) We shall show that there is an orthogo- argmaxyy{I (uy ; )}= argmin {(|)},H u y and it follows that nalization procedure that can be used to assure that no intercomponent redun- where the postquery conditional en - T dancy may exist (i.e., zero mutual infor- UU^hRRxx- u d = tropy is u 2 mation). Then the additive property that Eq. 10 vu H()uu= / H (i ) allows each compo- 1 n TTRR i He(|)uy =+log22 (||)R u log (2r ). = UffUxx 2 2 2 T 2 nent of u to be treated individually, just Eq. 2 vvffRx + u () like the scalar case. The orthogonaliza- TTRR Let us denote a scalar ou / |Ru |. By = fUxxUf tion hinges upon a proper transforma- ||f ||= 1 vv2 T R + 2 the variational principle, the gradient on u ([fIfx ]) tion matrix, denoted by C, i.e., replacing T C R the entropy (constant ignored) bears the = ff Ω (12) U by U for u in (4) and for Ω in (8). T 2 . following form: Eq. 9 fI[]Rx + v f Note that

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 97

THE WORLD’S NEWSSTAND®

new TT-1 TT DIG =-abII(;)uy (;) py (17) Ω =RCCRCCxxUUU[] U R x Privacy intrusive query TT-1 From the intruder’s perspective, it makes =RRxxUU[] U U R x so as to further broaden the application sense to set a = 0 and b =- 1 because = old Ω. scope. For example, its only objective is on the privacy It is important to note that X remains ■ First, a and b are adjustable to attack, paying no attention to the utility invariant with respect to any post- account for the relative reward/pen- gain. This results in a “most intrusive” transformation. Thus, in our analysis, alty associated with the utility gain query vector: f =-[.0 695 0 . 719 ],T we shall pretend as if U were already versus the privacy loss. with the post-query distribution nar- T preorthogonalized, i.e., UURKxu= , ■ Second, for constrained optimiza- rowing sideways (from the blue “4”s nn# where Ku ! 0 is a diagonal matrix tion problems such as “information to the red “*”s), indicating that the pri- " 2 , with diagonal elements: vui . Let bottleneck” [4] and “privacy funnel” vacy variance is severely compromised the ith column of U be denoted by Ui [2], a or b play the role of being the (from 30 down to 1.87) while the utility and then Lagrangian multipliers. remains little affected (from six down We have previously focused on to 5.86). 4 -1 T Ω=RKx UUu Rx the utility-based (mutual) information =/ RRv-2 T optimization and, obviously, the same Discrimant component analysis x UUi ui i x . (13) i analysis carries through to the privacy (DCA) machine learning analysis. (Detail omitted.) To study the and variants The proof is now trivial having the fol- joint utility-privacy tradeoff, we define The utility projection matrix U is some- lowing additive property: the privacy amplification matrix (PAM) times known in advance but some- as follows: times not. In machine learning, it is ==//d()i HH()uu (i )u . (14) possible to develop learning algorithms i i -1 T PRK= x PPp Rx . (18) to estimate U and X from the train- Fidelity preservation query ing data set, made available during the A special situation for the aforemen- Theorem 2. DUCA: Joint Utility-Privacy Optimization learning phase: tioned analysis is when the design The query vector f in (1) that maximizes objective is for the fidelity preservation. the (weighted) “differential informa- 6XY,@ = {[,],[,],,[,xx11yy 22f xNN y ]}, For this case, UI= and it follows that tion gain,” abII(;)uy- (;), py can be X / Rx and (7) becomes derived via the following optimizer: where the teacher values, denoted as y , represent two types of class labels: ffT 6abΩ - P@ i T R argmaxf . (19) 1) the utility-classes, e.g., the family d = ffx T 2 argmaxfu()f argmax fT 2 , fIf[]Rx + v fIf[]Rx +v income level: H/M/L (say, high/middle/ (15) The optimal solution can be computed low), and 2) the privacy-classes, e.g., the from the principal eigenvectors of income disparity between the couple 2 whose optimal solution is again the eig^habΩ -+PR,.x vI 4 (i.e., who earns more). same as PCA, just like (3). Example 2. The DIP (continued) Represent UAM by between-class 2 2 Differential utility/cost Note that vu = 6 and vp = 30 , via (8) scatter matrices analysis (DUCA) and (18) The “center-adjusted” scatter matrix is With reference to the Venn diagram [5] 46/ 86/ in Figure 4(b), the basic differential = ; E N Ω and r rrT vvT information gain (DIG) can be char- 86/ 16/ 6 SXX/ =--/ [][], xi nn xi acterized as the difference of the areas 196/ 30 -224/ 30 i = 1 (20) P = ; E. corresponding to the utility and privacy -224/ 30 256/ 30 which assumes the role of the covari- mutual information ance matrix Rx in the estimation con- Optimal PP query text. The classic unsupervised learning DIG =-II(;)uy (;). py (16) The optimal query vector is f = PCA algorithm is typically computed [.0 727 0 . 687 ].T This boosts the DIG from from Sr. More exactly, the PCA sub- Since the entropy (or information) is a DIG = 161. (no query) to DIG = 297. space is represented by a projection Mm# log function of its corresponding covari- (with query). Moreover, it improves the matrix WPCA ! 0 : ances, thus “differential gain” in infor- utility variance by fourfold (from six = ^hT r mation is in some sense corresponding down to 1.54) while keeping the pri- WWSWPCA argmax tr . (21) {:WWT W= I } to the “ratio gain” in covariance. vacy loss to within 6% (from 30 down While the default setting is a = 1 to 29.83). Pictorially, in Figure 5(a), the The PCA solution can be computed and b = 1, it serves many practical pur- variance of the distribution narrows from the m principal eigenvectors of poses to adopt a more flexible variant: from the blue “4”s to the red “*”s. eig^hSr .

98 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

In supervised learning, the scatter r matrix S can be further divided into two 4 4 additive parts [5] 3 3 r 2 2 SS=+B SW, 1 1

(2) 0 (2) 0

where the within-class scatter matrix x x SW is defined as –1 –1 –2 –2 L N, (), vv (), T –3 –3 SxxW =--/ / [][(].j nn, j , , = 1 j = 1 –4 –4 (22) –5 –4 –3 –2 –1 0 1 2 3 4 5 –5 –4 –3 –2 –1 0 1 2 3 4 5 x (1) x (1) The between-class scatter matrix SB is (a) (b) defined as T L FIGURE 5. A display of DUCA-reduced covariance matrices for the DIP example, where x = 6xx()12 ()@ . 66vvvv@@T SB =--/ N,,nnnn , (23) (a) PP query, with the setting a = 1 and b = 1 , and (b) privacy-intrusive query, with a = 0 and , = 1 b =-1. where N, denotes the number of training vectors associated with the ,th class, nv, denotes the centroid of the ,th In the whitened space, the between- solution can be derived from the prin- l class, for lL= 1,,,f and L denotes class utility scatter matrix SBU points to cipal eigen-subspace of the following the number of different classes. The the L best utility-class-discriminating “discriminant matrix” [6]: between-class scatter matrix represents vectors, spanning an ()L - 1 –dimen- r -1 the so-called signal subspace as it is sional subspace. Therefore, pursuant to 6SIS+ t @ BU . (31) formed from the L class-discriminating (26), it is natural to associate each col- vectors learnable from the data set. umn of Uu l with a vector pointing from Equivalently, they can be derived from To facilitate the training of Ω from the mass center to the centroid of a util- the first m principal eigenvectors of the supervised data set, we adopt two ity class. This results in r coordinate transformations to simplify eig^h SBU,. S+ t I (32) ll? uullT = Xt the mathematical analysis. SUUBU Eq. 26 . (28) ■ Orthogonalization Transforma- The extracted queries are rank- tion on U. Without loss of generali- Note also that ordered according to their “signal to

ty, we assume that U is already 1 T power ratios,” which are equivalent to uuT 2 ull u T 2 XR? xxUU R= Rxx U U R pre-orthogonalized in (8), which can Eq. 24 Eq. 27 their corresponding eigenvalues: 1 T then be expressed as = RXR2 l 2 (29) m = Eq. 26 xx. i T = RRuuT vSi BiU v =-f Ω xxUU (24) It follows that T r if iL11,, * vSi ()+ t Ivi 1 T 1 T # n tt 0 if iL$ . u ! 0M 2 ll2 2 2 where U w i t h c o l u m n ve c t o r s SSBUU==RRB ? RXRX. Eq. 25 xxEq..28 xxEq 29 (33) u ==vn-1 f defined as UUiiui ,i 1 , , . (30) 4 ■ Whitening Transformation on x. 1 2 Let Rx denote the square-root of Utility-driven DIM-DCA Metric for interclass separability the covariance matrix Rx i.e., for supervised learning The trace-norm of the discriminant matrix, 1 T 2 2 RRx = xxR . By transforming the Now we are ready to establish a machine defined in (31), may be used as a simple original vector space to a “canonical” learning variant, called DIM-DCA, cor- metric to measure the inter-class separa- (or “whitened”) vector space via responding to (7). bility of a supervised data set. It offers a convenient tool to evaluate the the suit- - 1 l = R 2 xxx , (25) Algorithm 1. Utility-driven DCA ability of a certain similarity function (or learning algorithm kernel function [3]) to be chosen for non- the new covariance matrix becomes The optimization formulation of DIM- linear data analysis (see the next section). an identity matrix, i.e., Rxl = I. It DCA involves searching for the projec- Mm# follows that tion matrix WDCA ! 0 : Theoretical connection between two variants of DCA XRt lllll==uuTT R uu = ^hT xxllUU UU , (26) WWSWDCA argmax tr BU . In [6], another variant of DCA was {:WWT 6@ Sr +=t IW I } developed for finding the optimal sub- where (Note that t here assumes the role of space projection matrix via the principal T uu2 2 UUl / Rx . (27) the variance v .) The optimal DCA eigenvectors of

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 99

THE WORLD’S NEWSSTAND®

PPFR simulation results same class align almost perfectly, Face Apply PCA and DCA to the Yale data sometimes making the whole class Recognition Performance (Yale) set for PP face recognition (FR) (PPFR) of data projected to a single point. 1.0 0.9 applications, we have the following 0.8 observations: Privacy-driven desensitized 0.7 0.6 ■ PCA/DCA Classification Ac- DCA via ridge DCA (RDCA) 0.5 curacies. There are only L - 1 In the previous section, the utility-driv- 0.4 Accuracy 0.3 DCA meaningful eigenvectors, because en learning algorithms are good for sce- PCA 0.2 rank^h SBU # L - 1. Note also that narios where the intended utility is well 0.1 Random usually LM% , it implies that the defined but the privacy policy is still

5 2 DCA eigen-components can enjoy a open. Conversely, there are other sce- 14 10

4,096 2,000 1,000 win-win advantage in improving pri- narios where the intended utility is yet Dimension (Log Scale) vacy without sacrificing utility. to be determined but the privacy policy – First, the L - 1 principal compo- is already pre-defined. This calls for a FIGURE 6. The FR accuracies on the Yale data nents can capture key features DCA variant tailored designed for the set, with respect to different reduced-dimen- fully adequate for very high per- extraction of desensitized components. sions via DCA, PCA, and random projection. formance, as evidenced by the To this end, we further incorporate (Figure courtesy of T. Chanyaswad.) performance curves shown in another ridge parameter tl to regulate

Figure 6. Note that DCA far out- the (privacy) signal matrix SBP, result- performs PCA and random pro- ing in the following privacy-driven r eig^h S,, SWU + t I (34) jection in terms of FR accuracies. learning algorithm [7]. – The DCA dimension reduction

where SWU denotes the within-(utility)- results in removal of a large pro- Algorithm 2: Privacy-driven class scatter matrix. This variant is portion of components, making it ridge DCA algorithm an extension of Fisher’s LDA and is an effective compression tool for Find the projection matrix Mm# indeed a very close sibling of the DIM- privacy preservation. WRDCA ! 0 : type DCA. To prove that the Fisher- ■ Data Visualization by PCA/DCA WRDCA = argmax type DCA is exactly the same as the Projection. The high-dimensional {:WWT 6 Sr +=t I@ W I } DIM-type DCAs, when t = 0, we Yale data set may be visualized by ^hT l tr WS6 BP - t IW@ . let {mii ,v } and {mllii ,v } respectively means of two-dimensional PCA or

denote the eigenvalues/eigenvectors DCA subspace projection. Fig- where SBP denotes the between- for DIM-DCA [see (32)] and Fisher- ure 7(a) displays the PCA visualiza- (privacy)-class scatter matrix and tl DCA [see (34)]. It can then be shown tion, showing that many classes are is a small positive value. The optimal -1 that vvlii= and mmlii=- (1 ) . The nonseparable by PCA. In contrast, RDCA solution can be derived from latter guarantees "mli, and {mi } to be the DCA visualization in Figure 7(b) the m eigenvectors corresponding to the sorted in the same order, thus verify- shows very well separated classes. LLmth,,(f +-1 )th eigenvalues of ing the equivalence. In fact, many data points from the l r eig^h SBP -+tt I,. S I (35)

The component powers are closely relat- PCA-Subspace Visualization DCA-Subspace Visualization 4,000 100 ed to their corresponding eigenvalues: 3,000 50 tl 2,000 PiL()vi .$--t, for . (36) 0 m 1,000 i 0 –50 4 DCA 2 PCA 2 –1,000 –100 –2,000 Equation (36) assures that the desen- –3,000 –150 sitized components can be orderly –4,000 –200 extracted according to their eigenval- 0 0 ues, just like PCA. This is why RDCA 100 200 300 400 –200 –100 2,000 4,000 is sometimes referred to as desensitized –4,000 –2,000 DCA or, more simply, desensitized PCA. PCA 1 DCA 1 (a) (b) Let us highlight some key properties of RDCA’s eigen-components: ■ FIGURE 7. The high-dimensional Yale data set may be visualized by means of two-dimensional (a) PCA Signal-Subspace Components, i.e., or (b) DCA subspace projection. Each mark represents a data point, and data points in the same class when iL1 : The first L - 1 eigen- share the same shape and color. (Figure courtesy of T. Chanyaswad.) components are potentially most

100 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

intrusive, so they must be filtered out for the privacy sake. DCA 15 DCA 16 DCA 17 DCA 18 DCA 19 ■ Noise Subspace Components, i.e., T when iL$ : In contrast, vSi BP vi - 0 for all iL$ , indicating that they do not leak sensitive information. Moreover, their utilizable component powers are rank ordered by the corre- (a) sponding eigenvalues mi . RDCA 15 RDCA 16 RDCA 17 RDCA 18 RDCA 19 Antirecognition utility maximization results Antirecognition utility maximization (ARUM) is an exemplifying application scenario, in which the privacy policy calls for deidentification. More exactly, (b) while we want to conceal the person’s identity, we would like to retain as much FIGURE 8. The DCA and RDCA basically have the same principal (14) eigen-faces in their signal sub- information as possible to serve other space (not shown here). However, they have completely different eigen-faces in the noise subspace. purposes such as 1) eyeglasses detection (a) The next five privatized DCA-eignfaces are random and useless. (b) In contrast, rich information or 2) mood detection, e.g., happy versus is revealed by the five highest-power desensitized RDCA-eigenfaces, corresponding to the 15th–19th eigenvalues (r'= 0.05). (Figure courtesy of T. Chanyaswad and A. Filipowicz.) sad faces. Further ARUM applications may include, for example, anomaly/ intruder detection without leaking the tized and contains no useful informa- both the eyeglasses detection and mood exact identity of the user. tion for authentication. detections. (More visibly so for the former As an example, we have applied but somewhat subjective for the latter.) DCA and RDCA to the Yale data set, Reconstructed images with 15 different persons (i.e., L = 15), by DCA and RDCA Quantitative tradeoff and compared their performances for The performance distinction lies in the analysis via RDCA both PPFR and ARUM applications. power-based rank-ordering (or lack of it) An experimental study on an in-house ■ Eigenfaces in the Signal-Subspaces of the DCA’s and RDCA’s desensitized glasses data set (with seven different of DCA and RDCA. Note that, components: persons) was conducted for an ARUM- when tl is relatively small, DCA and ■ Figure 8(a) shows the DCA’s desen- type application[7], where 1) the utility RDCA share almost the same signal sitized eigenfaces, which apparently is to determine whether or not a person subspace. Specifically, their contain no useful information. wears glasses and 2) the privacy involves 14()=-L 1 principal eigenfaces ■ Figure 8(b) shows the RDCA’s concealing the person’s identity, i.e., are very similar. How to treat the 14 desensitized eigenfaces, which exhib- deidentification. eigenfaces depends on the intended it much higher component powers, Our study involved 1,000 trials. utility/privacy goals: rendering them possibly amenable to RDCA appears to be promising: Upon – For PPFR, the principal eigenfaces ARUM-type applications. the RDCA desensitization, the privacy alone are sufficient to yield high By the following ARUM-type exam- accuracy drops significantly from 97.6% accuracies shown in Figure 6. ple, we show how to harness the recon- to 44.4%, implying a much improved pro- – For ARUM, on the other hand, the struction for some useful applications. tection, while its utility accuracy remains same principal eigenfaces are ■ Shown in Figure 9(a) is the original fairly high, from 98.3% to 95.5%. deemed to be most intrusive and, image of the first sample in the Yale therefore, they should be cast away data set, which is sampled/represent- DUCA machine learning for the sake of desensitization. ed by a full-dimensional vector and variants ■ Eigenfaces in the Noise-Subspace. (,).M = 4 096 It would be idealistic if we could benefit Shown in Figure 8(a) are the first five ■ Figure 9(b) displays the reconstruct- from having both types of the teacher desensitized DCA eigenfaces and, in ed face images using 3,986 DCA values made available during the learn- Figure 8(b), the first five RDCA components. ing phase, one for the utility class and eigenfaces. Each of the noise-sub- ■ Figure 9(c) displays the reconstruct- one for privacy class. Together with the space components yields a very low ed face images via 3,986 (power- training data set, these teacher values can classification accuracy around 6.6%, sorted) RDCA components. be used to produce two between-class

no better than random guess out of By comparing the two reconstructed scatter matrices, SBU (for utility) and SBP L = 15 choices. This confirms that images with the original image, we note (for privacy), which can, in turn, be used each of them is perfectly desensi- that the RDCA outperforms DCA for to estimate X and P , respectively.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 101

THE WORLD’S NEWSSTAND®

T and f2 =-[.221 . 535 . 733 . 357 ] , leading to the DCA projections in Figure 10(c) and (d). The good news is that the query “H” is correctly classified as middle-income. Moreover, when compared with PCA, cf. Figure 10(b) and (d), the (a) (b) (c) (d) two privacy classes seem to overlap more, which suggests an enhanced FIGURE 9. The original and reconstructed image of the first sample of the Yale data set. (a) The privacy. Indeed, although the query “ original face image, (b) the DCA reconstructed image, (c) the RDCA reconstructed image (with H” is leaning towards the “3 ” tl = 005.), and (d) the reconstructed image from 399 selected wavelet components. (Figure courtesy of T. Chanyaswad and A. Filipowicz.) class, its uncertainty represents a noticeable improvement over PCA. ■ With ab==1, the two DUCA DUCA: Joint utility-privacy where “H/M/L” denotes the three (high/ principle eigenvectors of (37) are: T learning algorithm middle/low) utility classes (i.e., family f1 = [.142 . 872 . 168 . 438 ] and T For utility-privacy tradeoff analysis, a income) and “34/ ” denotes the two f2 =-[.002 -. 114 . 682 . 723]. The natural combination of Algorithm 1 and privacy classes (i.e., who earns more DUCA subspace projection enjoys (19) leads to the following DUCA super- between the couple). the best of the two worlds: vised learning algorithm. Recall that, via (20), the scatter ma- – As shown in Figure 10(e), the trix can be learned from the given data query “H” can be confidently clas- Algorithm 3. DUCA supervised set {}X and UAM and CAM can be sified into the middle-income class. learning algorithm learned, via (23), by further incorporat- – As shown in Figure 10(f), the two Find the projection matrix WDUCA ing their respective class labels. Let us set privacy classes now overlap much ! 0Mm# such that the query vector as f =-[.941 . . . 1 ]T , more than before. Indeed, the pri- whose projection to the two-dimen- vacy label for the query “H” = WDUCA argmax sional subspace is marked as “H.” appears to be totally undecided. {:WWT 6 Sr +=t I@ W I } ^hT Given the full-dimensional query it is tr WS6abBBUP- SW@ . clear that it belongs to the middle (util- Supervised DUCA-based filtering whose solution can be derived from the ity) class, because ux()=+=xx1215 for feature selection m principal eigenvectors of and to the “3” privacy class, because An alternative approach to the dimension px()=-=xx12502 . The objec- reduction of the query vector is the fea- ^hab-+r t eig SBBUP S,. S I (37) tive of CP design is to find a dimension- ture selection strategy. Its design objec- 4 reduced query to correctly classify the tive is to retain only a small number of Note that there are only LC+-2 utility class but not to reveal its privacy selective features for CP. In this case, the meaningful eigenvectors, because class. By comparing the PCA, DCA, and feature vector is restricted to an indica-

rank^hab SBBUP-+- S # LC2 where DUCA subspace projections, depicted in tor-type vector: f = 60gg 010 0@, L and C denote the numbers of utility Figure 10, DUCA is noticeably the far where only the ith entry is nonzero. and privacy classes, respectively. best in meeting both objectives. 4 This brings about the following DUCA- It is worth elaborating further on the scores useful for ranking the features in Example 3 performance comparison between PCA, feature selection: Machine learning for DIP. Let us DCA, and DUCA. abSSBBUP(,ii )- (, ii ) revisit the DIP example, now with two ■ The two PCA principle eigen- DUCA()i = . r + t = S(,ii ) more features. Suppose that we are vectors of (21) are f1 (38) T given a training data set: [.984 . 174 . 039- . 016 ] and f2 = - T Example 4 { X } = [.163 . 899 . 042 . 405 ] . As shown R VR VR VR VR VR VR VR V by the PCA projections in Fig- DUCA Ranking for DIP Feature 11 18 17 4 5 4 1 4 S WS WS WS WS WS WS WS W ure 10(a), the utility class of the Selection: According to (38), the S 7 WS 8 WS 5 WS10WS6WS 7 WS 2 WS 1 W S WS WS WS WS WS WS WS W, query “H” is somewhat undeter- DUCA-scores are DUCA (1234 , , , ) = 1 2 -1 -1 2 1 -1 1 S WS WS WS WS WS WS WS W mined because it is possible to be [.- 005073019 . . 012 .]. This is con- 2 -1 -1 --4 2 1 1 -1 T XT XT XT XT XT XT XT X either middle-income or high- sistent with the previous finding that with the utility/privacy teacher labels, income. More seriously, Figure 10(b) (1) x1, being most DUCA-costly, is denoted by strongly hints its “3” class, poten- given the lowest weight in both f1 and f2 ; tially exposing the privacy. and (2) x2, being most DUCA-rewarding, "Y, = ;;;;;;;;HHMMMMLLEEEEEEEE ■ The two DCA principle eigenvectors of receives the highest weight in f1 . T 33344443 (32) are f1 = [.204 . 838 . 245 . 443 ] 4

102 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

Example 5 DUCA Filtering for PPFR Application: PCA By applying SVM to the full-dimen- 4 4 sional Yale data set, it yields a recognition accuracy of 82%. We have also 2 2 applied a utility-driven DUCA score, 0 0 r i.e., –SSBU (,ii) /^h (,) ii + t , to select –2 –2 the best 399 (of 4,096) Wavelet-transformed components. We have found –4 –4 that the DUCA-filtered CP method actu- –6 –6 ally offers a higher accuracy at 82.3%, again via SVM. At the same time, it –8 –6 –4 –2 0 2 4 6 8 10 –8 –6 –4 –2 0 2 4 6 8 10 also totally obfuscates the face images, (a) (b) as exemplified by Figure 9(d). In short, DCA the DUCA-filtering feature selection is 2 3 promising for PPFR since it offers PP 1 2 compression without compromising the 0 1 0 FR accuracy. –1 4 –1 –2 –2 –3 Extension to kernel –3 –4 –4 DCA and kernel DUCA –5 –5 In the kernel learning models [3],ux ( ) andpx ( ) will be nonlinear functions –5 –4–3 –2 –1 0 1 2 3 4 –6–5 –4–3 –2 –1 0 1 2 3 4 in general. As such, it can induce an (c) (d) expanded solution space and thus further DUCA improve the performance. It involves 3 3 a simple kernelization procedure to 2 2 extend from DCA to kernel DCA [6]. 1 1 For example, the discriminant matrix 0 0 in (31) can be extended to the following –1 –1 kernel-DCA discriminant matrix: –2 –2 –3 –3 rr21- []KKK+ t BU (39) –4 –4

r –6–5 –4–3 –2 –1 0 1 2 3 –5 –4–3 –2 –1 0 1 2 3 where K and KBU denote the kernelized r (e) (f) counterparts of S and SBU, respectively. Again, applying eigen-space analysis to FIGURE 10. Visualization of a query, marked as H, mapped to the (a) and (b) optimal two-dimensional this kernelized matrix will lead to the PCA, (c) and (d), DCA, and (e) and (f) DUCA subspaces. The high/middle/low utility labels are marked optimal query solution in the kernel vec- by+ /)# / , and the two privacy labels are marked by 3/ 4. The results suggest that DUCA-sub- tor space. space offers a promising approach to optimal utility-privacy tradeoff. For applications to CP problems, a kernel-DUCA discriminant matrix = / L c may be derived by substituting KBU by KK(,xy )l ll (, xy ). In this case, the privacy subspace, leading to a new v vT v KKBBUP- in (39). (Again, the prin- the trace-norm of kernel-DCA discrimi- query: y =+f (),x e with colored noise ciple eigen-subspace analysis would nant matrix in (39) may be used as an ef- covariance /e = tSB p . This compels yield the optimal queries.) However, fective evaluation criterion for finding the the eigen-solution (37) to be modified as: for such applications, the reduced di- optimal coefficients: {,cl lL= 1 ,f ,}. r mension must be strictly lower than (Detail omitted.) eig(,)ab SBBUP-+ S S t S B P the original dimension M, since the CP encoding scheme must necessarily Tailor designed noise Intuitively speaking, such a design be lossy. for privacy preservation aims to dampen Eve’s ability to intrude

Recently, there has been growing in- Pursuant to (23), the privacy matrix SB p privacy while leaving the utility gain terest in multikernel research[3], where may be learned from labeled training for Bob relatively unaffected. a multikernel function is expressed as data and the original data xv be pur- linear combination of many kernels: posefully perturbed by noise parallel to (continued on page 112)

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 103

THE WORLD’S NEWSSTAND®

APPLICATIONS CORNER

Muhammad Salman Khan, James Stewart Jenkins, and Nestor Becerra Yoma

Discovering New Worlds A review of signal processing methods for detecting exoplanets from astronomical radial velocity data

xoplanets, short for extra solar plan- E ets, are planets outside our solar system. They are objects with masses Discovering Exoplanets fewer than around 15 Jupiter-masses that orbit stars other than the sun. They are Radial velocities: where the gravitational tug of the planet on the star is mea- small enough so they cannot burn deu- sured by analyzing the stellar spectral fingerprint to search for the Doppler terium in their cores, yet large enough shift of these lines as the star and planet orbit their common center of mass. that they are not so-called dwarf planets Transits: where the planet passes in front of the star toward our line of sight, block- like Pluto. ing the star’s light as it does so, and inducing a slight dimming of the light profile. To discover life elsewhere in the uni- Transit timing variations: where the time of center of transit is measured over verse, particularly outside our own so- many transits, and variations in that time that are due to the gravitational inter- lar system, a good starting point would action from another planet in the system can be measured. be to search for planets orbiting nearby Photometric variations: a series of methods that model variations in a star’s photo- sun-like stars, since the only example metric light curve to infer the presence of planets orbiting the star (e.g., planetary of life we know of thrives on a planet reflected light or thermal emission, Doppler boosting, and ellipsoidal variations). we call Earth that orbits a G-type dwarf Gravitational microlensing: where a foreground star passes across the line star. Furthermore, understanding the of sight of a far-off star, and the gravitational field of the foreground star acts population of exoplanetary systems in as a lens to intensify the light of the background star, and also intensifies the the nearby solar neighborhood allows us light from a planet orbiting that star. to understand the mechanisms that built Pulsar timing: where the precise arrival time of the pulses of light are mea- our own solar system and gave rise to the sured, and small differences in the timing can be introduced due to small plan- conditions necessary for our tree of life ets orbiting these dead stars. to flourish. Direct imaging: where we point large telescopes at stars and directly image Signal processing is an integral part any orbiting planets. of exoplanet detection. From improving Astrometric wobble: where we measure the position of a star on the sky and the signal-to-noise ratio of the observed search for changes in that position, or a wobble, due to the gravitational tug data to applying advanced statistical sig- of orbiting planets. nal processing methods (among others), to detect signals (potential planets) in the data, astronomers have tended, and In this article, we focus on the radial ■ What are the different signal pro- continue to tend, toward signal process- velocity method of exoplanet detection, cessing methods that astronomers ing in their quest of finding Earth-like the most successful method for dis- have been using for detecting exoplanets. Methods that have been used to covering planets orbiting the nearest planets and what are their pros and detect exoplanets are listed in “Discov- stars to the sun [1]–[3]. We address cons? ering Exoplanets.” basic questions such as ■ What is the statistical significance ■ How is the radial velocity data of signal detection? ■ Digital Object Identifier 10.1109/MSP.2016.2617293 obtained? What are the potential directions for Date of publication: 11 January 2017 ■ Why is the data nonuniformly sampled? future research?

THE WORLD’S NEWSSTAND®

Introduction velocity technique, which is the focus of from the high signal-to-noise ratio data, The radial velocity method works by this article, the much smaller mass of and the spectra can be extracted, or col- breaking a star’s light up into its constit- the planet compared to the star means lapsed, into its two-dimensional format. uent colors using a high-resolution ech- we need spectrographs that are stable at Finally, all cosmic rays or bad pixels can elle spectrograph. The observed stellar the meters/second level to detect even be cleaned from the extracted spectrum spectral lines can then be used as mark- the most massive planets, and to detect and a highly precise wavelength correc- ers for the star’s velocity. If the star’s Earths in Earth-like orbits around sun- tion is applied. Interested readers can velocity changes, the Doppler effect like stars, we need spectrographs that refer to, e.g., [4] and [5] for more details tells us that electromagnetic waves are are stable at the centimeters/second of this process. affected by this movement by present- level. This is very difficult to accom- The radial velocity data is unevenly ing a shift in frequency, depicted in plish since small pressure and tempera- sampled because we can only observe Figure 1. We can measure that frequen- ture variations, illumination problems, the star when it is visible in the night cy and then correct for any additional mechanical stability problems, and even sky. Stars are not visible all year round, velocity shifts from noise sources, such the stars themselves can introduce noise as sometimes they are in the same area as temperature and pressure variations in the measurements at levels higher of sky as the sun. Furthermore, we must in the laboratory, mechanical instabili- than this. compete for telescope time, so we can- ties throughout the optical train, obser- The radial velocity data is obtained not always observe when we want—we vational airmass chromatic effects, by observing a star with a telescope and are at the mercy of schedulers and pro- stellar magnetic activity affects, and feeding the light to an echelle spectro- posal reviewers. Also, the number of stellar convective blueshift, and mea- graph, as mentioned previously. We stars we can observe per night is lim- sure the star’s radial velocity toward can observe the star as many times ited, e.g., 30–40 or more, so we cannot or away from us. Over the course of a as we want in a single night and as observe all of them every night that we planet’s orbital period, we can measure many times as we can when the star actually have telescope time. the star’s spectral redshift and blue- is in the sky throughout the year. The In the following sections, we review shift, and by analyzing the amplitude, more observations we get, the better the different signal processing methods phase, shape, and period of this signal, we sample the signal of the star’s radi- used by the astronomical community we can understand characteristics about al velocity. The data is first reduced, for detecting exoplanets based on radial the companion that is causing the star’s which is a way of using calibrations to velocity data. These methods include velocity variations. prepare the spectra for analysis, and the Lomb–Scargle (LS) periodogram, Exoplanets are difficult to detect then we can measure the velocity. The Keplerian periodogram, prewhitening due to their extreme contrasting char- typical reduction procedure for such method, maximum-likelihood (ML) acteristics compared to the stars they data is to perform a bias correction to periodograms, Bayesian analysis, and orbit. Planets are so much smaller and the image, so-called debiasing, then we the minimum mean square error (MMSE)- fainter than their host stars that all of correct for the pixel-to-pixel variations based method. All of the methods the aforementioned methods have a dif- through a process called flatfielding, assume that xt() is the radial velocity ficult time detecting them. For the radial any scattered-light is then removed data, t is the timestamp of observations,

“Wobbling” Host Star Not to Scale Blueshifted Light from the Star Center of Mass

Earth-Based Telescope Unevenly Sampled Radial Velocity Data 500 Unseen Exoplanet 0 Redshifted Light from the Star

Radial –500 0 1,000 2,000 3,000 Velocity (m/s) Velocity Time (Days)

When an exoplanet orbits a star, its gravitational pull causes the star to wobble. When the star moves toward the Earth, the light moves to the blue end of the spectrum, as its wavelengths get shorter. While when the star moves away, the light is shifted to the red end of the spectrum, as its wavelengths get longer.

FIGURE 1. Detecting exoplanets using the radial velocity method.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 105

THE WORLD’S NEWSSTAND®

i.e., tT= 123, , , ..., , where T is the total is applied. Yet many planetary systems approach, but we remind the reader number of observations. are found to contain more than one that the Keplerian periodogram is open planet, which means the radial veloc- to including other models to calculate LS periodogram ity time series should exhibit more than powers. In contrast to the LS method, The LS method [6] of signal detec- one signal. This assumption also under- the Keplerian method is relatively slow tion has been extensively used in the lies the Fourier transform analysis [7]. and complicated to apply to long time search for exoplanets, particularly using Therefore, signals must be subtracted series data, but since it is more robust the radial velocity technique of planet out of the data fits, before reapplica- in detecting signals that deviate from detection. In its simplest form, the tion of the method on the residuals sinusoids, it is more applicable in the method works in a Fourier-like man- is performed, and since the signals are search for exoplanetary systems orbit- ner by applying a number of sines and generally not orthonormal, this gradi- ing nearby stars [11]. However, it again cosines to the radial velocity data xt() ent-based approach introduces problems makes the assumption that there is only across a grid of frequencies chosen by for detecting low-mass and multiple one signal in the data, which means it the user, and the amplitude of these planet systems. It is worth mention- suffers from the same problems as the functions are minimized to fit the data ing that in [8] a date-compensated dis- LS method if the data contains more and a power is calculated. When one of crete Fourier transform was proposed than one signal, and there is no corre- the functions provides a good match to that gives better estimates of the power lated noise component. the radial velocity time series, the power spectrum of nonuniformly sampled will be maximized at the selected fre- data, aiding in accurate determination Prewhitening method quency, indicating to the user that there of spectral peak heights. Finally, the The method of prewhitening to search is a signal at that frequency, and this can LS method only considers white noise, for Doppler signals in radial velocity be visually viewed by a periodogram: which can be problematic since starlight time series is similar to the LS method, often involves correlated noise. except that the method is applied in the Z 2 ] T Fourier domain to search for any signals ] =/ xt() cos ~x( t- )G = in the data, but again using sines and ~ = 1 [ t 1 Keplerian periodogram Px () T 2 ] 2 Given some of the problems mentioned cosines (e.g. [12]). As the name suggests, ] / cos ~x()t - \ t =1 with the LS periodogram method of signal the method works by whitening the data 2 _ detection—particularly the fact that sig- as much as possible to remove all noise T b =/ xt() sin ~x( t- )G b nals are not always well described by sines sources with fitted functions, until a + t =1 ` T , or cosines—the Keplerian periodogram real Doppler signal is found. The data is 2 b / sin ~x()t - b was developed [9], [10]. The Keplerian translated into Fourier space and a search t =1 a periodogram allows the user to consider for frequencies that pass a significance (1) factors of noncircularity as part of the threshold is performed. The strongest where Px is the periodogram powers as a analysis when calculating the powers for signal is fit, the corresponding residual function of frequency, and x is defined as the periodogram, since the chi-squared to the fit calculated, and then the process comparison used is open to any model is repeated again. This goes on until the T eo/ sin 2~t that can be fit to the data, for instance residual data is just the noise-floor of the = ~x = t 1 ||~2 - 2 observations, meaning no peaks are found tan()2 T . 0 Kep () pKep ()~ = . above the significance threshold. Simi- e / ~ o |2 cos 2 t 0 t =1 lar to the LS and generalized LS (GLS) ~ |2 Here pKep ( ) is the power, 0 is the methods, this is quick and easy to apply, Although the technique is easy to chi-squared for the weighted mean, and but it has the same problems as these other 2 implement and fast to apply even to |Kep ()~ is the chi-squared of the Keple- two methods. However, the prewhitening large data sets, it has some major draw- rian model. The Keplerian model in this part is done to clean noise from the time backs when searching for Doppler sig- case can be written as series, but that requires knowledge of the nals induced on stars from orbiting noise source, like aliases of real signals, or planets. For instance, not all exoplanets xt( )=+csosK [etcos + cos ( ( ) + )], in the case of stars, stellar activity signals/ are found to be on circular orbits around (2) timescales, and again, this is a gradient- their stars. In fact, there is a high frac- based approach that does not consider tion that have significant eccentricities, where c is the systemic offset of the correlated noise. and once the eccentricity of the orbit is data, K is the amplitude of the signal, larger than ~.,06 the LS method finds e is the eccentricity of the orbit, s is ML periodograms it more difficult to detect these signals. the longitude of periastron of the orbit, Given that the aforementioned ap- Another issue with this method is that it and o()t is the true anomaly of the proaches focus on searching for one makes the assumption there is only one orbit. Keplerian signals can be detected signal at a time in the time series when signal in the data, each time the method in radial velocity data following this searching for planets, and none deal

106 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

with correlated noise, ML periodogram and assesses the parameter space using MMSE-based method methods have been developed to cir- Markov chains (e.g., [15), where the In [18]–[20], the independent sinu- cumvent these issues [13], [14]. The ML model is assessed by covering a given soidal components in nonuniformly periodogram method does not generate frequency/period domain. The maxi- sampled radial velocity data are deter- a periodogram that shows power on the mum of the posterior density distribu- mined by means of the MMSE method y-axis, but instead it shows the log-like- tion can be used to detect a signal in the or its direct extension, the ML estima- lihood of the model that is compared to data (e.g., [16] and [17]) tion scheme. According to [19], signifi- the data at each step. In this way, any cance tests are employed to filter out =+ + model can be compared to the data di- mdfdfkf,()cc t Ft the parasitic solutions appearing on the q rectly across a grid of frequencies or or- ++fp/ way. In [18], the MMSE-based method fz,,,dcfdd z bital periods, and the log-likelihood can z = 1 applies a trellis-based optimal global be calculated for each, with a detected p search and returns the optimal number ttfz- - f signal having the maximum likelihood + / zz,,.d exp' fz d1 of sinusoidal components including = xd z 1 their frequencies, phases, and ampli- N (4) i = % 1 tudes. This technique employs the Lm(|) 2 2 f = 1 2rv()f + vl Here model m for a given Keplerian MMSE criterion as an objective func- --o 2 ) [(mtflf)] 3 k and velocity data point f, previous tion in all the analysis. exp 2 2 . (3) 2()vvf + l measurement z, and data set d can be If Ci is the ith sinusoidal compo- described by the Keplerian model as a nent, and NC is their number, each The likelihood to be maximized function of time (Ft ( )), a systemic off- component may be written in the form can be described as in (3) with Lm(|)i set velocity c , a linear trend as a func- Caiiii= (,,),~z where ~ziii,,a are the being the likelihood of the data m given tion of time ct , a Gaussian noise model frequency, amplitude, and phase of the the model parameters i. v f and vl to describe the random noise f , a red ith component, respectively. represent the stellar and instrumental noise component described by a mov- The MMSE technique tries to find = " ~z,NC white noise components, respectively, ing average (MA) model with exponen- the set Sa(,,)iiii =1 that mini- and olf()t is the Keplerian model to tial smoothing (parameters z and x ), mizes the mean square error between fit, similar to (2) but with correlated and a set of linear correlations c with the original signal and S by optimizing noise terms included. Maximization of activity indicators that parameterise ~ii,,a and zi of each component [18]. this likelihood function allows signal the activity state of the star at the time First, the target frequency bandwidth detection to be performed and prob- of the observation p. The Bayesian is divided into K~ levels. Each level abilities can be calculated directly from approach is the least efficient of these ~k is represented by ~rk = # kK/,~ the log-likelihood values. Although signal detection methods, since long where 1 ##kK ~ .For each ~k an in practice this method is slower than chains are required to properly search optimal amplitude and phase, a~~kk , z the aforementioned methods, it has the the multidimensional parameter space are obtained by performing an MMSE- desired effect of allowing multiple sig- in a robust manner. However, currently based Fourier analysis: for each ~k, nals to be detected at the same time this method is the most flexible, allow- a~k and z~k are optimized to mini- (i.e., a global model approach), and it ing the user to assess the parameter mize the mean square error between also means the model can include cor- space in many different ways. It also the original signal and the components related noise components, along with allows visualization of the full param- atkkcos()~z+ ~k . the white noise component(s). There- eter space after the chains are com- The number of components to analyze, fore, given the continuing increase in plete, meaning nonlinear correlations N, is then estimated for all the frequen- computer processing power, the extra between parameters can be scruti- cies having local minimum of MMSE information and flexibility of ML peri- nized. Finally, this method was shown values and/or higher amplitudes with odograms outweigh the inefficiency of to be the most robust signal detection respect to a defined threshold. There- its application to real radial velocity and false-positive suppression method fore, a subset Samin = "(,~zi ~~ii , ), data. However, as with all model fit- currently used, given the results of an is constructed out of the set S P , which ting methods, one must be careful not International Challenge (Extreme Pre- includes only these components. Next, to overfit the data by adding unneces- cision Radial Velocities, Yale 2015) a neighborhood band Vi is defined = sary terms to the applied model, which [30] issued to the radial velocity planet for each component, Ci in Smin as, Vi is where proper model comparison sta- detection community. "(,~zaS~~ , )!!Pii / ~~d~d [-+ , ],, tistical tests should be applied.

Bayesian analysis T NC 2 ttj j t j NC j j j Like the ML approach, Bayesian anal- (,,)~zi axtati i i = 1 =-+arg min/ eo ()/ i cos ( ~zi i ) (5) ~zj j j ## ysis applies a global model to the data, (,,)i aiNi i 1 C t = 1 i = 1 including correlated noise components,

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 107

THE WORLD’S NEWSSTAND®

^httj j t j previous page [18], where ~zi,,ai i C1,1 C2,1 C3,1 CN ,1 j C ! Vi, 1 # i # NC , providing an optimal

set of triplets for each NC.Finally, the C C C C optimal set of triplets having the global 1,2 2,2 3,2 NC,2 minimum MMSE is selected at which its length, defines the number of the most important sinusoidal components in the nonuniformly sampled signal, C1,m C2,m C3,m CN ,m 1 2 3 C NC while its elements are their frequencies, amplitudes, and phases, respectively. It is worth mentioning that the problem of

C1,M C2,M C3,M CN ,M order selection has also been addressed C1 C2 C3 C CN C by using statistical significance analysis FIGURE 2. A trellis diagram representing all the possible combinations of components and their local [20] and extreme value theory [21]. neighborhoods [18]. Figure 3 illustrates, as an example, the Keck and the High Accuracy Radial where d defines half of the neighbor- where NNC = "1,.., ,. For each value Velocity Planetary Searcher radial veloc- j = " jj j , hood band around each component and of NC, let ACCC12 , , ..., NC be ity data of the M-dwarf planet host star set to some value that incorporates all a set of triplets for one of the possible GJ876 and the corresponding LS and the significant components around the combinations, where j is from 1 to MMSE periodograms. GJ876 is known j selected peaks of S P . Hence, for each NN!/CC !( N- N )! and Ci corresponds to host a system of planets that contains

component Caii(~z ,~~ii , ) there are to the ith component in the jth combina- at least two short-period gas giants [18].

MCi candidates, where MCi is the car- tion. The corresponding set of neighbor- Signals can be searched for using the jj j j = " , dinality of Vi . hoods is VVVVA 12, ,...,NC , where gradient-based approach that starts by j j j j Subsequently, a trellis analysis for Vai = (,,)~zi i i denotes the neighbor- searching for one signal only, and when j all of the possible combinations of com- hood of candidate components Ci . The one is detected it is then subtracted out of j t j ponents and their local neighborhoods optimal set of A ,,A for a specific NC the data and a new search is made using is performed, as schematically shown value, is the one associated with the the residuals all over again by treating in Figure 2. Now, all possible combi- lowest MMSE and corresponds to (5), them as an independent time series from nations of candidates NC is evaluated, shown in the box at the bottom of the the original observed data. This process is then repeated until the noise floor of the data is reached. By applying this method, the following signals [with periods in days 500 (d)] were detected with the MMSE method [18]: 61.03 d, 30.23 d, 15.04 d, 1.94 d, 0 10.01 d, and 124.69 d. This system was Radial −500 chosen because the two large-amplitude Velocity (m/s) Velocity 0 1,000 2,000 3,000 4,000 5,000 signals could be detected in both halves Time (Days) of the time series separately. The MMSE (a) and trellis technique allows studying the 100 phase of the detected signals as a function 50 of time, showing that the phase difference between both planets is stable over

Periodogram 0

Lomb−Scargle the length of the time series and therefore 0 0.2 0.4 0.6 0.8 1 1.2 Frequency (Hz) × 10−5 adding weight to the reality of these sig- (b) nals. This analysis shows the power of this method over previous periodogram tech- 40 niques, such as the LS method, that gives 20 no information on the signal parameters

MMSE other than the frequency. However, phase

Periodogram 0 0 0.2 0.4 0.6 0.8 1 1.2 variations with time for the 1.94 d, 10.01 d, Frequency (Hz) × 10−5 and 15.04 d signals were found, which (c) could cast doubt on the origin of these signals as being from orbiting planets. This FIGURE 3. Radial velocity data of (a) star GJ876, (b) the LS and MMSE periodograms, and (c) the was consistent with previous Newtonian planets initially detected shown by asterisks. integrational methods. This highlights that

108 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

the MMSE method provides the flexibil- Finally, the ML periodograms and CATA PFB-06. Muhammad Salman ity of further validating the authenticity Bayesian method allow probabilities Khan’s work was funded by the CONI- of the signals, runs a global search for all to be drawn directly from the data. We CYT-PIA project ACT 1120. the signals in the entire data, and outputs previously discussed that the ML meth- the frequency, phase, and amplitude of od allows probabilities to be calculated Authors the signals. Nevertheless, the MMSE and for each frequency as part of the meth- Muhammad Salman Khan (salman_____ trellis search method does not include any odology. For the Bayesian approach, [email protected])______is an assis- correlated red noise model, whereas the statistical comparison tests can be per- tant professor in the Department of ML and Bayesian analysis do, and since formed to assess if certain models are Electrical Engineering at the University correlated noise does indeed appear to be better suited to the data in comparison of Engineering and Technology, a very important part of high-precision to flat noise models, for instance. It is Peshawar, Jalozai campus, Pakistan. He radial velocity analysis at the ~m/s level, common to calculate the Bayes factors was a postdoctoral researcher in the as mentioned previously, avenues to test to evaluate if one model is statistically Department of Electrical Engineering, here would be the application of Gaussian favored over another, since this method Universidad de Chile, Santiago between processes (e.g., [22] and [23]), MAs [24], is based on marginalization of the like- 2013 and 2015. His research interests among others applied in the field and those lihood, a process that naturally applies a include signal processing, pattern rec- yet to be tested. penalty to models with increasing com- ognition, machine learning, and big plexity (so-called Occam’s penalty; see data. He is a Member of the IEEE and Statistical significance [27]). In fact, teams who employ these the IEEE Signal Processing Society. of signal detection types of methods are known to favor James Stewart Jenkins (jjenkins@______For any signal detection method, a robust certain models over others, only if they ______das.uchile.cl) is an assistant professor in statistical validation should be made of are at least 10,000 times more probable the Departamento de Astronomia, any detected signal, likely calculating the (e.g., [24]). Universidad de Chile, Santiago. His probability directly that the signal could research interests are mainly focused on be due to random noise fluctuations. For Potential directions the search for extrasolar planets using instance, for the LS method [6], the prob- for future research the radial velocity method. In particular, ability of a signal at any given frequen- We want to detect exo-Earths so future he employs correlated noise models to cy follows an exponential distribution, directions for the radial velocity method detect low-mass planets orbiting nearby where the larger the number of frequen- are better calibrations. One big avenue of stars and was a member of the team cies sampled, the larger the probability research is the implementation of laser who discovered the habitable zone ter- that a matching frequency is found. They comb technology, which recent tests have restrial planet Proxima Centauri b. defined a false alarm probability (FAP) told us will allow velocity stability at the Nestor Becerra Yoma (nbecerra@______analytically, such that one can determine centimeter/second level, necessary for the ing.uchile.cl)______is a full professor in the the probability of any given frequency discovery of Earth-like worlds. Further- Department of Electrical Engineering, being real, solely based on the signal’s more, some areas of stellar astrophysics Universidad de Chile, Santiago. Since measured power and the number of fre- needs to be better understood, particu- August 2016, he has been a visiting quencies sampled. larly the impact of stellar activity on radial professor at Carnegie Mellon Uni - Given the nature of problems in velocity measurements. All of the methods versity, Pittsburgh, Pennsylvania. At astronomy, the deviations from normal- reviewed in this article have the potential the Universidad de Chile, he started ity, the excess noise in measurements, to be optimized to further enhance the the Speech Processing and Transmission etc., it has become normal to instead cal- detection results. We need to better model Laboratory to carry out research on culate FAPs directly from the data using the impact of magnetic activity on radial speech technology applications on nonparametric statistical methods, like velocities. In fact, this impacts transits, human-robot interaction, Internet, and bootstrap analysis, for example (see [25] transit timing variations, and astrometry telephone. He was also the director and and [26]). Bootstrapping is performed measurements. New signal processing principal investigator of the Center for by scrambling the radial velocity data methods for signal enhancement and red Multidisciplinary Research on Signal with replacement, maintaining the time noise modeling and removal also need to Processing project. He is a member of stamps, then reconstructing the peri- be investigated (e.g., see [28] and [29]). the IEEE Signal Processing Society and odogram and selecting the highest peak. the International Speech Communi - Each of the strongest peaks are recorded Acknowledgments cation Association. from a series of 10,000 or more inde- Our work was partially funded by the pendent trials, and the total number CONICYT-PIA project ACT 1120 “Cen- References of peaks found to be stronger than the ter for Multidisciplinary Research on [1] M. Mayor, M. Marmier, C. Lovis, S. Udry, D. Segransan, F. Pepe, W. Benz, J.-L. Bertaux, F. Bouchy, observed peak power provides a direct Signal Processing,” Chile. James Stewart X. Dumusque, G. Lo Curto, C. Mordasini, D. Queloz, measure of the FAP, or how much such Jenkins also acknowledges funding from a power can arise from random chance. Fondecyt grant 1161218 and BASAL (continued on page 115)

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 109

THE WORLD’S NEWSSTAND®

BOOK DIGEST

EDITORS’ INTRODUCTION Books focusing on signal processing are constantly pub- timeliness of the topic, track record of the authors, training lished by academic publishers and researchers. To enhance materials for students, and signal processing focus. If an the visibility of new signal processing books and to inform expert volunteer is available to review a book that has a our readers of recently published books in a timely fashion, high impact on signal processing, a review will be consid- IEEE Signal Processing Magazine (SPM) launched the first ered for publication in the “Book Review” column. Should “Book Digest” column in its January 2016 issue. Different you have any comments or wish to have your book consid- from the “Book Review” column, which requires capable ered for publication in this column, do not hesitate to con- reviewers and takes a lengthy time to complete a review, tact Kenneth Lam ([email protected]),______SPM’s area the “Book Digest” column provides a list of books with a editor, columns and forums, or Danilo Mandic (d.mandic@______concise summary for each one. Books are selected by a imperial.ac.uk),______SPM’s associate editor, “Book Digest” and pool of senior editors and are based on criteria such as “Book Review” columns.

David R. Bull. Communicating Pictures: resilience; 2) conflicts between conven- tistical means and correlations, asymp- A Course in Image and Video Coding. tional video compression, based on vari- totic analysis, sampling, and effective Academic Press/Elsevier, Year: 2014, able length coding and spatiotemporal algorithms. Key topics covered include ISBN: 9780124059061. prediction, and the requirements for error calculus of random processes in linear Communicating resilient transmission; 3) how to assess systems, Kalman and Wiener filtering, Pictures starts with the quality of coded images and video hidden Markov models for statistical a unique historical content, both through subjective trials and inference, the estimation maximization perspective of the by using perceptually optimised objective algorithm, and an introduction to martin- role of images in metrics; and 4) features, operation, and gales and concentration inequalities. communications performance of the state-of-the-art high- Understanding of the key concepts is and then builds on efficiency video coding standard. reinforced through more than 100 this to explain the worked examples and 300 thoroughly applications and requirements of a mod- tested homework problems. ern video coding system. It draws on the Bruce Hajek. Random Processes for author’s extensive academic and profes- Engineers. Cambridge University Press, sional experience of signal processing and Year: 2015, ISBN: 9781107100121. Lingyang Song, Dusit Niyato, Zhu Han, video coding to deliver a text that is algo- This engaging intro- and Ekram Hossain. Wireless Device-to- rithmically rigorous, yet accessible, rele- duction to random Device Communications and Networks. vant to modern standards, and practical. It processes provides Cambridge University Press, Year: 2015, offers a thorough grounding in visual per- students with the ISBN: 9781107063570. ception and demonstrates how modern critical tools needed Covering the funda- image and video compression methods to design and evalu- mental theory can be designed to meet the rate-quality ate engineering sys- together with the performance levels demanded by today’s tems that must state of the art in applications, networks, and users. operate reliably in uncertain environments. research and devel- With this book you will learn: 1) prac- A brief review of probability theory opment, this practi- tical issues when implementing a codec, and a real analysis of deterministic func- cal guide provides such as picture boundary extension and tions set the stage for understanding ran- the techniques complexity reduction, with particular dom processes, while the underlying needed to design, analyze, and optimize emphasis on efficient algorithms for measure theoretic notions are explained device-to-device (D2D) communications transforms, motion estimators and error in an intuitive, straightforward style. Stu- in wireless networking. dents will learn to manage the complexi- With an ever-increasing demand for ty of randomness through the use of higher-data-rate wireless access, D2D Digital Object Identifier 10.1109/MSP.2016.2618578 Date of publication: 11 January 2017 simple classes of random processes, sta- communication is set to become a key

110 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

feature supported by next-generation cel- uate students, scientific researchers, and From the origins of the six degrees of lular networks. This book introduces industry practitioners alike. separation to explaining why networks are D2D-based wireless communications robust to failures and fragile to attacks, from the physical-, media access control-, the author explores how viruses like network-, and application-layer perspec- Guowang Miao, Jens Zander, Ki Won Ebola and H1N1 spread and why it is that tives, providing all the key background Sung, and Slimane Ben Slimane. our friends have more friends than we do. information before moving on to discuss Fundamentals of Mobile Data Networks. Using numerous real-world examples, real-world applications as well as potential Cambridge University Press, Year: 2016, this innovative text includes clear delinea- future developments. Key topics are dis- ISBN: 9781107143210. tion between undergraduate- and gradu- cussed in detail, such as dynamic resource This unique text ate-level material. The mathematical sharing (e.g., of spectrum and power) provides a compre- formulas and derivations are included between cellular and ad hoc D2D commu- hensive and system- within advanced topics sections, enabling nications to accommodate larger volumes atic introduction to use at a range of levels. Extensive online of traffic and provide better service to the theory and prac- resources, including films and software users. Readers will understand the practi- tice of mobile data for network analysis, make this a multi- cal challenges of resource management, networks. Covering faceted companion for anyone with an optimization, security, standardization, basic design princi- interest in network science. and network topology, and learn how the ples as well as analytical tools for network design principles are applied in practice. performance evaluation, and with a focus on system-level resource management, Vikram Krishnamurthy. Partially Observed you will learn how state-of-the-art net- Markov Decision Processes: From Filtering Shuguang Cui, Alfred O. Hero III, Zhi-quan work design can enable you to flexibly to Controlled Sensing. Cambridge Luo, and José M.F. Moura (Editors). Big and efficiently manage and trade off vari- University Press, Year: 2016, Data over Networks. Cambridge University ous resources such as spectrum, energy, ISBN: 9781316471104. Press, Year 2016, ISBN: 9781107099005. and infrastructure investments. Topics Covering formula- Utilizing both key covered range from traditional elements tion, algorithms, mathematical tools such as medium access, cell deployment, and structural re- and state-of-the-art capacity, handover, and interference man- sults and linking research results, this agement, to more recent cutting-edge theory to real-world text explores the topics such as heterogeneous networks, applications in con- principles underpin- energy and cost-efficient network design, trolled sensing ning large-scale and a detailed introduction to long-term (including social information pro- evolution (4G). Numerous worked exam- learning, adaptive radars, and sequential cessing over networks and examines the ples and exercises illustrate the key theo- detection), this book focuses on the con- crucial interaction between big data and retical concepts and help you put your ceptual foundations of partially observ- its associated communication, social, and knowledge into practice, making this an able Markov decision processes biological networks. essential resource whether you are a stu- (POMPDs). It emphasizes structural Written by experts in the diverse fields dent, researcher, or practicing engineer. results in stochastic dynamic program- of machine learning, optimization, statis- ming, enabling graduate students and tics, signal processing, networking, com- researchers in engineering, operations munications, sociology, and biology, this Albert-László Barbási. Network Science. research, and economics to understand book employs two complementary Cambridge University Press, Year: 2016, the underlying unifying themes without approaches: 1) analyzing how the under- ISBN: 9781107076266. getting weighed down by mathematical lying network constrains the upper layer Networks are every- technicalities. Bringing together research of collaborative big data processing and where, from the from across the literature, the book pro- 2) examining how big data processing In ternet, to social vides an introduction to nonlinear filtering may boost performance in various net- networks, and the followed by a systematic development of works. Unifying the broad scope of the genetic networks that stochastic dynamic programming, lattice book is the rigorous mathematical treat- determine our biolog- programming, and reinforcement learning ment of the subjects, which is enriched by ical existence. Illu- for POMDPs. in-depth discussion of future directions strated throughout in The abstraction of POMDPs becomes and numerous open-ended problems that full color, this pioneering textbook, spanning alive with applications. This book con- conclude each chapter. Readers will be a wide range of topics from physics to com- tains several examples starting from target able to master the fundamental principles puter science, engineering, economics and tracking in Bayesian filtering to optimal for dealing with big data over large sys- social sciences, introduces network science search, risk measures, active sensing, tems, making it essential reading for grad- to an interdisciplinary audience. adaptive radars, and social learning.

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 111

THE WORLD’S NEWSSTAND®

The supplement of this book contains communications This book is a comprehensive and errata and problem sets and can be since the second detailed guide to all signal processing tech- accessed via: http://www.cambridge generation of cel- niques employed in 5G wireless networks. .org/gb/academic/subjects/engineering/ lular systems. It is Uniquely organized into four categories, communications-and-signal-processing/______anticipated that “New Modulation and Coding, “New Spa-

partially-obser______ved-markov-decision- new techniques tial Processing,” “New Spectrum Opportu- processes-filtering-controlled-sensing?______employed in fifth- nities,” and “New System-Level Enabling format=HB.______generation (5G) Technologies,” it covers everything from wireless networks will not only improve network architecture, physical layer peak service rates significantly but also (down-link and up-link), protocols and air Fa-Long Luo and Charlie Jianzhong Zhang enhance capacity, coverage, reliability, interface, to cell acquisition, scheduling (Editors). Signal Processing for 5G: low-latency, efficiency, flexibility, com- and rate adaption, access procedures, and Algorithms and Implementations. patibility, and convergence to meet the relaying to spectrum allocations. All tech- Wiley-IEEE Press, Year: 2016, increasing demands imposed by applica- nology aspects and major roadmaps of ISBN: 9781119116462. tions such as big data, cloud service, global 5G standard development and Signal processing techniques have played machine-to-machine, and mission-crit- deployments are included in the book. the most important role in wireless ical communications. SP

LECTURE NOTES (continued from page 103)

Conclusions Wu, Yuan Zhou, Ying Li, Dr. Shibiao received multiple awards and recogni- This article introduces a new paradigm Wan, Thee Chanyaswad, Mert Al, Chang tions. He is a Life Fellow of the IEEE. of PP techniques—CP—which repre- Chang Liu, and Artur Filipowicz sents a dimension-reduced subspace (Princeton University) for invaluable dis- References approach to PP machine learning. Built cussions and assistance. [1]C.Dwork,K. Kenthapadi,F, McSherry, I. Mironov, upon the information and estimation and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” in Advancves in Cryptology- theory, CP methods tackle joint optimi- Author EUROCRYPT. New York: Springer, 2006. zation over feature/utility/privacy spaces. S.Y. Kung ([email protected])______is a [2] F. du Pin Calmon and N. Fawaz, “Privacy against This leads to several eigen-system-based professor in the Department of Electrical statistical inference,” in Proc. Allerton Conf. Communication, Control, and Computing, 2012, subspace methods, including PCA, Engineering at Princeton University, New pp. 1401–1408. DCA, and DUCA. To confirm the theo- Jersey. His research areas include [3] S. Y. Kung, Kernel Methods and Machine Learning. retical analysis, we have conducted machine learning, compressive privacy, Cambridge, U.K.: Cambridge Univ. Press, 2014. experimental studies on various DIP and data mining and analysis, statistical esti- [4] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” in Proc. 37th Allerton FR problems. The latter also demon- mation, system identification, wireless Conf. Communication and Computation, 1999. strates possible real-world applications communication, very-large-scale integra- [5] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern of the proposed CP methodology. tion (VLSI) array processors, genomic Classification. 2nd ed. New York: Wiley, 2011. [6] S. Y. Kung, “Discriminant component analysis for signal processing, and multimedia infor- privacy protection and visualization of big data,” Acknowledgments mation processing. He was a founding Multimed. Tools Appl., 2015, doi:10.1007/s11042-015- This material is based upon work sup- member of several technical committees 2959-9. [7]S. Y.Kung,T.Chanyaswad,J.Morris Chang, and ported in part by the Brandeis Program of the IEEE Signal Processing Society. He P. Wu, “Collaborative PCA/DCA learning methods for of DARPA and the Space and U.S. served as a member of the Board of compressive privacy,” ACM Trans. Embed. Comput. Syst. (Special Issue on Effective Divide-and-Conquer, Naval Warfare System Center Pacific Governors of the IEEE Signal Processing Incremental, or Distributed Mechanisms of (SSC Pacific) under contract 66001-15- Society (1989–1991). He has been the Embedded Designs for Extremely Big Data in Large- Scale Devices), to be published. C-4068. I would like to thank Prof. editor-in-chief of Journal of VLSI Signal Morris J. Chang (ISU) and Prof. Peiyuan Processing Systems since 1990. He has SP

112 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

DATES AHEAD

Please send calendar submissions to: Dates Ahead, Att: Jessica Barragué, E-mail: [email protected]______

2017 OCTOBER

MARCH IEEE Workshop on Applications IEEE International Conference of Signal Processing to Audio and on Acoustics, Speech, and Acoustics (WASPAA) 15–18 October, New Paltz, New York. Signal Processing (ICASSP) 5–9 March, New Orleans, Louisiana, USA. General Chairs: Patrick A. Naylor General Chair: Magdy Bayoumi and Meinard Müller URL: http://www.ieee-icassp2017.org/ URL: http://www.waspaa.com/ 19th IEEE International Workshop on Multimedia Signal Processing (MMSP) APRIL 16–18 October, London-Luton, United Kingdom. General Chairs: Vladan Velisavljevic, IMAGE LICENSED BY INGRAM PUBLISHING LICENSED BY IMAGE IEEE International Symposium Vladimir Stankovic, and Zixiang Xiong on Biomedical Imaging (ISBI) ICASSP 2017 will be held in New Orleans, URL: http://mmsp2017.eee.strath.ac.uk/ 18–21 April, Melbourne, Australia. Louisiana, 5-9 March. General Chairs: Olivier Salvado and Gary Egan URL: http://biomedicalimaging.org/2017/ IEEE International Conference on NOVEMBER 16th ACM/IEEE International Conference on Multimedia and Expo (ICME) Information Processing in Sensor Networks 10–14 July, Hong Kong, China. 5th IEEE Global Conference on Signal and (IPSN) General Chairs: Jörn Ostermann Information Processing (GlobalSIP) 18–21 April 2017, Pittsburgh, Pennsylvania, USA. and Kenneth K.M. Lam 14–16 November 2017, Montreal, Canada. General Chair: Pei Zhang URL: http://www.icme2017.org/ General Cochairs: Warren Gross URL: http://ipsn.acm.org/2017/ and Kostas Plataniotis URL: http://2017.ieeeglobalsip.org AUGUST MAY 25th European Signal Processing DECEMBER IEEE Radar Conference (RADARCONF) Conference (EUSIPCO) 8–12 May, Seattle, Washington, USA. 28 August–2 September, Kos Island, Greece. Seventh IEEE Conference of the Sensor General Chair: Daniel J. Sego General Chairs: Petros Maragos and Signal Processing for Defence (SSPD) URL: http://www.radarconf17.org Sergios Theodoridis 6–7 December, Edinburgh, Great Britain. URL: www.eusipco2017.org General Chairs: Mike Davies, Jonathon Chambers, and Paul Thomas 14th IEEE International Conference URL: www.sspd.eng.ed.ac.uk/ JULY on Advanced Video and Signal-Based Surveillance (AVSS) 17th IEEE International Workshop on 18th IEEE International Workshop on 29 August–1 September, Lecce, Italy. Computational Advances in Multisensor Signal Processing Advances in Wireless General Chairs: Cosimo Distante and Adaptive Processing (CAMSAP) Communications (SPAWC) Larry S. Davis 10–13 December, Curacao, Dutch Antilles. 3–6 July, Hokkaido, Japan. URL: www.avss2017.org General Chairs: André L.F. de Almeida General Chairs: Yasutaka Ogawa, Wei Yu, and Martin Haardt and Fumiyuki Adachi URL: http://www.cs.huji.ac.il/conferences/ URL: http://www.spawc2017.org/ CAMSAP17/______SP SEPTEMBER

IEEE International Conference on Image Processing (ICIP) 17–20 September, Beijing, China. General Chairs: Xinggang Lin,

Digital Object Identifier 10.1109/MSP.2016.2636082 Anthony Vetro, and Min Wu Date of publication: 11 January 2017 URL: http://2017.ieeeicip.org/

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 113

THE WORLD’S NEWSSTAND®

IN THE SPOTLIGHT (continued from page 116) ADOBE STOCK

much like data science, many of the dis- unknown to the engineer and application with the “scientific method” drummed ciplines within signal processing are developer. In short, the new era will have into you like a mantra. We’ve all grown fundamentally about looking for correla- more data than you could ever dream. up in an era of slow, methodical research tions and dependencies in data to effec- But, as we move forward in this new and development. In fact, one way of tively make decisions. era, we need to relish in the opportuni- looking at both the scientific method and That is not to say that signal process- ties it will provide, yet also retain an the engineering design process is that it ing is the same as data science. Perhaps appropriate level of caution. The prom- leads to implicit practice of quality con- the most notable difference between the ise of being able to analyze large trol—almost bordering on pessimism past era and the new amounts of data to and overt caution. era of data science As this new era of big data find a cure for cancer, Big data will often involve others and big data is the and data science unfolds, integrate infrastruc- unintentionally conducting experiments tearing down of let us issue a challenge ture and vehicle for the data scientist. The allure of hunt- boundaries associated to scientists, engineers, sensor data to allow ing through more and more data to find with how data is pro- for automated driving patterns without vetting that data is dan- duced and accessed. and signal processors and more efficient gerous. Data science will have some Big data is fundamen- to establish new forms transportation, or the growing pains, especially as the vast tally heterogeneous, of collaboration. potential to analyze amount of data being examined guaran- involving data from a the data being gener- tees that data will be haphazardly ana- vast collection of sources that report data ated by the broad collection of astro- lyzed and spurious correlations will be of various modalities for analysis. Where- nomical observatories to discover new proclaimed as scientific truths. Data sci- as the previous generation of scientific stellar phenomena are certainly fantastic ence will need quality control. discovery involved scientists conducting and truly important to society. We And this is where the signal pro- (and planning) experiments to intentional- would not be able to make such cessing community can advance big ly measure specific data for the purpose advancements or build new systems data and data science. Over the years, of discovery, oftentimes big data involves without the emergence of this new field the signal processing community has the opportunistic sharing of data from of data science. However, we must be carefully built up a sophisticated tool- nonvetted sources often provided in careful as this explosion of data and data box full of algorithms designed to unstructured representations. Thus, the science could take on a life of its own. analyze data, as well as the deep under- new era of big data and data analytics will Regardless of whether you are a scien- standing of when and how to use these likely lead to new engineered systems that tist, mathematician, engineer, or in some algorithms, and how they can be made utilize data from sources previously other profession, you were likely raised to work efficiently. Signal processors

114 IEEE SIGNAL PROCESSING MAGAZINE | January 2017 |

THE WORLD’S NEWSSTAND®

are a mixed breed of statisticians engineers who are making the next wave Computer Engineering Department at crossed with control theorists crossed of data and offer your services. Now Rutgers University, New Jersey, and with computer engineers who have, more than ever is the time for those associate director of the Wireless over the decades, folded performance engaged in signal processing to reach Information Network Laboratory assurance into their algorithms to across the boundaries of technical fields (WINLAB), where he directs ensure that video looks good after and contribute their tools to the analysis WINLAB’s research in wireless security. compressed, targets are accurately of the vast amounts of data that are being He coauthored the textbook Introduction tracked, and tumors can be effectively generated everywhere. Signal processing to Cryptography with Coding Theory as classified with low rates of false alarm has had a fantastic record of success, and, well as several monographs on wireless and missed detections. as we move to this new world of data security, including Securing Wireless Hence, as this new era of big data and treasure hunting, signal processing can Communications at the Physical Layer data science unfolds, let us issue a chal- ensure the success of data science— and Securing Emerging Wireless lenge to scientists, engineers, and signal ensuring that the hidden correlations one Systems: Lower-Layer Approaches. He processors to establish new forms of col- finds are truly golden treasures and not was an editor of IEEE Transactions on laboration: To the data scientists, reach spurious pyrite counterfeits. Information Forensics and Security, out and ask a signal processor whether IEEE Signal Processing Magazine, and they know of any signal processing tools Author IEEE Transactions on Mobile that might work on your data. To the sig- Wade Trappe ([email protected]______Computing. He is an IEEE Fellow.

nal processor, find the scientists and ___.edu) is a professor in the Electrical and SP

APPLICATIONS CORNER (continued from page 109)

and N. C. Santos “The HARPS search for southern ic significance thresholds,” Mon. Not. R. Astron. Soc., [21] F. Talebi and T. Pratt, “Model order selection for extra-solar planets xxxiv. occurrence, mass distribution vol. 446, no. 2, pp. 1478–1492, 2015. complex sinusoids in the presence of unknown correlat- and orbital properties of super-Earths and Neptune-mass [12] P. Lenz and M. Breger, “Period04: A software ed Gaussian noise,” IEEE Trans, Signal Process., vol. planets,” Astron. Astrophys., Sept. 2011 no. package to extract multiple frequencies from real 63, no. 7, pp. 1664–1674, 2015. arXiv:1109.2497. data,” in The A-Star Puzzle: IAUS 224. Cambridge, [22] R. V. Baluev, “The impact of red noise in radial [2] G. Anglada-Escud, et al. “Two planets around U.K.: Cambridge Univ. Press, 2004, pp. 786–790. velocity planet searches: Only three planets orbiting GJ Kapteyn’s star: a cold and a temperate super-Earth orbit- [13] R. V. Baluev, “A radial-velocity time-series analy- 581?” Mon. Not. R. Astron. Soc., vol. 429, no. 3, pp. ing the nearest halo red dwarf,” Mon. Not. R. Astron. sis tool facilitating exoplanets detection, characteriza- 2052–2068, 2012. Soc., vol. 443, no. 89, pp. L89-L93, 2014. tion, and dynamical simulations,” Astron. Comput., vol. [23] V. Rajpaul1, S. Aigrain, M. A. Osborne, S. Reece, [3] J. S. Jenkins, et al. “New planetary systems from the 2, pp. 18–26, Aug. 2013. and S. Roberts, “A Gaussian process framework for Calan-Hertfordshire extrasolar planet search and core [14] G. Anglada-Escud, M. Tuomi, et al. “A dynamical- modelling stellar activity signals in radial velocity data,” accretion mass limit,” Mon. Not. R. Astron. Soc., Oct. ly-packed planetary system around GJ 667C with three Mon. Not. R. Astron. Soc., vol. 452, no. 3, pp. 2269– 2015. super-Earths in its habitable zone,” Astron. Astrophys., 2291, 2015. [4] A. Baranne, et al. “ELODIE: A spectrograph for vol. 556, pp. 126–150, June 2013. [24] M. Tuomi, et al. “Signals embedded in the radial Astron. x accurate radial velocity measurements,” [15] H. Haario, E. Saksman, and J. Tamminen, “An velocity noise. Periodic variations in the Ceti veloci- Astrophys. , vol. 119, pp. 373–390, Oct. 1996. adaptive metropolis algorithm,” Bernoulli, vol. 7, no. 2, ties,” Astron. Astrophys., vol. 551, pp. A79, Dec. 2012. [5] J. S. Jenkins, H. R. A. Jones, Y. Pavlenko, D. J. 2001. [25] B. Efron, “Bootstrap methods: Another look at the Pinfield, J. R. Barnes, and Y. Lyubchik “Metallicities [16] M. Tuomi, “A new cold sub-Saturnian candidate jacknife,” Ann. Stat., vol. 7, no 1, pp. 1–126, 1979. and activities of southern stars,” Astron. Astrophys., vol. planet orbiting GJ 221,” Mon. Not. R. Astron. Soc., vol. [26] J. S. Jenkins, et al. “A hot Uranus orbiting the 485, no. 2, pp. 571–584, 2008. 440, pp. L1–L5, Jan. 2014. super metal-rich star HD77338 and the metallicity-mass [6] J. D. Scargle, “Studies in astronomical time series connection,” Astrophys. J., vol. 766, no. 2, 2013. analysis. II - Statistical aspects of spectral analysis of [17] J. Jenkins and M. Tuomi, “The curious case of HD [27] P. C. Gregory, “Bayesian re-analysis of the Gliese unevenly spaced data,” Astrophy. J., vol. 263, pp. 835– 41248. A pair of static signals buried behind red noise,” Astrophys. J. 581 exoplanet system,” Mon. Not. R. Astron. Soc., vol. 853, Dec. 1982. , vol. 794, no. 2, 2014. 415, pp. 2523–2545, Jan. 2011. [7]G.L.Bretthorst,Bayesian Spectrum Analysis and [18] J. S. Jenkins, N. B. Yoma, P. Rojo, R. Mahu, and J. [28] J. A. Carter and J. N. Winn, “Parameter estimation Parameter Estimation (Lecture Notes in Statistics, Wuth, “Improved signal detection algorithms for from time-series data with correlated errors: A wavelet- vol. 48). New York: Springer-Verlag, 1988. unevenly sampled data. Six signals in the radial velocity data for GJ876,” Mon. Not. R. Astron. Soc., vol. 441, based method and its application to transit light curves,” [8] S. Ferraz-Mello, “Estimation of periods from pp. 2253–2265, Mar. 2014. Astrophys. J., vol. 704, pp. 51–67, Oct. 2009. unequally spaced observations,” Astron. J., vol. 86, no. [29] B. Placek, K. H. Knuth, D. Angerhausen, and J. 4, pp. 619–624, 1981. [19] R. V. Baluev, “Detecting multiple periodicities in observational data with the multifrequency periodo- M. Jenkins, “Characterization of Kepler-91b and the [9] A. Cumming, “Detectability of extrasolar planets in gram-II. Frequency decomposer, a parallelized time- investigation of a potential trojan companion using radial velocity surveys,” Mon. Not. R. Astron. Soc., vol. series analysis algorithm,” Astron.Comput., vol. 3–4, EXONEST,” Astrophys. J., vol. 814, pp. 147, Dec. 2015. 354, no. 4, pp. 1165–1176, 2004. pp. 50–57, Nov.–Dec. 2013. [30] X. Dumusque, “Radial velocity fitting challenge-I: [10] M. Zechmeister and M. Kürster, “The generalised [20] R. V. Baluev, “Detecting multiple periodicities in Simulating the data set including realistic stellar radial- Lomb–Scargle periodogram,” Astron. Astrophys., vol. observational data with the multifrequency periodo- velocity signals,” Astron. Astrophys., vol. 593, vol. 5, 496, no. 2, pp. 577–584, 2009. gram–I. Analytic assessment of the statistical signifi- 2016. [11] R. V. Baluev, “Keplerian periodogram for doppler cance,” Mon. Not. R. Astron. Soc., vol. 436, pp. exoplanet detection: optimized computation and analyt- 807–818, Aug. 2013. SP

IEEE SIGNAL PROCESSING MAGAZINE | January 2017 | 115

THE WORLD’S NEWSSTAND®

IN THE SPOTLIGHT

Wade Trappe

Data Treasure Hunters: Science Expanding to New Frontiers

well as deep into the Earth and across EDITOR’S NOTE its ecosystems. Matching this explosion In June 2016, the IEEE Signal Processing Society (SPS) launched an SPS blog in data is a commensurate advance in website, which provides a nontechnical supplement to highly technical signal pro- computing: computing resources are cessing topics. There are seven SPS blogs so far, and these blogs help students now sophisticated enough to be able to and the general public to become more aware of signal processing. We selected perform immense amounts of computa- the first SPS blog in the web series, written by Wade Trappe, for this issue’s “In tion on this data. the Spotlight” column. We hope you enjoy reading it. You can find more SPS This is the emergence of the new blogs by visiting http://signalprocessingsociety.org/publications-resources/blog. field of big data and data analytics. Should you have any suggestions regarding the SPS blogs, please do not hesi- Data scientists are the postmodern trea- tate to contact IEEE Signal Processing Magazine’s Area Editor, Columns and sure hunters. They will reach into their Forums Kenneth Lam ([email protected])______and SPS Membership and Content toolboxes of algorithms and dig into Administrator Jessica Perry ([email protected]).______data looking for hidden correlations, trying to find never-before-seen patterns with the hope of advancing the frontier cience and engineering are rapidly We are experiencing an explosion in of knowledge and supporting the devel- Sheading toward a major culture the amount of data available for scien- opment of new products. change—a change in how we think tists and engineers to do their jobs. The The frank truth, though, is that data about data. world around us is becoming increas- science isn’t really new. Many technical This change is already happening, ingly “connected” as communication fields have been performing analysis on and it will be dramatic and exciting! It technologies have proliferated and the large amounts of data before the term will completely change how most of us costs of digital data storage have plum- big data was ever coined. The signal think about data and how we tackle sci- meted. Seemingly mundane items and processing community has been analyz- ence and engineering problems. With it devices that never before had a bit or ing data since its inception. After all, will come a flood of new discoveries— byte associated with them are now what is the Fourier transform but a tool advances in the sciences and in new tech- streaming a constant flow of data to to find periodic phenomena in data? Or nologies—that were never before data warehouses located in the cloud. take a quick survey of papers over the possible. What is this revolution? How Advancements in medical devices are past 25 years (or more), and you will find did we get here? Where is it going, and leading to the emergence of miniatur- signal processing is involved in every- how is signal processing involved? ized, nonintrusive medical sensors that thing from analyzing geological data for The short answer is that we are enter- will be integrated with communication oil discovery, to face recognition for ing an era of treasure hunting. Rather technologies to report real-time glucose domestic security, to processing genomic than digging through dirt like archaeolo- levels, monitor respiratory conditions, data and looking for patterns that indicate gists looking for ancient artifacts, the track immune responses, and allow for the onset of cancer. Signal processing future will involve digging through data. the analysis of a wide array of other was fundamental to advancements in data associated with the human condi- multimedia processing and storage, and tion. Meanwhile, scientific equipment Digital Object Identifier 10.1109/MSP.2016.2619918 Date of publication: 11 January 2017 is being aimed both out into space as (continued on page 114)

THE WORLD’S NEWSSTAND®

Call for Symposium Proposals

We invite Symposium proposals for the fifth IEEE Global Conference on Signal and Information Processing (GlobalSIP) which will be held in Montreal, Quebec, Canada on November 14-16, 2017. GlobalSIP is a flagship IEEE Signal Processing Society conference. It focuses on signal and information processing with an emphasis on up-and-coming signal processing themes. The conference features world-class plenary speeches, distinguished Symposium talks, tutorials, exhibits, oral and poster sessions, and panels. GlobalSIP is comprised of co-located General Symposium and symposia selected based on responses to the call-for-symposia proposals. Topics include but are not limited to:

Signal and information processing for Human machine interfaces o communications and networks, including green Multimedia transmission, indexing, retrieval, and quality communications of experience o optical communications Selected topics in statistical signal processing o forensics and security Cognitive communications and radar o finance Graph-theoretic signal processing o energy and power systems (e.g., smart grid) Machine learning o genomics and bioengineering (physiological, Compressed sensing and sparsity aware processing pharmacological and behavioral) Seismic signal processing o neural networks, including deep learning Big data and social media challenges Image and video processing Hardware and real-time implementations Selected topics in speech processing and human Other (industrial) emerging applications of signal and language technologies information processing.

Symposium proposals should contain the following information: title; duration (e.g., full day or half day); paper length, acceptance rate; name, address, and a short CV (up to 250 words) of the organizers, including the technical chairs (if any); a 1-page or 2-page description of the topics to be addressed, including timeliness and relevance to the signal processing community; names of (potential) members of the technical program committee; invited speakers' name; a draft call for papers. Please pack everything together in a single pdf document. More detailed information can be found in GlobalSIP2017 Symposium Proposal Preparation Guide.

Proposed Timeline Jan. 20, 2017: Symposium proposals due Jan. 25, 2017: Symposium selection decision made Feb. 1, 2017: Call for Papers for accepted Symposia May 15, 2017: Paper submission due June 30, 2017: Notification of Acceptance July 22, 2017: Camera-ready paper due.

Digital Object Identifier 10.1109/MSP.2016.2636084

THE WORLD’S NEWSSTAND®

MATLAB SPEAKS WIRELESS DESIGN

You can simulate, prototype, and verify wireless systems right in MATLAB. Learn how today’s MATLAB supports RF, LTE, WLAN and 5G development and SDR hardware. mathworks.com/wireless ©2016 The MathWorks, Inc MathWorks, The ©2016

THE WORLD’S NEWSSTAND®

IEEE SIGNAL PROCESSING SOCIETY ContentGazette

JANUARY 2017 ISSN 2167-5023

T-SP January 1 Vol. 65 #1 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7755888 T-SP January 15 Vol. 65 #2 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7762278 T-ASLP December Vol. 24 #12 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7755865 T-CI December Vol. 2 #4 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7736166 T-IP December Vol. 25 #12 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7744749 T-IFS December Vol. 11 #12 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7747716 T-MM December Vol.18 #12 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7744756 T-JSTSP December Vol. 10 #8 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7746026 T-SIPN December Vol. 2 #4 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7736190 T-SPL December Vol. 23 #12 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7769278

http://www.ieee.org/publications_standards/publications/authors/author_ethics.html

THE WORLD’S NEWSSTAND®

IEEE

ASRU 2017 Okinawa, Japan, December 16-20, 2017

IEEE Automatic Speech Recognition and Understanding Workshop General Chairs: John R. Hershey, MERL The biennial IEEE ASRU workshop has a tradition of bringing Tomohiro Nakatani, NTT together researchers from academia and industry in an intimate and collegial setting to discuss problems of common interest in automatic speech recognition, understanding, and related fields of research. The workshop includes keynotes, Important Dates: invited talks, poster sessions and will also feature challenge Paper Submission: tasks, panel discussions, and demo sessions. June 29, 2017

Paper Notification: tĞŝŶǀŝƚĞƉĂƉĞƌƐŝŶĂůůĂƌĞĂƐŽĨƐƉŽŬĞŶůĂŶŐƵĂŐĞƉƌŽĐĞƐƐŝŶŐ͕ August 31, 2017 ǁŝƚŚĞŵƉŚĂƐŝƐƉůĂĐĞĚŽŶƚŚĞĨŽůůŽǁŝŶŐƚŽƉŝĐƐ͗ Automatic speech recognition Early Registration Period: August 31 - Oct 5, 2017 ASR in adverse environments New applications of ASR Camera Ready Deadline: Speech-to-speech translation Sept 21, 2017 Spoken document retrieval Multilingual language processing More Information: Spoken language understanding Spoken dialog systems ǣȀȀ͖͔͕͛Ǥ Text-to-speech systems [email protected]

THE WORLD’S NEWSSTAND®

______

JANUARY 1, 2017 VOLUME 65 NUMBER 1 ITPRED (ISSN 1053-587X)

REGULAR PAPERS

Second-Generation Curvelets on the Sphere http://dx.doi.org/10.1109/TSP.2016.2600506 ...... J.Y.H. Chan, B. Leistedt, T. D. Kitching, and J. D. McEwen 5 Theoretical Bounds in Minimax Decentralized Hypothesis Testing http://dx.doi.org/10.1109/TSP.2016.2613072 .... G. Gül and A. M. Zoubir 15 A Signal-Space Aligned Network Coding Approach to Distributed MIMO http://dx.doi.org/10.1109/TSP.2016.2616335 ...... T.Yang,X.Yuan, and Q. T. Sun 27 Precoder Designs for the Relay-Aided X Channel Without Source CSI http://dx.doi.org/10.1109/TSP.2016.2616329 ...... K. Anand, E. Gunawan, and Y. L. Guan 41

www.signalprocessingsociety.org [1] JANUARY 2017

THE WORLD’S NEWSSTAND®

Compressed and Quantized Correlation Estimators http://dx.doi.org/10.1109/TSP.2016.2597128 ...... A.G. Zebadua, P.-O. Amblard, E. Moisan, and O. J. J. Michel 56 Perfect Recovery Conditions for Non-negative Sparse Modeling http://dx.doi.org/10.1109/TSP.2016.2613067 ...... Y. Itoh, M. F. Duarte, and M. Parente 69 Generalized Coprime Sampling of Toeplitz Matrices for Spectrum Estimation http://dx.doi.org/10.1109/TSP.2016.2614799 ...... S. Qin, Y. D. Zhang, M. G. Amin, and A. M. Zoubir 81 Model-Based Nonuniform Compressive Sampling and Recovery of Natural Images Utilizing a Wavelet-Domain Universal Hidden Markov Model http://dx.doi.org/10.1109/TSP.2016.2614654 ...... B. Shahrasbi and N. Rahnavard 95 Robust Sparse Recovery in Impulsive Noise via - Optimization http://dx.doi.org/10.1109/TSP.2016.2598316 ...... F.Wen,P. Liu, Y. Liu, R. C. Qiu, and W. Yu 105 Broadcast Gossip Ratio Consensus: Asynchronous Distributed Averaging in Strongly Connected Networks http://dx.doi.org/10.1109/TSP.2016.2614790 ...... A. Khosravi and Y. S. Kavian 119 Learning-Based Distributed Detection-Estimation in Sensor Networks With Unknown Sensor Defects http://dx.doi.org/10.1109/TSP.2016.2613062 ...... Q. Zhou, D. Li, S. Kar, L. M. Huie, H. V. Poor, and S. Cui 130 Network Newton Distributed Optimization Methods http://dx.doi.org/10.1109/TSP.2016.2617829 .... A. Mokhtari, Q. Ling, and A. Ribeiro 146 Optimized Spectrum Permutation for the Multidimensional Sparse FFT http://dx.doi.org/10.1109/TSP.2016.2599483 ...... A. Rauh and G. R. Arce 162 Closed-Loop Autonomous Pilot and Compressive CSIT Feedback Resource Adaptation in Multi-User FDD Massive MIMO Systems http://dx.doi.org/10.1109/TSP.2016.2616326 ...... A. Liu, F. Zhu, and V. K. N. Lau 173 A Bayesian Approach for Online Recovery of Streaming Signals From Compressive Measurements http://dx.doi.org/10.1109/TSP.2016.2614489 ...... U.L.Wijewardhana and M. Codreanu 184 Cooperative Simultaneous Localization and Mapping by Exploiting Multipath Propagation http://dx.doi.org/10.1109/TSP.2016.2616324 ...... H. Naseri and V. Koivunen 200 Joint Optimization of Power and Data Transfer in Multiuser MIMO Systems http://dx.doi.org/10.1109/TSP.2016.2614794 ...... J. Rubio, A. Pascual-Iserte, D. P. Palomar, and A. Goldsmith 212 Adaptive Cloud Radio Access Networks: Compression and Optimization http://dx.doi.org/10.1109/TSP.2016.2617826 ...... T.X.Vu,H.D. Nguyen, T. Q. S. Quek, and S. Sun 228 Distributed Fusion With Multi-Bernoulli Filter Based on Generalized Covariance Intersection http://dx.doi.org/10.1109/TSP.2016.2617825 ...... B.Wang,W.Yi,R. Hoseinnezhad, S. Li, L. Kong, and X. Yang 242 Learning From Hidden Traits: Joint Factor Analysis and Latent Clustering http://dx.doi.org/10.1109/TSP.2016.2614491 ...... B.Yang,X.Fu,andN.D. Sidiropoulos 256

www.signalprocessingsociety.org [2] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

JANUARY 15, 2017 VOLUME 65 NUMBER 2 ITPRED (ISSN 1053-587X)

REGULAR PAPERS

Autoregressive Moving Average Graph Filtering http://dx.doi.org/10.1109/TSP.2016.2614793 ...... E. Isuﬁ, A. Loukas, A. Simonetto, and G. Leus 274 Chernoff Test for Strong-or-Weak Radar Models http://dx.doi.org/10.1109/TSP.2016.2616323 ...... M.Franceschetti, S. Marano, and V. Matta 289 Massive MIMO Channel Subspace Estimation From Low-Dimensional Projections http://dx.doi.org/10.1109/TSP.2016.2616336 ...... S.Haghighatshoar and G. Caire 303 Persistent Homology Lower Bounds on High-Order Network Distances http://dx.doi.org/10.1109/TSP.2016.2620963 ...... W. Huang and A. Ribeiro 319

www.signalprocessingsociety.org [3] JANUARY 2017

THE WORLD’S NEWSSTAND®

Joint BS-User Association, Power Allocation, and User-Side Interference Cancellation in Cell-free Heterogeneous Networks http://dx.doi.org/10.1109/TSP.2016.2620962 ...... A.LiuandV.K.N.Lau 335 A Tensor-Based Method for Large-Scale Blind Source Separation Using Segmentation http://dx.doi.org/10.1109/TSP.2016.2617858 ...... M. Boussé, O. Debals, and L. De Lathauwer 346 Maximum-Likelihood Adaptive Filter for Partially Observed Boolean Dynamical Systems http://dx.doi.org/10.1109/TSP.2016.2614798 ...... M. Imani and U. M. Braga-Neto 359 Sparse Reconstruction Algorithm for Nonhomogeneous Counting Rate Estimation http://dx.doi.org/10.1109/TSP.2016.2620104 ...... T.Trigano, Y. Sepulcre, and Y. Ritov 372 A Pessimistic Approximation for the Fisher Information Measure http://dx.doi.org/10.1109/TSP.2016.2617824 ...... M.S. Stein and J. A. Nossek 386 Low-Rank Positive Semidefinite Matrix Recovery From Corrupted Rank-One Measurements http://dx.doi.org/10.1109/TSP.2016.2620109 ...... Y.Li,Y. Sun, and Y. Chi 397 Extending Classical Multirate Signal Processing Theory to Graphs—Part I: Fundamentals http://dx.doi.org/10.1109/TSP.2016.2617833 ...... O.TekeandP.P.Vaidyanathan 409 Extending Classical Multirate Signal Processing Theory to Graphs—Part II: M-Channel Filter Banks http://dx.doi.org/10.1109/TSP.2016.2620111 ...... O.TekeandP.P.Vaidyanathan 423 A Unified Framework for Low Autocorrelation Sequence Design via Majorization–Minimization http://dx.doi.org/10.1109/TSP.2016.2620113 ...... L. Zhao, J. Song, P. Babu, and D. P. Palomar 438 Mismatched Filter Design and Interference Mitigation for MIMO Radars http://dx.doi.org/10.1109/TSP.2016.2620960 ...... T. Aittomäki and V. Koivunen 454 Consensus +Innovations Distributed Kalman Filter With Optimized Gains http://dx.doi.org/10.1109/TSP.2016.2617827 ...... S.DasandJ.M.F. Moura 467 Sequential Estimation of Hidden ARMA Processes by Particle Filtering—Part I http://dx.doi.org/10.1109/TSP.2016.2598309 ...... I. Urteaga and P. M. Djuric´ 482 Sequential Estimation of Hidden ARMA Processes by Particle Filtering—Part II http://dx.doi.org/10.1109/TSP.2016.2598324 ...... I. Urteaga and P. M. Djuric´ 494 Optimally Modulated Illumination for Rapid and Accurate Time Synchronization http://dx.doi.org/10.1109/TSP.2016.2612176 ...... M. Sugimoto, H. Kumaki, T. Akiyama, and H. Hashizume 505 Multidimensional Harmonic Retrieval via Coupled Canonical Polyadic Decomposition—Part I: Model and Identifiability http://dx.doi.org/10.1109/TSP.2016.2614796 ...... M.Sørensen and L. De Lathauwer 517 Multidimensional Harmonic Retrieval via Coupled Canonical Polyadic Decomposition—Part II: Algorithm and Multirate Sampling http://dx.doi.org/10.1109/TSP.2016.2614797 ...... M.Sørensen and L. De Lathauwer 528

www.signalprocessingsociety.org [4] JANUARY 2017

THE WORLD’S NEWSSTAND® 0063 ,(((:RUNVKRSRQ0XOWLPHGLD6LJQDO3URFHVVLQJ 2FWREHU/RQGRQ/XWRQ8. ______KWWSPPVSHHHVWUDWKDFXN &$//)253$3(56 *HQHUDO&RFKDLUV 9ODGDQ9(/,6$9/-(9,& 0063LVWKH,(((WK,QWHUQDWLRQDO:RUNVKRSRQ0XOWLPHGLD 6LJQDO3URFHVVLQJ7KH 8QLYHUVLW\RI%HGIRUGVKLUH8. ZRUNVKRSLVRUJDQL]HGE\WKH0XOWLPHGLD6LJQDO3URFHVVLQJ7HFKQLFDO&RPPLWWHHRIWKH,((( 9ODGLPLU67$1.29,& 8QLYHUVLW\RI6WUDWKFO\GH8. 6LJQDO 3URFHVVLQJ 6RFLHW\ 7KLV \HDU¶V HYHQW KDV D WKHPH RI µ0XOWLPHGLD 3URFHVVLQJ IRU =L[LDQJ;,21*7H[DV$ 0 +HDOWKFDUHDQG$VVLVWHG/LYLQJ ¶ 8QLYHUVLW\7;86$ 5HFHQWDGYDQFHVLQPXOWLPHGLDSURFHVVLQJDQGFRPPXQLFDWLRQVKDY HSRWHQWLDOWRVLJQLILFDQWO\ 73&&RFKDLUV DGYDQFHFXUUHQWKHDOWKFDUHDQGDVVLVWHGOLYLQJVHUYLFHVE\HQDE OLQJUHPRWHKHDOWKPRQLWRULQJ +RPHU+&+(11DWLRQDO 7DLZDQ8QLYHUVLW\7DLZDQ UHPRWH GLDJQRVWLFV LQFUHDVHG SDWLHQW SULYDF\ URERWLFDVVLVWHG VXUJHU\ DQG KRPHEDVHG )UHGHULF'8)$8; WUHDWPHQW $ KXJH GLYHUVLW\ RI PXOWLPHGLD SURFHVVLQJ WHFKQLTXHV UDQJLQJ IURP LPDJHDXGLR &156)UDQFH VHQVLQJ FRPSUHVVLRQ DQG QHWZRUNLQJ GHQRLVLQJ IHDWXUH H[WUDFWLRQ VHFXULW\ GLVWULEXWHG 5DRXI+$0=$28, SURFHVVLQJGHSWKLPDJHSURFHVVLQJFORXGDQGVRFLDOFRPSXWLQJ YLVXDOL]DWLRQDQGPXOWLPHGLD 'H0RQWIRUW8QLYHUVLW\8. ELJGDWDDQDO\WLFVFDQDOOILQGWKHLUDSSOLFDWLRQVLQIXWXUHK HDOWKFDUHDQGHOGHUO\FDUH+RZHYHU )LQDQFH&RFKDLUV WR IXOO\ UHDOL]H WKLV SRWHQWLDO DQG HPEHG PXOWLPHGLD VROXWLRQV LQWR GD\WRGD\ KHDOWKFDUH &KLQJ7H&+,81DWLRQDO SUDFWLFHPDQ\FKDOOHQJHVQHHGWREHRYHUFRPHWKDWFDOOIRUVLJ QLILFDQWHQJLQHHULQJLQQRYDWLRQ 7VLQJ+XD8QLYHUVLW\7DLZDQ ZKLFKFDQRQO\KDSSHQWKURXJKFORVHLQWHUGLVFLSOLQDU\HIIRUW7 KH0063:RUNVKRSZLOO /LQD67$1.29,& EULQJ WRJHWKHU H[SHUWV IURP GLIIHUHQW ILHOGV LQFOXGLQJ VLJQDO SURFHVVLQJ FRPSXWHU VFLHQFH 8QLYHUVLW\RI6WUDWKFO\GH8. FRPPXQLFDWLRQVPHGLFLQHUHKDELOLWDWLRQSV\FKRORJ\WRH[FKDQ JHLGHDVRQKRZPXOWLPHGLD 6SHFLDO6HVVLRQ 3DQHO&RFKDLUV UHVHDUFKFDQVXSSRUWDGYDQFHPHQWVRIIXWXUHKHDOWKFDUHDQGKRZEHVWWRIDFLOLWDWHLQWHUDFWLRQV *HQH&+(81*1DWLRQDO EHWZHHQPXOWLPHGLDUHVHDUFKHUVDQGKHDOWKFDUHDQGDVVLVWHGOLYL QJSURYLGHUV ,QVWLWXWHIRU,QIRUPDWLFV-DSDQ 7DVRV'$*,8./$6/RQGRQ 3DSHUV DUH VROLFLWHGLQ EXWQRWOLPLWHGWR WKHIROORZLQJDUHDVFRYHULQJQRWRQO\WKLV \HDU¶V 6RXWK%DQN8QLYHUVLW\8. ZRUNVKRSWKHPHEXWDOVRWKHJHQHUDOVFRSHRIPXOWLPHGLDVLJQDO SURFHVVLQJ .H\QRWH&RFKDLUV x 0XOWLPHGLDELJGDWDDQDO\WLFV 1LNRODRV%28/*285,6 x 'LVWULEXWHGPXOWLPHGLDIRUERG\QHWZRUNV %UXQHO8QLYHUVLW\/RQGRQ8. x 'HHSOHDUQLQJIRUKHDOWKVSHFLILFHYHQWGHWHFWLRQDQGFODVVLILFDWLRQ

3URSRVDOVIRU6SHFLDO6HVVLRQVDQG7XWRULDOV$SULO 1RUWK 6RXWK$PHULFD/LDLVRQ 1RWLILFDWLRQRI$FFHSWDQFHIRU6SHFLDO6HVVLRQDQG7XWRULDO3URSRVDOV$SULO -DFRE&+$.$5(6.,8QLYHUVLW\ 6XEPLVVLRQRI5HJXODUDQG6SHFLDO6HVVLRQ3DSHUV-XQH RI$ODEDPD$/86$

1RWLILFDWLRQRI$FFHSWDQFHIRU5HJXODUDQG6SHFLDO6HVVLRQ3DSHUV-XO\ /RFDO$UUDQJHPHQWV&KDLU 6XEPLVVLRQRI6NHWFKDQG'HPR3DSHUV$XJXVW 1LFKRODV*$5'1(5 &DPHUD5HDG\'HDGOLQH$XJXVW 8QLYHUVLW\RI%HGIRUGVKLUH8.

www.signalprocessingsociety.org [5] JANUARY 2017

THE WORLD’S NEWSSTAND®

IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS Now accepting paper submissions The new /dƌĂŶƐĂĐƚŝŽŶƐŽŶ^ŝŐŶĂůĂŶĚ/ŶĨŽƌŵĂƚŝŽŶWƌŽĐĞƐƐŝŶŐŽǀĞƌEĞƚǁŽƌŬƐ publishes high-quality papers that extend the classical notions of processing of signals defined over vector spaces (e.g. time and space) to processing of signals and information (data) defined over networks, potentially dynamically varying. In signal processing over networks, the topology of the network may define structural relationships in the data, or may constrain processing of the data. Topics of interest include, but are not limited to the following:

Adaptation, Detection, Estimation, and Learning Modeling and Analysis (cont.) R Distributed detection and estimation R Simulations of networked information processing R Distributed adaptation over networks systems R Distributed learning over networks R Social learning R Distributed target tracking R Bio-inspired network signal processing R Bayesian learning; Bayesian signal processing R Epidemics and diffusion in populations R Sequential learning over networks Imaging and Media Applications R Decision making over networks R Image and video processing over networks R Distributed dictionary learning R Media cloud computing and communication R Distributed game theoretic strategies R Multimedia streaming and transport R Distributed information processing R Social media computing and networking R Graphical and kernel methods R Signal processing for cyber-physical systems R Consensus over network systems R Wireless/mobile multimedia Optimization over network systems R Data Analysis Communications, Networking, and Sensing R Processing, analysis, and visualization of big data R Distributed monitoring and sensing R Signal and information processing for crowd R Signal processing for distributed communications and computing networking R Signal and information processing for the Internet of R Signal processing for cooperative networking Things R Signal processing for network security R Emergence of behavior Optimal network signal processing and resource R Emerging topics and applications allocation R Emerging topics Modeling and Analysis R Applications in life sciences, ecology, energy, social R Performance and bounds of methods networks, economic networks, finance, social R Robustness and vulnerability sciences, smart grids, wireless health, robotics, R Network modeling and identification transportation, and other areas of science and engineering

Editor-in-ŚŝĞĨ͗WĞƚĂƌD͘ũƵƌŝđ͕^ƚŽŶǇƌŽŽŬhŶŝǀĞƌƐŝƚǇ;h^Ϳ dŽƐƵďŵŝƚĂƉĂƉĞƌ͕ŐŽƚŽ͗Śƚƚ______ƉƐ͗ͬͬŵĐ͘ŵĂŶƵƐĐƌŝƉƚĐĞŶƚƌĂů͘ĐŽŵͬƚƐŝƉŶ-ieee

______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJ

www.signalprocessingsociety.org [6] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

DECEMBER 2016 VOLUME 24 NUMBER 12 ITASFA (ISSN 2329-9290)

REGULAR PAPERS

Context-Dependent Piano Music Transcription With Convolutional Sparse Coding http://dx.doi.org/10.1109/TASLP.2016.2598305 ...... A.Cogliati, Z. Duan, and B. Wohlberg 2218 Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition http://dx.doi.org/10.1109/TASLP.2016.2598308 ...... Y. Qian, T. Tan, and D. Yu 2231 Factorized Hidden Layer Adaptation for Deep Neural Network Based Acoustic Modeling http://dx.doi.org/10.1109/TASLP.2016.2601146 ...... L. Samarakoon and K. C. Sim 2241 On MMSE-Based Estimation of Amplitude and Complex Speech Spectral Coefﬁcients Under Phase-Uncertainty http://dx.doi.org/10.1109/TASLP.2016.2602549 ...... M.Krawczyk-Becker and T. Gerkmann 2251 Very Deep Convolutional Neural Networks for Noise Robust Speech Recognition http://dx.doi.org/10.1109/TASLP.2016.2602884 ...... Y. Qian, M. Bi, T. Tan, and K. Yu 2263 Generation of Affective Accompaniment in Accordance With Emotion Flow http://dx.doi.org/10.1109/TASLP.2016.2603006 ...... Y.-C. Wu and H. H. Chen 2277

www.signalprocessingsociety.org [7] JANUARY 2017

THE WORLD’S NEWSSTAND®

Scalable Audio Coding Using Trellis-Based Optimized Joint Entropy Coding and Quantization http://dx.doi.org/10.1109/TASLP.2016.2607339 ...... M.Movassagh and P. Kabal 2288 Composition of Deep and Spiking Neural Networks for Very Low Bit Rate Speech Coding http://dx.doi.org/10.1109/TASLP.2016.2604566 ...... M. Cernak, A. Lazaridis, A. Asaei, and P. N. Garner 2301 Kernel Method for Voice Activity Detection in the Presence of Transients http://dx.doi.org/10.1109/TASLP.2016.2566919 ...... D.Dov,R.Talmon, and I. Cohen 2313 Bayesian Networks to Model the Variability of Speaker Verification Scores in Adverse Environments http://dx.doi.org/10.1109/TASLP.2016.2607343 ...... J.Villalba, A. Miguel, A. Ortega, and E. Lleida 2327 Novel Unsupervised Auditory Filterbank Learning Using Convolutional RBM for Speech Recognition http://dx.doi.org/10.1109/TASLP.2016.2607341 ...... H.B. Sailor and H. A. Patil 2341 Instantaneous Fundamental Frequency Estimation With Optimal Segmentation for Nonstationary Voiced Speech http://dx.doi.org/10.1109/TASLP.2016.2608948 ...... S.M. Nørholm, J. R. Jensen, and M. G. Christensen 2354 Robust Variable Step-Size Decorrelation Normalized Least-Mean-Square Algorithm and its Application to Acoustic Echo Cancellation http://dx.doi.org/10.1109/TASLP.2016.2556280 ...... S. Zhang, J. Zhang, and H. Han 2368 Blind Separation of Audio Mixtures Through Nonnegative Tensor Factorization of Modulation Spectrograms http://dx.doi.org/10.1109/TASLP.2016.2602546 ...... T. Barker and T. Virtanen 2377 Adaptive Compensation of Misequalization in Narrowband Active Noise Equalizer Systems http://dx.doi.org/10.1109/TASLP.2016.2604212 ...... J.LiuandX. Chen 2390 Estimating Speech Recognition Accuracy Based on Error Type Classification http://dx.doi.org/10.1109/TASLP.2016.2603599 ...... A. Ogawa, T. Hori, and A. Nakamura 2400 Score-Aging Calibration for Speaker Verification http://dx.doi.org/10.1109/TASLP.2016.2602542 ...... F.Kelly and J. H. L. Hansen 2414 An Approach to Score Following for Piano Performances With the Sustained Effect http://dx.doi.org/10.1109/TASLP.2016.2611938 ...... B.LiandZ. Duan 2425 Integration of Optimized Modulation Filter Sets Into Deep Neural Networks for Automatic Speech Recognition http://dx.doi.org/10.1109/TASLP.2016.2615239 ...... N. Moritz, B. Kollmeier, and J. Anemüller 2439 Multichannel Audio Source Separation With Probabilistic Reverberation Priors http://dx.doi.org/10.1109/TASLP.2016.2614140 ...... S.Leglaive, R. Badeau, and G. Richard 2453 Single Snapshot Detection and Estimation of Reflections From Room Impulse Responses in the Spherical Harmonic Domain http://dx.doi.org/10.1109/TASLP.2016.2615238 ...... S.Tervo 2466 Extraction of Acoustic Sources Through the Processing of Sound Field Maps in the Ray Space http://dx.doi.org/10.1109/TASLP.2016.2615242 ...... D. Markovic´, F. Antonacci, L. Bianchi, S. Tubaro, and A. Sarti 2481 Long-Term SNR Estimation of Speech Signals in Known and Unknown Channel Conditions http://dx.doi.org/10.1109/TASLP.2016.2615240 ...... P.Papadopoulos, A. Tsiartas, and S. Narayanan 2495 Sensitivity of Source–Filter Interaction to Specific Vocal Tract Shapes http://dx.doi.org/10.1109/TASLP.2016.2616543 ...... I.R.Titze and A. Palaparthi 2507 A Hybrid Approach for Speech Enhancement Using MoG Model and Neural Network Phoneme Classifier http://dx.doi.org/10.1109/TASLP.2016.2618007 ...... S.E. Chazan, J. Goldberger, and S. Gannot 2516 Superdirective Beamforming Based on the Krylov Matrix http://dx.doi.org/10.1109/TASLP.2016.2618003 ...... G. Huang, J. Benesty, and J. Chen 2531

Reviewers List http://dx.doi.org/10.1109/TASLP.2016.2618499 ...... 2531

EDICS—Editor's Information Classiﬁcation Scheme http://dx.doi.org/10.1109/TASLP.2016.2625091 ...... 2549 Information for Authors http://dx.doi.org/10.1109/TASLP.2016.2625087 ...... 2551

Introducing IEEE Collabratec http://dx.doi.org/10.1109/TASLP.2016.2630135 ...... 2553

2016 INDEX http://dx.doi.org/10.1109/TASLP.2016.2628026 ...... 2554

www.signalprocessingsociety.org [8] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

DECEMBER 2016 VOLUME 25 NUMBER 12 IIPRE4 (ISSN 1057-7149)

PAPERS

Dynamic Parallel and Distributed Graph Cuts http://dx.doi.org/10.1109/TIP.2016.2609819 ...... M.Yu,S. Shen, and Z. Hu 5511 Multi-View 3D Object Retrieval With Deep Embedding Network http://dx.doi.org/10.1109/TIP.2016.2609814 ...... H. Guo, J. Wang, Y. Gao, J. Li, and H. Lu 5526 Pedestrian Detection Inspired by Appearance Constancy and Shape Symmetry http://dx.doi.org/10.1109/TIP.2016.2609807 ...... J. Cao, Y. Pang, and X. Li 5538 Unsupervised Domain Adaptation With Label and Structural Consistency http://dx.doi.org/10.1109/TIP.2016.2609820 ...... C.-A. Hou, Y.-H. H. Tsai, Y.-R. Yeh, and Y.-C. F. Wang 5552 Incorporating Spatial Information and Endmember Variability Into Unmixing Analyses to Improve Abundance Estimates http://dx.doi.org/10.1109/TIP.2016.2601269 ...... T. Uezato, R. J. Murphy, A. Melkumyan, and A. Chlingaryan 5563 Deep Transfer Metric Learning http://dx.doi.org/10.1109/TIP.2016.2612827 ...... J.Hu,J.Lu,Y.-P.Tan,andJ. Zhou 5576 A Faster, Unbiased Path Opening by Upper Skeletonization and Weighted Adjacency Graphs http://dx.doi.org/10.1109/TIP.2016.2609805 ...... T. Asplund and C. L. Luengo Hendriks 5589 Hash-Based Line-by-Line Template Matching for Lossless Screen Image Coding http://dx.doi.org/10.1109/TIP.2016.2612884 ...... X.PengandJ.Xu 5601 A Fast Optimization Method for General Binary Code Learning http://dx.doi.org/10.1109/TIP.2016.2612883 ...... F. Shen, X. Zhou, Y. Yang, J. Song, H. T. Shen, and D. Tao 5610 Contour Restoration of Text Components for Recognition in Video/Scene Images http://dx.doi.org/10.1109/TIP.2016.2607426 ...... Y.Wu,P. Shivakumara, T. Lu, C. L. Tan, M. Blumenstein, and G. H. Kumar 5622 Structure Integral Transform Versus Radon Transform: A 2D Mathematical Tool for Invariant Shape Recognition http://dx.doi.org/10.1109/TIP.2016.2609816 ...... B.WangandY.Gao 5635 Hyperspectral Image Recovery via Hybrid Regularization http://dx.doi.org/10.1109/TIP.2016.2614131 ...... R.Arablouei and F. de Hoog 5649 Underwater Image Enhancement by Dehazing With Minimum Information Loss and Histogram Distribution Prior http://dx.doi.org/10.1109/TIP.2016.2612882 ...... C.-Y. Li, J.-C. Guo, R.-M. Cong, Y.-W. Pang, and B. Wang 5664 Beyond Object Proposals: Random Crop Pooling for Multi-Label Image Recognition http://dx.doi.org/10.1109/TIP.2016.2612829 ...... M.Wang,C. Luo, R. Hong, J. Tang, and J. Feng 5678 Web Video Event Recognition by Semantic Analysis From Ubiquitous Documents http://dx.doi.org/10.1109/TIP.2016.2614136 ...... L.Yu,Y.Yang,Z. Huang, P. Wang, J. Song, and H. T. Shen 5689

www.signalprocessingsociety.org [9] JANUARY 2017

THE WORLD’S NEWSSTAND®

Blind Bleed-Through Removal for Scanned Historical Document Image With Conditional Random Fields http://dx.doi.org/10.1109/TIP.2016.2614133 ...... B. Sun, S. Li, X.-P. Zhang, and J. Sun 5702 Robust Nanoparticles Detection From Noisy Background by Fusing Complementary Image Information http://dx.doi.org/10.1109/TIP.2016.2614127 ...... Y. Qian, J. Z. Huang, X. Li, and Y. Ding 5713 Joint Facial Action Unit Detection and Feature Fusion: A Multi-Conditional Learning Approach http://dx.doi.org/10.1109/TIP.2016.2615288 ...... S. Eleftheriadis, O. Rudovic, and M. Pantic 5727 Learning Collaborative Sparse Representation for Grayscale-Thermal Tracking http://dx.doi.org/10.1109/TIP.2016.2614135 ...... C.Li,H. Cheng, S. Hu, X. Liu, J. Tang, and L. Lin 5743 Tree-Structured Nuclear Norm Approximation With Applications to Robust Face Recognition http://dx.doi.org/10.1109/TIP.2016.2612885 ...... L. Luo, L. Chen, J. Yang, J. Qian, and B. Zhang 5757 Low-Rankness Transfer for Realistic Denoising http://dx.doi.org/10.1109/TIP.2016.2612820 ...... H. Badri, H. Yahia, and D. Aboutajdine 5768 A Coupled Hidden Conditional Random Field Model for Simultaneous Face Clustering and Naming in Videos http://dx.doi.org/10.1109/TIP.2016.2601491 ...... Y. Zhang, Z. Tang, B. Wu, Q. Ji, and H. Lu 5780 Image Denoising via Bandwise Adaptive Modeling and Regularization Exploiting Nonlocal Similarity http://dx.doi.org/10.1109/TIP.2016.2614160 ...... R. Xiong, H. Liu, X. Zhang, J. Zhang, S. Ma, F. Wu, and W. Gao 5793 Codewords Distribution-Based Optimal Combination of Equal-Average Equal-Variance Equal-Norm Nearest Neighbor Fast Search Algorithm for Vector Quantization Encoding http://dx.doi.org/10.1109/TIP.2016.2615292 ...... Y.-F.Xie,J.-H. Liu, C.-F. Zhang, L.-S. Kong, and J.-L. Yi 5806 Multi-View Object Retrieval via Multi-Scale Topic Models http://dx.doi.org/10.1109/TIP.2016.2614132 ...... R. Hong, Z. Hu, R. Wang, M. Wang, and D. Tao 5814 Multimodal Stereoscopic Movie Summarization Conforming to Narrative Characteristics http://dx.doi.org/10.1109/TIP.2016.2615289 ...... I. Mademlis, A. Tefas, N. Nikolaidis, and I. Pitas 5828 Subpixel Image Quality Assessment Syncretizing Local Subpixel and Global Pixel Features http://dx.doi.org/10.1109/TIP.2016.2615429 ...... J. Zeng, L. Fang, J. Pang, H. Li, and F. Wu 5841 Dynamic Programming Using Polar Variance for Image Segmentation http://dx.doi.org/10.1109/TIP.2016.2615809 ...... J.A. Rosado-Toro, M. I. Altbach, and J. J. Rodríguez 5857 Visual Tracking Under Motion Blur http://dx.doi.org/10.1109/TIP.2016.2615812 ...... B.Ma,L. Huang, J. Shen, L. Shao, M.-H. Yang, and F. Porikli 5867 High-Efﬁciency 3D Depth Coding Based on Perceptual Quality of Synthesized Video http://dx.doi.org/10.1109/TIP.2016.2615290 ...... Y. Zhang, X. Yang, X. Liu, Y. Zhang, G. Jiang, and S. Kwong 5877 Deep Learning Driven Visual Path Prediction From a Single Image http://dx.doi.org/10.1109/TIP.2016.2613686 ...... S. Huang, X. Li, Z. Zhang, Z. He, F. Wu, W. Liu, J. Tang, and Y. Zhuang 5892 Spatial Pyramid Covariance-Based Compact Video Code for Robust Face Retrieval in TV-Series http://dx.doi.org/10.1109/TIP.2016.2616297 ...... Y.Li,R.Wang,Z. Cui, S. Shan, and X. Chen 5905 Learning Expressionlets via Universal Manifold Model for Dynamic Facial Expression Recognition http://dx.doi.org/10.1109/TIP.2016.2615424 ...... M. Liu, S. Shan, R. Wang, and X. Chen 5920 Real-Time Superpixel Segmentation by DBSCAN Clustering Algorithm http://dx.doi.org/10.1109/TIP.2016.2616302 ...... J. Shen, X. Hao, Z. Liang, Y. Liu, W. Wang, and L. Shao 5933 Defocus Map Estimation From a Single Image Based on Two-Parameter Defocus Model http://dx.doi.org/10.1109/TIP.2016.2617460 ...... S. Liu, F. Zhou, and Q. Liao 5943 Efﬁcient Non-Consecutive Feature Tracking for Robust Structure-From-Motion http://dx.doi.org/10.1109/TIP.2016.2607425 ...... G. Zhang, H. Liu, Z. Dong, J. Jia, T.-T. Wong, and H. Bao 5957 Adaptive Subspace-Based Inverse Projections via Division Into Multiple Sub-Problems for Missing Image Data Restoration http://dx.doi.org/10.1109/TIP.2016.2616286 ...... T. Ogawa and M. Haseyama 5971 A Spatial Compositional Model for Linear Unmixing and Endmember Uncertainty Estimation http://dx.doi.org/10.1109/TIP.2016.2618002 ...... Y. Zhou, A. Rangarajan, and P. D. Gader 5987

EDICS-Editor’s Information Classiﬁcation Scheme http://dx.doi.org/10.1109/TIP.2016.2625027 ...... 6003 Information for Authors http://dx.doi.org/10.1109/TIP.2016.2625026 ...... 6004

List of Reviewers http://dx.doi.org/10.1109/TIP.2016.2617578 ...... Available online at http://ieeexplore.ieee.org 2016 Index ...... Available online at http://ieeexplore.ieee.org

www.signalprocessingsociety.org [10] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

www.signalprocessingsociety.org [11] JANUARY 2017

THE WORLD’S NEWSSTAND®

IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING 1(:

(GLWRULQ&KLHI The IEEE Transactions on Computational Imaging W. Clem Karl publishes research results where computation plays Boston University an integral role in the image formation process. All areas 7HFKQLFDO&RPPLWWHH of computational imaging are appropriate, ranging from &KDUOHV%RXPDQ the principles and theory of computational imaging, to mod- (ULF0LOOHU 3HWHU&RUFRUDQ eling paradigms for computational imaging, to image for- -RQJ&KXO

Computational Imaging Methods and Computational Consumer Tomographic Imaging Models Imaging x X-ray CT x Coded image sensing x Mobile imaging, cell phone imaging x PET x Compressed sensing x Camera-array systems x SPECT x Sparse and low-rank models x Depth cameras, multi-focus imaging Magnetic Resonance Imaging x Learning-based models, dictionary methods x Pervasive imaging, camera networks x Diffusion tensor imaging x Graphical image models Computational Acoustic Imaging x Perceptual models x Fast acquisition x Multi-static ultrasound imaging Computational Image Formation Radar Imaging x Photo-acoustic imaging x Sparsity-based reconstruction x Acoustic tomography x Synthetic aperture imaging x Inverse synthetic aperture imaging x Statistically-based inversion methods Computational Microscopy x Multi-image and sensor fusion Geophysical Imaging Holographic microscopy x Optimization-based methods; proximal itera- x tive methods, ADMM x Quantitative phase imaging x Multi-spectral imaging Multi-illumination microscopy x Ground penetrating radar Computational Photography x x Lensless microscopy x Seismic tomography x Non-classical image capture x Light field microscopy Multi-spectral Imaging x Generalized illumination Imaging Hardware and Software x Time-of-flight imaging x Multi-spectral imaging x High dynamic range imaging x Embedded computing systems x Hyper-spectral imaging x Plenoptic imaging x Big data computational imaging x Spectroscopic imaging x Integrated hardware/digital design

For more information on the IEEE Transactions on Computational Imaging see

______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVWFL

www.signalprocessingsociety.org [12] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

www.signalprocessingsociety.org [13] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

DECEMBER 2016 VOLUME 11 NUMBER 12 ITIFA6 (ISSN 1556-6013)

REGULAR PAPERS

A High-Security EEG-Based Login System with RSVP Stimuli and Dry Electrodes http://dx.doi.org/10.1109/TIFS.2016.2577551 ...... Y. Chen, A. D. Atnafu, I. Schlattner, W. T. Weldtsadik, M.-C. Roh, H. J. Kim, S.-W. Lee, B. Blankertz, and S. Fazli 2635 Reconstruction Attacks Against Mobile-Based Continuous Authentication Systems in the Cloud http://dx.doi.org/10.1109/TIFS.2016.2594132 ...... M. Al-Rubaie and J. M. Chang 2648 Quaternion-Based Image Hashing for Adaptive Tampering Localization http://dx.doi.org/10.1109/TIFS.2016.2594136 ...... C.-P. Yan, C.-M. Pun, and X.-C. Yuan 2664 A Data-Centric Approach to Quality Estimation of Role Mining Results http://dx.doi.org/10.1109/TIFS.2016.2594137 ...... L. Dong, K. Wu, and G. Tang 2678 Exploiting Multi-Antenna Non-Reciprocal Channels for Shared Secret Key Generation http://dx.doi.org/10.1109/TIFS.2016.2594143 ...... D.QinandZ. Ding 2693 Toward Efﬁcient Multi-Keyword Fuzzy Search Over Encrypted Outsourced Data With Accuracy Improvement http://dx.doi.org/10.1109/TIFS.2016.2596138 ...... Z.Fu,X.Wu,C. Guan, X. Sun, and K. Ren 2706 Detecting Byzantine Attacks Without Clean Reference http://dx.doi.org/10.1109/TIFS.2016.2596140 ...... R. Cao, T. F. Wong, T. Lv, H. Gao, and S. Yang 2717 Full Frame Encryption and Modulation Obfuscation Using Channel-Independent Preamble Identiﬁer http://dx.doi.org/10.1109/TIFS.2016.2582560 ...... H. Rahbari and M. Krunz 2732 SARRE: Semantics-Aware Rule Recommendation and Enforcement for Event Paths on Android http://dx.doi.org/10.1109/TIFS.2016.2596141 ...... Y.Li,F.Yao,T. Lan, and G. Venkataramani 2748 Deceptive Deletion Triggers Under Coercion http://dx.doi.org/10.1109/TIFS.2016.2598523 ...... L. Zhao and M. Mannan 2763 New Framework for Reversible Data Hiding in Encrypted Domain http://dx.doi.org/10.1109/TIFS.2016.2598528 ...... F. Huang, J. Huang, and Y.-Q. Shi 2777 Opting Out of Incentive Mechanisms: A Study of Security as a Non-Excludable Public Good http://dx.doi.org/10.1109/TIFS.2016.2599005 ...... P.Naghizadeh and M. Liu 2790 Analytic and Simulation Results about a Compact, Reliable, and Unbiased 1-bit Physically Unclonable Constant http://dx.doi.org/10.1109/TIFS.2016.2599008 ...... R. Bernardini and R. Rinaldo 2804

www.signalprocessingsociety.org [14] JANUARY 2017

THE WORLD’S NEWSSTAND®

Continuous Authentication Using One-Dimensional Multi-Resolution Local Binary Patterns (1DMRLBP) in ECG Biometrics http://dx.doi.org/10.1109/TIFS.2016.2599270 ...... W. Louis, M. Komeili, and D. Hatzinakos 2818 Server-Aided Public Key Encryption With Keyword Search http://dx.doi.org/10.1109/TIFS.2016.2599293 ...... R. Chen, Y. Mu, G. Yang, F. Guo, X. Huang, X. Wang, and Y. Wang 2833 Reconstructing High-Resolution Face Models From Kinect Depth Sequences http://dx.doi.org/10.1109/TIFS.2016.2601059 ...... E. Bondi, P. Pala, S. Berretti, and A. Del Bimbo 2843 On the Race of Worms and Patches: Modeling the Spread of Information in Wireless Sensor Networks http://dx.doi.org/10.1109/TIFS.2016.2594130 ...... M. Sayad Haghighi, S. Wen, Y. Xiang, B. Quinn, and W. Zhou 2854 Still-to-Video Face Matching Using Multiple Geodesic Flows http://dx.doi.org/10.1109/TIFS.2016.2601060 ...... Y. Zhu, Y. Li, G. Mu, S. Shan, and G. Guo 2866

List of Reviewers http://dx.doi.org/10.1109/TIFS.2016.2617259 ...... 2876

EDICS-Editor’s Information Classiﬁcation Scheme http://dx.doi.org/10.1109/TIFS.2016.2625404 ...... 2887 Information for Authors http://dx.doi.org/10.1109/TIFS.2016.2625405 ...... 2888

ANNOUNCEMENTS Call for Papers-IEEE Journal of Selected Topics in Signal Processing Special Issue on Light Field Image Processing http://dx.doi.org/10.1109/TIFS.2016.2626022 ...... 2890 Call for Papers-IEEE Transactions on Audio, Speech, and Language Processing Special Issue on Biosignal-Based Spoken Communication http://dx.doi.org/10.1109/TIFS.2016.2626002 ...... 2891 Call for Papers-IEEE Transactions on Signal and Information Processing Over Networks Special Issue on Distributed Signal Processing for Security and Privacy in Networked Cyber-Physical Systems http://dx.doi.org/10.1109/TIFS.2016.2626003 ...... 2892

2016 Index http://dx.doi.org/10.1109/TIFS.2016.2637459 ...... Available online at http://ieeexplore.ieee.org

www.signalprocessingsociety.org [15] JANUARY 2017

THE WORLD’S NEWSSTAND®

3rd IEEE International Conference on Network Softwarization SOFTWARIZATION SUSTAINING A HYPER-CONNECTED WORLD: EN ROUTE TO 5G

3-7 JULY 2017, BOLOGNA, ITALY http://sites.ieee.org/netsoft—CALL FOR PAPERS

rd The 3 IEEE International Conference on Network Softwarization (NetSoft 2017) will be held July –at the University of Bologna in Bologna, Italy. IEEE NetSoft is the flagship conference of the IEEE SDN Initiative which aims to address ³Softwarization´of networks and systemic trends concerning the convergence of Cloud Computing, Software-Defined Networks, and Network Function Virtualization.

TOPICS OF INTEREST Authors are invited to submit papers that fall into the area of software-defined and virtualized infrastructures. Topics of interest include, but are not limited to, the following: x SDN and NFV as enabling technologies for 5G x Real time operations and efficient network/service moni- x From Cloud Computing to Edge-Fog Computing toring in SDN/NFV x 5G Functional Decomposition and Infrastructure slicing x Performance and scalability issues in NFV implementation scenarios x 5G sustainable ecosystems: IoT, Industry 4.0, Pervasive Robotics, Self-driving vehicles, Tactile Internet, Immersi- x Traffic Engineering and QoS/QoE in SDN/NFV ve Communications, Artificial Intelligence applications x APIs, protocols and languages for programmable net- x Software Defined infrastructures for Public Protection works and Software-Defined Infrastructure and Disaster Relief (PPDR) network services x SDN switch/router architectures/designs x Service Function Chaining for NFV: Modeling, composi- x SDN/NFV issues and opportunities for security, trust tion algorithms, deployment and privacy x Intent-based interfacing for NFV x Experience reports from experimental testbeds and de- x SDN/NFV Network & Service Orchestration and Mana- ployment gement x Softwarized platforms for Internet-of-Things (IoT) x Management of federated SDN/NFV infrastructure and x New value chains and business models frameworks SCOPE The telecommunications landscape will change radically in the next few years. Pervasive ultra-broadband, programmable networks, and cost reduction of IT systems are paving the way to new services and commoditization of telecommunications infrastructure while lowering entrance barriers for new players and giving rise to new value chains. While this results in considerable challenges for service providers, this transformation also brings unprecedented opportunities for the Digital Society and the Digital Economy related to emerging new services and applications. Examples include Tactile Internet of Things, Industry 4.0, Cloud Robotics, and Artificial Intelligence. 5G will both exploit and accelerate this transformation.

NetSoft 2017 aims to capture the theme of ³Softwarization Sustaining a Hyper-connected World: en route to 5G´and serve as forum for researchers to discuss the latest advances in this area. NetSoft 2017 will feature technical paper, keynotes, tutorials, and demos and exhibits from world-leading experts representing operators, vendors, research institutes, open source projects, and academia.

PAPER SUBMISSION Authors are invited to submit original contributions (written in English) in PDF format. Only original papers not published or submitted for publication elsewhere can be submitted. Papers can be of two types: full (up to 9 pages) or short (up to 5 pages) papers. Full Papers accepted as short Papers will be required to be reduced to 5-pages length. Papers should be in IEEE 2-column US-Letter style using IEEE Conference template (KWWSZZZLHHHRUJ FRQIHUHQFHVBHYHQWVFRQIHUHQFHVSXEOLVKLQJWHPSODWHVKWPO) and submitted in PDF format via JEMS at: KWWSV MHPVVEFRUJEUKRPHFJL"F . Papers exceeding these limits, multiple submissions, and self-plagiarized papers will be rejected without further review. All submitted papers will be subject to a peer-review process. The accepted papers will be published in IEEE Xplore, provided that the authors do present their paper at the conference. IMPORTANT DATES GENERAL CO-CHAIRS December 5, 2016: Technical Papers deadline Antonio Manzalini (Telecom Italia Mobile, Italy) Workshop Submission deadline December 15, 2016: Roberto Verdone (University of Bologna, Italy) March 6, 2017: Paper submission acceptance notification Full Conference Pa April 10, 2017: SHU6SHDNHUUHJLVWUDWLRQ STEERING COMMITTEE CHAIR May 30, 2017: Early-Bird Registration Prosper Chemouil (Orange Labs, France)

TPC CO-CHAIRS &ƌĂŶĐŽĂůůĞŐĂƟ;hŶŝǀĞƌƐŝƚǇŽĨŽůŽŐŶĂ͕/ƚĂůǇͿ Alexander Clemm (Cisco, USA) Kohei Shiomoto (NTT Labs, Japan)

www.signalprocessingsociety.org [16] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

DECEMBER 2016 VOLUME 18 NUMBER 12 ITMUF8 (ISSN 1520-9210)

PAPERS Compression and Coding

Quadtree Degeneration for HEVC http://dx.doi.org/10.1109/TMM.2016.2598481 ...... Y. Gao, P. Liu, Y. Wu, and K. Jia 2321 Clustering-Based Content Adaptive Tiles Under On-chip Memory Constraints http://dx.doi.org/10.1109/TMM.2016.2600439 ...... X.JinandQ.Dai 2331 Watermarking, Encryption, and Data Hiding

Data Hiding Robust to Mobile Communication Vocoders http://dx.doi.org/10.1109/TMM.2016.2599149 ...... R. Kazemi, F. Pérez-González, M. A. Akhaee, and F. Behnia 2345 Image/Video/Graphics Analysis and Synthesis

Interactive Multilabel Image Segmentation via Robust Multilayer Graph Constraints http://dx.doi.org/10.1109/TMM.2016.2600441 ...... T.Wang,Z.Ji,Q. Sun, Q. Chen, and X.-Y. Jing 2358 Task-Driven Progressive Part Localization for Fine-Grained Object Recognition http://dx.doi.org/10.1109/TMM.2016.2602060 ...... C. Huang, Z. He, G. Cao, and W. Cao 2372 System Design Methodology and Tools A Constellation Design Methodology Based on QoS and User Demand in High-Altitude Platform Broadband Networks http://dx.doi.org/10.1109/TMM.2016.2595260 ...... F. Dong, H. Han, X. Gong, J. Wang, and H. Li 2384 Video Surveillance and Semantic Analysis

Exemplar-AMMs: Recognizing Crowd Movements From Pedestrian Trajectories http://dx.doi.org/10.1109/TMM.2016.2598091 ...... W. Liu, R. W. H. Lau, X. Wang, and D. Manocha 2398 Multimedia Using Haptic and Physiological Information

Deep Learning for Surface Material Classiﬁcation Using Haptic and Visual Information http://dx.doi.org/10.1109/TMM.2016.2598140 ...... H. Zheng, L. Fang, M. Ji, M. Strese, Y. Özer, and E. Steinbach 2407 Multimodal Perception, Integration, and Multisensory Fusion Mean-Shift and Sparse Sampling-Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking http://dx.doi.org/10.1109/TMM.2016.2599150 ...... V. Kiliç, M. Barnard, W. Wang, A. Hilton, and J. Kittler 2417

www.signalprocessingsociety.org [17] JANUARY 2017

THE WORLD’S NEWSSTAND®

Media Translation for Sense Substitution

Perceiving Graphical and Pictorial Information via Hearing and Touch http://dx.doi.org/10.1109/TMM.2016.2601029 ...... P.M. Silva, T. N. Pappas, J. Atkins, and J. E. West 2432 Subjective and Objective Quality Assessment, and User Experience Perceptual Annoyance Models for Videos With Combinations of Spatial and Temporal Artifacts http://dx.doi.org/10.1109/TMM.2016.2601027 ...... A.F. Silva, M. C. Q. Farias, and J. A. Redi 2446 Blind Image Quality Assessment Using Statistical Structural and Luminance Features http://dx.doi.org/10.1109/TMM.2016.2601028 ...... Q.Li,W. Lin, J. Xu, and Y. Fang 2457 Ubiquitous Media Access

Optimal Incentive Design for Cloud-Enabled Multimedia Crowdsourcing http://dx.doi.org/10.1109/TMM.2016.2604080 ...... S. Maharjan, Y. Zhang, and S. Gjessing 2470 Multimedia Search and Retrieval

Robust Latent Poisson Deconvolution From Multiple Features for Web Topic Detection http://dx.doi.org/10.1109/TMM.2016.2598439 ...... J.Pang,F.Tao,C. Zhang, W. Zhang, Q. Huang, and B. Yin 2482 Web and Internet

Image Classiﬁcation by Cross-Media Active Learning With Privileged Information http://dx.doi.org/10.1109/TMM.2016.2602938 ...... Y.Yan,F.Nie,W.Li,C. Gao, Y. Yang, and D. Xu 2494 Multimedia Streaming and Transport

Trend-Aware Video Caching Through Online Learning http://dx.doi.org/10.1109/TMM.2016.2596042 ...... S.Li,J.Xu,M.vanderSchaar, and W. Li 2503 Resource-Efficient Mobile Multimedia Streaming With Adaptive Network Selection http://dx.doi.org/10.1109/TMM.2016.2604565 ...... J.Lee,K.Lee,C. Han, T. Kim, and S. Chong 2517 Multimedia Sentiment Analysis and Synthesis; Affective Multimedia Processing A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition http://dx.doi.org/10.1109/TMM.2016.2598092 ...... T. Zhang, W. Zheng, Z. Cui, Y. Zong, J. Yan, and K. Yan 2528 Multimedia Networking and Processing in the Cloud and Data Centers Media Query Processing for the Internet-of-Things: Coupling of Device Energy Consumption and Cloud Infrastructure Billing http://dx.doi.org/10.1109/TMM.2016.2600438 ...... F. Renna, J. Doyle, V. Giotsas, and Y. Andreopoulos 2537 Ultra-Efficient Surveillance Video and Coding Of Multimedia Features Person Reidentification via Ranking Aggregation of Similarity Pulling and Dissimilarity Pushing http://dx.doi.org/10.1109/TMM.2016.2605058 ...... M.Ye,C. Liang, Y. Yu, Z. Wang, Q. Leng, C. Xiao, J. Chen, and R. Hu 2553

ANNOUNCEMENTS

Introducing IEEE Collabratec http://dx.doi.org/10.1109/TMM.20162.2625923 ...... 2567

CALLS FOR PAPERS

IEEE International Conference on Multimedia and Expo Call for Papers http://dx.doi.org/10.1109/TMM.2016.2625925 ...... 2568 IEEE TRANSACTIONS ON MULTIMEDIA Call for Papers for the Special Issue on Video Over Future Networks: Emerging Technologies, Infrastructures, and Applications http://dx.doi.org/10.1109/TMM.2016.2625899 ...... 2569

Information for Authors http://dx.doi.org/10.1109/TMM.2016.2625921 ...... 2570

2016 INDEX ...... Available online at http://ieeexplore.ieee.org

www.signalprocessingsociety.org [18] JANUARY 2017

THE WORLD’S NEWSSTAND®

2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA 2017) October 15–18, 2017 www.waspaa.com

Mohonk Mountain House New Paltz, New York, USA Workshop Committee General Chairs Patrick A. Naylor Imperial College London Meinard Müller International AudioLabs Erlangen Technical Program Chairs Gautham Mysore Adobe Research The 2017 IEEE Workshop on Applications of Signal Processing to Audio and Mads Christensen Acoustics (WASPAA 2017) will be held at the Mohonk Mountain House in New Aalborg University Paltz, New York, and is supported by the Audio and Acoustic Signal Processing Finance Chair technical committee of the IEEE Signal Processing Society. The objective of this Michael S. Brandstein workshop is to provide an informal environment for the discussion of problems M.I.T. Lincoln Laboratory in audio, acoustics and signal processing techniques leading to novel solutions. Publications Chair Technical sessions will be scheduled throughout the day. Afternoons will be left free Toon van Waterschoot for informal meetings among workshop participants. Papers describing original KU Leuven research and new concepts are solicited for technical sessions on, but not limited to, the following topics: Registration Chair Tiago H. Falk Acoustic Signal Processing INRS, Montréal A Source separation: single- and multi-microphone techniques Industrial Liaison Chair A Acoustic source localization and tracking Tao Zhang A Signal enhancement: dereverberation, noise reduction, echo reduction Starkey Hearing Technologies A Microphone and loudspeaker array processing A Acoustic sensor networks: distributed algorithms, synchronization Far East Liaison A Acoustic scene analysis: event detection and classification Shoko Araki NTT A Room acoustics: analysis, modeling and simulation Local Arrangements Chair Audio and Music Signal Processing Youngjune Gwon A Content-based music retrieval: fingerprinting, matching, cover song retrieval M.I.T. Lincoln Laboratory A Musical signal analysis: segmentation, classification, transcription A Music signal synthesis: waveforms, instrument models, singing Demonstrations Chair A Music separation: direct-ambient decomposition, vocal and instruments Christine Evers A Audio effects: artificial reverberation, amplifier modeling Imperial College London A Upmixing and downmixing Awards Chair Audio and Speech Coding Sebastian Ewert A Waveform and parametric coding Queen Mary University of London A Spatial audio coding Important Dates A Sparse representations Submission of four-page paper A Low-delay audio and speech coding April 20, 2017 A Digital rights Notification of acceptance Hearing and Perception June 27, 2017 A Hearing aids A Computational auditory scene analysis Early registration until A Auditory perception and spatial hearing August 15, 2017 A Speech and audio quality assessment Workshop A Speech intelligibility measures and prediction October 15–18, 2017

www.signalprocessingsociety.org [19] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

DECEMBER 2016 VOLUME 10 NUMBER 8 IJSTGY (ISSN 1932-4553)

ISSUE ON SIGNAL PROCESSING FOR EXPLOITING INTERFERENCE TOWARD ENERGY EFFICIENT AND SECURE WIRELESS COMMUNICATIONS

EDITORIAL Introduction to the Issue on Signal Processing for Exploiting Interference Toward Energy Efﬁcient and Secure Wireless Communications http://dx.doi.org/10.1109/JSTSP.2016.2620703 ...... I. Krikidis, C. Masouros, G. Zheng, R. Zhang, and R. Schober 1331

PAPERS

Phase-Only Zero Forcing for Secure Communication With Multiple Antennas http://dx.doi.org/10.1109/JSTSP.2016.2611483 ...... W. Zhao, S.-H. Lee, and A. Khisti 1334 A Collaboration Incentive Exploiting the Primary-Secondary Systems’ Cross Interference for PHY Security Enhancement http://dx.doi.org/10.1109/JSTSP.2016.2600514 ...... K.Tourki and M. O. Hasna 1346 Sum Secrecy Rate Maximization for Full-Duplex Two-Way Relay Networks Using Alamouti-Based Rank-Two Beamforming http://dx.doi.org/10.1109/JSTSP.2016.2603970 ...... Q.Li,W.-K. Ma, and D. Han 1359 Secrecy and Energy Efﬁciency in Massive MIMO Aided Heterogeneous C-RAN: A New Look at Interference http://dx.doi.org/10.1109/JSTSP.2016.2600520 ...... L.Wang, K.-K. Wong, M. Elkashlan, A. Nallanathan, and S. Lambotharan 1375 Secrecy Capacity Scaling by Jamming-Aided Hierarchical Cooperation in Ad Hoc Networks http://dx.doi.org/10.1109/JSTSP.2016.2616842 ...... M.G. Kang, Y.-b. Kim, J. H. Lee, and W. Choi 1390 Regularized Channel Inversion for Simultaneous Conﬁdential Broadcasting and Power Transfer: A Large System Analysis http://dx.doi.org/10.1109/JSTSP.2016.2608792 ...... B.He,N.Yang,S.Yan,andX. Zhou 1404

www.signalprocessingsociety.org [20] JANUARY 2017

THE WORLD’S NEWSSTAND®

Secrecy Rate Optimization for Secure Multicast Communications http://dx.doi.org/10.1109/JSTSP.2016.2600518 ...... K. Cumanan, Z. Ding, M. Xu, and H. V. Poor 1417 Improving Wireless Physical Layer Security via Exploiting Co-Channel Interference http://dx.doi.org/10.1109/JSTSP.2016.2600516 ...... L.Li,A.P.Petropulu, Z. Chen, and J. Fang 1433 Wireless Information Surveillance via Proactive Eavesdropping with Spoofing Relay http://dx.doi.org/10.1109/JSTSP.2016.2600519 ...... Y. Zeng and R. Zhang 1449 Energy Efficiency of Confidential Multi-Antenna Systems With Artificial Noise and Statistical CSI http://dx.doi.org/10.1109/JSTSP.2016.2607690 ...... A. Zappone, P.-H. Lin, and E. Jorswieck 1462 Directional Modulation Via Symbol-Level Precoding: A Way to Enhance Security http://dx.doi.org/10.1109/JSTSP.2016.2600521 ...... A. Kalantari, M. Soltanalian, S. Maleki, S. Chatzinotas, and B. Ottersten 1478 Secure Multiple Amplify-and-Forward Relaying With Cochannel Interference http://dx.doi.org/10.1109/JSTSP.2016.2607692 ...... L.Fan,X. Lei, N. Yang, T. Q. Duong, and G. K. Karagiannidis 1494 Multipair Two-Way Relay Network With Harvest-Then-Transmit Users: Resolving Pairwise Uplink-Downlink Coupling http://dx.doi.org/10.1109/JSTSP.2016.2612162 ...... S.Wang,M. Xia, and Y.-C. Wu 1506 On the Performance of mmWave Networks Aided by Wirelessly Powered Relays http://dx.doi.org/10.1109/JSTSP.2016.2610944 ...... S. Biswas, S. Vuppala, and T. Ratnarajah 1522 Accumulate and Jam: Towards Secure Communication via A Wireless-Powered Full-Duplex Jammer http://dx.doi.org/10.1109/JSTSP.2016.2600523 ...... Y.BiandH. Chen 1538 Quadrature Amplitude Modulation Division for Multiuser MISO Broadcast Channels http://dx.doi.org/10.1109/JSTSP.2016.2607684 ...... Z. Dong, Y.-Y. Zhang, J.-K. Zhang, and X.-C. Gao 1551

CORRECTION Corrections to “A Characterization of Deterministic Sampling Patterns for Low-Rank Matrix Completion” http://dx.doi.org/10.1109/JSTSP.2016.2618502 ...... D.L. Pimentel-Alarcón, N. Boston, and R. D. Nowak 1567

List of Reviewers http://dx.doi.org/10.1109/JSTSP.2016.2618498 ...... 1568

Information for Authors http://dx.doi.org/10.1109/JSTSP.2016.2622188 ...... 1575

2016INDEX______HTTP://DX.DOI.ORG/10.1109/JSTSP.2016.2628027...... 1577

www.signalprocessingsociety.org [21] JANUARY 2017

THE WORLD’S NEWSSTAND®

______

DECEMBER 2016 VOLUME 23 NUMBER 12 ISPLEM (ISSN 1070-9908)

LETTERS

Improved Wideband DOA Estimation Using Modiﬁed TOPS (mTOPS) Algorithm http://dx.doi.org/10.1109/LSP.2016.2614310 ...... A.K.Shaw 1697 Structure-Aware Slow Feature Analysis for Age Estimation http://dx.doi.org/10.1109/LSP.2016.2602538 ...... Z.He,X.Li,Z. Zhang, Y. Zhang, J. Xiao, and X. Zhou 1702 CRLB for I/Q Imbalance Estimation in FMCW Radar Receivers http://dx.doi.org/10.1109/LSP.2016.2614982 ...... F. Bandiera, A. Coluccia, V. Dodde, A. Masciullo, and G. Ricci 1707 Distributed Differential Modulation Over Asymmetric Fading Channels http://dx.doi.org/10.1109/LSP.2016.2609418 ...... S. AlMaeeni, P. C. Sofotasios, S. Muhaidat, G. K. Karagiannidis, and M. Valkama 1712 Color Gaussian Jet Features For No-Reference Quality Assessment of Multiply-Distorted Images http://dx.doi.org/10.1109/LSP.2016.2617743 ...... H. Hadizadeh and I. V. Bajic´ 1717 RGBD Co-saliency Detection via Bagging-Based Clustering http://dx.doi.org/10.1109/LSP.2016.2615293 ...... H. Song, Z. Liu, Y. Xie, L. Wu, and M. Huang 1722 Constrained Perturbation Regularization Approach for Signal Estimation Using Random Matrix Theory http://dx.doi.org/10.1109/LSP.2016.2615683 ...... M. Suliman, T. Ballal, A. Kammoun, and T. Y. Al-Naffouri 1727 Antenna Placement for MIMO Localization Systems With Varying Quality of Receiver Hardware Elements http://dx.doi.org/10.1109/LSP.2016.2616648 ...... V.Aggarwal and L. Huie 1732 Visual Saliency Detection via Sparse Residual and Outlier Detection http://dx.doi.org/10.1109/LSP.2016.2617340 ...... H.Tang,C. Chen, and X. Pei 1736 Online Dereverberation for Dynamic Scenarios Using a Kalman Filter With an Autoregressive Model http://dx.doi.org/10.1109/LSP.2016.2616888 ...... S.BraunandE.A.P. Habets 1741 Data Consistency Conditions for Cone-Beam Projections on a Circular Trajectory http://dx.doi.org/10.1109/LSP.2016.2616026 ...... R. Clackdoyle, L. Desbat, J. Lesaint, and S. Rit 1746 MIMO Detection and Equalization for Single-Carrier Systems Using the Alternating Direction Method of Multipliers http://dx.doi.org/10.1109/LSP.2016.2618959 ...... N. Souto and R. Dinis 1751 Performance Analysis of Low-Flux Least-Squares Single-Pixel Imaging http://dx.doi.org/10.1109/LSP.2016.2617329 ...... D. Shin, J. H. Shapiro, and V. K. Goyal 1756 Mixed Pulse Accumulation for Compressive Sensing Radar http://dx.doi.org/10.1109/LSP.2016.2618958 ...... Y.TaoandG. Zhang 1761 A Virtual User Pairing Scheme to Optimally Utilize the Spectrum of Unpaired Users in Non-orthogonal Multiple Access http://dx.doi.org/10.1109/LSP.2016.2619371 ...... M.B. Shahab, M. F. Kader, and S. Y. Shin 1766 A Topological Low-Pass Filter for Quasi-Periodic Signals http://dx.doi.org/10.1109/LSP.2016.2619678 ...... M. Robinson 1771 Revealing Hidden 3-D Reﬂection Symmetry http://dx.doi.org/10.1109/LSP.2016.2620380 ...... R.NagarandS. Raman 1776

www.signalprocessingsociety.org [22] JANUARY 2017

THE WORLD’S NEWSSTAND®

Novel Channel Estimation for Non-orthogonal Multiple Access Systems http://dx.doi.org/10.1109/LSP.2016.2617897 ...... Y.Tan,J. Zhou, and J. Qin 1781 Transient Performance Analysis of Zero-Attracting LMS http://dx.doi.org/10.1109/LSP.2016.2616890 ...... J. Chen, C. Richard, Y. Song, and D. Brie 1786 Low-Complexity 2D Direction-of-Arrival Estimation for Acoustic Sensor Arrays http://dx.doi.org/10.1109/LSP.2016.2614107 ...... K.Yu,R.E. Hudson, Y. D. Zhang, K. Yao, C. Taylor, and Z. Wang 1791 The Impact of Efficient Transport Blocks Management on the Downlink Power in MIMO Spatial Multiplexing of LTE-A http://dx.doi.org/10.1109/LSP.2016.2622479 ...... B. Al-Doori and X. Liu 1796 Blind Picture Upscaling Ratio Prediction http://dx.doi.org/10.1109/LSP.2016.2603842 ...... T.R. Goodall, I. Katsavounidis, Z. Li, A. Aaron, and A. C. Bovik 1801 Robust Semi-Supervised Classification for Noisy Labels Based on Self-Paced Learning http://dx.doi.org/10.1109/LSP.2016.2619352 ...... N.Gu,M.Fan,andD. Meng 1806 Spatial Filtering Based on Differential Spectrum for Improving ML DOA Estimation Performance http://dx.doi.org/10.1109/LSP.2016.2605006 ...... R.P. Lemos, H. V. Leao e Silva, E. L. Flores, J. A. Kunzler, and D. F. Burgos 1811 Optimal Binary Periodic Almost-Complementary Pairs http://dx.doi.org/10.1109/LSP.2016.2600586 ...... A.R. Adhikary, Z. Liu, Y. L. Guan, S. Majhi, and S. Z. Budishin 1816 Probabilistic Latent Semantic Analysis for Multichannel Biomedical Signal Clustering http://dx.doi.org/10.1109/LSP.2016.2623801 ...... J.WangandM.She 1821 Uncertainty Exchange Through Multiple Quadrature Kalman Filtering http://dx.doi.org/10.1109/LSP.2016.2618397 ...... J.Vilà-Valls, P. Closas, and Á. F. García-Fernández 1825 On the Maximal Invariant Statistic for Adaptive Radar Detection in Partially Homogeneous Disturbance With Persymmetric Covariance http://dx.doi.org/10.1109/LSP.2016.2618619 ...... D. Ciuonzo, D. Orlando, and L. Pallotta 1830 A Geometrical Look at MOSPA Estimation Using Transportation Theory http://dx.doi.org/10.1109/LSP.2016.2614774 ...... G.M. Lipsa and M. Guerriero 1835 Large Inpainting of Face Images With Trainlets http://dx.doi.org/10.1109/LSP.2016.2616354 ...... J. Sulam and M. Elad 1839 Optimal Memory Size Formula for Moving-Average Digital Phase-Locked Loop http://dx.doi.org/10.1109/LSP.2016.2623520 ...... C.K. Ahn, P. Shi, and S. H. You 1844 Unified Maximum Likelihood Form for Bias Constrained FIR Filters http://dx.doi.org/10.1109/LSP.2016.2627001 ...... S. Zhao and Y. S. Shmaliy 1848 Symbol Detection for Faster-Than-Nyquist Signaling by Sum-of-Absolute-Values Optimization http://dx.doi.org/10.1109/LSP.2016.2625839 ...... H. Sasahara, K. Hayashi, and M. Nagahara 1853 Sparse Decomposition for Signal Periodic Model Over Complex Exponential Dictionary http://dx.doi.org/10.1109/LSP.2016.2619329 ...... S. Deng and J. Han 1858 Low-Rank and Sparsity Analysis Applied to Speech Enhancement Via Online Estimated Dictionary http://dx.doi.org/10.1109/LSP.2016.2627029 ...... P.SunandJ.Qin 1862 Online Anomaly Detection With Nested Trees http://dx.doi.org/10.1109/LSP.2016.2623773 ...... I. Delibalta, K. Gokcesu, M. Simsek, L. Baruh, and S. S. Kozat 1867 Second-Order Cone Relaxation for TDOA-Based Localization Under Mixed LOS/NLOS Conditions http://dx.doi.org/10.1109/LSP.2016.2627603 ...... W.Wang,G.Wang,F. Zhang, and Y. Li 1872 Speech Bandwidth Extension Using Recurrent Temporal Restricted Boltzmann Machines http://dx.doi.org/10.1109/LSP.2016.2621053 ...... Y.Wang,S. Zhao, J. Li, and J. Kuang 1877 Image Fusion With Convolutional Sparse Representation http://dx.doi.org/10.1109/LSP.2016.2618776 ...... Y. Liu, X. Chen, R. K. Ward, and Z. J. Wang 1882 Superpixel-Guided Adaptive Image Smoothing http://dx.doi.org/10.1109/LSP.2016.2630741 ...... H.EunandC.Kim 1887

List of Reviewers http://dx.doi.org/10.1109/LSP.2016.2617138 ...... 1892

EDICS–Editors’ Information Classiﬁcation Scheme http://dx.doi.org/10.1109/LSP.2016.2627696 ...... 1902 Information for Authors http://dx.doi.org/10.1109/LSP.2016.2627697 ...... 1903

2016 INDEX http://dx.doi.org/10.1109/LSP.2016.2636642 ...... 1905

www.signalprocessingsociety.org [23] JANUARY 2017

THE WORLD’S NEWSSTAND®

DECEMBER 2016 VOLUME 2 NUMBER 4 ITCIAJ (ISSN 2333-9403)

PAPERS Computational Imaging Methods and Models A Hybrid Markov Random Field/Marked Point Process Model for Analysis of Materials Images http://dx.doi.org/10.1109/TCI.2016.2579601 ...... H. Zhao and M. Comer 395 Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation http://dx.doi.org/10.1109/TCI.2016.2599778 ...... S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman 408 Mean Squared Error Based Excitation Pattern Design for Parallel Transmit and Receive SENSE MRI Image Reconstruction http://dx.doi.org/10.1109/TCI.2016.2610141 .... I. Y. Chun, S. Noh, D. J. Love, T. M. Talavage, S. Beckley, and S. J. Kisner 424 Spectral Super-Resolution in Colored Coded Aperture Spectral Imaging http://dx.doi.org/10.1109/TCI.2016.2612943 ...... A.Parada-Mayorga and G. R. Arce 440 Robust Bayesian Target Detection Algorithm for Depth Imaging From Sparse Single-Photon Data http://dx.doi.org/10.1109/TCI.2016.2618323 ...... Y. Altmann, X. Ren, A. McCarthy, G. S. Buller, and S. McLaughlin 456 Computational Image Formation Combining Local Regularity Estimation and Total Variation Optimization for Scale-Free Texture Segmentation http://dx.doi.org/10.1109/TCI.2016.2594139 ...... N. Pustelnik, H. Wendt, P. Abry, and N. Dobigeon 468

www.signalprocessingsociety.org [24] JANUARY 2017

THE WORLD’S NEWSSTAND®

Computational Imaging Systems A General Framework for Compressed Sensing and Parallel MRI Using Annihilating Filter Based Low-Rank Hankel Matrix http://dx.doi.org/10.1109/TCI.2016.2601296 ...... K.H. Jin, D. Lee, and J. C. Ye 480 Fast Multilayer Laplacian Enhancement http://dx.doi.org/10.1109/TCI.2016.2607142 ...... H.Talebi and P. Milanfar 496 Spectral CT Reconstruction With Image Sparsity and Spectral Mean http://dx.doi.org/10.1109/TCI.2016.2609414 ...... Y. Zhang, Y. Xi, Q. Yang, W. Cong, J. Zhou, and G. Wang 510 Two-dimensional Autofocus for Spotlight SAR Polar Format Imagery http://dx.doi.org/10.1109/TCI.2016.2612945 .... X. Mao and D. Zhu 524 Numerical Inversion of Circular arc Radon Transform http://dx.doi.org/10.1109/TCI.2016.2615806 ...... T.A. Syed, V. P. Krishnan, and J. Sivaswamy 540 Computational Imaging Hardware and Software

Vehicle Tracking in Video Using Fractional Feedback Kalman Filter http://dx.doi.org/10.1109/TCI.2016.2600480 ...... H. Kaur and J. S. Sahambi 550

List of Reviewers http://dx.doi.org/10.1109/TCI.2016.2618098 ...... 562

EDICS—Editor's Classiﬁcation Information Scheme http://dx.doi.org/10.1109/TCI.2016.2622275 ...... 564 Information for Authors http://dx.doi.org/10.1109/TCI.2016.2622276 ...... 565

2016 INDEX ...... Available online at http://ieeexplore.ieee.org

www.signalprocessingsociety.org [25] JANUARY 2017

THE WORLD’S NEWSSTAND®

DECEMBER 2016 VOLUME 2 NUMBER 4 ITSIBW (ISSN 2373-776X)

EDITORIAL

Guest Editorial Inference and Learning over Networks http://dx.doi.org/10.1109/TSIPN.2016.2615526 ...... V. Matta, C. Richard, V. Saligrama, and A. H. Sayed 423

PAPERS Distributed Constrained Recursive Nonlinear Least-Squares Estimation: Algorithms and Asymptotics http://dx.doi.org/10.1109/TSIPN.2016.2618318 ...... A.K. Sahu, S. Kar, J. M. F. Moura, and H. V. Poor 426 Distributed Detection Over Adaptive Networks: Refined Asymptotics and the Role of Connectivity http://dx.doi.org/10.1109/TSIPN.2016.2613682 ...... V. Matta, P. Braca, S. Marano, and A. H. Sayed 442 Uniform -Stability of Distributed Nonlinear Filtering Over DNAs: Gaussian-Finite HMMs http://dx.doi.org/10.1109/TSIPN.2016.2614902 ...... D.S. Kalogerias and A. P. Petropulu 461 A Distributed Quaternion Kalman Filter With Applications to Smart Grid and Target Tracking http://dx.doi.org/10.1109/TSIPN.2016.2618321 ...... S.P.Talebi, S. Kanna, and D. P. Mandic 477 Efficient Distributed Online Prediction and Stochastic Optimization With Approximate Distributed Averaging http://dx.doi.org/10.1109/TSIPN.2016.2620440 ...... K.I. Tsianos and M. G. Rabbat 489 A Decentralized Second-Order Method with Exact Linear Convergence Rate for Consensus Optimization http://dx.doi.org/10.1109/TSIPN.2016.2613678 ...... A. Mokhtari, W. Shi, Q. Ling, and A. Ribeiro 507 Data Injection Attacks in Randomized Gossiping http://dx.doi.org/10.1109/TSIPN.2016.2614898 ...... R. Gentz, S. X. Wu, H.-T. Wai, A. Scaglione, and A. Leshem 523 Signal Recovery on Graphs: Fundamental Limits of Sampling Strategies http://dx.doi.org/10.1109/TSIPN.2016.2614903 ...... S. Chen, R. Varma, A. Singh, and J. Kovacevicˇ ´ 539 Adaptive Least Mean Squares Estimation of Graph Signals http://dx.doi.org/10.1109/TSIPN.2016.2613687 ...... P.DiLorenzo, S. Barbarossa, P. Banelli, and S. Sardellitti 555 Information Diffusion of Topic Propagation in Social Media http://dx.doi.org/10.1109/TSIPN.2016.2618324 ...... S. Mahdizadehaghdam, H. Wang, H. Krim, and L. Dai 569 Metadata-Conscious Anonymous Messaging http://dx.doi.org/10.1109/TSIPN.2016.2605761 ...... G.Fanti, P. Kairouz, S. Oh, K. Ramchandran, and P. Viswanath 582 Evolutionary Information Diffusion Over Heterogeneous Social Networks http://dx.doi.org/10.1109/TSIPN.2016.2613680 ...... X. Cao, Y. Chen, C. Jiang, and K. J. Ray Liu 595 Dual Graph Regularized Dictionary Learning http://dx.doi.org/10.1109/TSIPN.2016.2605763 ...... Y.YankelevskyandM. Elad 611 On the Difficulty of Selecting Ising Models With Approximate Recovery http://dx.doi.org/10.1109/TSIPN.2016.2596439 ...... J. Scarlett and V. Cevher 625

www.signalprocessingsociety.org [26] JANUARY 2017

THE WORLD’S NEWSSTAND®

LIST OF REVIEWERS http://dx.doi.org/10.1109/TSIPN.2016.2618501 ...... 639

EDICS—Editor's Information Classiﬁcation Scheme http://dx.doi.org/10.1109/TSIPN.2016.2622166 ...... 643 Information for Authors http://dx.doi.org/10.1109/TSIPN.2016.2622168 ...... 644

2016 INDEX ...... Available online at http://ieeexplore.ieee.org

www.signalprocessingsociety.org [27] JANUARY 2017

THE WORLD’S NEWSSTAND®

*HQHUDO&KDLUV 63$:& ZLOO EH KHOG DW +RNNDLGR 8QLYHUVLW\ LQ 6DSSRUR -DSDQ RQ -XO\

7HFKQLFDO3URJUDP&KDLUV &DOOIRU3DSHUV 7RPRDNL2KWVXNL /XW]/DPSH 3URVSHFWLYH DXWKRUV DUH LQYLWHG WR VXEPLW SDSHUV LQ WKH IROORZLQJ DUHDV :LQJ.LQ .HQ 0D 6PDUW DQWHQQDV 0,02 V\VWHPV PDVVLYH 0,02 DQG VSDFHWLPH SURFHVVLQJ 6LQJOHFDUULHU PXOWLFDUULHU DQG PXOWLUDWH V\VWHPV 6SHFLDO6HVVLRQ&KDLU 0XOWLSOHDFFHVV DQG EURDGFDVW FKDQQHOV PXOWLXVHU UHFHLYHUV 7RQ\464XHN 6LJQDO SURFHVVLQJ IRU DGKRF PXOWLKRS DQG VHQVRU QHWZRUNV 7HFKQLFDO3URJUDP&RPPLWWHH &RRSHUDWLYH FRPPXQLFDWLRQ FRRUGLQDWHG PXOWLSRLQW WUDQVPLVVLRQ DQG UHFHSWLRQ :DKHHG%DMZD 'LVWULEXWHG UHVRXUFH DOORFDWLRQ DQG VFKHGXOLQJ &KRQJ

)LQDQFH&KDLUV 3XEOLFDWLRQ&KDLUV

/RFDO$UUDQJHPHQW&KDLUV 3XEOLFLW\&KDLUV 7DNHR2KJDQH -XOLDQ:HEEHU 7RVKLKLNR1LVKLPXUD .RLFKL$GDFKL

www.signalprocessingsociety.org [28] JANUARY 2017

THE WORLD’S NEWSSTAND®

Call for Symposium Proposals

www.signalprocessingsociety.org [29] JANUARY 2017

THE WORLD’S NEWSSTAND®

*OREDO6,3&DOOIRU6\PSRVLXP3URSRVDOV

We invite Symposium proposals for the fifth IEEE Global Conference on Signal and Information Processing (GlobalSIP2017) which will be held in Montreal, Quebec, Canada on 1RYHPEHU (http://2017.ieeeglobalsip.org/ ). GlobalSIP is a flagship IEEE Signal Processing Society conference. It focuses on signal and information processing with an emphasis on up-and-coming signal processing themes. The conference features world-class plenary speeches, distinguished Symposium talks, tutorials, exhibits, oral and poster sessions, and panels. GlobalSIP is comprised of co-located General Symposium and symposia selected based on responses to the call-for-symposia proposals. Topics include but are not limited to:

Signal and information processing for Multimedia transmission, indexing, retrieval, and R communications and networks, including quality of experience green communications Selected topics in statistical signal processing R optical communications Cognitive communications and radar R forensics and security Graph-theoretic signal processing R finance Machine learning R energy and power systems (e.g., smart grid) Compressed sensing and sparsity aware processing R genomics and bioengineering (physiological, Seismic signal processing pharmacological and behavioral) Big data and social media challenges R neural networks, including deep learning Hardware and real-time implementations Image and video processing Other (industrial) emerging applications of signal and Selected topics in speech processing and human information processing. language technologies Interdisciplinary theme symposia are strongly encouraged. Human machine interfaces

Symposium proposals should contain the following information: title; duration (e.g., full day or half day); paper length; name, address, and a short CV (up to 250 words) of the organizers, including the technical chairs (if any); a 1-page or 2-page description of the topics to be addressed, including timeliness and relevance to the signal processing community; names of (potential) members of the technical program committee; invited/potential speakers' names; a draft call for papers (up to 1 page). Please pack everything together in a single pdf document and email your proposal to the Technical Program Committee (TPC) Chairs. More detailed information can be found in “GlobalSIP2017 Symposium Proposal Preparation Guide” at http://2017.ieeeglobalsip.org/. For better consideration, proposers are strongly advised to submit their proposals as early as possible. 3URSRVHG7LPHOLQH -DQ: Symposium proposals due -DQ: Symposium selection decision made )HE: Call for Papers for accepted Symposia 0D\: Paper submission due -XQH: Notification of Acceptance -XO\: Camera-ready paper due.

www.signalprocessingsociety.org [30] JANUARY 2017

THE WORLD’S NEWSSTAND®

www.signalprocessingsociety.org [31] JANUARY 2017

THE WORLD’S NEWSSTAND® Contents Volume 33 | Number 6 | November 2016

RECENT ADVANCES IN ACTIVE SPECIAL SECTION 61 NOISE CONTROL INSIDE AUTOMOBILE CABINS SIGNAL PROCESSING Prasanga N. Samarasinghe, Wen Zhang, and FOR SMART VEHICLE Thushara D. Abhayapala TECHNOLOGIES COORDINATION OF COOPERATIVE 74 AUTONOMOUS VEHICLES 12 FROM THE GUEST EDITORS Robert Hult, Gabriel R. Campos, John H.L. Hansen, Erik Steinmetz, Lars Hammarstrand, Kazuya Takeda, Paolo Falcone, and Henk Wymeersch Sanjeev M. Naik, Mohan M. Trivedi, Gerhard U. Schmidt, and Yingying (Jennifer) Chen FEATURES DRIVER-BEHAVIOR MODELING 14 USING ON-ROAD DRIVING DATA ON THE COVER THE RACE TO IMPROVE Chiyomi Miyajima 85 RADAR IMAGERY and Kazuya Takeda Signal processing is playing an increasingly substantial role in advancing today’s vehicles into “smart” ones. The Lifan Zhao, Lu Wang, special section in this issue provides a venue for summa- Lei Yang, Abdelhak M. Zoubir, DRIVER STATUS rizing, educating, and sharing the state of the art in signal and Guoan Bi 22 MONITORING SYSTEMS FOR processing applied to automotive systems. SMART VEHICLES USING MATCHING THEORY PHYSIOLOGICAL SENSORS COVER IMAGE: ©ISTOCKPHOTO.COM/NADLA 103 Youjun Choi, Sang Ik Han, Siavash Bayat, Yonghui Li, Seung-Hyun Kong, Lingyang Song, and Zhu Han and Hyunwoo Ko SMART DRIVER MONITORING: 35 WHEN SIGNAL PROCESSING MEETS HUMAN FACTORS Amirhossein S. Aghaei, Birsen Donmez, Cheng Chen Liu, Dengbo He, George Liu, Konstantinos N. Plataniotis, Huei-Yen Winnie Chen, and Zohreh Sojoudi

CONVERSATIONAL IN-VEHICLE 49 DIALOG SYSTEMS Fuliang Weng, Pongtep Angkititrakul, Elizabeth E. Shriberg, PG. 123 Larry Heck, Stanley Peters, PG. 8 and John H.L. Hansen

IEEE SIGNAL PROCESSING MAGAZINE (ISSN 1053-5888) (ISPREG) is published bimonthly by the Institute of Electrical and Electronics Engineers, Inc., 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA (+1 212 419 7900). Responsibility for the contents rests upon the authors and not the IEEE, the Society, or its members. Annual member subscriptions included in Society fee. Nonmember subscriptions available upon request. Individual copies: IEEE Members US$20.00 (first copy only), nonmembers US$213.00 per copy. Copyright and Reprint Permissions: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright Law for private use of patrons: 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without fee. Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For all other copying, reprint, or republication permission, write to IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Copyright © 2016 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals postage paid at New York, NY, and at additional mailing offices. Postmaster: Send address changes to IEEE Signal Processing Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188 Printed in the U.S.A.

Digital Object Identifier 10.1109/MSP.2016.2610483

www.signalprocessingsociety.org [32] JANUARY 2017 1

THE WORLD’S NEWSSTAND®

25'(5)250 )255(35,176 3XUFKDVLQJ,(((3DSHUVLQ3ULQWLVHDV\FRVWHIIHFWLYH DQGTXLFN &RPSOHWHWKLVIRUPVHQGYLDRXUVHFXUHID[ KRXUVDGD\ WRRUPDLO LWEDFNWRXV 3/($6( ),//2877+()2//2:,1* $XWKRU 5(78517+,6)25072 ,(((3XEOLVKLQJ6HUYLFHV 3XEOLFDWLRQ7LWOH +RHV/DQH 3LVFDWDZD\1- 3DSHU7LWOH (PDLOWKH5HSULQW'HSDUWPHQWDW UHSULQWV#LHHHRUJIRUTXHVWLRQVUHJDUGLQJ______WKLVIRUP 3/($6(6(1'0( LQPXOWLSOHVRI UHSULQWV RU ع ع ع ع ع ع SHU SHU126HOIFRYHULQJWLWOHSDJHUHTXLUHG&29(535,&( ع 6)> ع LU )UHLJKWPXVWEHDGGHGIRUDOORUGHUVEHLQJVKLSSHGRXWVLGHWKH86$ ع PXVWEHDGGHGIRUDOO86$VKLSPHQWV WRFRYHU WKHFRVW RI836VKLSSLQJ DQGKDQGOLQJ ع 3$<0(17 KHFNHQFORVHG3D\DEOHRQDEDQNLQWKH86$& ع LQHUV&OXE' ع ]PH$ ع 0DVWHUFDUG ع 9LVD ع \KDUJHP& ع

$FFRXQWB([SGDWH &DUGKROGHU¶V1DPH SOHDVHSULQW \RXPXVWDWWDFKDSXUFKDVHRUGHU 3XUFKDVH2UGHU1XPEHU LOOPH% ع 6HQG5HSULQWVWR %LOOWRDGGUHVVLIGLIIHUHQW %HFDXVHLQIRUPDWLRQDQGSDSHUVDUHJDWKHUHGIURPYDULRXVVRXUFHVWKHUHPD\EHDGHOD\LQUHFHLYLQJ\RXUUHSULQWUHTXHVW 7KLVLVHVSHFLDOO\WUXHZLWK SRVWFRQIHUHQFH SXEOLFDWLRQV3OHDVHSURYLGHXVZLWKFRQWDFW LQIRUPDWLRQLI\RXZRXOGOLNHQRWLILFDWLRQRIDGHOD\RIPRUHWKDQZHHNV 7HOHSKRQH )D[(PDLO$GGUHVV 5(35,1735,&(6 ZLWKRXWFRYHUV 1XPEHURI7H[W3DJHV /DUJHUT4XDQWLWLHVFDQEHRUGHUHG(PDLOUHSULQWV#LHHHRUJ______ZLWKVSHFLILFGHWDLOV 7D[$SSOLHV RQVKLSPHQWVRIUHJXODUUHSULQWVWR&$'&)/0, 1- 1<2+DQG &DQDGD *675HJLVWUDWLRQ QR 3ULFHVDUHEDVHGRQEODFN ZKLWHSULQWLQJ3OHDVHFDOOXVIRUIXOOFRORUSULFHTXRWHLIDSSOLFDEOH $XWKRUL]HG6LJQDWXUH 'DWH

www.signalprocessingsociety.org [33] JANUARY 2017

THE WORLD’S NEWSSTAND® 2017 IEEE MEMBERSHIP APPLICATION (students and graduate students must apply online)

Start your membership immediately: Join online www.ieee.org/join 3 Please Tell Us About Yourself Please complete both sides of this form, typing or printing in capital letters. Select the numbered option that best describes yourself. This infor- Use only English characters and abbreviate only if more than 40 characters and mation is used by IEEE magazines to verify their annual circulation. spaces per line. We regret that incomplete applications cannot be processed. Please enter numbered selections in the boxes provided.

A. Primary line of business W 1PERSONALName & INFORMATIONContact Information 1. Computers 2. Computer peripheral equipment Please PRINT your name as you want it to appear on your membership card and IEEE 3. Software correspondence. As a key identifier for the IEEE database, circle your last/surname. 4. Office and business machines 5. Test, measurement and instrumentation equipment Q Male Q Female Date of birth (Day/Month/Year) / / 6. Communications systems and equipment 7. Navigation and guidance systems and equipment 8. Consumer electronics/appliances 9. Industrial equipment, controls and systems Title First/Given Name Middle Last/Family Surname 10. ICs and microprocessors 11. Semiconductors, components, sub-assemblies, materials and supplies WPrimary Address Q Home Q Business (All IEEE mail sent here) 12. Aircraft, missiles, space and ground support equipment 13. Oceanography and support equipment 14. Medical electronic equipment Street Address 15. OEM incorporating electronics in their end product (not elsewhere classified) 16. Independent and university research, test and design laboratories and consultants (not connected with a mfg. co.) City State/Province 17. Government agencies and armed forces 18. Companies using and/or incorporating any electronic products in their manufacturing, processing, research or development activities Postal Code Country 19. Telecommunications services, telephone (including cellular) 20. Broadcast services (TV, cable, radio) 21. Transportation services (airline, railroad, etc.) Primary Phone 22. Computer and communications and data processing services 23. Power production, generation, transmission and distribution 24. Other commercial users of electrical, electronic equipment and services Primary E-mail (not elsewhere classified) 25. Distributor (reseller, wholesaler, retailer) WSecondary Address Q Home Q Business 26. University, college/other educational institutions, libraries 27. Retired 28. Other______Company Name Department/Division

B. Principal job function W Street Address City State/Province 1. General and corporate management 9. Design/development 2. Engineering management engineering—digital 3. Project engineering management 10. Hardware engineering Postal Code Country 4. Research and development 11. Software design/development management 12. Computer science Secondary Phone 5. Design engineering management 13. Science/physics/mathematics —analog 14. Engineering (not elsewhere 6. Design engineering management specified) Secondary E-mail —digital 15. Marketing/sales/purchasing 7. Research and development 16. Consulting engineering 17. Education/teaching To better serve our members and supplement member dues, your postal mailing address is made available to 8. Design/development engineering 18. Retired carefully selected organizations to provide you with information on technical services, continuing education, and —analog 19. Other conferences. Your e-mail address is not rented by IEEE. Please check box only if you do not want to receive these

postal mailings to the selected address. Q C. Principal responsibility W 1. Engineering and scientific management 6. Education/teaching 2. Management other than engineering 7. Consulting 3. Engineering design 8. Retired 2 Attestation 4. Engineering 9. Other 5. Software: science/mngmnt/engineering I have graduated from a three- to five-year academic program with a university-level degree. Yes No Q Q D. Title W This program is in one of the following fields of study: 1. Chairman of the Board/President/CEO 10. Design Engineering Manager 2. Owner/Partner 11. Design Engineer Q Engineering 3. General Manager 12. Hardware Engineer Q Computer Sciences and Information Technologies 4. VP Operations 13. Software Engineer Q Physical Sciences 5. VP Engineering/Dir. Engineering 14. Computer Scientist Q Biological and Medical Sciences 6. Chief Engineer/Chief Scientist 15. Dean/Professor/Instructor Q Mathematics 7. Engineering Management 16. Consultant 8. Scientific Management 17. Retired Q Technical Communications, Education, Management, Law and Policy 9. Member of Technical Staff 18. Other Q Other (please specify): ______Are you now or were you ever a member of IEEE? This academic institution or program is accredited in the country where the institution Q Yes Q No If yes, provide, if known: is located. Q Yes Q No Q Do not know

I have ______years of professional experience in teaching, creating, developing, Membership Number Grade Year Expired practicing, or managing within the following field:

Q Engineering 4 Please Sign Your Application Q Computer Sciences and Information Technologies I hereby apply for IEEE membership and agree to be governed by the Q Physical Sciences IEEE Constitution, Bylaws, and Code of Ethics. I understand that IEEE Q Biological and Medical Sciences will communicate with me regarding my individual membership and all Q Mathematics related benefits. Application must be signed. Q Technical Communications, Education, Management, Law and Policy Q Other (please specify): ______

Signature Date Over Please

(continued on next page)

www.signalprocessingsociety.org [34] JANUARY 2017

THE WORLD’S NEWSSTAND® 5 Add IEEE Society Memberships (Optional) 6 2017 IEEE Membership Rates (student rates available online) The 39 IEEE Societies support your technical and professional interests. Many society memberships include a personal subscription to the core journal, IEEE member dues and regional assessments are based on where magazine, or newsletter of that society. For a complete list of everything you live and when you apply. Membership is based on the calendar included with your IEEE Society membership, visit www.ieee.org/join. year from 1 January through 31 December. All prices are quoted All prices are quoted in US dollars. in US dollars. Please check the appropriate box. BETWEEN BETWEEN BETWEEN BETWEEN 16 AUG 2016- 1 MAR 2017- Please check the appropriate box. 16 AUG 2016- 1 MAR 2017- 28 FEB 2017 15 AUG 2017 28 FEB 2017 15 AUG 2017 RESIDENCE PAY PAY PAY PAY United States...... $199.00 Q...... $99.50 Q IEEE Aerospace and Electronic Systems a` AES010 25.00 Q 12.50 Q Canada (NB, NF, NS, and PEI HST)*...... $190.35 Q...... $95.18 Q IEEE Antennas and Propagation a` AP003 15.00 Q 7.50 Q Canada (ON HST) ...... $187.37 Q...... $93.69 Q IEEE Broadcast Technology a` BT002 15.00 Q 7.50 Q Canada (GST)*...... $175.45 Q...... $87.73 Q IEEE Circuits and Systems a` CAS004 22.00 Q 11.00 Q Canada (GST and QST Quebec)...... $190.31 Q...... $95.16 Q IEEE Communications a` C0M019 33.00 Q 16.50 Q Africa, Europe, Middle East...... $162.00 Q...... $81.00 Q IEEE Components, Packaging, & Manu. Tech. a` CPMT021 15.00 Q 7.50 Q Latin America...... $153.00 Q...... $76.50 Q IEEE Computational Intelligence a` CIS011 29.00 Q 14.50 Q Asia, Pacific...... $154.00 Q...... $77.00 Q IEEE Computer a` C016 60.00 Q 30.00 Q *IEEE Canada Business No. 125634188 IEEE Consumer Electronics a` CE008 20.00 Q 10.00 Q Minimum Income or Unemployed Provision IEEE Control Systems a` CS023 25.00 Q 12.50 Q Applicants who certify that their prior year income did not exceed US$14,900 (or equivalent) or were not employed are granted 50% reduction in: full-year dues, IEEE Dielectrics and Electrical Insulation a` DEI032 26.00 Q 13.00 Q regional assessment and fees for one IEEE Membership plus one Society Membership. IEEE Education a` E025 20.00 Q 10.00 Q If applicable, please check appropriate box and adjust payment accordingly. Student members are not eligible. IEEE Electromagnetic Compatibility a` EMC027 31.00 Q 15.50 Q Q I certify I earned less than US$14,900 in 2016 IEEE Electron Devices a` ED015 18.00 Q 9.00 Q Q I certify that I was unemployed in 2016 IEEE Engineering in Medicine and Biology a` EMB018 40.00 Q 20.00 Q IEEE Geoscience and Remote Sensing a` GRS029 19.00 Q 9.50 Q 7 More Recommended Options IEEE Industrial Electronics a` IE013 9.00 Q 4.50 Q Proceedings of the IEEE...... print $49.00 Qoronline $43.00 Q IEEE Industry Applications a` IA034 20.00 Q 10.00 Q Proceedings of the IEEE (print/online combination)...... $59.00 Q IEEE Information Theory a` IT012 30.00 Q 15.00 Q IEEE Standards Association (IEEE-SA) ...... $54.00 Q IEEE Instrumentation and Measurement a` IM009 29.00 Q 14.50 Q IEEE Women in Engineering (WIE) ...... $25.00 Q IEEE Intelligent Transportation Systems a` ITSS038 35.00 Q 17.50 Q IEEE Magnetics ` a MAG033 26.00 Q 13.00 Q 8 Payment Amount IEEE Microwave Theory and Techniques a` MTT017 24.00 Q 12.00 Q IEEE Nuclear and Plasma Sciences a` NPS005 35.00 Q 17.50 Q Please total the Membership dues, Society dues, and other amounts from this page: IEEE Oceanic Engineering a` OE022 19.00 Q 9.50 Q IEEE Membership dues 6 ...... $______IEEE Photonics a` PHO036 34.00 Q 17.00 Q IEEE Society dues (optional) 5 ...... $______IEEE Power Electronics a` PEL035 25.00 Q 12.50 Q IEEE-SA/WIE dues (optional) 7 ...... $______IEEE Power & Energy a` PE031 35.00 Q 17.50 Q Proceedings of the IEEE (optional) 7 ...... $______IEEE Product Safety Engineering PSE043 35.00 Q 17.50 Q Canadian residents pay 5% GST or appropriate HST (BC–12%; ON–13%; NB, NF, NS, PEI–15%) on Society payments & publications only ...... TAX $______IEEE Professional Communication a` PC026 31.00 Q 15.50 Q IEEE Reliability a` RL007 35.00 Q 17.50 Q AMOUNT PAID ...... TOTAL $______IEEE Robotics and Automation a` RA024 9.00 Q 4.50 Q Payment Method All prices are quoted in US dollars. You may pay for IEEE membership IEEE Signal Processing a` SP001 22.00 Q 11.00 Q by credit card (see below), check, or money order payable to IEEE, IEEE Social Implications of Technology a` SIT030 33.00 Q 16.50 Q drawn on a US bank. IEEE Solid-State Circuits a` SSC037 22.00 Q 11.00 Q Q Check Q Q Q Q Q IEEE Systems, Man, & Cybernetics a` SMC028 12.00 Q 6.00 Q IEEE Technology & Engineering Management a` TEM014 35.00 Q 17.50 Q IEEE Ultrasonics, Ferroelectrics, & Frequency Control a` UFFC020 20.00 Q 10.00 Q Credit Card Number IEEE Vehicular Technology a` VT006 18.00 Q 9.00 Q

MONTH YEAR CARDHOLDER’S 5-DIGIT ZIP CODE Legend—Society membership includes: EXPIRATION DATE (BILLING STATEMENT ADDRESS) USA ONLY a One or more Society publications ` Online access to publication Name as it appears on card Society newsletter CD-ROM of selected society publications Signature Auto Renew my Memberships and Subscriptions (available when paying by credit card). Complete both sides of this form, sign, and return to: Q I agree to the Terms and Conditions located at www.ieee.org/autorenew IEEE MEMBERSHIP APPLICATION PROCESSING 445 HOES LN, PISCATAWAY, NJ 08854-4141 USA 9 Were You Referred to IEEE? or fax to +1 732 981 0225 Q Yes Q No If yes, provide the following: or join online at www.ieee.org/join Member Recruiter Name IEEE Recruiter’s Member Number (Required)

CAMPAIGN CODE PROMO CODE Please reprint your full name here 16-MEM-031 P 6/16

www.signalprocessingsociety.org [35] JANUARY 2017

THE WORLD’S NEWSSTAND®

,QIRUPDWLRQ IRU $XWKRUV 8SGDWHG(IIHFWLYH -DQXDU\ )RU 7UDQVDFWLRQV DQG -RXUQDOV YHUVLRQ 3URRIUHDG \RXU VXEPLVVLRQ FRQ¿UPLQJ WKDW DOO $XWKRUV DUH HQFRXUDJHG WR VXEPLW PDQXVFULSWV RI 5HJXODU SDSHUV SD ¿JXUHV DQG HTXDWLRQV DUH YLVLEOH LQ \RXU GRFXPHQW EHIRUH SHUV ZKLFK SURYLGH D FRPSOHWH GLVFORVXUH RI D WHFKQLFDO SUHPLVH RU \RX ³68%0,7´ \RXU PDQXVFULSW 3URRIUHDGLQJ LV FULWLFDO &RPPHQW &RUUHVSRQGHQFHV EULHI LWHPV WKDW SURYLGH FRPPHQW RQ D RQFH \RX VXEPLW \RXU PDQXVFULSW WKH PDQXVFULSW FDQQRW EH SDSHU SUHYLRXVO\ SXEOLVKHG LQ WKHVH 75$16$&7,216 FKDQJHG LQ DQ\ ZD\ 7KLV GRHV QRW DSSO\ WR WKH -RXUQDO RI 6HOHFWHG 7RSLFV LQ 6LJQDO OHQJWK UHVWULFWLRQV ZLOO EH GHWHUPLQHG E\ WKH GRXEOHFROXPQ 3URFHVVLQJ 3OHDVH FRQWDFW WKH (GLWRULQ&KLHI@

'LJLWDO 2EMHFW ,GHQWL¿HU

www.signalprocessingsociety.org [36] JANUARY 2017

THE WORLD’S NEWSSTAND®

0DQXVFULSW /HQJWK )RU WKH LQLWLDO VXEPLVVLRQ RI D UHJXODU SDSHU WKH 7KH DEVWUDFW PXVW EH EHWZHHQ ZRUGV PDQXVFULSW PD\ QRW H[FHHG GRXEOHFROXPQ SDJHV SRLQW IRQW LQ 7KH DEVWUDFW VKRXOG LQFOXGH D IHZ NH\ZRUGV RU SKUDVHV DV WKLV FOXGLQJ WLWOH QDPHV RI DXWKRUV DQG WKHLU FRPSOHWH FRQWDFW LQIRUPDWLRQ ZLOO KHOS UHDGHUV WR ¿QG LW $YRLG RYHUUHSHWLWLRQ RI VXFK SKUDVHV DEVWUDFW WH[W DOO LPDJHV ¿JXUHV DQG WDEOHV DSSHQGLFHV DQG SURRIV DQG DV WKLV FDQ UHVXOW LQ D SDJH EHLQJ UHMHFWHG E\ VHDUFK HQJLQHV DOO UHIHUHQFHV 6XSSOHPHQWDO PDWHULDOV DQG JUDSKLFDO DEVWUDFWV DUH QRW ,Q DGGLWLRQ WR ZULWWHQ DEVWUDFWV SDSHUV PD\ LQFOXGH D JUDSKLFDO LQFOXGHG LQ WKH SDJH FRXQW )RU UHJXODU SDSHUV WKH UHYLVHG PDQXVFULSW DEVWUDFW VHH______KWWSZZZLHHHRUJSXEOLFDWLRQVBVWDQGDUGVSXEOLFDWLRQV PD\ QRW H[FHHG GRXEOHFROXPQ SDJHV SRLQW IRQW LQFOXGLQJ DXWKRUVDXWKRUVBMRXUQDOVKWPO IRU RSWLRQV DQG IRUPDW UHTXLUHPHQWV WLWOH QDPHV RI DXWKRUV DQG WKHLU FRPSOHWH FRQWDFW LQIRUPDWLRQ DEVWUDFW ______WH[W DOO LPDJHV ¿JXUHV DQG WDEOHV DSSHQGLFHV DQG SURRIV DQG DOO ,((( VXSSRUWV WKH SXEOLFDWLRQ RI DXWKRU QDPHV LQ WKH QDWLYH ODQ UHIHUHQFHV )RU 2YHUYLHZ 3DSHUV WKH PD[LPXP OHQJWK LV GRXEOH WKDW JXDJH DORQJVLGH WKH (QJOLVK YHUVLRQV RI WKH QDPHV LQ WKH DXWKRU OLVW RI DQ DUWLFOH )RU PRUH LQIRUPDWLRQ VHH ³$XWKRU QDPHV LQ QDWLYH ODQ IRU UHJXODU VXEPLVVLRQV DW HDFK VWDJH SOHDVH UHIHUHQFH______KWWSZZZ JXDJHV´ KWWSZZZLHHHRUJSXEOLFDWLRQVBVWDQGDUGVSXEOLFDWLRQV______ ______VLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVRYHUYLHZDUWLFOHV IRU PRUH LQIRUPDWLRQ ______DXWKRUVDXWKBQDPHVBQDWLYHBODQJSGI RQ WKH ,((( $XWKRU 'LJLWDO 1RWH WKDW DQ\ SDSHU LQ H[FHVV RI SDJHV ZLOO EH VXEMHFW WR PDQGDWRU\ 7RROER[ SDJH RYHUOHQJWK SDJH FKDUJHV 6LQFH FKDQJHV UHFRPPHQGHG DV D UHVXOW RI SHHU 2SHQ $FFHVV 7KH SXEOLFDWLRQ LV D K\EULG MRXUQDO DOORZLQJ HLWKHU UHYLHZ PD\ UHTXLUH DGGLWLRQV WR WKH PDQXVFULSW LW LV VWURQJO\ UHFRP 7UDGLWLRQDO PDQXVFULSW VXEPLVVLRQ RU 2SHQ $FFHVV DXWKRUSD\V 2$ PHQGHG WKDW \RX SUDFWLFH HFRQRP\ LQ SUHSDULQJ RULJLQDO VXEPLVVLRQV PDQXVFULSW VXEPLVVLRQ 8SRQ VXEPLVVLRQ LI \RX FKRRVH WR KDYH \RXU 1RWH 3DSHUV VXEPLWWHG WR WKH 75$16$&7,216 21 08/7,0(',$ LQ H[ PDQXVFULSW EH DQ 2SHQ $FFHVV DUWLFOH \RX FRPPLW WR SD\ WKH GLV FHVV RI SDJHV ZLOO EH VXEMHFW WR PDQGDWRU\ RYHUOHQJWK SDJH FKDUJHV FRXQWHG 2$ IHH LI \RXU PDQXVFULSW LV DFFHSWHG IRU SXEOLFDWLRQ ([FHSWLRQV WR PDQXVFULSW OHQJWK UHTXLUHPHQWV PD\ XQGHU H[WUDRUGL LQ RUGHU WR HQDEOH XQUHVWULFWHG SXEOLF DFFHVV $Q\ RWKHU DSSOLFDWLRQ QDU\ FLUFXPVWDQFHV EH JUDQWHG E\ WKH (GLWRULQ&KLHI +RZHYHU VXFK FKDUJHV VXFK DV RYHUOHQJWK SDJH FKDUJH DQGRU FKDUJH IRU WKH XVH RI H[FHSWLRQ GRHV QRW REYLDWH \RXU UHTXLUHPHQW WR SD\ DQ\ DQG DOO RYHU FRORU LQ WKH SULQW IRUPDW ZLOO EH ELOOHG VHSDUDWHO\ RQFH WKH PDQXVFULSW OHQJWK RU DGGLWLRQDO FKDUJHV WKDW DWWDFK WR WKH PDQXVFULSW IRUPDWWLQJ LV FRPSOHWH EXW SULRU WR WKH SXEOLFDWLRQ ,I \RX ZRXOG OLNH 5HVXEPLVVLRQ RI 3UHYLRXVO\ 5HMHFWHG 0DQXVFULSWV $XWKRUV RI PDQ \RXU PDQXVFULSW WR EH D 7UDGLWLRQDO VXEPLVVLRQ \RXU DUWLFOH ZLOO EH XVFULSWV UHMHFWHG IURP DQ\ MRXUQDO DUH DOORZHG WR UHVXEPLW WKHLU PDQX DYDLODEOH WR TXDOL¿HG VXEVFULEHUV DQG SXUFKDVHUV YLD ,((( ;SORUH 1R VFULSWV RQO\ RQFH $W WKH WLPH RI VXEPLVVLRQ \RX ZLOO EH DVNHG ZKHWKHU 2$ SD\PHQW LV UHTXLUHG IRU 7UDGLWLRQDO VXEPLVVLRQ \RXU PDQXVFULSW LV D QHZ VXEPLVVLRQ RU D UHVXEPLVVLRQ RI DQ HDUOLHU 3DJH &KDUJHV UHMHFWHG PDQXVFULSW ,I LW LV D UHVXEPLVVLRQ RI D PDQXVFULSW SUHYLRXVO\ UHMHFWHG E\ DQ\ MRXUQDO \RX DUH H[SHFWHG WR VXEPLW VXSSRUWLQJ GRFX 9ROXQWDU\ 3DJH &KDUJHV 8SRQ DFFHSWDQFH RI D PDQXVFULSW IRU SXEOLFD PHQWV LGHQWLI\LQJ WKH SUHYLRXV VXEPLVVLRQ DQG GHWDLOLQJ KRZ \RXU QHZ WLRQ WKH DXWKRU V RU KLVKHUWKHLU FRPSDQ\ RU LQVWLWXWLRQ ZLOO EH DVNHG YHUVLRQ DGGUHVVHV DOO RI WKH UHYLHZHUV¶ FRPPHQWV 3DSHUV WKDW GR QRW WR SD\ D FKDUJH RI SHU SDJH WR FRYHU SDUW RI WKH FRVW RI SXEOLFDWLRQ GLVFORVH FRQQHFWLRQ WR D SUHYLRXVO\ UHMHFWHG SDSHU RU WKDW GR QRW SUR RI WKH ¿UVW WHQ SDJHV WKDW FRPSULVH WKH VWDQGDUG OHQJWK WZR SDJHV LQ YLGH GRFXPHQWDWLRQ DV WR FKDQJHV PDGH PD\ EH LPPHGLDWHO\ UHMHFWHG WKH FDVH RI &RUUHVSRQGHQFHV $XWKRU 0LVFRQGXFW $XWKRU PLVFRQGXFW LQFOXGHV SODJLDULVP 0DQGDWRU\ 3DJH &KDUJHV 7KH DXWKRU V RU KLVKHUWKHLU FRPSDQ\ RU VHOISODJLDULVP DQG UHVHDUFK PLVFRQGXFW LQFOXGLQJ IDOVL¿FDWLRQ RU LQVWLWXWLRQ ZLOO EH ELOOHG SHU HDFK SDJH LQ H[FHVV RI WKH ¿UVW WHQ PLVUHSUHVHQWDWLRQ RI UHVXOWV $OO IRUPV RI PLVFRQGXFW DUH XQDFFHSWDEOH SXEOLVKHG SDJHV IRU UHJXODU SDSHUV DQG VL[ SXEOLVKHG SDJHV IRU FRUUH DQG PD\ UHVXOW LQ VDQFWLRQV DQGRU RWKHU FRUUHFWLYH DFWLRQV 3ODJLDULVP VSRQGHQFH LWHPV 127( 3DSHUV DFFHSWHG WR ,((( 75$16$&7,216 LQFOXGHV FRS\LQJ VRPHRQH HOVH¶V ZRUN ZLWKRXW DSSURSULDWH FUHGLW 21 08/7,0(',$ LQ H[FHVV RI SDJHV ZLOO EH VXEMHFW WR PDQGDWRU\ QGDWRU\ SDJH FKDUJHV DQG WKH XVLQJ VRPHRQH HOVH¶V ZRUN ZLWKRXW FOHDU GHOLQHDWLRQ RI FLWDWLRQ RYHUOHQJWK SDJH FKDUJHV 7KHVH DUH PD DXWKRU V ZLOO EH KHOG UHVSRQVLEOH IRU WKHP 7KH\ DUH QRW QHJRWLDEOH RU DQG WKH XQFLWHG UHXVH RI DQ DXWKRU¶V SUHYLRXVO\ SXEOLVKHG ZRUN WKDW DOVR LQYROYHV RWKHU DXWKRUV 6HOISODJLDULVP LQYROYHV WKH YHUEDWLP YROXQWDU\ 7KH DXWKRU V VLJQL¿HV KLV ZLOOLQJQHVV WR SD\ WKHVH FKDUJHV KHLU PDQXVFULSW WR WKH 75$16$&7,216 FRS\LQJ RU UHXVH RI DQ DXWKRUV RZQ SULRU ZRUN ZLWKRXW DSSURSULDWH VLPSO\ E\ VXEPLWWLQJ KLVKHUW 7KH 3XEOLVKHU KROGV WKH ULJKW WR ZLWKKROG SXEOLFDWLRQ XQGHU DQ\ FLUFXP FLWDWLRQ LQFOXGLQJ GXSOLFDWH VXEPLVVLRQ RI D VLQJOH MRXUQDO PDQXVFULSW WR WZR GLIIHUHQW MRXUQDOV DQG VXEPLVVLRQ RI WZR GLIIHUHQW MRXUQDO VWDQFH DV ZHOO DV SXEOLFDWLRQ RI WKH FXUUHQW RU IXWXUH VXEPLVVLRQV RI PDQGDWRU\ SDJH FKDUJH GHEW 1R PDQGD PDQXVFULSWV ZKLFK RYHUODS VXEVWDQWLDOO\ LQ ODQJXDJH RU WHFKQLFDO DXWKRUV ZKR KDYH RXWVWDQGLQJ WRU\ RYHUOHQJWK SDJH FKDUJHV ZLOO EH DSSOLHG WR RYHUYLHZ DUWLFOHV LQ WKH FRQWULEXWLRQ )RU PRUH LQIRUPDWLRQ RQ WKH GH¿QLWLRQV LQYHVWLJD WLRQ SURFHVV DQG FRUUHFWLYH DFWLRQV UHODWHG WR DXWKRU PLVFRQGXFW 6RFLHW\¶V MRXUQDOV VHH WKH 6LJQDO 3URFHVVLQJ 6RFLHW\ 3ROLFLHV DQG 3URFHGXUHV 0DQXDO &RORU &KDUJHV &RORU ¿JXUHV ZKLFK DSSHDU LQ FRORU RQO\ LQ WKH HOHF 6HFWLRQ ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJDERXWVSVJRY WURQLF ;SORUH YHUVLRQ FDQ EH XVHG IUHH RI FKDUJH ,Q WKLV FDVH WKH ______HUQDQFHSROLF\SURFHGXUHSDUW $XWKRU PLVFRQGXFW PD\ DOVR EH ¿JXUH ZLOO EH SULQWHG LQ WKH KDUGFRS\ YHUVLRQ LQ JUD\VFDOH DQG WKH DFWLRQDEOH E\ WKH ,((( XQGHU WKH UXOHV RI 0HPEHU &RQGXFW DXWKRU LV UHVSRQVLEOH WKDW WKH FRUUHVSRQGLQJ JUD\VFDOH ¿JXUH LV LQWHO OLJLEOH &RORU UHSURGXFWLRQ FKDUJHV IRU SULQW DUH WKH UHVSRQVLELOLW\ RI ([WHQVLRQV RI WKH $XWKRU¶V 3ULRU :RUN ,W LV DFFHSWDEOH IRU FRQIHU HQFH SDSHUV WR EH XVHG DV WKH EDVLV IRU D PRUH IXOO\ GHYHORSHG MRXUQDO WKH DXWKRU 'HWDLOV RI WKH DVVRFLDWHG FKDUJHV FDQ EH IRXQG RQ WKH ,((( H VXEPLVVLRQ 6WLOO DXWKRUV DUH UHTXLUHG WR FLWH WKHLU UHODWHG SULRU ZRUN 3XEOLFDWLRQV SDJ WKH SDSHUV FDQQRW EH LGHQWLFDO DQG WKH MRXUQDO SXEOLFDWLRQ PXVW LQFOXGH 3D\PHQW RI IHHV RQ FRORU UHSURGXFWLRQ LV QRW QHJRWLDEOH RU YROXQ VXEVWDQWLYHO\ QRYHO DVSHFWV VXFK DV QHZ H[SHULPHQWDO UHVXOWV DQG DQDO WDU\ DQG WKH DXWKRU¶V DJUHHPHQW WR SXEOLVK WKH PDQXVFULSW LQ WKHVH \VLV RU DGGHG WKHRUHWLFDO ZRUN 7KH MRXUQDO SXEOLFDWLRQ VKRXOG FOHDUO\ 75$16$&7,216 LV FRQVLGHUHG DFFHSWDQFH RI WKLV UHTXLUHPHQW VSHFLI\ KRZ WKH MRXUQDO SDSHU RIIHUV QRYHO FRQWULEXWLRQV ZKHQ FLWLQJ WKH SULRU ZRUN /LPLWHG RYHUODS ZLWK SULRU MRXUQDO SXEOLFDWLRQV ZLWK D FRPPRQ DXWKRU LV DOORZHG RQO\ LI LW LV QHFHVVDU\ IRU WKH UHDGDELOLW\ RI ,((( 75$16$&7,216 21 6,*1$/ 352&(66,1* WKH SDSHU DQG WKH SULRU ZRUN PXVW EH FLWHG DV WKH SULPDU\ VRXUFH ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVWVS763(',&6 6XEPLVVLRQ )RUPDW $XWKRUV DUH UHTXLUHG WR SUHSDUH PDQXVFULSWV ,((( 75$16$&7,216 21 ,0$*( 352&(66,1* HPSOR\LQJ WKH RQOLQH VW\OH ¿OHV GHYHORSHG E\ ,((( ZKLFK LQFOXGH ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVLPDJHSURFHVVLQJWLSHGLFV JXLGHOLQHV IRU DEEUHYLDWLRQV PDWKHPDWLFV DQG JUDSKLFV $OO PDQX ,((($&0 75$16$&7,216 21 $8',263((&+ $1' /$1*8$*( $&0 VFULSWV DFFHSWHG IRU SXEOLFDWLRQ ZLOO UHTXLUH WKH DXWKRUV WR PDNH ¿QDO ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVWDVOSWDVOSHGLFV VXEPLVVLRQ HPSOR\LQJ WKHVH VW\OH ¿OHV 7KH VW\OH ¿OHV DUH DYDLODEOH RQ ,((( 75$16$&7,216 21 ,1)250$7,21)25(16,&6 $1' 6(&85,7< WKH ZHE DW WKH ,((( $XWKRU 'LJLWDO 7RROER[ XQGHU ³7HPSODWH IRU DOO ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVIRUHQVLFVIRUHQVLFVHGLFV 75$16$&7,216´ /D7H; DQG 06 :RUG 3OHDVH QRWH WKH IROORZLQJ ,(((75$16$&7,216 21 08/7,0(',$ UHTXLUHPHQWV DERXW WKH DEVWUDFW ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJWPPWPPHGLFV 7KH DEVWUDFW PXVW EH D FRQFLVH \HW FRPSUHKHQVLYH UHÀHFWLRQ RI ,((( 75$16$&7,216 21 &20387$7,21$/ ,0$*,1* ZKDW LV LQ \RXU DUWLFOH ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVWFLWFLHGLFV 7KH DEVWUDFW PXVW EH VHOIFRQWDLQHG ZLWKRXW DEEUHYLDWLRQV IRRW ,((( 75$16$&7,216 21 6,*1$/ $1' ,1)250$7,21 352&(66,1* 29(5 1(7:25.6 QRWHV GLVSOD\HG HTXDWLRQV RU UHIHUHQFHV ______KWWSZZZVLJQDOSURFHVVLQJVRFLHW\RUJSXEOLFDWLRQVSHULRGLFDOVWVLSQWVLSQHGLFV

www.signalprocessingsociety.org [37] JANUARY 2017

THE WORLD’S NEWSSTAND®

,(((6,*1$/352&(66,1*62&,(7<0(0%(56+,3$33/,&$7,21 0DLOWR,(((23(5$7,216&(17(5$7710DWWKHZ3ORWQHU0HPEHUDQG*HRJUDSKLF$FWLYLWLHV+RHV/DQH3LVFDWDZD\1HZ-HUVH\86$ RU)D[WR FUHGLWFDUGSD\PHQWVRQO\ )RULQIRFDOO RU ,(((RU(PDLOQHZ______PHPEHUVKLS#LHHHRUJ 3(5621$/,1)250$7,21 1$0($6,76+28/'$33($521,(((0$,/,1*66(1'0$,/72+RPH$GGUHVV25%XVLQHVV6FKRRO$GGUHVV ,IQRWLQGLFDWHGPDLOZLOOEHVHQWWRKRPHDGGUHVV1RWH(QWHU\RXUQDPHDV\RXZLVKLWWRDSSHDURQPHPEHUVKLSFDUGDQGDOOFRUUHVSRQGHQFH 3/($6(35,17'RQRWH[FHHGFKDUDFWHUVRUVSDFHVSHUOLQH$EEUHYLDWHDVQHHGHG3OHDVHFLUFOH\RXUODVWVXUQDPHDVDNH\LGHQWLILHUIRUWKH,(((GDWDEDVH BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB 7,7/( ),56725*,9(11$0( 0,''/(1$0( 6851$0(/$671$0( BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB +20($''5(66 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB &,7< 67$7(3529,1&( 3267$/&2'( &28175< $UH\RXQRZRUZHUH\RXHYHUDPHPEHURI,((("

www.signalprocessingsociety.org [38] JANUARY 2017

THE WORLD’S NEWSSTAND®

,(((6,*1$/352&(66,1*62&,(7<678'(170(0%(56+,3$33/,&$7,21 &XUUHQWDQGUHLQVWDWLQJ,(((PHPEHUVMRLQLQJ636FRPSOHWHDUHDV 0DLOWR,(((23(5$7,216&(17(5$7710DWWKHZ3ORWQHU0HPEHUDQG*HRJUDSKLF$FWLYLWLHV+RHV/DQH3LVFDWDZD\1HZ-HUVH\86$ RU)D[WR FUHGLWFDUGSD\PHQWVRQO\ )RULQIRFDOO RU ,(((RU(PDLOQHZ______PHPEHUVKLS#LHHHRUJ 3(5621$/,1)250$7,21 1$0($6,76+28/'$33($521,(((0$,/,1*66(1'0$,/72+RPH$GGUHVV25%XVLQHVV6FKRRO$GGUHVV ,IQRWLQGLFDWHGPDLOZLOOEHVHQWWRKRPHDGGUHVV1RWH(QWHU\RXUQDPHDV\RXZLVKLWWRDSSHDURQPHPEHUVKLSFDUGDQGDOOFRUUHVSRQGHQFH 3/($6(35,17'RQRWH[FHHGFKDUDFWHUVRUVSDFHVSHUOLQH$EEUHYLDWHDVQHHGHG3OHDVHFLUFOH\RXUODVWVXUQDPHDVDNH\LGHQWLILHUIRUWKH,(((GDWDEDVH BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB 7,7/( ),56725*,9(11$0( 0,''/(1$0( 6851$0(/$671$0( BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB +20($''5(66 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB &,7< 67$7(3529,1&( 3267$/&2'( &28175< 636678'(170(0%(55$7(6$XJ)HE0DU$XJ $UH\RXQRZRUZHUH\RXHYHUDPHPEHURI,((("

www.signalprocessingsociety.org [39] JANUARY 2017

THE WORLD’S NEWSSTAND®

,(((6,*1$/352&(66,1*62&,(7<$)),/,$7(0(0%(56+,3$33/,&$7,21 0DLOWR,(((23(5$7,216&(17(5$7710DWWKHZ3ORWQHU0HPEHUDQG*HRJUDSKLF$FWLYLWLHV+RHV/DQH3LVFDWDZD\1HZ-HUVH\86$ RU)D[WR FUHGLWFDUGSD\PHQWVRQO\ )RULQIRFDOO RU ,(((RU(PDLOQHZ______PHPEHUVKLS#LHHHRUJ 3(5621$/,1)250$7,21 1$0($6,76+28/'$33($521,(((0$,/,1*66(1'0$,/72+RPH$GGUHVV25%XVLQHVV6FKRRO$GGUHVV ,IQRWLQGLFDWHGPDLOZLOOEHVHQWWRKRPHDGGUHVV1RWH(QWHU\RXUQDPHDV\RXZLVKLWWRDSSHDURQPHPEHUVKLSFDUGDQGDOOFRUUHVSRQGHQFH 3/($6(35,17'RQRWH[FHHGFKDUDFWHUVRUVSDFHVSHUOLQH$EEUHYLDWHDVQHHGHG3OHDVHFLUFOH\RXUODVWVXUQDPHDVDNH\LGHQWLILHUIRUWKH,(((GDWDEDVH BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB 7,7/( ),56725*,9(11$0( 0,''/(1$0( 6851$0(/$671$0( BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB +20($''5(66 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB &,7< 67$7(3529,1&( 3267$/&2'( &28175< $UH\RXQRZRUZHUH\RXHYHUDPHPEHURI,((("

THE WORLD’S NEWSSTAND®

THE WORLD’S NEWSSTAND®