DOCSLIB.ORG

Curriculum Vitae

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Curriculum Vitae Yisong Yue Contact California Institute of Technology website: www.yisongyue.com Information 1200 E. California Blvd. email: [email protected] CMS, 305-16 Pasadena, CA 91125 Research Theory and application of statistical machine learning, with a particular focus on developing Interests novel methods for interactive machine learning and structured machine learning. Research California Institute of Technology September 2014 - Present Appointments Position: Professor (May 2020 - Present) Previously: Assistant Professor (September 2014 - May 2020) Disney Research August 2013 - August 2014 Position: Research Scientist Carnegie Mellon University September 2010 - August 2013 Position: Postdoctoral Researcher Supervisors: Carlos Guestrin and Ramayya Krishnan Cornell University May 2006 - August 2010 Position: Research Assistant Supervisors: Thorsten Joachims and Robert Kleinberg Google June 2009 - September 2009 Position: Search Quality Analyst Intern Supervisor: Rajan Patel Microsoft Research May 2007 - August 2007 Position: Research Intern Supervisor: Christopher Burges Education Cornell University Ph.D. January 2011 Ph.D. in Computer Science Graduate Minor in Statistics Dissertation: New Learning Frameworks for Information Retrieval Thesis Committee: Thorsten Joachims (advisor), Robert Kleinberg, Christopher Burges, Ping Li, John Hopcroft University of Illinois at Urbana-Champaign B.S. June 2005 Bachelor of Science in Computer Science Graduated with Highest Honors (Summa Cum Laude) Illinois Math and Science Academy 1998 - 2001 Honors and Okawa Foundation Grant Recipient, 2018 Awards Best Reviewer, ICLR 2018 Best Paper Award, ICRA 2020 Best Student Paper Award, CVPR 2021 Best Paper Nomination, WSDM 2011, ICDM 2014, SSAC 2017, R-AL 2020 Microsoft Research Graduate Fellowship, 2008-2010 Google Student Award Winner, NYAS Machine Learning Symposium, 2009 Yahoo! Key Scientific Challenges Award, 2008 Advising Postdocs Supervised • Uriah Israel, active • Kamyar Azizzadenesheli, 2019-2020, Assistant Professor at Purdue • Yuxin Chen, 2017-2019, Assistant Professor at University of Chicago • Angie Liu, 2018-2021, Assistant Professor at Johns Hopkins Univeresity • Taehwan Kim, 2015-2017, Assistant Professor at UNIST • Ugo Rosolia, 2020-2021, Research Scientist at Amazon • Yanan Sui, 2016-2018, Assistant Professor at Tsinghua University • Romann Weber, 2015-2016, Research Scientist at Disney Research • Rose Yu, 2017-2018, Assistant Professor at UCSD Ph.D. Students Advised • Jeremy Bernstein, California Institute of Technology, active • Victor Dorobantu, California Institute of Technology, active • Alex Farhang, California Institute of Technology, active • Ivan Jimenez Rodriguez, California Institute of Technology, active • Amy Kejun Li, California Institute of Technology, active • Hao Liu, California Institute of Technology, active • Guanya Shi, California Institute of Technology, active • Jennifer Sun, California Institute of Technology, active • Sabera Talukder, California Institute of Technology, active • Cameron Voloshin, California Institute of Technology, active • Christopher Yeh, California Institute of Technology, active • Eric Zhan, California Institute of Technology, active • Hoang Le, California Institute of Technology, Ph.D. 2019, Postdoctoral Researcher at Microsoft Research • Joseph Marino, California Institute of Technology, Ph.D. 2021, Research Scientist at DeepMind • Ellen Novoseller, California Institute of Technology, Ph.D. 2020, Postdoctoral Re- searcher at UC Berkeley • Jialin Song, California Institute of Technology, Ph.D. 2021, Research Scientist at NVIDIA AI • Stephan Zheng, California Institute of Technology, Ph.D. 2018, Research Scientist at Salesforce AI Teaching Machine Learning & Data Mining. Core machine learning and data mining course offered to graduate students and advanced undergraduates. Taught at Caltech: Winter 2015, Winter 2016, Winter 2017, Winter 2018, Winter 2019, Winter 2020. Advanced Topics in Machine Learning. Advanced course on contemporary research topics in machine learning. Taught at Caltech: Spring 2016, Spring 2017, Spring 2018, Spring 2019, Spring 2020, Spring 2021. Projects in Machine Learning. Project-based course matching students to mentors on projects of mutual interest. Taught at Caltech: Fall 2016, Fall 2017, Winter 2018, Fall 2018, Winter 2019, Fall 2019, Winter 2020. Tutorials • \Imitation Learning." co-taught with Hoang M. Le, ICML 2018 Tutorial, Stockholm, Sweden, July 2018. • \Practical Online Retrieval Evaluation." co-taught with Filip Radlinski, SIGIR 2011 Tutorial, Beijing, China, July 2011. • \Learning to Rank." co-taught with Filip Radlinski, NESCAI 2008 Tutorial, Ithaca, NY, May 2008 • \An Introduction to Structured Output Learning Using Support Vector Machines." Microsoft Research Web Learning Group, Redmond, WA, August 2007. Professional Organizing Committee Activites • Fundraising Chair, AISTATS 2016 Journal Reviewing • Data Mining and Knowledge Discovery • Information Processing & Management • Information Retrieval • Journal of Artificial Intelligence Research • Neural Networks • Transactions on Knowledge and Data Engineering • Transactions on the Web Conference Reviewing • AAAI 2014, 2015, 2017 (SPC), 2020 (SPC) • ACL 2012 • ACML 2011, 2012, 2014 • AISTATS 2019 (SPC) • CIKM 2012 • COLING 2010, 2014 • COLT 2015 • ECML/PKDD 2008 • EMNLP 2011, 2012 • ICLR 2018, 2019, 2020 • ICML 2007, 2008, 2009, 2010, 2011, 2012, 2013 (AC), 2014, 2016 (AC), 2017 (AC), 2018 (AC), 2019 (AC), 2020 (AC), 2021 (SAC) • IJCAI 2016 (SPC), 2019 (SPC) • KDD 2011, 2015 (SPC), 2016 (SPC), 2017 (SPC) • L4DC 2021 • NAACL-HLT 2012, 2013 • NeurIPS 2008, 2009, 2010, 2011, 2012, 2014, 2015, 2016, 2017, 2018 (AC), 2019 (AC), 2020 (SAC), 2021 (SAC) • SIGIR 2008, 2009, 2010, 2013, 2014 • SoCG 2010 • UAI 2020 (AC) • UBICOMP 2014 • UIST 2015 • WSDM 2011, 2012, 2013, 2014, 2015, 2016 (SPC) • WWW 2011, 2012, 2013, 2014, 2017 Book Reviewing & Editing • Introduction to Information Retrieval, Chapter 18, Matrix decompositions & latent semantic indexing Other Multi-Agent Behavior Modeling Workshop, @CVPR 2021 Service Learning Meets Combinatorial Algorithms Workshop, @NeurIPS 2020 Real-World Experiment Design and Active Learning Workshop, @ICML 2020 Safety and Robustness in Decision Making Workshop, @NeurIPS 2019 Real-world Sequential Decision Making Worksop, @ICML 2019 AI for Science Workshop, @Caltech, 2018, 2019 Southern California Machine Learning Symposium, @Caltech, November 2016 Large-Scale Sports Analytics Workshop, @KDD 2014, @KDD 2015, @KDD 2016 Personalization Workshop, @NeurIPS 2014, @ICML 2016 Invited Thorsten Joachims, Thomas Hofmann, Yisong Yue, Chun-Nam Yu. (2009) \Predicting Articles Structured Objects with Support Vector Machines." Communications of the ACM (CACM), Research Highlight, 52(11), 97{104, November 2009. Journal Guanya Shi, Wolfgang Hnig, Xichen Shi, Yisong Yue, Soon-Jo Chung. (2021) \Neural- Papers Swarm2: Planning and Control of Heterogeneous Multirotor Swarms using Learned Inter- actions." IEEE Transactions on Robotics (T-RO), 2021. Bruce Wittmann, Yisong Yue, Frances Arnold. (2021) \Machine Learning-Assisted Directed Evolution Navigates a Combinatorial Epistatic Fitness Landscape with Minimal Screening Burden." Cell Systems, August 2021. Andrew J. Taylor, Victor D. Dorobantu, Yisong Yue, Paulo Tabuada, Aaron D. Ames. (2021) \Sampled-Data Stabilization with Control Lyapunov Functions via Quadratically Constrained Quadratic Programs." IEEE Control Systems Letters (L-CSS), June 2021. Yidan Qin, Max Allan, Yisong Yue, Joel Burdick, Mahdi Azizian. (2021) \Learning Invari- ant Representation of Tasks for Robust Surgical State Estimation." IEEE Robotics and Automation Letters (RA-L), 2021. Yashwanth Kumar Nakka, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, Soon- Jo Chung. (2021) \Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems." IEEE Robotics and Automation Letters (RA-L), April, 2021. Logan Cross, Jeff Cockburn, Yisong Yue, John ODoherty. (2020) \Using deep reinforce- ment learning to reveal how the brain encodes abstract state-space representations in high- dimensional environments." Neuron, February, 2021. Michael R. Maser, Alexander Y. Cui, Serim Ryou, Travis J. DeLano, Yisong Yue, and Sarah E. Reisman. (2021) \Multilabel Classification Models for the Prediction of Cross-Coupling Reaction Conditions." Journal of Chemical Information and Modeling (JCIM), January, 2021. Andrew J. Taylor, Andrew Singletary, Yisong Yue, Aaron D. Ames. (2020) \A Control Barrier Perspective on Episodic Learning via Projection-to-State Safety." In IEEE Control Systems Letters (L-CSS), July 2020. Benjamin Rivire, Wolfgang Hoenig, Yisong Yue, Soon-Jo Chung. (2020) \GLAS: Global-to- Local Safe Autonomy Synthesis for Multi-Robot Motion Planning with End-to-End Learn- ing." IEEE Robotics and Automation Letters, June, 2020. Zachary Ross, Yisong Yue, Men-Andrin Meier, Egill Hauksson, Thomas Heaton. (2019) \PhaseLink: A Deep Learning Approach to Seismic Phase Association." Journal of Geo- physical Research - Solid Earth, DOI:0.1029/2018JB016674R, January, 2019. Men-Andrin Meier, Zachary Ross, Anshul Ramachandran, Ashwin Balakrishna, Suraj Nair, Peter Kundzicz, Zefeng Li, Jennifer Andrews, Egill Hauksson, Yisong Yue. (2019) \Reli- able Real-time Seismic Signal/Noise Discrimination with Machine Learning." Journal of Geophysical Research - Solid Earth, DOI:0.1029/2018JB016661, January, 2019. Long Sha, Patrick Lucey, Yisong Yue, Xinyu Wei, Jennifer Hobbs, Charlie Rohlf,
Load more

Recommended publications
  • Anima Anandkumar
    Anima Anandkumar Director of Machine Learning Research, NVIDIA "Anima is at the forefront of AI, as both a theorist and a praconer" Anima Anandkumar holds dual posions in academia and industry. She is a Bren professor at Caltech CMS department and a director of machine learning research at NVIDIA. At NVIDIA, she is leading the research group that develops next-generaon AI algorithms. At Caltech, she is the co-director of Dolcit and co-leads the AI4science iniave. TOPICS: IN DETAIL: AI Security Anima is a former Principal Scienst at Amazon Web Services (AWS) where she Artificial Intelligence was head of research and development providing enterprises with arficial Big Data intelligence tools. Anima is also a champion for enabling AI methods and tools in Empowering Women every part of the cloud and for the kind of progressive, incremental Futurism Technology improvements in performance that can make a big difference over me - the Diversity in Technology same kind of processes with which machines learn. Anima holds the endowed Machine Learning Bren chair at CalTech. She has received numerous awards and honors, including a Can Artificial Intelligence be Conscious Microso Faculty Fellowship, a Google faculty award, and a number of other too? research and best paper awards. She has been featured in several documentaries Bridging the Gap Between Artificial and and media arcles. Human Intelligence WHAT SHE OFFERS YOU: LANGUAGES: Anima Anandkumar offers that rare combinaon of leading-edge academic She presents in English. research with equally deep experience in business and praccal applicaons. She specialises in using AI in cloud compung to solve real world problems.
    [Show full text]
  • Tensor Regression Networks
    Journal of Machine Learning Research 21 (2020) 1-21 Submitted 7/18; Published 7/20 Tensor Regression Networks Jean Kossaifi [email protected] NVIDIA & Imperial College London Zachary C. Lipton [email protected] Carnegie Mellon University Arinbj¨ornKolbeinsson [email protected] Imperial College London Aran Khanna [email protected] Amazon AI Tommaso Furlanello [email protected] University of Southern California Anima Anandkumar [email protected] NVIDIA & California Institute of Technology Editor: Aapo Hyvarinen Abstract Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy.
    [Show full text]
  • Deep Learning in Unconventional Domains
    Deep Learning in Unconventional Domains Thesis by Milan Cvitkovic In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2020 Defended March 10, 2020 ii © 2020 Milan Cvitkovic ORCID: 0000-0003-4188-452X All rights reserved iii ACKNOWLEDGEMENTS To my parents, Kate Turpin and Michael Cvitkovic, and my sister, Adriana Cvitkovic: thank you for your boundless love and support. And all the teasing. To my thesis committee, Prof. Anima Anandkumar, Prof. Yisong Yue, Prof. Adam Wierman, and Prof. Thomas Vidick: thank you for your invaluable guidance and encouragement throughout my time at Caltech, both in your roles as committee members and beyond. To Prof. Anima Anandkumar, Prof. Pietro Perona, and Prof. Stefano Soatto: thank you for facilitating my affiliation with Amazon Web Services, without which this dissertation would never have been possible. To all my coauthors who helped with the work in this thesis — Badal Singh, Prof. Anima Anandkumar, Dr. Günther Koliander, and Dr. Zohar Karnin — and to all my other collaborators during my PhD — Joe Marino, Dr. Grant Van Horn, Dr. Hamed Alemohammad, Prof. Yisong Yue, Keng Wah Loon, Laura Graesser, Jiahao Su, and Prof. Furong Huang: thank you for your generosity and your brilliance. To Lee, Carmen, Kirsten, Leslie, James, Alex, Josh, Claudia, and Patricia: thank you for being cheerful stewards of daily rituals. And most of all to my advisor, Prof. Thomas Vidick: thank you for your unwavering support of a non-traditional advisee. iv ABSTRACT Machine learning methods have dramatically improved in recent years thanks to advances in deep learning (LeCun, Bengio, and Hinton, 2015), a set of methods for training high-dimensional, highly-parameterized, nonlinear functions.
    [Show full text]
  • Anima Anandkumar
    ANIMA ANANDKUMAR DISTRIBUTED DEEP LEARNING PRACTICAL CONSIDERATIONS FOR MACHINE LEARNING SOFTWARE PACKAGES TITLE OF SLIDE UTILITY DIVERSE & COMPUTING LARGE DATASETS CHALLENGES IN DEPLOYING LARGE-SCALE LEARNING TITLE OF SLIDE CHALLENGES IN DEPLOYING LARGE-SCALE LEARNING • Complex deep network TITLE OF SLIDE • Coding from scratch is impossible • A single image requires billions floating-point operations • Intel i7 ~500 GFLOPS • Nvidia Titan X: ~5 TFLOPS • Memory consumption is linear with number of layers DESIRABLE ATTRIBUTES IN A ML SOFTWARE PACKAGE {…} TITLE OF SLIDE PROGRAMMABILITY PORTABILITY EFFICIENCY Simplifying network Efficient use In training definitions of memory and inference TensorFlow TITLE OF SLIDE Torch CNTK MXNET IS AWS’S DEEP LEARNING TITLE OF SLIDE FRAMEWORK OF CHOICE MOST OPEN BEST ON AWS Apache (Integration with AWS) {…} {…} PROGRAMMABILITY frontend backend TITLE OF SLIDE single implementation of performance guarantee backend system and regardless which front- common operators end language is used IMPERATIVE PROGRAMMING PROS • Straightforward and flexible. import numpy as np a = np.ones(10) • Take advantage of language b = np.ones(10) * 2 native features (loop, c = b * a condition, debugger) TITLE OF SLIDE cd = c + 1 • E.g. Numpy, Matlab, Torch, … Easy to tweak CONS with python codes • Hard to optimize DECLARATIVE PROGRAMMING A = Variable('A') PROS B = Variable('B') • A B More chances for optimization C = B * A • Cross different languages D = C + 1 X 1 f = compile(D) • E.g. TensorFlow, Theano, + TITLE OF SLIDE d = f(A=np.ones(10),
    [Show full text]
  • Communication-Efficient Distributed SGD with Sketching
    Communication-efficient Distributed SGD with Sketching Nikita Ivkin ∗y Daniel Rothchild ∗ Enayat Ullah ∗ Amazon UC Berkeley Johns Hopkins University [email protected] [email protected] [email protected] Vladimir Braverman z Ion Stoica Raman Arora Johns Hopkins University UC Berkeley Johns Hopkins University [email protected] [email protected] [email protected] Abstract Large-scale distributed training of neural networks is often limited by network band- width, wherein the communication time overwhelms the local computation time. Motivated by the success of sketching methods in sub-linear/streaming algorithms, we introduce SKETCHED-SGD4, an algorithm for carrying out distributed SGD by communicating sketches instead of full gradients. We show that SKETCHED-SGD has favorable convergence rates on several classes of functions. When considering all communication – both of gradients and of updated model weights – SKETCHED- SGD reduces the amount of communication required compared to other gradient compression methods from O(d) or O(W ) to O(log d), where d is the number of model parameters and W is the number of workers participating in training. We run experiments on a transformer model, an LSTM, and a residual network, demonstrating up to a 40x reduction in total communication cost with no loss in final model performance. We also show experimentally that SKETCHED-SGD scales to at least 256 workers without increasing communication cost or degrading model performance. 1 Introduction Modern machine learning training workloads are commonly distributed across many machines using data-parallel synchronous stochastic gradient descent. At each iteration, W worker nodes split a mini-batch of size B; each worker computes the gradient of the loss on its portion of the data, and then a parameter server sums each worker’s gradient to yield the full mini-batch gradient.
    [Show full text]
  • View a List of the Invited Faculty Members
    COURSE DIRECTORS Michael Wallace, MD, MPH, FASGE Prateek Sharma, MD, FASGE Professor of Medicine Professor of Medicine Gastroenterology and Hepatology Gastroenterology and Hepatology Mayo Clinic University of Kansas School of Medicine Jacksonville, Florida Kansas City, Kansas FACULTY Omer Ahmad, BSc, MBBS, MRCP Philip Chiu, MD, MBChB, FRCSEd, FCSHK, FHKAM Clinical Research Fellow (Surgery) Wellcome/EPSRC Centre for Interventional and Professor and Head of Division of Upper Surgical Sciences (WEISS) Gastrointestinal & Metabolic Surgery Surgical Robot Vision Research Group University College London Department of Surgery, The Chinese University of London, England Hong Kong Director of Endoscopy, Prince of Wales Hospital Anima Anandkumar, PhD Hong Kong Bren Professor Microsoft and Sloan Fellow Jonathan Cohen, MD, FASGE California Institute of Technology Clinical Professor Director of Machine Learning, NVIDIA New York University School of Medicine Irvine, California New York, New York David Armstrong, MD Giovanni Di Napoli, MBA Associate Professor President, Gastrointestinal McMaster University Medtronic, Inc. Ontario, Canada Sunnyvale, California Ulas Bagci, PhD Seth Gross, MD, FASGE SAIC Chair Professor Associate Professor, Department of Medicine Center for Research in Computer Vision Director, Clinical Care and Quality University of Central Florida Division of Gastroenterology Orlando, Florida NYU Langone Health New York, New York Tyler Berzin, MD, FASGE Co-Director, GI Endoscopy Aashima Gupta, MS Beth Israel Deaconess Medical Center Director, Global Healthcare Strategy and Solutions Director, Advanced Therapeutic Endoscopy Google Healthcare Cloud Fellowship San Francisco, California Assistant Professor of Medicine Harvard Medical School Cesare Hassan, MD Boston, Massachusetts Endoscopy Unit Nuovo Regina Margherita Hospital Rahul Bhotika, PhD Rome, Italy Amazon Computer Vision Director Shani Haugen, PhD Seattle, Washington Assistant Director, Gastroenterology and Endoscopy Devices Team (THT3A2) Anthony Borelli, MBA U.S.
    [Show full text]
  • Riseandgrind: Lessons from a Biased AI Conor Mcgarrigle Dublin School of Creative Arts, TU Dublin
    RADICAL IMMERSIONS: DRHA 2019 ---------------------------------------------------- #Riseandgrind: Lessons From a Biased AI Conor McGarrigle Dublin School of Creative Arts, TU Dublin Abstract: #RiseandGrind is a research-based artwork that, through a process of active engagement with the machine-learning tools of what is known as artificial intelligence, sought to make visible the complex relationship between the origins and context of training data and the results that are produced through the training process. The project using textual data extracted from Twitter hashtags that exhibit clear bias to train a recurrent neural network (RNN) to generate text for a Twitter bot, with the process of training and text generation represented in a series of gallery installations. The process demonstrated how original bias is consolidated, amplified, and ultimately codified through this machine learning process. It is suggested that this is not only reproductive of the original bias but also constitutive, in that black- box machine learning models shape the output but not in ways that are readily apparent or understood. This paper discusses the process of creating and exhibiting the work and reflects on its outcomes. Keywords: Twitter; Machine-learning; artificial intelligence; new media art; generative art; 1. Introduction #RiseandGrind, is an AI art project using textual data extracted from Twitter hashtags to train a recurrent neural network (RNN) to generate text for a Twitter bot, with the process of training and text generation represented in a series of gallery installations. The work was first commissioned in 2018 and exhibited in three different iterations in 2018 and 2019. The neural network was trained on two selected hashtags, #RiseandGrind and #Hustle, specifically chosen as representative of a Twitter filter bubble that I identify as embodied neoliberal precarity.
    [Show full text]
  • Lecture 17: March 16Th 2019 17.1 Introduction
    CS 165: Foundations of Machine Learning Winter 2019 Lecture 17: March 16th 2019 Lecturer: Anima Anandkumar Scribes: Qilin Li, Wen Gu We will label each section with the corresponding slides page number in () for your reference. 17.1 Introduction 17.1.1 Overview (2 ∼ 8) Machine learning, from a toy example to the applications in real life, has a broader implication in terms of bias, fairness, ethics and all the tricky issues. The mathematical model, data set used, and design of architecture for machine learning are all considered in ideal cases, but the things will change once they are brought to real life. In fact, there are more questions than answers regarding the broader implications rather than the traditional topic covered in the area of machine learning. What's more, the technology may not be able to reduce bias and prejudice as we expect. Our history, bias, and prejudices will impact the way we create the data sets. These biased data sets will be used to train the AI models, and the biased AI models will affect the way they make decisions. Consequently, these decisions will also influence the human's social issues such as bias and prejudices. This closed loop amplifies the social problems when we do not take consideration of the side effect of technology. For example, the ImageNet data set contains a lot of dogs and cats, but this data set cannot represent the whole picture of the real world in the sense that we have many things other than dogs and cats. In addition, modern technology is led by western civilization, and thus the value holds by western culture would also impact the way data set is created and the way technology is used.
    [Show full text]
  • Guaranteed Training of Neural Networks Using Tensor Methods
    Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods Majid Janzamin∗ Hanie Sedghi† Anima Anandkumar‡ Abstract Training neural networks is a challenging non-convex optimization problem, and backpropa- gation or gradient descent can get stuck in spurious local optima. We propose a novel algorithm based on tensor decomposition for guaranteed training of two-layer neural networks. We pro- vide risk bounds for our proposed method, with a polynomial sample complexity in the relevant parameters, such as input dimension and number of neurons. While learning arbitrary tar- get functions is NP-hard, we provide transparent conditions on the function and the input for learnability. Our training method is based on tensor decomposition, which provably converges to the global optimum, under a set of mild non-degeneracy conditions. It consists of simple embarrassingly parallel linear and multi-linear operations, and is competitive with standard stochastic gradient descent (SGD), in terms of computational complexity. Thus, we propose a computationally efficient method with guaranteed risk bounds for training neural networks with one hidden layer. Keywords: neural networks, risk bound, method-of-moments, tensor decomposition 1 Introduction Neural networks have revolutionized performance across multiple domains such as computer vision and speech recognition. They are flexible models trained to approximate any arbitrary target function, e.g., the label function for classification tasks. They are composed of multiple layers of neurons or activating functions, which are applied recursively on the input data, in order to predict the output. While neural networks have been extensively employed in practice, a complete theoretical understanding is currently lacking.
    [Show full text]
  • Conference on Learning Theory 2016: Preface
    JMLR: Workshop and Conference Proceedings vol 49:1–3, 2016 Conference on Learning Theory 2016: Preface Vitaly Feldman [email protected] IBM Research – Almaden Alexander Rakhlin [email protected] Department of Statistics, University of Pennsylvania These proceedings contain the 63 papers accepted to and presented at the 29th Conference on Learning Theory (COLT), held in New York, USA on June 23-26, 2016. These papers were selected by the program committee with additional help from external expert reviewers from 199 submissions. Of the 63 papers, 18 were given a 20-minute presentation, the remaining 45 a 10- minute presentation. These proceedings also contain the 6 submissions to the Open Problems session. Selection of the submissions was handled by the Open Problem Chair, Shai Shalev-Shwartz. In addition to the papers and open problems published in these proceedings, the conference program also included two invited talks: “Randomized Algorithms in Linear Algebra” by Ravi Kannan (Microsoft Research), and “Testing properties of distributions over big domains” by Ronitt Rubinfeld (MIT and Tel-Aviv University). The paper “Multi-scale Exploration of Convex Functions and Bandit Convex Optimization” by Sebastien´ Bubeck and Ronen Eldan received the Best Paper Award, and the paper “Provably Manipulation-resistant Reputation Systems” by Paul Christiano received the Best Student Paper Award. The local arrangements chairs were Daniel Hsu and Satyen Kale, and the publication chair was Ohad Shamir. We would like to express our gratitude to the entire program committee and to the external reviewers for their invaluable contributions to the success of conference. Finally, we would like to thank our generous sponsors: Two Sigma; Engineers Gate; Microsoft; Baidu Research; Facebook, Columbia University Data Science Institute; Google; Amazon; and Springer’s Machine Learning Journal.
    [Show full text]
  • Neural Operator: Graph Kernel Network for Partial Differential Equations
    Neural Operator: Graph Kernel Network for Partial Differential Equations Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar March 10, 2020 Abstract The classical development of neural networks has been primarily for mappings between a finite-dimensional Euclidean space and a set of classes, or between two finite-dimensional Euclidean spaces. The purpose of this work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators). The key innovation in our work is that a single set of network parameters, within a carefully designed network architecture, may be used to describe mappings between infinite-dimensional spaces and between different finite- dimensional approximations of those spaces. We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. The kernel integration is computed by message passing on graph networks. This approach has substantial practical consequences which we will illustrate in the context of mappings between input data to partial differential equations (PDEs) and their solutions. In this context, such learned networks can generalize among different approximation methods for the PDE (such as finite difference or finite element methods) and among approximations corresponding to different underlying levels of resolution and discretization. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers. 1 INTRODUCTION There are numerous applications in which it is desirable to learn a mapping between Banach spaces. In particular, either the input or the output space, or both, may be infinite-dimensional.
    [Show full text]
  • Fetchsgd: Communication-Efficient Federated Learning with Sketching
    FetchSGD: Communication-Efficient Federated Learning with Sketching 1 1 2 3 1 2 Daniel Rothchild∗ Ashwinee Panda∗ Enayat Ullah Nikita Ivkin Ion Stoica Vladimir Braverman Joseph Gonzalez 1 Raman Arora 2 Abstract 2018). Even without these concerns, training machine learn- ing models in the cloud can be expensive, and an effective Existing approaches to federated learning suf- way to train the same models on the edge has the potential fer from a communication bottleneck as well as to eliminate this expense. convergence issues due to sparse client participa- tion. In this paper we introduce a novel algorithm, When training machine learning models in the federated called FetchSGD, to overcome these challenges. setting, participating clients do not send their local data to FetchSGD compresses model updates using a a central server; instead, a central aggregator coordinates Count Sketch, and then takes advantage of the an optimization procedure among the clients. At each it- mergeability of sketches to combine model up- eration of this procedure, clients compute gradient-based dates from many workers. A key insight in the updates to the current model using their local data, and they design of FetchSGD is that, because the Count communicate only these updates to a central aggregator. Sketch is linear, momentum and error accumu- A number of challenges arise when training models in the lation can both be carried out within the sketch. federated setting. Active areas of research in federated learn- This allows the algorithm to move momentum ing include solving systems challenges, such as handling and error accumulation from clients to the central stragglers and unreliable network connections (Bonawitz aggregator, overcoming the challenges of sparse et al., 2016; Wang et al., 2019), tolerating adversaries (Bag- client participation while still achieving high com- dasaryan et al., 2018; Bhagoji et al., 2018), and ensuring pression rates and good convergence.
    [Show full text]