Dept. of Computer Science Phone: +1 (212) 998 3353

Rob Fergus

Dept. of Computer Science Phone: +1 (212) 998 3353 The Courant Institute of Mathematical Sciences Room 1226, 715 Broadway, Email: [email protected] New York, NY 10003, USA Web:http://cs.nyu.edu/~fergus

CURRENT POSITION

Assistant Professor of Computer Science 2007-present The Courant Institute of Mathematical Sciences New York University

PREVIOUS POSITIONS

Postdoctoral Research Associate 2005-2007 CSAIL, Massachusetts Institute of Technology Advisor: Professor William T. Freeman

EDUCATION

University of Oxford, UK 2002-2005 D.Phil. in Electrical Engineering, October 2005 Thesis title: Visual Object Category Recognition Advisor: Professor Andrew Zisserman

California Institute of Technology, Pasadena, CA 2000-2002 M.Sc. in Electrical Engineering, June 2002 Advisor: Professor Pietro Perona

University of Cambridge, UK 1996-2000 Pembroke College B.A., M.Eng. in Electrical and Information Engineering

RESEARCH INTERESTS

Computer Vision , Computational Photography, Machine Learning.

HONORS & AWARDS

NSF Career Award (Recommended) 2012 Joint 1st place, PASCAL VOC 2011 Detection Competition (with L. Zhu et al.) 2011 Sloan Research Fellowship 2011 Best Computer Science PhD thesis in UK, British Computer Society 2006 Best Computer Vision PhD thesis in UK, British Machine Vision Association 2006 Best Short Course Prize, IEEE International Conference on Computer Vision 2005 Best Paper Prize, IEEE Conf. on Computer Vision and Pattern Recognition 2003

PUBLICATIONS

Refereed conference papers

[R23] , Zeiler, M., Taylor, G., Sigal, L., Matthews, I. and Fergus, R., “Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines”, Proc. Neural Information Processing Systems 2011.

[R22] , Silberman, N. and Fergus, R., “Indoor Scene Segmentation using a Structured Light Sensor”, Workshop on 3D Representation and Recognition, Proc. of ICCV 2011.

[R21] , Zeiler, M., Taylor, G. and Fergus, R., “Adaptive Deconvolutional Networks for Mid and High Level Feature Learning”, Proc. of the International Conference on Computer Vision (ICCV), 2011.

[R20] Krishnan, D., Tay, T. and Fergus, R., “Blind Deconvolution using a Normalized Sparsity Measure”, Proc. of the IEEE Conf on Computer Vision and Pattern Recognition 2011.

[R19] Taylor, G., Spiro, I., Bregler, C. and Fergus, R., “Learning Invariance Through Imitation”, Proc. of the IEEE Conf on Computer Vision and Pattern Recognition 2011.

[R18] Taylor, G., Fergus, R., Sprio, I., Williams, G. and Bregler, C., “Pose-sensitive embedding by non-linear NCA”, Proc. Neural Information Processing Systems 2010.

[R17] Fergus, R., Bernal, H., Weiss, Y. and Torralba, A., “Semantic Label Sharing for Learning with Many Categories”, Proc. of the IEEE European Conference on Computer Vision 2010.

[R16] Taylor, G., Fergus, R., LeCun, Y. and Bregler, C., “Convolutional Learning of Spatio- Temporal Features”, Proc. of the IEEE European Conference on Computer Vision 2010.

[R15] , Zeiler, M., Krishnan, D., Taylor, G. and Fergus, R., “Deconvolutional Networks”,Proc. of the IEEE Conf on Computer Vision and Pattern Recognition 2010.

[R14] Silberman, N., Ahlrich, K., Fergus, R. and Subramanian, L., “Case for Automated Detection of Diabetic Retinopathy”, in AAAI Spring Symposium, 2010.

[R13] Krishnan, D. and Fergus, R., “Fast Image Deconvolution using Hyper-Laplacian Priors”, Proc. Neural Information Processing Systems 2009. [R12] Fergus, R., Weiss, Y. and Torralba, A., “Semi-Supervised Learning in Gigantic Image Collections”, Proc. Neural Information Processing Systems 2009. Oral presentation, (Top 2% submissions).

[R11] Kavukcuoglu, K., Ranzato, M., Fergus, R. and LeCun, Y. , “Learning Invariant Features through Topographic Filter Maps”, Proc. of the IEEE Conf on Computer Vision and Pattern Recognition 2009.

[R10] Weiss, Y., Torralba, A. and Fergus, R. , “Spectral Hashing”, Proc. Neural Information Processing Systems 2008.

[R9] Torralba, A. , Fergus, R. and Weiss, Y. , “Small Codes and Large Image Databases for Recognition”, Proc. of the IEEE Conf on Computer Vision and Pattern Recognition 2008.

[R8] Russell, B. , Torralba, A. , Liu, C. , Fergus, R. and Freeman, W.T. , ““Object Recognition by Scene Alignment” Proc. Neural Information Processing Systems 2007.

[R7] Fergus, R., Fei-Fei L., Perona, P. and Zisserman, A., “Learning Object Categories from Google's Image Search”, Proc. of the International Conference on Computer Vision (ICCV), Vol. 2, pp. 1816-1823, 2005.

[R6] Fergus, R., Perona, P. and Zisserman, A., “A Sparse Object Category Model for Efficient Learning and Exhaustive Recognition”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp. 380-387, 2005.

[R5] Fergus, R., Zisserman, A. and Perona, P., “Sampling Methods for Unsupervised Learning”, Advances in Neural Information Processing Systems (NIPS), pp. 433-440, 2004.

[R4] Fei-Fei, L., Fergus, R. and Perona, P., “Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories”, IEEE CVPR Workshop of Generative Model Based Vision, 2004.

[R3] Fergus, R., Perona, P. and Zisserman, A., “A Visual Category Filter for Google Images”, Proc. of the 8th European Conference on Computer Vision (ECCV), pp. 242-256, 2004.

[R2] Fei-Fei, L., Fergus, R. and Perona, P., “A Bayesian Approach to Unsupervised One-Shot Learning of Object Categories”, Proc. of the IEEE International. Conference on Computer Vision (ICCV), pp. 1134-1141, 2003.

[R1] Fergus, R., Perona, P. and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2, pp. 264-271, 2003. Awarded best paper prize out of ~1000 submissions. [1436 citations]. Journal papers

[J8] Fergus, R. , Fei-Fei L. , Perona, P. and Zisserman, A., “Learning Object Categories from Internet Image Searches”, Proc. of IEEE, Vol. 98, No. 8, Special Issue on Internet Vision, August 2010.

[J7] Krishnan, D. and Fergus, R., “Dark Flash Photography”, Vol. 28, Issue 3, ACM Trans. on Graphics (Proc. SIGGRAPH) 2009.

[J6] Torralba, A., Fergus, R. and Freeman, W. T., “80 Million Tiny Images: A Large Dataset for Nonparameteric Object and Scene Recognition”, IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), Vol. 30(11), pp. 1958 - 1970, 2008.

[J5] Levin, A., Fergus, R., Durand, F. and Freeman, W.T., “Image and Depth from a Conventional Camera with a Coded Aperture”, Vol. 26, Issue 3, pp. 70-79, ACM Trans. on Graphics (Proc. SIGGRAPH) 2007.

[J4] Fergus, R., Singh, B., Hertzmann A., Roweis, S. T. and Freeman, W.T., “Removing Camera Shake From a Single Photograph”, Vol. 25, Issue 3, pp. 787-794, ACM Trans. on Graphics (Proc. SIGGRAPH) 2006.

[J3] Fergus, R., Perona, P. and Zisserman, A., “Weakly Supervised Scale-Invariant Learning of Models for Visual Recognition”, Vol. 71, Issue 3, pp. 273-303, International Journal of Computer Vision (IJCV), 2007.

[J2] Fei-Fei, L., Fergus, R. and Perona, P., ”Learning Generative Visual Models for 101 Object Categories”, Computer Vision and Image Understanding, 2006.

[J1] Fei-Fei, L., Fergus, R. and Perona,P., “One-Shot Learning of Object Categories”, IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), Vol. 28(4), pp. 594 - 611, 2006.

Book chapters

[B3] Grauman, K. and Fergus, R., “Learning Binary Projections for Large Scale Image Search”, in Computer Vision, Editors: Cipolla, R., Battiato, S., Farinella, G., Springer, In Preparation, 2011.

[B2] Fergus, R., Perona, P. and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, in Cognitive Vision Systems, Editor: Nagel, H.H., Springer LNCS 3948, 2006.

[B1] Fergus, R., Perona, P. and Zisserman, A., “A Sparse Object Category Model for Efficient Learning and Complete Recognition”, in Toward Category-Level Object Recognition”, Editors: J. Ponce, M. Hebert, C. Schmid, and A. Zisserman, Springer LNCS 4170, 2006.

PATENTS Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T. and Freeman, W.T., “Removing Camera Shake from a Single Photograph using Statistics of a Natural Image”. US Patent 7616826, granted 11/10/ 09.

Fergus, R. and Krishnan, D., “Dark Flash Photography”. Patent application 475396-00251, filed 01/09/10.

GRADUATE STUDENTS/POSTDOCS SUPERVISED

Dilip Krishnan (Microsoft Fellowship) PhD, 4th year

Nathan Silberman PhD, 3rd year

Li Wan PhD, 3rd year

Matt Zeiler (NSERC Fellowship) PhD, 3rd year

David Eigen PhD, 2nd year

Leo Zhu (09/10—08/11) Postdoc, joint with Y. LeCun

Graham Taylor (09/09—08/11) Postdoc, joint with C. Bregler and Y. LeCun

UNDERGRADUATE STUDENTS SUPERVISED

Liz Balsam (09/11 – present)

Andrew Flockhart (09/11 – present)

Angjoo Kim (09/10 – 06/11) Now at U. Maryland

Melanie Clements (09/07 – 05/09) Now at Google NYC

TEACHING Computational Photography Spring 2008, 2009, 2010, Fall 2011

Introduction to Artificial Intelligence Fall 2009

Computer Vision Fall 2008, Spring 2011

Recognizing and Learning Object Categories ICCV 2005, 2009, CVPR 2007, ICML 2008 Conference tutorial with L.Fei-Fei and A. Torralba

New York City High School Outreach Jan/Feb 2011

FUNDING AWARDS

NSF Career (Recommended) “Large Scale Non-Parametric Image Dec 2011 Understanding” ($500,000), PI

NSF #1124794 “CDI: A Unified Probabilistic Model of Astronomical Imaging” ($675,000), co-PI June 2011

NSF #1116923 “RI: Small: Indoor Visual Navigation and Recognition for the Blind Using a Motion Sensing Input Device“ ($449,995), PI June 2011

Sloan Research Fellowship ($50,000), PI May 2011

Gift from Microsoft Research ($15,000) Jan 2010

DARPA “Deep Learning” grant of $1,242,610 (co-PI) Dec 2009

ONR #N00014-10-1-0294 “Learning Hierarchical Models for Information Dec 2009 Integration” ($906,309), co-PI

NYU URCF “Hyper-spectral Flash Photography” grant ($11,400) July 2009

Gift from NHK Corporation, Japan ($20,000) July 2009

Gift from Microsoft Research ($20,000) April 2009

Google Research Award ($50,000, with Y. LeCun) June 2008

Gift from Microsoft Research ($25,000) April 2008

PROFESSIONAL ACTIVITIES

Area chair: CVPR 2012 SIGGRAPH 2011 Technical Papers, ICCV 2011, NIPS 2011 SIGGRAPH 2010 Technical Papers, CVPR 2010, NIPS 2010

Conference reviewing: CVPR, NIPS, ECCV, ICCV, BMVC, SIGGRAPH, ICCP

Journal reviewing: IJCV, PAMI, JMLR, TOG, CVIU, TIP, JOV

PROFESSIONAL MEMBERSHIPS

Associate member of NCAP program of Canadian Institute for Advanced Research Member of IEEE, ACM

INVITED / ORGANIZED WORKSHOPS

Co-organizer NIPS 2011 workshop (with M. Hirsch, S. Harmelling, P. Milanfar) Dec 2011 “Machine Learning meets Computational Photography”

Frontiers of Computer Vision Workshop, MIT August 2011

Co-organizer CVPR 2011 workshop (wth A. Berg) June 2011 “Large Scale Learning for Vision”

Computational Photography Workshop, NIPS 2010 Dec 2010

International Workshop on Computer Vision, Italy May 2010

Banff International Research Station, CA June 2009

NCAP Summer School, Toronto, CA Aug 2008

May 2008

May 2008, Dec 2009

Snowbird Learning Workshop (Snowbird, UT) April 2008 “Small Codes and Large Image Databases for Object Recognition”

IPAM meeting (UCLA) Nov 2007 “80 Million Tiny Images”

International Object Recognition Workshop (Sicily) Sept 2006 Oct 2004 Object Class Recognition Using Unsupervised Scale-Invariant Learning” Aug 2003

Intl. Workshop on Current Trends in Computer Vision, Lhasa, Tibet Aug 2006 “Learning Object Categories from Google”

Neural Information Processing Systems (NIPS) Workshop Dec 2005 “Transferring Information Using Bayesian Priors on Object Categories”

RECENT INVITED TALKS

Deconvolutional Networks December 2010 May 2011 April 2011 July 2011

Semi-supervised Learning in Gigantic Image Collections December 2009 December 2009

Dark Flash Photography October 2009

May 2009

Small Codes and Large Image Databases for Object Recognition Nov 2011 Sept 2010 June 2008