Wright State University CORE Scholar

Computer Science & Engineering Syllabi College of Engineering & Computer Science

Fall 2007

CS 790: Information Theory, Machine Learning and Statistics

Shaojun Wang Wright State University - Main Campus, [email protected]

Follow this and additional works at: https://corescholar.libraries.wright.edu/cecs_syllabi

Part of the Computer Engineering Commons, and the Computer Sciences Commons

Repository Citation Wang, S. (2007). CS 790: Information Theory, Machine Learning and Statistics. . https://corescholar.libraries.wright.edu/cecs_syllabi/231

This Syllabus is brought to you for free and open access by the College of Engineering & Computer Science at CORE Scholar. It has been accepted for inclusion in Computer Science & Engineering Syllabi by an authorized administrator of CORE Scholar. For more information, please contact library[email protected]. Page 1of2

CS790: INFORMATION THEORY, MACHINE LEARNING AND STATISTICS FALL 2007

Infonnation theory deals with encoding data in order to transmit it correctly and effectively. Statistics and machine learning deal with estimating models of data and predicting future observations. ls there any relationship between the two? It turns out, perhaps not surprisingly, that the most compact encoding ofthe data is by the probabilistic model that describes it best. In other words, there is a fundamental link between information and probability.

This course starts with the basic notions of infonnation theory and explores its relationship to machine learning and statistics. The course will have a strong theoretical component, but will also focus on applications and computing. The topics to be covered are: • Entropy, mutual information, Kullback-Leibler and Bregman divergences; • Source-channel models; boosting and optimal betting strategy; • Maximum likelihood and Bayesian inference; • Channel capacity, rate distortion and information bottleneck method; • Maximum entropy principle, information geometry and alternating algorithms; • Large deviations, coding theory and approximate inference in graphical models.

LECTURES

Time: Tuesday/Thursday 8:00 pm -9:15 pm; Location: Russ Center 155

INSTRUCTOR

Shaojun Wang 428, Russ Engineering Center Building shaoj1m. wang( at)wright.edu (937) 775-5140 Oflice hours: Tuesday/Thursday 2:00PM-3:30PM

Thomas M. Cover and Joy A. Thomas El~m&111s__~lnfqrmmkl11~Thi!ory~l_£4i@11 Wiley-Interscience, 2006.

http://www.cs.wright.edu/~swang/cs790inf/ 9/11/2007 Page 2 of2

I. Csiszar and P. Shields l!Jfgrmation Theory and Statistics: A Tutorial Foundations and Trends in Communications and Information Theory, 1(4):417-528, 2004

COURSE GRADES AND WORKLOAD

Three Homeworks 60% Projects 40% (Presentation: 20%; Report: 20%)

PREREQUISITE

A rudimentary knowledge ofprobability and statistics, for example, familiar with the materials in the standard textbook, A First Course in Probability by Sheldon Ross.

http://www.cs.wright.edu/~swang/cs790inf/ 9/11/2007