Journal of Machine Learning Research 9 (2008) 371-421 Submitted 8/07; Published 3/08 A Tutorial on Conformal Prediction Glenn Shafer∗
[email protected] Department of Accounting and Information Systems Rutgers Business School 180 University Avenue Newark, NJ 07102, USA Vladimir Vovk
[email protected] Computer Learning Research Centre Department of Computer Science Royal Holloway, University of London Egham, Surrey TW20 0EX, UK Editor: Sanjoy Dasgupta Abstract Conformal prediction uses past experience to determine precise levels of confidence in new pre- dictions. Given an error probability ε, together with a method that makes a prediction yˆ of a label y, it produces a set of labels, typically containing yˆ, that also contains y with probability 1 − ε. Conformal prediction can be applied to any method for producing yˆ: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted succes- sively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right 1 − ε of the time, even though they are based on an accumulating data set rather than on independent data sets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples.