, 1 Lattice Duality: The Origin of Probability and Entropy Kevin H. Knuth NASA Ames Research Center, Mail Stop 269-3, Moffett Field CA 94035-1000, USA Abstract Bayesian probability theory is an inference calculus, which originates from a gen- eralization of inclusion on the Boolean lattice of logical assertions to a degree of inclusion represented by a real number. Dual to this lattice is the distributive lat- tice of questions constructed from the ordered set of down-sets of assertions, which forms the foundation of the calculus of inquiry-a generalization of information theory. In this paper we introduce this novel perspective on these spaces in which machine learning is performed and discuss the relationship between these results and several proposed generalizations of information theory in the literature. Key words: probability, entropy, lattice, information theory, Bayesian inference, inquiry PACS: 1 Introduction It has been known for some time that probability theory can be derived as a generalization of Boolean implication to degrees of implication repre- sented by real numbers [11,12]. Straightforward consistency requirements dic- tate the form of the sum and product rules of probability, and Bayes’ thee rem [11,12,47,46,20,34],which forms the basis of the inferential calculus, also known as inductive inference. However, in machine learning applications it is often times more useful to rely on information theory [45] in the design of an algorithm. On the surface, the connection between information theory and probability theory seems clear-information depends on entropy and entropy Email address: kevin.h.
[email protected] (Kevin H.