Belief in Information Flow Michael R. Clarkson Andrew C. Myers Fred B. Schneider Department of Computer Science Cornell University {clarkson,andru,fbs}@cs.cornell.edu Abstract include guards, which sit at the boundary between trusted and untrusted systems, and password checkers. Information leakage traditionally has been defined to oc- Defining the quantity of information flow is more diffi- cur when uncertainty about secret data is reduced. This cult than it might seem. Consider a password checker PWC uncertainty-based approach is inadequate for measuring that sets an authentication flag a after checking a stored information flow when an attacker is making assumptions password p against a (guessed) password g supplied by the about secret inputs and these assumptions might be incor- user. rect; such attacker beliefs are an unavoidable aspect of any satisfactory definition of leakage. To reason about informa- PWC : if p = g then a := 1 else a := 0 tion flow based on beliefs, a model is developed that de- scribes how attacker beliefs change due to the attacker’s For simplicity, suppose that the password is either A, B, observation of the execution of a probabilistic (or determin- or C. Suppose also that the user is actually an attacker at- istic) program. The model leads to a new metric for quanti- tempting to discover the password, and he believes the pass- tative information flow that measures accuracy rather than word is overwhelmingly likely to be A but has a minuscule uncertainty of beliefs. and equally likely chance to be either B or C. (This need not be an arbitrary assumption on the attacker’s part; per- haps the attacker was told by a usually reliable informant that the password is A.) If the attacker experiments by ex- 1. Introduction ecuting PWC and guessing A, he expects the outcome to be that a is equal to 1. Such a confirmation of the attacker’s Qualitative security properties, such as noninterference belief does seem to convey some small amount of informa- [10], typically either prohibit any flow of information from tion. But suppose that the informant was wrong: the real a high security level to a lower level, or they allow any password is C. The outcome of this experiment has a equal amount of flow so long as it passes through some release to 0, from which the attacker infers that A is not the pass- mechanism. For a program whose correctness requires flow word. Common sense dictates that his new belief is that B from high to low, the former property is too restrictive and and C each have a 50% chance of being the password. The the latter can lead to unbounded leakage of information. attacker’s belief has greatly changed—he is surprised to dis- Quantitative flow properties, such as “at most k bits leak cover the password is not A—so this outcome of his exper- per execution of the program”, allow information flows but iment seems to convey a larger amount of information than at restricted rates. Such properties are useful when analyz- the previous outcome. Thus, the information conveyed by ing programs whose nature requires that some—but not too executing PWC depends on what the attacker believes. much—information be leaked. Examples of these programs How much information flows from p to a in each of the above experiments? Answers to this question have tradition- This work was supported by the Department of the Navy, Office of Naval Research, ONR Grant N00014-01-1-0968; Air Force Office of Scientific ally been based on change in uncertainty [5, 20, 11, 1, 16, 2, Research, Air Force Materiel Command, USAF, grant number F49620-03- 17]: information flow is measured by the reduction in uncer- 1-0156; and National Science Foundation grants 0208642, 0133302, and tainty about secret data. Observe that, in the case where the 0430161. Michael Clarkson is supported by a National Science Founda- tion Graduate Research Fellowship; Andrew Myers is supported by an Al- password is C, the attacker initially is quite certain (though fred P. Sloan Research Fellowship. Opinions, findings, conclusions, or rec- wrong) about the value of the password and after the exper- ommendations contained in this material are those of the authors and do iment is rather uncertain about the value of the password; not necessarily reflect the views of these sponsors. The U.S. Government is authorized to reproduce and distribute reprints for Governmental pur- the change from “quite certain” to “rather uncertain” is an poses notwithstanding any copyright notation thereon. increase in uncertainty. So according to a reduction in un- Proceedings of the 18th IEEE Computer Security Foundations Workshop (CSFW’05) 1063-6900/05 $20.00 © 2005 IEEE certainty metric, no information flow occurred, which flatly guage semantics [21]. Henceforth, we write “distribution” contradicts our intuition. to mean “frequency distribution”. The problem with metrics based on uncertainty is The set of all program states is State, and the set of all twofold. First, they do not take accuracy into account. Ac- distributions is Dist. The structure of State is mostly unim- curacy and uncertainty are orthogonal properties of portant; it can be instantiated according to the needs of any the attacker’s belief—being certain does not make one particular language or system. For our examples, states map correct—and as the password checking example illus- variables to values, where Var and Val are both countable trates, the amount of information flow depends on accu- sets. ∈ racy rather than on uncertainty. Second, uncertainty-based v Var ∈ → metrics are concerned with some unspecified agent’s un- σ State Var Val + certainty rather than an attacker’s. The unspecified agent δ ∈ Dist State → R is able to observe a probability distribution over secret in- We write a state as a list of mappings; e.g. (g → A, a → 0) put values but cannot observe the particular secret in- is a state in which variable g has value A and a has value 0. put used in the program execution. If the attacker were The mass in a distribution δ is the sum of frequencies: the unspecified agent, there would be no reason in gen- eral to assume the probability distribution the attacker δ σ δ(σ) uses is correct. Because the attacker’s probability distri- bution is therefore subjective, it must be treated as a be- A probability distribution has mass 1, but a frequency distri- lief. Beliefs are thus an essential—though until now bution may have any non-negative mass. A point mass is a uninvestigated—component of information flow. probability distribution that maps a single state to 1. It is de- This paper presents a new way of measuring informa- noted by placing a dot over that single state: tion flow, based on these insights. Section 2 gives basic rep- σ˙ λσ . if σ = σ then 1 else 0 resentations and notations for beliefs and programs. Sec- tion 3 describes a model of the interaction between attack- 2.2. Programs ers and systems; it also describes how attackers update be- liefs by observing execution of programs. Section 4 defines Execution of program S is described by a denotational a new quantitative flow metric, based on information the- semantics in which the meaning [[ S]] of S is a function of ory, that characterizes the amount of information flow due type State → Dist. This semantics describes the frequency to changes in the accuracy of an attacker’s belief. The model of termination in a given state: if [[ S]] σ = δ, then the fre- and metric are formulated for use with any programming quency of S, when begun in σ, terminating in σ should language (or even any state machine) that can be given a de- be δ(σ). This semantics can be lifted to a function of type notational semantics compatible with the representation of Dist → Dist by the following definition: beliefs, and Section 5 illustrates with a particular program- ming language (while-programs plus probabilistic choice). [[ S]] δ σ δ(σ) · [[ S]] σ Section 6 discusses related work, and Section 7 concludes. Thus, the meaning of S over a distribution of inputs is completely determined by the meaning of S given a state 2. Incorporating beliefs as input. By defining programs in terms of how they oper- ate on distributions we permit analysis of probabilistic pro- A belief is a statement an agent makes about the state grams. Section 5 shows how to build such a semantics. of the world, accompanied by some measure of how cer- Our examples use while-programs extended with a prob- tain the agent is about the truthfulness of the statement. We abilistic choice construct. Let metavariables S, v, E,and begin by developing mathematical structures for represent- B range over programs, variables, arithmetic expressions, ing beliefs. and Boolean expressions, respectively. Evaluation of ex- pressions is assumed side-effect free, but we do not other- 2.1. Distributions wise prescribe their syntax or semantics. The syntax of the language is: A frequency distribution is a function δ that maps a pro- S ::= skip | v := E | S; S | if B then S else S gram state to a frequency, where a frequency is a non- | while B do S | S p S negative real number. A frequency distribution is essen- tially an unnormalized probability distribution over pro- The operational semantics for the deterministic subset of gram states; frequency distributions are often better than this language is standard. Probabilistic choice S1 p S2 ex- probability distributions as the basis for a programming lan- ecutes S1 with probability p or S2 with probability 1 − p.