Entropy in Classical and Quantum Information Theory William Fedus Physics Department, University of California, San Diego. Entropy is a central concept in both classical and quantum information theory, measuring the uncertainty and the information content in the state of a physical system. This paper reviews classical information theory and then proceeds to generalizations into quantum information theory. Both Shannon and Von Neumann entropy are discussed, making the connection to compressibility of a message stream and the generalization of compressibility in a quantum system. Finally, the paper considers the application of Von Neumann entropy in entanglement of formation for both pure and mixed bipartite quantum states. CLASSICAL INFORMATION THEORY information. For instance, if we use a block code which assigns integers to typical sequences, the information in In statistical mechanics, entropy is the logarithm of a string of n letters can be compressed to H(A) bits. the number of arrangements a system can be configured With this framework and definition, we may now con- and still remain consistent with the thermodyanmic ob- sider the maximum compression of a length n message servables. From this original formulation, entropy has without loss of information. The number of bits neces- grown to become an important element in many diverse sary to transmit the message is given by fields of study. One of the first examples was in 1948 when Claude Shannon adopted entropy as a measure of the uncertainty in a random variable, or equivalently, the H(An) = nH(A): (2) expected value of information content within a message. Classical information theory, as established by Claude Shannon, sought to resolve two central issues in signal which simply states that one needs (n times the entropy processing of the ensemeble A)-bits. However, a more interesting case may be considered 1. The compression achievable for a message while if we allow a small error δ > 0 in the encoding of the preserving the fidelity of the original information message. Specifically, we will only encode messages in 2. The rate at which a message can be communicated set B ⊂ A such that P (B) ≥ 1 − δ. The information size reliabily over a noisy channel is given by Hδ(A) which is equal to H(A) in the limit that δ ! 0. Shannon's noiseless coding theorem then both of these address the issue of the degree of redun- states that dancy embedded within a message and, as we will see, entropy is found to be a rigorous and useful approach in this context. 1 n lim Hδ(A ) = H(A) (3) n!1 n Shannon Entropy and Data Compression that is, the optimal compression rate is simply the Shan- non entropy; this is Shannon's noiseless coding theorem. As alluded to previously, classical information theory As a basic example of this theorem, consider an infor- is framed in terms of the information content and the mation source which produces messages composed of the transmission of messages. A message may be defined as four element alphabet, a string of letters chosen from an alphabet of k letters fa1; a2; :::; akg. Assuming that each letter ax occurs with probability p(ax) and is independently transmitted, this may be denoted as an ensemble A = fax; p(ax)g; the A = fa1; a2; a3; a4g: (4) Shannon entropy of an ensemble is then defined as Clearly, without any compression, two bits of storage X are necessary for each letter transmitted (bit strings 00, H(A) ≡ H (p(a ); :::; p(a )) ≡ − p(a )logp(a ): (1) x k x x 01, 10, 11) as depicted in the left side of Figure 1. How- x ever, suppose now that symbol a1 is produced with prob- where log is taken to be base-2 since we are transmitting ability 1/2, symbol a2 with probability 1/4 and symbols messages with binary bits. This formula is important as a3 and a4 with probability 1/8. We find that by applying it can be used to quantify the resources necessary to store Eq. 1, we can calculate the entropy to be, 2 quantum states, X = fpx; ρxg where ρx is the quantum state and px is the a priori probability of selecting that quantum state. This quantum state may be completely characterized via the density matrix X ρ = pxρx (7) x The Von Neumann entropy of a quantum state ρ is then defined as FIG. 1. Tree representation for the encoding of the above example. S(ρ) ≡ −Tr (ρlogρ) (8) where log is taken to be base d, d being the dimension of X the Hilbert space containing ρ. H(A) = −p(ax)logp(ax) (5a) Von Neumann entropy enters quantum information x theory in three important ways. First, it is a measure 1 1 1 1 1 1 = − ∗ log − log − 2 ∗ log of the quantum information content of letters in the en- 2 2 4 4 8 8 semble, specifically, how many quibits are needed to en- (5b) code the message without loss of information. Second, 7 it is also a measure of the classical information content = (5c) 4 per letter, where we find the maximum amount of infor- mation in bits that can be recovered from the message. Therefore, a coding scheme exists in which the mes- And finally, as we will see later, Von Neumann entropy sages may be transmitted with only 7/4 bits per letter is used to quantify the entanglement of pure and mixed on average if we assume that the message being transmit- bipartite quantum states [1]. ted is of length n and n 1. To realize this compression, Now, returning to Eq. 8, if λxs are the eigenvalues we choose to encode a1 as it string 0, a2 as bit string 10, that diagonalize ρ, then we may rewrite it as, a3 as bit string 110, and a4 as bit string 111, we find that the average length of the compressed string is X S(ρ) = − λxlogλx: (9) 1 1 1 1 7 x ∗ 1 + ∗ 2 + ∗ 3 + ∗ 3 = (6) 2 4 8 8 4 One now observes that in this orthonormal basis, the Von Neumann entropy is equivalent to the Shannon entropy, indicating that we have achieved the best possible com- Eq. 1, pression for an encoding. Also, it should be noted that this encoding will not compress all messages since clearly an unlucky and improbable message of repeated a4s S(ρ) = H(A) (10) would result in a higher average bit content per letter than both encodings. Also, note that Figure 1 explicitly for the ensemble A = fa; λag. This indicates that if a demonstrates that our encoding must not have duplicate quantum system is a pure separable system, it reduces prefixes, otherwise, the encoding is not valid. If letters to the classical system. For a separable quantum system, of the alphabet share a common prefix, message strings the Von Neumann entropy is another quantification of the may not be uniquely decodable. incompressibility of the information content, analagous to the Shannon entropy of a classical system. QUANTUM INFORMATION THEORY Connection to Thermodynamics In classical information theory, we have considered a message of n letters, where n 1 and each letter is Entropy in statistical thermodynamics and in informa- drawn independently from an ensemble A = fax; p(ax)g. tion theory are not disjoint concepts. For instance, if we The information content of the ensemble is equal to the take an open quantum system (in contact with its en- Shannon information, H(A), in the limit that n ! 1 vironment), we may use the mathematical properties of To generalize this formulation to the quantum regime, subadditivity and entropic invariance under unitary evo- the message may still be regarded as composed of n let- lution for Von Neumann entropy to imply the second law ters, but now the letters are drawn from an ensemble of of thermodynamics. 3 To see this, subadditivity may be succinctly expressed and the entire message of length n can then be charac- as, terized as the n-tensor product n S(ρAB) ≤ S(ρA) + S(ρB) (11) ρ = ρ ⊗ · · · ⊗ ρ (17) where ρA is the partial trace over B (ρA ≡ TrB(ρAB)) Now, the best optimal compression of this quantum and ρB is defined similarly. If we consider the system (A) message to a Hilbert space H as n ! 1 is given by and the environment (E) to be initially uncorrelated, we can decompose the density matrix as the tensor product log(dimH) = S(ρn) = nS(ρ) (18) ρAB = ρA ⊗ρB which by Eq. 11 implies that the entropy is simply the sum of the two states A and B where dim is the dimension of the Hilbert space. This implies that Von Neumann entropy is equal to the num- S(ρ ) = S(ρ ) + S(ρ ): (12) ber of quibits of quantum information carried for each AB A B letter of the message Note that compression is always If we now allow the system to evolve under a unitary possible provided that the density matrix is not a max- 1 1 operator U which acts on the full system, the density imially mixed state, ρ = 2 , since we cannot compress matrix becomes random quibits, directly analagous to the classical incom- pressibility of random bits. 0 −1 ρAE = UρAEU (13) ENTANGLEMENT MEASURES but we know that the entropy of a system is invariant under unitary evolution S(ρ) = S(UρU −1) so therefore, Entanglement is a feature of quantum states to exhibit correlations that cannot be accounted for classically. En- tanglement theory is underpinned by the paradigm of 0 S(ρAE) = S(ρAE): (14) Local Operations and Classical Communication (LOCC) formulated by Bennett et al [2].