DOCSLIB.ORG

Entropy in Classical and Quantum Information Theory

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Entropy in Classical and Quantum Information Theory Entropy in Classical and Quantum Information Theory William Fedus Physics Department, University of California, San Diego. Entropy is a central concept in both classical and quantum information theory, measuring the uncertainty and the information content in the state of a physical system. This paper reviews classical information theory and then proceeds to generalizations into quantum information theory. Both Shannon and Von Neumann entropy are discussed, making the connection to compressibility of a message stream and the generalization of compressibility in a quantum system. Finally, the paper considers the application of Von Neumann entropy in entanglement of formation for both pure and mixed bipartite quantum states. CLASSICAL INFORMATION THEORY information. For instance, if we use a block code which assigns integers to typical sequences, the information in In statistical mechanics, entropy is the logarithm of a string of n letters can be compressed to H(A) bits. the number of arrangements a system can be configured With this framework and definition, we may now con- and still remain consistent with the thermodyanmic ob- sider the maximum compression of a length n message servables. From this original formulation, entropy has without loss of information. The number of bits neces- grown to become an important element in many diverse sary to transmit the message is given by fields of study. One of the first examples was in 1948 when Claude Shannon adopted entropy as a measure of the uncertainty in a random variable, or equivalently, the H(An) = nH(A): (2) expected value of information content within a message. Classical information theory, as established by Claude Shannon, sought to resolve two central issues in signal which simply states that one needs (n times the entropy processing of the ensemeble A)-bits. However, a more interesting case may be considered 1. The compression achievable for a message while if we allow a small error δ > 0 in the encoding of the preserving the fidelity of the original information message. Specifically, we will only encode messages in 2. The rate at which a message can be communicated set B ⊂ A such that P (B) ≥ 1 − δ. The information size reliabily over a noisy channel is given by Hδ(A) which is equal to H(A) in the limit that δ ! 0. Shannon's noiseless coding theorem then both of these address the issue of the degree of redun- states that dancy embedded within a message and, as we will see, entropy is found to be a rigorous and useful approach in this context. 1 n lim Hδ(A ) = H(A) (3) n!1 n Shannon Entropy and Data Compression that is, the optimal compression rate is simply the Shan- non entropy; this is Shannon's noiseless coding theorem. As alluded to previously, classical information theory As a basic example of this theorem, consider an infor- is framed in terms of the information content and the mation source which produces messages composed of the transmission of messages. A message may be defined as four element alphabet, a string of letters chosen from an alphabet of k letters fa1; a2; :::; akg. Assuming that each letter ax occurs with probability p(ax) and is independently transmitted, this may be denoted as an ensemble A = fax; p(ax)g; the A = fa1; a2; a3; a4g: (4) Shannon entropy of an ensemble is then defined as Clearly, without any compression, two bits of storage X are necessary for each letter transmitted (bit strings 00, H(A) ≡ H (p(a ); :::; p(a )) ≡ − p(a )logp(a ): (1) x k x x 01, 10, 11) as depicted in the left side of Figure 1. How- x ever, suppose now that symbol a1 is produced with prob- where log is taken to be base-2 since we are transmitting ability 1/2, symbol a2 with probability 1/4 and symbols messages with binary bits. This formula is important as a3 and a4 with probability 1/8. We find that by applying it can be used to quantify the resources necessary to store Eq. 1, we can calculate the entropy to be, 2 quantum states, X = fpx; ρxg where ρx is the quantum state and px is the a priori probability of selecting that quantum state. This quantum state may be completely characterized via the density matrix X ρ = pxρx (7) x The Von Neumann entropy of a quantum state ρ is then defined as FIG. 1. Tree representation for the encoding of the above example. S(ρ) ≡ −Tr (ρlogρ) (8) where log is taken to be base d, d being the dimension of X the Hilbert space containing ρ. H(A) = −p(ax)logp(ax) (5a) Von Neumann entropy enters quantum information x theory in three important ways. First, it is a measure 1 1 1 1 1 1 = − ∗ log − log − 2 ∗ log of the quantum information content of letters in the en- 2 2 4 4 8 8 semble, specifically, how many quibits are needed to en- (5b) code the message without loss of information. Second, 7 it is also a measure of the classical information content = (5c) 4 per letter, where we find the maximum amount of infor- mation in bits that can be recovered from the message. Therefore, a coding scheme exists in which the mes- And finally, as we will see later, Von Neumann entropy sages may be transmitted with only 7/4 bits per letter is used to quantify the entanglement of pure and mixed on average if we assume that the message being transmit- bipartite quantum states [1]. ted is of length n and n 1. To realize this compression, Now, returning to Eq. 8, if λxs are the eigenvalues we choose to encode a1 as it string 0, a2 as bit string 10, that diagonalize ρ, then we may rewrite it as, a3 as bit string 110, and a4 as bit string 111, we find that the average length of the compressed string is X S(ρ) = − λxlogλx: (9) 1 1 1 1 7 x ∗ 1 + ∗ 2 + ∗ 3 + ∗ 3 = (6) 2 4 8 8 4 One now observes that in this orthonormal basis, the Von Neumann entropy is equivalent to the Shannon entropy, indicating that we have achieved the best possible com- Eq. 1, pression for an encoding. Also, it should be noted that this encoding will not compress all messages since clearly an unlucky and improbable message of repeated a4s S(ρ) = H(A) (10) would result in a higher average bit content per letter than both encodings. Also, note that Figure 1 explicitly for the ensemble A = fa; λag. This indicates that if a demonstrates that our encoding must not have duplicate quantum system is a pure separable system, it reduces prefixes, otherwise, the encoding is not valid. If letters to the classical system. For a separable quantum system, of the alphabet share a common prefix, message strings the Von Neumann entropy is another quantification of the may not be uniquely decodable. incompressibility of the information content, analagous to the Shannon entropy of a classical system. QUANTUM INFORMATION THEORY Connection to Thermodynamics In classical information theory, we have considered a message of n letters, where n 1 and each letter is Entropy in statistical thermodynamics and in informa- drawn independently from an ensemble A = fax; p(ax)g. tion theory are not disjoint concepts. For instance, if we The information content of the ensemble is equal to the take an open quantum system (in contact with its en- Shannon information, H(A), in the limit that n ! 1 vironment), we may use the mathematical properties of To generalize this formulation to the quantum regime, subadditivity and entropic invariance under unitary evo- the message may still be regarded as composed of n let- lution for Von Neumann entropy to imply the second law ters, but now the letters are drawn from an ensemble of of thermodynamics. 3 To see this, subadditivity may be succinctly expressed and the entire message of length n can then be charac- as, terized as the n-tensor product n S(ρAB) ≤ S(ρA) + S(ρB) (11) ρ = ρ ⊗ · · · ⊗ ρ (17) where ρA is the partial trace over B (ρA ≡ TrB(ρAB)) Now, the best optimal compression of this quantum and ρB is defined similarly. If we consider the system (A) message to a Hilbert space H as n ! 1 is given by and the environment (E) to be initially uncorrelated, we can decompose the density matrix as the tensor product log(dimH) = S(ρn) = nS(ρ) (18) ρAB = ρA ⊗ρB which by Eq. 11 implies that the entropy is simply the sum of the two states A and B where dim is the dimension of the Hilbert space. This implies that Von Neumann entropy is equal to the num- S(ρ ) = S(ρ ) + S(ρ ): (12) ber of quibits of quantum information carried for each AB A B letter of the message Note that compression is always If we now allow the system to evolve under a unitary possible provided that the density matrix is not a max- 1 1 operator U which acts on the full system, the density imially mixed state, ρ = 2 , since we cannot compress matrix becomes random quibits, directly analagous to the classical incom- pressibility of random bits. 0 −1 ρAE = UρAEU (13) ENTANGLEMENT MEASURES but we know that the entropy of a system is invariant under unitary evolution S(ρ) = S(UρU −1) so therefore, Entanglement is a feature of quantum states to exhibit correlations that cannot be accounted for classically. En- tanglement theory is underpinned by the paradigm of 0 S(ρAE) = S(ρAE): (14) Local Operations and Classical Communication (LOCC) formulated by Bennett et al [2].
Load more

Recommended publications
  • On Entropy, Information, and Conservation of Information
    entropy Article On Entropy, Information, and Conservation of Information Yunus A. Çengel Department of Mechanical Engineering, University of Nevada, Reno, NV 89557, USA; [email protected] Abstract: The term entropy is used in different meanings in different contexts, sometimes in contradic- tory ways, resulting in misunderstandings and confusion. The root cause of the problem is the close resemblance of the defining mathematical expressions of entropy in statistical thermodynamics and information in the communications field, also called entropy, differing only by a constant factor with the unit ‘J/K’ in thermodynamics and ‘bits’ in the information theory. The thermodynamic property entropy is closely associated with the physical quantities of thermal energy and temperature, while the entropy used in the communications field is a mathematical abstraction based on probabilities of messages. The terms information and entropy are often used interchangeably in several branches of sciences. This practice gives rise to the phrase conservation of entropy in the sense of conservation of information, which is in contradiction to the fundamental increase of entropy principle in thermody- namics as an expression of the second law. The aim of this paper is to clarify matters and eliminate confusion by putting things into their rightful places within their domains. The notion of conservation of information is also put into a proper perspective. Keywords: entropy; information; conservation of information; creation of information; destruction of information; Boltzmann relation Citation: Çengel, Y.A. On Entropy, 1. Introduction Information, and Conservation of One needs to be cautious when dealing with information since it is defined differently Information. Entropy 2021, 23, 779.
    [Show full text]
  • ENERGY, ENTROPY, and INFORMATION Jean Thoma June
    ENERGY, ENTROPY, AND INFORMATION Jean Thoma June 1977 Research Memoranda are interim reports on research being conducted by the International Institute for Applied Systems Analysis, and as such receive only limited scientific review. Views or opinions contained herein do not necessarily represent those of the Institute or of the National Member Organizations supporting the Institute. PREFACE This Research Memorandum contains the work done during the stay of Professor Dr.Sc. Jean Thoma, Zug, Switzerland, at IIASA in November 1976. It is based on extensive discussions with Professor HAfele and other members of the Energy Program. Al- though the content of this report is not yet very uniform because of the different starting points on the subject under consideration, its publication is considered a necessary step in fostering the related discussion at IIASA evolving around th.e problem of energy demand. ABSTRACT Thermodynamical considerations of energy and entropy are being pursued in order to arrive at a general starting point for relating entropy, negentropy, and information. Thus one hopes to ultimately arrive at a common denominator for quanti- ties of a more general nature, including economic parameters. The report closes with the description of various heating appli- cation.~and related efficiencies. Such considerations are important in order to understand in greater depth the nature and composition of energy demand. This may be highlighted by the observation that it is, of course, not the energy that is consumed or demanded for but the informa- tion that goes along with it. TABLE 'OF 'CONTENTS Introduction ..................................... 1 2 . Various Aspects of Entropy ........................2 2.1 i he no me no logical Entropy ........................
    [Show full text]
  • Lecture 4: 09.16.05 Temperature, Heat, and Entropy
    3.012 Fundamentals of Materials Science Fall 2005 Lecture 4: 09.16.05 Temperature, heat, and entropy Today: LAST TIME .........................................................................................................................................................................................2� State functions ..............................................................................................................................................................................2� Path dependent variables: heat and work..................................................................................................................................2� DEFINING TEMPERATURE ...................................................................................................................................................................4� The zeroth law of thermodynamics .............................................................................................................................................4� The absolute temperature scale ..................................................................................................................................................5� CONSEQUENCES OF THE RELATION BETWEEN TEMPERATURE, HEAT, AND ENTROPY: HEAT CAPACITY .......................................6� The difference between heat and temperature ...........................................................................................................................6� Defining heat capacity.................................................................................................................................................................6�
    [Show full text]
  • The Enthalpy, and the Entropy of Activation (Rabbit/Lobster/Chick/Tuna/Halibut/Cod) PHILIP S
    Proc. Nat. Acad. Sci. USA Vol. 70, No. 2, pp. 430-432, February 1973 Temperature Adaptation of Enzymes: Roles of the Free Energy, the Enthalpy, and the Entropy of Activation (rabbit/lobster/chick/tuna/halibut/cod) PHILIP S. LOW, JEFFREY L. BADA, AND GEORGE N. SOMERO Scripps Institution of Oceanography, University of California, La Jolla, Calif. 92037 Communicated by A. Baird Hasting8, December 8, 1972 ABSTRACT The enzymic reactions of ectothermic function if they were capable of reducing the AG* character- (cold-blooded) species differ from those of avian and istic of their reactions more than were the homologous en- mammalian species in terms of the magnitudes of the three thermodynamic activation parameters, the free zymes of more warm-adapted species, i.e., birds or mammals. energy of activation (AG*), the enthalpy of activation In this paper, we report that the values of AG* are indeed (AH*), and the entropy of activation (AS*). Ectothermic slightly lower for enzymic reactions catalyzed by enzymes enzymes are more efficient than the homologous enzymes of ectotherms, relative to the homologous reactions of birds of birds and mammals in reducing the AG* "energy bar- rier" to a chemical reaction. Moreover, the relative im- and mammals. Moreover, the relative contributions of the portance of the enthalpic and entropic contributions to enthalpies and entropies of activation to AG* differ markedly AG* differs between these two broad classes of organisms. and, we feel, adaptively, between ectothermic and avian- mammalian enzymic reactions. Because all organisms conduct many of the same chemical transformations, certain functional classes of enzymes are METHODS present in virtually all species.
    [Show full text]
  • A Mini-Introduction to Information Theory
    A Mini-Introduction To Information Theory Edward Witten School of Natural Sciences, Institute for Advanced Study Einstein Drive, Princeton, NJ 08540 USA Abstract This article consists of a very short introduction to classical and quantum information theory. Basic properties of the classical Shannon entropy and the quantum von Neumann entropy are described, along with related concepts such as classical and quantum relative entropy, conditional entropy, and mutual information. A few more detailed topics are considered in the quantum case. arXiv:1805.11965v5 [hep-th] 4 Oct 2019 Contents 1 Introduction 2 2 Classical Information Theory 2 2.1 ShannonEntropy ................................... .... 2 2.2 ConditionalEntropy ................................. .... 4 2.3 RelativeEntropy .................................... ... 6 2.4 Monotonicity of Relative Entropy . ...... 7 3 Quantum Information Theory: Basic Ingredients 10 3.1 DensityMatrices .................................... ... 10 3.2 QuantumEntropy................................... .... 14 3.3 Concavity ......................................... .. 16 3.4 Conditional and Relative Quantum Entropy . ....... 17 3.5 Monotonicity of Relative Entropy . ...... 20 3.6 GeneralizedMeasurements . ...... 22 3.7 QuantumChannels ................................... ... 24 3.8 Thermodynamics And Quantum Channels . ...... 26 4 More On Quantum Information Theory 27 4.1 Quantum Teleportation and Conditional Entropy . ......... 28 4.2 Quantum Relative Entropy And Hypothesis Testing . ......... 32 4.3 Encoding
    [Show full text]
  • Chapter 6 Quantum Entropy
    Chapter 6 Quantum entropy There is a notion of entropy which quantifies the amount of uncertainty contained in an ensemble of Qbits. This is the von Neumann entropy that we introduce in this chapter. In some respects it behaves just like Shannon's entropy but in some others it is very different and strange. As an illustration let us immediately say that as in the classical theory, conditioning reduces entropy; but in sharp contrast with classical theory the entropy of a quantum system can be lower than the entropy of its parts. The von Neumann entropy was first introduced in the realm of quantum statistical mechanics, but we will see in later chapters that it enters naturally in various theorems of quantum information theory. 6.1 Main properties of Shannon entropy Let X be a random variable taking values x in some alphabet with probabil- ities px = Prob(X = x). The Shannon entropy of X is X 1 H(X) = p ln x p x x and quantifies the average uncertainty about X. The joint entropy of two random variables X, Y is similarly defined as X 1 H(X; Y ) = p ln x;y p x;y x;y and the conditional entropy X X 1 H(XjY ) = py pxjy ln p j y x;y x y 1 2 CHAPTER 6. QUANTUM ENTROPY where px;y pxjy = py The conditional entropy is the average uncertainty of X given that we observe Y = y. It is easily seen that H(XjY ) = H(X; Y ) − H(Y ) The formula is consistent with the interpretation of H(XjY ): when we ob- serve Y the uncertainty H(X; Y ) is reduced by the amount H(Y ).
    [Show full text]
  • Quantum Thermodynamics at Strong Coupling: Operator Thermodynamic Functions and Relations
    Article Quantum Thermodynamics at Strong Coupling: Operator Thermodynamic Functions and Relations Jen-Tsung Hsiang 1,*,† and Bei-Lok Hu 2,† ID 1 Center for Field Theory and Particle Physics, Department of Physics, Fudan University, Shanghai 200433, China 2 Maryland Center for Fundamental Physics and Joint Quantum Institute, University of Maryland, College Park, MD 20742-4111, USA; [email protected] * Correspondence: [email protected]; Tel.: +86-21-3124-3754 † These authors contributed equally to this work. Received: 26 April 2018; Accepted: 30 May 2018; Published: 31 May 2018 Abstract: Identifying or constructing a fine-grained microscopic theory that will emerge under specific conditions to a known macroscopic theory is always a formidable challenge. Thermodynamics is perhaps one of the most powerful theories and best understood examples of emergence in physical sciences, which can be used for understanding the characteristics and mechanisms of emergent processes, both in terms of emergent structures and the emergent laws governing the effective or collective variables. Viewing quantum mechanics as an emergent theory requires a better understanding of all this. In this work we aim at a very modest goal, not quantum mechanics as thermodynamics, not yet, but the thermodynamics of quantum systems, or quantum thermodynamics. We will show why even with this minimal demand, there are many new issues which need be addressed and new rules formulated. The thermodynamics of small quantum many-body systems strongly coupled to a heat bath at low temperatures with non-Markovian behavior contains elements, such as quantum coherence, correlations, entanglement and fluctuations, that are not well recognized in traditional thermodynamics, built on large systems vanishingly weakly coupled to a non-dynamical reservoir.
    [Show full text]
  • Information Content and Error Analysis
    43 INFORMATION CONTENT AND ERROR ANALYSIS Clive D Rodgers Atmospheric, Oceanic and Planetary Physics University of Oxford ESA Advanced Atmospheric Training Course September 15th – 20th, 2008 44 INFORMATION CONTENT OF A MEASUREMENT Information in a general qualitative sense: Conceptually, what does y tell you about x? We need to answer this to determine if a conceptual instrument design actually works • to optimise designs • Use the linear problem for simplicity to illustrate the ideas. y = Kx + ! 45 SHANNON INFORMATION The Shannon information content of a measurement of x is the change in the entropy of the • probability density function describing our knowledge of x. Entropy is defined by: • S P = P (x) log(P (x)/M (x))dx { } − Z M(x) is a measure function. We will take it to be constant. Compare this with the statistical mechanics definition of entropy: • S = k p ln p − i i i X P (x)dx corresponds to pi. 1/M (x) is a kind of scale for dx. The Shannon information content of a measurement is the change in entropy between the p.d.f. • before, P (x), and the p.d.f. after, P (x y), the measurement: | H = S P (x) S P (x y) { }− { | } What does this all mean? 46 ENTROPY OF A BOXCAR PDF Consider a uniform p.d.f in one dimension, constant in (0,a): P (x)=1/a 0 <x<a and zero outside.The entropy is given by a 1 1 S = ln dx = ln a − a a Z0 „ « Similarly, the entropy of any constant pdf in a finite volume V of arbitrary shape is: 1 1 S = ln dv = ln V − V V ZV „ « i.e the entropy is the log of the volume of state space occupied by the p.d.f.
    [Show full text]
  • The Dynamics of a Central Spin in Quantum Heisenberg XY Model
    Eur. Phys. J. D 46, 375–380 (2008) DOI: 10.1140/epjd/e2007-00300-9 THE EUROPEAN PHYSICAL JOURNAL D The dynamics of a central spin in quantum Heisenberg XY model X.-Z. Yuan1,a,K.-D.Zhu1,andH.-S.Goan2 1 Department of Physics, Shanghai Jiao Tong University, Shanghai 200240, P.R. China 2 Department of Physics and Center for Theoretical Sciences, National Taiwan University, Taipei 10617, Taiwan Received 19 March 2007 / Received in final form 30 August 2007 Published online 24 October 2007 – c EDP Sciences, Societ`a Italiana di Fisica, Springer-Verlag 2007 Abstract. In the thermodynamic limit, we present an exact calculation of the time dynamics of a central spin coupling with its environment at finite temperatures. The interactions belong to the Heisenberg XY type. The case of an environment with finite number of spins is also discussed. To get the reduced density matrix, we use a novel operator technique which is mathematically simple and physically clear, and allows us to treat systems and environments that could all be strongly coupled mutually and internally. The expectation value of the central spin and the von Neumann entropy are obtained. PACS. 75.10.Jm Quantized spin models – 03.65.Yz Decoherence; open systems; quantum statistical meth- ods – 03.67.-a Quantum information z 1 Introduction like S0 and extend the operator technique to deal with the case of finite number environmental spins. Recently, using One of the promising candidates for quantum computa- numerical tools, the authors of references [5–8] studied the tion is spin systems [1–4] due to their long decoherence dynamic behavior of a central spin system decoherenced and relaxation time.
    [Show full text]
  • Entanglement Entropy and Decoupling in the Universe
    Entanglement Entropy and Decoupling in the Universe Yuichiro Nakai (Rutgers) with N. Shiba (Harvard) and M. Yamada (Tufts) arXiv:1709.02390 Brookhaven Forum 2017 Density matrix and Entropy In quantum mechanics, a physical state is described by a vector which belongs to the Hilbert space of the total system. However, in many situations, it is more convenient to consider quantum mechanics of a subsystem. In a finite temperature system which contacts with heat bath, Ignore the part of heat bath and consider a physical state given by a statistical average (Mixed state). Density matrix A physical observable Density matrix and Entropy Density matrix of a pure state Density matrix of canonical ensemble Thermodynamic entropy Free energy von Neumann entropy Entanglement entropy Decompose a quantum many-body system into its subsystems A and B. Trace out the density matrix over the Hilbert space of B. The reduced density matrix Entanglement entropy Nishioka, Ryu, Takayanagi (2009) Entanglement entropy Consider a system of two electron spins Pure state The reduced density matrix : Mixed state Entanglement entropy : Maximum: EPR pair Quantum entanglement Entanglement entropy ✓ When the total system is a pure state, The entanglement entropy represents the strength of quantum entanglement. ✓ When the total system is a mixed state, The entanglement entropy includes thermodynamic entropy. Any application to particle phenomenology or cosmology ?? Decoupling in the Universe Particles A are no longer in thermal contact with particles B when the interaction rate is smaller than the Hubble expansion rate. e.g. neutrino decoupling Decoupled subsystems are treated separately. Neutrino temperature drops independently from photon temperature.
    [Show full text]
  • Chapter 4 Information Theory
    Chapter Information Theory Intro duction This lecture covers entropy joint entropy mutual information and minimum descrip tion length See the texts by Cover and Mackay for a more comprehensive treatment Measures of Information Information on a computer is represented by binary bit strings Decimal numb ers can b e represented using the following enco ding The p osition of the binary digit 3 2 1 0 Bit Bit Bit Bit Decimal Table Binary encoding indicates its decimal equivalent such that if there are N bits the ith bit represents N i the decimal numb er Bit is referred to as the most signicant bit and bit N as the least signicant bit To enco de M dierent messages requires log M bits 2 Signal Pro cessing Course WD Penny April Entropy The table b elow shows the probability of o ccurrence px to two decimal places of i selected letters x in the English alphab et These statistics were taken from Mackays i b o ok on Information Theory The table also shows the information content of a x px hx i i i a e j q t z Table Probability and Information content of letters letter hx log i px i which is a measure of surprise if we had to guess what a randomly chosen letter of the English alphab et was going to b e wed say it was an A E T or other frequently o ccuring letter If it turned out to b e a Z wed b e surprised The letter E is so common that it is unusual to nd a sentence without one An exception is the page novel Gadsby by Ernest Vincent Wright in which
    [Show full text]
  • Shannon Entropy and Kolmogorov Complexity
    Information and Computation: Shannon Entropy and Kolmogorov Complexity Satyadev Nandakumar Department of Computer Science. IIT Kanpur October 19, 2016 This measures the average uncertainty of X in terms of the number of bits. Shannon Entropy Definition Let X be a random variable taking finitely many values, and P be its probability distribution. The Shannon Entropy of X is X 1 H(X ) = p(i) log : 2 p(i) i2X Shannon Entropy Definition Let X be a random variable taking finitely many values, and P be its probability distribution. The Shannon Entropy of X is X 1 H(X ) = p(i) log : 2 p(i) i2X This measures the average uncertainty of X in terms of the number of bits. The Triad Figure: Claude Shannon Figure: A. N. Kolmogorov Figure: Alan Turing Just Electrical Engineering \Shannon's contribution to pure mathematics was denied immediate recognition. I can recall now that even at the International Mathematical Congress, Amsterdam, 1954, my American colleagues in probability seemed rather doubtful about my allegedly exaggerated interest in Shannon's work, as they believed it consisted more of techniques than of mathematics itself. However, Shannon did not provide rigorous mathematical justification of the complicated cases and left it all to his followers. Still his mathematical intuition is amazingly correct." A. N. Kolmogorov, as quoted in [Shi89]. Kolmogorov and Entropy Kolmogorov's later work was fundamentally influenced by Shannon's. 1 Foundations: Kolmogorov Complexity - using the theory of algorithms to give a combinatorial interpretation of Shannon Entropy. 2 Analogy: Kolmogorov-Sinai Entropy, the only finitely-observable isomorphism-invariant property of dynamical systems.
    [Show full text]