DOCSLIB.ORG

6.02 Lecture 1: Overview: Information and Entropy

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
6.02 Lecture 1: Overview: Information and Entropy 6.02 Fall 2012 Lecture #1 • Digital vs. analog communication • The birth of modern digital communication • Information and entropy • Codes, Huffman coding 6.02 Fall 2012 Lecture 1, Slide #1 A9AR(DWQf F(DG9GEEVFA0(SGF • GEEVFA0(SF9(0GFSFVGVQsSE2X(W23GPE t2f9fdWGDR(923PGE(EA0PGH@GF2udWA((EHDARV12EG1VD(SGF t uGP3P2IV2F0`EG1VD(SGFt&uGPfff – F(DG92D20RPGFA0Q – &A12DAR`RGR@2X(W23GPE • ! GEEVFA0(SF9(E2QQ(920GEHPAQAF9(1AQ0P2R2s SE2Q2IV2F02G3Q`E)GDQ3PGEQGE2QGVP02(DH@()2R – 82F0G121GFRGQGE2GR@2PQ2IV2F02G3Q`E)GDQR@(RhQ(1(HR21RG R@20GEEVFA0(SGF0@(FF2Dd2f9fd)AF(P`1A9ARQd(F1f – 82FAFWGDWAF9(F(DG90GEEVFA0(SGF(0PGQQR@2H@`QA0(D0@(FF2D – &A12DAR`RGR@2E2QQ(92 – $2DDQVAR21RGPA1AF9R@2QR(992PAF99PGXR@AF0GEHVR(SGF(DHGX2Pd QRGP(92d)A91(R(dg 6.02 Fall 2012 Lecture 1, Slide #3 6.02 Syllabus Point-to-point communication channels (transmitterreceiver): • Encoding information BITS • Models of communication channels SIGNALS • Noise, bit errors, error correction • Sharing a channel Multi-hop networks: • Packet switching, efficient routing PACKETS • Reliable delivery on top of a best-efforts network 6.02 Fall 2012 Lecture 1, Slide #4 Samuel&f0fGPQ2 • FW2FR21tGFX(P1QdH(R2FRwdAFuR@2EGQRHP(0S0(D3GPE G32D20RPA0(DR2D29P(H@`dAF0DV1AF9C2`QdXAP2(PP(F92E2FRQd 2D20RPGE(9F2RQdE(PCAF912WA02QdP2D(`Qdgd(F1GPQ20G12c • $GPC21SP2D2QQD`RG2QR()DAQ@R@2R20@FGDG9` • 82PAFAS(DQRPV99D2QdR2D29P(H@`X(QIVA0CD`(1GHR21(F1XA12D` 12HDG`21 – !P(FQs RD(FS00()D2(U2EHRQ t@GVPQRGQ2F1XGP1Q3PGEV22F #A0RGPA(RGP2QA12FR0V0@(F(Fcud d d5F(DD`QV002QQAFtXGP1Qq EAFVR2u – !P(FQs0GFSF2FR(D" AFt2420SW2D`2F121R@2GF`EYHP2QQu – !P(FQs(0A50 • !2D29P(H@`RP(FQ3GPE210GEEVFA0(SGFtRP(FQs RD(FS0SE23PGE1(`Q )`Q@AHRGEAFVR2Q)`R2D29P(H@u(F10GEE2P02d(DQGQHVPP21E(BGP 12W2DGHE2FRQAFEER@2GP`vHP(0S02tI2FP`d2DWAFdI2(WAQA12dVHAFdgu 6.02 Fall 2012 Lecture 1, Slide #7 &(QRs3GPX(P1`2(PQ • #A( – !2D2H@GF2tiEHPGW2E2FRAF!2D29P(H@`pdH(R2FRw d d02DDu – $AP2D2QQR2D29P(H@`t(P0GFAu – P(1AGt&2QQ2F12Fu – &P(1AGt PEQRPGF9u – !2D2WAQAGF)PG(10(QSF9)`R@200tu – g • 02DD()Q9(D(Y`G3P2Q2(P0@2PQ – `IVAQRd0G12dI(PRD2`dg 6.02 Fall 2012 Lecture 1, Slide #13 Claude E. Shannon, 1916-2001 (QR2PQR@2QAQdEE2HRd! FRPG1V021(HHDA0(SGFG30GGD2(F (D92)P(RGDG9A00AP0VARQd(F1WA02W2PQ(f #2P`AF7V2FS(DAF1A9AR(D0AP0VAR12QA9Ff iGQRAEHGPR(FR(QR2PQR@2QAQG3R@202FRVP`p @d(R@2HRd! !G(F(D`a2R@21`F(EA0QG32F12DA(F Photograph © source unknown. All rights reserved.This HGHVD(SGFQf content is excluded from our Creative Commons license. For more information, see http://ocw.mit.edu/fairuse. GAF2102DD()QAFf MIT faculty 1956-1978 i E(R@2E(S0(DR@2GP`G30P`HRG9P(H@`p q ! "%(&' 6.02 Fall 2012 Lecture 1, Slide #14 A probabilistic theory requires at least a one-slide checklist on Probabilistic Models! • "FAW2PQ2G32D2E2FR(P`GVR0GE2QQdQdgdQdgfF2(F1GFD`GF2 GVR0GE2AF2(0@2YH2PAE2FRGPPVFG3R@2EG12Df • EW2FRQ d0ddg(P2QV)Q2RQG3GVR0GE2Qf$2Q(`2W2FR @(QG00VPP21A3R@2 GVR0GE2G3R@22YH2PAE2FRDA2QAF f • EW2FRQ3GPE(Fi(D92)P(pG3Q2RQdAf2fd 0tVFAGFd(DQGXPAU2F 0uAQ(F 2W2FRd 0tAFR2PQ20SGFd(DQGXPAU2F 0uAQ(F2W2FRd t0GEHD2E2FRd (DQGXPAU2F 0uAQ(F2W2FRf G(F1R@2FVDDQ2R$(P2(DQG2W2FRQf • PG)()ADAS2Q(P2125F21GF2W2FRQdQV0@R@(Rt udtud(F1 t 0ut ut0uA3 (F10(P2EVRV(DD`2Y0DVQAW2dAf2fA3 0$fGP2 92F2P(DD`dt 0ut ut0ust 0uf • EW2FRQ d0dddEd2R0fd(P2Q(A1RG)2tEVRV(DD`uAF12H2F12FRA3BGAFR HPG)()ADAR`G32W2P`0GE)AF(SGFG3R@2Q22W2FRQ3(0RGPQAFRGHPG1V0RG3 AF1AWA1V(DHPG)()ADAS2QdQGt 0Eut ut0utututEudt 0ut u t0ututudt Eut ututEud2R0f • GF1ASGF(DHPG)()ADAR`t d9AW2FR@(R0@(QG00VPP21ut r0ut 0uqt0uf • EYH20R21W(DV2G3(P(F1GEW(PA()D2AQR@2HPG)()ADAR`sX2A9@R21(W2P(92f 6.02 Fall 2012 Lecture 1, Slide #15 Measuring Information Shannon’s (and Hartley’s) definition of the information obtained on being told the outcome s i of a probabilistic experiment S : ⎛ ⎞ = = ⎜ 1 ⎟ I(S si ) log2 ⎝ pS (si )⎠ = where p S ( s i ) is the probability of the event S s i . The unit of measurement (when the log is base-2) is the bit (binary information unit --- not the same as binary digit!). 1 bit of information corresponds to = p S ( s i ) 0.5 . So, for example, when the outcome of a fair coin toss is revealed to us, we have received 1 bit of information. “Information is the resolution of uncertainty” Image by MIT OpenCourseWare. Shannon 6.02 Fall 2012 Lecture 1, Slide #16 Examples We’re drawing cards at random from a standard N=52-card deck. Elementary outcome: card that’s drawn, probability 1/52, information log2(52/1) = 5.7 bits. For an event comprising M such (mutually exclusive) outcomes, the probability is M/52. Q. If I tell you the card is a spade , how many bits of information have you received? A. Out of N=52 equally probable cards, M=13 are spades , so probability of drawing a spade is 13/52, and the amount of information received is log2(52/13) = 2 bits. This makes sense, we can encode one of the 4 (equally probable) suits using 2 binary digits, e.g., 00=, 01=, 10=, 11=. Q. If instead I tell you the card is a seven, how much info? A. N=52, M=4, so info = log2(52/4) = log2(13) = 3.7 bits 6.02 Fall 2012 Lecture 1, Slide #17 PGH2PS2QG3F3GPE(SGF125FASGF • DGX2PsHPG)()ADAR`GVR0GE2`A2D1Q@A9@2PAF3GPE(SGF • @A9@D`AF3GPE(SW2GVR0GE21G2QFGRF202QQ(PAD`E2(F( EGP2W(DV()D2GVR0GE2dGFD`(EGP2QVPHPAQAF9GVR0GE2dAf2fd R@2P2hQFGAFRPAFQA0W(DV2)2AF9(QQ2QQ21t0(FR@AFCG3 AF3GPE(SGF(Q129P22G3QVPHPAQ2u • 82FVQ213(0Re!@2AF3GPE(SGFAFAF12H2F12FR2W2FRQAQ (11ASW2ft(VSGFeR@GV9@AF12H2F12F02AQQV60A2FR3GP (11ASWAR`dARAQFGRF202QQ(P`d)20(VQ2X20(F@(W2t 0u t ut0utu2W2FX@2F d0d(P2FGRAF12H2F12FRsss AF12H2F12F02P2IVAP2QR@2H(APXAQ2BGAFRHPG)()ADAS2QRG(DQG 3(0RGPfu 6.02 Fall 2012 Lecture 1, Slide #18 Expected Information as Uncertainty or Entropy Consider a discrete random variable S , which may represent the set of possible symbols to be transmitted at a particular time, taking possible values s 1 , s 2 ,..., s N , with respective probabilities p S ( s 1 ), p S ( s 2 ),..., p S ( s N ) . The entropy H ( S ) of S is the expected (or mean or average) value of the information obtained by learning the outcome of S : N N ⎛ ⎞ = = = ⎜ 1 ⎟ H(S) ∑ pS (si )I(S si ) ∑ pS (si )log2 i=1 i=1 ⎝ pS (si )⎠ When all the p S ( s i ) are equal (with value 1/ N ), then = = H (S) H(S) log2 N or N 2 6.02 Fall 2012 This is the maximum attainable value! Lecture 1, Slide #19 e.g., Binary entropy function h(p) h(p) 1.0 Heads (or C=1) with probability p Tails (or C=0) with 0.5 probability 1 − p p 0 0 0.5 1.0 =− − − − = H(C) plog2 p (1 p)log2 (1 p) h(p) 6.02 Fall 2012 Lecture 1, Slide #20 Connection to (Binary) Coding • Suppose p=1/1024, i.e., very small probability of getting a Head, typically one Head in 1024 trials. Then = + h(p) (1/1024)log2 (1024 /1) (1023/1024)log2 (1024 /1023) =.0112 bits of uncertainty or information per trial on average • So using 1024 binary digits (C=0 or 1) to code the results of 1024 tosses of this particular coin seems inordinately wasteful, i.e., 1 binary digit per trial. Can we get closer to an average of .0112 binary digits/trial? • Yes! Confusingly, a binary digit is also referred to as a bit! • Binary coding: Mapping source symbols to binary digits 6.02 Fall 2012 Lecture 1, Slide #21 Significance of Entropy Entropy (in bits) tells us the average amount of information (in bits) that must be delivered in order to resolve the uncertainty about the outcome of a trial. This is a lower bound on the number of binary digits that must, on the average, be used to encode our messages. If we send fewer binary digits on average, the receiver will have some uncertainty about the outcome described by the message. If we send more binary digits on average, we’re wasting the capacity of the communications channel by sending binary digits we don’t have to. Achieving the entropy lower bound is the “gold standard” for an encoding (at least from the viewpoint of information compression). 6.02 Fall 2012 Lecture 1, Slide #22 Fixed-length Encodings An obvious choice for encoding equally probable outcomes is to choose a fixed-length code that has enough sequences to encode the necessary information • 96 printing characters 7-”bit” ASCII • Unicode characters UTF-16 • 10 decimal digits 4-”bit” BCD (binary coded decimal) Fixed-length codes have some advantages: • They are “random access” in the sense that to decode the nth message symbol one can decode the nth fixed- length sequence without decoding sequence 1 through n-1. • Table lookup suffices for encoding and decoding 6.02 Fall 2012 Lecture 1, Slide #23 Now consider: choicei pi log2(1/pi) A 1/3 1.58 bits B 1/2 1 bit C 1/12 3.58 bits D 1/12 3.58 bits The expected information content in a choice is given by the entropy: = (.333)(1.58) + (.5)(1) + (2)(.083)(3.58) = 1.626 bits Can we find an encoding where transmitting 1000 choices requires 1626 binary digits on the average? The natural fixed-length encoding uses two binary digits for each choice, so transmitting the results of 1000 choices requires 2000 binary digits. 6.02 Fall 2012 Lecture 1, Slide #24 Variable-length encodings (David Huffman, in term paper for MIT graduate class, 1951) Use shorter bit sequences for high probability choices, longer sequences for less probable choices BC A BA D 011010010111 Expected length choicei pi encoding =(.333)(2)+(.5)(1)+(2)(.083)(3) A 1/3 10 0 1 = 1.666 bits B 1/2 0 B 0 1 Transmitting 1000 C 1/12 110 A 0 1 choices takes an average of 1666 bits… D 1/12 111 C D better but not optimal Huffman Decoding Tree Note: The symbols are at the leaves of the tree; necessary and sufficient for instantaneously decodability. 6.02 Fall 2012 Lecture 1, Slide #25 Huffman’s Coding Algorithm • Begin with the set S of symbols to be encoded as binary strings, together with the probability p(s) for each symbol s in S.
Load more

Recommended publications
  • Information Content and Error Analysis
    43 INFORMATION CONTENT AND ERROR ANALYSIS Clive D Rodgers Atmospheric, Oceanic and Planetary Physics University of Oxford ESA Advanced Atmospheric Training Course September 15th – 20th, 2008 44 INFORMATION CONTENT OF A MEASUREMENT Information in a general qualitative sense: Conceptually, what does y tell you about x? We need to answer this to determine if a conceptual instrument design actually works • to optimise designs • Use the linear problem for simplicity to illustrate the ideas. y = Kx + ! 45 SHANNON INFORMATION The Shannon information content of a measurement of x is the change in the entropy of the • probability density function describing our knowledge of x. Entropy is defined by: • S P = P (x) log(P (x)/M (x))dx { } − Z M(x) is a measure function. We will take it to be constant. Compare this with the statistical mechanics definition of entropy: • S = k p ln p − i i i X P (x)dx corresponds to pi. 1/M (x) is a kind of scale for dx. The Shannon information content of a measurement is the change in entropy between the p.d.f. • before, P (x), and the p.d.f. after, P (x y), the measurement: | H = S P (x) S P (x y) { }− { | } What does this all mean? 46 ENTROPY OF A BOXCAR PDF Consider a uniform p.d.f in one dimension, constant in (0,a): P (x)=1/a 0 <x<a and zero outside.The entropy is given by a 1 1 S = ln dx = ln a − a a Z0 „ « Similarly, the entropy of any constant pdf in a finite volume V of arbitrary shape is: 1 1 S = ln dv = ln V − V V ZV „ « i.e the entropy is the log of the volume of state space occupied by the p.d.f.
    [Show full text]
  • Chapter 4 Information Theory
    Chapter Information Theory Intro duction This lecture covers entropy joint entropy mutual information and minimum descrip tion length See the texts by Cover and Mackay for a more comprehensive treatment Measures of Information Information on a computer is represented by binary bit strings Decimal numb ers can b e represented using the following enco ding The p osition of the binary digit 3 2 1 0 Bit Bit Bit Bit Decimal Table Binary encoding indicates its decimal equivalent such that if there are N bits the ith bit represents N i the decimal numb er Bit is referred to as the most signicant bit and bit N as the least signicant bit To enco de M dierent messages requires log M bits 2 Signal Pro cessing Course WD Penny April Entropy The table b elow shows the probability of o ccurrence px to two decimal places of i selected letters x in the English alphab et These statistics were taken from Mackays i b o ok on Information Theory The table also shows the information content of a x px hx i i i a e j q t z Table Probability and Information content of letters letter hx log i px i which is a measure of surprise if we had to guess what a randomly chosen letter of the English alphab et was going to b e wed say it was an A E T or other frequently o ccuring letter If it turned out to b e a Z wed b e surprised The letter E is so common that it is unusual to nd a sentence without one An exception is the page novel Gadsby by Ernest Vincent Wright in which
    [Show full text]
  • Shannon Entropy and Kolmogorov Complexity
    Information and Computation: Shannon Entropy and Kolmogorov Complexity Satyadev Nandakumar Department of Computer Science. IIT Kanpur October 19, 2016 This measures the average uncertainty of X in terms of the number of bits. Shannon Entropy Definition Let X be a random variable taking finitely many values, and P be its probability distribution. The Shannon Entropy of X is X 1 H(X ) = p(i) log : 2 p(i) i2X Shannon Entropy Definition Let X be a random variable taking finitely many values, and P be its probability distribution. The Shannon Entropy of X is X 1 H(X ) = p(i) log : 2 p(i) i2X This measures the average uncertainty of X in terms of the number of bits. The Triad Figure: Claude Shannon Figure: A. N. Kolmogorov Figure: Alan Turing Just Electrical Engineering \Shannon's contribution to pure mathematics was denied immediate recognition. I can recall now that even at the International Mathematical Congress, Amsterdam, 1954, my American colleagues in probability seemed rather doubtful about my allegedly exaggerated interest in Shannon's work, as they believed it consisted more of techniques than of mathematics itself. However, Shannon did not provide rigorous mathematical justification of the complicated cases and left it all to his followers. Still his mathematical intuition is amazingly correct." A. N. Kolmogorov, as quoted in [Shi89]. Kolmogorov and Entropy Kolmogorov's later work was fundamentally influenced by Shannon's. 1 Foundations: Kolmogorov Complexity - using the theory of algorithms to give a combinatorial interpretation of Shannon Entropy. 2 Analogy: Kolmogorov-Sinai Entropy, the only finitely-observable isomorphism-invariant property of dynamical systems.
    [Show full text]
  • Understanding Shannon's Entropy Metric for Information
    Understanding Shannon's Entropy metric for Information Sriram Vajapeyam [email protected] 24 March 2014 1. Overview Shannon's metric of "Entropy" of information is a foundational concept of information theory [1, 2]. Here is an intuitive way of understanding, remembering, and/or reconstructing Shannon's Entropy metric for information. Conceptually, information can be thought of as being stored in or transmitted as variables that can take on different values. A variable can be thought of as a unit of storage that can take on, at different times, one of several different specified values, following some process for taking on those values. Informally, we get information from a variable by looking at its value, just as we get information from an email by reading its contents. In the case of the variable, the information is about the process behind the variable. The entropy of a variable is the "amount of information" contained in the variable. This amount is determined not just by the number of different values the variable can take on, just as the information in an email is quantified not just by the number of words in the email or the different possible words in the language of the email. Informally, the amount of information in an email is proportional to the amount of “surprise” its reading causes. For example, if an email is simply a repeat of an earlier email, then it is not informative at all. On the other hand, if say the email reveals the outcome of a cliff-hanger election, then it is highly informative.
    [Show full text]
  • Information, Entropy, and the Motivation for Source Codes
    MIT 6.02 DRAFT Lecture Notes Last update: September 13, 2012 CHAPTER 2 Information, Entropy, and the Motivation for Source Codes The theory of information developed by Claude Shannon (MIT SM ’37 & PhD ’40) in the late 1940s is one of the most impactful ideas of the past century, and has changed the theory and practice of many fields of technology. The development of communication systems and networks has benefited greatly from Shannon’s work. In this chapter, we will first develop the intution behind information and formally define it as a mathematical quantity, then connect it to a related property of data sources, entropy. These notions guide us to efficiently compress a data source before communicating (or storing) it, and recovering the original data without distortion at the receiver. A key under­ lying idea here is coding, or more precisely, source coding, which takes each “symbol” being produced by any source of data and associates it with a codeword, while achieving several desirable properties. (A message may be thought of as a sequence of symbols in some al­ phabet.) This mapping between input symbols and codewords is called a code. Our focus will be on lossless source coding techniques, where the recipient of any uncorrupted mes­ sage can recover the original message exactly (we deal with corrupted messages in later chapters). ⌅ 2.1 Information and Entropy One of Shannon’s brilliant insights, building on earlier work by Hartley, was to realize that regardless of the application and the semantics of the messages involved, a general definition of information is possible.
    [Show full text]
  • Entropy in Classical and Quantum Information Theory
    Entropy in Classical and Quantum Information Theory William Fedus Physics Department, University of California, San Diego. Entropy is a central concept in both classical and quantum information theory, measuring the uncertainty and the information content in the state of a physical system. This paper reviews classical information theory and then proceeds to generalizations into quantum information theory. Both Shannon and Von Neumann entropy are discussed, making the connection to compressibility of a message stream and the generalization of compressibility in a quantum system. Finally, the paper considers the application of Von Neumann entropy in entanglement of formation for both pure and mixed bipartite quantum states. CLASSICAL INFORMATION THEORY information. For instance, if we use a block code which assigns integers to typical sequences, the information in In statistical mechanics, entropy is the logarithm of a string of n letters can be compressed to H(A) bits. the number of arrangements a system can be configured With this framework and definition, we may now con- and still remain consistent with the thermodyanmic ob- sider the maximum compression of a length n message servables. From this original formulation, entropy has without loss of information. The number of bits neces- grown to become an important element in many diverse sary to transmit the message is given by fields of study. One of the first examples was in 1948 when Claude Shannon adopted entropy as a measure of the uncertainty in a random variable, or equivalently, the H(An) = nH(A): (2) expected value of information content within a message. Classical information theory, as established by Claude Shannon, sought to resolve two central issues in signal which simply states that one needs (n times the entropy processing of the ensemeble A)-bits.
    [Show full text]
  • An Introduction to Information Theory and Entropy
    An introduction to information theory and entropy Tom Carter CSU Stanislaus http://astarte.csustan.edu/~ tom/SFI-CSSS [email protected] Complex Systems Summer School Santa Fe March 14, 2014 1 Contents . Measuring complexity 5 . Some probability ideas 9 . Basics of information theory 15 . Some entropy theory 22 . The Gibbs inequality 28 . A simple physical example (gases) 36 . Shannon's communication theory 47 . Application to Biology (genomes) 63 . Some other measures 79 . Some additional material . Examples using Bayes' Theorem 87 . Analog channels 103 . A Maximum Entropy Principle 108 . Application: Economics I 111 . Application: Economics II 117 . Application to Physics (lasers) 124 . Kullback-Leibler information measure 129 . References 135 2 The quotes } Science, wisdom, and counting } Being different { or random } Surprise, information, and miracles } Information (and hope) } H (or S) for Entropy } Thermodynamics } Language, and putting things together } Tools To topics 3 Science, wisdom, and counting \Science is organized knowledge. Wisdom is organized life." - Immanuel Kant \My own suspicion is that the universe is not only stranger than we suppose, but stranger than we can suppose." - John Haldane \Not everything that can be counted counts, and not everything that counts can be counted." - Albert Einstein (1879-1955) \The laws of probability, so true in general, so fallacious in particular ." - Edward Gibbon 4 Measuring complexity • Workers in the field of complexity face a classic problem: how can we tell that the system we are looking at is actually a complex system? (i.e., should we even be studying this system? :-) Of course, in practice, we will study the systems that interest us, for whatever reasons, so the problem identified above tends not to be a real problem.
    [Show full text]
  • Introduction to Information Theory
    Introduction to Information Theory Entropy as a Measure of Information Content Entropy of a random variable. Let X be a random variable that takes on values from its domain P fx1; x2; : : : ; xng with respective probabilities p1; p2; : : : ; pn, where i pi = 1. Then the entropy of X, H(X), represents the average amount of information contained in X and is defined by n X H(X) = − pi log2 pi: i=1 Note that entropy is measured in bits. Also, Hn(p1; : : : ; pn) denotes another way of writing the entropy function. Arguments in favor of the above definition of information: • The definition is consistent with the following extreme cases: 1 1. If n = 2 and p1 = p2 = 2 , then H(X) = 1 bit; i.e. when an event (e.g. X = x1) has an equally likely chance of occurring or not occurring, then its outcome possesses one bit of information. This is the maximum amount of information a binary outcome may possess. 2. In the case when pi = 1 for some 1 ≤ i ≤ n, then H(X) = 0; i.e. any random variable whose outcome is certain possesses no information. • Moreover, the above definition is the only definition which satisfies the following three properties of information which seem reasonable under any definition: 1 1 { Normalization: H2( 2 ; 2 ) = 1 { Continuity: H2(p; 1 − p) is a continuous function of p on the interval (0; 1) { Grouping: p1 p2 Hm(p1; : : : ; pm) = Hm−1(p1 + p2; p3; : : : ; pm) + (p1 + p2)H2( ; ) p1 + p2 p1 + p2 1 Claude Shannon (1912-2001). Pioneer in • applying Boolean logic to electronic circuit design • studying the complexity of Boolean circuits • signal processing: determined lower bounds on the amount of samples needed to achieve a desired estimation accuracy • game theory: inventor of minimax algorithm • information theory: first to give a precise definition for the concept of information • coding theory: Channel Coding Theorem, Optimal Coding Theorem 2 Example 1.
    [Show full text]
  • Exploring Deep Learning Using Information Theory Tools and Patch Ordering
    Unde r revie w as a conf er e nc e pape r at ICLR 2019 EXPLORING DEEP LEARNING USING INFORMATION THEORY TOOLS AND PATCH ORDERING Anony m o us author s Pape r unde r double -blin d revie w ABSTRACT We present a framework for automatically ordering image patches that enables in-depth analysis of dataset relationship to learnability of a classification task using convolutional neural network. An image patch is a grou p of pixels resid ing in a contin u ou s area contain e d in the sample . Our preliminary experimental results show that an informed smart shuffling of patches at a sample level can expedite training by exposing important features at early stages of training. In addition, we conduct systematic experiments and provide evidence that CNN’s generalization capabilities do not correlate with human recognizable features present in training samples. We utilized the framework not only to show that spatial locality of features within samples do not correlate with generalization, but also to expedite convergence while achieving similar generalization performance. Using multiple network architectures and datasets, we show that ordering image regions using mutual information measure between adjacent patches, enables CNNs to converge in a third of the total steps required to train the same network without patch ordering. 1 INTRODUCTION Adva nc e s in Deep Lear nin g (DL) and Convolutional Neura l Netw or ks (CNN ) have dramatically impro ve d the state- of-the - ar t in compu te r vision tasks. Many of these breakthroughs are attributed to the succe ssiv e featu re extrac tion and an increa sin g abstr a ct representation of the underly ing training dat a using multi- stag e simple oper ation s such as convolution.
    [Show full text]
  • Interpretable Convolution Methods for Learning Genomic Sequence Motifs
    bioRxiv preprint doi: https://doi.org/10.1101/411934; this version posted September 8, 2018. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC 4.0 International license. Interpretable Convolution Methods for Learning Genomic Sequence Motifs MS Ploenzke1*, RA Irizarry1,2 1 Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA 2 Department of Biostatistics and Computational Biology, Dana-Farber Cancer Institute, Boston, MA, USA * Corresponding author: ploenzke(at)g.harvard.edu Abstract The first-layer filters employed in convolutional neural networks tend to learn, or extract, spatial features from the data. Within their application to genomic sequence data, these learned features are often visualized and interpreted by converting them to sequence logos; an information-based representation of the consensus nucleotide motif. The process to obtain such motifs, however, is done through post-training procedures which often discard the filter weights themselves and instead rely upon finding those sequences maximally correlated with the given filter. Moreover, the filters collectively learn motifs with high redundancy, often simply shifted representations of the same sequence. We propose a schema to learn sequence motifs directly through weight constraints and transformations such that the individual weights comprising the filter are directly interpretable as either position weight matrices (PWMs) or information gain matrices (IGMs). We additionally leverage regularization to encourage learning highly-representative motifs with low inter-filter redundancy. Through learning PWMs and IGMs directly we present preliminary results showcasing how our method is capable of incorporating previously-annotated database motifs along with learning motifs de novo and then outline a pipeline for how these tools may be used jointly in a data application.
    [Show full text]
  • Lecture 1: August 29 1.1 About the Class
    10-704: Information Processing and Learning Fall 2016 Lecture 1: August 29 Lecturer: Aarti Singh Note: These notes are based on scribed notes from Spring15 offering of this course. LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. 1.1 About the class This class focuses on information theory, signal processing, machine learning, and the connections between these fields. Both signal processing and machine learning are about how to extract useful information from signals and data, respectively. The distinction between these fields is vanishing nowadays, however it is still useful to point out some classic differences that still often serve as motivation for the respective fields. Classically signals in signal processing involve a temporal component and are usually transmitted over a channel, while data in machine learning is much more general (but we will see in this course how channels do arise in machine learning problems as well, and help characterize the fundamental limits of achievable error in machine learning problems). A more fundamental difference between signal processing and machine learning is that signals are often designed in the former or modeled using an understanding of the system physics. Such physics-driven models can enable inference i.e. answering related questions about the data generating system, whereas in ML, we do not have much control or understanding of the data generating distribution and the models are primarily used to make predictions. Information theory is a common thread that lays the foundations of signal processing and machine learning, helping us characterize the fundamental limits in these problems - number of bits needed to recover signals accurately and number of samples needed to learn models well.
    [Show full text]
  • Information Diffusion Prediction Via Recurrent Cascades Convolution
    2019 IEEE 35th International Conference on Data Engineering (ICDE) Information Diffusion Prediction via Recurrent Cascades Convolution Xueqin Chen∗, Fan Zhou∗†, Kunpeng Zhang‡, Goce Trajcevski§, Ting Zhong∗, Fengli Zhang∗ ∗School of Information and Software Engineering, University of Electronic Science and Technology of China ‡Department of Decision, Operations & Information Technologies, University of Maryland, college park MD §Department of Electrical and Computer Engineering, Iowa State University, Ames IA †Corresponding author: [email protected] Abstract—Effectively predicting the size of an information cas- for cascade prediction; (2) feature-based approaches – mostly cade is critical for many applications spanning from identifying focusing on identifying and incorporating complicated hand- viral marketing and fake news to precise recommendation and crafted features, e.g., structural [20]–[22], content [23]–[26], online advertising. Traditional approaches either heavily depend on underlying diffusion models and are not optimized for popu- temporal [27], [28], etc. Their performance strongly depends larity prediction, or use complicated hand-crafted features that on extracted features requiring extensive domain knowledge, cannot be easily generalized to different types of cascades. Recent which is hard to be generalized to new domains; (3) generative generative approaches allow for understanding the spreading approaches – typically relying on Hawkes point process [6], mechanisms, but with unsatisfactory prediction accuracy. [29],
    [Show full text]