Lecture 21: Computing Capacities, Coding in Practice, & Review

Lecture 21: Computing Capacities, Coding in Practice, & Review

COMP2610/6261 - Information Theory Lecture 21: Computing Capacities, Coding in Practice, & Review ANU Logo UseMark Guidelines Reid and Aditya Menon Research School of Computer Science The ANU logo is a contemporary The Australian National University refection of our heritage. It clearly presents our name, our shield and our motto: First to learn the nature of things. To preserve the authenticity of our brand identity, there are rules that govern how our logo is used. Preferred logo Black version Preferred logo - horizontal logo October 14, 2013 The preferred logo should be used on a white background. This version includes black text with the crest in Deep Gold in either PMS or CMYK. Black Where colour printing is not available, the black logo can be used on a white background. Reverse Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 1 / 21 The logo can be used white reversed out of a black background, or occasionally a neutral dark background. Deep Gold Black C30 M50 Y70 K40 C0 M0 Y0 K100 PMS Metallic 8620 PMS Process Black PMS 463 Reverse version Any application of the ANU logo on a coloured background is subject to approval by the Marketing Offce, contact [email protected] LOGO USE GUIDELINES 1 THE AUSTRALIAN NATIONAL UNIVERSITY 1 Computing Capacities 2 Good Codes vs. Practical Codes 3 Linear Codes 4 Coding: Review Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 2 / 21 1 Computing Capacities 2 Good Codes vs. Practical Codes 3 Linear Codes 4 Coding: Review Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 3 / 21 1 Compute the mutual information I (X ; Y ) for a general pX = (p0; p1) 2 Determine which choice of pX maximises I (X ; Y ) 3 Use that maximising value to determine C Binary Symmetric Channel: We first consider the binary symmetric channel with X = Y = 0; 1 A A f g and flip probability f . It has transition matrix 1 f f Q = − f 1 f − Computing Capacities Recall the definition of capacity for a channel Q with inputs X and A ouputs Y A C = max I (X ; Y ) pX How do we actually calculate this quantity? Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 4 / 21 Binary Symmetric Channel: We first consider the binary symmetric channel with X = Y = 0; 1 A A f g and flip probability f . It has transition matrix 1 f f Q = − f 1 f − Computing Capacities Recall the definition of capacity for a channel Q with inputs X and A ouputs Y A C = max I (X ; Y ) pX How do we actually calculate this quantity? 1 Compute the mutual information I (X ; Y ) for a general pX = (p0; p1) 2 Determine which choice of pX maximises I (X ; Y ) 3 Use that maximising value to determine C Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 4 / 21 Computing Capacities Recall the definition of capacity for a channel Q with inputs X and A ouputs Y A C = max I (X ; Y ) pX How do we actually calculate this quantity? 1 Compute the mutual information I (X ; Y ) for a general pX = (p0; p1) 2 Determine which choice of pX maximises I (X ; Y ) 3 Use that maximising value to determine C Binary Symmetric Channel: We first consider the binary symmetric channel with X = Y = 0; 1 A A f g and flip probability f . It has transition matrix 1 f f Q = − f 1 f − Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 4 / 21 In general, q := pY = QpX , so above calculation is just (1 f ) f p0 q = pY = − f (1 f ) p1 − Using H2(q) = q log q (1 q) log (1 q) and letting − 2 − − 2 − q = q1 = P(y = 1) we see the entropy H(Y ) = H2(q1) = H2(fp0 + (1 f )p1) − Computing Capacities Binary Symmetric Channel - Step 1 The mutual information can be expressed as I (X ; Y ) = H(Y ) H(Y X ). − j We therefore need to compute two terms: H(Y ) and H(Y X ) so we need j the distributions P(y) and P(y x). j Computing H(Y ): P(y = 0) = (1 f )P(x = 0) + fP(x = 1) = (1 f )p0 + fp1 − − P(y = 1) = (1 f )P(x = 1) + fP(x = 0) = fp0 + (1 f )p1 − − Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 5 / 21 Using H2(q) = q log q (1 q) log (1 q) and letting − 2 − − 2 − q = q1 = P(y = 1) we see the entropy H(Y ) = H2(q1) = H2(fp0 + (1 f )p1) − Computing Capacities Binary Symmetric Channel - Step 1 The mutual information can be expressed as I (X ; Y ) = H(Y ) H(Y X ). − j We therefore need to compute two terms: H(Y ) and H(Y X ) so we need j the distributions P(y) and P(y x). j Computing H(Y ): P(y = 0) = (1 f )P(x = 0) + fP(x = 1) = (1 f )p0 + fp1 − − P(y = 1) = (1 f )P(x = 1) + fP(x = 0) = fp0 + (1 f )p1 − − In general, q := pY = QpX , so above calculation is just (1 f ) f p0 q = pY = − f (1 f ) p1 − Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 5 / 21 Computing Capacities Binary Symmetric Channel - Step 1 The mutual information can be expressed as I (X ; Y ) = H(Y ) H(Y X ). − j We therefore need to compute two terms: H(Y ) and H(Y X ) so we need j the distributions P(y) and P(y x). j Computing H(Y ): P(y = 0) = (1 f )P(x = 0) + fP(x = 1) = (1 f )p0 + fp1 − − P(y = 1) = (1 f )P(x = 1) + fP(x = 0) = fp0 + (1 f )p1 − − In general, q := pY = QpX , so above calculation is just (1 f ) f p0 q = pY = − f (1 f ) p1 − Using H2(q) = q log q (1 q) log (1 q) and letting − 2 − − 2 − q = q1 = P(y = 1) we see the entropy H(Y ) = H2(q1) = H2(fp0 + (1 f )p1) − Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 5 / 21 X X = H2(f )P(x) = H2(f ) P(x) = H2(f ) x x So, X H(Y X ) = H(Y x)P(x) j x j Computing I (X ; Y ): Putting it all together gives I (X ; Y ) = H(Y ) H(Y X ) = H2(fp0 + (1 f )p1) H2(f ) − j − − Computing Capacities Binary Symmetric Channel - Step 1 Computing H(Y X ): j Since P(y x) is described by the matrix Q, we have j H(Y x = 0)= H2(P(y = 1 x = 0)) = H2(Q1 0) = H2(f ) j j ; and similarly, H(Y x = 1)= H2(P(y = 1 x = 1)) = H2(Q0 1) = H2(f ) j j ; Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 6 / 21 X X = H2(f )P(x) = H2(f ) P(x) = H2(f ) x x Computing I (X ; Y ): Putting it all together gives I (X ; Y ) = H(Y ) H(Y X ) = H2(fp0 + (1 f )p1) H2(f ) − j − − Computing Capacities Binary Symmetric Channel - Step 1 Computing H(Y X ): j Since P(y x) is described by the matrix Q, we have j H(Y x = 0)= H2(P(y = 1 x = 0)) = H2(Q1 0) = H2(f ) j j ; and similarly, H(Y x = 1)= H2(P(y = 1 x = 1)) = H2(Q0 1) = H2(f ) j j ; So, X H(Y X ) = H(Y x)P(x) j x j Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 6 / 21 Computing Capacities Binary Symmetric Channel - Step 1 Computing H(Y X ): j Since P(y x) is described by the matrix Q, we have j H(Y x = 0)= H2(P(y = 1 x = 0)) = H2(Q1 0) = H2(f ) j j ; and similarly, H(Y x = 1)= H2(P(y = 1 x = 1)) = H2(Q0 1) = H2(f ) j j ; So, X X X H(Y X ) = H(Y x)P(x) = H2(f )P(x) = H2(f ) P(x) = H2(f ) j x j x x Computing I (X ; Y ): Putting it all together gives I (X ; Y ) = H(Y ) H(Y X ) = H2(fp0 + (1 f )p1) H2(f ) − j − − Mark Reid and Aditya Menon (ANU) COMP2610/6261 - Information Theory Oct. 14, 2014 6 / 21 Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981 You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links. 9.6: The noisy-channel coding theorem 151 How much better can we do? By symmetry, the optimal input distribu- tion is 0.5, 0.5 and the capacity is { } I(X; Y ) BSC (f = 0) and p = (0:5; 0:5): C(Q ) = H (0.5) H (0.15)X= 1.0 0.61 = 0.39 bits. (9.11) BSC 2 − 2 − 0.4 I (X ; Y ) = H2(0:5) H2(0) = 1 We’ll justify the symmetry argumen−t later. If there’s any doubt about 0.3 the symmetryBSCargumen (f =t, 0:w15)e can andalwaysp resort= (0to:5explicit; 0:5):maximization X 0.2 of the mutual information I(X; Y ), I (X ; Y ) = H2(0:5) H2(0:15) 0:39 − ≈ 0.1 I(X; Y ) = H2((1 f)p1 + (1 p1)f) H2(f) (figure 9.2).

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    54 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us