Digital Communication Systems ECS 452

Home , Abbas El Gamal, Alain Glavieux, Claude Berrou, David Tse, Gottfried Ungerboeck, Jacob Wolfowitz

Asst. Prof. Dr. Prapun Suksompong (ผศ.ดร.ประพันธ ์ สขสมปองุ ) [email protected] 1. Intro to Digital Communication Systems Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40

1 Thursday 14:20-15:30 “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.”

Shannon, Claude. A Mathematical Theory Of Communication. (1948)

2 Shannon: Father of the Info. Age

 Documentary  Co-produced by the Jacobs School, UCSD-TV, and the California Institute for Telecommunications and Information Technology  Won a Gold award in the Biography category in the 2002 Aurora Awards.

3 [http://www.uctv.tv/shows/Claude-Shannon-Father-of-the-Information-Age-6090] [http://www.youtube.com/watch?v=z2Whj_nL-x8] C. E. Shannon (1916-2001)  1938 MIT master's thesis: A Symbolic Analysis of Relay and Switching Circuits  Insight: The binary nature of Boolean logic was analogous to the ones and zeros used by digital circuits.  The thesis became the foundation of practical digital circuit design.  The first known use of the term bit to refer to a “binary digit.”  Possibly the most important, and also the most famous, master’s thesis of the century.  It was simple, elegant, and important.

4 C. E. Shannon: Master Thesis

5 Boole/Shannon Celebration  Events in 2015 and 2016 centered around the work of  George Boole, who was born 200 years ago, and  Claude E. Shannon, born 100 years ago.  Events were scheduled both at  the University College Cork (UCC), Ireland and  the Massachusetts Institute of Technology (MIT)

 6 http://www.rle.mit.edu/booleshannon/ An Interesting Book  The Logician and the Engineer: How George Boole and Claude Shannon Created the Information Age  by Paul J. Nahin  ISBN: 9780691151007  http://press.princeton.edu/titles/ 9819.html

7 C. E. Shannon (Con’t)  1948: A Mathematical Theory of Communication  Bell System Technical Journal, vol. 27, pp. 379-423, July- October, 1948.  September 1949: Book published. Include a new section by Warren Weaver that applied Shannon's theory to  Invent Information Theory: human communication. Simultaneously founded the subject, introduced all of the  Create the architecture and major concepts, and stated and concepts governing digital proved all the fundamental communication. theorems. 8 A Mathematical Theory of Communication  Link posted in the “references” section of the website.

9 [An offprint from the Bell System Technical Journal] C. E. Shannon

10 …with some remarks by Toby Berger. Claude E. Shannon Award

Claude E. Shannon (1972) Elwyn R. Berlekamp (1993) Sergio Verdu (2007) David S. Slepian (1974) Aaron D. Wyner (1994) Robert M. Gray (2008) Robert M. Fano (1976) G. David Forney, Jr. (1995) Jorma Rissanen (2009) Peter Elias (1977) Imre Csiszár (1996) Te Sun Han (2010) Mark S. Pinsker (1978) Jacob Ziv (1997) Shlomo Shamai (Shitz) (2011) Jacob Wolfowitz (1979) Neil J. A. Sloane (1998) Abbas El Gamal (2012) W. Wesley Peterson (1981) Tadao Kasami (1999) Katalin Marton (2013) Irving S. Reed (1982) Thomas Kailath (2000) János Körner (2014) Robert G. Gallager (1983) Jack KeilWolf (2001) Arthur Robert Calderbank (2015) Solomon W. Golomb (1985) Toby Berger (2002) Alexander S. Holevo (2016) David Tse (2017) William L. Root (1986) Lloyd R. Welch (2003) James L. Massey (1988) Robert J. McEliece (2004) Thomas M. Cover (1990) Richard Blahut (2005) Andrew J. Viterbi (1991) Rudolf Ahlswede (2006)

11 [ http://www.itsoc.org/honors/claude-e-shannon-award ] [http://www.ieee.org/documents/hamming_rl.pdf] IEEE Richard W. Hamming Medal 1988 - Richard W. Hamming 2006 -Vladimir I. Levenshtein 1989 - Irving S. Reed 2007 - Abraham Lempel 1990 - Dennis M. Ritchie and Kenneth L. Thompson 2008 - Sergio Verdú 1991 - Elwyn R. Berlekamp 2009 - Peter Franaszek 1992 - Lotfi A. Zadeh 2010 -Whitfield Diffie, 1993 - Jorma J. Rissanen Martin Hellman, 1994 - Gottfried Ungerboeck and Ralph Merkle 1995 - Jacob Ziv 2011 -Toby Berger 1996 - Mark S. Pinsker 2012 - Michael Luby, Amin Shokrollahi 1997 -Thomas M. Cover 2013 - Arthur Robert Calderbank 1998 - David D. Clark 2014 -Thomas Richardson 1999 - David A. Huffman and Rüdiger L. Urbanke 2000 - Solomon W. Golomb 2015 - Imre Csiszar 2001 - A. G. Fraser 2016 - Abbas El Gamal 2002 - Peter Elias 2017 - Shlomo Shamai 2003 - Claude Berrou and Alain Glavieux 2018 - Erdal Arikan 2004 - Jack K. Wolf 2005 - Neil J.A. Sloane “For contributions to Information Theory, including 12 source coding and its applications.” [http://www.cvaieee.org/html/toby_berger.html] Information Theory The science of information theory tackles the following questions [Berger] 1. What is information, i.e., how do we measure it quantitatively? 2. What factors limit the reliability with which information generated at one point can be reproduced at another, and what are the resulting limits? 3. How should communication systems be designed in order to achieve or at least to approach these limits?

13 Elements of communication sys. (ECS 332)

Analog (continuous) Digital (discrete) Transmitted Received Message Message Signal Signal Information Transmitter Channel Receiver Destination Source Modulation Amplification Coding Demodulation Decoding Noise, Filtering Interference, Distortion + Transmission loss (attenuation)

[Shannon, 1948] 14 The Switch to Digital TV

Japan: Starting July 24, 2011, the analog broadcast has ceased and only digital broadcast is available. US: Since June 12, 2009, full- power television stations nationwide have been broadcasting exclusively in a digital format. Thailand: Use DVB-T2. Launched in 2014.

15 [https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Digital_broadcast_standards.svg/800px-Digital_broadcast_standards.svg.png] News: The Switch to Digital Radio  Norway (the mountainous nation of 5 million) is the first country to shut down its national FM radio network in favor of digital radio.  Start on January 11, 2017  At which point, 99.5% of the population has access to DAB reception with almost three million receivers sold.  70% of Norwegian households regularly tune in digitally  Take place over a 12-month period, conducting changes region by region.  December 13, 2017: All national networks are DAB-only.  Local broadcasters have five years to phase out their FM stations.  New format: Digital Audio Broadcasting (DAB)

http://gizmodo.com/norway-is-killing-fm-radio-tomorrow-1791019824 http://www.worlddab.org/country-information/norway http://www.smithsonianmag.com/smart-news/norway-killed-radio-star-180961761/ http://www.latimes.com/world/la-fg-norway-radio-20170114-story.html 16 https://www.newscientist.com/article/2117569-norway-is-first-country-to-turn-off-fm-radio-and-go-digital-only/ http://fortune.com/2017/12/18/norway-fm-radio-digital-audio-broadcasting/ Digital Audio Broadcasting  Initiated as a European research project in the 1980s.  The Norwegian Broadcasting Corporation (NRK) launched the first DAB channel in the world on 1 June 1995 (NRK Klassisk)  The BBC and Swedish Radio (SR) launched their first DAB digital radio broadcasts in September 1995.  Audio quality varies depending on the bitrate used.

17 The Switch to DAB in Norway  Co-exist with FM since 1995.  Provide a clearer and more reliable network that can better cut through the country's sparsely populated rocky terrain.  FM has always been problematic in Norway since the nation’s mountains and fjords makes getting clear FM signals difficult.  Offer more channels at a fraction of the cost.  Allow 8 times as many radio stations  Norway currently has five national FM radio stations.  With DAB, it will be able to have around 40.  The FM radio infrastructure was coming to the end of its life,  Need to either replace it or fully commit to DAB anyway  Can run at lower power levels  the infrastructure electricity bills are lower 18 The Switch to Digital Radio  Switzerland and Denmark are also interested in phasing out FM  Great Britain says it will look at making the switch  once 50 percent of listeners use digital formats  currently at 35 percent  Unlikely to happen before 2020.  and when the DAB signal reaches 90 percent of the population.  Germany had set a 2015 date for dumping FM many years ago, but lawmakers reversed that decision in 2011.  In North America,  FM radio, which has been active since the 1940s, shows no sign of being replaced any time soon, either in the United States or Canada.  There are around 4,000 stations using HD radio technology in the United States, and HD radio receivers are now common fixtures in new cars.  In Thailand, NBTC planed to start digital radio trial within 2018.

http://thaidigitalradio.com/ความคบหนื ้าล่าสดุ -วิทยุ/

19 [ https://en.wikipedia.org/wiki/HD_Radio

 Selected by the U.S. FCC in 2002 as a digital audio broadcasting method for the United States.  Embed digital signal “on-frequency” immediately above and below a station’s standard analog signal  Provide the means to listen to the same program in either HD ] (digital radio with less noise) or as a standard broadcast (analog radio with standard sound quality).

Spectrum of FM broadcast station

20 without HD Radio with HD Radio Countries using DAB/DMB

21 https://en.wikipedia.org/wiki/Digital_audio_broadcasting 2017 hurricane season in the US

22 http://edition.cnn.com/2017/10/10/weather/hurricane-nate-maria-irma-harvey-impact-look-back-trnd/index.html https://www.vox.com/energy-and-environment/2017/9/28/16362522/hurricane-maria-2017-irma-harvey-rain-flooding-climate-change Radio broadcasts are critical during a disaster  In areas hit hardest by things like hurricanes, earthquakes, fires, or even shootings:  Vulnerabilities of mobile phone infrastructure  Cell phone infrastructure is often knocked out.  Overwhelmed from everyone trying to access information.  Three weeks after Hurricane Maria pummeled Puerto Rico, more than 76 percent of cell sites still aren’t functioning.  Radio broadcast signals, which use low frequencies and can travel much further distances and penetrate through obstacles, usually remain up.

23 FM capability in modern cellphone  FM capability is baked into the Qualcomm LTE modem inside nearly every cellphone.  You can easily turn your phone into an FM radio if it has an embedded chipset and the proper circuitry to connect that chip to an FM antenna.  Need  an app like NextRadio  something to act as an antenna, such as headphones or nonwireless speakers.  Until a few years ago, device manufacturers disabled the function.  Wireless carriers wanted customers to stream music and podcasts, and consume more data.  Broadcasters and public safety officials have long urged handset manufacturers and wireless carriers to universally activate the FM chip.  ITU (International Telecommunications Union) issued an opinion in March 2017 urging all mobile phone makers to include and turn on FM radios on their devices.  Major US carriers now allow FM chips to be turned on. Manufacturers like Samsung, LG, HTC and Motorola have activated FM radio on their phones.  September 28, 2017: FCC blasted Apple for not activating FM receivers built into iPhones. Apple responded that iPhone 7 and iPhone 8, and iPhone X don’t use a chipset with an embedded FM radio.

https://www.cnet.com/news/everything-you-need-to-know-about-fm-radio-on-your-phone/ 24 https://spectrum.ieee.org/tech-talk/consumer-electronics/gadgets/fcc-wants-apple-to-turn-on-iphone-fm-receivers-that-may-not-exist https://www.wired.com/2016/07/phones-fm-chips-radio-smartphone/ Pokémon Communications

25 Pikachu's language

 Some of Pikachu's speech is consistent enough that it seems that some phrases actually mean something.  Pikachu always uses "Pikapi" when referring to Ash (notice that it sounds somewhat similar to "Satoshi").  Pi-Kachu: He says this during the sponsor spots in the original Japanese, Pochama (Piplup)  Pikachu-Pi: Kasumi (Misty)  Pika-Chu: Takeshi (Brock), Kibago (Axew)  Pikaka: Hikari (Dawn)  PiPiPi: Togepy (Togepi)  PikakaPika: Fushigidane (Bulbasaur)  PikaPika: Zenigame (Squirtle), Mukuhawk (Staraptor), Goukazaru (Infernape) or Gamagaru (Palpitoad)  PiPi-kachu: Rocket-dan (Team Rocket)  Pi-Pikachu: Get da ze! (He says this after Ash wins a Badge, catches a new Pokémon or anything similar.)  Pikachu may not be the only one to use this phrase, as other Pokémon do this as well. For example, when Iris caught Emolga, Axew said Ax-Axew (Ki-Kibago in the Japanese).  Pika-Pikachu: He says this when referring to himself.  Four-symbol variable-length code?

26 [https://www.youtube.com/watch?v=XumQrRkGXck] Rate-Distortion Theory  The theory of lossy source coding

27 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 2. Source Coding

Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40

1 Thursday 14:20-15:30 Elements of digital commu. sys.

Message Transmitter

Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add Signal redundancy systematic redundancy Channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator

2 System Under Consideration

Message Transmitter

3 Main Reference  Elements of Information Theory  2006, 2nd Edition  Chapters 2, 4 and 5 ‘the jewel in Stanford's crown’ One of the greatest information theorists since Claude Shannon (and the one most like Shannon in approach, clarity, and taste).

4 English Alphabet (Non-Technical Use)

5 US UK The ASCII Coded Character Set (American Standard Code for Information Interchange)

016 32 48 64 80 96 112

6 [The ARRL Handbook for Radio Communications 2013] Example: ASCII Encoder

Character Codeword x c(x) ⋮ MATLAB: E 1000101 >> M = 'LOVE'; ⋮ >> X = dec2bin(M,7); >> X = reshape(X',1,numel(X)) L 1001100 X = ⋮ 1001100100111110101101000101 O 1001111 Remark: ⋮ numel(A) = prod(size(A)) V 1010110 (the number of elements in matrix A) ⋮

Information “LOVE” Source “1001100100111110101101000101” Source Encoder 7 English Redundancy: Ex. 1

J-st tr- t- r--d th-s s-nt-nc-.

8 English Redundancy: Ex. 2

yxx cxn xndxrstxnd whxt x xm wrxtxng xvxn xf x rxplxcx xll thx vxwxls wxth xn 'x' (t gts lttl hrdr f y dn't vn kn whr th vwls r).

9 English Redundancy: Ex. 3

To be, or xxx xx xx, xxxx xx xxx xxxxxxxx

10 Entropy Rate of Thai Text

11 Introduction to Data Compression

12 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ] Introduction to Data Compression

13 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ] ASCII: Source Alphabet of Size = 128 (American Standard Code for Information Interchange)

016 32 48 64 80 96 112

14 [The ARRL Handbook for Radio Communications 2013] Ex. Source alphabet of size = 4

15 Ex. DMS (1)

 1 ,,,,,xabcde px 5 X  abcde,,,, X  0, otherwise Information Source a c a c e c d b c e d a e e d a b b b d b b a a b e b e d c c e d b c e c a a c a a e a c c a a d c d e e a a c a a a b b c a e b b e d b c d e b c a e e d d c d a b c a b c d d e d c e a b a a c a d

16 Approximately 20% are letter ‘a’s [GenRV_Discrete_datasample_Ex1.m] Ex. DMS (1) clear all; close all;

S_X = 'abcde'; p_X = [1/5 1/5 1/5 1/5 1/5];

n = 100; MessageSequence = datasample(S_X,n,'Weights',p_X) MessageSequence = reshape(MessageSequence,10,10)

>> GenRV_Discrete_datasample_Ex1

MessageSequence =

eebbedddeceacdbcbedeecacaecedcaedabecccabbcccebdbbbeccbadeaaaecceccdaccedadabceddaceadacdaededcdcade

MessageSequence =

eeeabbacde eacebeeead bcadcccdce bdcacccaed ebabcbedac dceeeacadd dbccbdcbac deecdedcca eddcbaaedd cecabacdae

17 [GenRV_Discrete_datasample_Ex1.m]  1 ,1,x  Ex. DMS (2)  2  1 ,2,x  px 4 X  1, 2, 3, 4 X   1 ,3,4x  8 0, otherwise Information Source 2 1 1 2 1 4 1 1 1 1 1 1 4 1 1 2 4 2 2 1 3 1 1 2 3 2 4 1 2 4 2 1 1 2 1 1 3 3 1 1 1 3 4 1 4 1 1 2 4 1 4 1 4 1 2 2 1 4 2 1 4 1 1 1 1 2 1 4 2 4 2 1 1 1 2 1 2 1 3 2 2 1 1 1 1 1 1 2 3 2 2 1 1 2 1 4 2 1 2 1

18 Approximately 50% are number ‘1’s [GenRV_Discrete_datasample_Ex2.m] Ex. DMS (2)

clear all; close all;

S_X = [1 2 3 4]; p_X = [1/2 1/4 1/8 1/8];

n = 20;

MessageSequence = randsrc(1,n,[S_X;p_X]); %MessageSequence = datasample(S_X,n,'Weights',p_X);

rf = hist(MessageSequence,S_X)/n; % Ref. Freq. calc. 0.5 stem(S_X,rf,'rx','LineWidth',2) % Plot Rel. Freq. Rel. freq. from sim. 0.45 pmf p (x) hold on X stem(S_X,p_X,'bo','LineWidth',2) % Plot pmf 0.4 xlim([min(S_X)-1,max(S_X)+1]) legend('Rel. freq. from sim.','pmf p_X(x)') 0.35 xlabel('x') 0.3 grid on 0.25

0.2

0.15

0.1

0.05

0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 x

19 [GenRV_Discrete_datasample_Ex2.m] DMS in MATLAB

clear all; close all;

S_X = [1 2 3 4]; p_X = [1/2 1/4 1/8 1/8]; n = 1e6;

SourceString = randsrc(1,n,[S_X;p_X]);

Alternatively, we can also use SourceString = datasample(S_X,n,'Weights',p_X);

rf = hist(SourceString,S_X)/n; % Ref. Freq. calc. stem(S_X,rf,'rx','LineWidth',2) % Plot Rel. Freq. hold on stem(S_X,p_X,'bo','LineWidth',2) % Plot pmf xlim([min(S_X)-1,max(S_X)+1]) legend('Rel. freq. from sim.','pmf p_X(x)') xlabel('x') 20 grid on [GenRV_Discrete_datasample_Ex.m] A more realistic example of pmf:

Relative freq. of letters in the English language

21 [http://en.wikipedia.org/wiki/Letter_frequency] A more realistic example of pmf:

Relative freq. of letters in the English language ordered by frequency

22 [http://en.wikipedia.org/wiki/Letter_frequency] Example: ASCII Encoder

Character Codeword x c(x) ⋮ MATLAB: E 1000101 >> M = 'LOVE'; ⋮ >> X = dec2bin(M,7); >> X = reshape(X',1,numel(X)) L 1001100 X = ⋮ 1001100100111110101101000101

Codebook O 1001111 Remark: ⋮ numel(A) = prod(size(A)) V 1010110 (the number of elements in matrix A) ⋮

c(“L”) c(“O”) c(“V”) c(“E”) Information “LOVE” Source “1001100100111110101101000101” Source Encoder 23 The ASCII Coded Character Set

016 32 48 64 80 96 112

24 [The ARRL Handbook for Radio Communications 2013] A Byte (8 bits) vs. 7 bits

>> dec2bin('I Love ECS452',7) >> dec2bin('I Love ECS452',8) ans = ans = 1001001 01001001 0100000 00100000 1001100 01001100 1101111 01101111 1110110 01110110 1100101 01100101 0100000 00100000 1000101 01000101 1000011 01000011 1010011 01010011 0110100 00110100 0110101 00110101 0110010 00110010 25 Geeky ways to express your love >> dec2bin('I Love You',8) >> dec2bin('i love you',8) ans = ans = 01001001 01101001 00100000 00100000 01001100 01101100 01101111 01101111 01110110 01110110 01100101 01100101 00100000 00100000 01011001 01111001 01101111 01101111 01110101 01110101 https://www.etsy.com/listing/91473057/binary-i-love-you-printable-for- your?ref=sr_gallery_9&ga_search_query=binary&ga_filters=holidays+- supplies+valentine&ga_search_type=all&ga_view_type=gallery http://mentalfloss.com/article/29979/14-geeky-valentines-day-cards https://www.etsy.com/listing/174002615/binary-love-geeky-romantic-pdf- cross?ref=sr_gallery_26&ga_search_query=binary&ga_filters=holidays+- supplies+valentine&ga_search_type=all&ga_view_type=gallery https://www.etsy.com/listing/185919057/i-love-you-binary-925-silver-dog-tag- 26 can?ref=sc_3&plkey=cdf3741cf5c63291bbc127f1fa7fb03e641daafd%3A185919057&ga_search_query=binary &ga_filters=holidays+-supplies+valentine&ga_search_type=all&ga_view_type=gallery http://www.cafepress.com/+binary-code+long_sleeve_tees w/o extension Summary: Source Encoder source string encoded string c(“L”) c(“O”) c(“V”) c(“E”) Information “LOVE” Source “1001100100111110101101000101” Source Encoder

Discrete Memoryless • An encoder · is a • The codeword Source (DMS) function that maps each of corresponding to a source • The source the symbol in the source symbol is denoted by alphabet is the alphabet into a . collection of all corresponding (binary) • the length of possible source codeword. symbols. • The list for such mapping • Each codeword is • Each symbol that the is called the codebook. constructed from a code Source Symbol Codeword alphabet. source generates is x c(x) ⋮ • assumed to be E 1000101 For binary codeword, ⋮ randomly selected L 1001100 the code alphabet is ⋮ 0,1 from the source O 1001111 ⋮ alphabet. V 1010110 27 ⋮ Morse code (wired and wireless)  Telegraph network  Samuel Morse, 1838  A sequence of on-off tones (or , lights, or clicks)

28 Example

29 [http://www.wolframalpha.com/input/?i=%22I+love+you.%22+in+Morse+code] Example

30 Morse code: Key Idea

Frequently-used characters are mapped to short codewords.

31 Relative frequencies of letters in the English language Morse code: Key Idea Frequently-used characters (e,t) are mapped to short codewords.

32 Relative frequencies of letters in the English language Morse code: Key Idea Frequently-used characters (e,t) are mapped to short codewords.

Basic form of compression.

33 รหสมอรั ์สภาษาไทย

34 Example: ASCII Encoder

Character Codeword ⋮ MATLAB: E 1000101 >> M = 'LOVE'; ⋮ >> X = dec2bin(M,7); L 1001100 >> X = reshape(X',1,numel(X)) ⋮ X = 1001100100111110101101000101 O 1001111 ⋮ V 1010110 ⋮

Information “LOVE” Source “1001100100111110101101000101” Source Encoder 35 Another Example of non-UD code  Suppose we want to convey the sequence of outcomes from rolling a dice.

xc(x)  A sequence of throws such as 53214 is 11 encoded as 10111101100 210 311 4 100 5 101 6 110

36 Another Example of non-UD code  Suppose we want to convey the sequence of outcomes from rolling a dice.

xc(x)  The encoded string 11 could be 11 interpreted as 210  11: 1 1  311 3: 11  The encoded string 110 could be 4 100 interpreted as 5 101  12: 1 10 6 110  6: 110

37 Another Example of non-UD code

xc(x) A1 B 011 C 01110 D 1110 E 10011

38 Another Example of non-UD code

xc(x)  Consider the encoded string A1 011101110011. B 011  It can be interpreted as C 01110  CDB: 01110 1110 011  D 1110 BABE: 011 1 011 10011 E 10011

39 [ https://en.wikipedia.org/wiki/Sardinas%E2%80%93Patterson_algorithm ] Game: 20 Questions  20 Questions is a classic game that has been played since the 19th century.  One person thinks of something (an object, a person, an animal, etc.)  The others playing can ask 20 questions in an effort to guess what it is.

40 20 Questions: Example

41 Prof. Robert Fano (1917-2016) Shannon Award (1976 ) Shannon–Fano coding  Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948  The method was attributed to Fano, who later published it as a technical report.  Fano, R.M. (1949). “The transmission of information”. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT.  Should not be confused with  Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with  Shannon–Fano–Elias coding (also known as Elias coding), the precursor to arithmetic coding.

42 David Huffman (1925–1999) Huffman Code Hamming Medal (1999)  MIT, 1951  Information theory class taught by Professor Fano.  Huffman and his classmates were given the choice of  a term paper on the problem of finding the most efficient binary code. or  a final exam.  Huffman, unable to prove any codes were the most efficient, was about to give up and start studying for the final when he hit upon the idea of using a frequency-sorted binary tree and quickly proved this method the most efficient.  Huffman avoided the major flaw of the suboptimal Shannon-Fano coding by building the tree from the bottom up instead of from the top down.

43 Huffman’s paper (1952)

[D. A. Huffman, "A Method for the Construction of Minimum-Redundancy 44 Codes," in Proceedings of the IRE, vol. 40, no. 9, pp. 1098-1101, Sept. 1952.] [ http://ieeexplore.ieee.org/document/4051119/ ] All codes Summary: Nonsingular codes  [2.16-17] A good code must be UD codes uniquely decodable (UD). Prefix-free [Defn 2.18] codes  Difficult to check. Huffman  [2.24] Consider a special family codes of codes: prefix(-free) code.  No codeword is a prefix of any other Always UD. codeword.  Same as being instantaneous. Each source symbol can be decoded as soon as we come to the end of the  [Defn 2.30] Huffman’s recipe codeword corresponding to it  Repeatedly combine the two least-likely (combined) symbols.  Automatically give prefix-free code. [Defn 2.36]  [2.37] For a given source’s pmf, Huffman codes are optimal 45 among all UD codes for that source. Huffman coding

46 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ] Ex. Huffman Coding in MATLAB [Ex. 2.31] Observe that pX = [0.5 0.25 0.125 0.125]; % pmf of X MATLAB SX = [1:length(pX)]; % Source Alphabet automatically give [dict,EL] = huffmandict(SX,pX); % Create codebook the expected length of the %% Pretty print the codebook. codewords codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook

%% Try to encode some random source string n = 5; % Number of source symbols to be generated sourceString = randsrc(1,10,[SX; pX]) % Create data using pX encodedString = huffmanenco(sourceString,dict) % Encode the data

47 [Huffman_Demo_Ex1] Ex. Huffman Coding in MATLAB

codebook =

[1] '0' [2] '1 0' [3] '1 1 1' [4] '1 1 0'

sourceString =

1 4 4 1 3 1 1 4 3 4

encodedString =

0 1 1 0 1 1 0 0 1 1 1 0 0 1 1 0 1 1 1 1 1 0

48 [Huffman_Demo_Ex1] Ex. Huffman Coding in MATLAB [Ex. 2.32] pX = [0.4 0.3 0.1 0.1 0.06 0.04]; % pmf of X SX = [1:length(pX)]; % Source Alphabet [dict,EL] = huffmandict(SX,pX); % Create codebook

%% Pretty print the codebook. >> Huffman_Demo_Ex2 codebook = dict; for i = 1:length(codebook) codebook = codebook{i,2} = num2str(codebook{i,2}); end codebook [1] '1' [2] '0 1' EL [3] '0 0 0 0' [4] '0 0 1' [5] '0 0 0 1 0' The codewords can be different [6] '0 0 0 1 1' from our answers found earlier. EL = The expected length is the same. 2.2000 49 [Huffman_Demo_Ex2] Ex. Huffman Coding in MATLAB [Exercise] pX = [1/8, 5/24, 7/24, 3/8]; % pmf of X SX = [1:length(pX)]; % Source Alphabet [dict,EL] = huffmandict(SX,pX); % Create codebook

%% Pretty print the codebook. codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook

codebook = [1] '0 0 1' >> -pX*(log2(pX)).' [2] '0 0 0' ans = [3] '0 1' 1.8956 [4] '1' EL = 1.9583 50 0

-0.05

-0.1

-0.15

-0.2

-0.25

-0.3

-0.35

-0.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x 56 Entropy and Description of RV

57 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy ] Entropy and Description of RV

58 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy ] Summary: Optimality of Huffman Codes Consider a given DMS with All codes known pmf … Nonsingular codes  [Defn 2.36] A code is optimal if it is UD and its corresponding UD codes expected length is the shortest Prefix-free among all possible UD codes for codes that source. Huffman  [2.37] Huffman codes are codes optimal.  [2.49-2.54] Bounds on expected lengths: Expected length Expected length (per source (per source 1 symbol) of an symbol) of a optimal code Huffman code 59 Summary: Entropy  Entropy measures the amount of uncertainty (randomness) in a RV.  Three formulas for calculating entropy:  [Defn 2.41] Given a pmf of a RV ,  ≡∑ log . Set 0log00.  [2.44] Given a probability vector ,

 ≡∑ log.  [Defn 2.47] Given a number , binary entropy  ≡ log 1 log 1 function  [2.56] Operational meaning: Entropy of a random variable is the average length of its shortest description. 60 Examples  Example 2.31 Huffman HX1.75   X  Efficiency = 100%

 Example 2.32

Huffman HX2.14 2.2   X  Efficiency  97%

61 Examples  Example 2.33

Huffman HX2.29 2.3   X  Efficiency  99%

 Example 2.34

Huffman A HX1.86 2   X  B   C Efficiency  93% D

62 Summary: Entropy  Important Bounds deterministic uniform  The entropy of a uniform (discrete) random variable:  The entropy of a Bernoulli random variable:  binary entropy function

63 [Ex.2.40] Huffman Coding: Source Extension

1 1

i.i.d. 0.9 X k  Bernoullip

p  0.1 0.8

Ln 0.7 0.645

0.6

0.533 0.5

0.4 1 2 3 4 5 6 7 8 n: order of extension 66 [Ex.2.40] Huffman Coding: Source Extension

1.8

i.i.d. 1.6

X k  Bernoullip p  0.1 1.4

1.2

1 1 HX n 0.8 Ln 0.6 HX  0.4 1 2 3 4 5 6 7 8 Order of source extension 67 n Summary: Source Extension  The encoder operates on the blocks rather than on individual symbols.  [Defn 2.39] -th extension coding: 1 block = successive source symbols

 = expected (average) codeword length per source symbol when Huffman coding is used with -th extension 1.8  1.6

1.4  → 1.2 1 HX 1 n

0.8

0.6 Ln

68 HX  0.4 1 2 3 4 5 6 7 8 Order of source extension Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 3 Discrete Memoryless Channel (DMC)

Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40

1 Thursday 14:20-15:30 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 3.1 DMC Models

2 Summary: DMC

Put together Channel Input | All values at corresponding Alphabet positions in the row vector matrix.  | matrix

 , matrix row vector “AND” ,

y  y yyy1 j n x p x x  1   x1  x    1        p x i   xi Qxy ji x pxy, Q   i  XY, i j   P      p xm   xm  x  m  x1 y1      Qyx j i  xi y j   11 qy j  qpQ xm yn  Review: Evaluation of Probability from the Joint PMF Matrix  Consider two random variables X and Y.  Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3  Find

Step 1: Find the pairs (x,y) that y satisfy the condition x 2 3 4 5 6 “x+y < 7” 1 3 4 5 6 7 xy 3 5 6 7 8 9 One way to do this is to first 4 6 7 8 9 10 construct the matrix of x+y. 6 8 9 10 11 12 12 Review: Evaluation of Probability from the Joint PMF Matrix  Consider two random variables X and Y.  Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3  Find

Step 2: Add the corresponding y probabilities from the x 2 3 4 5 6 joint pmf (matrix) 1 3 4 5 6 7 xy 3 5 6 7 8 9 7 0.10.10.1 4 6 7 8 9 10 0.3 6 8 9 10 11 12

13 Review: Evaluation of Probability from the Joint PMF Matrix  Consider two random variables X and Y.  Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3  Find

14 Review: Sum of two discrete RVs  Consider two random variables X and Y.  Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3  Find

y x 2 3 4 5 6 7 0.1 1 3 4 5 6 7 xy 3 5 6 7 8 9 4 6 7 8 9 10 6 8 9 10 11 12

15 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 3.2 Decoder and

17 Recipe for finding  of any decoder y xyˆ 0 0.5 1 1 1 XY2 2 1 1 0.3 3 110xˆ y 3 0 x y 123 1 2 3y x 000.5 0.2 0.3 0.2 0.10 0.04 0.06 Q   P 0.8  1 0.3 0.4 0.3  0.24 0.32 0.24 1  Use the matrix.  If unavailable, can be found by scaling each row of the matrix by its corresponding .  Write values on top of the values for the matrix.  For column in the matrix, circle the element whose corresponding value is the same as .   the sum of the circled probabilities.  1. 25 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 3.3 Optimal Decoder

27 Review: ECS315 (2017)

29 Guessing Game 1  There are 15 cards.  Each have a number on it.  Here are the 15 cards: 1 2 2 3 3 3 4 4 4 4 5 5 5 5 5  One card is randomly selected from the 15 cards.  You need to guess the number on the card.  Have to pay 1 Baht for incorrect guess.  The game is to be repeated n = 10,000 times.  What should be your guess value?

30 close all; clear all;

n = 5; % number of time to play this game

D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5]; X = D(randi(length(D),1,n));

if n <= 10 X end

g = 1 cost = sum(X ~= g) >> GuessingGame_4_1_1 X = if n > 1 3 5 1 2 5 averageCostPerGame = cost/n g = end 1 cost = 4 averageCostPerGame = 0.8000 31 close all; clear all;

n = 5; % number of time to play this game

D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5]; X = D(randi(length(D),1,n));

if n <= 10 X end

g = 3.3 cost = sum(X ~= g) >> GuessingGame_4_1_1 X = if n > 1 5 3 2 4 1 averageCostPerGame = cost/n g = end 3.3000 cost = 5 averageCostPerGame = 1 32 close all; clear all;

n = 1e4; % number of time to play this game

D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5]; X = D(randi(length(D),1,n));

if n <= 10 X end

g = ? cost = sum(X ~= g)

if n > 1 averageCostPerGame = cost/n end

33 Guessing Game 1

0.95

0.9

0.85

0.8

AVerage Cost Per Game 0.75

0.7

0.65 1 1.5 2 2.5 3 3.5 4 4.5 5 34 Guess value Guessing Game 1

0.95

0.9 Optimal Guess:

0.85 The most-likely value

0.8

AVerage Cost Per Game 0.75

0.7

0.65 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 Guess value 35 Summary: MAP vs. ML Decoders likelihood funtion

xyˆMAP  arg max pxyXY ,  ,  xyˆML   arg maxQy x x x  xyˆoptimal a posteriori probability Optimal at least when is uniform (the channel inputs are  arg max P XxYy equally likely) x prior probability Can be derived without knowing the  arg maxQyx px x channel input probabilities.  Decoder is derived from the matrix  Decoder is derived from the matrix  Select (by circling) the maximum value  Select (by circling) the maximum value in each column (for each value of ) in in each column (for each value of ) in the matrix. the matrix.  The corresponding value is the value  The corresponding value is the value of . of .  Once the decoder (the decoding table) is derived  and  are calculated from adding the corresponding probabilities in the matrix.

36 Example: MAP Decoder [Ex. 3.36]

0 0.5 1 y xˆMAP y X 2 Y 1 1 1 0.3 3 111xyˆ   MAP 2 1 y y x 123 1 2 3x 3 1 000.5 0.2 0.3 0.2 0.10 0.04 0.06 Q   P 0.8  1 0.3 0.4 0.3  0.24 0.32 0.24 1 P()  1P( )12 0.24 0.32 0.24 0.

37 Example: ML Decoder [Ex. 3.47]

0.5 0 1 [Same as Ex. 3.27] y xˆ y X 2 Y ML 1 3 1 0 0.3 010xyˆML   2 1 x y 123 1 2 3y x Sol 1: 000.5 0.2 0.3 0.2 0.10 0.04 0.06 3 0 Q   P 0.8  1 0.3 0.4 0.3  0.24 0.32 0.24 1 P()1 P( ) 1 0.10 0.32 0.06  0.52

0.5 0 1 y xˆ y X 2 Y ML 1 3 1 0 0.3 011xyˆML   2 1 y 123 1 2 3y Sol 2: x x 000.5 0.2 0.3 0.2 0.10 0.04 0.06 3 1 Q   P 0.8  1 0.3 0.4 0.3  0.24 0.32 0.24 1 [Agree with Ex. 3.33] 38 P()1 P( ) 1 0.10 0.32 0.24  0.34 MAP Decoder %% MAP Decoder P = diag(p_X)*Q; % Weight the channel transition probability by the % corresponding prior probability. [V I] = max(P); % For I, the default MATLAB behavior is that when there are % multiple max, the index of the first one is returned. Decoder_Table = S_X(I) % The decoded values corresponding to the received Y

%% Decode according to the decoder table x_hat = y; % preallocation for k = 1:length(S_Y) I = (y==S_Y(k)); x_hat(I) = Decoder_Table(k); end

PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation

%% Calculation of the theoretical error probability PC = 0; for k = 1:length(S_X) I = (Decoder_Table == S_X(k)); Q_row = Q(k,:); PC = PC+ p_X(k)*sum(Q_row(I)); end PE_theretical = 1-PC

39 [DMC_decoder_MAP_demo.m] ML Decoder %% ML Decoder [V I] = max(Q); % For I, the default MATLAB behavior is that when there are % multiple max, the index of the first one is returned. Decoder_Table = S_X(I) % The decoded values corresponding to the received Y

%% Decode according to the decoder table x_hat = y; % preallocation for k = 1:length(S_Y) I = (y==S_Y(k)); x_hat(I) = Decoder_Table(k); end

PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation

%% Calculation of the theoretical error probability PC = 0; for k = 1:length(S_X) I = (Decoder_Table == S_X(k)); Q_row = Q(k,:); PC = PC+ p_X(k)*sum(Q_row(I)); end PE_theretical = 1-PC

40 [DMC_decoder_ML_demo.m] Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 3.5 Repetition Code in Communications Over BSC

47 Hypercube

48 https://www.youtube.com/watch?v=Q_B5GpsbSQw Hypercube

49 https://www.youtube.com/watch?v=Q_B5GpsbSQw n-bit space

n = 4 n = 1

n = 2

n = 3

50 Channel Encoder and Decoder

Message Transmitter S Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Add X: channelSignal input systematic redundancy 0 1-p 0 p Channel p 1 1-p 1 Noise & Interference Recovered Message Receiver Received Y: channelSignal output Source Channel Digital Destination Decoder Decoder Demodulator

51 Better Equivalent Channel

Message Transmitter S Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add Signal redundancy systematic redundancy Better Equivalent Channel Channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator

52 Example: Repetition Code  Original Equivalent Channel: 0 1-p 0 p p 1 1-p 1  BSC with crossover probability p = 0.01  New (and Better) Equivalent Channel:

1-p Repetition 0 0 Majority 0 0 Code with p p Vote 1 1 n = 5 1 1-p 1

 Use repetition code with n = 5 at the transmitter  Use majority vote at the receiver 5 5 5 1 1 1  New BSC with 3 4 5 10 53 Example: ASCII Encoder and BSC

Message Transmitter

Information Source “1001100100111110101101000101”Channel Digital Source Encoder Encoder Modulator Transmitted “LOVE” Add X: channelSignal input systematic redundancy 0 1-p 0 p Channel p 1 1-p 1 Noise & Interference Recovered Message Receiver Received Y: channelSignal output Source Channel Digital Destination Decoder Decoder Demodulator

54 The ASCII Coded Character Set

016 32 48 64 80 96 112

55 [The ARRL Handbook for Radio Communications 2013] Example: ASCII Encoder

Character Codeword ⋮ MATLAB: E 1000101 >> M = 'LOVE'; ⋮ >> X = dec2bin(M,7); L 1001100 >> X = reshape(X',1,numel(X)) ⋮ X = 1001100100111110101101000101 O 1001111 ⋮ V 1010110 ⋮

c(“L”) c(“O”) c(“V”) c(“E”) Information “LOVE” Source “1001100100111110101101000101” Source Encoder 56 System considered 101010010010001000101010000010101111001001101101010000011010010100010001000001001111100011001000001001111101101001000000100000010000001010000110001011100 101100000110000010100100010100001001111011111100101101001111010011101001100101110111001000001100010111100101000001001100010111001000001000110111001011000 011101110110101101000001000010110000111101011101101000101000010100001010100100111011101110100111001011011111100100111010111000111110100110100111011111101 110000101000010100100000010000010001101101111110110011010111101100110111111100101100101010110001000001101100110010111001111100101110111011001001110011010 110001000001101101111100111101001101000111001101000001100001110111011001000100000110011011000011101001111001011110010100000111010011000011101100110010111 100110100000110100011000011110110110010101000001100110110111111011001101100110111111101111100101110010001000001100011110100011010011101100110010011010001 Message Transmitter101111110111111001000001010111010011010001110010110111111101011100111110100001000001110100110100011001010100000110000111001111100101111001101011000100000 110011011011111110010010000011001011110110110010111100101111001010000011010001100101110000111011001110100110100011110010100000111100111011111110101110111 011001111110011111010011001011110010010000011010001100001111001101000001100001010000011101111101000110111111011001100101111001111011111101101110010101000 001100001110111011001000001010110100111011101110011111010011010011101110110001111101001101001111011011001010100000110110011011111110110110010101000001100 110110111111100100100000111001111101001101111111001011010011100101111001101000001100110110000111011101110100110000111100111110100110100111000110101100010 000011011011100001111001011101101100101110110011011111110101111001101000001100001110111011001000100000110110111000011101110110100111001101100101111001111 Information Source Channel 101001101100111100100010101110101110111011100101100101110000111011000101110010000010101001101000110010101000001110111110100111011101Digital 100111110010111001000 100000110011011000011101001111001011010011100101111001101000001101111110011001000001000111111001011010011101101110110101000001100001110111011001000100000 100000111011101100100110010111100101110011110010111011100100000110100011000011110110110010101000001100010111001011011111110101110011111010001110100010000 Source Encoder Encoder011011011101111111001011001010001010110100011000011110000111000011010011101110110010111100111110011010000011101001101111010000011000Modulator 111101000110100111011 001100100110100111100111101000010000011010001100101110000111100101110100111001101000001110100110100011000011101110010000011000011101100110110001000001101 111111010011010001100101111001001000001101000111010111011011100001110111001000001100011111001011001011100001111010011010011101111110111011100110101110000 101001000000100000101100111001011110100010000011101001101000110010101000001101111110110011001000100000111010011010011101101110010101Transmitted000001100110110000111 010011110010111100101000001110100110000111011001100101010110001000001101000110000111101101101001110111011001110100000111001111001011110010111011011001011 100100010000011001101101111111001001000001100111110010111011101100101111001011000011110100110100111011111101110111001101011000100000110110111000011111001 THE WIZARD OF OZ (1900) Signal written by L. Frank Baum Add systematic X: channel input Introduction redundancy 1-p Folklore, legends, myths and fairy tales have followed childhood 0 0 through the ages, for every healthy youngster has a wholesome and p instinctive love for stories fantastic, marvelous and manifestly Channel 0.01 unreal. The winged fairies of Grimm and Andersen have brought more p happiness to childish hearts than all other human creations. Yet the old time fairy tale, having served for generations, may 1 1-p 1

Y: channel output Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator

57 [ErrorProbabilityoverBSC.m] Results

THE WIZARD OF OZ (1900) written by L. Frank Baum 101010010010001000101010000010101111001001101101010000011010010100010001000001001111100011001000001001111101101001000000100000010000001010000110001011100 101100000110000010100100010100001001111011111100101101001111010011101001100101110111001000001100010111100101000001001100010111001000001000110111001011000 011101110110101101000001000010110000111101011101101000101000010100001010100100111011101110100111001011011111100100111010111000111110100110100111011111101 110000101000010100100000010000010001101101111110110011010111101100110111111100101100101010110001000001101100110010111001111100101110111011001001110011010 110001000001101101111100111101001101000111001101000001100001110111011001000100000110011011000011101001111001011110010100000111010011000011101100110010111 100110100000110100011000011110110110010101000001100110110111111011001101100110111111101111100101110010001000001100011110100011010011101100110010011010001 Introduction 101111110111111001000001010111010011010001110010110111111101011100111110100001000001110100110100011001010100000110000111001111100101111001101011000100000 110011011011111110010010000011001011110110110010111100101111001010000011010001100101110000111011001110100110100011110010100000111100111011111110101110111 011001111110011111010011001011110010010000011010001100001111001101000001100001010000011101111101000110111111011001100101111001111011111101101110010101000 001100001110111011001000001010110100111011101110011111010011010011101110110001111101001101001111011011001010100000110110011011111110110110010101000001100 Folklore, legends, myths and fairy tales have followed childhood 110110111111100100100000111001111101001101111111001011010011100101111001101000001100110110000111011101110100110000111100111110100110100111000110101100010 000011011011100001111001011101101100101110110011011111110101111001101000001100001110111011001000100000110110111000011101110110100111001101100101111001111 101001101100111100100010101110101110111011100101100101110000111011000101110010000010101001101000110010101000001110111110100111011101100111110010111001000 through the ages, for every healthy youngster has a wholesome and 100000110011011000011101001111001011010011100101111001101000001101111110011001000001000111111001011010011101101110110101000001100001110111011001000100000 100000111011101100100110010111100101110011110010111011100100000110100011000011110110110010101000001100010111001011011111110101110011111010001110100010000 instinctive love for stories fantastic, marvelous and manifestly 011011011101111111001011001010001010110100011000011110000111000011010011101110110010111100111110011010000011101001101111010000011000111101000110100111011 001100100110100111100111101000010000011010001100101110000111100101110100111001101000001110100110100011000011101110010000011000011101100110110001000001101 111111010011010001100101111001001000001101000111010111011011100001110111001000001100011111001011001011100001111010011010011101111110111011100110101110000 unreal. The winged fairies of Grimm and Andersen have brought more 101001000000100000101100111001011110100010000011101001101000110010101000001101111110110011001000100000111010011010011101101110010101000001100110110000111 010011110010111100101000001110100110000111011001100101010110001000001101000110000111101101101001110111011001110100000111001111001011110010111011011001011 happiness to childish hearts than all other human creations. 100100010000011001101101111111001001000001100111110010111011101100101111001011000011110100110100111011111101110111001101011000100000110110111000011111001 Yet the old time fairy tale, having served for generations, may

THE WIZARD _F OZ (19009 written by L. Frank0Baum

101010010010001000101010000010101111001001101101010000011010010100010001000001011111100011001000001001111101101001000000100000010000001010000110001011100 101100000110000011100100110100001001111011111100101101001111010011101001100101110111001000001100010111100101000001001100010111001000001000110111001011000 011101110110101101100001000010110000111101011101101000101000010100001010100100111011101110100111001011011111100100111010111000111110100110100111011111101 110000101000010100100000011000010001101101111110110011010111101100110111111100101100101010111000000001101100110010111001111100101110111011001001010011000 110001000001101101111100111101001101000111001101000001100001110111011001000100000110011011000011101001110001011110010100000111010011000011101100110011111 Introduction 100110100000110100011000011110110110010101000001100110110110111011001101100110111111101111100101110010001000001100011110100011010011101100110010011010001 101111110111111001000001010111010011010001110010110111111101011100111110100001000001110100110100011001010100000110000111001111100101111001101011000000000 110011011011111110010010010011001011110110110010111100101111001010000011011101100101110000111011001110100110100011110010100000111100111011111110101110111 011001111110011111010011001011110010010000011010001100001111001101000001100001010000011101111101000110111111011001100101111001111011111101101110010101000 0Folklore. legendS myths and faiby talgs have fmllowed childhood 001100001110111011001000001010110100111011001110011111000011110011101110110001111101001101001111001011001010100000110110011011111110110110010101000001100 110110111111100100100000111001111101001101111111001011010011100101111000101000001100110110000101011101110100110000111100111110100110100111000110101100010 000011011011100001111001011101101100101110110011011111110101011001101000001100101110111011001000100000110110111000011101110110100111001101100101111001111 through the ages, for$every nealthy youngster has a wholesome and 101001101100111100100010101110101110111011100101100101110000111011000101110010000010101001101000110010101000001110111110100111011101100111110010111001000 100000110011011000011101001111001011010011100101111001101000001101111110011001000001000111111001011010011101101110110101000001100001110111011001000101010 ilspynctire love for storieq fa.tastic, marvelou3 end manifestly 100000111011101100100110010111100101110011110010111011100100000110100011000011110110100010101000001100010111001011011111110101110011111010001110100010000 011011011101111111001011001010001010110100011000011110000111000011010011101110110010111100111110011000000011101001101111010000011000111101000110100111010 001100100110100111100111101000010000011010001100101110000111100101110100111001101000001110100110100011000011101110010000011000011101100110110001000001101 unreal. The winged fairies of Grimm and*Andersen havE brought more 111110010011010001110101111001001000001101000111010111011011100001110111001000001100011111001011001011100001111010011010011101111110111011100110101111000 101001000000100000101100111001011110100010000011101001101000110010101100001101111110110011001000100010111010011010011101101110110101000001100110110000111 happiness to chihdish hearts than all odhur human creations/ 010010110010111100101000001110100110000111011001100101010110001000001101000110000111101101101001110111011001110100000111000111001011110010111011011001011 Yet the0old"timm fai2y tale, having qerved for generationq, may 100100010000011001101101111111001001000001100111110010111011101100101111001011000011110100110100111011111101110111000101011000100000110110111000011111001

58 Results

THE WIZARD OF OZ (1900) written by L. Frank Baum  The whole book which is saved in the file “OZ.txt” has Introduction 207760 characters Folklore, legends, myths and fairy tales have followed childhood through the ages, for every healthy youngster has a wholesome and instinctive love for stories fantastic, marvelous and manifestly (symbols). unreal. The winged fairies of Grimm and Andersen have brought more happiness to childish hearts than all other human creations.  The ASCII encoded string has Yet the old time fairy tale, having served for generations, may 207760×7 = 1454320 bits. THE WIZARD _F OZ (19009 written by L. Frank0Baum  The channel corrupts 14545 Introduction bits. 0Folklore. legendS myths and faiby talgs have fmllowed childhood through the ages, for$every nealthy youngster has a wholesome and  This corresponds to 14108 ilspynctire love for storieq fa.tastic, marvelou3 end manifestly unreal. The winged fairies of Grimm and*Andersen havE brought more erroneous characters. happiness to chihdish hearts than all odhur human creations/ Yet the0old"timm fai2y tale, having qerved for generationq, may

59 >> ErrorProbabilityoverBSC biterror = Results 14545 BER = 0.010001237691842 theoretical_BER =  0.010000000000000 The file “OZ.txt” has 207760 characterErrror = 14108 characters (symbols). CER = 0.067905275317674  The ASCII encoded string has theoretical_CER = 0.067934652093010 207760×7 = 1454320 bits.  The channel corrupts 14545 bits.  This corresponds to 14108 erroneous characters.

60 Results  The file “OZ.txt” has 207760 BSC’s crossover probability characters (symbols).  The ASCII encoded string has 207760×7 = 1454320 bits.  The channel corrupts 14545 bits.  This corresponds to 14108 erroneous characters (symbols).

CER A character (symbol) is successfully recovered if and only if none of its bits are 61 corrupted. Crossover probability and readability When the first novel of the series, Harry Potter and the Philosopher's Stone (published in some countries as Harry Potter and the Sorcerer's Stone), opens, it is apparent that some significant event has taken place in the wizarding world--an event so very remarkable, even the Muggles notice signs of it. The full background to this event and to the person of Harry Potter is only revealed gradually through the series. After the introductory chapter, the book leaps forward to a time shortly before Harry Potter's eleventh birthday, and it is at this point that his magical background begins to be revealed. Original

Human may be able to correct some (or even all) of these errors.

When the first novel of the series, Harry Pottez and the Philosopher's Stone (p5blished in some countries as Harry Potter cnd the Sorcerep's Stone), opens, it i3 apparent that soMe cignifacant event!haS taken0place in the wi~arding 7orld--ao event so`very!bemark!blu, even the Mufgles nodice signs"of it. The fuld background to this event and to the person of Harry P/tTer is only revealed gradually through th series. After the introfuctory chapter, the boo+ leaps forward to a time shortly before Harpy Potteb7s eleventh`birthday, and )t is at this poi~t that his -agikal bac{ground begins to be revealed. 0.01 CER 0.07 63 Crossover probability and readability w(en th% birst .ovo,òf the serieq, Hcrry Potter(and the Phidosopèr's Suone (xub|ishe$(in some!Countries as @arry Potter and tè Snr#erer's S|ong)- opens, it is apparent thatsoee smgfificant erent ha3 taieN place in the wizardino world-,an event!so very remarkable, even thE Eugglds notiae qigns of it. Tledfull back'tound to this event ane to the perron of Harry Popter is onl{ reveqned gsadeally thro}gH th%$serias. After the int2oducpory chcptur, the0jook deaps forward to"a!tmme shmrtly befosE Harry"Potter's eleventh jirthdiy cnd ht is a| thi3 po{nt tHat @is mAgiial background begijs to rm rerealed. 0.02 CER 0.13

Whethe first nOwgl nf the susi-Q-@Harr} PoutEr(and |he PhilosoxhEr's Ctonepuclyshed in som% coultries as Harrx @ tter and the S_rcermr7s Spone), opdns, id is apparent that {omg`signifikant evmnt ias taKen!Placa in tHe 7ijardIng world--an event so Very remaroqble, eve.!thE MugglaC fotice signc of"it. Uhe full backf2ound`to thas even| ant`0o the pEssoj of @arry Qotteb iw only revealed gradu!lly vhvoug` the rerier. Afte2 the IndRoductori chaptar,t`e book leats ForwaRf tc a 4imE shostl= before!Hcssy potter's u|Eveoth$firthdA{, and iT is ad this pomNt uhav `ir magica, back'bound cegins to bE 2evealed. 0.03 CER 0.19 64 Crossover probability and readability

When phe fir v okval ov"th% serie3, @`rry0Pntter efdxtxeThil soph%rs Stone0(p}blisjed!in {ooe c un|pye{ agav0y Potter aj`(the sorcerer"s S4o|e)< opdns- mt"Is!apParEnt 4hat somu siwnidiga.v evant iAs take."plhge in(uhe w)zard)ng wo{|d--An event so very Rumar{ablel eteN0Dhe %ugcles$n t)ce signs of$At. Tje!&ul|!backep/und Dk thkw`event ajt(to vhd per{On of8Ikxry P_Pter is oN,y rereAeud gredualli 4hroufh5ie qeriesn Af|ir the )~trofUctkry!ciapter,$tle r%ok lE`ps for erd8to!a d)hg 3Hostly redobd HArry(Potter/r elaventI(birpl%ay,))nd(iD i3 1t tlis hohlt vhat iis$iagical bac+gropnd bedans to bg rEve!ied/ 0.05 CER 0.30

Whef th% &i2sv nkvdl"On(txE"serm-s< HaRtY Qo|p%R$anlthe Phi$)qop8gb'r YtoNe(puclirhed in23/ee c uNpr9es aZ Harby!PovDdZ qnd0THA!Uorojev's Qpof'), pegsL iT is0aqazenP Tiet`{nle sau*!fICQ~t eve.t`xA# raken pOqb%%)D }Hm`wizprdYjv"wOrnd--a~%W%Jv s' tury 2maskABdd$`eden(tl| LuxGxec`nOtike c)gzq of ktTiu!f5mm"cackG@ Ud(to"vhhQ a~aNd alt tn0vid veRckn of HaRvq$Xntter#isxohk{ regea,ed@&saduadLy u(2otGh"tau griEs."AfTex0T`g mntr DUCt ry kh `ter,$thd(fomN0j`apv ngrwarTt-0c t,me"1xortly bEemsL |ar2q Pnfter'3 aMen-n5i@Fipth$`q, aoh It i3d1t piac0pmhnP d*if Zas mafibin"je#k7poUndpb%dins tk`be qe6e!lgd. 0.10 CER 0.52 65 BER vs. CER

0.9

0.8

0.7

0.6

0.5

0.4

0.3 Character Error rate (CER)

0.2

0.1

0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 66 Bit Error Rate (BER) [BERCER.m] BER vs. CER

0 10

-1 10 Character Error rate (CER)

-2 10 -2 -1 0 67 10 10 10 Bit Error Rate (BER) [BERCER.m] Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 4. Mutual Information and Channel Capacity Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40

1 Thursday 14:20-15:30 Reference for this chapter  Elements of Information Theory  By Thomas M. Cover and Joy A. Thomas  2nd Edition (Wiley)  Chapters 2, 7, and 8  1st Edition available at SIIT library: Q360 C68 1991

2 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] Information-Theoretic Quantities

3 Conditional Entropies

Amount of randomness in ≡log ≡ ∈

given a particular value |

Amount of randomness | ≡ | ≡ | log | still remained in when ∈ we know that . Apply the entropy calculation to a row from the matrix

The average amount of | ≡ | randomness still remained in ∈ when we know , 4 ; Diagrams [Figure 16] Venn Diagram Probability Diagram

\B ∩ \A \B ∩ \A

∪ Information Diagram ∪

| ; |

5 , Diagrams [Figure 16]

Probability Diagram Information Diagram

\B ∩ \A | ; |

∪ ,

6 Diagrams

Probability Diagram Information Diagram

\B ∩ \A | ; |

∪ ,

7 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] Operational Channel Capacity

8 Example: Repetition Code [Figure 14]  Original Equivalent Channel: 0 1-p 0 p p 1 1-p 1  BSC with crossover probability p = 0.01  New (and Better) Equivalent Channel:

1-p Repetition 0 0 Majority 0 0 Code with p p Vote 1 1 n = 5 1 1-p 1

 Use repetition code with n = 5 at the transmitter  Use majority vote at the receiver 5 5 5 1 1 1  New BSC with 3 4 5 10 9 Example: Repetition Code  Original Equivalent Channel: 0 1-p 0 p p 1 1-p 1  BSC with crossover probability p = 0.1  New (and Better) Equivalent Channel:

0 1-p 0 0 0 Repetition p Majority p Vote Code 1 1 1 1-p 1

 Use repetition code at the transmitter  Use majority vote at the receiver  New BSC with new crossover probability 10 Reminder [From ECS315]

11 [From ECS315]

12 Example: Repetition Code

0 1-p 0 0 0 Repetition p Majority p Vote Code 1 1 1 1-p 1

1 0.1 3 3 3 1 0.0280 2 3 5 5 5 5 1 1 0.0086 3 4 5 7 0.0027 9 8.9092 10 11 2.9571 10

13 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] Information Channel Capacity

14 Channel Capacity

“Operational”: max rate at which reliable communication is possible

Channel Capacity Arbitrarily small error probability can be achieved.

“Information”: [bpcu]

Shannon [1948] shows that these two quantities are actually the same.

15 MATLAB

function H = entropy2s(p) % ENTROPY2 accepts probability mass function % as a row vector, calculate the corresponding % entropy in bits. p=p(find(abs(sort(p)-1)>1e-8)); % Eliminate 1 p=p(find(abs(p)>1e-8)); % Eliminate 0 if length(p)==0 H = 0; else H = simplify(-sum(p.*log(p))/log(sym(2))); end

function I = informations(p,Q) X = length(p); q = p*Q; HY = entropy2s(q); temp = []; for i = 1:X temp = [temp entropy2s(Q(i,:))]; end HYgX = sum(p.*temp); I = HY-HYgX; 16 Capacity calculation for BAC

0.1 0.1 0 0 0.09 0.9 0.08 X 0.4 Y 0.07 0.06 1 1 0.05

0.6 I(X;Y) 0.04 0.1 0.9 0.03 Q  0.02 0.4 0.6 0.01 0  0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p 0

Capacity of 0.0918 bits is achieved by p  0.5380, 0.4620

17 Capacity calculation for BAC

0 0.1 0 0.9 0.1 0.9 XYQ  0.4 0.4 0.6 1 0.6 1   close all; clear all; >> Capacity_Ex_BAC syms p0 I = p = [p0 1-p0]; (log(2/5 - (3*p0)/10)*((3*p0)/10 - 2/5) - log((3*p0)/10 + 3/5)*((3*p0)/10 + Q = [1 9; 4 6]/sym(10); 3/5))/log(2) + (log((5*2^(3/5)*3^(2/5))/6)*(p0 - 1))/log(2) + (p0*log((3*3^(4/5))/10))/log(2) I = simplify(informations(p,Q)) p0o = (27648*2^(1/3))/109565 - (69984*2^(2/3))/109565 + 135164/109565 p0o = simplify(solve(diff(I)==0)) po = 0.5376 0.4624 C = po = eval([p0o 1-p0o]) (log((3*3^(4/5))/10)*((27648*2^(1/3))/109565 - (69984*2^(2/3))/109565 + 135164/109565))/log(2) - (log((104976*2^(2/3))/547825 - (41472*2^(1/3))/547825 + 16384/547825)*((104976*2^(2/3))/547825 - (41472*2^(1/3))/547825 + C = simplify(subs(I,p0,p0o)) 16384/547825) + log((41472*2^(1/3))/547825 - (104976*2^(2/3))/547825 + 531441/547825)*((41472*2^(1/3))/547825 - (104976*2^(2/3))/547825 + 531441/547825))/log(2) + (log((5*2^(3/5)*3^(2/5))/6)*((27648*2^(1/3))/109565 - eval(C) (69984*2^(2/3))/109565 + 25599/109565))/log(2) ans = 18 0.0918 Same procedure applied to BSC

0 0.6 0 0.4 0.6 0.4 XY0.4 Q  0.4 0.6 1 0.6 1   close all; clear all; >> Capacity_Ex_BSC syms p0 I = p = [p0 1-p0]; (log((5*2^(3/5)*3^(2/5))/6)*(p0 - 1))/log(2) - Q = [6 4; 4 6]/sym(10); (p0*log((5*2^(3/5)*3^(2/5))/6))/log(2) - (log(p0/5 + 2/5)*(p0/5 + 2/5) - log(3/5 - p0/5)*(p0/5 - I = simplify(informations(p,Q)) 3/5))/log(2) p0o = p0o = simplify(solve(diff(I)==0)) 1/2 po = po = eval([p0o 1-p0o]) 0.5000 0.5000 C = C = simplify(subs(I,p0,p0o)) log((2*2^(2/5)*3^(3/5))/5)/log(2) ans = eval(C) 0.0290

19 Blahut–Arimoto algorithm

function [ps C] = capacity_blahut(Q) % Input: Q = channel transition probability matrix % Output: C = channel capacity % ps = row vector containing pmf that achieves capacity

tl = 1e-8; % tolerance (for the stopping condition) n = 1000; % max number of iterations (in case the stopping condition % is "never" reached") nx = size(Q,1); pT = ones(1,nx)/nx; % First, guess uniform X. for k = 1:n qT = pT*Q; % Eliminate the case with 0 % Column-division by qT temp = Q.*(ones(nx,1)*(1./qT)); %Eliminate the case of 0/0 l2 = log2(temp); l2(find(isnan(l2) | (l2==-inf) | (l2==inf)))=0; logc = (sum(Q.*(l2),2))'; CT = 2.^(logc); A = log2(sum(pT.*CT)); B = log2(max(CT)); if((B-A)

0 0.1 0 0.9 0.1 0.9 XYQ  0.4 0.4 0.6 1 0.6 1  

close all; clear all; >> Capacity_Ex_BAC_blahut ps = Q = [1 9; 4 6]/10; 0.5376 0.4624 C = [ps C] = capacity_blahut(Q) 0.0918

21 Richard Blahut  Former chair of the Electrical and Computer Engineering Department at the University of Illinois at Urbana-Champaign  Best known for Blahut–Arimoto algorithm (Iterative Calculation of C)

22 Claude E. Shannon Award

23 [ http://www.itsoc.org/honors/claude-e-shannon-award ] Berger plaque

24 Raymond Yeung  BS, MEng and PhD degrees in electrical engineering from Cornell University in 1984, 1985, and 1988, respectively.