Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong (ผศ.ดร.ประพันธ ์ สขสมปองุ ) [email protected] 1. Intro to Digital Communication Systems Office Hours: BKD, 6th floor of Sirindhralai building Tuesday 14:20-15:20 Wednesday 14:20-15:20
1 Friday 9:15-10:15 “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.”
Shannon, Claude. A Mathematical Theory Of Communication. (1948)
2 Shannon: Father of the Info. Age
Documentary Co-produced by the Jacobs School, UCSD-TV, and the California Institute for Telecommunications and Information Technology Won a Gold award in the Biography category in the 2002 Aurora Awards.
3 [http://www.uctv.tv/shows/Claude-Shannon-Father-of-the-Information-Age-6090] [http://www.youtube.com/watch?v=z2Whj_nL-x8] C. E. Shannon (1916-2001) 1938 MIT master's thesis: A Symbolic Analysis of Relay and Switching Circuits Insight: The binary nature of Boolean logic was analogous to the ones and zeros used by digital circuits. The thesis became the foundation of practical digital circuit design. The first known use of the term bit to refer to a “binary digit.” Possibly the most important, and also the most famous, master’s thesis of the century. It was simple, elegant, and important.
4 C. E. Shannon: Master Thesis
5 Boole/Shannon Celebration Events in 2015 and 2016 centered around the work of George Boole, who was born 200 years ago, and Claude E. Shannon, born 100 years ago. Events were scheduled both at the University College Cork (UCC), Ireland and the Massachusetts Institute of Technology (MIT)
6 http://www.rle.mit.edu/booleshannon/ An Interesting Book The Logician and the Engineer: How George Boole and Claude Shannon Created the Information Age by Paul J. Nahin ISBN: 9780691151007 http://press.princeton.edu/titles/ 9819.html
7 C. E. Shannon (Con’t) 1948: A Mathematical Theory of Communication Bell System Technical Journal, vol. 27, pp. 379-423, July- October, 1948. September 1949: Book published. Include a new section by Warren Weaver that applied Shannon's theory to Invent Information Theory: human communication. Simultaneously founded the subject, introduced all of the Create the architecture and major concepts, and stated and concepts governing digital proved all the fundamental communication. theorems. 8 A Mathematical Theory of Communication Link posted in the “references” section of the website.
9 [An offprint from the Bell System Technical Journal] C. E. Shannon
10 …with some remarks by Toby Berger. Claude E. Shannon Award
Claude E. Shannon (1972) Elwyn R. Berlekamp (1993) Sergio Verdu (2007) David S. Slepian (1974) Aaron D. Wyner (1994) Robert M. Gray (2008) Robert M. Fano (1976) G. David Forney, Jr. (1995) Jorma Rissanen (2009) Peter Elias (1977) Imre Csiszár (1996) Te Sun Han (2010) Mark S. Pinsker (1978) Jacob Ziv (1997) Shlomo Shamai (Shitz) (2011) Jacob Wolfowitz (1979) Neil J. A. Sloane (1998) Abbas El Gamal (2012) W. Wesley Peterson (1981) Tadao Kasami (1999) Katalin Marton (2013) Irving S. Reed (1982) Thomas Kailath (2000) János Körner (2014) Robert G. Gallager (1983) Jack KeilWolf (2001) Arthur Robert Calderbank (2015) Solomon W. Golomb (1985) Toby Berger (2002) Alexander S. Holevo (2016) William L. Root (1986) Lloyd R. Welch (2003) James L. Massey (1988) Robert J. McEliece (2004) Thomas M. Cover (1990) Richard Blahut (2005) Andrew J. Viterbi (1991) Rudolf Ahlswede (2006)
11 [ http://www.itsoc.org/honors/claude-e-shannon-award ] [http://www.ieee.org/documents/hamming_rl.pdf] IEEE Richard W. Hamming Medal
1988 - Richard W. Hamming 2003 - Claude Berrou and Alain Glavieux 1989 - Irving S. Reed 2004 - Jack K. Wolf 1990 - Dennis M. Ritchie and Kenneth L. 2005 - Neil J.A. Sloane Thompson 2006 -Vladimir I. Levenshtein 1991 - Elwyn R. Berlekamp 2007 - Abraham Lempel 1992 - Lotfi A. Zadeh 2008 - Sergio Verdú 1993 - Jorma J. Rissanen 2009 - Peter Franaszek 1994 - Gottfried Ungerboeck 2010 -Whitfield Diffie, Martin Hellman 1995 - Jacob Ziv and Ralph Merkle 1996 - Mark S. Pinsker 2011 -Toby Berger 1997 -Thomas M. Cover 2012 - Michael Luby, Amin Shokrollahi 1998 - David D. Clark 2013 - Arthur Robert Calderbank 1999 - David A. Huffman 2014 -Thomas Richardson and Rüdiger L. 2000 - Solomon W. Golomb Urbanke 2001 - A. G. Fraser 2015 - Imre Csiszar 2002 - Peter Elias 2016 - Abbas El Gamal “For contributions to Information Theory, including 12 source coding and its applications.” [http://www.cvaieee.org/html/toby_berger.html] Information Theory The science of information theory tackles the following questions [Berger] 1. What is information, i.e., how do we measure it quantitatively? 2. What factors limit the reliability with which information generated at one point can be reproduced at another, and what are the resulting limits? 3. How should communication systems be designed in order to achieve or at least to approach these limits?
13 Elements of communication sys. (ECS 332)
Analog (continuous) Digital (discrete) Transmitted Received Message Message Signal Signal Information Transmitter Channel Receiver Destination Source Modulation Amplification Coding Demodulation Decoding Noise, Filtering Interference, Distortion + Transmission loss (attenuation)
14 The Switch to Digital TV
Japan: Starting July 24, 2011, the analog broadcast has ceased and only digital broadcast is available. US: Since June 12, 2009, full- power television stations nationwide have been broadcasting exclusively in a digital format. Thailand: Use DVB-T2. Launched in 2014.
15 [https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Digital_broadcast_standards.svg/800px-Digital_broadcast_standards.svg.png] News: The Switch to Digital Radio Norway (the mountainous nation of 5 million) is the first country to start shutting down its national FM radio network in favor of digital radio. Start on January 11, 2017 At which point, 99.5% of the population has access to DAB reception with almost three million receivers sold. 70% of Norwegian households regularly tune in digitally Take place over a 12-month period, conducting changes region by region. By the end of the year all national networks will be DAB-only, while local broadcasters have five years to phase out their FM stations. New format: Digital Audio Broadcasting (DAB)
http://gizmodo.com/norway-is-killing-fm-radio-tomorrow-1791019824 http://www.worlddab.org/country-information/norway http://www.smithsonianmag.com/smart-news/norway-killed-radio-star-180961761/ 16 http://www.latimes.com/world/la-fg-norway-radio-20170114-story.html https://www.newscientist.com/article/2117569-norway-is-first-country-to-turn-off-fm-radio-and-go-digital-only/ Digital Audio Broadcasting Initiated as a European research project in the 1980s. The Norwegian Broadcasting Corporation (NRK) launched the first DAB channel in the world on 1 June 1995 (NRK Klassisk) The BBC and Swedish Radio (SR) launched their first DAB digital radio broadcasts in September 1995. Audio quality varies depending on the bitrate used.
17 The Switch to DAB in Norway Co-exist with FM since 1995. Provide a clearer and more reliable network that can better cut through the country's sparsely populated rocky terrain. FM has always been problematic in Norway since the nation’s mountains and fjords makes getting clear FM signals difficult. Offer more channels at a fraction of the cost. Allow 8 times as many radio stations Norway currently has five national FM radio stations. With DAB, it will be able to have around 40. The FM radio infrastructure was coming to the end of its life, Need to either replace it or fully commit to DAB anyway Can run at lower power levels the infrastructure electricity bills are lower 18 The Switch to Digital Radio Switzerland and Denmark are also interested in phasing out FM Great Britain says it will look at making the switch once 50 percent of listeners use digital formats currently at 35 percent Unlikely to happen before 2020. and when the DAB signal reaches 90 percent of the population. Germany had set a 2015 date for dumping FM many years ago, but lawmakers reversed that decision in 2011. In North America, FM radio, which has been active since the 1940s, shows no sign of being replaced any time soon, either in the United States or Canada. There are around 4,000 stations using HD radio technology in the United States, and HD radio receivers are now common fixtures in new cars. In Thailand, NBTC planed to start digital radio trial within 2018.
19 [ https://en.wikipedia.org/wiki/HD_Radio
Selected by the U.S. FCC in 2002 as a digital audio broadcasting method for the United States. Embed digital signal “on-frequency” immediately above and below a station’s standard analog signal Provide the means to listen to the same program in either HD ] (digital radio with less noise) or as a standard broadcast (analog radio with standard sound quality).
Spectrum of FM broadcast station
20 without HD Radio with HD Radio Countries using DAB/DMB
21 https://en.wikipedia.org/wiki/Digital_audio_broadcasting Pokémon Communications
22 Pikachu's language
Some of Pikachu's speech is consistent enough that it seems that some phrases actually mean something. Pikachu always uses "Pikapi" when referring to Ash (notice that it sounds somewhat similar to "Satoshi"). Pi-Kachu: He says this during the sponsor spots in the original Japanese, Pochama (Piplup) Pikachu-Pi: Kasumi (Misty) Pika-Chu: Takeshi (Brock), Kibago (Axew) Pikaka: Hikari (Dawn) PiPiPi: Togepy (Togepi) PikakaPika: Fushigidane (Bulbasaur) PikaPika: Zenigame (Squirtle), Mukuhawk (Staraptor), Goukazaru (Infernape) or Gamagaru (Palpitoad) PiPi-kachu: Rocket-dan (Team Rocket) Pi-Pikachu: Get da ze! (He says this after Ash wins a Badge, catches a new Pokémon or anything similar.) Pikachu may not be the only one to use this phrase, as other Pokémon do this as well. For example, when Iris caught Emolga, Axew said Ax-Axew (Ki-Kibago in the Japanese). Pika-Pikachu: He says this when referring to himself. Four-symbol variable-length code?
23 [https://www.youtube.com/watch?v=XumQrRkGXck] Rate-Distortion Theory The theory of lossy source coding
24 Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong [email protected] 2. Source Coding
Office Hours: BKD, 6th floor of Sirindhralai building Tuesday 14:20-15:20 Wednesday 14:20-15:20
1 Friday 9:15-10:15 Elements of digital commu. sys.
Message Transmitter
Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add Signal redundancy systematic redundancy Channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator
2 Main Reference Elements of Information Theory 2006, 2nd Edition Chapters 2, 4 and 5 ‘the jewel in Stanford's crown’ One of the greatest information theorists since Claude Shannon (and the one most like Shannon in approach, clarity, and taste).
3 US UK The ASCII Coded Character Set (American Standard Code for Information Interchange)
016 32 48 64 80 96 112
4 [The ARRL Handbook for Radio Communications 2013] Introduction to Data Compression
5 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ] English Redundancy: Ex. 1
J-st tr- t- r--d th-s s-nt-nc-.
6 English Redundancy: Ex. 2
yxx cxn xndxrstxnd whxt x xm wrxtxng xvxn xf x rxplxcx xll thx vxwxls wxth xn 'x' (t gts lttl hrdr f y dn't vn kn whr th vwls r).
7 English Redundancy: Ex. 3
To be, or xxx xx xx, xxxx xx xxx xxxxxxxx
8 Entropy Rate of Thai Text
9 Ex. Source alphabet of size = 4
10 Ex. DMS (1)
1 ,,,,,xabcde px 5 X abcde,,,, X 0, otherwise Information Source a c a c e c d b c e d a e e d a b b b d b b a a b e b e d c c e d b c e c a a c a a e a c c a a d c d e e a a c a a a b b c a e b b e d b c d e b c a e e d d c d a b c a b c d d e d c e a b a a c a d
11 Approximately 20% are letter ‘a’s [GenRV_Discrete_datasample_Ex1.m] Ex. DMS (1) clear all; close all;
S_X = 'abcde'; p_X = [1/5 1/5 1/5 1/5 1/5];
n = 100; MessageSequence = datasample(S_X,n,'Weights',p_X) MessageSequence = reshape(MessageSequence,10,10)
>> GenRV_Discrete_datasample_Ex1
MessageSequence =
eebbedddeceacdbcbedeecacaecedcaedabecccabbcccebdbbbeccbadeaaaecceccdaccedadabceddaceadacdaededcdcade
MessageSequence =
eeeabbacde eacebeeead bcadcccdce bdcacccaed ebabcbedac dceeeacadd dbccbdcbac deecdedcca eddcbaaedd cecabacdae
12 [GenRV_Discrete_datasample_Ex1.m] 1 ,1,x Ex. DMS (2) 2 1 ,2,x px 4 X 1, 2, 3, 4 X 1 ,3,4x 8 0, otherwise Information Source 2 1 1 2 1 4 1 1 1 1 1 1 4 1 1 2 4 2 2 1 3 1 1 2 3 2 4 1 2 4 2 1 1 2 1 1 3 3 1 1 1 3 4 1 4 1 1 2 4 1 4 1 4 1 2 2 1 4 2 1 4 1 1 1 1 2 1 4 2 4 2 1 1 1 2 1 2 1 3 2 2 1 1 1 1 1 1 2 3 2 2 1 1 2 1 4 2 1 2 1
13 Approximately 50% are number ‘1’s [GenRV_Discrete_datasample_Ex2.m] Ex. DMS (2)
clear all; close all;
S_X = [1 2 3 4]; p_X = [1/2 1/4 1/8 1/8];
n = 20;
MessageSequence = randsrc(1,n,[S_X;p_X]); %MessageSequence = datasample(S_X,n,'Weights',p_X);
rf = hist(MessageSequence,S_X)/n; % Ref. Freq. calc. 0.5 stem(S_X,rf,'rx','LineWidth',2) % Plot Rel. Freq. Rel. freq. from sim. 0.45 pmf p (x) hold on X stem(S_X,p_X,'bo','LineWidth',2) % Plot pmf 0.4 xlim([min(S_X)-1,max(S_X)+1]) legend('Rel. freq. from sim.','pmf p_X(x)') 0.35 xlabel('x') 0.3 grid on 0.25
0.2
0.15
0.1
0.05
0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 x
14 [GenRV_Discrete_datasample_Ex2.m] DMS in MATLAB
clear all; close all;
S_X = [1 2 3 4]; p_X = [1/2 1/4 1/8 1/8]; n = 1e6;
SourceString = randsrc(1,n,[S_X;p_X]);
Alternatively, we can also use SourceString = datasample(S_X,n,'Weights',p_X);
rf = hist(SourceString,S_X)/n; % Ref. Freq. calc. stem(S_X,rf,'rx','LineWidth',2) % Plot Rel. Freq. hold on stem(S_X,p_X,'bo','LineWidth',2) % Plot pmf xlim([min(S_X)-1,max(S_X)+1]) legend('Rel. freq. from sim.','pmf p_X(x)') xlabel('x') 15 grid on [GenRV_Discrete_datasample_Ex.m] A more realistic example of pmf:
Relative freq. of letters in the English language
16 [http://en.wikipedia.org/wiki/Letter_frequency] A more realistic example of pmf:
Relative freq. of letters in the English language ordered by frequency
17 [http://en.wikipedia.org/wiki/Letter_frequency] Example: ASCII Encoder
Character Codeword x c(x) ⋮ MATLAB: E 1000101 >> M = 'LOVE'; ⋮ >> X = dec2bin(M,7); >> X = reshape(X',1,numel(X)) L 1001100 X = ⋮ 1001100100111110101101000101
Codebook O 1001111 Remark: ⋮ numel(A) = prod(size(A)) V 1010110 (the number of elements in matrix A) ⋮
c(“L”) c(“O”) c(“V”) c(“E”) Information “LOVE” Source “1001100100111110101101000101” Source Encoder 18 The ASCII Coded Character Set
016 32 48 64 80 96 112
19 [The ARRL Handbook for Radio Communications 2013] A Byte (8 bits) vs. 7 bits
>> dec2bin('I Love ECS452',7) >> dec2bin('I Love ECS452',8) ans = ans = 1001001 01001001 0100000 00100000 1001100 01001100 1101111 01101111 1110110 01110110 1100101 01100101 0100000 00100000 1000101 01000101 1000011 01000011 1010011 01010011 0110100 00110100 0110101 00110101 0110010 00110010 20 Geeky ways to express your love >> dec2bin('I Love You',8) >> dec2bin('i love you',8) ans = ans = 01001001 01101001 00100000 00100000 01001100 01101100 01101111 01101111 01110110 01110110 01100101 01100101 00100000 00100000 01011001 01111001 01101111 01101111 01110101 01110101 https://www.etsy.com/listing/91473057/binary-i-love-you-printable-for- your?ref=sr_gallery_9&ga_search_query=binary&ga_filters=holidays+- supplies+valentine&ga_search_type=all&ga_view_type=gallery http://mentalfloss.com/article/29979/14-geeky-valentines-day-cards https://www.etsy.com/listing/174002615/binary-love-geeky-romantic-pdf- cross?ref=sr_gallery_26&ga_search_query=binary&ga_filters=holidays+- supplies+valentine&ga_search_type=all&ga_view_type=gallery https://www.etsy.com/listing/185919057/i-love-you-binary-925-silver-dog-tag- 21 can?ref=sc_3&plkey=cdf3741cf5c63291bbc127f1fa7fb03e641daafd%3A185919057&ga_search_query=binary &ga_filters=holidays+-supplies+valentine&ga_search_type=all&ga_view_type=gallery http://www.cafepress.com/+binary-code+long_sleeve_tees Morse code (wired and wireless) Telegraph network Samuel Morse, 1838 A sequence of on-off tones (or , lights, or clicks)
22 Example
23 [http://www.wolframalpha.com/input/?i=%22I+love+you.%22+in+Morse+code] Example
24 Morse code: Key Idea
Frequently-used characters are mapped to short codewords.
25 Relative frequencies of letters in the English language Morse code: Key Idea Frequently-used characters (e,t) are mapped to short codewords.
26 Relative frequencies of letters in the English language Morse code: Key Idea Frequently-used characters (e,t) are mapped to short codewords.
Basic form of compression.
27 รหสมอรั ์สภาษาไทย
28 Example: ASCII Encoder
Character Codeword ⋮ MATLAB: E 1000101 >> M = 'LOVE'; ⋮ >> X = dec2bin(M,7); L 1001100 >> X = reshape(X',1,numel(X)) ⋮ X = 1001100100111110101101000101 O 1001111 ⋮ V 1010110 ⋮
Information “LOVE” Source “1001100100111110101101000101” Source Encoder 29 Another Example of non-UD code
xc(x) Consider the string A1 011101110011. B 011 It can be interpreted as C 01110 CDB: 01110 1110 011 D 1110 BABE: 011 1 011 10011 E 10011
30 Game: 20 Questions 20 Questions is a classic game that has been played since the 19th century. One person thinks of something (an object, a person, an animal, etc.) The others playing can ask 20 questions in an effort to guess what it is.
31 20 Questions: Example
32 Prof. Robert Fano (1917-2016) Shannon Award (1976 ) Shannon–Fano coding Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948 The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). “The transmission of information”. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT. Should not be confused with Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with Shannon–Fano–Elias coding (also known as Elias coding), the precursor to arithmetic coding.
33 David Huffman (1925–1999) Huffman Code Hamming Medal (1999) MIT, 1951 Information theory class taught by Professor Fano. Huffman and his classmates were given the choice of a term paper on the problem of finding the most efficient binary code. or a final exam. Huffman, unable to prove any codes were the most efficient, was about to give up and start studying for the final when he hit upon the idea of using a frequency-sorted binary tree and quickly proved this method the most efficient. Huffman avoided the major flaw of the suboptimal Shannon-Fano coding by building the tree from the bottom up instead of from the top down.
34 Huffman’s paper (1952)
[D. A. Huffman, "A Method for the Construction of Minimum-Redundancy 35 Codes," in Proceedings of the IRE, vol. 40, no. 9, pp. 1098-1101, Sept. 1952.] [ http://ieeexplore.ieee.org/document/4051119/ ] Huffman coding
36 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/compressioncodes ] Ex. Huffman Coding in MATLAB [Ex. 2.31] Observe that pX = [0.5 0.25 0.125 0.125]; % pmf of X MATLAB SX = [1:length(pX)]; % Source Alphabet automatically give [dict,EL] = huffmandict(SX,pX); % Create codebook the expected length of the %% Pretty print the codebook. codewords codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook
%% Try to encode some random source string n = 5; % Number of source symbols to be generated sourceString = randsrc(1,10,[SX; pX]) % Create data using pX encodedString = huffmanenco(sourceString,dict) % Encode the data
37 [Huffman_Demo_Ex1] Ex. Huffman Coding in MATLAB
codebook =
[1] '0' [2] '1 0' [3] '1 1 1' [4] '1 1 0'
sourceString =
1 4 4 1 3 1 1 4 3 4
encodedString =
0 1 1 0 1 1 0 0 1 1 1 0 0 1 1 0 1 1 1 1 1 0
38 [Huffman_Demo_Ex1] Ex. Huffman Coding in MATLAB [Ex. 2.32] pX = [0.4 0.3 0.1 0.1 0.06 0.04]; % pmf of X SX = [1:length(pX)]; % Source Alphabet [dict,EL] = huffmandict(SX,pX); % Create codebook
%% Pretty print the codebook. >> Huffman_Demo_Ex2 codebook = dict; for i = 1:length(codebook) codebook = codebook{i,2} = num2str(codebook{i,2}); end codebook [1] '1' [2] '0 1' EL [3] '0 0 0 0' [4] '0 0 1' [5] '0 0 0 1 0' The codewords can be different [6] '0 0 0 1 1' from our answers found earlier. EL = The expected length is the same. 2.2000 39 [Huffman_Demo_Ex2] Ex. Huffman Coding in MATLAB [Exercise] pX = [1/8, 5/24, 7/24, 3/8]; % pmf of X SX = [1:length(pX)]; % Source Alphabet [dict,EL] = huffmandict(SX,pX); % Create codebook
%% Pretty print the codebook. codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook
EL
codebook = [1] '0 0 1' >> -pX*(log2(pX)).' [2] '0 0 0' ans = [3] '0 1' 1.8956 [4] '1' EL = 1.9583 40 Let’s talk about TV series on HBO
41 Silicon Valley (TV series) Focus around six young men who found a startup company in Silicon Valley. In the first season, the company develop a “revolutionary” data compression algorithm: The “middle-out” algorithm
42 Behind the Scene When Mike Judge set out to write Silicon Valley, he wanted to conceive a simple, believable widget for his characters to invent. He teamed with Stanford electrical engineering professor TsachyWeissman and a PhD student Vinith Misra They came up with a fictional breakthrough compression algorithm. “We had to come up with an approach that isn’t possible today, but it isn’t immediately obvious that it isn’t possible,” says Misra. The writers also coined a metric, the “Weissman Score,” for characters to use when comparing compression codes.
43 [May 2014 issue of Popular Science] Middle-Out Algorithm: Into the Real World Something like it can be found in Lepton A new lossless image compressor created by Dropbox. Lepton reduces the file size of JPEG-encoded images by as much as 22 percent, yet without losing a single bit of the original. How is this possible? Middle-out. Well, it’s much more complicated than that, actually. The middle-out bit comes toward the end of the decompression bit. Lepton is open source, and Dropbox has put the code for it on GitHub.
44 [ https://techcrunch.com/2016/07/14/dropboxs-lepton-lossless-image-compression-really-uses-a-middle-out-algorithm/ ] Weissman Score: Into the Real World It’s hard to convey to a lay audience that one compression algorithm is better than another. Created by Misra (Prof. Weissman’s PhD student) for the show. Metrics for compression algorithms that rate not only the amount of compression but the processing speed, are hard to find.
45 http://spectrum.ieee.org/view-from-the-valley/computing/software/a-madefortv-compression-metric-moves-to-the-real-world http://spectrum.ieee.org/view-from-the-valley/computing/software/a-madefortv-compression-algorithm Summary A good code must be uniquely decodable (UD). Difficult to check. A special family of codes called prefix(-free) code is always UD. They are also instantaneous. Huffman’s recipe Repeatedly combine the two least-likely (combined) symbols Automatically give prefix code For a given source’s pmf, Huffman codes are optimal among all UD codes for that source.
46 0
-0.05
-0.1
-0.15
-0.2
-0.25
-0.3
-0.35
-0.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x 47 Entropy and Description of RV
48 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy ] Entropy and Description of RV
49 [ https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy ] Kronecker Product An operation on two matrices of arbitrary size Named after German mathematician Leopold Kronecker. If A is an m-by-n matrix and B is a p-by-q matrix, then the Kronecker product AB is the mp-by-nq matrix
Use aB11 aB 1n kron(A,B) AB . in MATLAB. aBmmn1 aB Example 11··2·02·50501005 1 2··2·62·7671214051 617 . 3 467 3·0 3·5 4·0 4·5 0 15 0 20 3·6 3·7 4·6 4·7 18 21 24 28 50 Kronecker Product
>> p = [0.9 0.1] p = 0.9000 0.1000 >> p2 = kron(p,p) p2 = 0.8100 0.0900 0.0900 0.0100 >> p3 = kron(p2,p) p3 = Columns 1 through 7 0.7290 0.0810 0.0810 0.0090 0.0810 0.0090 0.0090 Column 8 0.0010
51 [Ex.2.40] Huffman Coding: Source Extension
1 1
i.i.d. 0.9 X k Bernoullip
p 0.1 0.8
Ln 0.7 0.645
0.6
0.533 0.5
0.4 1 2 3 4 5 6 7 8 n: order of extension 52 [Ex.2.40] Huffman Coding: Source Extension
1.8
i.i.d. 1.6
X k Bernoullip p 0.1 1.4
1.2
1 1 HX n 0.8 Ln 0.6 HX 0.4 1 2 3 4 5 6 7 8 Order of source extension 53 n Final Remarks about Huffman Code Huffman coding compresses an i.i.d. source with a known
pmf pX(x) to its entropy limit H(X). Sensitive to the assumed distribution. If the code is designed for some incorrect distribution, a penalty is incurred.
What compression can be achieved if the true pmf pX(x) is unknown? One may assume uniform pmf Inefficient if the actual pmf is not uniform. Lempel-Ziv algorithm
54 Lempel-Ziv algorithm Often used in practice to compress data that cannot be modeled simply Could also be described as adaptive dictionary compression algorithms. Ziv and Lempel wrote their papers in 1977 and 1978. The two papers describe two distinct versions of the algorithm. LZ77: sliding window Lempel–Ziv LZ78: tree-structured Lempel–Ziv
55 Arithmetic Coding The Huffman coding procedure is optimal for encoding a random variable with a known pmf that has to be encoded symbol by symbol. Coding Inefficiency of Huffman Code: The codeword lengths for a Huffman code were restricted to be integer-valued There could be a loss of up to 1 bit per symbol in coding efficiency. We could alleviate this loss by encoding blocks of source symbols The complexity of this approach increases exponentially with block length n. In arithmetic coding, instead of using a sequence of bits to represent a symbol, we represent it by a subinterval of the unit interval.
56 Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong [email protected] 3 Discrete Memoryless Channel (DMC)
Office Hours: BKD, 6th floor of Sirindhralai building Tuesday 14:20-15:20 Wednesday 14:20-15:20
1 Friday 9:15-10:15 Elements of digital commu. sys.
Message Transmitter
Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add Signal redundancy systematic (compression) redundancy to combat errors Channel introduced by the channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator
2 System considered previously
Message Transmitter
Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add Signal redundancy systematic redundancy Channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator
3 System considered in this section
Message Transmitter
Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add X: channelSignal input redundancy systematic redundancy EquivalentChannel Channel Noise & Interference Recovered Message Receiver Received Y: channelSignal output Source Channel Digital Destination Decoder Decoder Demodulator
4 Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong [email protected] 3.1 DMC Models
5 MATLAB
%% Generating the channel input x x = randsrc(1,n,[S_X;p_X]); % channel input
%% Applying the effect of the channel to create the channel output y y = DMC_Channel_sim(x,S_X,S_Y,Q); % channel output
function y = DMC_Channel_sim(x,S_X,S_Y,Q) %% Applying the effect of the channel to create the channel output y y = zeros(size(x)); % preallocation for k = 1:length(x) % Look at the channel input one by one. Choose the corresponding row % from the Q matrix to generate the channel output. y(k) = randsrc(1,1,[S_Y;Q(find(S_X == x(k)),:)]); end [DMC_Channel_sim.m]
6 [DMC_Analysis_demo.m] %% Simulation parameters % The number of symbols to be transmitted [Example 3.2] n = 20; % Channel Input Ex: BSC S_X = [0 1]; S_Y = [0 1]; p_X = [0.3 0.7]; >> BSC_demo % Channel Characteristics ans = p = 0.1; Q = [1-p p; p 1-p]; 1 0 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1 ans = 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1
p_X = Q = 0.3000 0.7000 0.9000 0.1000 p_X_sim = 0.1000 0.9000 0.1500 0.8500 Q_sim = q = 0.6667 0.3333 0.3400 0.6600 0.0588 0.9412 q_sim = PE_sim = 0.1500 0.8500 0.1000 PE_theretical = 0.1000
7 Elapsed time is 0.134992 seconds. [BSC_demo.m] Rel. freq. from the simulation %% Statistical Analysis % The probability values for the channel inputs p_X % Theoretical probability p_X_sim = hist(x,S_X)/n % Relative frequencies from the simulation % The probability values for the channel outputs q = p_X*Q % Theoretical probability q_sim = hist(y,S_Y)/n % Relative frequencies from the simulation % The channel transition probabilities from the simulation Q_sim = []; for k = 1:length(S_X) I = find(x==S_X(k)); LI = length(I); rel_freq_Xk = LI/n; yc = y(I); cond_rel_freq = hist(yc,S_Y)/LI; Q_sim = [Q_sim; cond_rel_freq]; end Q % Theoretical probability Q_sim % Relative frequencies from the simulation
8 [DMC_Analysis_demo.m] %% Simulation parameters % The number of symbols to be transmitted [Example 3.2] n = 20; % Channel Input Ex: BSC S_X = [0 1]; S_Y = [0 1]; p_X = [0.3 0.7]; >> BSC_demo % Channel Characteristics ans = p = 0.1; Q = [1-p p; p 1-p]; 1 0 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1 ans = 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1
p_X = Q = 0.3000 0.7000 0.9000 0.1000 Because there are only 20 p_X_sim = 0.1000 0.9000 samples, we can’t expect the 0.1500 0.8500 Q_sim = relative freq. from the q = 0.6667 0.3333 simulation to match the 0.3400 0.6600 0.0588 0.9412 specified or calculated q_sim = PE_sim = probabilities. 0.1500 0.8500 0.1000 PE_theretical = 0.1000
9 Elapsed time is 0.134992 seconds. [BSC_demo.m] %% Simulation parameters % The number of symbols to be transmitted n = 1e4; % Channel Input Ex: BSC S_X = [0 1]; S_Y = [0 1]; p_X = [0.3 0.7]; % Channel Characteristics p = 0.1; Q = [1-p p; p 1-p]; >> BSC_demo
p_X = Q = 0.3000 0.7000 0.9000 0.1000 p_X_sim = 0.1000 0.9000 0.3037 0.6963 Q_sim = q = 0.9078 0.0922 0.3400 0.6600 0.0934 0.9066 q_sim = PE_sim = 0.3407 0.6593 0.0930 PE_theretical = 0.1000
Elapsed time is 0.922728 seconds.
10 [BSC_demo.m] Ex: DMC %% Simulation parameters p_X = % The number of symbols to be transmitted n = 20; 0.2000 0.8000 % General DMC p_X_sim = % Ex. 3.6 amd 3.12 in lecture note % Channel Input 0.2000 0.8000 S_X = [0 1]; S_Y = [1 2 3]; q = p_X = [0.2 0.8]; % Channel Characteristics 0.3400 0.3600 0.3000 Q = [0.5 0.2 0.3; 0.3 0.4 0.3]; q_sim = 0.4000 0.3500 0.2500 >> DMC_demo Q = ans = 0.5000 0.2000 0.3000 x: 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0.3000 0.4000 0.3000 ans = Q_sim = y: 1 3 2 2 1 2 1 2 2 3 1 1 1 3 1 3 2 3 1 2 0.7500 0 0.2500 0.3125 0.4375 0.2500 . 0 0.5 1 >> sym(Q_sim) ans = X 2 Y [ 3/4, 0, 1/4] . [ 5/16, 7/16, 1/4] 11 1 0.3 3 PE_sim = [DMC_demo.m] 0.7500 Ex: DMC
>> p = [0.2 0.8] p = 0.2000 0.8000 >> p = [0.2 0.8]; >> Q = [0.75 0 0.25; 0.3125 0.4375 0.25]; >> p*Q ans = 0.4000 0.3500 0.2500
12 Block Matrix Multiplications
225102 102 5 334510536C D 106643 33411556 9A 7359B 7253106103F 83698365E 108 73 136 175 150 193 126 149 155 85 164 224 213 197 158 165 AC+BE AD+BF 225102 102 5 334510536 106643 33411556 9 7359X GH 7253106103 83698365 108 73 136 175 150 193 126 149 155 85 164 224 213 197 158 165 13 XG XH Review: Evaluation of Probability from the Joint PMF Matrix Consider two random variables X and Y. Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3 Find
Step 1: Find the pairs (x,y) that y satisfy the condition x 2 3 4 5 6 “x+y < 7” 1 3 4 5 6 7 xy 3 5 6 7 8 9 One way to do this is to first 4 6 7 8 9 10 construct the matrix of x+y. 6 8 9 10 11 12 14 Review: Evaluation of Probability from the Joint PMF Matrix Consider two random variables X and Y. Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3 Find
Step 2: Add the corresponding y probabilities from the x 2 3 4 5 6 joint pmf (matrix) 1 3 4 5 6 7 xy 3 5 6 7 8 9 7 0.1 0.1 0.1 4 6 7 8 9 10 0.3 6 8 9 10 11 12
15 Review: Evaluation of Probability from the Joint PMF Matrix Consider two random variables X and Y. Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3 Find
16 Review: Sum of two discrete RVs Consider two random variables X and Y. Suppose their joint pmf matrix is y x 2 3 4 5 6 1 0.1 0.1 0 0 0 PXY, 3 0.1 0 0 0.1 0 4 0 0.1 0.2 0 0 6 0 0 0 0 0.3 Find
y x 2 3 4 5 6 7 0.1 1 3 4 5 6 7 xy 3 5 6 7 8 9 4 6 7 8 9 10 6 8 9 10 11 12
17 Ex: DMC
>> p = [0.2 0.8]; >> Q = [0.5 0.2 0.3; 0.3 0.4 0.3]; >> p*Q ans = 0.3400 0.3600 0.3000 >> P = (diag(p))*Q P = 0.1000 0.0400 0.0600 0.2400 0.3200 0.2400 >> sum(P) ans = 0.3400 0.3600 0.3000
18 Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong [email protected] 3.2 Decoder and
19 for Naïve Decoder
%% Naive Decoder x_hat = y;
%% Error Probability PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
% Calculation of the theoretical error probability PC = 0; for k = 1:length(S_X) t = S_X(k); i = find(S_Y == t); if length(i) == 1 PC = PC+ p_X(k)*Q(k,i); Formula derived in 3.19 of lecture notes end end PE_theretical = 1-PC
20 [DMC_Analysis_demo.m] Ex: Naïve Decoder and BAC [Ex. 3.18] %% Simulation parameters p_X = % The number of symbols to be transmitted n = 20; 0.5000 0.5000 % Binary Assymmetric Channel (BAC) p_X_sim = % Ex 3.8 in lecture note (11.3 in [Z&T, 2010]) % Channel Input 0.7000 0.3000 S_X = [0 1]; S_Y = [0 1]; q = p_X = [0.5 0.5]; % Channel Characteristics 0.5500 0.4500 Q = [0.7 0.3; 0.4 0.6]; q_sim = 0.6500 0.3500 >> BAC_demo Q = ans = 0.7000 0.3000 x: 0 0 0 1 1 0 0 1 0 0 0 010 01 0 100 0.4000 0.6000 ans = Q_sim = y: 0 0 1 1 0 0 0 1 1 1 0 0 1 0 0 0 0 0 1 0 0.7143 0.2857 0.7 0.5000 0.5000 . 0 0 7 PE_sim = 0.3500 X 0.3 Y 20 0.4 PE_theretical = . 0.3500 21 1 0.6 1 [BAC_demo.m] Ex: Naïve Decoder and BAC [Ex. 3.18] %% Simulation parameters p_X = % The number of symbols to be transmitted n = 1e4; 0.5000 0.5000 % Binary Assymmetric Channel (BAC) p_X_sim = % Ex 3.8 in lecture note (11.3 in [Z&T, 2010]) % Channel Input 0.5043 0.4957 S_X = [0 1]; S_Y = [0 1]; q = p_X = [0.5 0.5]; % Channel Characteristics 0.5500 0.4500 Q = [0.7 0.3; 0.4 0.6]; q_sim =
10 10 0.5532 0.4468
20 20
30 30 Q = 40 40 0.7000 0.3000 50 50 60 60 0.4000 0.6000 70 70 80 80 Q_sim = 90 90
100 100 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 0.7109 0.2891 0.7 0.3928 0.6072 . 0 0 PE_sim = 0.3405 X 0.3 Y 0.4 PE_theretical = . 0.3500 22 1 0.6 1 [BAC_demo.m] Ex: Naïve Decoder and DMC [Ex. 3.21] %% Simulation parameters p_X = % The number of symbols to be transmitted n = 20; 0.2000 0.8000 % General DMC p_X_sim = % Ex. 3.16 in lecture note % Channel Input 0.2000 0.8000 S_X = [0 1]; S_Y = [1 2 3]; q = p_X = [0.2 0.8]; % Channel Characteristics 0.3400 0.3600 0.3000 Q = [0.5 0.2 0.3; 0.3 0.4 0.3]; q_sim = 0.4000 0.3500 0.2500 >> DMC_demo [Same samples as in Ex. 3.6] Q = ans = 0.5000 0.2000 0.3000 x: 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0.3000 0.4000 0.3000 ans = Q_sim = y: 1 3 2 2 1 2 1 2 2 3 1 1 1 3 1 3 2 3 1 2 0.7500 0 0.2500 0.3125 0.4375 0.2500 0.5 . 0 1 20 4 PE_sim = 20 0.7500 X 2 Y PE_theretical = . 0.7600 23 1 0.3 3 [DMC_demo.m] Ex: Naïve Decoder and DMC [Ex. 3.21] %% Simulation parameters p_X = % The number of symbols to be transmitted n = 1e4; 0.2000 0.8000 % General DMC p_X_sim = % Ex. 3.16 in lecture note % Channel Input 0.2011 0.7989 S_X = [0 1]; S_Y = [1 2 3]; q = p_X = [0.2 0.8]; % Channel Characteristics 0.3400 0.3600 0.3000 Q = [0.5 0.2 0.3; 0.3 0.4 0.3]; q_sim =
10 10 0.3387 0.3607 0.3006 20 20 Q = 30 30 40 40 0.5000 0.2000 0.3000 50 50 60 60 0.3000 0.4000 0.3000
70 70
80 80 Q_sim =
90 90
100 100 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 0.4943 0.1914 0.3143 0.2995 0.4033 0.2972 . 0 0.5 1 PE_sim = 0.7607 X 2 Y PE_theretical = . 0.7600 24 1 0.3 3 [DMC_demo.m] DIY Decoder [Ex. 3.22]
%% Simulation parameters % The number of symbols to be transmitted >> DMC_decoder_DIY_demo n = 20; ans = % General DMC X 1 0 1 1 1 1 1 0 1 1 0 1 1 1 1 0 0 1 0 1 % Ex. 3.16 in lecture note % Channel Input ans = S_X = [0 1]; S_Y = [1 2 3]; Y 2 1 1 3 3 1 2 2 1 2 1 2 3 1 1 3 1 3 1 1 p_X = [0.2 0.8]; ans = % Channel Characteristics Q = [0.5 0.2 0.3; 0.3 0.4 0.3]; 1 0 0 0 0 0 1 1 0 1 0 1 0 0 0 0 0 0 0 0 PE_sim = 0.5500 PE_theretical = %% DIY Decoder 0.5200 Decoder_Table = [0 1 0]; % The decoded Elapsed time is 0.081161 seconds. values corresponding to the received Y
25 [DMC_decoder_DIY_demo.m] DIY Decoder [Ex. 3.22] %% DIY Decoder Decoder_Table = [0 1 0]; % The decoded values corresponding to the received Y
% Decode according to the decoder table x_hat = y; % preallocation for k = 1:length(S_Y) I = (y==S_Y(k)); x_hat(I) = Decoder_Table(k); end
PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
% Calculation of the theoretical error probability PC = 0; for k = 1:length(S_X) I = (Decoder_Table == S_X(k)); q = Q(k,:); PC = PC+ p_X(k)*sum(q(I)); end PE_theretical = 1-PC
26 [DMC_decoder_DIY_demo.m] DIY Decoder [Ex. 3.22] %% Simulation parameters % The number of symbols to be transmitted n = 1e4; % General DMC >> DMC_decoder_DIY_demo % Ex. 3.16 in lecture note PE_sim = % Channel Input 0.5213 S_X = [0 1]; S_Y = [1 2 3]; PE_theretical = p_X = [0.2 0.8]; % Channel Characteristics 0.5200 Q = [0.5 0.2 0.3; 0.3 0.4 0.3]; Elapsed time is 2.154024 seconds. %% DIY Decoder Decoder_Table = [0 1 0]; % The decoded values corresponding to the received Y
10 10 10
20 20 20
30 30 30
40 40 40
50 50 50
60 60 60
70 70 70
80 80 80
90 90 90
100 100 100 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 27 [DMC_decoder_DIY_demo.m] Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong [email protected] 3.3 Optimal Decoder
28 Searching for the Optimal Detector
>> DMC_decoder_ALL_demo ans = 1 2 3 0 0 0 0.8000 0 0 1.0000 0.6200 0 1.0000 0 0.5200 Ex. 3.22 0 1.0000 1.0000 0.3400 1.0000 0 0 0.6600 1.0000 0 1.0000 0.4800 1.0000 1.0000 0 0.3800 Ex. 3.23 1.0000 1.0000 1.0000 0.2000 Min_PE = 0.2000 Optimal_Detector = 1 1 1 Elapsed time is 0.003351 seconds.
29 Review: ECS315 (2016)
30 Guessing Game 1 There are 15 cards. Each have a number on it. Here are the 15 cards: 1 2 2 3 3 3 4 4 4 4 5 5 5 5 5 One card is randomly selected from the 15 cards. You need to guess the number on the card. Have to pay 1 Baht for incorrect guess. The game is to be repeated n = 10,000 times. What should be your guess value?
31 close all; clear all;
n = 5; % number of time to play this game
D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5]; X = D(randi(length(D),1,n));
if n <= 10 X end
g = 1 cost = sum(X ~= g) >> GuessingGame_4_1_1 X = if n > 1 3 5 1 2 5 averageCostPerGame = cost/n g = end 1 cost = 4 averageCostPerGame = 0.8000 32 close all; clear all;
n = 5; % number of time to play this game
D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5]; X = D(randi(length(D),1,n));
if n <= 10 X end
g = 3.3 cost = sum(X ~= g) >> GuessingGame_4_1_1 X = if n > 1 5 3 2 4 1 averageCostPerGame = cost/n g = end 3.3000 cost = 5 averageCostPerGame = 1 33 close all; clear all;
n = 1e4; % number of time to play this game
D = [1 2 2 3 3 3 4 4 4 4 5 5 5 5 5]; X = D(randi(length(D),1,n));
if n <= 10 X end
g = ? cost = sum(X ~= g)
if n > 1 averageCostPerGame = cost/n end
34 Guessing Game 1
1
0.95
0.9
0.85
0.8
AVerage Cost Per Game 0.75
0.7
0.65 1 1.5 2 2.5 3 3.5 4 4.5 5 35 Guess value Guessing Game 1
1
0.95
0.9 Optimal Guess:
0.85 The most-likely value
0.8
AVerage Cost Per Game 0.75
0.7
0.65 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 Guess value 36 MAP Decoder %% MAP Decoder P = diag(p_X)*Q; % Weight the channel transition probability by the % corresponding prior probability. [V I] = max(P); % For I, the default MATLAB behavior is that when there are % multiple max, the index of the first one is returned. Decoder_Table = S_X(I) % The decoded values corresponding to the received Y
%% Decode according to the decoder table x_hat = y; % preallocation for k = 1:length(S_Y) I = (y==S_Y(k)); x_hat(I) = Decoder_Table(k); end
PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
%% Calculation of the theoretical error probability PC = 0; for k = 1:length(S_X) I = (Decoder_Table == S_X(k)); Q_row = Q(k,:); PC = PC+ p_X(k)*sum(Q_row(I)); end PE_theretical = 1-PC
37 [DMC_decoder_MAP_demo.m] ML Decoder %% ML Decoder [V I] = max(Q); % For I, the default MATLAB behavior is that when there are % multiple max, the index of the first one is returned. Decoder_Table = S_X(I) % The decoded values corresponding to the received Y
%% Decode according to the decoder table x_hat = y; % preallocation for k = 1:length(S_Y) I = (y==S_Y(k)); x_hat(I) = Decoder_Table(k); end
PE_sim = 1-sum(x==x_hat)/n % Error probability from the simulation
%% Calculation of the theoretical error probability PC = 0; for k = 1:length(S_X) I = (Decoder_Table == S_X(k)); Q_row = Q(k,:); PC = PC+ p_X(k)*sum(Q_row(I)); end PE_theretical = 1-PC
38 [DMC_decoder_ML_demo.m] Digital Communication Systems ECS 452
Asst. Prof. Dr. Prapun Suksompong [email protected] 4. Mutual Information and Channel Capacity Office Hours: BKD, 6th floor of Sirindhralai building Tuesday 14:20-15:20 Wednesday 14:20-15:20
1 Friday 9:15-10:15 Reference for this chapter Elements of Information Theory By Thomas M. Cover and Joy A. Thomas 2nd Edition (Wiley) Chapters 2, 7, and 8 1st Edition available at SIIT library: Q360 C68 1991
2 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] Operational Channel Capacity
3 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] Information Channel Capacity
4 Channel Capacity
“Operational”: max rate at which reliable communication is possible
Channel Capacity Arbitrarily small error probability can be achieved.
“Information”: [bpcu] ܘഫ
Shannon [1948] shows that these two quantities are actually the same.
5 MATLAB
function H = entropy2s(p) % ENTROPY2 accepts probability mass function % as a row vector, calculate the corresponding % entropy in bits. p=p(find(abs(sort(p)-1)>1e-8)); % Eliminate 1 p=p(find(abs(p)>1e-8)); % Eliminate 0 if length(p)==0 H = 0; else H = simplify(-sum(p.*log(p))/log(sym(2))); end
function I = informations(p,Q) X = length(p); q = p*Q; HY = entropy2s(q); temp = []; for i = 1:X temp = [temp entropy2s(Q(i,:))]; end HYgX = sum(p.*temp); I = HY-HYgX; 6 Capacity calculation for BAC
0.1 0.1 ൌ 0 0 0.09 0.9 0.08 X 0.4 Y 0.07 ൌെ 0.06 1 1 0.05
0.6 I(X;Y) 0.04 0.1 0.9 0.03 Q 0.02 0.4 0.6 0.01 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p 0
Capacity of 0.0918 bits is achieved by p 0.5380, 0.4620
7 Capacity calculation for BAC
0 0.1 0 0.9 0.1 0.9 XYQ 0.4 0.4 0.6 1 0.6 1 close all; clear all; >> Capacity_Ex_BAC syms p0 I = p = [p0 1-p0]; (log(2/5 - (3*p0)/10)*((3*p0)/10 - 2/5) - log((3*p0)/10 + 3/5)*((3*p0)/10 + Q = [1 9; 4 6]/sym(10); 3/5))/log(2) + (log((5*2^(3/5)*3^(2/5))/6)*(p0 - 1))/log(2) + (p0*log((3*3^(4/5))/10))/log(2) I = simplify(informations(p,Q)) p0o = (27648*2^(1/3))/109565 - (69984*2^(2/3))/109565 + 135164/109565 p0o = simplify(solve(diff(I)==0)) po = 0.5376 0.4624 C = po = eval([p0o 1-p0o]) (log((3*3^(4/5))/10)*((27648*2^(1/3))/109565 - (69984*2^(2/3))/109565 + 135164/109565))/log(2) - (log((104976*2^(2/3))/547825 - (41472*2^(1/3))/547825 + 16384/547825)*((104976*2^(2/3))/547825 - (41472*2^(1/3))/547825 + C = simplify(subs(I,p0,p0o)) 16384/547825) + log((41472*2^(1/3))/547825 - (104976*2^(2/3))/547825 + 531441/547825)*((41472*2^(1/3))/547825 - (104976*2^(2/3))/547825 + 531441/547825))/log(2) + (log((5*2^(3/5)*3^(2/5))/6)*((27648*2^(1/3))/109565 - eval(C) (69984*2^(2/3))/109565 + 25599/109565))/log(2) ans = 8 0.0918 Same procedure applied to BSC
0 0.6 0 0.4 0.6 0.4 XY0.4 Q 0.4 0.6 1 0.6 1 close all; clear all; >> Capacity_Ex_BSC syms p0 I = p = [p0 1-p0]; (log((5*2^(3/5)*3^(2/5))/6)*(p0 - 1))/log(2) - Q = [6 4; 4 6]/sym(10); (p0*log((5*2^(3/5)*3^(2/5))/6))/log(2) - (log(p0/5 + 2/5)*(p0/5 + 2/5) - log(3/5 - p0/5)*(p0/5 - I = simplify(informations(p,Q)) 3/5))/log(2) p0o = p0o = simplify(solve(diff(I)==0)) 1/2 po = po = eval([p0o 1-p0o]) 0.5000 0.5000 C = C = simplify(subs(I,p0,p0o)) log((2*2^(2/5)*3^(3/5))/5)/log(2) ans = eval(C) 0.0290
9 Blahut–Arimoto algorithm
function [ps C] = capacity_blahut(Q) % Input: Q = channel transition probability matrix % Output: C = channel capacity % ps = row vector containing pmf that achieves capacity
tl = 1e-8; % tolerance (for the stopping condition) n = 1000; % max number of iterations (in case the stopping condition % is "never" reached") nx = size(Q,1); pT = ones(1,nx)/nx; % First, guess uniform X. for k = 1:n qT = pT*Q; % Eliminate the case with 0 % Column-division by qT temp = Q.*(ones(nx,1)*(1./qT)); %Eliminate the case of 0/0 l2 = log2(temp); l2(find(isnan(l2) | (l2==-inf) | (l2==inf)))=0; logc = (sum(Q.*(l2),2))'; CT = 2.^(logc); A = log2(sum(pT.*CT)); B = log2(max(CT)); if((B-A) 0 0.1 0 0.9 0.1 0.9 XYQ 0.4 0.4 0.6 1 0.6 1 close all; clear all; >> Capacity_Ex_BAC_blahut ps = Q = [1 9; 4 6]/10; 0.5376 0.4624 C = [ps C] = capacity_blahut(Q) 0.0918 11 Berger plaque 12 Richard Blahut Former chair of the Electrical and Computer Engineering Department at the University of Illinois at Urbana-Champaign Best known for Blahut–Arimoto algorithm (Iterative Calculation of C) 13 Raymond Yeung BS, MEng and PhD degrees in electrical engineering from Cornell University in 1984, 1985, and 1988, respectively. 14