Hamming Distance, We Can Work with the Distances Instead of the % Conditional Probability

Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 4. Mutual Information and Channel Capacity Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] Operational Channel Capacity

14 Channel Capacity

[Section 4.2] “Operational”: max rate at which reliable communication is possible

Channel Capacity Arbitrarily small error probability can be achieved.

“Information”: [bpcu] 𝐩 [Section 4.3]

Shannon [1948] shows that these two quantities are actually the same.

15 System Model for Section 3.4

Message Transmitter

Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add X: channelSignal input redundancy systematic redundancy EquivalentChannel Channel Recovered Decoded Message value Received Y: channelSignal output Source Channel Digital Destination Decoder Decoder (Detector)Demodulator

Receiver In Chapter 3, we studied how to find the optimal decoder. Some results from Section 3.3-3.4

17 Under appropriate assumptions, minimum distance decoder is optimal. System Model for Section 3.5

Message Transmitter

Information Source Channel Source Encoder Encoder

Remove Add systematic X: channel input redundancy redundancy Equivalent Channel Recovered Decoded Message value Y: channel output Source Channel Destination Decoder Decoder (Detector)

Receiver We then introduced the channel encoder box. [3.62] Block Encoding

Channel X Encoder k bits k bits k bits n bits n bits n bits 𝒏, 𝒌 code 𝒌 Code rate = 𝒏 [3.62] Block Encoding

Channel X Encoder k bits k bits k bits n bits n bits n bits 𝒏, 𝒌 code 𝒌 Code rate = 𝒏 Example: Repetition Code

Channel X 1 0 1 111110000011111 Encoder 1bit 1bit 1bit 5 bits 5 bits 5 bits 𝟓, 𝟏 code 𝟏 Code rate = 𝟓 [3.62] Block Encoding

Channel X Encoder k bits n bits

𝑀2 possibilities 𝐱 Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be [Figure 13] 𝐬 𝐱 used as codewords. 𝐬 𝐱

Codebook Repetition Code

22 Review: Channel Encoder and Decoder 𝑀2 possibilities 𝐱 Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be 𝐬 𝐱 used as codewords. 𝐬 Message (Data block) 𝐱 Channel Digital Encoder Modulator Transmitted k bits k bits k bits Signal Add n bits n bits n bits systematic redundancy 0 1-p 0 p Channel Binary Symmetric p 1 1-p 1 Channel with p < 0.5 Received Noise & Interference Signal Channel Digital Recovered Message Decoder Demodulator

23 minimum distance decoder Example: Repetition Code [Figure 14]  Original Equivalent Channel: 0 1-p 0 p p 1 1-p 1  BSC with crossover probability p = 0.01  New (and Better) Equivalent Channel:

1-p Repetition 0 0 Majority 0 0 Code with p p Vote 1 1 n = 5 1 1-p 1

 Use repetition code with n = 5 at the transmitter  Use majority vote at the receiver 5 5 5 𝑝 1𝑝 𝑝 1𝑝 𝑝 1𝑝  New BSC with 𝑝 3 4 5 10 24 [From ECS315]

25 MATLAB

close all; clear all;

% ECS315 Example 6.58 % ECS452 Example 3.66 C = [0 0 0 0 0; 1 1 1 1 1]; % repetition code

p = (1/100); PE_minDist(C,p)

Code C is defined by putting all its (valid) codewords as its rows. For repetition >> PE_minDist_demo1 code, there are two codewords: 00..0 and 11..1. ans = 9.8506e-06 Crossover probability of the binary symmetric channel. 26 function PE = PE_minDist(C,p) MATLAB % Function PE_minDist_3 computes the error probability P(E) when code C % is used for transmission over BSC with crossover probability p. % Code C is defined by putting all its (valid) codewords as its rows. M = size(C,1); k = log2(M); n = size(C,2);

% Generate all possible received vectors Y = dec2bin(0:2^n-1)-'0';

% Normally, we need to construct an extended Q matrix. However, because % each conditional probability in there is a decreasing function of the % Hamming distance, we can work with the distances instead of the % conditional probability. In particular, instead of selecting the max in % each column of the Q matrix, we consider min distance in each column. dmin = zeros(1,2^n); for j = 1:(2^n) % for each received vector y, y = Y(j,:); % find the minimum distance (the distance from y to the closest % codeword) d = sum(mod(bsxfun(@plus,y,C),2),2); dmin(j) = min(d); end

% From the distances, calculate the conditional probabilities. % Note that we compute only the values that are to be selected (instead of % calculating the whole Q first). n1 = dmin; n0 = n-dmin; Qmax = (p.^n1).*((1-p).^n0); % Scale the conditional probabilities by the input probabilities and add % the values. Note that we assume equally likely input. PC = sum((1/M)*Qmax); PE = 1-PC; end 27 MATLAB

28 Example: Repetition Code  Original Equivalent Channel: 0 1-p 0 p p 1 1-p 1  BSC with crossover probability p  New (and Better) Equivalent Channel:

1-p Repetition 0 0 Majority 0 0 Code with p p Vote 1 1 n = 5 1 1-p 1

 Use repetition code at the transmitter  Use majority vote at the receiver  New BSC with new crossover probability 29 MATLAB close all; clear all;

0.5

% ECS315 Example 6.58 0.45 % ECS452 Example 3.66 0.4 C = [0 0 0 0 0; 1 1 1 1 1]; 0.35 0.3

0.25 syms p; P(E) PE = PE_minDist(C,p) 0.2 pp = linspace(0,0.5,100); 0.15 0.1 PE = subs(PE,p,pp); 0.05

plot(pp,PE,'LineWidth',1.5) 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 xlabel('p') p ylabel('P(E)') grid on >> PE_minDist_demo2

PE = (p - 1)^5 + 10*p^2*(p - 1)^3 - 5*p*(p - 1)^4 + 1 30 Searching for the best encoder  Now that we have MATLAB function PE_minDist, for specific values of n, k, we can try to search for the encoder that minimizes the error probability.  Recall that, from Example 3.64, there are reasonable encoders.  Even for small n and k, this is a large space to look at every possible cases.

31 Example: Repetition Code

0 1-p 0 0 0 Repetition p Majority p Vote Code 1 1 1 1-p 1 𝑝0.1

𝒏 1 𝑝0.1 3 3 3 𝑝 1𝑝 𝑝 0.0280 2 3 5 5 5 5 𝑝 1𝑝 𝑝 1𝑝 𝑝 0.0086 3 4 5 7 0.0027 9 8.9092 10 11 2.9571 10

32 Channel Capacity

[Section 4.2] “Operational”: max rate at which reliable communication is possible

Channel Capacity Arbitrarily small error probability can be achieved.

“Information”: [bpcu] 𝐩 [Section 4.3]

Shannon [1948] shows that these two quantities are actually the same.

33 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 5. Channel Coding

Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD Review: Channel Encoder and Decoder

Message Transmitter S Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Add X: channelSignal input systematic redundancy 0 1-p 0 p Channel p 1 1-p 1 Noise & Interference Recovered Message Receiver Received Y: channelSignal output 𝐒 Source Channel Digital Destination Decoder Decoder Demodulator

2 System Model for Chapter 5

Transmitter s Channel Digital Message Encoder Modulator Transmitted Add x: channelSignal input systematic redundancy 0 1-p 0 p Channel p 1 1-p 1

Received Noise & Interference Receiver y: channel output 𝐬 Signal Channel Digital Recovered Message Decoder Demodulator

3 𝑣 Vector Notation 𝑣 0,0: the zero vector ⋮ (the all-zero vector)  : column vector 𝑣 ⋮ 1,1: the one vector 𝑣 (the all-one vector)  : row vector 𝑟,𝑟,…,𝑟,…𝑟  Subscripts represent element indices inside individual vectors. th  and refer to the i elements inside the vectors and , respectively.  When we have a list of vectors, we use superscripts in parentheses as indices of vectors.  is a list of M column vectors  is a list of M row vectors  and refer to the ith vectors in the corresponding lists. 4 Harpoon  a long, heavy spear attached to a rope, used for killing large fish or whales

5 Review: Channel Decoding  Recall 1. The MAP decoder is the optimal decoder. 2. When the codewords are equally-likely, the ML decoder the same as the MAP decoder; hence it is also optimal. 3. When the crossover probability of the BSC p is < 0.5, ML decoder is the same as the minimum distance decoder.  In this chapter, we assume the use of minimum distance decoder.  𝐱 𝐲 arg min 𝑑 𝐱,𝐲 𝐱  Also, in this chapter, we will focus  less on probabilistic analysis,  but more on explicit codes.

6 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 5.1 Binary Linear Block Codes

Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 7 BKD Review: Block Encoding  We mentioned the general form of channel coding over BSC.  In particular, we looked at the general form of block codes. Block Encoder

k bits k bits k bits n bits n bits n bits Code length “Dimension” of the code  (n,k) codes: n-bit blocks are used to conveys k-info-bit blocks  Assume n > k codewords “messages” Max. achievable rate  Rate: . Recall that the capacity of BSC is 𝐶 1𝐻 𝑝 . For 𝑝∈ 0,1 , we also have 𝐶∈ 0,1 . Achievable rate is < 1. 8 System Model for Section 5.1

Transmitter s Channel Digital Message Encoder Modulator n bits Transmitted k bits Add x: channelSignal input systematic redundancy 0 1-p 0 p Channel p 1 1-p 1

Received Noise & Interference Receiver y: channel output 𝐬 Signal Recovered Message Channel Digital Decoder Demodulator k bits n bits

9    = the collection of all codewords for the code considered  Each n-bit block is selected from .  The message (data block) has k bits, so there are 2k possibilities.  A reasonable code would not assign the same codeword to different messages.  Therefore, there are 2k (distinct) codewords in .  Ex. Repetition code with n = 3

10 MATHEMATICAL SCRIPT CAPITAL C

11 [ http://www.charbase.com/1d49e-unicode-mathematical-script-capital-c ] GF(2)  The construction of the codes can be expressed in matrix form using the following definition of addition and multiplication of bits:  01 01 0 01 000 110 101  These are modulo-2 addition and modulo-2 multiplication, respectively.  The operations are the same as the exclusive-or (XOR) operation and the AND operation.  We will simply call them addition and multiplication so that we can use a matrix formalism to define the code.  The two-element set {0, 1} together with this definition of addition and multiplication is a number system called a finite field or a Galois field, and is denoted by the label GF(2). 12 Modulo operation  The modulo operation finds the remainder after division of one number by another (sometimes called modulus).  Given two positive numbers, (the dividend) and (the divisor),  (abbreviated as ) is the remainder of the division of by . quotient 13  “ ” divisor 6 83 dividend  “ ” 6  In MATLAB, mod(5,2) = 1. 23  Congruence relation quotient 2 18 divisor 2 5 dividend  5 ≡ 1 mod 2 5 remainder 4 13 1 remainder GF(2) and modulo operation  Normal addition and multiplication (for 0 and 1):  01 01 001 000 112 101

 Addition and multiplication in GF(2):  01 01 0 01 000 110 101

14 GF(2)  The construction of the codes can be expressed in matrix form using the following definition of addition and multiplication of bits:  01 01 0 01 000 110 101  Note that x  0  x x 1  x xx  0 The above property implies x  x By definition, “-x” is something that, when added with x, gives 0.  Extension: For vector and matrix, apply the operations to the elements the same way that addition and multiplication would normally apply 15 (except that the calculations are all in GF(2)). Examples  Normal vector addition:

1121 2 3 0 1 1 2 2 2

 Vector addition in GF(2): Alternatively, one can also apply normal vector addition first, then apply “mod 2” to each element:

1011⊕ 0101

1011⊕ 0101 1110 1112 1110

16 Examples  Normal matrix multiplication: 7 2 4 3 + 3 7 1412 21

7432 4 23 14 256 38 31 4 1897 6 41 6  Matrix multiplication in GF(2):

1 · 1  0 · 0  1 · 1 101 Alternatively, one can also apply normal matrix multiplication first, then apply “mod 2” to each element: 10111 01 10111 21 01 00101 10 00101 10 10 11110 00 11110 22 00 17 Galois: Introduction

18 [ https://www.youtube.com/watch?v=Mc0bvea6G3I ] Galois: Introduction

On the morning of May 30th, 1832, two men in Paris fought a duel. Not an unusual event for those days. One of the men was shot in the gut and died the following day... 19 Galois: Introduction

Do not cry Alfred I need all my courage to die at 20.

20 Galois: Introduction

The night before the duel, Galois sent several letters. Some were to his political colleagues but one of his letters in particular has become famous amongst mathematicians. Fearing that he might die, Galois assembled his mathematical discoveries and sent them to his friend with instructions to pass him along to two of 21 the best mathematicians of the day: Gauss and Jacobi. Galois: Introduction

The papers laid dormant until over a decade later when the letter made its way to the mathematician Liouville who took the time to read through the manuscripts and sought to their publication.

The world finally learned that as a teenager Galois had solved one of the most important problems in algebra.

22 Galois: Contribution

23 [ https://www.youtube.com/watch?v=Mc0bvea6G3I ] Galois: Contribution

In algebra, you learn to solve equations. To solve quadratic equations you use a quadratic formula. To solve cubic equations, you use the less well-known cubic formula and to solve equations of degree four, you use the quartic formula... Galois proved that for degrees five and higher, there are no general formulas.

To prove this Galois created new mathematics which we now call Galois theory in his honor. 24 Galois: Life

25 [ https://www.youtube.com/watch?v=Mc0bvea6G3I ] Galois: Life

Galois’ tale was tinged with frustration, trouble, and tragedy.

26 Galois: Life

27 [ https://www.youtube.com/watch?v=Mc0bvea6G3I ] Galois: Life

Unfortunately, Galois was not very good at patiently explaining his ideas to others. He entered math contests and sent his work to leading mathematicians but his writing was considered incomprehensible.

28 Galois: Life

troubled genius, indeed!

29 BSC and the Error Pattern  For one use of the channel,

x BSC y

 Again, to transmit k information bits, the channel is used n times.

b Encoder x BSC y 1 k 1 n y  xe

error pattern Its nonzero elements mark the 30 positions of transmission error in y Additional Properties in GF(2)  The following statements are equivalent 1.  Having one of these is the same 2.  as having all three of them. 3.   The following statements are equivalent 1.  2.  Having one of these is the same 3.  as having all three of them.  In particular, because  , if we are given two quantities, we can find the third quantity by summing the other two.

2 Linear Block Codes  Definition:  is a (binary) linear (block) code if and only if  forms a vector (sub)space (over GF(2)). In case you forgot about the concept of vector space,…  Equivalently, this is the same as requiring that f and  then  

 Note that any (non-empty) linear code  must contain 0.

 Ex. The code that we considered in Problem 5 of HW3 is  00000,01000,10001,11111 Is it a linear code?

31 Linear Block Codes: Motivation (1)  Why linear block codes are popular?  Recall: General block encoding  Characterized by its codebook.

𝑀2 possibilities 𝐱 Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be 𝐬 𝐱 used as codewords. 𝐬 𝐱 [See p. 51 in Ch. notes.] 3 of the lecture Ch. 51 in [See p.

 Can be realized by combinational/combinatorial circuit.  32 If lucky, can used K-map to simplify the circuit. Linear Block Codes: Motivation (2)  Why linear block codes are popular?  Linear block encoding is the same as matrix multiplication.  See next slide.  The matrix replaces the table for the codebook.  The size of the matrix is only bits.  Compare this against the table (codebook) of size 2 𝑘𝑛 bits for general block encoding.  Linearity  easier implementation and analysis  Performance of the class of linear block codes is similar to performance of the general class of block codes.  Can limit our study to the subclass of linear block codes without sacrificing system performance. 33 Linear Block Codes: Generator Matrix

𝐠 𝐠 For any linear code, there is a matrix = ⋮ 𝐠 called the generator matrix such that, for any codeword , there is a message vector which produces by mod-2 summation Note: (1) Any codeword can be expressed as a linear combination of the rows of G Note also that, given a matrix 𝐆, the (block) (2) 34 code that is constructed by (2) is always linear. Block Matrices  A block matrix or a partitioned matrix is a matrix that is interpreted as having been broken into sections called blocks or submatrices.  Examples:

225102 102 5 334510536C D 106643 334 1 1 5 5 6 9A 7359B 7253106103F 836E 9 8 3 6 5

4 Block Matrix Multiplications  Matrix multiplication can also be carried out blockwise (assuming that the block sizes are compatible).

[ https://youtu.be/FX4C-JpTFgY?t=1103 ] 5 Ex: Block Matrix Multiplications

225102 102 5 334510536C D 106643 334 1 1 5 5 6 9A 7359B 7253106103F 836E 9 8 3 6 5 108 73 136 175 150 193 126 149 155 85 164 224 213 197 158 165 AC+BE AD+BF 225102 102 5 334510536 106643 334 1 1 5 5 6 9 7359X GH 7253106103 836 9 8 3 6 5 108 73 136 175 150 193 126 149 155 85 164 224 213 197 158 165 6 XG XH From to  Any codeword is simply a linear combination of the rows of G.  The weights are given by the bits in the message

7 Linear Combination in GF(2)  A linear combination is an expression constructed from a set of terms by multiplying each term by a constant (weight) and adding the results.  For example, a linear combination of x and y would be any expression of the form ax + by, where a and b are constants.  General expression:  In GF(2), is limited to being 0 or 1. So, a linear combination is simply a sum of a sub-collection of the vectors.

3 Example

100101  G  010011  001110

 Find the codeword for the message b = [1 0 0]

 Find the codeword for the message b = [0 1 1]

35 Example 1110000 1001100 G    0010110   1010101  Find the codeword for the message b = [1 0 0 0]

 Find the codeword for the message b = [0 1 1 0]

36 Linear Block Codes: Examples

 Repetition code: 𝑏 𝐱  0000 1111  

 Single-parity-check code:  parity bit  𝐛 𝐱 00000 01011 10101 37 11110 Fact: If a code is generated by plugging in every possible b into xbG , then the code will automatically be linear.

Proof

k If G has k rows. Then, b will have k bits. We can list them all as bb12,,, b 2. The corresponding codewords are

xbGii  for i 1, 2, , 2 k .

i i ii ii Let’s take two codewords, say, x1 and x2 . By construction, xbG11  and xbG22  . Now, consider the sum of these two codewords:

iii   i ii xxbGbGbbG121  2  12 

ii Note that because we plug in every possible b to create this code, we know that bb12  should be one of these b . Let’s suppose bbbiii123  for some bi3 . This means

iii xxbG123 .

i i But, again, by construction, bG3 gives a codeword x3 in this code. Because the sum of any two codewords is still a codeword, we conclude that the code is linear.

9 [Exercise Solution] Vectors representing 3-bit codewords

Representing the codewords in the two examples on the previous slide as vectors:

Triple-repetition code Parity-check code

38 Related Idea: Even Parity vs. Odd Parity  Parity bit checking is used occasionally for transmitting ASCII characters, which have 7 bits, leaving the 8th bit as a parity bit.  Two options:  Even Parity: Added bit ensures an even number of 1s in each codeword.  A: 10000010  Odd Parity: Added bit ensures an odd number of 1s in each codeword.  A: 10000011

39 Even Parity vs. Odd Parity  Even parity and odd parity are properties of a codeword (a vector), not a bit.  Note: The generator matrix 𝐆 𝐈;𝟏 previously considered produces even parity codeword

𝐱 𝐛 ;𝑏

 Q: Consider a code that uses odd parity. Is it linear?

40 Error Control using Parity Bit  If an odd number of bits (including the parity bit) are transmitted incorrectly, the parity will be incorrect, thus indicating that a parity error occurred in the transmission.  Ex.  Suppose we use even parity.  Consider the codeword 10000010

 Suitable for detecting errors; cannot correct any errors 41 US UK The ASCII Coded Character Set (American Standard Code for Information Interchange)

016 32 48 64 80 96 112

42 [The ARRL Handbook for Radio Communications 2013] Two types of error control: 1. error detection Error Detection 2. error correction  Error detection: the determination of whether errors are present in a

received word 𝑀2 possibilities  𝐱 usually by checking whether Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be the received word is one of the 𝐬 𝐱 used as codewords. valid codewords. 𝐬 𝐱  When a two-way channel exists between source and destination, the receiver can request retransmission of information containing detected errors.  This error-control strategy is called automatic-repeat-request (ARQ).  An error pattern is undetectable if and only if it causes the received word to be a valid codeword other than that which was transmitted.  Ex: In single-parity-check code, error will be undetectable when the number of bits in error is even. 43 Error Correction  In FEC (forward error correction) system, when the decoder detects error, the arithmetic or algebraic structure of the code is used to determine which of the valid codewords was transmitted.

 It is possible for a detectable error pattern to cause the decoder to select a codeword other than that which was actually transmitted. The decoder is then said to have committed a decoding error.

44 Square array for error correction by parity checking.   The codeword is formed by arranging k message bits in a square array whose rows and columns are checked by parity bits.  A transmission error in one message bit causes a row   and column parity failure with the error at the intersection, so single errors can be corrected.

45 [Carlson & Crilly, p 594] Example: square array   parity bits.    ______

46 [Carlson & Crilly, p 594] Weight and Distance  The weight of a vector is the number of nonzero coordinates in the vector.  The weight of a vector 𝐱 is commonly written as 𝒘 𝐱 .  Ex. 𝑤 010111  For BSC with cross-over probability 𝑝0.5, error pattern with smaller weight (less #1s) are more likely to occur.  The Hamming distance between two n-bit blocks is the number of coordinates in which the two blocks differ.  Ex. 𝑑 010111,011011  Note:  The Hamming distance between any two vectors equals the weight of their sum.  The Hamming distance between the transmitted codeword 𝐱 and the received vector 𝐲 is the same as the weight of the corresponding error pattern 𝐞. 47 Probability of Error Patterns  Recall: We assume that the channel is BSC with crossover probability 𝒑.  For the discrete memoryless channel that we have been considering since Chapter 3,  the probability that error pattern 𝐞 00101 is 1𝑝 1𝑝 𝑝 1𝑝 𝑝.  Note also that the error pattern is independent from the transmitted vector 𝐱  In general, from Eq. (11) in Section 3.4, the probability the error pattern 𝐞 occurs is 𝐱,𝐲 𝐞 𝑝 𝑝 𝑝 𝐱,𝐲 1𝑝 𝐱,𝐲 1𝑝 1𝑝 1𝑝 1𝑝  As in 3.58, if we assume 𝒑𝟎.𝟓, the error patterns that have larger weights are less likely to occur.  This also supports the use of minimum distance decoder.

8 Review: Minimum Distance (dmin)

The minimum distance (dmin) of a block code is the minimum Hamming distance between all pairs of distinct codewords.  Ex. Problem 5 of HW4:

48  Ex. Repetition code: dmin: two important facts

 For any linear block code, the minimum distance (dmin) can be found from minimum weight of its nonzero codewords.  So, instead of checking 2 pairs, 2 simply check the weight of the 2 codewords.

 A code with minimum distance dmin can

 detect all error patterns of weight w ≤ dmin-1.  correct all error patterns of weight w ≤ .

the floor function

49 dmin is an important quantity

Recall: Codebook construction Choose 𝑀2 from 2 possibilities to be used as codewords.

50 Triple-repetition code Parity-check code dmin is an important quantity

Recall: Codebook construction Choose 𝑀2 from 2 possibilities to be used as codewords.

𝐜 𝐜 𝐜 𝐜 𝐜

𝐜 𝐜 𝐜

51 dmin is an important quantity

 To be able to detect all w-bit errors, we need 𝑑 𝑤1.  With such a code there is no way that w errors can change a valid codeword into another valid codeword.  When the receiver observes an illegal codeword, it can tell that a transmission error has occurred.

𝐜 𝐜 𝐜 𝐜 𝐜

𝐜 𝐜 𝐜 Consider all the (valid) codewords (in the codebook). 52 dmin is an important quantity

𝐜 𝐜 𝐜 𝐜 𝐜

𝐜 𝐜 𝐜 We can find the distances between them. 53 dmin is an important quantity

𝐜 𝐜 𝐜 We can then find dmin. dmin 𝐜

𝐜

54 dmin is an important quantity

𝐜 𝐜 𝐜 When we draw a circle (sphere, dmin hypersphere) of radius dmin around a codeword, we know 𝐜 that there can not be another 𝐜 codeword inside this circle.

55 dmin is an important quantity

𝐜 𝐜 d 𝐜min When we draw a circle (sphere, hypersphere) of radius dmin around a codeword, we know 𝐜 that there can not be another 𝐜 codeword inside this circle.

56 dmin is an important quantity

𝐜 𝐜 𝐜 When we draw a circle (sphere, hypersphere) of radius dmin around a codeword, we know d min 𝐜 that there can not be another 𝐜 codeword inside this circle.

57 dmin is an important quantity

𝐜 𝐜 dmin Suppose the codeword 𝐜 is chosen to be transmitted; that is 𝒙 𝐜 𝒙 𝐜 . 𝐜

𝐜

58 dmin is an important quantity

𝒚 𝒙 ⊕𝒆 𝐜 𝐜 𝒆 The received vector 𝒚 can be calculated from 𝒙 𝐜 𝒚 𝒙 ⊕𝒆. 𝐜

𝐜

59 dmin is an important quantity

When 𝑑 𝑤, there is no way that 𝑤 errors can change a valid 𝐜 𝐜 𝒆 codeword into another valid codeword. 𝒙

𝐜

60 dmin is an important quantity

For some codewords,… 𝐜 𝐜 𝐜 𝒆 When 𝑑 𝑤, it is possible 𝒙 𝐜 that 𝑤 errors can change a valid 𝐜 codeword into another valid codeword.

61 dmin is an important quantity

When 𝑑 𝑤, there is no way that 𝑤 errors can change a valid 𝐜 𝐜 𝒆 codeword into another valid codeword. 𝒙 When 𝑑 𝑤, it is possible 𝐜 that 𝑤 errors can change a valid 𝐜 codeword into another valid codeword.

62 dmin is an important quantity

 To be able to correct all w-bit errors, we need 𝑑 2𝑤1.  This way, the legal codewords are so far apart that even with w changes, the original codeword is still closer than any other codeword.

𝐜 𝐜 𝐜

𝐜 dmin 𝐜 2

63 dmin is an important quantity

𝐜 𝐜 𝐜

𝒙 𝐜 dmin 𝐜 2

64 Example Consider the code  0000000000, 0000011111, 1111100000, and 1111111111

 Is it a linear code? ⊕ 𝐜 𝐜 𝐜 𝐜 0000000000 𝐜 0000011111 𝐜 1111100000 𝐜  d = min 1111111111 𝐜

 It can detect (at most) ___ errors.

 It can correct (at most) ___ errors.

65 Review: Block Codes  k = number of bits in each data block.  There are possibilities for the k-bit data block.  n = number of bits in each codeword.  There are 2k valid codewords.  One for each possible data block.

𝑀2 possibilities 𝐱 Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be 𝐬 𝐱 used as codewords. 𝐬 𝐱

1 Review: Channel Encoder and Decoder 𝑀2 possibilities 𝐱  = the collection of all Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be codewords. 𝐬 𝐱 used as codewords. m,d,b,s 𝐬 Message (Data block) 𝐱 x,c Channel Digital codeword Encoder Modulator Transmitted k bits k bits k bits Signal Add n bits n bits n bits systematic redundancy 0 1-p 0 p Channel Binary Symmetric p 1 1-p 1 Channel with p < 0.5 Received Noise & Interference 𝐦,𝐝,𝐛 ,𝐬 Signal Channel Digital yxe  Recovered Message Decoder Demodulator

2 minimum distance decoder Review: Even Parity  A binary vector (or a collection of 1s and 0s) has even parity if and only if the number of 1s in there is even.  Suppose we are given the values of all the bits except one bit.  We can force the vector to have even parity by setting the value of the remaining bit to be the sum of the other bits.

Single-parity-check code Square array [1 0 1 1 0 _]

5 Review: Linear Block Codes  Given a list of codewords for a code , we can determine whether  is linear by  Definition: if 𝐱 and 𝐱 ∈ , then 𝐱 𝐱 ∈   Shortcut:  First check that  must contain 0.  Then, check only pairs of the non-zero codewords.  One check = three checks  Codewords can be generated by a generator matrix th  𝐱𝐛𝐆 𝑏𝐠 where 𝐠 is the i row of 𝐆  Codebook can be generated by  working row-wise: generating each codeword one-by-one, or  working column-wise: first, reading, from 𝐆, how each bit in the codeword is created from the bits in 𝐛; then, in the codebook, carry 3 out the operations on columns 𝐛. Listing all possible codewords from G

0111 G  b x 1001  00 0000 0111 01 1001 xbGbb 1 2   0111 1001 10 11 1110 b21bbb1 1 b 2

66 Hamming codes  One of the earliest codes studied in coding theory.  Named after Richard W. Hamming  The IEEE Richard W. Hamming Medal, named after him, is an award given annually by Institute of Electrical and Electronics Engineers (IEEE), for "exceptional contributions to information sciences, systems and technology“.  Sponsored by Qualcomm, Inc  Some Recipients:  1988 - Richard W. Hamming  1997 -Thomas M. Cover  1999 - David A. Huffman  2011 -Toby Berger  The simplest of a class of (algebraic) error correcting codes that can correct one error in a block of bits

67 Hamming codes

68 [https://www.youtube.com/watch?v=cBBTWcHkVVY] Hamming codes

Alice Bob

Alice and Bob exchange messages by plucking a wire either hard or soft to transmit a zero versus a one. Due to gusts of wind, false zerosor ones can occur during transmission resulting in errors.

In the 1940s, Richard Hamming faced a similar problem while working at Bell Laboratories. 69 Hamming codes

At the time the computers used stored information on punch cards representing one versus zero with hole versus no hole.

This system was error-prone because it was common for cards to get bent or miss punched in the first place so holes could be missed or no holes could be accidentally punctured causing flipped bits.

Hamming took it upon himself to devise a method which could automatically detect and correct single bit errors without interrupting calculations. 70 Hamming codes

71 Hamming codes

72 [https://www.youtube.com/watch?v=cBBTWcHkVVY] Hamming codes: Ex. 1

73 [https://www.youtube.com/watch?v=cBBTWcHkVVY] Hamming codes: Ex. 1 This is an example of Hamming (7,4) code In the video, the codeword is constructed from the data by p1

d1 d2 where d4 p2 d3 p3

 The message bits are also referred to as the data bits or information bits.  The non-message bits are also referred to as parity check bits, checksum 74 bits, parity bits, or check bits. Writing the generator matrix from the code “structure” Generator matrix: a revisit  Fact: The 1s and 0s in the jth column of G tells which positions of the data bits are combined (⊕) to produce the jth bit in the codeword.  For the Hamming code in the previous slide,

x  pd11 pd 2 2 pdd 33 4

11  𝑝 𝑑 ⊕𝑑 ⊕𝑑 10  𝑝 𝑑 ⊕𝑑 ⊕𝑑    dddd1234 𝑝 𝑑 ⊕𝑑 ⊕𝑑 00    10 

75 G Generator matrix: a revisit  From 𝐱𝐛𝐆 𝑏 , we see that the j element of the codeword 𝐱 of a linear code is constructed from a linear combination of the bits in the message:

𝑥 𝑏𝑔 .  The elements in the jth column of the generator matrix become the weights for the combination.  Because we are working in GF(2), 𝑔 has only two values: 0 or 1.  When it is 1, we use 𝑏 in the sum.  When it is 0, we don’t use 𝑏 in the sum.  Conclusion: For the jth column, the ith element is determined from whether the ith message bit is used in the sum that produces the jth element of the codeword 𝐱. 76 Codebook of a linear block code

𝐝 𝐱  Now that we have a 00000000000 sufficiently-large example 00011010101 of a codebook, let’s 00100010110 00111000011 consider some important 01001001100 types of problems. 01010011001  Given a codebook, how can 01101011010 01110001111 we check that the code is 10001110000 linear and generated by a 10010100101 generator matrix? 10101100110  10110110011 Given a codebook, how can 11000111100 we find the corresponding 11011101001 generator matrix? 11100101010 11111111111 77 Codebook of a linear block code

𝐝 𝐱 Note that 00000000000  Each bit of the codeword for 00011010101 linear code is either 00100010110  the same as one of the message 00111000011 bits  Here, the second bit (x2) of the 01001001100 codeword is the same as the first 01010011001 bit (b1) of the message 01101011010  the sum of some bits from the 01110001111 message

10001110000  Here, the first bit (x1) of the 10010100101 codeword is the sum of the first, second and fourth bits of the 10101100110 message. 10110110011  So, each column in the codebook 11000111100 should also satisfy the above 11011101001 structure (relationship). 11100101010 11111111111 78 “Reading” the structure from the codebook.

𝐝 𝐱  One can “read” the 00000000000 structure (relationship) 𝑑 00011010101 from the codebook. 𝑑 00100010110 00111000011  From 𝑥 𝑑𝑔 𝑑 01001001100 01010011001 when we look at the 01101011010 01110001111 message block with a 𝑑 10001110000 single 1 at position , 10010100101 then 10101100110  𝑥 10110110011 the value of in the 11000111100 corresponding 11011101001 codeword gives 𝑔 11100101010  𝑥 𝑑 ⊕𝑑 ⊕𝑑 11111111111 79  𝑥 𝑑 ⊕𝑑 ⊕𝑑 “Reading” the generator matrix from the codebook.

𝐝 𝐱  One can also “read” 𝐆 from the 00000000000 codebook. 𝐠  𝐱𝐝𝐆 𝑑 00011010101 From 𝑑 𝐠 00100010110 𝑑𝐠 , 00111000011 𝑑 01001001100𝐠 when we look at the 01010011001 message block with a single 01101011010 1 at position 𝑖, then 01110001111 the corresponding 𝑑 10001110000𝐠 codeword is the same as 10010100101 𝐠 . 10101100110 10110110011 1110000 11000111100 1001100 11011101001  G 11100101010 0010110  11111111111 1010101 80 Checking whether a code is generated by some generator matrix G

𝐝 𝐱  If a code is generated by a generator 00000000000 matrix, it is automatically a linear code.  𝑑 00011010101 When the codebook is provided, look at each column of the codeword part. 𝑑 00100010110 00111000011  Write down the equation by reading the 𝑑 01001001100 structure from appropriate row discussed 01010011001 earlier. 01101011010  For example, here, we read 01110001111 𝑥 𝑑 ⊕𝑑 ⊕𝑑.  𝑑 10001110000 Then, we add the corresponding 10010100101 columns of the message part and check 10101100110 whether the sum is the same as the corresponding codeword column. 10110110011 11000111100  So, we need to check n summations. 11011101001  Direct checking that we discussed 𝑀1 11100101010 earlier consider 2 11111111111 summations. 81 Example:

𝐝 𝐱  We read 00000000000 . 𝑑 00011010101𝐠 𝐠 𝑑 00100010110  We add the message 00111000011 𝑑 01001001100𝐠 columns corresponding 01010011001 to , 01101011010 01110001111  We see that the first bit 𝐠 𝑑 10001110000 of the 13th codeword 10010100101 10101100110 does not conform with 10110110011 the structure above. 11001 111100 11011101001  Conclusion: This code is 11100101010 not generated by a 11111111111 82 generator matrix. Example:

𝐝 𝐱  The case found in the previous 00000000000 slide may help with the search to 𝑑 00011010101𝐠 show that the code is not linear. 𝑑 00100010110𝐠  00111000011 The corresponding message is 𝑑 01001001100𝐠 1100. 01010011001  The codeword corresponding to 01101011010 this message should be 𝐠 ⊕ 01110001111 𝐠 . 𝑑 10001110000𝐠 10010100101  If 𝐠 ⊕𝐠 , is not a codeword, 10101100110 then we can quickly conclude 10110110011 that the code is not linear: 11001 111100 𝐠 and 𝐠 are codewords but 11011101001 𝐠 ⊕𝐠 0111100 is not one 11100101010 of the codewords. 11111111111 83 Implementation  Linear block codes are typically implemented with modulo-2 adders tied to the appropriate stages of a shift register.

𝑝 𝑝 𝑝

where

𝑑 𝑑 𝑑 𝑑

84 Back to Hamming codes: Ex. 1 p1

d1 d2 d4 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 d3 𝐱 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 p2 p3 Structure in the codewords: 𝑝 𝑑 ⊕𝑑 ⊕𝑑 𝑝 ⊕𝑑 ⊕𝑑 ⊕𝑑 0 𝑝 𝑑 ⊕𝑑 ⊕𝑑 𝑝 ⊕𝑑 ⊕𝑑 ⊕𝑑 0 𝑝 𝑑 ⊕𝑑 ⊕𝑑 𝑝 ⊕𝑑 ⊕𝑑 ⊕𝑑 0 100 At the receiver, we check whether the received 110 vector 𝐲 still satisfies these conditions via computing  010 the syndrome vector: s0 yyyyyyy101 ? 1234567 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 001 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑  0 11 85 111 Parity Check Matrix: Ex 1  Intuitively, the parity check matrix 𝐇, as the name suggests, tells which bits in the observed vector 𝐲 are used to “check” for validity of 𝐲.  The number of rows is the same as the number of conditions to check (which is the same as the number of parity check bits).  For each row, a one indicates that the bits (including the bits in the parity positions) are used in the validity check calculation.

𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 Structure in the codeword: 1101001 𝑝 ⊕𝑑 ⊕𝑑 ⊕𝑑 0 H  0110011 𝑝 ⊕𝑑 ⊕𝑑 ⊕𝑑 0   𝑝 ⊕𝑑 ⊕𝑑 ⊕𝑑 0 0001111

86 Parity Check Matrix: Ex 1 Relationship between and .

𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 1110000 1101001 1001100 G  H  0110011 0010110   0001111 1010101

87 Parity Check Matrix: Ex 1 Relationship between and .

88 Parity Check Matrix: Ex 1 Relationship between and .

(columns of) identity matrix (columns of) identity matrix in the data positions in the parity check positions

89 Parity Check Matrix: Ex 1 Relationship between and .

90 Review: Linear Block Codes

Code structure

𝐱 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 p1 where 𝑝 𝑑 ⊕𝑑 ⊕𝑑 d1 d2 𝑝 𝑑 ⊕𝑑 ⊕𝑑 d4

d3 𝑝 𝑑 ⊕𝑑 ⊕𝑑 p2 p3

Codebook 𝐝 𝐱

𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 1110000 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 1001100 1101001   G  H  0110011 0010110     0001111 1010101   Generator Matrix Parity Check Matrix 91 Remark: Location of the Message Bits

𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 1110000 1101001 1001100 G    H  0110011 0010110     0001111 1010101

 The “identity-matrix” columns in corresponds to positions of the message (data) bits in each codeword.  Ex. For this code, codeword 𝐱 1 1 0 0 1 1 0 corresponds to message 𝐛 1 0 1 0.  The “identity-matrix” columns in corresponds to positions of the parity (check) bits in each codeword.

91 Review: Linear Block Codes  The code structure is built into each codeword at the encoder (transmitter) via the generator matrix  Each codeword is created by .  The code structure is checked at the decoder (receiver) via the parity check matrix.  A valid codeword must satisfy .

𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑦 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑥 𝑝 𝑑 𝑝 𝑑 𝑝 𝑑 𝑑 1110000 1101001 1001100 G    H  0110011 0010110     0001111 1010101 92 Parity Check Matrix Key property: T GH 0knk Proof:  When there is no error , the syndrome vector calculation should give .  By definition, .  Therefore, when , we have .  To have for any , we must have . A matrix of zeroes

91 Systematic Encoding  Code constructed with distinct information bits and check bits in each codeword are called systematic codes.  Message bits are “visible” in the codeword.  Popular forms of G: GI P xGb bb bP I  knkk 1 2 k  knkk 

 xx1 2  xnk bb1 2  bk xnk 1 xnk 2 xn

GI P xGb b b  b  IP  k knk 1 2 k  k knk 

 b12b  bk xk1 xk 2  xn x x x 92 1 2 k Parity check matrix  For the generators matrices we discussed in the previous slide, the corresponding parity check matrix can be found easily:

GP  I HI PT  knkk  nk 

I Check: GHT  P I P P 0  knk -P

GIP HPI T  k knk  nk 

93 Hamming codes: Ex. 2  Systematic (7,4) Hamming Codes

0111000 1010100 G    1100010   1110001

1000111 H  01 01 011   0011101

94 Hamming codes Now, we will give a general recipe for constructing Hamming codes. Parameters:  number of parity bits    It can be shown that, for Hamming codes,

 dmin = 3.  Error correcting capability:

95 Construction of Hamming Codes  Start with m. 1. Parity check matrix H:  Construct a matrix whose columns consist of all nonzero binary m-tuples.  The ordering of the columns is arbitrary. However, next step is easy when the columns are arranged so

that HI   m P  . 2. Generator matrix G:  HI P  TT  When  m  , we have GPIPI   kk    .

96 Hamming codes: Ex. 2  Systematic (7,4) Hamming Codes

𝐏 1000111  Columns are all possible nonzero 3- bit vectors H  01 01 011  We arrange the columns so that I3 is 0011101 on the left to make the code systematic. (One can also put I3 on the right.)

0111000  Note that the size of the identity 1010100 matrices in 𝐆 and 𝐇 are not the same. G    1100010   1110001 97 𝐏 Review: Hamming Code Recipe Here, 𝑚3 𝐏 𝑛2 1 𝐈  Start with the parity-check matrix 7 1000111  𝑚 rows  𝑚𝑛𝑘 𝑘4 H  01 01 011  Columns are all possible nonzero 𝑚-bit 0011101 vectors  𝑛2 1columns  Arranged to have 𝐈 on the left (or on the right).  This simplifies conversion to 𝐆. 0111000  Get 𝐆 from 𝐇. 1010100   G  GP I HI PT  1100010  knkk   nk    1110001    Note that the size of the identity 𝐏 𝐈 matrices in 𝐆 and 𝐇 can be different.

98 Minimum Distance Decoding  At the decoder, suppose we want to use minimum distance decoding, then  The decoder needs to have the list of all the possible codewords so that it can compare their distances to the received vector .  There are 2k codewords each having n bits. Therefore, saving these takes bits.  Also, we will need to perform the comparison 2k times.  Alternatively, we can utilize the syndrome vector (which is computed from the parity-check matrix).  The syndrome vector is computed from the parity-check matrix .  Therefore, saving takes bits.

98 Minimum Distance Decoding  Recall that 𝑑 𝐱,𝐲 𝒘 𝐱⊕𝐲 𝒘 𝐞  Therefore, minimizing the distance is the same as minimizing the weight of the error pattern.  New goal:  find the decoded error pattern 𝐞 with the minimum weight  then, the decoded codeword is 𝐱𝐲 ⊕𝐞  Once we know 𝐱 we can directly extract the message part from the decoded codeword if we are using systematic code.  For example, consider 0111000 1010100 G    1100010   1110001 Suppose 𝐱 1011010, then we know that the decoded message is 𝐛 1010.

99 Properties of Syndrome Vector  Recall that, from , we have    Thinking of as a matrix with many columns inside,

h1     h2 TT T Hvvv 12 n     h   nk nk n

 Therefore, is a (linear combination of the columns of )T. 100 Hamming Codes: Ex. 2

1000111 H  01 01 011   0011101 Error pattern e Syndrome = eHT (0,0,0,0,0,0,0) (0,0,0) 𝐬 𝐞𝐇 𝑒𝐯 (0,0,0,0,0,0,1) (1,1,1) (0,0,0,0,0,1,0) (1,1,0) Linear combination of (0,0,0,0,1,0,0) (1,0,1) the columns of H (0,0,0,1,0,0,0) (0,1,1) Note that for an error pattern with a single one in the jth (0,0,1,0,0,0,0) (0,0,1) coordinate position, the (0,1,0,0,0,0,0) (0,1,0) syndrome is the same as the jth column of H. (1,0,0,0,0,0,0) (1,0,0) 101 Properties of Syndrome Vector  We will assume that the columns of 𝐇 are nonzero and distinct.  This is automatically satisfied for Hamming codes constructed from our recipe.  Case 1: When 𝐞 𝟎, we have 𝐬 𝟎.  When 𝐬 𝟎, we can conclude that 𝐞 𝟎.  There can also be 𝐞 𝟎 that gives 𝐬 𝟎.  For example, any nonzero 𝐞 ∈ , will also give 𝐬 𝟎.  However, they have larger weight than 𝐞 𝟎.  The decoded codeword is the same as the received vector. 0, 𝑖 𝑗,  𝑒 th Case 2: When, 1, 𝑖 𝑗, (a pattern with a single one in the j position) th T we have 𝐬 𝐯 (the j column of 𝐇) 0, 𝑖 𝑗,  𝐬 th 𝐇 T 𝑒̂ When ( the j column of ) , we can conclude that 1, 𝑖 𝑗,  There can also be other 𝐞 that give 𝐬 𝐯. However, their weights  can not be 0 (because, if so, we would have 𝐬 𝟎 but the columns of 𝐇 are nonzero)  nor 1 (because the columns of 𝐇 are distinct).  We flip the jth bit of the received vector to get the decoded codeword.

102 Decoding Algorithm  Assumption: the columns of are nonzero and distinct.  Compute the syndrome for the received vector.  Case 1: If , set .  Case 2: If ,  determine the position j of the column of H that is the same as (the transposition) of the syndrome,  set but with the jth bit complemented.  For Hamming codes, because the columns are constructed from all possible non-zero m-tuples, the syndrome vectors must fall into one of the two cases considered above.  For general linear block codes, the two cases above may not cover every cases. 103 Review: Linear Block Code 𝑀2 possibilities 𝐱 Choose 𝑀2 from 𝒔 𝐱 𝐬 2 possibilities to be 𝐬 𝐱 used as codewords. m,d,b,s 𝐬 Message (Data block) 𝐱 x,c Channel Digital codeword Encoder Modulator Transmitted k bits k bits k bits 𝐱𝐝𝐆 Signal n bits n bits n bits Generator matrix 0 1-p 0 p Channel Binary Symmetric p 𝐆𝐇 𝟎 1 1-p 1 Channel with Parity-check matrix p < 0.5 Syndrome 𝐬 𝒚𝐇 𝐞𝐇 Received Noise & Interference 𝐦,𝐝,𝐛 ,𝐬 Signal Channel Digital yxe  Recovered Message Decoder Demodulator

107 minimum distance decoder Hamming Codes: Ex. 1  Consider the Hamming code with

1110000 1101001 1001100 G  H  0110011 0010110   0001111 1010101  

 Suppose we observe at the receiver. Find the decoded codeword and the decoded message.

104 [To be explored in the HW] Hamming Codes: The original method  Encoding  The bit positions that are powers of 2 (1, 2, 4, 8, 16, etc.) are check bits.  The rest (3, 5, 6, 7, 9, etc.) are filled up with the k data bits.  Each check bit forces the parity of some collection of bits, including itself, to be even (or odd).  To see which check bits the data bit in position i contributes to, rewrite i as a sum of powers of 2. A bit is checked by just those check bits occurring in its expansion  Decoding  When a codeword arrives, the receiver initializes a counter to zero. It then examines each check bit at position i (i = 1, 2, 4, 8, ...) to see if it has the correct parity.  If not, the receiver adds i to the counter. If the counter is zero after all the check bits have been examined (i.e., if they were all correct), the codeword is accepted as valid. If the counter is nonzero, it contains the position of the incorrect bit.

105 Interleaving  Conventional error-control methods such as parity checking are designed for errors that are isolated or statistically independent events.  Some errors occur in bursts that span several successive bits.  Errors tend to group together in bursts. Thus, errors are no longer independent  Examples  impulse noise produced by lightning and switching transients  fading or in wireless systems  channel with memory  Such multiple errors wreak havoc on the performance of conventional codes and must be combated by special techniques.  One solution is to spread out the transmitted codewords.  We consider a type of interleaving called block interleaving. 106 Interleave as a verb  To interleave = to combine different things so that parts of one thing are put between parts of another thing  Ex. To interleave two books together:

107 Interleaving: Example

Consider a sequence of m blocks of coded data:

ℓ ℓ ℓ 𝑥 𝑥 ⋯𝑥 𝑥 𝑥 ⋯𝑥 ⋯ 𝑥 𝑥 ⋯𝑥

 Arrange these blocks as rows of a table.  Normally, we get the bit sequence simply by 𝑥 𝑥 ⋯𝑥 reading the table by rows.  𝑥 𝑥 ⋯𝑥 With interleaving (by an interleaver), transmission ⋮⋮⋱⋮ is accomplished by reading out of this table by columns. 𝑥 ℓ 𝑥 ℓ ⋯𝑥ℓ  Here, ℓ blocks each of length n are interleaved to form a sequence of length ℓn.

ℓ ℓ ℓ 𝑥 𝑥 ⋯ 𝑥 𝑥 𝑥 ⋯ 𝑥 ⋯ 𝑥 𝑥 ⋯ 𝑥

The received symbols must be deinterleaved (by a deinterleaver) prior to decoding.

108 Interleaving: Advantage  Consider the case of a system that can only correct single errors.  If an error burst happens to the original bit sequence, the system would be overwhelmed and unable to correct the problem.

ℓ ℓ ℓ original bit sequence 𝑥 𝑥 ⋯𝑥 𝑥 𝑥 ⋯𝑥 ⋯ 𝑥 𝑥 ⋯𝑥

ℓ ℓ ℓ interleaved transmission 𝑥 𝑥 ⋯ 𝑥 𝑥 𝑥 ⋯ 𝑥 ⋯ 𝑥 𝑥 ⋯ 𝑥

 However, in the interleaved transmission,  successive bits which come from different original blocks have been corrupted  when received, the bit sequence is reordered to its original form and then the FEC can correct the faulty bits  Therefore, single error-correction system is able to fix several errors. 109 Interleaving: Advantage  If a burst of errors affects at most consecutive bits, then each original block will have at most one error.  If a burst of errors affects at most consecutive bits (assume 𝑟𝑛), then each original block will have at most errors.  Assume that there are no other errors in the transmitted stream of n bits.  A single error-correcting code can be used to correct a single burst spanning upto ℓ symbols.  A double error-correcting code can be used to correct a single burst spanning upto 2ℓ symbols.

110 References: Linear Codes  Lathi and Ding, Modern Digital and Analog Communication Systems, 2009  [TK5101 L333 2009]  Chapter 15 p. 907-918  Carlson and Crilly, Communication Systems: An Introduction to Signals and Noise in Electrical Communication, 2010  [TK5102.5 C3 2010]  Chapter 13 p. 591-597, 604-611  Cover and Thomas, Elements of Information Theory, 2006  1991 Version: [Q360 C68 1991]  Section 7.11 p. 210-215  Sklar and F. J. Harris, The ABCs of linear block codes, IEEE Signal Process. Mag., 21(4), (2004).  111 p. 14–24 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 5.2 Binary Convolutional Codes

Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 112 BKD Binary Convolutional Codes  Introduced by Elias in 1955  There, it is referred to as convolutional parity-check symbols codes.  Peter Elias received  Claude E. Shannon Award in 1977  IEEE Richard W. Hamming Medal in 2002  for "fundamental and pioneering contributions to information theory and its applications  The encoder has memory.  In other words, the encoder is a sequential circuit or a finite- state machine.  Easily implemented by shift register(s).  The state of the encoder is defined as the contents of its memory.

113 Binary Convolutional Codes  The encoding is done on a continuous running basis rather than by blocks of k data digits.  So, we use the terms bit streams or sequences for the input and output of the encoder.  In theory, these sequences have infinite duration.  In practice, the state of the convolutional code is periodically forced to a known state and therefore code sequences are produced in a block-wise manner.

114 Binary Convolutional Codes 𝒌  In general, a rate-𝒏 convolutional encoder has  k shift registers, one per input information bit, and  n output coded bits that are given by linear combinations (over the binary field, of the contents of the registers and the input information bits.  k and n are usually small.  For simplicity of exposition, and for practical purposes, 𝟏 only rate-𝒏 binary convolutional codes are considered here.  k = 1.  These are the most widely used binary codes.

115 (Serial-in/Serial-out) Shift Register  Accept data serially: one bit at a time on a single line.  Each clock pulse will move an input bit to the next FF. For example, a 1 is shown as it moves across.  Example: five-bit serial-in serial-out register.

FF0 FF1 FF2 FF3 FF4 Serial 1 1 1 1 1 1 Serial data D0 Q0 D1 Q1 D2 Q2 D3 Q3 D4 Q4 data input output C C C C C

CLKCLK

116 Example 1: n = 2, k = 1

+ +

117 Graphical Representations  Three different but related graphical representations have been devised for the study of convolutional encoding: 1. the state diagram 2. the code tree 3. the trellis diagram

118 Ex. 1: State (Transition) Diagram  The encoder behavior can be seen from the perspective of a finite state machine with its state (transition) diagram.

0/00 +

00 1/11 0/11

1/00 + 10 01 0/10 1/01 0/01 A four-state directed graph that 11 uniquely represents the input-output relation of the encoder. 1/10 119 Drawing State Diagram

10 01

+ +

120 Input bit Drawing State Diagram 0

00 + 1

0 0 0 + 10 01

+ 11

1 0 0 +

+ +

121 Corresponding Input bit output bits Drawing State Diagram 0/00

00 + 0 1/11

0 0 0 + 10 01 0

+ 1 11 1 0 0

+ 1

+ +

122 Drawing State Diagram 0/00

00 + 1 1/11 0/11

1/00

0 0 1 + 10 01 1

+ 0 11 1 0 1

+ 0

+ +

123 Drawing State Diagram 0/00

00 1/11 0/11 + 1/00 + 10 01 (1) (2) bs1 s2 x x 0/10 0000 0 1/01 0/01 1001 1 11 0011 1 1010 0 1/10 0101 0 + 1100 1

0110 1 1111 0 +

124 Tracing the State Diagram to Find the Outputs 0/00

00 1/11 0/11 1/00 10 01 0/10 1/01 0/01 Input 1 1 0 1 1 1 11 Output 11 01 01 00 01 10 1/10

+ +

125 Directly Finding the Output

+ +

(1) (2) bs1 s2 x x 100 1 Input 1 1 0 1 1 1 0 Output 1 1 +

1 +

126 Directly Finding the Output

+ +

(1) (2) bs1 s2 x x 1001 1 1100 1 Input 1 1 0 1 1 1 0110 1 Output 11 01 01 00 01 10 1010 0 1100 1 +

1111 0 +

127 Parts for Code Tree 0/00

00 1/11 0/11 0/11 0/00 00 00 1/00 00 01 1/11 1/00 10 01 10 10 0/10 1/01 0/01 0/01 11 0/10 01 01 10 11 1/01 11 1/10 11 1/10

Two branches initiate from each node, +

the upper one for 0 and the lower one for 1. +

128 Show the coded output for any possible sequence of data digits.

0/00 Code Tree 00 0/00 00 1/11 0/00 0/11 10 00 00 0/00 00 01 00 0/10 1/11 1/00 01 10 10 1/11 10 1/01 11 0/10 0/00 01 0/01 01 Initially, we 00 0/11 10 11 00 1/01 11 1/10 always assume 0/10 01 11 that all the 1/00 10 1/11 10 contents of the 0/01 10 1/01 11 register are 0. 1/10 11 Start 00 0/00 0/11 00 00 1/11 0/10 10 01 0/10 1/00 01 10 + 1/01 11 10 0/11 1/11 0/01 00

01 1/00 10 + 1/01 11 0/01 10 1/10 11 129 1/10 11 0/00 Code Tree 00 0/00 00 1/11 10 0/00 00 0/10 01 1/11 10 1/01 11 Input 1 1 0 1 0/00 00 0/11 00 Output 11 01 01 00 0/10 01 1/00 10 1/11 10 0/01 10 1/01 11 1/10 11 Start 00 0/00 0/11 00 00 1/11 0/10 10 01 0/10 1/00 01 10 + 1/01 11 10 0/11 1/11 0/01 00

01 1/00 10 + 1/01 11 0/01 10 1/10 11 130 1/10 11 Code Trellis [Carlson & Crilly, 2009, p. 620] 0/00 0/11 00 00 00 01 1/11 1/00 Current Next 10 10 State State 0/10 01 0/01 01 00 0/00 00 10 11 1/01 1/10 11 11

10 10

01 01

11 11 1/10

+ +

131 Review: Finding the Code Trellis

+ 0/00 Current Next State State 00 1/11 0/11 00 0/00 00

+ 1/00 10 01 10 10

0/00 0/11 0/10 00 00 00 01 01 01 1/11 1/00 1/01 0/01 10 10 11

0/10 01 0/01 01 11 11 10 11 1/10 1/01 1/10 11 11 1/10

2 Another useful way of Towards the Trellis Diagram representing the code tree. 0/00 0/00 00 0/00 0/00 0/00 0/00 00

10 10

01 01

11 11 1/10 1/10 1/10 1/10 1/10 1/10

+ +

132 Trellis Diagram

0/00 0/00 00 0/00 0/00 0/00 0/00 00

10 10

Initially, we 01 01 always assume that all the contents of the 11 11 1/10 1/10 1/10 1/10 register are 0.

+ Each path that traverses through the

trellis represents a valid codeword. +

133 Trellis Diagram

0/00 0/00 00 0/00 0/00 0/00 0/00 00

10 10

01 01

11 11 1/10 1/10 1/10 1/10

Input 1 1 0 1 1 1 +

Output 11 01 01 00 01 10 +

134 Trellis

โครงลูกไม้,โครงสร้างบงตาทั ่เปี ็นช่อง,โครงสร้างสาหรํ ับปลูกไม้เล้ือย

135 Convolutional Encoder: Graphical Representations

+ +

0/00 00 0/00 0/00 00 1/11 10 0/00 0/00 0/00 0/00 00 00 00 1/11 0/11 0/10 01 1/11 1/00 10 1/01 11 10 01 10 Start 00 0/10 0/11 00 0/01 0/10 1/01 01 01 11 1/00 10 10 1/11 0/01 1/10 01 11 1/01 11 1/10

1/10 11 136 +

The Codebook +

0/00 0/00 00 0/00 00 𝐛 𝐱 1/11 10 00 0/00 00 1/11 0/11 0/10 01 1/11 b b b x x x x x x 1/00 10 1 2 3 1 2 3 4 5 6 1/01 11 10 01 Start 00 0/11 0/10 00 0000 0 0 0 0 0 0/10 01 1/00 1/01 0/01 10 11 10 0010 0 0 0 1 1 1/11 0/01 01 1/01 11 1/10 1/10 11 0100 0 1 1 1 0 00 0/00 0/00 0/00 0110 0 1 1 0 1

10 1001 1 1 0 1 1

01 1011 1 1 0 0 0 1101 1 0 1 0 1 11 1/10 1111 1 0 1 1 0

137 Convolutional Encoder and Decoder

d,b + Message (Stream of data) x,c Convolutional Digital Encoder Modulator Encoded stream Transmitted Signal

0 1-p 0 p Channel Binary Symmetric p 1 1 Channel with 2 0/00 (2) 0/00 (1) 3 2 0/00 (0) 2 0/00 (1) 3 3 1-p 00

0 3 3 1 3 3 10 p < 0.5

2 1 2 2 3

01 Noise & Interference

0 1 2 1 1 Received 11 1/10 (1) 1/10 (1) 1/10 (0) 𝐝,𝐛 Signal Channel Digital yxe  Recovered Message Decoder Demodulator

138 minimum distance decoder Viterbi decoder Direct Minimum Distance Decoding  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .  𝐱 arg min 𝑑 𝐱,𝐲 𝐱

+ +

139 Direct Minimum Distance Decoding 𝐲

y1 y2 y3 y4 y5 y6 110111

𝐛 𝐱

b1 b2 b3 x1 x2 x3 x4 x5 x6 d(x,y) 000000000 5 001000011 3 010001110 4 011001101 4 100111011 2 101111000 4 𝐱argmin 𝑑 𝐱,𝐲 𝐱 1 110110101 110101 or 110110 111110110 1 140 𝐛 110 or 111 Direct Minimum Distance Decoding +  Suppose = [11 01 11].

 Find . +  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .

𝒚 = [ 11 01 11 ].

00 0/00 0/00 0/00 For 3-bit message, 3 10 there are 2 = 8 possible codewords. We can list all possible codewords. However, here, let’s first try to work 01 on the distance directly.

11 1/10 141 Direct Minimum Distance Decoding  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from . The number in parentheses on each 𝒚 = [ 11 01 11 ]. branch is the branch metric, obtained 0/00 (2) 0/00 (1) 0/00 (2) by counting the differences between 00 the encoded bits and the corresponding bits in 𝒚. 10

11 1/10 (1) 142 Direct Minimum Distance Decoding  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .

𝒚 = [ 11 01 11 ]. b d(x,y) 0/00 (2) 0/00 (1) 0/00 (2) 00 000 2+1+2 = 5 001 2+1+0 = 3 010 2+1+1 = 4 10 011 2+1+1 = 4 100 0+2+0 = 2 01 101 0+2+2 = 4 110 0+0+1 = 1 11 1/10 (1) 111 0+0+1 = 1 143 Viterbi decoding  Developed by Andrew J. Viterbi  Also co-founded Qualcomm Inc.  Published in the paper “Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm”, IEEE Transactions on Information Theory, Volume IT-13, pages 260-269, in April, 1967. https://en.wikipedia.org/wiki/Andrew_Viterbi

144 Viterbi and His Decoding Algorithm

[http://viterbi.usc.edu/about/viterbi/viterbi_video.htm] 145 [https://www.youtube.com/watch?v=A_xHptEqgp0] Viterbi and His Decoding Algorithm

We knew it was an important piece of work, but of course we had no idea of how it was going to revolutionize communications altogether.

I came up with some ideas which were kind of based on a tennis tournament, eliminating This is original copy of losers but always having new the Viterbi algorithm. I players coming to the game. read the paper and I really didn't see the significance of it; needless to say I was certainly wrong. 146 Viterbi and His Decoding Algorithm

Satellite communications, cellular communications, quite a few other things that we now take for granted, it made them really feasible to use on a large scale.

With 45 years of experience and working in communications, [Viterbi algotrithm] is the most important thing I've seen and it's relatively easy to teach.

147 Viterbi and His Decoding Algorithm

I teach the Viterbi algorithm in several classes ... and I can even teach it to undergrads because it's such a simple algorithm in hindsight and it makes such good intuitive sense. It's not just some theoretical concept that never really made it out of the academic community. Every cell phone in use today has at least one Viterbi processor in it and sometimes more.

Other things that you might not associate with communications like the disk drive in your computer has a Viterbi detector in it.

148 Andrew J. Viterbi  1991: Claude E. Shannon Award  1952-1957: MIT BS & MS  Studied electronics and communications theory under such renowned scholars as Norbert Wiener, Claude Shannon, Bruno Rossi, and Roberto Fano.  1962: Earned one of the first doctorates in electrical engineering granted at the University of Southern California (USC)  Ph.D. dissertation: error correcting codes  2004: USC Viterbi School of Engineering 149 named in recognition of his $52 million gift Andrew J. Viterbi

This donation from the Viterbi is the largest naming gift ever received by an American engineering school… Thanks to their gift of 52 million dollars, the USC school of engineering will become the USC Andrew and Erna Viterbi school of engineering.

[http://viterbi.usc.edu/about/viterbi/viterbi_video.htm] 150 [https://www.youtube.com/watch?v=A_xHptEqgp0] Andrew J. Viterbi  Cofounded Qualcomm  Helped to develop the CDMA standard for cellular networks.  1998 Golden Jubilee Award for Technological Innovation  To commemorate the 50th Anniversary of Information Theory  Given to the authors of discoveries, advances and inventions that have had a profound impact in the technology of information transmission, processing and compression. 1. Norman Abramson: For the invention of the first random-access communication protocol. 2. Elwyn Berlekamp: For the invention of a computationally efficient algebraic decoding algorithm. 3. Claude Berrou, Alain Glavieux and Punya Thitimajshima (ปัญญา ฐิติมชฌั ิมา): For the invention of turbo codes. 4. Ingrid Daubechies: For the invention of wavelet-based methods for signal processing. 5. Whitfield Diffie and Martin Hellman: For the invention of public-key cryptography. 6. Peter Elias: For the invention of convolutional codes. 7. G. David Forney, Jr: For the invention of concatenated codes and a generalized minimum-distance decoding algorithm. 8. Robert M. Gray: For the invention and development of training mode vector quantization. 9. David Huffman: For the invention of the Huffman minimum-length lossless data- compression code. 10. Kees A. Schouhamer Immink: For the invention of constrained codes for commercial recording systems. 11. Abraham Lempel and Jacob Ziv: For the invention of the Lempel-Ziv universal data compression algorithm. 12. Robert W. Lucky: For the invention of pioneering adaptive equalization methods. 13. Dwight O. North: For the invention of the matched filter. 14. Irving S. Reed: For the co-invention of the Reed-Solomon error correction codes. 15. Jorma Rissanen: For the invention of arithmetic coding. 16. Gottfried Ungerboeck: For the invention of trellis coded modulation. 17.Andrew J. Viterbi: For the invention of the Viterbi algorithm.

151 [ http://www.itsoc.org/honors/golden-jubilee-awards-for-technological-innovation ] Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .

𝒚 = [ 11 01 11 ]. 2 0/00 (2) 0/00 (1) 30/00 (2) 00 Each circled number at a node is the running 0 3 10 (cumulative) path metric, obtained by summing 2 01 branch metrics (distance) up to that node. Here, it is 0 11 1/10 (1) simply the cumulative 152 distance. Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .  For the last column of nodes, 𝒚 = [ 11 01 11 ]. 2 each of the nodes has two 0/00 (2) 0/00 (1) 30/00 (2) 00 branches going into it.  So, there are two possible 0 3 cumulative distance values. 10  We discard the larger- 2 metric path because, 01 regardless of what happens subsequently, this path will 0 have a larger Hamming 11 1/10 (1) distance from y . 153 Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .

𝒚 = [ 11 01 11 ]. 2 3 5 0/00 (2) 0/00 (1) 0/00 (2) min(3+2,2+0) 00 2 We discard the larger-distance 0 3 3 path because, 10 min(3+0,2+2) 4 regardless of what happens subsequently, 2 4 min(3+1,0+1) this path will have a 01 1 larger Hamming 0 4 distance from y. 11 min(3+1,0+1) 1/10 (1) 1 154 Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .  For the last column of nodes, 𝒚 = [ 11 01 11 ]. each of the nodes has two 2 3 5 0/00 (2) 0/00 (1) 0/00 (2) branches going into it. 00 × 2  So, there are two possible 0 3 3 cumulative distance values. 10 × 4  We discard the larger- 2 × 4 distance path because, 01 1 regardless of what happens subsequently, this path will 0 × 4 have a larger Hamming 11 1/10 (1) 1 distance from y. 155 Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .  For the last column of nodes, 𝒚 = [ 11 01 11 ]. 2 each of the nodes has two 0/00 (2) 0/00 (1) 3 2 00 branches going into it.  So, there are two possible 0 3 3 cumulative distance values. 10  We discard the larger- 2 1 distance path because, 01 regardless of what happens subsequently, this path will 0 1 have a larger Hamming 11 1/10 (1) distance from y . 156 Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .

𝒚 = [ 11 01 11 ]. 2 0/00 (2) 0/00 (1) 3 2 00 Note that we keep exactly one 0 3 3 10 (optimal) survivor path to each state. 2 1 01 (Unless there is a tie, then we keep both 0 1 or choose any.) 11 1/10 (1) 157 Viterbi Decoding: Ex. 1  Suppose = [11 01 11].  Find .  Find the message which corresponds to the (valid) codeword with minimum (Hamming) distance from .

𝒚 = [ 11 01 11 ].  So, the codewords 2 0/00 (2) 0/00 (1) 3 2 00 which are nearest to is [11 01 01] or [11 01 0 3 3 10 10].  2 1 The corresponding 01 messages are [110] or 0 1 [111], respectively. 11 1/10 (1) 158 Viterbi Decoding: Ex. 2  Suppose = [11 01 11 00 01 10].  Find . same as before

𝒚 = [ 11 01 11 00 01 10 ]. 2 0/00 (2) 0/00 (1) 3 2 00 The first part is the same as 0 3 3 10 before. So, we simply copy the diagram that we had. 2 1 01

0 1 11 1/10 (1)

159 Viterbi Decoding: Ex. 2  Suppose = [11 01 11 00 01 10].  Find . same as before

𝒚 = [ 11 01 11 00 01 10 ]. 2 0/00 (2) 0/00 (1) 3 2 0/00 (0) 00 min(2+0,1+2)

0 3 3 10 min(2+2,1+0)

2 1 01 min(3+1,1+1)

0 1 min(3+1,1+1) 11 1/10 (1) 1/10 (1)

160 Viterbi Decoding: Ex. 2  Suppose = [11 01 11 00 01 10].  Find .

𝒚 = [ 11 01 11 00 01 10 ]. 2 0/00 (2) 0/00 (1) 3 2 0/00 (0) 00 min(2+0,1+2) = 2 × 0 3 3 10 × min(2+2,1+0) = 1

2 1 01 × min(3+1,1+1) = 2

0 1 × min(3+1,1+1) = 2 11 1/10 (1) 1/10 (1)

161 Viterbi Decoding: Ex. 2  Suppose = [11 01 11 00 01 10].  Find .

𝒚 = [ 11 01 11 00 01 10 ]. 2 0/00 (2) 0/00 (1) 3 2 0/00 (0) 2 0/00 (1) 00 min(2+1,2+1) = 3

0 3 3 1 10 min(2+1,2+1) = 3

2 1 2 01 min(1+2,2+0) = 2

0 1 2 min(1+0,2+2) = 1 11 1/10 (1) 1/10 (1) 1/10 (2)

162 Viterbi Decoding: Ex. 2  Suppose = [11 01 11 00 01 10].  Find .

𝒚 = [ 11 01 11 00 01 10 ]. 2 0/00 (2) 0/00 (1) 3 2 0/00 (0) 2 0/00 (1) 3 0/00 (1) 00

0 3 3 1 3 10

2 1 2 2 01

0 1 2 1 11 1/10 (1) 1/10 (1) 1/10 (0)

163 Viterbi Decoding: Ex. 2  Suppose = [11 01 11 00 01 10].  Find .

𝒚 = [ 11 01 11 00 01 10 ]. 2 0/00 (2) 0/00 (1) 3 2 0/00 (0) 2 0/00 (1) 3 3 00

0 3 3 1 3 3 10

2 1 2 2 3 01

0 1 2 1 1 11 1/10 (1) 1/10 (1) 1/10 (0)

𝐱 = [11 01 01 00 01 10] 𝐛 = [1 1 0 1 1 1] 164 Viterbi Decoding: Ex. 3  Suppose = [01 10 11 10 00 00].

𝐱 = [11 10 11 00 00 00] 𝐛 = [1 0 0 0 0 0] 165 References: Conv. Codes  Lathi and Ding, Modern Digital and Analog Communication Systems, 2009  [TK5101 L333 2009]  Section 15.6 p. 932-941  Carlson and Crilly, Communication Systems: An Introduction to Signals and Noise in Electrical Communication, 2010  [TK5102.5 C3 2010]  Section 13.3 p. 617-637

166 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 6. Digital Modulation

Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD Elements of digital commu. sys.

Message Transmitter

Information Source Channel Digital Source Encoder Encoder Modulator Transmitted Remove Add Map digital Signal redundancy systematic sequence to (compression) redundancy to analog signal combat errors Channel introduced by the channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder Decoder Demodulator

2 Digital Modulation/Demodulation

Message Transmitter [Chapter 6]

Information Source Channel Digital Source Encoder101001 Encoder Modulator Transmitted Remove Add Map digital Signal redundancy systematic sequence to (compression) redundancy to analog signal combat errors Channel introduced by the channel Noise & Interference Recovered Message Receiver Received Signal Source Channel Digital Destination Decoder101001Decoder Demodulator [Chapter 7]

[Chapter 8] 3 Digital Modulation/Demodulation

Message Transmitter

Information Source Channel Digital Source Encoder101001 Encoder Modulator Digital Modulator Transmitted Index Binary Waveform st w Block Remove Add Map digital Signal

1 0000 st1  redundancy systematic sequence to (compression)t redundancy to analog signal combat errors 2 0001 st2  Channel t introduced by the channel    Received Noise & Interference Recovered Message t Receiver b M  2 1111 stM   Signal Source Channel Digital Destination Decoder101001Decoder Demodulator

4 Standard Quaternary PAM

-3 -1 1 3 @ Tx The points are chosen randomly (according to the bits that are fed into the digital modulator) 100 times.

Add noise 1

@ Rx 0

-1 -3 -1 1 3

7 Review: From Waveforms to Constellation

1 1  What should be the axes?  1 axis = 1 dimension 0 2 t 0 2 3 t

-1 -1  How many do we need?  1 1 How to represent the waveform on these axes? 0 1 2 t 0 2 3 t

-1 -1

1 s (t) 3  A waveform contains infinitely 0.8

s (t) many points. To represent all 0.6 4 (t)] possible waveforms, we would 3  need to work in infinite- [ 0.4 0.2 s (t) dimensional space. 2 0  However, we only have to 1.5 s (t) 1 2 1 consider four possible waveforms 1 0.5 0 here. -1 8 0 -2 [ (t)] [ (t)] 2 1 Review: From Waveforms to Constellation  In Section 6.2, we start with a collection of vectors instead of a collection of waveforms.  The calculation for vector case is easier.  It may seem redundant because some implicit axes are already there when we specify any vector.  Still useful because the calculation can remove the redundant axes.  The formula is exactly the same for the waveform case.  In Section 6.3, the waveform case is studied.

1 1 1 s (t) 3 0.8

s (t) 0 2 t 0 2 3 t 0.6 4 (t)] 3  -1 -1 [ 0.4

0.2 s (t) 2 1 1 0 1.5 s (t) 1 2 1 2 t t 1 0 0 2 3 1 0.5 0 9 -1 -1 -1 0 -2 [ (t)] [ (t)] 2 1 Example 6.26

34 From Waveforms to Constellation

1 1 Find N orthonormal basis functions t t 0 2 0 2 3 𝜙 𝑡,𝜙 𝑡,…,𝜙 𝑡 -1 -1 to use as axes for 𝑠 𝑡,𝑠 𝑡,…,𝑠 𝑡. 1 1

0 1 2 t 0 2 3 t

-1 -1

1 s (t) 3 This gives vector representations for the 0.8 s (t) waveforms 𝑠 𝑡,𝑠 𝑡,…,𝑠 𝑡: 0.6 4 (t)] 3    [ 0.4 ss12,,,   sM  0.2 s (t) 2

which can be visualized in the form of 0 1.5 s (t) 1 2 signal constellation 1 1 0.5 0 -1 35 0 -2 [ (t)] [ (t)] 2 1 Star Constellations

[http://iamintellectuallypromiscuous.com/science/star-right-straight-morning/]

36 [http://68.media.tumblr.com/89eed4669ec511bbb6413acb8da8b0e2/tumblr_mz3qmyQeLH1rhb9f5o1_r1_400.jpg] Standard Quaternary PAM

-3 -1 1 3 

-1 -10 -8 -6 -4 -2 0 2 4 6 8 10 R

37 Standard Quaternary PSK

4 2

2 0 1 r

-2

-4

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5

r 1

38 Standard Quaternary QAM

4 2 3

1 2

r 0 1 -1

-2

-3

-4

-5 -6 -4 -2 0 2 4 6 r 1

41 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 7. The Waveform Channel

Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD The Big Plan  Although we are thinking about digital communication systems,  actual signaling in the wire or air is in continuous time which is described by the waveform channel:

 Directly finding the optimal (MAP) detector or evaluating the performance of such system is difficult.  Our approach is to first construct an equivalent vector channel that preserves the relevant features. (Chapter 7)  Then, at the end, we can use what we learn to go back to the original waveform channel and directly work with the

2 waveforms. Review: ECS315

3 [ECS315 2018] Autocorrelation Function

For random variables, correlation: XY 

expectation operator

[ECS315 2018]

For a random process, autocorrelation function:

RX tt12,    Xt 1 Xt 2 For two random processes, cross-correlation function:

RXY,12tt,    XtYt 1  2 4 Review: ECS315

5 [ECS315 2018] Standard Quaternary PSK

4 2

2 0 1 r

-2

-4

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5

r 1

6 Standard Quaternary QAM

4 2 3

1 2

r 0 1 -1

-2

-3

-4

-5 -6 -4 -2 0 2 4 6 r 1

7 Modulator and Waveform Channel Goal: Want to transmit the message (index) W  {1, 2, 3, …, M}

Prior Probabilities: p j  PW  j M-ary Scheme Waveform Channel: M =2: Binary St  Rt  M = 3: Ternary W Digital Digital ˆ Modulator DeModulator W M = 4: Quaternary Nt 

M possible messages requires RtStNt     Additive White Noise M possibilities for S(t): (Independent of S(t)) st,,, st s t Transmitted waveform  12    M   Received waveform Transmission of the message 𝑊𝑗is done by inputting the corresponding waveform 𝑠 𝑡 into the channel.   Prior Probabilities: p jjPW j PSt   s t M Energy: E  st ,st  E p E log M E 8 j jj s  jj 2 b j1 Conversion to Vector Channels

Waveform Channel: RtStNt     Find K orthonormal basis functions 𝜙 𝑡,𝜙 𝑡,…,𝜙 𝑡 for the space spanned by    𝑠 𝑡,𝑠 𝑡,…,𝑠 𝑡. Vector Channel RSN  This gives vector representations for the ⇀ 𝑆 , th 𝐒 waveforms 𝑠 𝑡,𝑠 𝑡,…,𝑠 𝑡: Note that the i component of the vector,   comes from the inner-product: ss12,,,   sM   j which can be visualized in the form of SSttii  ,   ⇀ signal constellation The received vector 𝑅 is computed in the same way: the j component is given by Prior Probabilities: pPWjPStst  Rii rt ,  t jj     ⇀   In which case, the corresponding noise vector 𝐍 is P Ssj computed in the same way: the j component is given by  

NNttii  ,   For additive white Gaussian noise (AWGN) process N(t),

 2 1 n   NN2     1 2 00 2  NNi 0, 0, N0    , I   fN  n K e 9 22K    22

 Digital Communication Systems ECS 452

Asst. Prof. Dr. Prapun Suksompong [email protected] 8. Optimal Detection for Additive Noise Channels 1-D Case

Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD Elements of digital commu. sys.

Message Transmitter

2 Digital Modulation/Demodulation

Message Transmitter [Chapter 6]

[Chapter 8] 3 Digital Modem: Ex 1

Message Transmitter

Information Source Channel Digital Source Encoder101001 Encoder Modulator Digital Modulator Transmitted Index Binary Waveform st w Block Remove Add Signal

1 0000 st1   redundancy systematic (compression)t redundancy to combat errors 2 0001 st2   Channel t introduced by the channel    Received Noise & Interference Recovered Message t Receiver b M  2 1111 stM   Signal Source Channel Digital Destination Decoder101001Decoder Demodulator

4 Digital Modem: Ex 2

Message Transmitter

Information Source Channel Digital Source Encoder101001 Encoder Modulator Transmitted PAM Example: M = 4 Remove Add Signal Index Binary Amplitude Waveform s t m b A m Block m redundancy systematic 0 Ts s t t 1 00 ‐3 1   (compression) redundancy to combat errors Ts s t 0 t Channel 2 01 ‐1 2 introduced by

3 10 1 s3 t t the channel 0 Ts Noise & Interference s t M  4 11 3 4  t Received 0 Recovered Message Ts Receiver Signal Source Channel Digital Destination Decoder101001Decoder Demodulator

5 Analysis of Digital Modem

Transmitted waveform Received waveform

Digital Digital Modulator Demodulator M possibilities. For example, if 𝒃 has 4 Digital Modulator Index Binary 4 Waveform st bits, then there are 2 w Block

= 16 possibilities. 1 0000 s t 1 t

2 0001 s2 t M-ary Scheme t Additive White Noise

M =2: Binary    (Independent of S(t)) M = 3: Ternary t b M  2 1111 sM t M = 4: Quaternary

M possible messages requires M possibilities for S(t):

st12 ,,, st   sM  t 6 Analysis of Digital Modem

𝑠 𝑡,𝑠 𝑡,…,𝑠 𝑡

Waveform Channel: 𝑅 𝑡 𝑆 𝑡 𝑁 𝑡 Find orthonormal basis (possibly by GSOP): 𝜙 𝑡,𝜙 𝑡,…,𝜙 𝑡 𝑆 𝑡,𝜙 𝑡 𝑆 𝑡,𝜙 𝑡 ⋮ Ex: 𝑆 𝑡,𝜙 𝑡 𝐾1: PAM, ASK 𝑁 𝑡,𝜙 𝑡 𝑅 𝑡 ,𝜙 𝑡 𝑁 𝑡,𝜙 𝑡 𝐾2: PSK, QAM 𝑅 𝑡 ,𝜙 𝑡 ⋮ ⋮ 𝑁 𝑡,𝜙 𝑡 𝑅 𝑡 ,𝜙 𝑡 ⇀ ⇀ ⇀ Vector Channel: 𝐑 𝐒 𝐍

⇀ ⇀ ⇀ 𝐑 , 𝐒 and 𝐍 are all  ⇀𝐬 ,𝐬⇀ ,…,⇀ 𝐬 random vectors. Each vector contain K can be visualized in the form of signal constellation random variables. 7 Analysis of Digital Modem: 1D

𝑅 𝑆 𝑁 Continuous RV Discretes RV st1   st2   st3   stM    t 𝑠 𝑠 𝑠 𝑠 p P St s t PS s j p s j Prior Probabilities: j   jS      2 2 j pmf of the “message” Energy: E j stjjj stst,   s M Average Energy (per Symbol): Es   p j E j j1 1 Average Energy (per Bit): Eb  Es log2 M

8 Analysis of Digital Modem: MAP  Model: 𝑆 𝑅  We know the pmf of S and the pdf of N. 𝑁  We assume that S and N are independent.  Goal: Use the (observed) value of R to infer back to the value of S that was transmitted.  Note that once we recover the value of S, then we can map this back to the corresponding waveform , and consequently, recover the corresponding bit block .

9 [ECS452 2018 Section 3.3 p. 44-45] Review: MAP decoder

10 [ECS452 2018 Section 3.3 p. 45] Review: MAP decoder

11 Analysis of Digital Modem: MAP  Model: 𝑆 𝑅  We know the pmf of S and the pdf of N. 𝑁  We assume that S and N are independent.  Suppose, at the receiver, we get .  Optimal (MAP) Detector: ∈ , ,…,

12 Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 pf2 N rs  0.2

0.15 1 pf1 N rs 

0.1 Area  P  0.05 2 s1 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 13  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 pf2 N rs  0.2

0.15 1 pf1 N rs 

0.1

0.05 2 s1 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 14  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 pf2 N rs  0.2

0.15 1 pf1 N rs 

0.1

0.05 Area  p1 2 s1 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 15  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 pf2 N rs  0.2

0.15 1 pf1 N rs 

0.1

0.05 p 1A A 2 11s 1 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 16  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 pf2 N rs  0.2

0.15 1 pf1 N rs 

0.1

Area  p2 0.05 2 s1 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 17  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 pf2 N rs  0.2

0.15 1 pf1 N rs 

0.1

0.05 1 p  A2 s A2 22s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 18  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 PpApA      pf2 N rs  11 2 2

0.2 1 AA12  PP1  0.15 pf rs 1 1 N   AA12 0.1

0.05 p  A p  A2 11s1 22s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 19  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 PpApA      pf2 N rs  11 2 2

0.2 1 AA12  PP1  0.15 pf rs 1 1 N   AA12 0.1

0.05 2 1 A1 s A2 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 20  MAP Error Probability Ex. Binary PAM under “Triangular” Noise

0.35

0.3

0.25 2 P  pA  pA  pf rs   11 2 2 0.4,0.105 2 N    1 AA12  0.2   PP1 0.15 1 AA pf1 N rs  12 1 0.1 20.105 2

0.05  0.105 2 1 A1 s A2 s 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 r 21  MAP Similar Triangles

ℎ Similar Triangles ℎ 𝑦  𝑑 𝑑 

𝑏