Today’s Topics Information Theory • Communication Channel • Noiseless binary channel Mohamed Hamada • Binary Symmetric Channel (BSC) Software Engineering Lab The University of Aizu • Symmetric Channel Email: [email protected] URL: http://www.u-aizu.ac.jp/~hamada • Mutual Information • Channel Capacity 1 Digital Communication Systems Digital Communication Systems 1. Huffman Code． 1. Memoryless 2. Two-pass Huffman Code． 2. Stochastic 3. Lemple-Ziv Code． 3. Markov 4. Fano code． Information User of Information 4. Ergodic User of Source 5. Shannon Code． Information Source Information 6. Arithmetic Code． Source Source Source Source Encoder Decoder Encoder Decoder Channel Channel Channel Channel Encoder Decoder Encoder Decoder Modulator De-Modulator Modulator De-Modulator Channel Channel 2 3 INFORMATION TRANSFER ACROSS CHANNELS Communication Channel Sent Received A (discrete ) channel is a system consisting of an input alphabet messages messages X and output alphabet Y and a probability transition matrix symbols p(y|x) that expresses the probability of observing the output symbol y given that we send the symbol x Channel Channel Source Source Channel sourcecoding coding decoding decoding receiver Examples of channels: Compression Error Correction Decompression CDs, CD – ROMs, DVDs, phones, Source Entropy Channel Capacity Ethernet, Video cassettes etc. Rate vs Distortion Capacity vs Efficiency 4 5 1 Communication Channel Noiseless binary channel Noiseless binary channel Channel Channel 00 input x p(y|x) output y Transition probabilities 11 Memoryless: - output only on input Transition Matrix - input and output alphabet finite 01 p(y | x) = 0 10 1 01 6 7 Binary Symmetric Channel (BSC) Binary Symmetric Channel (BSC) (Noisy channel) (Noisy channel) 00BSC Channel 1 p 1-p 1-p Error Source 00 e BSC Channel p 110 y = x e p 1-p xi i i + 11 Input Output p 1-p 00 1-p 1 1 p BSC Channel 8 9 Symmetric Channel (Noisy channel) Channel XY In the transmission matrix of this channel , all the rows are permutations of each other and so the columns. Mutual Information Example: Transition Matrix y1 y2 y3 x1 0.3 0.2 0.5 p(y | x) = x2 0.5 0.3 0.2 x3 0.2 0.5 0.3 10 11 2 Mutual Information (MI) Mutual Information (MI) Channel Relationship between Entropy and Mutual Information X p(y|x) Y I(X,Y)=H(Y)-H(Y|X) Proof: MI is a measure of the amount of information that one p(y | x) I(X ,Y) p(x, y)log2 ( ) random variable contains about another random variable. xy p(y) It is the reduction in the uncertainty of one random I (X ,Y ) p(x, y) log 2 ( p( y)) p(x, y) log 2 ( p( y | x)) variable due to the knowledge of the other. xy xy For the two random variable X and Y with a joint I(X,Y) p(y)log (p(y)) p(x) p(y| x)log (p(y| x)) probability p(x, y) and marginal probabilities p(x) and p(y). 2 2 y xy The mutual information I(X,Y) is the relative entropy between = H(Y) – H(Y|X) Note that I(X,Y) 0 the joint distribution and the product distribution p(x) p(y) i.e. p(x, y) i.e. the reduction in the description length of X given Y I(X ,Y) p(x, y)log2 ( ) p(x) p(y) 12 or: the amount of information that Y gives about X 13 Mutual Information (MI) Mutual Information with 2 channels Note That: • I(X,Y)=H(Y)-H(Y|X) Channel 1 Channel 2 YZ • I(X,Y)=H(X)-H(X|Y) X p(y|x) p(z|y) • I(X,Y)=H(X)+H(Y)-H(X,Y) Let X, Y and Z form a Markov chain: X Y Z • I(X,Y)=I(Y,X) and Z is independent from X given Y • I(X,X)=H(X) i.e. p(x,y,z) = p(x) p(y|x) p(z|y) The amount of information that Y gives about X given Z is: I(X, Y|Z) = H(X|Z) – H(X|YZ) H(X|Y) I(X,Y) H(Y|X) H(X) H(Y) 14 15 Mutual Information with 2 channels Theory: I(X,Y) I(X, Z) (I.e. Multi-channels may destroy information) Proof: I(X, (Y,Z) ) =H(Y,Z) - H(Y,Z|X) =H(Y) + H(Z|Y) - H(Y|X) - H(Z|YX) = I(X, Y) + I(X; Z|Y) Channel Capacity I(X, (Y,Z) ) = H(X) - H(X|YZ) = H(X) - H(X|Z) + H(X|Z) - H(X|YZ) = I(X, Z) + I(X,Y|Z) now I(X, Z|Y) = 0 (independency) Thus: I(X, Y) I(X, Z) 16 17 3 Transmission efficiency Channel Capacity The channel capacity C is the highest rate, in bits per channel I need on the average H(X) bits/source output to describe the use, at which information can be sent with arbitrarily low source symbols X After observing Y, I need H(X|Y) bits/source probability of error output. C = Max I(X , Y) Channel p(x) H(X) XYH(X|Y) Where the maximum is taken over all possible input distributions p(x) Note that : Reduction in description length is called the transmitted • During data compression, we remove all redundancy in the Information. data to form the most compressed version possible. Transmitted R = I(X, Y) = H(X) - H(X|Y) • During data transmission, we add redundancy in a controlled = H(Y) – H(Y|X) from earlier calculations fashion to combat errors is the channel We can maximize R by changing the input probabilities. Capacity depends on input probabilities The maximum is called CAPACITY 18 because the transition probabilites are fixed 19 Channel Capacity Channel Capacity Example 1: Noiseless binary channel Example 2: (Noisy) Binary symmetric channel Channel If P is the probability of an error ( noise ) Probability Transition Matrix p 1/2 00 00 01 1-p Transition Matrix p(y|x) = 0 10 1 1 p 01 1 01 BSC Channel p(y|x)= 0 p1-p 1/2 11 Here the mutual information 1 1-p p I(X,Y) = H(Y) - H(Y | X) = H(Y) - ∑ p(x) H(Y | X=x) Here the entropy is: x = H(Y) - ∑ p(x) H(P) = H(Y) - H(P) ∑ p(x) (Note that ∑p(x) = 1) x x x H(X) = H(p, 1-p)= H(1/2,1/2) = -1/2 log 1/2 - 1/2 log 1/2 = 1 bit = H(Y) - H(P) (Note that H(Y) ≤ 1 ) ≤ 1 - H(P) I(X,Y) ≤ 1 – H(P) H(X|Y) = -1 log 1 - 0 log 0 - 0 log 0 - 1 log 1 = 0 bit Hence the information capacity of the BSC channel is I(X, Y) = H(X) - H(X | Y) = 1 - 0 = 1 bit 20 C = 1 - H (P) bits (maximum information ) 21 Channel Capacity Channel Capacity Channel Example 3: (Noisy) Symmetric channel XY(Noisy) Symmetric channel The capacity of such channel can be found as follows: Channel let r be a row of the transition matrix =1 I(X, Y) = H(Y) - H(Y | X) = H (Y) - ∑ p(x) H(Y | X = x) x XY = H(Y) - ∑ p(x) H(r) = H(Y) - H(r) ∑ p(x) = H(Y) - H(r) x x ≤ log |Y | - H (r) Y is the output alphabet Hence the capacity C of such channel is: C = max I(X; Y) = log |Y | - H (r) Note that: In general for symmetric channel the capacity C is For example: Consider a symmetric channel with the following transmission matrix It is clear that such channel is symmetric y1 y2 y3 C = log |Y | - H (r) since rows (and columns) are permutations. x1 0.3 0.2 0.5 x Hence the capacity C of this channel is: P (y|x) = 2 0.5 0.3 0.2 Where Y is the output alphabet and r is any row in the x3 0.2 0.5 0.3 transmission matrix. C = log 3 - H(0.2, 0.3, 0.5) = 0.1 bits (approx.) Transition Matrix 23 22 4.