Introduction Why Coding Theory ? Introduction Info Info Noise ! Info »?# Sink to Source Heh ! Sink Heh ! Coding Theory Encoder Channel Decoder Samuel J. Lomonaco, Jr. Dept. of Comp. Sci. & Electrical Engineering Noisy Channels University of Maryland Baltimore County Baltimore, MD 21250 • Computer communication line Email: [email protected] WebPage: http://www.csee.umbc.edu/~lomonaco • Computer memory • Space channel • Telephone communication • Teacher/student channel
Error Detecting Codes Applied to Memories
Redundancy Input Encode Storage Detect Output
1 1 1 1 0 0 0 0 1 1 Error 1 Erasure 1 1 31=16+5 Code 1 0 ? 0 0 0 0 EVN THOUG LTTRS AR MSSNG 1 16 Info Bits 1 1 1 0 5 Red. Bits 0 0 0 0 0 0 0 FRM TH WRDS N THS SNTNCE 1 1 1 1 0 0 0 0 1 1 1 Double 1 IT CN B NDRSTD 0 0 0 0 1 Error 1 Error 1 Erasure 1 1 Detecting 1 0 ? 0 0 0 0 1 1 1 1 Error Control Coding 0 0 1 1 0 0 31% Redundancy 1 1 1 1
Error Correcting Codes Applied to Memories
Input Encode Storage Correct Output Space Channel
1 1 1 1 0 0 0 0 Mariner Space Probe – Years B.C. ( ≤ 1964) 1 1 Error 1 1 • 1 31=16+5 Code 1 0 1 0 0 0 0 Eb/N0 Pe 16 Info Bits 1 1 1 1 -3 0 5 Red. Bits 0 0 0 6.8 db 10 0 0 0 0 -5 1 1 1 1 9.8 db 10 0 0 0 0 1 1 1 Single 1 0 0 0 0 Error 1 1 • Mariner Space Probe – Years A.C. 1 Error 1 • 1 Correcting 1 1 1 0 0 0 0 Eb/N0 Pe 1 1 1 1 0 0 -1.6 db Essentially Zero 1 1 0 0 31% Redundancy 1 1 1 1 1 db = $1,000,000
1 Types of Channels 2-Repeat Code 1 − p 0 0 Info. Words Code Words p p 0 00 1 11 1 1 − p 1 Binary Symmetric Channel (BSC) Detects all single errors
If we use BEC with probability of transition p = 10-2, Erasure then the probability P of undetectable error is 1 − p then the probability PU of undetectable error is 00 ==24− p PpU 10 p ? Moreover, 1 1 #1InfoBits 1 − p Rate== R = Binary Erasure Channel (BEC) #2All Bits
A Surprise in Communication Theory Shannon’s Theorem 1 −2 Repetition 10 Coding 10−4 Within a large class of coding schemes −6 there exist some schemes – nearly all, 10 Limit Given −8 10 by Shannon’s actually – that give arbitrarily low error − Theorem rates at any information rate up to a 10 10 critical rate C, called channel capacity. 10−12 Prob. Of Msg. Error Msg. Of Prob. Prob. Of Msg. Error Msg. Of Prob. 10−14 ε 1 1 1 1 1 1 Folk Theorem Channel Choose as 3 5 7 9 11 small as “All codes are good, except those we can think of.” Capacity small as Info Bits you like Msg Rate = C All Bits
Hamming (8,4) 4 Code Info Words Code Words Types of Codes 0000 0000 0000 • Corrects all single errors 0001 1101 0001 0010 0111 0010 • A block code is a code that uses sequences • Detects all single, double, 0011 1010 0011 of n channel symbols, or n-tuples. and triple errors 0100 1011 0100 0101 0110 0101 Rate R = 1/2 • 0110 1100 0110 Only certain selected n-tuples, called 0111 0001 0111 We will now apply this code code blocks or code words are sent. 1000 1110 1000 code to the BEC with p=10-2 1001 0011 1001 <−≈−88844kk− 1010 1001 1010 Convolutional Codes: PUkC pp(1 )C4 pp (1 ) 1011 0100 1011 • ∑ Each output bit depends on all the previous k −−44 1100 0101 1100 ∴<()()22 − bits. PU 70 10 1 10 1101 1000 1101 1110 0010 1110 − ∴< ×7 1111 1111 1111 PU 6.72 10
2 Decoding Table for a Block Code Terminology Codewords 11000 00110 10011 01101 Code 11001 00111 10010 01100 11010 00100 10001 01111 00 11000 11100 00010 10111 01001 01 00110 10000 01110 11011 00101 10 10011 01000 10110 00011 11101 Received 11110 00000 01011 10101 11 01101 Words 01010 10100 11111 00001
Info Encoding Code Words Procedure Words
Convolutional Codes Types of Error Correcting Codes
Infinite Infinite Block Codes Input Output • Orthogonal Codes • Linear Codes aann+1210 aaa Convolver bbnn+1210 bbb • Cyclic Codes • BCH Codes
Convolutional Codes • Threshold Decoding • Sequential Decoding
Block Codes Definitions Def. A code is a set of binary vectors of the same fixed length Non-Linear
Parameters of codes Linear n = length of code vectors
Cyclic R = code rate = (# info bits)/n
PU = Prob. of undetectable error BCH
3 Problem: PU depends on the channel Problem: PU depends on the channel (Cont.)
Def. Let u and v be n bit vectors. Def. The Hamming weight H(u) of u is The Hamming distance H(u,v) between the number of 1’s in u. u and v is the number of bits at which they differ. For example, H ()0110= 2 For example,
H ()0110, 1011= 3 Please also note that H ()10011, 00011= 1 Huv(), =+ Hu ( v )
Problem: PU depends on the channel (Cont.) A Recurring Theme
Def. Let V be a code. Then the minimum distance d(V) is uv≠ Add More Algebra dV()= MinH() uv , | uv, ∈ V Observation: & Gain ≤−++−nd nd−− nn nn PCpUd(1 p ) Cp n (1 p ) • A trade of space for time, i.e., memory So channel independent Code Parameters are: for computation n = length of code vectors • Simplifications R = code rate = (# info bits)/n d = minimum distance
Linear Codes Linear Codes GF ()20,1,,=+= { } i Galois field of two elts. + 0 1 i 0 1 Enter stage right 0 0 1 0 0 0 … Addition 1 1 0 1 0 1 nn=={() ∈∀} E GF(2) b12 , b ,… , bni : b GF (2) i We now adjoin addition “+” to the = n-dim vector space over GF(2) code space E n ,,+ i ()()(+=+++ ) bb12, ,…… , bnn b 1 ', b 2 ', , b ' b 1 b 1 ', b 2 b 2 ', … , b nn b ' ()()=∈ a b12,,, b…… bnn ab 1 , ab 2 ,, ab , a GF (2)
4 Linear Codes A Simplification Def. A linear code V is a subspace of E, i.e., V is linear iff uv, ∈ V⇒ u+∈ v V Min. Dist. = Min. Non-Zero Wt.
Parameters of linear codes: NOTE: R=k/n uv, ∈ V n = length of code vectors dV()= MinHuv (,): ≠ k = dim(V) = log2(#V) = # Info. Bits uv d = d(V) = minimum distance uV∈ = Min H(): u u ≠ 0 V is said to be a linear (n,k) d code
Length Dim Min. Dist.
Error Model An example of the Error Model
Error Error Vector Vector ε = 000010 ε Sent Received Vector Vector Sent Received Vector Vector v =001110 rv=+ε v rv=+ε =001100 000010 = ε + 001110 = v 001100 = r
An Example: The Hamming (7,4) 3 Code The Hamming (7,4) 3 Code (Cont.) Infoword Codeword ()(bbbb,,, bbbbbbb ,,,,,, ) 1011000 3210 6543210 Infoword Codeword bbbb=++ 1110100 6023=++ ()bbbb,,,= ( bbbbbbb ,,,,,, ) where bbbb5012 3210 6543210 bbbb=++ 0110010 4123 Infoword Codeword Infoword Codeword 1110001 0000 000 0000 1000 110 1000 0001 101 0001 A 1001 011 1001 0010 111 0010 1010 001 1010 Generator Matrix G 0011 010 0011 Linear 1011 100 1011 74 0100 011 0100 Code 1100 101 1100 VvEuEstuGv=∈{ :.. ∃∈ =} 0101 110 0101 1101 000 1101 0110 100 0110 1110 010 1110 0111 001 0111 1111 111 1111 The rows of G span the linear code V.
5 The Hamming (7,4) 3 Code (Cont.) Linear Codes
T Codeword 1001101 • The rows of the generator matrix G span ()bbbbbbb,,,,,, 0100111=() 0,0,0 the code V. 6543210 0011110 The columns of the parity check matrix H • The columns of the parity⊥ check matrix H span the dual code V . Parity Check Matrix HT VvEvH=∈{ 7 :0T =}
⊥ The rows of HT span the linear code V .
Linear Codes Linear Codes
Def. Let V be a linear code in En. Then Observation: vV∈⇔ vHT =0 the dual code V⊥ of V is Hence, ()veHvHeHeH+=+=TTTT VvEvuuV⊥ =∈{ n :0,i =∀∈} where Def. Let r be a received vector. Then vui = vu T ∑ ii rHi i is called the syndrome of r .
The syndrome depends only on the error pattern e.
Maximum Likelihood Decoding Maximum Likelihood Decoding Coset Leaders Decoding Error Patterns Code Table Most Probable Error Pattern for Given Syndrome 000000 001110 010101 100011 011011 101101 110110 11100 000001 001111 010100 100010 011010 101100 110111 111001 Error Syndrome 000010 001100 010111 100001 011001 101111 110100 111010 000000 000 000100 001010 010001 100111 011111 101001 110010 111100 000001 001 001000 000110 011101 101011 010011 100101 111110 110000 Error/Syndrome 000010 010 010000 011110 000101 110011 001011 111101 100110 101000 Table 000100 100 100000 101110 110101 000011 111011 001101 010110 011000 001000 110 001001 000111 011100 101010 010010 100100 111111 110001 010000 101 100000 011 Standard Array for a Linear (6,3) 3 Code 001001 111
6 A Recurring Theme Cyclic Codes
Add More Algebra Enter stage right & Gain … Multiplication
• • A trade of space for time, i.e., memory We now adjoin Multiplication “ ” for computation to the code space • Simplifications
Cyclic Codes: Preliminaries Cyclic Codes: Preliminaries
Each n-bit binary number can be considered to be a polynomial with coefficients over GF(2) . 23 1011 1 ++x x + 1101 ++1 x +x3 10101011=+++++++ 1iiiiiiiix01234567 0xxxxxxx 1 0 1 0 1 1 0110 00++xx2 + =++++1 x2467xxx Addition
Cyclic Codes: Preliminaries Cyclic Codes: Preliminaries
101 2 1 + x Problem: Multiplication of code vectors × 011 ×+xx2 (thought of as polynomials) may increase 101 code vector length. xx+ 3 101 Example: xx24+ 000 22 +++234 ()()101ii 011=+()() 1 x xx + 01111 xx x x =+++=xx234 x x 01111 Multiplication Both Length 3 Length 5
7 Cyclic Codes: Preliminaries Cyclic Codes: Preliminaries
A quick way to fix the problem: Hence, under this identification, the Assume xn=1 or xn+1=0, where n linear space denotes the codeword length. n +={() ∈} + EbbbbGF,,,,:(2),01… ni− 1 Example: n=3 Therefore, x3=1 ∴=x4 x, ∴=x52x , ∴=xx63 =1 becomes a RING
234 2 ∴01111 =+++=+++x xxxxx 1 x GF(2)[ x ] n−1 RbbxbxbGF=+=+++∈+,,i {}− : (2),, i =+1111x +xx22 + =+() + xx + nni()1+xn 01 1 =+1x2 = 101
Cyclic Codes Cyclic Codes Def. A linear code V in En is a cyclic code if
()()vv,, , v−−−∈ V⇒ v ,,, vv , v∈ V Proposition. A cyclic code V is an ideal in 01nnn 1 101 2 n the ring GF (2)[ x ]/ () 1 + x , i.e.,
()vv,, , v Identify 01n− 1 with the polynomial 1) vv,'∈ V⇒ v−∈ v ' V =+ +21 ++ n− vx() v01 vx vx 2 vn− 1 x Then a cyclic shift is ∈∈n ∈ vx() i xvx () 2) uEvV, ⇒ uvVi
in the ring E nn=+GF(2)[ x ]/() 1 x
Generator Polynomial of a Cyclic Code Generator Polynomial of a Cyclic Code
But En is a principal ideal domain. Proposition. k = dim(V) = n - deg(g) Hence, for every cyclic code V, there exists a polynomial g(x) such that n Proposition. The generator matrix G of V ==()g()xuxvxuxE{ ()():() ∈} a cyclic code can be written in the form g()x We choose g(x) so that it is a factor of 1+xn . Such a choice is unique. xg() x G = The polynomial g(x) is called a generator k−1 polynomial of the cyclic code V . xg()x
8 An Example Encoding and Decoding Procedures for Cyclic Codes Consider the cyclic code V given by the generator polynomial g(x) = 1+x+x3
Let n be the smallest positive integer s.t. Encoding Procedure = Multiplication g(x) g(x) (1+xn)
ix() ix ()()g x Then n = 7 and dim(V) = k = n – deg(g) = 7 – 3 = 4 This is the Hamming (7,4) 3 code. Decoding Procedure = Division by g(x) If i(x) = 1+x, then the encoded vector is rx() rx ()/()g x ()11+x () ++xx3324 =++ 1 xx ++ xx + x =+++1 x234xx ∴1100 1011100
Another Recurring Theme Encoding Circuit = Circuit which Multiplies by g(x)
Infoword 1011100 2 3 1100 1 x x2 x3 Codeword A circuit which multiplies by 1+x+x3
Output Algebra = Computing Input State Algebra = Computing 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 1 0 1 0 0 1 0 1 1 1 0 0 0 1 0 1 1 1 0 0 0 0 1 0 1 1 1 0 0 0 0 0 1 0 1 1 1 0 0
Decoding Circuit = Circuit that Divides by General Encoding Circuit which Multiplies g(x) = 1+x+x3 by an Arbitrary Generator Polynomial g(x)
Output Input ix()()g x … Input State 3 Initial State g0 g g … g g g1 g2 gn-1 gn xg()x = 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 1 1 0 0 ix() … 0 0 0 1 0 0 1 0 Input 0 0 0 1 0 0 1 Syndrome 0 0 0 0 1 0 Linear Circuit for multiplying by 0 0 0 0 1 2 n-1 n 0 1 1 0 g(x)=g0+g1x+g2x + … +gn-1x +gnx 2 0 1 1 ==+S()xxx
9 General Encoding Circuit which Divides by An Example an Arbitrary Generator Polynomial g(x)
Output ++3 ix()/()g x ix()/1() x x … Output -1 - g0 - g1 - g2 - gn-1 gn … ix() ix() … Input Input
Linear Circuit for dividing by Linear Circuit for dividing by 2 n-1 n 3 g(x)=g0+g1x+g2x + … +gn-1x +gnx g(x)=1+x+x
A Recurring Theme BCH Codes
Add More Algebra Enter stage right & Gain … More Algebraic Structure
• A trade of space for time, i.e., memory for computation BCH = Bose-Chaudhuri-Hocquenghem • Simplifications
Galois Fields (Characteristic 2) Galois Fields (Characteristic 2)
Simplified Approach GF(2k) = GF[2]/(p(x)) , where GF(2k) = polynomials in ξ subject to the relation p(ξ) = 0 . 1) deg( p(x) ) = k 2) p(x) is irreducible 3 3) p(x) is primitive, i.e., the residue Example: Construction of GF(2 ) 3 3 class ξ containing x generates the Let p(x) = 1+ξ+ξ ; hence ξ = 1+ξ . k multiplicative group of GF(2 ). Therefore, every polynomial in ξ reduces to one of the form 2 a0+a1x+a2x ,
where ai e GF(2) for i = 0,1,2
10 Example (Cont.) Example (Cont.)
GF(23) 0 = 0 = 000 We can create other Galois fields using the following “relations” ξ0 = 1 = 100 ξ1 = ξ = 010 GF(22) with the relation 1+ξ+ξ2 = 0 ξ2 = ξ2 = 001 GF(2 ) with the relation 3 ξ4 = 1+ξ = 110 GF(23) “ “ “ 1+ξ+ξ = 0 4 ξ5 = ξ +ξ2 = 011 GF(24) “ “ “ 1+ξ+ξ = 0 ξ6 = 1+ξ+ξ2 = 111 GF(25) “ “ “ 1+ξ2+ξ5 = 0 7 2 ξ = 1 +ξ = 101 GF(26) “ “ “ 1+ξ+ξ6 = 0 ξ ξ2 where a0 + a1 + a2 a0 a1 a2
Another way to describe cyclic linear codes BCH Codes Another way to describe a cyclic code, i.e., in terms of the roots of the generator polynomial g(x) . Def. Let ξ be a primitive root of GF(2m). A V = ( g(x) ) = { h(x) : h(α) = 0 for all roots of g(x) } cyclic linear code V generated by a polynomial g(x) is a BCH code with design parameter δ if g(x) is the polynomial of smallest degree over Example. Let V = Hamming (7,4) 3 linear code. GF(2) having Then g(x) = 1+x+x3, and the roots of g(x) are ξ, ξ2, ξ4 in GF(23). Hence, the linear code is ξ, ξ2,ξ3, … , ξδ-1 V ={ h(x) : h(ξ) = h(ξ2) = h(ξ4) =0 } as roots. Moreover, the syndrome is given by r(ξ) = h(ξ)+e(ξ) = e(ξ)
An Example BCH Codes (Cont.) Let ξ be a primitive root of GF(24) , where p(x) = 1+x3+x4 δ Id = 2t0+1. Then the BCH code can Let V be the cyclic linear code consisting of all correct t0 errors, and detect 2t0 errors. vectors h(x) having the roots ξ, ξ2, ξ3, ξ4, ξ5, ξ7−1 Such a BCH code is a cyclic linear (2m-1,k) d Then ξ , ξ2, ξ4 , ξ8 code, where ξ , ξ , ξ , ξ m1(x)=m2(x)=m4(x) (deg 4) 3 6 12 9 ξ3, ξ6, ξ12, ξ9 m (x)=m (x) (deg 4) kmt≥−−21m m3(x)=m6(x) (deg 4) 0 5 10 ξ , ξ m5(x) (deg 2) d ≥ δ So g(x) = m1(x)m3(x)m5(x) ; and hence of deg 10
Therefore, (2m-1,k) d = (15,5) d, where d ≥ 7
11 TheThe EndEnd
12