INFORMATION THEORY in the Benelux and FIRE CONTROL

Han Vinck Univ. Duisburg‐Essen Univ. Johannesburg, South Africa IT starts with Shannon

The book co‐authored with Warren Weaver, The Mathematical Theory of Communication,reprintsShannon's1948 article and Weaver's popularization of it, which is accessible to the non‐specialist.[5] In short, Weaver reprinted Shannon's two‐part paper, wrote a 28 page introduction for a Weaver changed the title from“ "transformed cryptography 144 pages book, and changed the title from Amathematicaltheory...“ from an art to a science." "A mathematical theory..." to "The to "The mathematical theory..." mathematical theory..." BUT ...

WIC meeting Gent May 2019 2 BEFORE ... (1946)

“there is an obvious analogy between the problem of smoothing the data to eliminate or reduce the effect of tracking errors and the problem of separating a signal from interfering noise in communications systems.”

WIC meeting Gent May 2019 3 RVO –TNO (Rijks Verdedigings Organisatie RVO ‐ Nederlandse Organisatie voor Toegepast‐ Natuurwetenschappelijk Onderzoek TNO.

Ir. J.l. van Soest: Director RVO‐TNO 1927 – 1957

Extra ordinary Professor TU Delft Information and Communication Theory 1949 ‐ 1964

Task: NEW Research Directions. From mechanical (pre‐war) via analogue to digital methods for fire control

Prof. van Soest, started (1955) a group on Information theory (Topics: game theory, cryptography and correlators

WIC meeting Gent May 2019 4 But also this: hearing aids

WIC meeting Gent May 2019 5 #1 Prof. IJ. Boxma successor of van Soest(director) in 1957:

• After 1947: Digital Fire‐control development by ir. IJ. Boxma in the group Electronic computing, later Information Processing Systems

(Militaire Spectator, 1958) 1950, Head Engineer: E.W. Gröneveld

1970 Ordinary Prof in Information and Communication theory at TU Delft. WIC meeting Gent May 2019 6 #2 Willem Gröneveld, successor Boxma in 1957:

(Militaire Spectator, 1960)

1965 Ordinary Professor in InformationWIC and meeting Communication Gent May 2019 theory at TU Twente 7 #3: Piet (J.P.M.) Schalkwijk

1959‐1961 HSA Hengelo, 1st lieutenant Dutch armed forces Project digital fire control system DIPHYSA developed together with HSA Hengelo. 1961‐1963 RVO‐TNO (Rijks verdedigings organisatie)

1972 Ordinary Professor in InformationWIC theory meeting Gent at May TU2019 Eindhoven 8 Piet Schalkwijk

1965! Kailath’s first Ph.D student, is said to have complained that he was made to rewrite his thesis 6 times!

In response Kailath (Stanford) says: “That’s an exaggeration. After all he graduated in two years after his Master’s degree. Almost all my students graduated within 3‐4 years from their Master’s degrees.”

Best IEEE IT paper 1970

WIC meeting Gent May 2019 9 All chairs in information theory in the Netherlands are occupied in 1972

•Prof.ir. IJ. Boxma, Delft University of Technology (deceased) •Prof.ir. E. Willem Gröneveld, University of Twente (deceased) WIC founding fathers1986 •Prof.dr. Edward C. van der Meulen, KU Leuven •Prof.dr.ir. J.Pieter.M. Schalkwijk, TU Eindhoven

Edward van der Meulen, 1976, KUL

IT TOPICS COVERED: Data Processing (Boxma, Delft), Coding techniques (Schalkwijk, Eindhoven, Estimation and detection (Gröneveld, Twente) Algebraic Coding theory (Van Lint, Eindhoven and Goethals, MBLE Brussels), Multi User theory (vd Meulen, Leuven), WIC meeting Gent May 2019 10 Thanks Ludo! ISIT Sorento,

ISIT Nice

WIC meeting Gent1st May Benelux 2019 Japan workshop, 1989 (now AEW, 2019 R‘dam) 11 Next topic: memory with restrictions;. Some examples

‐ limit the number of changes to pn (Phase Change)

‐ memory with defects or permanent error

‐ memory with periodic writing direction: MO disk

‐ memory with erase phase: flash memory

‐ write once memory (WOM): only change 0 into 1

WIC meeting Gent May 2019 12 We start with defects: What is a defect?

• defect‐00 0 1 • Image of a Poly‐Silicon defect‐10 Bridging Defect 1 1 Writing does not change the memory cell content

We assume a fraction pn defects in a word of length n 50% defect‐0; 50% defect‐1

WIC meeting Gent May 2019 13 Key result for memories with a fraction p‐defects

writer reader CAPACITY knows knows bits/cell

YY 1 ‐ p(obvious) Sacha Kuznetsov

YN 1 ‐ psurprise! (K‐T)

In a word of length n with pn defects we can write (1‐p)n bits! Boris Tsybakov

Kuznetsov‐Tsybakov (1974!) show that codes exist (45 years ago)

A. V. Kuznetsov, B. S. Tsybakov , Coding in a Memory with Defective Cells , Probl. Peredachi Inf., 1974, Pages 52–60 (Mi ppi1029)

‐"On the General Defect Channel wi th Informed Encoder and Capacities of Some Constrained Channels A.V.Kuznetsov and A.J. Han Vinck, IEEE Trans. on Information Theory, Vol. 40, November 1994, pp. 1866‐1871. 14 WIC meeting Gent May 2019 2 examples: problem to find codes/coding methods

Example 1: 1 defect ( 1 ) ‐ store x or its complement x Example: ‐ efficiency: (n‐1)/n = 1 –1/n = optimum! Defect‐0 in 2nd position

000 111 001 110 010 101 100 011

Example 2: (n‐1) defects ( 1 0 1 1 ? 0) ‐ 1 position left to change the word parity ‐ efficiency: (1/n) = 1 ‐ (n‐1)/n = optimum !

In General: Linear binary codes can be used. (MDS, or RS (dmin = n‐k+1) codes are optimum)

WIC meeting Gent May 2019 15 An application: Writing more than once in a write‐once memory!

FLASH: we can change 0 into 1 only.

• First writing: write words with pn „1“s 0 0 1 0 0 0 1 0 • write nh(p) bits pn defects 0 0 1 0 0 0 1 0 • Second writing: consider „1“s as fixed (1‐p)n „free“ positions • write (1‐p)n bits

Sum rate: { h(p) + (1‐p) } log23 = 1.58 bits/memory cell

We cannot do better!

WIC meeting Gent May 2019 16 Writing twice: 2 bit in 3 cells (R = 67%)

• First writing 2 bits: 000 001 010 100 0 1 2 3

After inspection of the content, write again 2 bits

• Second writing: 0 111 111 111 111 1 110 001 110 110 2 101 101 010 101 3 011 011 011 100

Uniquely direct decodable: unique rows! Efficiency: R = 2 bits /3 cells = 2/3. At the cost of only 1 additional read!

WIC meeting Gent May 2019 17 General result for q‐ary cells and T writings –before erase‐:

T = 1 2 3 4 … Level Example: q = 4 Increment any step size 0 1 2 3

• Capacity: log2|C| ≈ (q‐1) log2 (1 + T/(q‐1)); for q =2, |C| = log2(T+1)

On the Capacity of Generalized Write‐Once Memory with State Transitions Described by an Arbitrary Directed Acyclic Graph Fang‐Wei Fu and A. J. Han Vinck, IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 45, NO. 1, JANUARY 1999 WIC meeting Gent May 2019 18 q‐ary flash: different type of errors need for special codes

• Writing errors (incorrect level) • Leaking cells • Interference • Cell Wear

3214 5 4 error ‐1 +1

Codes over the Ring of Integers Modulo, IEICE Transactions, November 1998 pp. 2013–2018, (A.J. Han Vinck and Hiro Morita)

Perfect (d,k)‐Codes Capable of Correcting Single Peak‐Shifts, IEEE Trans. on Inf. Th., pp. 656‐662, March 1993, (V.I. Levenshtein and A.J. Han Vinck) WIC meeting Gent May 2019 19 Applications using error correcting codes (ECC)– coding techniques

• Write‐once memories (WOM): FLASH. Reuse old locations • Principle: only use clean pages. Delay erase cycle (expensive in time + ..)

• Use multi level Flash: more bits per cell. Write more than once! • Problems: leaking, overshoot. Use of special ECC, integer codes. • Capacity improvements?

• Phase change: much better performance. Use defect codes • Problems: do not overheat. Use fraction p of the storage capacity • Writing capacity?

• Write unidirectional: Technical writing constraints. Needs special coding techniques • Problem: calculating capacity.

WIC meeting Gent May 2019 20 Multi user theory in Eindhoven

• Frans Willems as staff member from Leuven (vdMeulen) • Example: Coding for Optical Rewriting Disk (Sony)

After a visit to the Walhalla at TUE About 40 years At Philips research: try to prove that .5 efficiency is optimal good colleagues We showed the .51 can be reached in a simple way

Coding problem for higher efficiency still open. "Repeated recording for an optical disk", Proc. 7th Symp. Information Theory in the Benelux, pp.49 ‐53 1986

WIC meeting Gent May 2019 21 Recent: Inf. Th. applications of defect matching codes: old stuff in new jacket

Fraction T position in a word should not be changed: too hot

Fraction W changes in the rest are allowed To be published IEEE IT transactions Capacity: (1‐T)h(W) for W = .5, we have (1‐T)

To be published IEEE IT transactions, and ISIT 2019 Paris ex: for q‐ary cells, maximum change of charge < q

WIC meeting Gent May 2019 22 Single cell storage: one step‐up strategy

average increase 2p(1‐p) per writing

q 1 • Example: average number of writes to reach top level T 2p(1 p)

Input Sequence 7 = 1 6 = 0 1, 0, 1, 0, 0, 0, 1, 1, 0, 1 5 = 1 4 = 0 1, 2, 3, 4, 4, 4, 5, 5, 6, 7 3 = 1 Level 2 = 0 1 = 1 0 = 0 8 level flash

T q 1 the # of bits stored is : log 2 log 2 (T 1) pT 2 which is 50% of the capacity!

WIC meeting Gent May 2019 23 Our research approach: Find appropriate model ( use of the Walhalla) Find codes/coding techniques Calculate capacity/efficiency Use un‐ or informed writer/reader WIC meeting Gent May 2019

Often very unpractical!

Memory research examples: • Magnetic optical rewriting: Write Unidirectional Codes (WUM) • Peak‐shift error correcting codes: especiall in magnetic, but also in flash • Life time prediction for memories with defects: improvement from coding?

24 One more picture

WIC meeting Gent May 2019 25 1990, IEEE IT workshop

Paul de Bot, Ineke van Overveld, Han Vinck Door „gebrek“ aan mankracht naar ULM 1997 in Duitsland

WIC meeting Gent May 2019 26 The first Japan Benelux Workshop on IT, 1989 now Asia‐Europe Workshop (AEW), AEW‐11 in R‘dam, July 3‐5, 2019

WIC meeting Gent May 2019 27 Large non‐volatile memories (RAM) are based on the principle of a variable resistor

NEW RAM MEMORY TYPES:

CHANGE RESISTOR

Flash: change FET resistance

Phase Change RAM: heating DRAM, capacitor load Magnetic RAM: magnetization charge

WIC meeting Gent May 2019 28 Example: Flash memory. Disadvantage: needs erase before writing cycle which is costly in time

• Before writing: find clean page or erase a whole block

done by flash translation layer (FTL)

1. it takes many times (100) longer to perform an (block) erase than it does to perform a read or program

2. Max # of erase operation: 10.000- 100.000

• Info Theory alternative Solution?: use write more than once!

WIC meeting Gent May 2019 29 Writing twice: 1 bit in two cells (R = 50%, fill half of the page)

• First writing 1 bit: 01 0 0 0 1

Without inspection of the content, write

• Second writing: 000 01 110 11

Uniquely and direct decodable: unique rows!

Efficiency: R = 1 bits /2 cells = ½ . Can we do better? How much?

WIC meeting Gent May 2019 30 Extension to: 3‐write‐construction (idea)

Idea of coding:

first writing: 0 0 0 0 10 0 0 1 0 0 0 1 0 0 0 1 0 0 02 bits 0 0 0 0 10 0 0 1 0 0 0 1 0 0 0 1 0 0 0 second writing 0 0 0 1 1 0 0 0 1 10 1 1 0 0 0 1 1 0 0 0 0 1 0 10 1 0 1 0 0 0 1 0 1 0 1 0 1 0 2 bits 0 1 0 0 1 0 0 1 1 0 0 0 1 1 0 0 1 0 0 1

third writing use 2‐defect matching code, store 2 bits

1 0 0 1 0 0 1 0 0 1 0 0 1 1 1 RATE: 2/5 = 0.4 for 3 writings

More results: G. D. Cohen, Philippe Godlewski and Frans Merkx, “Linear binary code for write‐once memories”, IEEE Trans. on Information Theory, 32(5):697‐700, Sep. 1986.

WIC meeting Gent May 2019 31 Extension to q‐ary flash model and T steps T = 1 2 3 4 … Level Example: q = 4 Increment any step size 0 1 2 3

q=2 T+1 T q 1 q=3 (T+1)(T+2)/2 q 1 q=4 (T+1)(T+2)(T+3)/6 q=5 (T+1)(T+2)(T+3)(T+4)/24 …

• Capacity: log2|C| ≈ (q‐1) log2 (1 + T/(q‐1)); for q =2, |C| = log2(T+1)

WIC meeting Gent May 2019 32 ISIT 2019

Level

Example: q = 4 0 1 step increment 1

3 T = 1 2 3 4 …

• Capacity: for large T=> log2|C| ≈ (q‐1) log2 (T+1)

• a factor (q‐1) more than for q = 2

WIC meeting Gent May 2019 33 q‐ary flash: different type of errors need for special codes

• Writing errors (incorrect level) • Leaking cells • Interference • Cell Wear

3214 5 4 error ‐1 +1

Codes over the Ring of Integers Modulo, IEICE Transactions, November 1998 pp. 2013–2018, (A.J. Han Vinck and Hiro Morita)

Perfect (d,k)‐Codes Capable of Correcting Single Peak‐Shifts, IEEE Trans. on Inf. Th., pp. 656‐662, March 1993, (V.I. Levenshtein and A.J. Han Vinck) WIC meeting Gent May 2019 34 Inter cell interference

• Binary: avoid the combination …101…

• Problem solved: Shannon

• Code rate obtainable: 0.694 bits/cell

• Problem for 2D with constrained + 1 + (also multi level) 1 0 1 + 1 +

WIC meeting Gent May 2019 35 Inter cell interference 2‐D constraint

• 0 1 1 1 0 0 No 0 1 1 1 0 0 0 1 0 1 0 or 1 0 1 0 0 0 1 1 1 0 1 1 1 1 1 0 0 0 0 0 1 1 1

WIC meeting Gent May 2019 36 Phase change Problem: do not heat to often

Programmable resistor

allow a fraction p of changes: do not heat too much

Question: What is the obtainable writing efficiency?

WIC meeting Gent May 2019 37 Summary efficiencies

Writer/Reader knows efficiency efficiency efficiency previous content phase change MO defect allow p changes

Y/Y h(p) 0.69 1 - p For IT Y/N h(p) 0.69 (0.55) 1 - p N/Y h(p/2) 0.69 (0.53) 1 - p N/N p 0.54 1-h(p/2)

Efficiency Efficiency h(p) Phase change defect

h(p/2) 1-p 1-h(p/2)p p p p 1/2 1/2 WIC meeting Gent May 2019 38 Sony mini‐disc as an example Low power laser High power laser beam beam Polarization of reflected beam depends on direction of magnetization

Polarization process slow, keep direction for n time instants

F. M. J. Willems and A. J. H. Vinck, "Repeated recording for an optical disk", Proc. 7th Symp. Information Theory in the Benelux,pp.49 ‐53 1986

WIC meeting Gent May 2019 39 A method with R > ½ !

Example: 6 messages, word length n = 5 => R = 0.51 > 0.5 !

code book 0 1 code book 1 0 0 0 0 0 0 1 1 1 1 1 0 1 0 0 0 00 1 0 0 1 0 1 1 1 1 1 0 1 1 0 1 6 messages 0 1 0 0 0 1 0 1 0 0 1 0 1 1 1 0 1 0 1 1 2 0 0 1 0 00 1 0 1 0 1 1 0 1 1 1 0 1 0 1 3 0 0 0 1 00 0 1 0 1 1 1 1 0 1 1 1 0 1 0 4 0 0 0 0 11 0 0 1 0 1 1 1 1 0 0 1 1 0 1 5

Property:Example: 0 0 0 1 0 0 1 0 1 1 0 0 0 1 0 1 0 1 1 0 ... from any code word0 in code1 book1 0 0 1 to any0 word1 in code book 1 0 and back

F. M. J. Willems and A. J. H. Vinck, "Repeated recording for an optical disk", Proc. 7th Symp. Information Theory in the Benelux,pp.49 ‐53 1986 WIC meeting Gent May 2019 40 Some more results (< .69)

R =

[2] F.M.J. Willems & A.J. Vinck, "Repeated recording for an optical disc," Proc. Willems‐Vinck 0,51 n = 5 7th Symp. on Information Theory in the Benelux, Noordwijkerhout, The Netherlands, May 22‐23, 1986. Ed. by D.E. Boekee, Delft University Press, 1986, p. 49‐53. Simonyi 0,5325 n = 11 On write‐unidirectional memory codes Gabor Simonyi, 1989, Information Theory, IEEE Transactions on

Koschnik 0,5637 n = 17 K.U. KOSCHNICK, Coding for Write‐Unidirectional Memories, Oberwolfach Tagungsbericht, 21/1989, May 1989.

WIC meeting Gent May 2019 41 WIC meeting Gent May 2019 42 1-Error correction started by Hamming (1950)

2 1 3

(3,1)

123 (7,4) (31,26) 000 3567 124 111 0000 000 Every circle has even parity 1000 110 0100 101 ••• 1111 111

info parities minimum distance dmin = 3

WIC meeting Gent May 2019 43 Example integer code

• Define calculations modulo m; numbers < m

• Codewords c such that cHT = 0;

For noisy codewords: syndrome S = (c+n)HT = nHT

Correctable error patterns give different syndromes

ex: store 6, read 7 or 5 => noise value +1 or ‐1

WIC meeting Gent May 2019 44 Examples of correction of +/‐ 1 error

• For binary flash: q=2: n = 7, k = 4, d = 3 (Hamming code)

• 1 0 0 0 1 1 1 • 0 1 0 0 1 0 1 • 0 0 1 0 1 1 0 • 0 0 0 1 0 1 1

• For quarternary flash: q = 4: n = 6, k = 4, d=3, corrects one +1/ ‐1 error

• 1 0 0 0 1 1 • 0 1 0 0 1 2 • 0 0 1 0 1 3 • 0 0 0 1 2 1

WIC meeting Gent May 2019 45 General idea used

Information about Information about content? previous content?

Y/N Y/N

writer reader

REMARK: In a normal memory word of length n:

we may store information without restriction

WIC meeting Gent May 2019 46 Problem: do not heat to often

Situation 1: writer and reader know the previous disk content:

change a fraction pn of bits => R = h(p) p ≤ ½ R = 1 p = ½

write read

This gives an upperbound on the performance

WIC meeting Gent May 2019 47 Problem: do not heat to often (do only pn changes)

Situation 2: writer does reader does not know previous content

write read

Start: all 0 memory; use linear code that corrects pn errors

Idea: do only pn to existing words in the memory

Step 1 x: pn changes => syndrome Sx => message m = Sx

Step 2 y: pn changes => syndrome Sy => message m = Sx Sy

Step 3 z: pn changes => syndrome Sz => message m = Sx Sy Sz …

Efficiency: nh(p) bits per writing!

WIC meeting Gent May 2019 48 Problem: do not heat to often (pn changes)

Situation 3: writer does not , reader does know the old disk content:

write read

new 0

p/2 change into 0 old 0 1/2 1‐pno change C = h(p/2) 1/2 p/2 change into 1 old 1 new 1 Coding strategy?

WIC meeting Gent May 2019 49 Problem: do not heat to often (pn changes)

Situation 4: writer and reader do not know the disk content

write read

p/2 change into 0 0 1/2 1‐pno change 1/2 p/2 change into 1 1

C = H(Y)-H(Y|X) = 1 – (1-p) = p

WIC meeting Gent May 2019

50 Memory cell structure

• Gate charge influences the resistor value between S and D

Floating gate

Load the layer unload the layer Problems: 1. Wear out of the material: wear‐leveling algorithms (addressing) every time a cell is written to, it suffers some wear‐out damage. 2. leaking charge in multi level flash 3. incorrect write charge (too high or too low charge) 4. intercell interference

WIC meeting Gent May 2019 51 Application defect matching code

Fraction T position in a word should not be changed: too hot

Fraction W changes in the rest are allowed

Capacity: (1‐T)h(W) for W = .5, we have (1‐T)

WIC meeting Gent May 2019 52 Overview of related research done

Defects: (Kuznetsov/Tsybakov, 1976)

WOM write once memory (Rivest, 1983)

WUM write unidirectional (Willems Vinck, 1986)

WIM write inhibited memory (Cohen, 1998)

WEM write efficient memory (Ahlswede, 1990)

WAM write addressfault memory (Fuja, 1995)

All possible W*M

WIC meeting Gent May 2019 53 Important organized events

• WIC symposium, yearly • Midwintermeeting (January, yearly) • Japan –Benelux workshops on Concepts in Information Theory • First in Eindhoven, 1989 • Follow up –Asia –Europe workshop on Concepts in Information Theory • AEW11, July 3‐5 in Rotterdam • European winterschool on Information theory (Veldhoven ‐ 1994, Zandvoort ‐ 2015) • Van der Meulen Seminar (#8 in 2019?) • IEEE IT workshop 1990

WIC meeting Gent May 2019 54 Information theory from 1950 ‐ 1970

• Louis Stumpers. Member of Board of Governors 1955 ‐ 1962 • Organizer Information Theory events • Important for Benelux: ISIT 1962 –Brussel, 1970 ‐ Noordwijk • Member of many committees, URSI, etc

WIC meeting Gent May 2019 55 Important source of information about local research: 1920 –2013

Archief NERG Tijdschriften Nederlands Electronica en Radiogenootschap

1985

https://www.kivi.nl/afdelingen/telecommunicatie/nerg/archiefWIC meeting‐ nergGent May‐tijdschriften 2019‐nederlands‐electronica‐en‐ 56 radiogenootschap Members with special distinction

• Frans Willems: Marconi young scientist + 2 best papers • Han V. President IT society 2003 + Aaron Wyner award IT society • Kees Schouhamer Immink: IEEE medal of Honor + ....

WIC meeting Gent May 2019 57 WIC meeting Gent May 2019 58