Design of Sequential Stochastic Computing Systems by Paishun Ting A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science and Engineering) in the University of Michigan 2019 Doctoral Committee: Professor John P. Hayes, Chair Professor Scott Mahlke Professor Karem A. Sakallah Professor Zhengya Zhang Paishun Ting [email protected] ORCID iD: 0000-0002-5675-3816 © Paishun Ting 2019 DEDICATION This dissertation is dedicated to my family and my friends. ii ACKNOWLEDGEMENTS I am truly grateful for all the help and advice I have received during my pursuit of the Ph.D. degree. I am specially thankful to my Ph.D. advisor Professor John P. Hayes for his guidance and kindness. I was able to learn so much from him, including but not limited to, identifying and solving critical research problems, writing technical documents and presenting scientific results. It has been a pleasure and an honor to be able to work with him for the past few years. I would also like to thank my Ph.D. committee members Professor Scott Mahlke, Professor Karem A. Sakallah and Professor Zhengya Zhang for their constructive suggestions and their tremendous help that made my research and dissertation better. I am thankful to my former and current colleagues at the University of Michigan, including Dr. Armin Alaghi, Dr. Te-Hsuan Chen, I-Che Chen, William Sullivan and Timothy Baker, for their helpful discussions on research problems and for their useful advice on my career path. I would also like to express my gratefulness to my friends who supported me through these challenging but fruitful years. The good and tough moments we had together are something that will never be forgotten. Finally, I would like to thank my family, especially my parents and my brother, for their company and their encouragement. Their unconditional support has been my emotional and spiritual anchor during the toughest times in my life. I must have been the luckiest person in the world to be a member of this wonderful family. Thank you Dad, Mom, and my younger brother. iii TABLE OF CONTENTS DEDICATION .................................................................................................................... ii ACKNOWLEDGEMENTS ............................................................................................... iii LIST OF FIGURES ........................................................................................................... vi LIST OF ABBREVIATIONS ............................................................................................ xi LIST OF SYMBOLS ....................................................................................................... xiii ABSTRACT ..................................................................................................................... xiv CHAPTER 1 Introduction ................................................................................................... 1 1.1 Motivation ............................................................................................. 1 1.2 Stochastic Computing ............................................................................ 4 1.3 Research Challenges .............................................................................. 7 1.4 Dissertation Outline ............................................................................. 10 CHAPTER 2 Random Fluctuation .................................................................................... 12 2.1 Errors Induced by Randomness ........................................................... 12 2.2 The Role of Constants ......................................................................... 15 2.3 Constant Elimination ........................................................................... 19 2.4 Case Studies ......................................................................................... 23 2.5 Summary .............................................................................................. 29 CHAPTER 3 Correlation .................................................................................................. 31 3.1 Correlation in Stochastic Circuits ........................................................ 31 3.2 Mitigating Correlation Errors .............................................................. 35 3.3 Isolation-Based Decorrelation ............................................................. 40 3.4 Optimizing Isolator Placement ............................................................ 43 3.5 Case Studies ......................................................................................... 46 3.6 Summary .............................................................................................. 52 CHAPTER 4 Sequential Stochastic Circuits..................................................................... 53 4.1 Role of Sequential Elements in SC...................................................... 53 4.2 Behavior of Sequential Stochastic Circuits ......................................... 56 iv 4.3 Shift-Register-Based Circuits .............................................................. 59 4.4 Optimal-Modulo-Counting Circuits .................................................... 66 4.5 Autocorrelation .................................................................................... 71 4.6 Summary .............................................................................................. 78 CHAPTER 5 Stochastic-Binary Hybrid Systems ............................................................. 79 5.1 Integration of SC and Binary Components.......................................... 79 5.2 Latency Minimization for Complex Functions ................................... 83 5.3 Implementation of Maxflow ................................................................ 91 5.4 Case Study: Artificial Neural Network ............................................... 94 5.5 Summary .............................................................................................. 96 CHAPTER 6 Exploiting Randomness in SC .................................................................... 98 6.1 Randomness in Stochastic Numbers.................................................... 98 6.2 Controlling Randomness in SC ......................................................... 102 6.3 Case Study: Image Dithering ............................................................. 105 6.4 Case Study: Neural-Network Hardening ........................................... 108 6.5 Summary ............................................................................................ 113 CHAPTER 7 Conclusions ............................................................................................... 114 7.1 Summary of Contributions ................................................................ 114 7.2 Directions for Future Work ............................................................... 117 APPENDIX ..................................................................................................................... 122 A.1 Proof of Theorem 2.1......................................................................... 122 A.2 Proof of Theorem 2.2......................................................................... 122 A.3 Proof of Theorem 3.1......................................................................... 130 A.4 Proof of Theorem 4.4......................................................................... 131 A.5 Proof of Theorem 6.1......................................................................... 132 BIBLIOGRAPHY ........................................................................................................... 135 v LIST OF FIGURES Figure 1.1 (a) Stochastic sensor array for edge detection [8]. (b) An edge image generated by the stochastic edge detector [8]. ................................................ 3 Figure 1.2 Key SC arithmetic circuits: (a) unipolar stochastic multiplier, and (b) stochastic scaled adder. .................................................................................. 5 Figure 1.3 Data-format converters: (a) binary-to-stochastic converter (BSC), and (b) stochastic-to-binary converter (SBC). ...................................................... 6 Figure 1.4 Stochastic squarer: (a) incorrect implementation due to input correlation, and (b) correct implementation enabled by an isolator D. .......... 8 Figure 1.5 (a) A constant-free stochastic scaled adder [48]. (b) A design for the update node used in the stochastic LDPC code decoder [33]......................... 9 Figure 2.1 Distribution of the number of 1s in SNs with value 0.5 for (a) N = 8, and (b) N = 128, respectively. ...................................................................... 13 Figure 2.2 Three stochastic implementations of scaled addition: (a) conventional MUX-based design CMA with a constant input R = 0.5; (b) ad hoc sequential design CAA with no constant input [72]; (c) sequential design CCA produced by CEASE; (d) error comparison of the three designs. ......................................................................................................... 14 Figure 2.3 MUX-based stochastic scaled adder CMA. On receiving 11 (blue) or 00 (green), the output is 1 or 0, respectively. On receiving 01 or 10 (red), the output is 1 with probability 0.5. .................................................... 18 Figure 2.4 STGs for the sequential scaled adders corresponding to (a) ad hoc design CAA of Figure