A DEEP DIVE INTO AUTOMATIC CODE GENERATION USING CHARACTER BASED RECURRENT NEURAL NETWORKS A Thesis Presented to the Faculty of California State Polytechnic University, Pomona In Partial Fulfillment Of the Requirements for the Degree Master of Science In Computer Science By Renita Priya 2017 SIGNATURE PAGE THESIS: A DEEP DIVE INTO AUTOMATIC CODE GENERATION USING CHARACTER BASED RECURRENT NEURAL NETWORKS AUTHOR: Renita Priya DATE SUBMITTED: Summer 2017 Computer Science Department Dr. Yu Sun Thesis Committee Chair Computer Sciences Dr. Mohammad Husain Computer Sciences Dr. Gilbert Young Computer Sciences ii ACKNOWLEDGEMENTS I would like to acknowledge my thesis advisor, Dr. Yu Sun. He was always there to encourage me and support me in my thoughts and ideas. He has inspired me to always do the best that I can and taught me to always push forward. Dr. Sun has provided me with so much grace and understanding and I am so grateful for the opportunity to have been his student. I would also like to thank the other committee members, Dr. Mohammad Husain and Dr. Gilbert Young for being excellent professors and advisors who have given their time and excellent feedback for this research. Lastly, I would like to give the sincerest acknowledgement to my parents. My parents have constantly supported and encouraged me throughout the completion of my Master Degree. They were the ones who first believed in me that I was even capable of accomplishing this. They have been my support system from day one and have always provided for me. I want to thank all my friends as well. I would like to give special thanks to Brianna who was always there to listen, pray, encourage, and for always having a coffee in hand for me for all those late nights. Thank you Lisa and Ben for all encouragement, prayers and for cheering me on. Above all, I want to give all glory to God. Thank you, Renita Priya iii ABSTRACT Deep Learning is an emerging field in Artificial Intelligence that uses biologically inspired neural networks to recognize patterns in the natural world. These neural networks have an amazing ability to process large amounts of data and learn from them. Recurrent Neural Networks (RNN) are used in applications involving natural language processing like text translations and text generation. This research evaluates the effectiveness of a RNN to be able to automatically generate programming code. Programming languages are different from natural languages in that they have unique structure and syntax. The goal for this research is to conduct experiments on a character RNN model with for three programming languages; Java, Python and C#, and evaluate the results by testing and analyzing the ability for the RNN to automatically produce code that is able to compile. iv TABLE OF CONTENTS SIGNATURE PAGE ......................................................................................................... ii ACKNOWLEDGEMENTS ............................................................................................ iii ABSTRACT ...................................................................................................................... iv TABLE OF CONTENTS ................................................................................................. v LIST OF TABLES .......................................................................................................... vii LIST OF FIGURES ....................................................................................................... viii GLOSSARY ..................................................................................................................... xi INTRODUCTION ............................................................................................................ 1 1.1 Motivation ................................................................................................................. 1 1.2 Problem Description ................................................................................................. 2 1.3 Research Goals .......................................................................................................... 3 1.3.1 Programming Language ..................................................................................... 3 1.3.2 Dataset Size ........................................................................................................ 3 1.3.3 Training Length ................................................................................................. 4 1.4 Background and History ........................................................................................... 4 1.4.1 Deep Learning and Neural Networks ................................................................ 4 1.4.2 Recurrent Neural Networks ............................................................................... 5 1.4.3 Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) .......... 7 1.4.4 Language Model ................................................................................................ 8 v 1.5 Structure of the Experiments .................................................................................... 8 RNN MODEL .................................................................................................................. 10 2.1 Framework and Resources ...................................................................................... 10 2.2 Implementation ....................................................................................................... 10 TRAINING ...................................................................................................................... 16 ANALYSIS AND EVAULATION ................................................................................ 23 FUTURE WORK AND CONCLUSION ...................................................................... 48 5.1 Future Work ............................................................................................................ 48 5.2 Conclusion .............................................................................................................. 49 REFERENCES ................................................................................................................ 50 APPENDIX A .................................................................................................................. 53 APPENDIX B .................................................................................................................. 55 APPENDIX C .................................................................................................................. 58 APPENDIX D .................................................................................................................. 63 APPENDIX E .................................................................................................................. 67 vi LIST OF TABLES Table 1 Programming language datasets and source ........................................................ 12 Table 2 Character codes .................................................................................................... 13 Table 3 Training experiments ........................................................................................... 16 Table 4 Batch loss summary for all training sessions ...................................................... 24 Table 5 Batch Accuracy Summary for All Training Sessions .......................................... 25 vii LIST OF FIGURES Figure 1. General Structure of a Neural Network ............................................................... 5 Figure 2. Unrolled Recurrent Neural Network ................................................................... 6 Figure 3. Example of a two layer RNN model ................................................................... 7 Figure 4. Example of unrolled character RNN ................................................................. 15 Figure 5. Java large - batch accuracy ................................................................................ 17 Figure 6. Java large - batch loss ........................................................................................ 17 Figure 7. Java small - batch accuracy ............................................................................... 18 Figure 8. Java small - batch loss ....................................................................................... 18 Figure 9. Python large - batch accuracy ............................................................................ 19 Figure 10. Python large - batch loss .................................................................................. 19 Figure 11. Python small - batch accuracy ......................................................................... 20 Figure 12. Python small - batch loss ................................................................................. 20 Figure 13. C# large - batch accuracy ................................................................................ 21 Figure 14. C# large - batch loss ........................................................................................ 21 Figure 15. C# small - batch accuracy ................................................................................ 22 Figure 16. C# small - batch loss ........................................................................................ 22 Figure 17. Batch loss summary ......................................................................................... 24