Leveraging Language Tooling for Better Voice Coding Implementing program awareness and structural editing for Talon Gaute Berge Thesis submitted for the degree of Master in Informatics: Programming and System Architecture 60 credits Department of Informatics Faculty of mathematics and natural sciences UNIVERSITY OF OSLO Spring 2021 Leveraging Language Tooling for Better Voice Coding Implementing program awareness and structural editing for Talon Gaute Berge © 2021 Gaute Berge Leveraging Language Tooling for Better Voice Coding http://www.duo.uio.no/ Printed: Reprosentralen, University of Oslo Abstract Background: Increasingly many programmers become unable to work due to developing pain from using computers too much over a long period of time. Thanks to recent developments, voice control has become a very vi- able option for people with a variety of conditions. At the same time, tools that easily enable analysis of programs in many different languages, such as Language Servers and TreeSitter, are becoming readily available. Aim: This thesis aims to investigate how modern software development tools can be integrated with general purpose voice control systems in order to increase the efficiency of programmers using voice control. Method: A functional prototype of an extension to Talon was developed and evaluated through a qualitative user study of 6 participants. Results: The results showed that participants had a very positive impres- sion of the prototype. The prototype itself also shows that TreeSitter is an excellent tool for the kind of syntactic analysis used in this project. Conclusion: The techniques presented in this thesis can indeed improve the efficiency of programmers using voice controls. i ii Acknowledgments When I first decided that I wanted to write about this topic for my thesis, many of the details were still very unclear. I would like to extend my deepest gratitude to my supervisor Yngve Lindsjørn for putting his faith in me, and agreeing to supervise this project. I would also like to thank the Talon community for their dedication, hard work, and willingness to help anyone struggling to overcome the challenges with learning to become proficient with the voice controls. This project would not have been possible had it not been for their tremendous efforts. Lastly, I would like to thank my family and my partner Lisbeth for their unwavering love and support throughout this process, as well as my friends from the university for staying in touch online and encouraging me when we were not able to meet in person. Gaute Berge May, 2021 iii iv Contents 1 Introduction 1 1.1 Motivation . .1 1.2 Research Questions . .3 1.3 Approach . .3 1.4 Chapter Overview . .3 2 Background 5 2.1 Vocal Programming . .5 2.1.1 History . .5 2.1.2 Dictation . .6 2.2 Structural Editing . .7 2.3 Language Server Protocol . .8 2.3.1 Client/Server Communication . .8 2.4 TreeSitter . .9 2.4.1 Libraries and Bindings . .9 2.4.2 Using Parsers . .9 2.4.3 Queries . 10 2.5 Elm . 12 2.5.1 Introduction . 12 2.5.2 Functions and Values . 12 2.5.3 Types . 13 2.5.4 Modules and Imports . 15 2.5.5 Namespaces . 16 3 Talon 19 3.1 Current state of the project . 19 3.2 Distribution and Monetization . 20 3.3 Community . 20 3.4 Learning Resources and Documentation . 20 3.5 Using Talon . 20 3.5.1 The Basics . 21 3.5.2 Dictation . 22 3.6 Writing Voice Commands with Talon . 23 3.7 Dealing With Homophones . 26 v 4 Methodology and Evaluation 29 4.1 Research Methods . 29 4.1.1 Prototype . 29 4.1.2 Usability Testing . 30 4.1.3 Interviews . 30 4.2 Participants and Recruitment . 30 4.3 The Study . 31 4.3.1 Technical Details . 31 4.4 Ethical Considerations . 31 4.5 Alternative Methods . 32 5 The Prototype 33 5.1 System Overview . 33 5.2 Classified Identifier Extraction . 34 5.2.1 Identifier Classes . 35 5.2.2 Extracting with Regular Expressions . 37 5.2.3 Extracting From Documentation . 37 5.2.4 Extracting From Language Server . 41 5.2.5 Extracting With TreeSitter . 43 5.3 Voice Command Optimization . 45 5.3.1 Automatic Acronyms . 46 5.3.2 Prefix Elimination . 46 5.3.3 Overrides . 46 5.4 Code Navigation . 47 5.5 Structural Editing . 48 5.6 Architecture . 48 5.7 Implementation Details . 48 5.7.1 Tools . 48 5.7.2 Programming Languages and Frameworks . 49 5.7.3 Platforms . 49 5.7.4 Interprocess Communication . 49 5.7.5 Project Statistics . 50 5.8 Prototype Completeness . 51 5.9 Extending the System . 52 5.9.1 Adding Languages . 52 5.9.2 Adding Editors . 53 6 Results 55 6.1 Interview (Part 1) . 55 6.1.1 Prior Experience . 55 6.1.2 Issues With Using Talon . 55 6.2 Usability Test . 56 6.2.1 How Well Did Participants Understand the System? . 57 6.2.2 Engine Differences . 57 6.2.3 User Experience Considerations . 57 6.3 Interviews (Part 2) . 58 6.3.1 General Impressions . 58 6.3.2 Was the Prototype Intuitive? . 58 vi 6.3.3 Favorite Features . 58 6.3.4 Requested Features . 58 6.3.5 System Relevance . 59 6.3.6 System Interest . 59 6.4 Assessment of Tools . 59 7 Discussion 61 7.1 Increase in programming efficiency . 61 7.2 Structural editing by voice . 62 7.3 Implications for Theory . 62 7.4 Implications for Practice . 63 7.5 Limitations . 63 8 Conclusions and Future Work 65 8.1 Future Work . 65 Appendices 69 A Interview Guide 71 A.1 Questions Round 1 . 71 A.2 User Test . 72 A.3 Questions Round 2 . 72 vii viii List of Figures 2.1 Example Communication During Editing Session . .9 2.2 TreeSitter Playground . 11 3.1 Talon Logo . 19 5.1 System Architecture . 49 5.2 Language Distribution . 50 6.1 Challenges of Talon Users . 57 ix x List of Tables 2.1 Constrained Type Variables . 15 3.1 Alphabet used in knausj_talon . 22 3.2 Formatters in knausj (excerpt) . 23 3.3 Syntax of Rules . 24 5.1 Node Types of Interest . 35 5.2 Module Structure in docs.json . 38 5.3 Files and lines per language . 50 6.1 Participants Experience . 56 6.2 Participants Setup . 56 xi xii Listings 2.1 JSON-RPC Example . .8 2.2 Simple TreeSitter Example . 10 2.3 Result from running Listing 2.2 . 10 2.4 Query for Declarations of “x” . 11 2.5 Simple Value and Function Declaration . 12 2.6 Anonymous Records . 13 2.7 Extensible Records . 13 2.8 Type Alias . 14 2.9 Record Alias . 14 2.10 The Maybe Union Type . 15 2.11 Polymorphic Function . 15 2.12 Simple Elm Program . 17 3.1 Example Talon Script . 21 3.2 Talon Module Declaration . 24 3.3 Talon Action . 24 3.4 Talon Context Declaration with Action Override . 25 3.5 Talon lists for functions . 25 3.6 Capture Example . 26 5.1 Simple Search Command . 34 5.2 Type, Constructor and Function with same name . 35 5.3 Excerpt from docs.json . ..