Docable: Evaluating the Executability of Software Tutorials Note: When Installing from the Nodesource PPA, the Node.Js Executable Is Called Nodejs , Rather Than Node

$ 7/22/2019sudo apt-get install build-essentialHow To Set Up a Node.js Application for Production on Ubuntu 16.04 | DigitalOcean The Node.js runtime is now installed, and ready to run an application! Let's write a Node.js application. Docable: Evaluating the Executability of Software Tutorials Note: When installing from the NodeSource PPA, the Node.js executable is called nodejs , rather than node . Samim Mirhosseini Chris Parnin Create Node.js Application NC State University NC State University We will write a Hello World application that simply returns "Hello World" to any HTTP requests. This is Raleigh, North Carolina, USA a sample applicationRaleigh, that will help y Northou get your Carolina, Node.js set up, w USAhich you can replace with your own [email protected] application--just make sure [email protected] you modify your application to listen on the appropriate IP addresses and ports. ABSTRACT Hello World Code The typical software tutorial includes step-by-step instructions for First, create and open your Node.js application for editing. For this tutorial, we will use nano to edit a installing developer tools, editing files and code, and running com- sample application called hello.js : mands. When these software tutorials are not executable, either due $ cd ~ to missing instructions, ambiguous steps, or simply broken com- $ nano hello.js mands, their value is diminished. Non-executable tutorials impact Insert the following code into the file. If you want to, you may replace the highlighted port, 8080 , in developers in several ways, including frustrating learning experi- both locations (be sure to use a non-admin port, i.e. 1024 or greater): ences, and limiting usability of developer tools. To understand to what extent software tutorials are executable— hello.js #!/usr/bin/env nodejs and why they may fail—we conduct an empirical study on over var http = require('http'); http.createServer(function (req, res) { 600 tutorials, including nearly 15,000 code blocks. We find a naive res.writeHead(200, {'Content-Type': 'text/plain'}); execution strategy achieves an overall executability rate of only res.end('Hello World\n'); }).listen(8080, 'localhost'); 26%. Even a human-annotation-based execution strategy—while console.log('Server running at http://localhost:8080/'); doubling executability—still yields no tutorial that can successfully execute all steps. We identify several common executability barri- Now save and exit. ers, ranging from potentially innocuous causes, such as interactive This Node.js application simply listens on the specified address ( localhost ) and port ( 8080 ), and returns "Hello World" with a 200 HTTP success code. Since we're listening on localhost, remote prompts requiring human responses, to insidious errors, such as FigureSign up fo 1:r our annewsle excerpttter. Get the latest of tutor theials on Sy technicalsAdmin and open so softwareurce topics. × tutorial,SCROLL “ THowO TOP clients won't be able to connect to our applicatio n. missing steps and inaccessible resources. We validate our findings Tohttps://www Set.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for Up a Node.js Application-production-on-ubuntu-16-04 for Production on Ubuntu3/21 1 with major stakeholders in technical documentation and discuss 16.04”. You can see the full instructions , and even give it possible strategies for improving software tutorials, such as pro- a try. viding accessible alternatives for tutorial takers, and investing in automated tutorial testing to ensure continuous quality of software Ubuntu 16.04” as shown in Figure 1. Tutorials are featured promi- tutorials. nently in developer-related search results [33], and serve a vari- ety of purposes, such as integrating with instructional materials KEYWORDS for classrooms [10], supporting documentation efforts [27], and software tutorials, documentation, testing, continuous integration training underrepresented and professional developers through community-driven [16] or paid workshops 2. The ideal tutorial, as described in DigitalOcean’s guidelines for tutorial creators [34], should follow several principles. First, tu- 1 INTRODUCTION torials should be accessible for all tutorial takers, being “as clear and detailed as possible without making assumptions about the Many software tutorials, include step-by-step instructions for in- reader’s background knowledge.” Furthermore, tutorials should be stalling, configuring, and using software tools, which are essential executable from start-to-finish: “We explicitly include every com- in the software development process. For example, software tuto- mand a reader needs to go from their first SSH connection ona rials hosted by DigitalOcean have been viewed over 409 million brand new server to the final, working setup.” Finally, tutorials are times, including tutorials such as “How To Secure Nginx with Let’s not merely scripts, but should explain and impart knowledge: “We Encrypt” and “How To Set Up a Node.js Application for Production on also provide readers with all of the explanations and background information they need to understand the tutorial. The goal is for 1https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js- application-for-production-on-ubuntu-16-04 our readers to learn, not just copy and paste.” Unfortunately, tutorials can fall far from this ideal. Despite their Permission to make digital or hard copies of all or part of this work for personal or importance, software tutorials require considerable effort to pro- classroom use is granted without fee provided that copies are not made or distributed duce, test, and maintain in order to ensure a high-quality learning for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM experience. Tutorial creators face several barriers: supporting dif- must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, ferent levels and environments of tutorial takers [30], preventing to post on servers or to redistribute to lists, requires prior specific permission and/or a instructions from becoming stale as tools or environments quickly fee. Request permissions from [email protected]. ESEC/FSE 2020, 8 - 13 November, 2020, Sacramento, California, United States evolve [18], and overcoming the expert blind spot effect [25], when © 2021 Association for Computing Machinery. tutorial creators do not anticipate steps where novice tutorial takers ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn 2https://learnk8s.io/ ESEC/FSE 2020, 8 - 13 November, 2020, Sacramento, California, United States Samim Mirhosseini and Chris Parnin may have difficulty [23]. As a result, undiscovered issues [17] in 2.2 Data Collection software tutorials can lead to frustrating and ineffective learning We selected three popular sources of software tutorials: Vultr3, Dig- experiences [15, 22]. italOcean4, and Linuxize5. All tutorials were related to installing To systematically understand what issues software tutorials software tools, security, and configuring and operating virtual com- contain—and what tutorial creators can do to avoid them—we inves- puting environments, topics frequently important in continuous tigate software tutorials through the lens of executability, measuring deployment processes in software engineering [26]. We used a web- to what extent we can follow step-by-step instructions to their “fi- scraping script to crawl and download all tutorials for the Ubuntu nal, working setup”. To this end, we conduct an empirical study of operating system hosted on the sources, yielding a total of 780 663 tutorials, first by measuring executability with a naive execu- tutorials. tion strategy (as one would “just copy and paste”), and finally with We organized our collected data into target platform specified a more sophisticated strategy using human-annotation for inter- by the tutorial (e.g. Ubuntu 18.04), and then removed duplicate tu- preting and executing instructions. Through a qualitative analysis, torials with the same content but targeting different platforms. We we identified several issues in tutorials that limited executability, also excluded tutorials that targeted a deprecated Linux distro (i.e. and we validated these issues with 6 informants, who were expert Ubuntu 12.10) for which we could not find a stable base image, and stakeholders in technical documentation. tutorials that were primarily focused on using GUI interfaces (e.g. Our findings show that with a naive execution strategy, we “How To Set Up Continuous Integration Pipelines with Drone on achieve overall executability rate of only 26%. Even with a more Ubuntu 16.04” 6). After filtering, our collected dataset of 780 tutori- sophisticated strategy using annotated tutorials, executability rate als was reduced to 663 tutorials. In summary, our filtered dataset only increases to 52.3%—and even more concerning no tutorial included 339 tutorials from Vultr, 224 tutorials from DigitalOcean successfully executed all steps to its “final, working setup”. Our and 100 tutorials from Linuxize. qualitative analysis revealed several issues, such as inaccessible We then drew a stratified random sample of tutorials inour resources, missing steps, inconsistency in handling file content, and dataset (6%), in order to facilitate qualitative analysis, as done in Kim documentation rot, which detract from the usability and

Load more