Docable: Evaluating the Executability of Software Tutorials Note: When Installing from the Nodesource PPA, the Node.Js Executable Is Called Nodejs , Rather Than Node

$ 7/22/2019sudo apt-get install build-essentialHow To Set Up a Node.js Application for Production on Ubuntu 16.04 | DigitalOcean The Node.js runtime is now installed, and ready to run an application! Let's write a Node.js application. Docable: Evaluating the Executability of Software Tutorials Note: When installing from the NodeSource PPA, the Node.js executable is called nodejs , rather than node . Samim Mirhosseini Chris Parnin Create Node.js Application NC State University NC State University We will write a Hello World application that simply returns "Hello World" to any HTTP requests. This is Raleigh, North Carolina, USA a sample applicationRaleigh, that will help y Northou get your Carolina, Node.js set up, w USAhich you can replace with your own [email protected] application--just make sure [email protected] you modify your application to listen on the appropriate IP addresses and ports. ABSTRACT Hello World Code The typical software tutorial includes step-by-step instructions for First, create and open your Node.js application for editing. For this tutorial, we will use nano to edit a installing developer tools, editing files and code, and running com- sample application called hello.js : mands. When these software tutorials are not executable, either due $ cd ~ to missing instructions, ambiguous steps, or simply broken com- $ nano hello.js mands, their value is diminished. Non-executable tutorials impact Insert the following code into the file. If you want to, you may replace the highlighted port, 8080 , in developers in several ways, including frustrating learning experi- both locations (be sure to use a non-admin port, i.e. 1024 or greater): ences, and limiting usability of developer tools. To understand to what extent software tutorials are executable— hello.js #!/usr/bin/env nodejs and why they may fail—we conduct an empirical study on over var http = require('http'); http.createServer(function (req, res) { 600 tutorials, including nearly 15,000 code blocks. We find a naive res.writeHead(200, {'Content-Type': 'text/plain'}); execution strategy achieves an overall executability rate of only res.end('Hello World\n'); }).listen(8080, 'localhost'); 26%. Even a human-annotation-based execution strategy—while console.log('Server running at http://localhost:8080/'); doubling executability—still yields no tutorial that can successfully execute all steps. We identify several common executability barri- Now save and exit. ers, ranging from potentially innocuous causes, such as interactive This Node.js application simply listens on the specified address ( localhost ) and port ( 8080 ), and returns "Hello World" with a 200 HTTP success code. Since we're listening on localhost, remote prompts requiring human responses, to insidious errors, such as FigureSign up fo 1:r our annewsle excerpttter. Get the latest of tutor theials on Sy technicalsAdmin and open so softwareurce topics. × tutorial,SCROLL “ THowO TOP clients won't be able to connect to our applicatio n. missing steps and inaccessible resources. We validate our findings Tohttps://www Set.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for Up a Node.js Application-production-on-ubuntu-16-04 for Production on Ubuntu3/21 1 with major stakeholders in technical documentation and discuss 16.04”. You can see the full instructions , and even give it possible strategies for improving software tutorials, such as pro- a try. viding accessible alternatives for tutorial takers, and investing in automated tutorial testing to ensure continuous quality of software tutorials. 1 INTRODUCTION Many software tutorials, include step-by-step instructions for in- CCS CONCEPTS stalling, configuring, and using software tools, which are essential • Software and its engineering ! Documentation. in the software development process. For example, software tutorials hosted by DigitalOcean have been viewed over 409 million KEYWORDS times, including tutorials such as “How To Secure Nginx with Let’s software tutorials, documentation, testing, continuous integration Encrypt” and “How To Set Up a Node.js Application for Production on Ubuntu 16.04” as shown in Figure1. Tutorials are featured promi- ACM Reference Format: nently in developer-related search results [34], and serve a vari- Samim Mirhosseini and Chris Parnin. 2020. Docable: Evaluating the Exe- ety of purposes, such as integrating with instructional materials cutability of Software Tutorials. In Proceedings of the 28th ACM Joint Euro- pean Software Engineering Conference and Symposium on the Foundations for classrooms [11], supporting documentation efforts [28], and of Software Engineering (ESEC/FSE ’20), November 8–13, 2020, Virtual Event, training underrepresented and professional developers through 2 USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3368089. community-driven [17] or paid workshops . 3409706 The ideal tutorial, as described in DigitalOcean’s guidelines for tutorial creators [35], should follow several principles. First, tutorials should be accessible for all tutorial takers, being “as clear 1https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js- and detailed as possible without making assumptions about the application-for-production-on-ubuntu-16-04 reader’s background knowledge.” Furthermore, tutorials should be executable from start-to-finish: “We explicitly include every com- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed mand a reader needs to go from their first SSH connection ona for profit or commercial advantage and that copies bear this notice and the full citation brand new server to the final, working setup.” Finally, tutorials are on the first page. Copyrights for components of this work owned by others than the not merely scripts, but should explain and impart knowledge: “We author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission also provide readers with all of the explanations and background and/or a fee. Request permissions from [email protected]. information they need to understand the tutorial. The goal is for ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA our readers to learn, not just copy and paste.” © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-7043-1/20/11...$15.00 https://doi.org/10.1145/3368089.3409706 2https://learnk8s.io/ ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Samim Mirhosseini and Chris Parnin Unfortunately, tutorials can fall far from this ideal. Despite their remain that prevent fully automated testing of a software importance, software tutorials require considerable effort to pro- tutorial? duce, test, and maintain in order to ensure a high-quality learning experience. Tutorial creators face several barriers: supporting dif- 2.2 Data Collection ferent levels and environments of tutorial takers [31], preventing We selected three popular sources of software tutorials: Vultr3, Dig- instructions from becoming stale as tools or environments quickly italOcean4, and Linuxize5. All tutorials were related to installing evolve [19], and overcoming the expert blind spot effect [26], when software tools, security, and configuring and operating virtual com- tutorial creators do not anticipate steps where novice tutorial takers puting environments, topics frequently important in continuous may have difficulty [24]. As a result, undiscovered issues [18] in deployment processes in software engineering [27]. We used a web- software tutorials can lead to frustrating and ineffective learning scraping script to crawl and download all tutorials for the Ubuntu experiences [16, 23]. operating system hosted on the sources, yielding a total of 780 To systematically understand what issues software tutorials tutorials. contain—and what tutorial creators can do to avoid them—we inves- We organized our collected data into target platform specified tigate software tutorials through the lens of executability, measuring by the tutorial (e.g. Ubuntu 18.04), and then removed duplicate tu- to what extent we can follow step-by-step instructions to their “fi- torials with the same content but targeting different platforms. We nal, working setup”. To this end, we conduct an empirical study of also excluded tutorials that targeted a deprecated Linux distro (i.e. 663 tutorials, first by measuring executability with a naive execu- Ubuntu 12.10) for which we could not find a stable base image, and tion strategy (as one would “just copy and paste”), and finally with tutorials that were primarily focused on using GUI interfaces (e.g. a more sophisticated strategy using human-annotation for inter- “How To Set Up Continuous Integration Pipelines with Drone on preting and executing instructions. Through a qualitative analysis, Ubuntu 16.04” 6). After filtering, our collected dataset of 780 tutori- we identified several issues in tutorials that limited executability, als was reduced to 663 tutorials. In summary, our filtered dataset and we validated these issues with 6 informants, who were expert included 339 tutorials from Vultr, 224 tutorials from DigitalOcean stakeholders in technical documentation. and 100 tutorials from Linuxize. Our findings show that with a naive execution strategy, we We then drew a stratified random sample of tutorials inour

Load more