Integrating a Command Shell Into a Web Browser Robert C. Miller and Brad A. Myers Carnegie Mellon University {rcm,bam}@cs.cmu.edu Abstract This command uses ps to list information about running processes, grep to find just the xterm processes, awk The transition from command-line interfaces to graphi- to select just the process identifiers, and finally kill to cal interfaces has resulted in programs that are easier to kill those processes. learn and use, but harder to automate and reuse. Another These capabilities are lost in the transition to a graphical transition is now underway, to HTML interfaces hosted user interface (GUI). GUI programs accept mouse clicks by a web browser. To help users automate HTML in- and keystrokes as input and generate raster graphics as terfaces, we propose the browser-shell, a web browser output. Automating graphical interfaces is hard, unfor- that integrates a command interpreter into the browser’s tunately, because mouse clicks and pixels are too low- Location box. The browser-shell’s command language level for effective automation and interprocess commu- is designed for extracting and manipulating HTML and nication. Attempts to introduce Unix shell features like text, and commands can also invoke local programs. pipelining into graphical user interfaces [3, 6, 7, 8, 15, Command input is drawn from the current browser page, 16] have been unsuccessful, largely because they were and command output is displayed as a new page. The not integrated well with existing applications, required browser-shell brings to web browsing many advantages extra work from application developers to expose hooks of the Unix shell, including scripting web services and and programming interfaces, or were too hard to use. creating pipelines of web services and local programs. A browser-shell also allows legacy command-line pro- With the advent of the World Wide Web, another tran- grams to be wrapped with an HTML/CGI interface that sition is underway, this time to distributed web applica- is graphical but still scriptable, and offers a new shell tions that run on a web server and interact with the user interaction model, different from the conventional type- through a web browser. Most web services accept input script model, which may improve usability in some re- from HTML forms and generate output as HTML pages. spects. Since HTML is textual and capable of being parsed and manipulated, we have the opportunity to recover some of the interactive automation capabilities that were avail- able in the Unix shell, but missing in graphical inter- 1 Introduction faces. Consider the following web-browsing tasks that could be partially or totally automated: The transition from command-line interfaces to graph- ical interfaces carries with it a significant cost. In the • Download and print a group of links on a page; Unix shell, for example, programs accept plain text as input and generate plain text as output. This makes it • Compare airfares and schedules for several choices easy to write scripts that automate user interaction. An of departure and arrival dates; expert Unix user can create sophisticated programs on the spur of the moment, by hooking together simpler • Look up a colleague in the online university phone- programs with pipelines and command substitution. For book, obtain a home address, locate the address on example: a map, and get driving directions; kill ‘ps ax | grep xterm | awk ’{print $1;}’‘ • Given a list of books to read, search for each book in the local library catalog, and if the book is not on the shelves, buy it from an online bookstore; Appeared in Proceedings of USENIX 2000 Annual Technical Conference, San Diego, CA, June 2000, pp 171-182. Best • Make a smart alarm clock that announces the cur- student paper award. rent temperature from an online weather report, and the time until the next bus departs from an online button to correct mistakes, and then copying the com- schedule, while you dress in the morning. mand sequence out of the browsing history and saving it as a script. As a step towards automating these tasks and others, we The browser-shell concept has implications beyond web have extended a web browser in several ways: automation, two of which are considered in this paper: 1. Embedding a pattern language for matching text and HTML, and a suite of text-processing tools for 1. HTML interfaces for local programs. Currently, extracting and manipulating web page data. High- programs with HTML interfaces must be installed level pattern-matching and text manipulation are in a web server in order to handle form submis- essential to web automation, acting as a glue lan- sions. LAPIS can submit forms to local programs guage for connecting unrelated web services and by the Common Gateway Interface (CGI) [17], an programs. existing standard used by web servers. This opens 2. Embedding a scripting language and integrating a the possibility of running HTML applications en- command interpreter into the Location box. In ad- tirely locally. HTML offers benefits of both a dition to accepting a typed URL, the browser win- graphical user interface (GUI) and a command-line dow’s Location box can also accept a typed com- interface (CLI). An HTML interface can be as easy mand with arguments. A command may be a built- to learn and use as a GUI, yet still open to au- in command, a user-defined script, or an external tomation like a CLI. As a demonstration, we have program. The built-in scripting language includes wrapped an HTML interface around the Unix find commands for automatic web browsing, such as program. clicking on hyperlinks, filling out forms, and ex- tracting data from web pages. 2. Using the browser as a command shell, in place of the Unix shell or MS-DOS command prompt. The 3. Using the browser window to display command browser-shell can be used to invoke local programs, output and construct pipelines of commands. When but it behaves differently from a conventional type- a command is invoked, it takes its input from the script shell. Whereas a typescript shell interleaves current page in the browser window, and sends its commands with program output in the same win- output back to the browser window as a new page. dow, a browser-shell displays commands and pro- 4. Including executed commands in the browsing his- gram output in separate parts of the browser win- tory. Forward and Back navigate through command dow, and automatically redirects a program’s in- output pages as well as web pages. Part of the his- put from the current page. These differences make tory can be extracted and saved as a script for later some tasks easier, such as viewing program output execution. and constructing pipelines, but others harder, such as running legacy programs that use standard input to interact with the user. The tradeoffs are discussed We have implemented these extensions in a proto- in more detail in section 5. type web browser named LAPIS (Lightweight Archi- tecture for Processing Information Structure). The first extension, consisting of a pattern language and text- The remainder of this paper is organized as follows. Sec- processing tools, was described in a previous paper [14], tion 2 covers related work. Section 3 describes impor- which is summarized below. This paper focuses on the tant features of the LAPIS browser-shell, including the other three features, which integrate a command shell pattern language, the scripting language, and invocation into the web browser to create a browser-shell. of external programs. Section 4 describes our proto- The browser-shell addresses the problem of interactive type implementation of LAPIS and contrasts some im- web automation by allowing the user to apply patterns, plementation alternatives. Section 5 discusses some of script commands, and external programs directly to the the implications of integrating a command shell into a browser page. For one-shot tasks, commands can be in- web browser, in particular creating local programs with terleaved with manual browsing to perform the task as HTML interfaces and using the browser as an alternative quickly and directly as possible. For repeated tasks, the interface to the system command prompt. Section 6 re- user can interactively define a script by invoking a se- ports on the status of the LAPIS prototype, and Section 7 quence of commands on example data, using the Back concludes. 2 Related Work how the controls are mapped to command-line options. In that sense, it resembles an HTML interface, but is Several systems have addressed the problem of web au- more platform-dependent than HTML. tomation. One approach is macro recording, typified by Others have investigated wrapping HTML interfaces LiveAgent [11]. LiveAgent automates a task by record- around command-line programs on a web server, but not ing a sequence of browsing actions in Netscape through on the client. For example, Phanouriou and Abrams [19] a local HTTP proxy. Macro recording requires little described an HTML interface that presented status infor- learning on the part of the user, but recorded macros suf- mation about a web server (network, filesystem, mem- fer from limited expressiveness, often lacking variables, ory, kernel, etc.) obtained from Unix commands. conditionals, and iteration. The browser-shell is not the first alternative to the stan- Another approach is scripting, writing a program in dard typescript Unix shell. Another is Sam [21], a a scripting language such as Perl, Tcl, or Python. graphical text editor which integrates external program These scripting languages are fully expressive, Turing- execution in three ways: “< command” replaces the complete programming languages, but programs written current selection with the output of a command, “> in these languages must be developed, tested, and in- command” runs the command with the current selec- voked outside the web browser, making them difficult to tion as input, and “| command” redirects both input incorporate into a web user’s work flow.