RegressionTesting and Conformance TestingInteractive Programs Don Libes- NationalInstitute of Standardsand Technology ABSTRACT _ Testinginteractive programs, by its nature,requires interaction - usuallyby real people. Such testing is an expensiveprocess and hencerarely done. Some interactivetools can ¡e used non-interactivelyto a limited extent, and are often tested only this way. Purely interactiveprograms are rarely testedin any systematicway. _ This paper describestesting of interactive line and character-orientedprograms via Expect. An immediate use of this is to build a test suite for automatingstandards conformanceof all of the interactiveprograms in POSH t003.2a (interactiveshells and tools),something which hasnot yet beenaccomplished by any means. Introduction does not produceoutput that is immediatelycon- sumed by another process, Dennis Ritchie said that "A program but may for example, tl] write designedfor inputs from people is usually stressed to a disk or display.) This is the environment beyond the breaking point by computer-generated in which Ritchie'sremark arose. inputs." I would add the following: Any program In practice,there are forms of input that pro- useful to people- interactively- is likely to be use- duction programsdo not generate.For example, ful to programs- non-interactively.A corollary of programsdo not make typing errorsand thereforedo Ritchie's statementis that conect softwarefunction not (press the backspaceor delete key to) delete during normalhuman use is not a very goodtest of a charactersjust produced.Similarly, programs do not program'stotal correctness. enter control characters,such as might be used to process. I claim that even when humansare explicitly interrupt a This suggeststhat Ritchie was too - teqtinginteractive software, the resultsare still quite optimistic even computergenerated inputs still unreliable. Humanshave many drawbacks: testonly a subsetof a program'sinterface. o Humansknow what is reasonable,and strive Another problem is that some programs are to avoid inconectinput. ni,everused non-interactively. For example, the o Humansassume programs c¿n do things that UNIX passwd program [2] is designedonly to be haveworked in earlierreleases. run interactively. passwd ignores I/O redirection o Humansget boredquickly, and skip tests. and cannotbe embeddedin a pipelineso that input a Humansforget tests. comes from anotherprogram or file. It insists on ¡ Humansare expensive. performingall I/O directlywith a real user. passwd was designedthis way Regressiontesting requiresthe same testing to for security reasons,but the result be performedmany times. For example,after fixing is that there is no way to test passwd non- interactively. a bug, a programshould be testedwithout regardto It is ironic that a programso critical to system security the particular change. Although a modified state- has no way of being reliably tested. ment is an obviousplace to look for new bugs,sub- tle bugs can manifestthemselves Ín distantpieces of Some programs can be run interactively or software. The likelihood of such bugs is low com- non-interactively, but detect the difference and pared to more blatant problems such as incorrect modify their behavior accordingly. For example, algorithms. Hence, they get short sh¡ift from pro- virtually all programs that prompt when running grammersduring testing. interactively disable prompting when running non- The UNIX tool-building paradigm encourages interactively. Unfortunately,this makes it difficult to automatic¿lly desþning programsthat can be usedinteractively as test their interactivebehavior non- interactively. well as non-interactively. Such programscan be embeddedin pipelines. Pipelinesare sets of pro- Commandlanguages, such as the UNIX shell, grams, where each program producesoutput that offer no way of dealingwith programsthat "know" becomesinput for the next programin the pipeline. they are interactingwith a real user. While such (The first program in a pipeline does not dynami- languagesare rich in control and data structuresand cally consumeoutput of anotherprogram, but may can interact with users þrompting and reading for example,read a disk. Similarly,the lastprogram responses),they cannotdo the samefrom programs. Commandlanguages in other popular environments '92 Summer USENIX - June 8-June12,lgg} - SanAntonio, TX 135 Testing Interactive Programs Libes suchas VMS and DOS are similarly lacking. Examplesand Guidelines Expect - A Tool for RegressionTesting Interac- This section of the paper presentsguidelines tions and examplesusing Expect to test commoninterac- tive UNIX tools, building upon earlier work Expect program [7]. [3] is a specificallydesigned to Familiarity with the rudimentsof Expectand UNIX interactwith interactiveprograms. Expectcommun- is assumed. icates with processesby interposingitself between them and acting as an intelligent communications Example- passwd switch. Pseudo-ttys are used so that proc€sses l2l The passwd program believethey are talking to a real user. UNIX takes a username as an argument,and interactivelyprompts for a pass- This is useful for regressiontesting interactive word. The Expect script in Listing 1 takes a user- programs. Expect readsa script that resemblesthe nameand a passwordas arguments,and can be run dialogue itself. By following the script, Expect non-interactivelv. knows what can be expectedfrom a program and what the correctresponses should be. The script can set passhrord Ilindex $argv 21 specify responsesby patterns,and can take different spa$rnpasswd [lindex $argv 1] actionson differentpatterns. expect "password: " gend Scriptsare written in a high-levellanguage (Tcl "$password\r" expect "passwords" - Tool ControlLanguage t4lt5l) andsupport: o send/expectsequences - expect patternscan send "$password\r" includeregular expressions. expect eof o high-level language Control flow Listing 1: Non-interactivepasswd scrípt. First (if/then/else, while, etc.) allows different argumentis username.Second argument is new actions on different inputs, along with pro, password. ceduredefinition, built-in expressionevalua- tion, and execution of arbitrary UNIX pro- In the first line of the script, the variablepass- grams. word is set to the value of the expressionin biack- o job control - Multiple programscan be con- ets. This expressionreturns the secondargument of trolled at the sametime. the scriptby usingthe lindex command(list index). a user interaction- Control can be passedfrom The first argumentof lindex is a list, from which it scriptedto interactivemode and vice versaat retrievesthe elementcorresponding to the positionof any time. The user can also be treatedas an the secondargument. argv refers to the arguments I/O source/sink. of the script, in the same style as the C language Expect is actually capable of general use in argv. automatingor partially automatinginteractive pro- The next line startsthe passwdprogram, with grams,however this paper will focus on its use in the usernamepassed as an argument. testing. In the third line, expect looks for the pattern I will not discussa high-leveltest harness. "password:". There is no action specified,so the This can be providedby any numberof extantpack- expectjust waits until the patternis found before agesor shell scriptsthat are alreadyin use for test- continuing. ing non-interactiveprograms. This paperfocuses on After receivingthe prompt, the next line sends the low-level problems with program interaction a passwordto the currentprocess. The \r indicatesa itself which differ significantlyfrom non-interactive carriage-return,(All the "usual" C conventionsare testing. supported.) There are two send/expectsequences becausepasswd asks the passwordto be typed twice as a spelling verification. There is no point to this Figure 1: An instanceof Expect,communicating with 5 interactiveprocesses directed by a script. '92 136 Summer USENIX - June 8-June12, L992- SanAntonio, TX Libes Testing Interactive Programs in a non-interactivepasswd, but the script has to do This script exits with a numericalindication of this becausepasswd assumes it is interactingwith a what happened.In this case,0 indicatespasswd ran humanthat doesnot type consistently. normally,1 that the username was bogus,etc. LX The final expect eof searchesfor an end-of-file indicatesit died unexpectedlyand 2X that it locked in the ouþut of passwdand demonstratesthe use of up, where X is the particular question in passwd keywordpatterns, Another one is timeout, used to being checked. Exit numbersare usedfor simplicity - denotethe failure of any patternto matchin a given here descriptivestrings could as easilybe returned, amountof time. Here,eof is necessaryonly because including messagesfrom the spawned program passwd is careñrlly written to check that all of its itself. In fact, it is typical to savethe entire interac- I/O succeeds,including the final newline produced tion to a file, deletingit only if the commandunder after the passwordhas been entered a secondtime. testbehaves as expected.Otherwise the log is avail- able for further examination. This script is sufficient to show the basic interactionof the passwd command. A more com- This passwd testing script is designedto be plete script would verify other behaviors.For exam- driven by anotherscript. This secondscript readsa ple, the script in Listing 2 checks several other file of ægumentsand expectedresults. For eachset, aspectsof the passwd program. Completeprompts it calls the first script and then comparesthe results are checked. Correct