Digital Technical Journal

<

Number 6

February 1988 Managing Editor Richard W. Beane

Editorial Staff Editor -Jane C. Blake

Production Staff Production Editor- Helen L. Patterson Designer- Charlotte Bell Interactive Page Makeup -jonathan M. Bohy

Advisory Board Samuel H. Fuller,Chairman Robert M. Glorioso John W. McCredie Mahendra R. Patel F. Grant Saviers William D. Strecker Victor A. Vyssotsky

The Digital Technical journal is published by Digital Equipment Corporation,77 Reed Road, Hudson, Massachusetts 0 I 7 4 9.

Changes of address should be sent to Digital Equipment Corporation,attention: List Maintenance, 10 Forbes Road,North boro, MA 01532. Please include the address label with changes marked.

Comments on the content of any paper are welcomed. Write to the editor at Mail Stop HL02·3/KI I at the published-by address. Comments can also be sent on the ENET to RDVAX::BLAKE or on the ARPANET to BLAKE%RDVAX.DEC@DECWRL.

Copyright© 1988 Digital Equipment Corporation. Copying without fee is permitted provided that such copies are made for use in educational institutions by faculty members and are not distributed for commercial advantage. Abstracting with credit of Digital Equipment Corporation's authorship is permitted. Requests for other copies for a fee may be made to the Digital Press of Digital Equipment Corporation. All rights reserved.

The information in this journal is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation. Digital Equipment Corporation assumes no responsibility for any errors that may appear in this document.

ISBN 1·55558-005-X

Documentation Number EY-8259E-DP

The following are trademarks of Digital Equipment Corporation: DATATRIEVE,DEC, DECjCMS,DEC/MMS, DECnet, the Digital logo,EDT, FMS, PjOS, RALLY,Rdb, ReGIS,RMS, RSTSjE, RSX,TEAMDATA, VAX, VAX COD, VAX COBOL GENERATOR,VAX DECjTest Manager, VAX GKS,VAX NOTES,VAX PHIGS, VAX SCAN, VAXset, VAX SOFTWARE PROJECT MANAGER, VAXstation, VAXstation lljGPX, VAXTPU, VAX VALU, VAX VTX, VMS,VT, VT52,VT IOO, VT200

BASIC is a trademark of Dartmouth College.

BASIS is a registered trademark of Battelle Development Corporation.

HPGL is a trademark of Hewlett-Packard Company.

IBM is a registered trademark of International Business Cover Design Machines Corporation. Lightspeed is a trademark of Lightspeed Computers,Inc. The helix of the DNA molecule depicted on our cover is a Macintosh is a trademark of Apple Computer,Inc. visual metaphor for software productivity tools, the theme of Postscript is a trademark of Adobe Systems, Inc. this issue. just as the encoded DNA molecule serves as a tem­ PowerHouse is a registered trademark of Cognos,Inc. plate for the synthesis of new forms, so software productivity UNIX is a registered trademark of American Telephone & languages and procedures serve as tools for the development Telegraph Company's Bell Laboratories. of new software programs. The image was created using the Xerox is a registered trademark of Xerox Corporation. Lightspeed System. X Window System is a trademark of the Massachusetts Institute of Technology.

The cover was designed by Barbara Grzeslo and David Carroll Book production was done by Digital's Educational of the Graphic Design Department. Services Media Communications Group in Bedford,MA. Contents

9 Foreword William J. Heffner

Software Productivity Tools 1 0 VAX/VMS Software Development Environment Bert Beander

20 Software Productivity Measurements Anne Smith Duncan and Thomas J. Harris

28 Language-Sensitive Editor Glenn Lupton

4 0 VAX SCAN: Rule-based Text Processing Software Stephen R. Greenwood

51 Software Productivity Features Provided by the Ada Language and the VAX Ada Compiler Robert A. Conti

62 Programmer Productivity Aspects of the VAX GKSand VAX PRIGS Products Brian A. Axtell, William H. Clifford , Jr., and Jeffrey S. Saltz

71 The VAXRALLY System - A Relational Fourth-Generation Language Lewis Lasher

80 V7X and VALU- Software Productivity Tools for Distributed Applications Development Linda E. Benson, Michael Gianatassio, Jr., and Karen L McKeen

91 Pragmatics in the Development of VAX Ada Ronald F. Brender, Bevin R. Brett, and Charles Z. Mitchell

I 0 1 Development of a Graphical Program Generator Steven J. Grass

110 Project Management of the VAX DECjTest Manager Software Version 2.0 Linda Ziman and Martin Dickau

117 Development of the VAX NOTES System Peter D Gilbert

I 2 5 Software Usability Engineering Michael D. Good Editor's Introduction

tures of the Ada language and the additional features provided by Digital's implementation. High-level, functional interfaces for graphics pro­ gramming, specifically the VAX GKS and VAX PHIGS implementations, are the topics of our next paper. 13rian A..-xtell, Bill Clifford, and jeff Saltz relate how these interfaces have made graphics programming easier and describe the common architecture on which both products are based. The designer of a software tool is sometimes faced with the dilemma of choosing between flexibility and ease of use. Lew Lasher discusses how the designers of VAX RALLY, a forms-based fourth-genera­ Jane C. Blake Editor tion language, resolved this issue through the design of RAI.LY's application definition system and run-time environment. This issue of the Digital Technical journal features Also designed for application development, the papers on software productivity tools that assist pro­ VAX VTX and VA.'( VALU tool set allows the develop­ grammers in the development of high-quality, reli­ ment and integration of applicatons in distributed, able software. In addition to papers about these heterogeneous environments. Linda Benson, Mike tools, we also present several papers that examine Gianatassio, and Karen McKeen describe the VTX and innovative project practices developed by Digital VALU features and how these serve to enhance pro­ engineers to improve productivity. ductivity. Our first paper looks at the set of tools developed The next four papers offer insights into some of to support all stages of the software life cycle, from the tools and techniques used by Digital software the requirements and specification 5tages through engineers to reduce project development time. In the maintenance stage. Bert Deander gives an the first paper, Ron Brender, Devin Brett, and Charlie overview of each of the tools and describes how their Mitchell describe how their use of automation, strong integration provides for a rich development instrumentation, self-checking, and self-description environment. not only saved development time but also con­ Our second paper is not about a software tool, but tdbuted to the VAX Ada compiler's performance. rather about a study to determine to what degree Next, Steve Grass discusses a new approach devised software wols and new development methods are to manage the development of a then unprecedented contributing to reductions in project cost and to graphical interface, the VAX COBOL GENERATOR increases in product quality. Anne Smith Duncan and software. In the third paper, Linda Ziman and Martin Tom Harris discuss the influences on sofrware pro­ Dickau attribute significant time savings and product ductivity and present findings for three productivity improvements to an iterative approach and the soft­ metrics. ware tools used to develop the VAX DECjTest Man­ The subject of the next paper is the VAX Language­ ager software. One of these tools was the VAX NOTES Sensitive Editor, an imporrant component of the computer conferencing system, which is the topic of VAXfVMS software development environment. Glenn the fourth paper. Peter Gilbert reviews the innova­ Lupton reviews the research on which the require­ tive design and development strategies that led to the ments for this advanced text editor were founded and success of NOTES and describes several key product then describes the design of various LSE features. features. The next two papers are about languages that have In our final paper, Michael Good discusses the been integrated with the VMS environment and three principal activities of software usability engi­ provide programmers increased efficiency in the cre­ neering. He also gives examples of how this user-ori­ ation of programs. First, Steve Greenwood describes ented approach has contributed tO software product the VAX SCAN product and gives examples of how design at Digital. this rule-based text processing language simplifies We thank John Henning for his help in preparing the building of software, thereby reducing program this issue. development time. Next, Bob Conti presents an infor­ mative discussion of the inherent productivity fea-

2 Biographies

Brian A. Axtell A principal software engineer in the Core Applications Group, Brian Axtell is the project leader and supervisor for the VAX PHIGS product. Joining Digital in 1980, Brian co-designed the Base Graphics Archi­ tecture, the VAX PHIGS and VAX GKS products, and was also the project leader in the development of VA X GKS. He earned a B.S. in meteorology ( 1978) and a B.S. in computer science ( 1980) from the Pennsylvania State University. Brian is a member of the ACM, the SIGGRAPH Special Interest Group, and the AmericanMeteorological Society.

Bert Beander Bert Beander, a consulting software engineer, joined Digital after receiving his Ph.D. and M.S. degrees in computer sciences from the University of Wisconsin in Madison. Prior to his current work in the area of programming env ironments with the Technical Languages and Environments Group, he supervised the VAX Debugger and the VA X Perfor­ mance and Coverage Analyzer (PCA) projects and also served as project leader in the development of PCA version 1 and the Debugger versions 3 and 4. Bert is a member of the ACM, SIGPLAN, and SlGSOIT.

Linda E. Benson Linda Benson, a principal engineer, is the project leader and supervisor for the VAX VTX and VAX VALU products. Previously, she was development engineer and project leader for the VAX Run-time Library and has also been involved in a project to produce terminal front-ends. Prior to joining Digital in 1979. Linda was a software engineer at Grumman Aerospace Corporation where she contributed to the development of com­ munication and navigation software systems. She received her B.S.C.S. in 1977 from Rochester Institute of Technology, Rochester, N.Y., and is cur­ rently working toward an M.B.A. degree.

Ronald F. Brender As a senior consultant software engineer, Ron is responsible for most static semantics processing in the VA X Ada compiler. Prior to this work, he supervised development of the first optimizing com­ piler on the PDP-11 and the development of the current generation of BLISS language and implementations. He has been appointed by the Ada Joint Pro­ gram Office to the Ada Board and was a member of X3)3, the ANSI commit­ tee that developed the current FORTRAN '77 standard. Ron received his B.S.E. in engineering science, M.S. in applied mathematics, and Ph.D. in computer and communication sciences from the University of Michigan.

3 Biographies

Bevin R. Brett Bevin Breu is a principal software engineer in the Te chni­ cal Languages and Environments Group, currently working on the VAX Ada project. Most recently, he has been involved in the Integrated Programming Support Environment task force and in the design and implementation of shared generic instantiations in VAX Ada compiler code . Before joining Digital in 1982, Bevin received an M.Sc . in computer science ( 1982) from the University of Adelaide , a B.Sc. (Honors, 1977) in mathematics from the University of Canterbury, and graduated as valedictorian from the Nelson Boys College ( 1 974), New Zealand.

William H. Clifford Jr. Bill Clifford, a principal software engineer, is a co-designer of Digita l's Base Graphics Architecture and the VAX PHIGS product, and of the proposed three-dimensional extension of X 1 1. Bi II is a representative from Digita l to the ANSI X3H3.1 (PHIGS) committee and to the ad hoc PHIGS+ committee. Before joining Digital in 1984, he developed real-time, distri buted control systems at Stable Te chnology, Inc. He received a B.S. ( 1968) and an M.S. ( 1970) in systems engineering from Case We stern Reserve University and pursued doctoral work as an NDEA Fellow at the uni­ versity.

Robert A. Conti A consulting software engineer, Bob Conti has con­ tributed to the VAX Ada project in the area of tasking and debugging support. He is currently pursuing advanced development work related to future enhancements to VAX Ada. Before joining Digital in 1981, Bob developed software for several radar systems, including AWAC S, at We stinghouse. He received a B.S. in engineering (1968) from Case Western Reserve University, an M.S.E.E. ( 197 1) from johns Hopkins, and an M.S.C.S ( 1981) from the Uni­ ve rsity of Maryland. Bob is a member of Ta u Beta Pi, Eta Kappa Nu, IEEE, and ACM.

Martin Dickau Martin Dickau is a Senior Software Engineer in the Com­ mercial Languages and To ols development group. Currently a developer with the VAX Software Project Manager project, he worked on the VAX DECjTest Manager versions 2.0 and 2 1 and was acting project leader on the DEC/CMS project. Martin was a coop student at Digital for two years and joined the company after receiving a B.S. in computer science from the Massachusetts Institute of Technology in 1985.

Anne Smith Duncan Anne Smith Duncan, a software engineering manager in the Commercial Languages and To ols Development Group, is working in the area of measuring and improving the software development process. She has also held various techniol and managerial positions in the Distributed Information Systems Group and the Software Standards Group. Prior to join­ ing Digital in 1978, Anne worked at the Department of the Navy as a Com­ puter Specialist in application systems development and technical support. She earned a Certificate of Management from the Smith College Management Program ( 1984) and a B.A. from George Wa shington University ( 1969).

4 Michael Gianatassio Jr. Mike Gianatassio joined the VTX and VALU engi· neering team in 1983 and with that team has had responsibilities in the areas of development, consulting, and product architecture . As part of the Video· tex Technical Partnership program, he worked with fi nancial institutions on the design of home banking systems. Mike is currently a principal engineer and project leader in the IBM Interconnect Group. Prior to joining Digital in 1982, he worked at Polaroid Corporation. He earned a B.S. degree (1982) in computer science from Northeastern University ofBost on.

Peter D Gilbert After earning a B.S. in computer science ( 1976) and an M.S. in computer engineering ( 1979) from the University of Illinois, Peter Gilbert joined Digital in January 1979. He is a member of the Com mer· cia! Languages and To ols Group and has been a developer on the VAX COBOL, VAX and PDP-I I sortjmergc, and VAX NOTES projects. and was a developer responsible for the col lating sequences, parallel processing, and mathematics software for the VAX Run-time Library project. Curre ntly work· ing on the design of a configuration management tool, Pe ter is a principal software engineer.

Michael D. Good As a principal software engineer in the Software Usabil· ity Engineering group, Michael Good is developing software usability engineering methodologies and contributing to the user-interface design of several products. He has co:1ducted usability research since joining Digita I in 1981 and has published a number of papers on usability engineering and text editing. He designed and implemented the Eve text editor for the VAXjVMS operating system ve rsion 4 2. Michae l re ce ived a B.S. (1979) and an M.S. (1981) in computer science from the MassachusettS Institute of Technology.

Steven J. Grass Steve Grass, a princi pal software engineer in the Commer· cial Languages and Tools Group, is the project leader of a group respon· sible for the development of common components for window-based applica­ tions. Previously, he worked on the VAX COBOL GENERATOR project, first as a member of the advanced development team and then as the project leader for implementations of ve rsions 1.0 and 1.1. Steve was also a devel· opcr on the PDP-1 1 COBOL and COBOL-81 compiler projects. He joined Digital in 1978 after earning a B.S. E. in computer engineering from the Uni· versity of Michigan.

Stephen R. Greenwood A consu ltil)g software engineer, Steve Green· wood is currently re sponsible for a tool for specifyi ng the end-user interfa ce to window-based applications. Previously, he has been involved with the design of run-time libraries for future architectures and the design and devel­ opment of the VAX SCAN language. Before joining Digital in 1981, Steve worked at Sperry Univac Corporation as a member of the compiler deve lop· me nt grou p. He received his B.S. in physics (cum laude, 1 973) from Buck­ ne ll University and an M.S. in computer science ( 1975) from the University of Wisconsin. He is a me mber of the ACM. Biographies

Thomas J. Harris Since joining Digital in 1978, Tom Harris has been a manager of the Commercial Languages and Tools Development Group and is currently the group's senior manager. He chairs Digital's Sponsored Research Board and is responsible for advanced development planning across the Sys­ tems Software Development Group. Prior to joining Digital, he worked for Sperry Univac Corporation in software and hardware product development. To m is a member of the CO DASY L Executive Committee. He pa rticipated in and chaired the CO DASYL Command La nguage Committee and was a member of the FO RTRAN Data Manipulation Committee. Tom earned a B.S. in engi­ neering at Case Western Reserve University in 1967.

Lewis Lasher Lew Lasher is a senior software engineer in the Software Development Technologies Group. Since joining Digital in 1985, he has worked on the VAX RAL LY project, primarily in the area of user interface. He earned an A.B. degree in applied mathematics in 1978 at Harvard College , where he served as a teaching fe llow in undergraduate courses in computer science. While at Harvard he also worked on PPL, an interpreted language used there in the introductory programming course. Lew earned a J.D. degree at Harvard in 1981 before returning to software engineering.

Glenn H. Lupton Glenn Lupton joined Digital in 1975 after receiving a B.S.E.E. ( 1973) and an M.E.E.E. ( 1974) from Rensselaer Polytechnic Insti­ tute. A consulting software engineer in the Technical Languages and Environ­ ments Group, he is the project leader of the VAX Language-Sensitive Editor project . Glenn has been associated with the BLISS compiler projects as devel­ oper, project leader, and supervisor. He has also supervi sed the development of a number of software programming environment tools, including the pro­ totypes for VAX SCA, DECJMMS and the DECjTe st Manager software.

Karen L. McKeen Karen McKeen is a senior software engineer working in the VAX VTXjVALU engineering group. She joined the VTX group in 1985 and has been responsible for designing and developing information provider components. She is currently the architect for VTX and VALU. Previ­ ously, Karen wa s project leader for the VA X DECgraph project. She has also worked in the Commercial Systems Evaluation Group developing perfor­ mance tools. Karen joined Digital in 1979 after earning a B.S in mathematics with a computer science interdisciplinaryoption from the University of New Hampshire.

Charles Z. Mitchell Charlie Mitchell, a consu lting software engineer in the Technical Languages and Environments Group, has been a member of the VA X Ada development project since its inception as an advanced develop­ ment project in 1979 Currently the project leader of the VAX Ada develop­ ment project, he joined Digital in 1976 as a developer in the LCG Languages Group. Charlie received a B.S. from the University of New Mexico and an M.S. in computer science from Rensselaer Polytechnic Institute . He is a member of the ACM and SIGAda .

6 JeffreyS. Saltz Jeff Sa ltz joined Digital after receiving a B.S. in computer science (honors, 1985) from Cornell Un iversity. A senior software engineer in the Core Applications Group, he is co-designer of Digital's Base Graphics Architecture, and the VAX PHlGS and VAX GKS products. jeff is a representa­ tive from Digital to the ad hoc committee for the proposed three-dimensional extension ro Xll and a co-architect of the X3D-PEX proposal. He is a mem­ be r of Tau Beta Phi.

Linda Ziman Linda Ziman is a development supervisor in the Commercial Languages and Tools Grou p. She is currently responsible for several projects, including the VAX DEC/Test Manager, VAX DEC/CMS, VAX Software Project Manager, and an advanced development project. Previously, Linda worked with integrated software envi ron ments and was project leader of the DEC/ Test Manager project. Sbe bas worked in the area of software product ivity tools since joining Digi tal in 1978 She received her B.S. degree from Union Col lege and is a member of ACM and IEEE.

7 Foreword

Brooks likens a programmer ro a poet in that a creative, intangible product is rhe result of a programmer's wo rk. Even though colleges and universities have fo rmalized the training of pro­ grammers in a discipline called software engi­ neering, we are certain on ly that programmers write programs; that the discipline is unique; and that because this discipline is unique, pro­ grammers require unique rools and products to effective ly complete their tasks. The papers in this issue of the Digital Te chni­ William]. Heffner cal jo urnal address a parr of our continual effort at Digital to produce the environment and Vice President, products that assist programmers in produc­ Systems Soft ware Group ing timely, we ll-defined, efficient, and reliable programs . HistOrically, there have been two What is a programmer' What does hejshe do1 major breakthroughs in reducing the elapsed Why does it rake so long' These are three of the time to produce a working program. First were questions most often asked of those of us in the compilers, which provide the programmer a software profession. more concise and error-free technique for pro­ Augusta Ada Byron (1815-1852), the ducing programs. Grace Hopper at UNIVAC and Countess of Lovelace, has been accorded the john Backus at IBM were leaders in this break­ title of the world's first programmer. Her notes through. The second major breakthrough, led published in London in 184 3 regarding Charles by Digital, was interactive timesharing. Inter­ Babbage's analytical engine included a formula active timesharing allowed the programmer for solving a problem on that machine. This greater access ro rhe computer, thus reducing formula is in effect the first example of a com­ the elapsed time for program development. puter program, hence her recognition as a pro­ In add ition to the effort ro reduce elapsed grammer. time, equal effort is being expended tO add dis­ For roughly the century fol lowing Byron's cipline and predictability ro the process of pro­ notes, the person who designed rhe computer ducing programs. Today, very few programs also built and used the computer. There was sel­ are deve loped by a single programmer. Instead, dom a separation of the builder and the user. In reams of programmers collaborate ro produce the middle of this century. however, computers larger and more comprehensive programs, for were being used by many people not involved example, the FORTRAl'J project and the VMS in either designing or building the computer. projecr. To accomplish such projects, program­ These users transformed their problem state­ mer productivity rools and the Computer Aided ment intO a computational method understOod Software Environment (CASE) have been devel­ by the computer. This computer representation oped . The VA XjVMS system has been the indus­ of the problem was called a program, and the try standard for programming development and person preparing it was ca lled a programmer. the sys tem of choice for programmers. The Now, what is a programmer' Simply pur, the papers herein demonstrate our continuing effort programmer is someone heretOfore unknown in tO be the leader. Our goal is ro produce the best the professions. Programmers in the 1950s and environment for programmers - an environ­ 1960s came from many disciplines. Many were ment in which they can exploit their creativity electrical engineers and mathematicians, but as they participate in a predictable and disci­ others were musicians, I iberal arts majors, and plined process. even dentists, hospital admin istratOrs and the like . What was the unique talent they possessed' In his book The My thical Ma n Mo nth, Fred

9 Bert Beander I

VAX/VMS Software Development Environment

The VAXjVMS software development environment comprises tools that support all stages of the software life cycle. Thesetools include documenta­ tion tools, a project management tool, code management and system build­ ing facilities, a rich editing and browsing environment, a powerful debug­ ger, static and dynamic analysis tools, test management facilities, and project communications tools. Moreover, these tools are strongly inte­ grated with each other: they share a common user interface philosophy, theyhave numerous tool-to-too/links that allow them to pass substantial amounts of program infonnation to each other, and they support multiple programming languages. As a result, the environment has both richness and internalcohesiveness.

Software development has become increasi ngl y productivity by reducing deve loper learning dependent on programming environments that rime . This paper describes how the separate tOols provide a rich set of software deve lopment rools. of the VAX/VMS software deve lopment environ ­ Such environments are attractive because they ment supporr the various stages of the software can increase both programmer productivity and life cyck and explains how the many rool-ro-rool software quality, while reducing deve lopment links and information flows make the environ­ costs. The programming environment that Digital ment so tightly integrated. has clcvelopeclfor the VAXjVMS operating system is an example of a commercially available envi­ Supporting the SoftwareLife Cycle ronment that provides a particularly rich set of Digita l's programming environment on the VA.,'

10 Di�ital Technical jo unwl No. o Fl!bmnr)• I

REQUIREMENTS STAGE

SPECIFICATION STAGE

DESIGN STAGE

'MCC NW,ON 5'AGE

TESTING STAG E r l MAINT ENANCE STAGE RUNOFF. DOCUMENT I I I I I I I VAX NOTES. VMS MAIL I I I I l J LANGUAGE-SENSITIVE EDITOR I I I I I I CODE MANAGEMENT SYSTEM I I I I _l J I SOFTWARE PROJECT MANAGER I I I I I I MODULE MANAGEMENT SYSTEM I I l COMPILERS. LINKER I I I I SYMBOLIC DEBUGGER I I _l J PERFORMANCE AND COVERAGE ANALYZER I I I I SOURCE CODE ANALYZER I I DEC/Test Manager

KEY

- PRIMARY TOOL USAGE c:::J OCCASIONAL TOOL USAGE

figure I The Software Life Cycle

documents and source files, and to manage pro­ Both requirements and specifications are usu­ ject activities. This section describes the stages of ally written in English or another natural lan­ the software life cycle and the tools that are used guage. The tools needed ar these two stages must at each stage. rhus faci I irate rhe production and organization Figure I summarizes the software develop­ of documents. To produce documents, develop­ ment stages and the associated tools. In this dia­ ers first need one of the environment's text gram, the life-cycle stages are listed along the editOrs, such as the VAX Texr Processing Utility top and selected tools are listed along the right or the VAX Language-Sensitive Editor (described side. Solid bars mark the life-cycle stages where further below), to compose the actual text. tools have their primary uses; light bars mark the They then need a rexr processing roo! ro format stages where tools are occasionally used. their documents. Two such roots are available on VAXfVMS. One is Runoff, which produces docu­ Requirements and Specification Stages ments as formatted ASCII text files. Runoff is sim­ During the requirements stage. the customers ple bur quite serviceable for documents that only or developers identify the requirements of the require typewriter quality. VAX DOCUMENT is a proposed software system. During the specifica­ newer tool, which has been used at Digital to tion stage that follows, developers formulate produce all VAXfVMS software documentation. detailed specifications that define what the sys­ DOCUMENT converts text files written in a tem will do and how it will be used. By com­ marh.'Up language into typeset-quality, formatted paring the specifications tO the requirements, output. The output can be printed on a laser the developers can show, at least informally, printer or processed on a typesetting system for that if the system is built as specified. it will meet final production. DOCUMENT is layered on top its requirements. of Donald Knuth's TeX text processing system,

Digital Technical journal 1 I No. 6 Februar)• I 'J88 VAXjVMS Software Development Environment

and thus supports multiple fonts, mathematical guages and arc thus created using editors. The typesetting, and extensive formatting capabili­ VAX Lmguagc-Sensirive EditOr (LSE) is usually ties. 2 rhe editor of choice.·• In this section, we discuss Documents also need to be stored. Although how LSE is used and also mention how designs they can be stored as ordinary files in ordinary may be represented graphically. Once a design is VMS directories, it often makes more sense ro use in piau:, the developers must formulate a plan the VAX DECjCMS (Code Management System) for building the desired software and create a tool ro store documents. CMS can store multiple development schedule based on that plan. A dis­ versions of document sources efficiently, and it cussion of a tool that helps developers do such a !lows old versions to be retrieved at any time. planning concludes this section. CMS also provides check-outjcheck-in control The design components written in program­ over document sources to prevent different ming languages are normally created using LSE. developers from inadvertently modifying the LSE is a full-featured, programmable. full-screen same sources at the same time. A developer thus text editor. Jt is "language-sensitive·· in several checks our (or "reserves") a source file from a senses. First. it provides templates for the con­ CMS library into a private work area, works on it structs in each supported programming language in the private area until satisfied with it, and then (about a dozen languages arc currently sup­ checks it back in (or "replaces" it) to the CMS ported and users can create templates for addi­ library. While the source module is reserved, no tional languages). Second, it al.lows placeholders other developer can modify it. (CMS allows mul­ in those templates to be expanded so that the tiple concurrent reservations. but developers valid possibilities for each syntactic entity can be who choose this option must be willing ro later displayed and selected. Third, it provides on-line merge the independently made changes. CMS has help for each supported language. And fourth, it faci I ities that partially automate such merges.) allows programs tO be com piled directly from Another tool that is very useful when collect­ tbe editor and compilation errors ro be reviewed ing requirements is the VAX NOTES electronic directly in the editor. 5 conferencing system NOTES allows multiple These capabilities are best explained by exam­ users ro share comments on a variety of topics. ple. Suppose a user wants ro enter a WHILE loop Each NOTES conference is organized into "top­ in a Pascal program. To do so, the user enters the ics," where a written note starts the discussion of WHILE keyword and then expands that construct each topic. Members of the conference can cr<:ate by pressing an "expand" key. In response, LSE new topics at any time, and they can reply to produces the following text: existing notes and other people's replies. All WHILE X{expression}X DO information is stored on line and is easily perused X{statement}X from any node in the users' computer network. VAX NOTES thus provides a very convenient and Within this template. there are two placeholders: expedient way to collect requirements and one for the Boolean expression. and one for the reviewers' comments for a software project. and statement that forms the loop body. Single is widely used within Digital during the require­ keystrokes move the editing cursor from place­ ments and specification stages of the software holder to placeholder. Any placeholder can in life cycle. The VMS Mail utility is also used exten­ turn be expanded to display a list of valid alterna­ sively for project communications and for infor­ tive expansions from which the user can choose mation exchange with other groups. one. For example. pressing the expand key when the cursor is on the X

12 Digital Tecbnical journal No. () Februarv I 'JHR Software Productivity Tools

tactically correct programs even if they do not schedule that shows when each task will begin know the language very well. In addition , LSE and end . By later recording the actual start and provides language help in the form of help text end dates of each task, the project leader can that explains the form and usage of each language use PM to track actual progress and compare it to construct. the schedule. The va lue of PM is that it automates Finally, developers can compile programs from much of the bookkeeping associated with LSE and review compilation errors in the editor. scheduling and controlling a software project, To compile a program, LSE writes the contents of thus helping to ensure that the project is com­ the current buffer to a fi le, creates a subprocess, pleted on schedule This kind of bookkeeping and runs the compiler on that file in the subpro­ wou ld otherwise have to be done manually. cess. The compiler records any error messages in a "diagnostics file," which it passes back to the Implementation Stage editor. The editor displays these error messages During the implementation stage, code is writ­ in one editing window while displaying source ten, debugged, and tested. The environment code in another window. The user can select suc­ provides numerous cools for this stage . These cessive error messages and direct the editor to tools include editors, compilers, a debugger, automatically position the source window on the code management facilities, a system builder, corresponding error locations. Errors are thus and static and dynamic analysis tOols. This section quickly located and corrected . Some compilers gives an overview of these tools. wi ll also suggest error corrections, in which case When writing code, developers using the LSE automatically displays those corrections in VAXjVMS software development environment the source window for the user's approval or dis­ can choose from among more than a dozen pro­ approval. gramming languages, and they may include mod­ When designing data structures, developers ules written in different languages in the same may choose to store their data definitions in the program. The developers write most code using VA X CDD (Common Data Dictionary) database. LSE, but may also use specialized edirors such as CDD serves as a repository fo r data definitions a forms ed itor. Developers compile programs common to many separate programs, where the using both the standard language compilers and programs access common databases and may be specialized compilers such as the message com­ written in many different languages. CDD is par­ piler (for error messages) and the help librarian ticularly well suited to commercial environments (for creating hierarchical help text) . They then where multiple applications programs access link and run the program. large central databases. To debug their code, programmers use the 5 Developers may also create graphical represen­ VA XjVMS debugger The debugger allows the tations of designs using techniques such as struc­ programmer to set breakpoints in the code , to set tured analysis, structured design , or data model­ watch points (data breakpoints) on data loca­ ing. Digital does not itself provide tools to tions, to single-step the program by source line or automate graphical software design, but suitable machine instruction , to examine variable values, tools are available fo r the VMS operating system and to deposit new values into memory, among from other vendors such as Intech, Cadre, Nastec, many other things. The debugger is fu lly sym­ Te ktronix, and Interactive Development Environ­ bolic, receiving its symbol information from the ments. compilers via the linker. The debugger uses mul­ Once a design is in place, the project leader tiple windows on the user's screen to display must fo rmulate a plan for building the desired extensive program state information to the user. software . To do so, he creates a work break­ This information allows the user tO find program down structure that identifies the individual bugs rapidly and efficiently. tasks or work assignments needed to implement To organize and maintain all program sources, the design. He associates time estimates with the developers use the VA.'<. DECJCMS code man­ the individual tasks, identifies dependencies agement system, described earlier as a tool for between tasks, and determines which pro­ managing document sources. To build the soft­ grammers are avai lable. Given this information, ware system being developed , programmers use the project leader then uses the VAX Software the VAX DECJMMS (Module Management Sys­ Project Manager (PM) tool tO construct a project tem) system builder. Like the UNIX Make utility,

Digital Technicaljournal 13 No. 6 Fe bruary 1988 VAX/VMS Software Development Environment

MMS performs a minimal system build based on ious software functions. The developers then module dependency information and knowledge have DTM capture the software's output when of which source modules have changed since the the software is mn under each script, and they last build. manually certi fy that the software produces cor­ To fo llow cross-references and perform static rect output for each script. DTM then saves the analysis, developers use the VAX Source Code correct outputs as "benchmark files" and orga­ Analyzer (SCA) . SCA receives cross-reference nizes the test scripts into user-defined categories. information from the compilers. This information Subsequently, the developers can use DTM is incorporated into a database that allows cross­ tO automatically run various categories of tests reference queries over an entire software project (or all tests) on later versions of the software. ro be answered quickly. SCA is rightly integrated When DTM runs a collection of rests, it runs a set with LSE so that LSE can display cross-reference of rest scripts through the software being tested, information and cross-referenced source code in collects the outputs from the software , compares editor windows. SCA can also perform static anal­ the actual outputs to the expected outputs (the ysis by showing call trees and by checking proce­ benchmark fi les) , and reports any differences dure calls for consistency with the corresponding to the user. DTM allows developers to build procedure declarations. up large regression rest systems for their soft­ To perform dynamic program analysis, devel­ ware . Experience indicates that such test systems opers use the VAX Performance and Coverage constitute the single best guarantee of software Analyzer (PCA) . PCA can collect several kinds of quality. performance data during program execution, The VAX Performance and Coverage Analyzer is including program counter sampling data, page important during testing because it can measure fa ult recording, 1/0 usage, and exact execution rest coverage, that is, identify the code paths counts at specified program locations. PCA can that are or are not executed by the regression later display all this data in a variety of his­ tests. (PCA measures what some people would tograms and tables. PCA can also show perfor­ call "statement coverage"; PCA determines what mance data at various resolutions, from the pro­ instructions are executed , nor what branches are gram module level down to the individual source taken.) The coverage is reported symbolically in line or even instruction. By using PCA, pro­ source code displays. Using this information, grammers can quickly locate performance bottle­ developers can write additional test scripts to necks, many of which usually turn out to be easy ensure that all code paths are tested at least once. to remove by reprogramming. PCA thus helps Once the software is implemented and passes programmers produce high-performance soft­ all regression tests, it is ready to be tested by ware, something that is hard to do without this actual users in a field test. During field test, prob­ kind of tool. lems must be reported to the developers. Pro­ vided the users and the developers are on the Te sting Stage same computer network, VA X NOTES has proven There are typically two kinds of software resting. to be an excellent problem reporting tool. A user First, developers test the software during the can report each new problem as a separate topic implementation stage to ensure that all individ­ and developers can reply to each topic. Other ual functions work. Second, actual users test the users can see the problem reports along with software to ensure that it works under normal their responses, which alerts them to known operating conditions. Several components of the problems; they can also enter additional VA XjYMS software development environment responses to supply fu rther information or to were designed to help make programmers more answer questions. productive by automating certain activities of the testing stage. Ma intenance Stage To test software during implementation, devel­ When a software system is released tO its opers use the DECjTesr Manager (DTM) testing users, it enters the maintenance stage of the tool. To use DTM, developers must first write test software life cycle. At this stage, bugs are fixed scripts for their software, where each "script" and minor enhancements are added. (Major consists of input tO the software that will test var- enhancements require a new pass through the

14 Digital Technical journal No. 6 February 1988 Software Productivity Tools

whole software life cyc le, and developers start generate a substantial amount of information this process by defining the req uirements for tools such as the debugger, the perfor­ for the next major version .) As during field test, mance analyzer, the editor, and the static NOTES can be an effective roo! for recording and analysis and cross-reference tool . Other tools respond ing tO problem reports, provided the can invoke each other, passing along enough users and developers are on the same computer information to create a smooth transition from network. As during implementation, the standard tool to tool . coding tools - LSE, the compilers, the linker. • Al l tools support the development of applica­ the debugger, and PCA - are used tO fix bugs tions written in mu ltiple programming lan­ and add en hancements. guages. Deve lopers are therefore free to pick During the maintenance stage, CMS and the language or languages they deem best for MMS remain essential. CMS's ability to keep their applications. track of multiple versions of the software sys­ The strong integration between tools gives tem and to maintain multiple parallel develop­ the programming environ ment a mature, cohe­ ment streams (variants) of the program sources sive feel to the user. Because tools have been is particularly important. For example, by using developed together, they can also give a wea lth CMS, developers can easi ly maintain a version of capabilities which wou ld not otherwise be 1 .1 maintenance stream of the sources (for bug possible. This section describes how the environ­ fixes) while also wo rking on a version 2.0 ment is integrated across tools and illustrates development stream (for major enhancements) . some of the capabi lities that this integration The Source Code Analyzer is also very useful makes possible. because it makes it easy tO browse through unfa­ miliar sources and quickly obtain the definitions Common Command Sy ntax of procedures, variables, and other program constmcts. All tools in the VMS environment have command Finally, the Te st Manager re mains very impor­ languages that are based on the same philosophy tant at this stage for maintaining software as the Digital Command Language (DCL) , the qual ity as changes and bug fixes are made . A top-l evel command language for VMS. In DCL, we ll-designed set of regression tests can ensure each command consists of a command name, that all major functions of the software system followed by zero or more "qualifiers," followed st ill work correctly after changes have been in turn by zero or more command parameters. made. Test ing can never demonstrate the absence The following command , which invokes the of errors, but the successfu l execution of we ll­ FORTRAN compiler, is an exam ple: designed tests can demonstrate that all com mon FORTRAN/DEBUG/NOOPT A,8 operations work correctly in typical circu m­ stances. Such tests can therefore give developers Here FORTRAN is the command keyword , /DEBUG a high degree of confidence in the integrity of the and /NOOPT are qualifiers, and A and 8 are softwa re . parameters. The command compiles files A. FOR and 8. FOR with debugging information enabled Integration among To ols and optimization disabled . All tools in the VMS environ ment have com­ To increase their usabi lity and to enhance the mands of the same syntactic form as DCL. Fur­ smoothness with which they can be used thermore , when tools have common capabilities, together, Digital's rools are strongly integrated they use the same command syntax. For exam­ with each other. This integration takes three ple, the command, which creates a new forms: SPAWN subprocess, has the same syntax in DCL, the • Al l tools share a common command language debugger, the Mail utility, the Language-Sensi­ phi losop hy. Consequently, commands have tive Ed itor, and many other tools. The help sys­ the same syntactic fo rm and general appear­ tem also works the same way in all tools. As a ance in all tools. result, all tools share a common "feel," and • A great deal of program information fl ows users can frequently guess how to use a given between tools. The compilers, in particular, tool from their knowledge of other tools. Future

Digital Technicaljournal 15 No (, Febt'tlal)' I 988 VA Xj VMS Software Development Environment

workstation interfaces will maintain this unifor­ available in suitable forms to all tools that need mity across tools by having all tools use a new it. Th is section discusses these information flows windowing interface conforming to the industry­ and certain other connections between tools. wide X Window standard . Figure 2 illustrates the many connections and information flows between tools that give the Information Flow between To ols VAX.jVMS programming environment its tight Th e integration of the VMS programming envi­ integration. The boxes represent tools, and the ronment stems in large part from the informa­ arrows represent information flows, either via tion flow between tools. The compilers in fi les or through direct calls between tools. particular generate a substantial amount of infor­ The debug symbol table (DST) contains the mation for other tools. They generate symbol name, type. and address or va lue of every symbol table information for the debugger and the Per­ in the user's program. This information is passed formance and Coverage Analyzer, diagnostic from the compiler to the linker, which performs information for the Language-Sensitive Editor, address relocation on the DST. The information is and cross- reference and calling-sequence infor­ then passed to the debugger. The DST contains mation for the Source Code Analyzer. The com­ scope information so that the scope of each sym­ pilers are thus the sole sources of semantic pro­ bol is known to the debugger. The DST also con­ gram information, but they make that information tains the correlation between program counter

PCA CALLS. COLLECTION INFORMATION SYMBOLIC PCA DEBUGGER

DEBUG SYMBOL TABLE EDIT COMMANDS

SOURCE FILES. COM PILE COMMANDS CMS COMMANDS COMPILERS LANGUAGE- AND LANGUAGE SENSITIVE CMS DEFINITIONS EDITOR DIAGNOSTICS. SOURCE FILES. TEMPLATES. CMS OUTPUT HELP FILES TEST SCRIPTS. CMS BENCHMARKS SCA SCA CALLS COMMANDS OUTPUT

ANALYSIS FILES DEC/Test Manager f-

SCA

KEY:

CMS - CODE MANAGEMENT SYSTEM

PCA - PERFORMANCE AND COVERAGE ANALYZER

SCA - SOURCE CODE ANALYZER

Figure 2 Information Flows between To ols

16 Digital Technical journal No. o February 1')88 Software Productivity Tools

values and source lines so that the debugger can corresponding to the current program location. If display the source code that corresponds to the user sees an error in that source code, he can specified run-time program addresses. PCA uses enter the EDIT command , which causes the the same information to display performance and debugger to invoke LSE in a separate process and coverage data symbol ically. to pass the current source location tO LSE. LSE The diagnostic information is passed from com­ positions the editing wi ndow to that source loca­ pilers to the LSE editor vi a a diagnostics fi le, as tion, and the user can correct the source code described earlier. This information includes the imme diately. PCA has the same connection to text of each error message along with the source LSE. After editing the code , LSE can invoke the location of the error. If the compiler suggests appropriate compiler, also in a separate process, error corrections, the suggested corrections are and pass along the edited source. included too. If the user wishes to browse through sources Language syntax (templates and placeholders) stored in a CMS library, LSE is able tO read those is passed to LSE through template fi les, and sources by calling CMS directly. There is also a language-specific help is passed to LSE via help RESERVE command in LSE which allows source fi les. Although template and help files are not modules to be checked out from a CMS library generated by the compilers as such, they are directly via the editor. Aga in, LSE calls CMS to do written by Digital's compiler developers. These this. The Te st Manager can also store test scripts files thus represent information flow from the and expected test resu lts in a CMS library and compilers to LSE. will call CMS directly to retrieve those fi les. The compilers create "analysis fi les" to hold A test run managed by the DECjTest Manager is all cross-reference and static analysis informa­ often a natural ve hicle for collecting perfor­ tion . These files can then be included in an SCA mance or coverage data. DTM therefore passes library, from which SCA can quickly answer information to PCA (via VMS logical names) that cross-reference queries and perform call-tree and tells PCA the name of each separate test script call-sequence analyses. The analysis file contains and the way the data should be collected. When the name, type , and scope of each symbol in the the developer later uses DTM tO review test user's program; its information thus partially results, he can invoke PCA directly from DTM to overlaps the DST information . However, the display the performance or coverage data associ­ analysis file also contains detailed information on ated with the current test execution. all symbol references, including the type of In all these cases, Digital's tool developers have each reference (read-reference, write-reference, created connections between tools whenever declaration, etc.), and detailed calling sequence they have been able to identify useful connec­ information on all procedure symbols. tions. Since most of these tools are developed in LSE and SCA are separate tools that can be the same organization and most tool groups ru n separately. However, they are strongly inte­ are physica lly close to each other, it is relatively grated with each other so that any SCA com­ easy for the developers of different tOols to work mand can be entered directly tO LS E. Also, together to develop the connections between LSE win-dows can be used to display cross­ tools that give the VA XfVMS programming envi ­ reference and static analysis information, and ronment its cohesiveness . cross-reference information can be used to auto­ matically position editor windows at specific Multilanguage Support symbol references. This tight coupling between One of the strengths of the VA XfVM S program­ the two tools makes them look like a single ming environment is its support of multiple tool to the user and gives the user a very rich edit­ programming languages. Software developers are ing and browsing environment for program thus free to choose the programming languages sources. In fact, SCA is seldom used alone except best su ited to their applications, and they can in batch runs. include modules written in different languages Other connections between tools pass more in the same program. At present, the environment modest amounts of information , but still help supports about a dozen languages. Only com­ provide a smooth , seamless fe el to the environ­ piled languages are supported ; interpreted lan­ ment. The debugger can display the source code guages have execution and editing models

Digital Tecbnical journal 17 No. 6 Fe bruary 1988 VAXjVMS Software Development Environment

that do not readily fit into a compiled-language sequence that may come up in a multilanguage environment. program, even though no one language has them The programming environment supports mul­ all. PCA and the debugger must both understand tiple languages in two ways. First, all Digital case-sensitivity, which occurs only in C. compilers generate code that adheres to the Multilanguage support thus complicates the VA XjVMS Calling Standard, wh ich standardizes design of most programming roots considerably. how programs call procedures and pass parame­ The tools must be designed to cope with a wide ters. Because all compiled languages use th is variety of language constructs. They must standard, modules written in different languages understand subtle semantic differences in appar­ can always call each other, provided both ently similar constructs in different languages. languages understand the data types of the They must also be very extensible since it is parameters. impossible to predict what languages they may Second, all the tools support multiple lan­ have to support in the future. As a result, the guages. The debugger can debug modules writ­ tools generally are very table-driven, and they are ten in any language whose compiler passes very dependent on having well-defined interfaces symbol table information to it. LSE can support with the compilers and the other tools. templates and placeholders for any language fo r However, there arc also substantial savings in which someone has constructed a template file, solving a given problem once for I 2 languages and it can review error messages from any com­ instead of solving it 1 2 times. Furthermore, there piler which passes diagnostics fi les to it. SCA is a lot of power in a multi language environment can provide cross-reference services and static because the programmer is free to choose the analysis for any language whose compiler creates programming language based on which language analysis files. The Common Data Dictionary is best for the application, and he is free to use (COD) can pass data definitions to any language existing program libraries regardless of what lan­ whose compiler accepts them. To fully partici­ guages they are written in. pate in the environment , each compiler must thus provide all the information needed by the Fu tureDirections various tools, and each compiler must call cer­ The VAXjVMS tools environment is st ill evolving. tain tools, such as COD. Some directions fo r future work include improv­ To support multiple languages, all tools use ing the integration between tools where suitable essentially the same implementation strategy. opportunities are perceived, providing fuller They define a single canonical representation for support for program design, providing better the data they need so that the same data from two configuration management tools, and continuing different languages is always represented the the trend to increasingly distributed software same way. LSE has only one template file format development. The environment is likely to main­ and one diagnostics file format. All compilers tain increasing amounts of project data and tO describe a given data type or programming con­ use that data for more kinds of project-control struct in the same way to the debugger. PCA uses and reporting functions. The increasing use of the same symbol information as the debugger. workstations and their capabilities is another SCA accepts only one format for its cross-refer­ trend that will affect practically all tools in ence information. If two languages pass a given the VAX jVMS programming environment to one piece of information to a given tool, they must degree or another. always do it the same way. However, all tools must also support the union References of all constructs in all the programming lan­ guages they support. The debugger must support 1. C. Mitche!J, "Engineering VAX Ada for a every data type that occurs in any language. It Multi-Language Programming Environ­ thus understands a numeric string type that ment," Proceedings of the ACM SIGSOFTj occurs only in RPG (a report generation lan­ SICPLAN Software Engineering Sy mposium guage) and tasking constructs that occur only in on Practical Software Development Envi­ the Ada language. SCA must understand every ronments, SIGPLAN No tices, vol. 22, no. 1 kind of cross-reference and every kind of caJling Qanuary 1987): 49-58

18 Digital Technicaljournal No. (, Februar)' I <)88 Software Productivity Tools

2. D. Knuth, The Te Xbook (Reading: Ad dison We sley, 1986).

3. P. Gi lbert, "Development of the VAX NOTES System," Digital Te chnical jo urnal (Febru­ ary 1988, this issue): 1 17-124.

4. G. Lupton , ''Language-Sensitive Editor ," Digital Te chnical journal (February 1988, this issue): 28-39

'5. B. Beander, "VAX DEBUG: An Interactive, Symbol ic, Mu ltilingual Debugger," Proceed­ ings of the ACM SIGSOFTjSI GPLAN Soft­ ware Engineering Sy mp osium on High­ Level Debugging, SIGPLAN Notices, vol. 18, no . 8 (August 1983) 173- 179.

Digital Te chnical journal 19 No. o Fe bmarv 1')88 Anne Smith Duncan ThomasJ. Ha rris I

Software Productivity Measurements

One objective of Digital Software Engineering is to build and maintain high-quality software products at reduced costs. To determine to what degree we are achieving this goal, Digital's Commercial Languages and To ols (CLT) Group is studying softwareproductivity in relation to their software development cycle. In today 's environment, engineers are build­ ing tools that assist in writing code and that automate project tasks. Fur­ ther, development teams share processes and reuse existing code. To mea­ sure the effectiveness of these and other steps, this group has begun to devise softwareproduct and project metrics and to collect project data. To date, .findingshave been madefo r three metrics: engineering productivity, deje ct rate, and cost to build.

Digital's CLT Group builds and supports high­ new one depended on the people who moved volume software products, including commer­ between projects. In the 1970s and early 1980s cial language compilers, software development few supported rools were available, and tool tools, and the VAXjVMS run-time library rou­ deve lopment was done at the project level, if at tines. CLT has been shipping native-mode all. Some processes were automated, most were VAXjVMS software products since 1978. not. Regression testing (testing that reveals Approximately one hundred software engineers, whether something that previously worked still managers, release engineers, operational ana­ does) was done by hand, bug lists were com­ lysts, and system managers work in this group. piled on blackboards, and debugging major inte­ User-documentation writers and editors. busi­ grations at base levels was difficult and time con­ ness product managers, and product marketing suming. The project members paid minimal specialists are also members of the project attention to tracing how and when things hap­ teams. pened, and they documented this activity on paper, if at all. TheIm portance of Productivity Another aspect of this culture was the sense Not long ago, computer users focused their that each project team had ro write all the code attention on increasing the productivity of hard­ needed for that project. This attitude meant ware because hardware was the component of that code to do common routines was duplicated greatest overall system cost. Although important, from project to project. Each team believed that software development costs were small com­ its problem was unique, that it could not share pared to the cost of running the software. There­ code with any other team. The belief was perva­ fore , software engineers stressed writing pro­ sive that each problem was different and that grams that ran fast, used small amounts of each project team had found the only appropri­ memory and disk, and minimized the number of ate techniques. compiles and tests needed during the develop­ By the mid- I 980s, our customers, and our soft­ ment process. ware engineers and managers started to pay much The Digital engineering culture allows each more attention to software costs, as the costs of software project team substantial freedom to software development and maintenance began to determine its own conventions, standards, and exceed the cost of hardware. Concurrently, cer­ infrastructure. In this culture, moving a success­ tain trends both inside and outside Digital were ful "process" from one completed project to a forcing us to shift our focus from improving the

20 Digital Technical Jo urnal No. 6 FebruarJ' 1988 Software Productivity Tools

hardware to improving the software development • We want our engineers to solve new problems process. These trends were as follows: in creative ways, and we want to solve each problem only once. • Marketplace expectations - Software cus­ tomers were becoming more sophisticated and • We want to reduce the costs of delivering new demanding. They needed software systems products.

that would provide them with a competitive • We want to reduce the costs of maintaining advantage in their marketplaces. They also and supporting the product set. wanted software that could be used safely by • We want all team members and their managers people of varied abilities and training. to feel more "in control" of their own work. • Increasing complexity - The complexity of And these objectives have to be accomplished developing software systems was increasing, within the constraints of our budgets and the and project management became more diffi­ availability of good software engineers. cult as projects became interrelated and some­ In order to determine how to better accom­ times were located in different facilities, plish these objectives, we needed to understand states, or countries. Communications between how we were doing at a point in time compared teams became increasingly difficult as the nor­ with how we had done in the past. This compari­ mal communications paths became clogged. son is frequently described as measuring pro­ • New technology - New technologies were grammer "productivity" or measuring software arriving at a faster rate and providing capabili­ engineering "productivity." ties we had not considered feasible 5, 1 0, or In February 1985, a graph was published 20 years earlier. under the topic of programmer productivity. • Software maintenance - Various studies indi­ The graph showed the actual and projected cated that from 50 to 70 percent of the cost rates of growth in lines-of-code per programmer 4 of software is spent on maintenance. 1 '2 Mainte­ from 1980 through 1990. It indicated that by nance includes defect correction, product 1990 the average software developer would pro­ support, and feature and capability evolution duce 1,075 lines-of-code per month, up from and extension. Software maintenance, espe­ 650 lines-of-code in 1985. cially defect correction and product support, This graph re-emphasized to us the need to consumes the human and hardware resources clarify how productivity should be defined and that should be used to build new products. measured . Productivity in software development

• Shortage of skilled software engineers - The is more complex than simply increasing the growing demand for highly skilled and trained lines-of-code produced by each programmer. software developers, projected to continue for The productivity of people, regardless of how it 3 the next 20 years, meant that experienced is measured, is only one part of the software engineers had more pressure to increase their development process. In any case, that projection output. caused us to seek answers to several important questions, such as the following: • High-quality software - The demand for con­ sistently high-quality software was increasing • What exactly is programmer productivity or as more businesses built their operations software engineer productivity? around software systems. These businesses had • How can we help our software project teams to little tolerance (nor should they have had) for become more productive, and how can we software systems with defects. measure whether or not their efforts and To address these trends, Digital's software engi­ achievements are better? neering managers and engineers have identified • How do we know if our products and the ways objectives for both the software product and the in which we develop those products are better software development process. now than in the past? What do we mean by

• We want to build and deliver high-quality, "better" ? dependable software products that meet The remainder of this paper describes some our customers' needs in predictable, cost­ answers to these questions and how the answers effective ways . were derived. A major benefit of this work has

Digital TechnicalJournal 21 No. 6 February 1988 Software Productivity Measurements

been the increased and continuing contribution • Requirements on the resultant software sys­ by all members of the CLT group to more precise tem, including reliability, storage use, and definitions of quality in our products and pro­ perfor mance cesses. Our findings indicate that CLT's produc­ • Characteristics of the development process, tivity has improved over the last seven years. And including the use of disciplined engineering our findings justify the costs of collecting and practices, the use of software rools, and the analyzing the data so that we can know whether availability of hardware for development and we are continuing to do better work. testing SoftwarePro ductivity The major factors that have changed at Digital during the last seven years are ( 1) increases in Software productivity encompasses more than size, complexity, and dependencies of products; just the programming of software products. A (2) increased use of shared tools; and (3) the software system is completed only when the fu nctional and performance requirements have sharing (reuse) of design, code, and documenta­ been met and when it is useful for the intended tion between projects. user. Therefore, the usual steps to reaching that The first fa ctor wou ld be expected to decrease state include analyzing the requirements; design­ the overall productivity of the project teams. The ing, coding, and testing the code; documenting second two factors would be expected to the system for both its users and maintenance increase productivi ty. software engineers; and providing training and Other papers in this issue of the Digital Te ch­ field support. The only way we can really exam­ nicaljournal describe specific tools used during ine productivity is to consider the software sys­ product development and give examples of 6 7 tem in the context of the entire development design and code reuse. · cycle. To ol Development and Us e Two major dimensions of software engi neering productivity are ( l) the change in quantity of In today's environment, each project team can software produced for a given period of time at a choose whether to use tools, define its project given cost and (2) the quality of the resultant infrastructure, and determine its own methods software system. for running the project. With the availability of Since a software system is built to solve a set of supported and useful tools, however , project members usually choose to automate some pro­ problems, nor as an end in itself, we need to con­ sider the product and the process in the context cesses, rhus avoiding the redundant effort of rein­ of each other. Then we can measure productivity venting designs and code that already exist. The and use the results to help us focus on whether paper "VAXjVMS Software Development Environ­ ment" (this issue) describes the tools and their our process is better , and what needs to be 8 changed in the development process. uses during the development process The same The quality attributes that are important to the tools are used across multiple development user of the software are important also to the phases. For example, the VAX DECjCMS Code engineer who supports and extends the software. Management System roo!, rhe version control sys­ For example, quality attributes include the tem, can be used fr om rhe beginning through the usability, usefulness, defect level and rate, and end of the process. At the beginning, this pro­ perfor mance of the software system. Also, we gram manages versions of the requirements docu­ must include the quality attributes that affect ments; ar the end. it manages the versions of fu ture costs; for example, the ease of modifying code, rests, command files, and documents. ro enhance or correct the software and the ease of Here are some examples of CLT's use of these porting the software to other hardware. tools:

• Previously, the procedures for building and Influences on Productivity controlling source code were usually listed on A number of studies indicate that software pro­ a blackboard, in a notebook, or in someone's ' ductivity is influenced by multiple factors. head . Now, the VAX DECjCMS and VAX DEC/ These factors include MMS Module Management System tOols auto­ • Personnel and ream capabili ties and experi­ mate the versioning of source code, the identi­ ence fication of modules that belong tO a particular

22 Digital Tecbnical journal No. o February I

base level or version, and the build processes. have been planned and designed specifically to The library structures and MMS build proce­ provide functions that are needed in multiple dures also serve as project documentation. products.

• Regression testing is now simplified by the Software Metrics VAX DEC/Test Manager software. By support­ The best way to gauge improvements is to have ing attribute-based subset test selection, this a set of measurements that compares how things tool makes it easier for software engineers have changed over time. A software metric working on optimizations or defect correc­ is a quantitative way to characterize an attribute tions to quickly run subsets of a major test sys­ of either the software system or the software tem. Being easier, testing is done more often. development process. For a metric to be mean­ Many projects routinely rebuild the project ingful, there must be a way to measure these code (using CMS and MMS) and run either the attributes consistently and objectively. Then entire test system or a part of it every night. various software systems and development The next morning, the software engineers projects can be compared to themselves over know immediately if their previous work time and to each other. (That assumes other caused a new problem or regression . Projects variables remain constant; for example, similar that have adopted this process for builds and types of organizations building similar types of tests have almost completely eliminated the software using similar methods and processes.) many hours of integration at base level. Only when metrics have the same definition (and • The VAX NOTES system, a distributed confer­ therefore are measured in the same way) should ? 10. 1 1 ence tool, helps in automating and tracking they be compared. . project design discussions and decisions. The Any metric process should guard against

project members can open a separate topic • Measuring only one dimension; for example, dealing with an issue. Subsequent responses quantity alone without considering quality; from the project members, and perhaps field time only without regard for the product support personnel (world-wide) , are available delivered (For the resu lts of a study on the to all interested parties on Digital's internal effects of measuring one dimension or crite­ network. AJthough the NOTES conference does rion in favor of another, see We inburg and not replace meetings of the project team for Schulman's study on computer programming 2 design discussions and reviews, it does goals and performance.1 ) provide an easy-to-use mechanism for describ­ • Measuring for the wrong reasons; for example, ing the history of the discussions. NOTES helps using measurements to appraise an individual to inform new project members of the pro­ ject's historyand status. • Comparing measurements with too many vari­ ables; for example, process control applica­ Reduced Redundancy tions are different than payroll applications; Our software engineers now search for code, the quantity of code per unit of time is less for designs, additional tools, and documentation high -level languages vis-a-vis assembly lan­ that can be reused. Both managers and engineers guage code consider reused code as an investment in In this paper, we discuss two sets of software design, programming, and testing that has metrics: product metrics to describe the software already been paid fo r. Moreover, the support for itself, and process metrics to reflect the process that code has been planned and is in place. of software development. Reusable run-time components have been used and available since the first version of the Software Product Metrics VAX jVMS operating system in 1977. The VAX Product metrics (also called system metrics) Common Run-Time Library (RTL) is used by all describe the attributes of the software system products. This library is a group of approxi­ or components of a system, and the related mately one thousand software routines used documentation, tests, and system control infor­ by hundreds of software components and prod­ mation (for example, command language batch ucts for run-time support of common functions. streams) . Size, usability, maintainability, number Recently, major components outside the RTL of defects, and performance are all attributes of a

Digital Te chnical journal 23 No. o February I 'J88 Software Productivity Measurements

software system. For this study, we used three in months that the software engineers and pro­ software product metrics: size, defects, and ject leader spent in the various phases of the defect rate. project. Thus cost Cis defined as

• The size S of a software product i is defined as C; = DM1p1 + DM1p2 + DM;P.J

+ 5, in which DM equals the number of months = 5,. 5I 1000 directly charged to the project by the software engineers and project leader, and P1, P2, and in which Se equals the number of lines of P3 are phases 1, 2, and 3, respectively. code , including data declarations, and S, equals the number of lines of comments. Each • The engineering productivity EP1 is defined as line is counted as one regardless of the number S; of operators, operands, and comments that the EPI = · C; line may include. Include fi les are counted once, and reused code shipped with the This metric provides a means for comparing product is counted . Blank lines are not various projects by normalizing the size and counted. Project tools, tests, test data , and the cost of each project. control fi les are also not counted. Indicators of Improvement • The number of defects D for a product i is To answer the question, Are we doing better now defined as than in the past1, we have tO gather data on older D; = Db + Dd + D, projects and then compare it with data from more recent projects. in which Db is the number of customer reports answered as a "bug" or "correction given," Dd Collecting the Data is the number of customer reports answered as "documentation error," and D, is the number For the produns in this study, we present three of customer reports answered "restriction on sets of data: size, number of defects, and develop­ the use of the software ." ment cost. We chose version 1 products and other major product versions in which more than 50 • The defect rate DR1 , which also describes the percent of the delivered code was new. Al l prod­ software product, is defined as ucts in this study were developed on the VA X/ D VMS system, and all but one were written in the DR1 = 1 S, VAX BLISS-32 language . Shipment of these prod­ ucts to custOmers began during the period from The defect rate provides a way to normalize late 1980 through summer 1987. the data associated with a particular product Collecting data from older projects was some­ such that the defect rates of multiple products what di fficult: there were few common tools that may be compared without regard for variances we cou ld use for data collection, files were fre­ in product size. quently lost, and memories of the project team members were not always clear or accurate. Some Soft ware Development Process Metrics projects did keep data, and these are included in Process metrics describe the attributes of devel­ the comparisons. Many projects of the late 1970s oping the software system, product, or compo­ and early 1980s, however, did not collect or save nent. Attributes of the process include the cost of needed data; so there are fewer data points for development (in human resources, hardware products shipped before 1983. Many of the early resources, and calendar time) , the predictability products developed in CLT were not written for of the schedule and delivered software capabil­ the VAXjVMS system and have not been included ity, the number of design and code reviews, and either. the length of time to respond to a customer Collecting data from recent projects was inquiry or problem report. For this study we are easie because most of them used the same tools using the cost and engineering productivity met­ as part of the project infrastructure. We were rics. able to collect product lines-of-code data in a • One definition for cost C of the developmenr consistent manner by using routines wriuen in of the software product i is the length of time the VA X SCAN language to access the project

24 Digital Te chnicaljournal No. 6 February 1988 Software Productivity Tools

I= VA X DECjCMS libraries. Customer-reported (fJ 0 defect data has been collected for many years >-0 4.00 t=,_ .. through a database that stores information from �z t-UJ 3 50 0::2; Software Performance Reports. The data for time ::>a_ 3.00 DO and effort to build the products was collected 0-' 2.50 C:UJ ll-> project phase review and accounting UJ 2.00 from (!) zQ_ 150 .. records. -UJ C:N .. .. UJ­ 1 00 UJ The Defect Rate Me tric UJ Z 0.60 ttW 0.40 Defect rate is one measure of the quality of the oSj .. t­ 0.20 software products that we ship to customers. Var­ u ::> 0 Cl 1980 1981 1982 1983 1984 ious published studies indicate that the "typical" 0 defect rate for Am erican industrial software is [ DATE OF RELEASE TO CUSTOMERS. 13 PRODUCTS 10 defects per 1 ,000 lines of code, 13 and that th:� rate varies from 5 to 30 defects per KLOC . Figure 2 Product Defect Rate

Digital Technicaljo urnal 25 No. 6 Fe bruary 1988 Software Productivity Measurements

We attribute the improvcm<..:nt in quality to 0 0 an increase in the availability and use of roots, (jJ 87 VA)( 1- 300 .. especially the DECjCMS, VAX DEC/MMS, z w VA,'( DEC/Test Manager, VAX Language-Sensitive ::2 250 ::2 Editor, and VAX Source Code An alyzer tools. (The 0 200 u last three tools first became available for internal �0 85 � 150 85 -- -- iii � .. ..__ Digital use during the second half of 19H4.) Dur­ 85 -- w 100 80 86 .. ing project quality reviews, each project ream is 0 86 84 -- -- ...... -- 50 83 questioned about the use of tools, which has led 8 84 -- 83 "- "- ---- to an increased usc of various tools during the 0 (f) 20 40 60 80 100 120 140 150 deve lopment process. An unpublished survey w z COST (DEVELOPER MONTHS), 14 PRODUCTS taken of the current CLT project groups in the :::!.

summer of 1987 ind icated that 100 percent of KEY:

the projects responding use the VA,'( DEC/CMS -- - PRE-1 985 TREND tool, 80 percent of the projects use or plan to us<..: -- POST-JANUARY 1985 TREND the VAX DEC/MMS root, I 00 percent of th<..: projects use or plan to use the VAX DECj'J'est Figure 3 Relationship ofProduct Size and Cost Manager tool, 100 percent use i nteracrive editors with 86 percent using the VAX Language-Sensi­ tive Editor roo!, and 79 percent use or plan to usc This data he lps both the project team and their the VAX Source Code Analyzer tool . managers to gauge how a project is doing. One use of the clara in Figure 1 is to check the va lidity Th e Cost- to -Build Rate of the estimates of a new project. We can usc this We also examined the relationship between the data to answer questions such as, Do the fore­ size of the product and the cost to build it. Fig­ casted costs appear to be realistic given the his­ ure 3 shows the data for the cost to deliver tested tory of past projects in this organization' and debugged code for 14 products. The hori­ The data also provides a known base that can zontal axis indicates the project cost C fo r devel­ be used for comparisons with newer clata when oping the product, the vertica l axis represents there arc changes in the methods, tools, or train­ the product size S, and the date at each data ing of engineers and their managers. Using these point is the year that the product began shipping comparisons, we can determine if our process to customers. Many studies show that the cost and products are getting "better" or not. of software has a direct relationship to the size We arc defining additional metrics, searching ' of the software produc<..:d. Figure 3 indicates for those that add ro our knowledge about the that CLT has delivered products with lower quality of the product and the process. Acldirional cost since January 1985 than for the period software proclucr merrics include the mean-rime­ 1980 ro 1984 . For example, in 1980, one project to-failure, module and product complexity, and

cost C = 145 ro design, implement, rest ancl maintainability and extendability of product del iver a product with size S = 83. In 1987, a code. documentation. ancl test systems. Ad di­ project delivered a product with a cost C = 131 tional development process metrics include the with size S = 294. That is a 254 percent increase ratios of defects fo uncl by inspections. unit rest­ in product size with a reduction in cost of 10 per­ ing, pre-customer testing, ancl customer resting, cent. In other words, a product that is over the mcan-time-ro-fix a problem, the cost ancl three-and-one-half times the size of another was effort for the various development phases, and produc<..:d with less cost. We consider that as one the rare of successfu I test com pterion . indicator of improved productivity. To col lect the data, we use the software deve l­ opment tools that the project reams usc as part Continuing Improvement in the Fu ture of their project infrastructure . For example, the By collecting project and process information. VAX OEC/CMS program maintains a complete we hav<..: the clara to compare past projects with history of the activity of a library, including add i­ new proj<..:crs and to compare projects with them­ tions and changes, when mack , by whom, and for selves over time. We can use that data to evaluate what reason . Using the VAX DECjCMS history-fi I. e the estimates and progress of current projects. data. we can analyze the reasons for changes to

26 Digital Te chnical jounwl No. (i Fehrttar)' I 'JR8 Software Productivity Tools

modules and the rates of changes and deliveries References fo r code. documents. and tests. A<> another exam­ 1. R. Kn ight, "COBOL Still Strategic After All ple, we can track test fa ilures and successes using These Years," Software Ne ws 7(7) Oune the VAX DECjTesr Manager software. The test I 987): 58-64 . manager tool also provides a history of additions and changes to tests, thus yielding data about the 2. R. Hall, "Seven Ways to Cur Software Mainte­ arrival of tests into the test system. nance Costs," Datamation Ouly 15, 1987) : 81-84 . Summary 3. "Help Wa nted," Business Week (August 10, The metrics and data discussed in this paper 1987): 50. demonstrate that one group in Digital is building higher quality software products at lower costs. 4. H. Davis, "Measuring the Programmer's Pro­ Particularly noteworthy is the increased quality ductivity," Electronic Engineering Man· of products, leading to reduced costs of mainte­ ager (February 1985): 44-48. nance. In several cases, one project ream is able 5. B. Boehm, Software Engineering Econom­ to support , maintain, and enhance one product, ics (Englewood Cliffs : Prentice-Hall, 1981) while providing support and maintenance for another. 6. S. Greenwood, "VAX SCAN : Rule-Based Te xt One central question we asked earlier was, Processing Software," Digital Te chnical Do we understand what software development jo urnal (February 1988, this issue): "productivity" means' Our answer is. More than 40-'50. we did in the past, but we have more work to do. 7. S. Grass, "Development of a Graphical Pro­ We look at software development "productivi ty" gram Generator," Digital Te chnicaljournal to include more than the productivity of individ­ (February 1988, this issue) : 10 1-109 uals or the project team. Software development productivity also includes the quality attributes 8. B. Beander, "VAXjVMS Software Develop­ of the product. ment Environment," Digital Te chnicaljour­ Quality and productivity improvement arc nal (February 1988. this issue) : 10-19. ongoing. They have become parr of our way 9. T. Capers Jones, Programming Productiv­ of doing business. The managers and software ity (New York: McGraw-Hill, 1986) . development team members consider software metrics and measurements as additional tools that 10. S. Conte, H. Dunsmore, and V. Shen, Soft­ help them ro manage projects. The ability ro col­ ware Engineering Me trics and Models lect data from the productivity tools themse lves (Menlo Park: Benjamin Cummings, 1986) . has assisted in this process of change . Thus data 11. R. Grady and D. Caswell, Software Metrics: col lection has become a nonintrusive by-product Establishing a Company- Wide Program of normal tool use. By using consistent collection (Englewood Cliffs : Prentice-Hall, 1987) . methods, team members can compare the data across projects in the organization. However, that 12. G. We inburg and E. Schulman, "Goals and data is never useel to measure an individual's pro­ Performance in Computer Programming," ductivity; rhe focus is always on the software Human Fa ctors , (16/1) (1974): 70-77. development process and software systems that 13. B. Beizer, Software Systems Te sting and we deliver. Quality Assurance (New Yo rk: Va n Nos­ Acknowledgments trand, 1984 ). We wanr to express our appreciation to the 14. W. Myers, "Can Software for the Strategic present and past CLT project team members and Defense Initiative Ever Be Error-Free'" Com­ managers for doing good work, for their um:ncl­ puter (November 1986): 61-67. ing search for improvement, and for their efforts with data col lection .

Digital Te chnicaljournal 27 No G Febma ry 1988 GlennLupton I

Language-Sensitive Editor

The VAX Language-Sensitive Editor, a component of the VAXjVMS program development environment, is an advanced text editor specificaUy designed to help programmers develop and maintain program code. Developers of the product required that it include a simple interface that wouldbe read­ ily accepted by the VMS user community, language-sensitive features that improve programmer productivity, support for multiple languages with the same user interface, and support for user extensions. In addition, the editor had to mesh well with the existing program development environ­ ment offe red by Digital. This paper provides the background of the devel­ opment effo rt, a close look at the design of various features and some of the insights gained, and a summary of the current status and future directions for the environment provided by the editor.

Background can determine the syntactic context of the cur­ re nt editing position and offer assistance on the In late 1982, the Tec hnical Languages and Envi­ language constructs or the identifiers that are ronments Group started a project to specify a Pro­ va lid at that position . There are also operations gramming Support Environment (PSE) . Several that can be performed conveniently on a parse Digital products related to supporting program tree, such as cursor movement by syntactic ele­ development were already ava ilable or under ment and elision, the suppression of selected development. The PSE project outlined a number program details in the display. Although tree of components needed to complete Digital's PSE editors can offer some ve ry useful programming offering. One component was an ed itor special­ support, their disadvantages are significant. ized for program development. The editors being Their main drawbacks relate to their user inter­ used to deve.lop software typically did not con­ fa ce, their performance, and their specialization tain any special features to support the program­ to a single programming language or subset of a ming process. The PSE project developers saw language . this as an interesting opportunity, and the pro­ Making mod ifications ro source fi les is often gram editor became the first target of the PSE awkward using a tree editor. Since tree editors development effort. insist that the contents of the fi le must always correspond ro a we ll-formed syntax tree, there Research of Progra m Editors are serious constraints on the intermediate At the time, various universities were research­ forms that the contents of the fi le can assume. ing program editors, and a number of papers In one tree editor, the language keywords are appeared in technical journals. The program­ not items that can be edited by the user. A ming-related features the editors provided often simple change, such as replacing the keyword varied with the language they supported, but WHILE with UNTIL, requires a number of they typically provided interactive syntax check­ steps, including saving the loop-body, deleting ing and special commands to insert language the WHILE-loop. and reconstructing it as an 1 statements. Most of these were tree editors, UNTIL-Ioop. which model a source fi le as a syntax tree, rather Another tree editor copes with such difficulties than text editors, which model a fi le as a stream by integrating a simple text editOr with the edi· of characters. An advantage of tree editOrs is that tor. 2 Users may edit a portion of the source file as syntax errors and some semantic errors are pre­ text by clipping a syntax subtree into the text cluded or diagnosed immediately. Tree edirors editor. Of course, thc benefits of the tree editor

28 Digital Te chnical journal No. 6 February I ')88 Software Productivity Tools

are not ava ilable when using the text editor fa cil­ does not place such a restriction on the contents ity. When returning the clipping tO the source of the file. Since the file may contain any frag­ fi le, the tree editOr parses the clipping to verify ment of a compilation, it might be impossible to that its syntax is valid at the point where it construct a pa rse tree for the contents of the fi le, is be ing inserted . In both of the above editors, and so, impossible to edit the file using a tree there are constraints on how users make changes editor. It may also be difficult to construct a to source code. Moreover, they must think in parse tree fo r a source fi le that contains #include terms of changes to a syntax tree, even if that is statements. not the most natural way to think of a particular Still another drawback is that tree editors can­ editing task. not be used for all editing needs. Most tree edi­ Tree editors rely on pretty-printers to convert tors are tailored to a particular language. Users their internal parse tree representation of a have to use other editors for other languages and source fi le into text for display to users. When for text files. the formatting style of the pretty-printer is agree­ Product Requirements able to the user, this is a time-saver. However, every pretty-printing algorithm has limitations, The primary requirement of the program ed itor is to improve programmer productivity by sup­ including cases that the user can format more porting program editing with language-sensitive readably. Thus, another drawback of tree editors is that users must accept the formatting style of features. Based on the insights gained from the above research, the PSE project assembled a list the pretty-printer even when they would prefer a of additional requirements and design consider­ diffe rent style. Tree editors also consume considerable com­ ations. puter resources, both processing power for Ease of Us e parsing and memory for storage of parse trees. Users must view the PSE editor as being easy to Only single-user systems or lightly loaded, mu lti­ learn. In particular, since most of our customer user systems could accommodate the resource base was using EDT, the progra m editor shou ld " requirements of these editors. have an interface that is compatible with EDT. Tree editors have difficulty in supporting cer­ Given that they know how to use EDT or tain language features, such as macros and another text editor, there should be very little conditional compilation. For example, the fol­ that users have to learn in order to begin using lowing fragment of C code uses conditional the program editor productively. compilation ro call the function PROCESS with Ad ditionally, the PSE editor must not compli­ an extra parameter when it is compiled for cate the users' programming environment by TARGET2: forc ing them to use the PSE editor for editing

process( source files and another editor for other text

input fi les. They should be able to use the PSE editor as

#if target2 a replacement for their current text editor. The

, length PSE editOr must enhance the programming pro­

#endif cess with minimal changes to users' editing > i styles. Such language features pose considerable prob­ Multiple Language Support lems in both the construction of a parse tree and The editor must support a variety of program­ pretty-printing. Typically, tree editors place ming languages. FORTRAN was the most widely restrictions on the usage of such constructs. 3 used programming language in our customer Another language fe ature that poses problems base; but many customers were using other lan­ for tree editors is file inclusion. This fe ature guages, and a significant portion were using more inserts a specified source file into the compiler's than one high-level language . Some customers input stream, temporarily suspending input from would soon start using the Ada language, and a the original source fi le. The #include preproces­ program editor that would help users make the sor control line in the C language is an example transition tO Ada wou ld be an important part of of this. Although a fi le specified by #include typi­ Digital's Ada Programming Support Environment cally contains only declarations, the language (APSE) . The editor must also make it easy for

Digital Technicaljournal 29 No. 6 February 1988 language-Sensitive Editor

users to work on software written in more than Given the above requirements, the PSE project one language and ro work on multiple programs deve loped a prototype program editor called wri rren in various languages. Therefore, the edi­ EDITH. This was a text ediror with an EDT-style tor must support a variety of languages with a interface. It maintained only a textual representa­ common user interface. tion of source code and could be used ro edit any text file. Fl exibility EDIT H supported program editing by supply­ In addition to supporting a variety of languages , ing templates for source constructs. A template is the editor must provide access to all the features usually a skeleton for a language construct. Users of each language. The editor must not limit users can in struct the editor tO insert a te mplate into a the way in to a subset of the language nor restrict source file. The foll owing is an example of a tem­ which language features are used. Users must be p.lare for an IF statement: able to construct any lega l program and format the code as they wish. if {cond ition} then {statem ent} ... Extensibility [elsif _partl ... Users must be able to modify and enhance rhe [else_part l editor to suit their preferences and their special end if; needs. The popularity of EMACS, a programmable Templates provide several henefits: 5 editor , was evidence of the need for this. Also, • Correct keywords and punctuation many customers had tailored EDT for specia l • uses. Proper formatting and indentation

In addition to providi ng a malleable editing • Consistent case conventions interface , the editor would have to accommodate • Source entry using fewer keystrokes user-defined languages and user modifications ro Templates typically contain syntactic markers the languages provided by Digital. Users should indicating where other program elements can not have to change their coding style or their pro­ or must appear. Templates also aid the user in ject coding conventions when switching to the entering correct source code . Markers, such as PSE editor. [else_part] and lsrare menrj , have templates or Pe1Jormance menus of templates associated with them rhat the Performance was an important design consider­ user can select. The user is free to use the text ation. Some users complained that the perfor­ ed iting capabilities of the editor to enter and mance of EDT was barely acceptable on a loaded modify program rext. By providing language tem­ timesharing system, which was the expected plates, the editor helps the user develop syntacti­ environment for the PSE ediror. Developers work­ cally correct programs, without restricting the ing on a text editor for such a target system were contents of the source file, as docs a tree editor. then wrestling with the problem of echoing char­ This design for syntax su pport also performed I acters as fast as the user typed the m. The PSE we.ll and could easi y accommodate a variery of editor must support program editing without languages. requiring significa ntly more system resources Additionally, EDITH interfaced to a parser to than typical text editors needed . perform syntactic checking and to report errors. An explicit PARSE command would pass the TheEDI TH Prototype source code ro a language-specific parser. As Al though program edirors that viewed a source errors were detected, the parser passed diagnos­ file as a parse tree showed promise and had tic information back to the editor. The editOr dis­ advantages over editors with a text view of a pro­ played this information ro rhe user, who could gram, their disadv antages were significant. An then step through the errors one at a rime. As alternate approach was ro add language-sensitive each error message was displayed, rhe editor enhancements to a text editor. This approach had positioned the editing cursor at the point in the been meeting with some success internally with source where the selected error was diagnosed. ' enhancements to the EDT editor." and externallv The user could the n make appropriate correc­ ' (' with EMACS. and rhe Z editor This is th� tion s. This interface allowed users ro find syntax approach that the PSE project chose to prototype . errors without leaving the editor to run the com-

.10 Dip,ita/ Te chnical journal No. (j Fe hmar1• 1988 Software Productivity Tools

piler, and provided a convenienr interface for Te mplates fo r Language Constructs reviewing the errors and locating the correspond­ To provide support for entering source code, LS E ing source code. adopted the template approach prototyped in A number of ideas prototyped in EDITH EDITH. In a very basic sense, a template is simply evolved into fe atures of LSE. text that is inserted into the editing buffer at the user's request . Within that text may be places Fo undationfo r the Implementation where the user will have to enter additional text Coding for the VAX La nguage-Sensitive Editor to complete the material. begin in Ja nuary, 1984 . At that time, a prelimi­ In the context of a programming language, nary version of the VAX Text Processing Util ity, the text of a template is usually a sequence of VA XTPU, was available. VAXTPU is a text ed itor lexemes, such as keywords and punctuation , users can program using a procedural language. that form some piece of the syntax of a program­ The language supplies a large number of built-in ming language. Places where the user must fill functions that are used to manage fi les, windows in more syntax to have valid program text are (portions of the terminal screen) . text, and even indicated hy syntactic markers called placehold­ subprocesses. VA XTPU provides compilation and ers. Te mplates are inserted inro a source fi le execution fa cilities for programs written in the by an EXPAN D operation . Te xt inserted as part VA XTPU language. To use the VA XTPU editor, of a template can be edited like any other text in the user needs a user interface written in the the source fi le. VA XTPU programming language. The first release of VAXTPU provided rwo interfaces. One inter­ To kens face was an EDT emulator; the other. called EVE, The keyword or text that a user types to identify was heavily oriented tO the keyboard for VT200- the template he wants inserted is called a roken. series terminals. VA."'\TPU was to replace EDT as An EXPAN D operation replaces the token with the VMS editor. Shortly after the release of the corresponding template. For example, typing VA XTPU, LSE would have to address compatibil­ IF into an Ada source fi le and then pressing the ity with VAXTPU and EVE . as we ll as with EDT To EXPAND key inserts the fo llowing into the file: meet this new re quirement, and the requirement for the editing capabilities to be user-tailorablc, if {condition} then 5 n LSE wou ld be a superset of VA XTPU. The high­ { tat erne t} ... performance design of VA.XTPU would also help [elsif_par tJ. .. LSE meet its re quirement to perform well on [else-par t] loaded timesharing systems. LSE wou ld also use end if; VA XTPU to provide EDT-compatible functions. A token is usually a keyword that introduces a As a result, the LSE project used the VA.XTPU language construct or the name of a predefined sources as a base when implementing LS E. function cal l. Only one EXPAN D key is needed to Although a number of the VA.X TPU source mod ­ expand any token into a template. Users do not ules had to be changed to interface ro LSE func­ have to type the entire token name before press­ tions, most of the VAXTPU code was used ing the EXPAN D key, just enough ro uniquely unchanged. Thus, LSE used the VA.XTPU code as a identify the token. If the prefix that they enter library of run-time rou tines for terminal interac­ marches more than one token, the editor displays tion and for support of the user-programmable a menu of possibilities from which to choose . features . This gave the LSE developers access to This style of interaction has proved to be easy a powerfu l set of functions for text manipulation and effective. Users have access to a large number and screen management. At this point, the code of rempbres, including templates for calls to for the EDITH prototype was abandoned. all the VAX jVMS system services and run-time library routines , and can access any template Syntax Support quickly and easily. One obvious way an editor can support pro­ gramming is ro assist in the construction of syn­ Placeholders tactically correct programs. To provide such As mentioned earlier, placeholders are the syntac­ assistance, LSE adopted the template approach tic markers that arc inserted into the editing prorotyped in EDITH. buffer as part of a template. Placeholders are

Digital Te chnical journal 31 No. 6 Fe bruary I 988 Language-Sensitive Editor

distinguished from program text by the brackets appears at a point where the user must enter ([ ]) or braces (! }) that de limit them. They one or more statements, whereas [statement]. are stored internally as text, and fi les that contain appears where the user may optionally enter placeholders are simple text files that require statements. no special processing for operations such as printing. Editor fu nctions that operate on place­ Expanding Placeholders holders recognize them by the characters chosen Li ke tokens, placeholders may be expanded . as delimiters for the language . For languages There are three types of placeholders: nontermi­ in wh ich braces or brackets arc valid lexemes, nal, menu, and terminal. other delimiters must be chosen. Typ ically, Nonterminal placeholders expand into a tem­ the delimiters in such cases are formed using plate . For example, in the above IF template, the braces and brackets in conjunction with some (else_part] placeholder expands into the fo llow­ other character, such as a tilde (for example, ing simple template: ! �statement�}) . else Op tional and Required Placeholders {stateme nt} . A placeholder may represent optional or requir­ A menu placeholder expands to a display of ed language syntax. The optional or required a set of choices. The !statement} ... placeholder nature of the placeholder is indicated by the above is a menu placeholder. Expanding this enclosing de limiters. In the IF example above, placeholder results in the display of a menu of the !condition} placeholder appears with braces, statements, as shown in Figure 1. delimiters that indicate tO the user and the edi­ Expanding a terminal placeholder simply dis­ tor that it is a placeholder for syntax that is plays a description of the program syntax that required at that point. The (else_pan] place­ must be entered at that placeholder. A typical holder appears with brackets as delimiters, example is the !identifier} placeholder; the indicating that the else_pan is optional. The expansion provides information such as the char­ placeholders terminating with ellipses ( ...) are acters that are legal in an identifier and the maxi­ called list placeholders. They indicate where mum length of an identifier. users may enter more than one of the correspond­ ing language constructs. Braces and brackets Filling in Placeholders indicate whether a list placeholder is for Once users have the level of detail they need from required or optional syntax. Thus, !statement} .. the templates, they can fi ll in the missing syntax

function PRIHE l'etul'n BOOLEAN is begin for I in 2 . . NUHBER/2 loop if (NUHBER - < • !)) = 0 then lll!tatel!lent l ... [e I si f_pai'tl ,., [ e I se_pa rtl end if; end loop; [ s tate111entl .•. return !expression!; [exception_partl •:n . . > abort : Prevents an� ful'ther l'endezvous with the named tasks accept : Spec ifies the resulting action of a task entl'� ca ll {assignl!lent_statel!lent l : Assigns the va lue of an expl'ess ion to a val'iable {block_statelllentl : An optiona l]� named block of dec l al'ations and state111ents CISI! : Chooses an action based on the value of an expression de la� : De l a�s execution fol' specified a111ount of time lentl'�_ca ll_statelllent l : Calls a task entl'� .. ;., . .

13 lines read fro111 file LSES: [DEHOJPRIHES .ADA; I

Figure 1 Editor Display with Choices fo r < stat ernen t} ...

32 Digital Te chnical journal No. 6 Fe bmarv I ')88 Software Productivity Tools

by typing text on the placeholders. In some whitespace wou ld change the indentation of the cases, such as an assignment statement, users wi ll line. In this case, it erases the whitespace that fol­ likely be sufficiently familiar with the syntax of lows the placeholder. the language tO type the complete statement on a In a number of cases, lines must also be erased . statement placeholder, rather than to expand the In the example of the IF statement shown earlier, statement placeholder and select the assignment after erasing [else_part] the editor must also erase template from the menu. In other cases there wi ll the now blank line . This can be slightly more be no more detail that can be supplied by the complicated, as in the fo llowing example of an IF template, either because the template designer statement in Pascal: did not provide the detail or because the user has IF {expression} reached a true terminal in the language syntax THEN such as {identifier} .

Digital Technical journal 33 No. 6 February 1988 Language-Sensitive Editor

allowing the keyword sequence for access ing a For example. to get from a placeholder for a particular node of the tree to he associated with a statement to a template for a while-loop might token or placeholder. For example. the token for req uire going from statement menu ro control­ the FORTRAl'J cosine fu nction, COS, has the statement menu to loop-statement menu to string "FORTRAN INTRINSIC COS" associated pn.:tested-loop menu ro while-loop. It wou ld be with it. When the user presses the language-help much better to include the template for the key while the cursor is on the token COS, the while-loop in the menu for the statement place­ information for COS will display. Similarly. the holder. In general, templates can be vastly !statement} placeholder can have a string associ­ improved by eliminating intermediate menus and ated with it that accesses a node of the tree whose red ucing the number of expansions required to suhnodes describe the diffe rent rypes of state­ access a template. ments for a language . This allows users to get In some cases, the formal syntax definition help on language constructs withou t leaving the for a language does not include some simple editor and wi thout typing HELP commands. semantics that the templates should include. For example, the syntax may define BEGIN and END Defining Language Support statements but not include their relation . The The ed itor supports a special definition language template for the BEGIN statement should include fo r describing a language to the editor and speci­ the matching END statement, plus placeholders fying the tokens and placeholders. For each of for the syntax that may appear between them. the languages supported by Digital, the corre­ There arc a number of cases like this where com­ sponding compiler project develops the lan­ mon sense and coding standards should influence guage, token, and placeholder definitions, and the template definitions. The template for a pro­ the on- line HELP library. The definition language cedure declaration is another good example. is documented for customers so they can modify Often a user site fo llows a coding standard that the definitions supplied by Digital or add defini­ requires a procedure to have a specially format­ tions for other languages. The editor accesses ted com ment associated with it. Also, the coding these definitions when it is invoked. By conven­ standard might require that the body of the pro­ tion on the VAX jVMS operating system, source cedure be a BEG IN/END block. although the fi les for diffe rent languages are distinguished by language does not require this. The predefined a naming convention for the file-type portion of templates for a language can be a greater pro­ the fi le specification. The description of each lan­ ductivity aid if they are tailored to the way in guage includes the fi le types that apply to which the language is typically used on a particu­ that language. The editOr uses this to determine lar project. which set of language definitions ro associate Flexibility in Detail of Te mplates with a source file. Users may edit several source The level of detail that templates provide is fi les written in diffe rent languages in one ed iting up to the author of the template definitions. session . For example, the p.laceholder !condition} cou ld

Te mplates Stray fr om BNF be a terminal placeholder that expands into a description of Boolean expressions in the lan­ A formal definition for the syntax of a language. guage; or it could expand into more detai led rem­ such as a BNF description, provides a good refer­ plates and menus that provide the syntactic ence when developing templates for a language . elements of a Boolean expression, such as AN D, However. strict adherence to such a description OR. and n:lational operators can produce templates that are very tedious to Highly detailed templates are especially useful use. in declarations, for example: The syntax for a language may be defined using many intermediate productions. A straight­ {iden tifier} ... : forward conversion of this grammar into tem­ [constant] [subtype-indication] plate definitions will resu lt in a menu place­ [init ial_va luel; holder for each production that has severa.l Compared to a control construct such as an IF alternatives as a right-hand side. Therefore many statement, the syntax of dec.larations is often menu placeholders wi.ll have elements that are complex, and likely to have a large number of also menus. options. Detailed templates help users with this

34 Digital Te chnical journal No. 6 February 1988 Software Productivity Tools

complexity by presenting menus of choices and it also provides tight integration with the fo llow­ placeholders for various declaration options. ing fa ci I ities:

• VMS language compi lers Observations on Us ing Te mp lates 7 • VA)( Source Code Analyzer One of the significant aspects of the editor's sup­ 8 port for entering source code is that its use does • VAX DECjCMS (Code Management System) not interfere with the use of the editor for arbi­ trary text manipulation. There are no restrictions Compiler Interfa ce on the intermediate contents of the text buffer The editOr provides users the means for diagnos­ when reorganizing or restructuring code. Text ing syntactic and semantic errors by interfac­ man ipulation operations that users have coded ing with the language compi lers. Each compiler themselves in the VAX TPU language are also for the languages that support LSE has been available. The fi nal formatting of the source is up en hanced to generate diagnostic fi les that specify to users. Te mplates supply fo rmatted language the compi lation errors and the related source constructs, but users can reformat the program locations. The editor creates VMS subprocesses co text without restrictions. This is not possible in perform the compilations. This is a very different most language editOrs. design from the EDITH protOtype that interfaced In practice, users frequently take advantage by procedure calls directly ro a parser. This of the source entry support afforded by tem­ design has the fo llow ing benefits. plates . There are two primary reasons to use • Users need not run the language processor templates. One reason is for assistance with unfa­ from the editor. miliar language syntax. Users who are just • Concurrent compilations are possible. beginning to learn a new programming language can use the templates to help them get details • Allcomp ilation errors are diagnosed, not just correct . Users who are genera lly knowledgeable syntax errors. about the language can rely on the templates • No special parsers are needed . when they have tO use a construct that they rarely • have occasion to use, or when they have to make a Implementation is straightforward. call to a system routine that takes many parame­ Since the compilers place the diagnostic infor­ ters. A second reason to use templates is that they mation in a fi le, the source processing does provide structured formatting and reduce typing. not need to be done from within the editor. Even the most experienced programmers can Compi lations can even be done overnight by a benefit from using the editor to insert properly batch procedure, and the diagnostics reviewed indented control constructs, matched BEG IN and later using the editor. Also, since the compila­ END statements, and comment blocks, such as tions are done in separate processes, the user those used to describe the function, parameters, is free to continue editing one source file while etc., of a procedure. compiling others. By helping users with language syntax and by A drawback of using a parser, as in EDITH , is inserting some of the source code for the user. that the parser does not catch semantic errors, the editor reduces errors and improves produc­ such as mismatched data types. These errors were tivity. caught by the compiler in a separate compilation step. In LSE, all compilation errors are caught by BeyondSy ntax - Integrationfo r a single mechanism and can often be fixed in one Programming Support step. If assistance in entering syntactically correct Another drawback of using a parser within the source code were all that the editor offered ed itor is the work required to develop the parser with respect to support for program deve lop­ and keep it consistent with the compiler. Not all ment, it wou ld be a very useful productivity compilers are designed in a way that makes it aid. However some of its most important produc­ easy to pick up the parser and use it as a separate , tivity features do not deal with ed iting source callable utility. Consequently, considerable time code . In fa ct if the editOr provided no support was saved by avoiding the implementation of a at all for language syntax, LSE wou ld still be callable parser for each of the many languages a valuable aid in program development because LSE had to su pport.

Digital Te chnical journal 35 No. 6 February I 988 Language-Sensitive Editor

Once the compiler has written the diagnostics pilcr will include a suggested correction . The information to a fi le, the editor reads rhe fi le and editor will make the correction , subject ro the displays the information in an editing window . user's confi rmation . Conventional editing commands for scrol ling and searching can be used within this window. Source Code Analyzer In terfa ce Special keystrokes allow the user to select a Perhaps the greatest productivity aid in LSE is diagnostic and bring the corresponding source its interface ro the VAX Source Code Analyzer fi le intO a second editing window, as shown (SCA) . SCA is a multilanguage , source code in Figure 2. cross-reference and static-analysis root . Compil­ Each diagnostic consists of message text and ers for supported VAX languages collect informa­ one or more I ines of related source code. The tion during the compilation process and write message text may be one line or many lines. this information to a special analysis file. SCA Usual ly one line of source re lated to the error loads this information into a library for a program is presentee!. The soph istication of the diagnostic system. For each occurrence of an identifier in information depends on the capabi lities of rhe rhe program system, the SCA library contains compiler being used . Most compilers present information such as the source fi le. line. and one-1 inc error messages and rhe corresponding col umn of rhe occurrence; rhe type of occur­ source line. The most sophisticated presents rence (read, write, declaration, call, etc.); derailed error messages that include references and the class of identifier (procedure, variable, tO sections of the language manual and multiple ere.). SCA can also display calls ro and from related source lines . One usc of multiple source a procedure, and can perform consistency checks lines is to display both the declaration and usc on calls and declarations between compilation of a variable with the error message. The user unirs. may select either source line in the diagnostic Through its integration with SCA, LSE becomes display, and the editor wi ll position the editing a browsing utility for a software system. By posi­ cursor on the corresponding line in the appro­ tioning the editing cursor on a use of an identifier priate source fi le. In some cases, such as a in a source fi le and pressing a key that invokes rhe missing semicolon or a misspelled keyword , GOTO DECLARATION command, the user can the diagnostic information supplied by the com- bring the source corresponding to the declaration

line 39: LOW := 1 XADAC-E-INSSEMI, Inser·ted "; " at end of I ine

ine 24 : LOW, HIGH : INTEGER; L1ne 43: HIGH := 10000.0; XADAC-E-ASSIGNNERESTYP, Resu lt t�pe INTEGER in predef ined STANDARD of variable HIGH at line 24 is not the sa�e as an� real t�pe of litera l 10000.0 [LRM 5.2(1>1

Buffer SREVIEW Read-on l No�odif Forward end if;

if HIGH > 10000 then HIGH B= 10000 .0; end if;

if HIGH > LOW then PUT ( ITEM => "Input er·r·or· : LOW > HIGH" ) ; NEW_LINE; -� ' I

60 lines read fro� fi le LSES :[DEMOlPRIMES_ERROR .ADA; l

Figure 2 Editor Display ll'ilh Source fo r Selected Diag nostic

36 Digital Te chnical journal No. (, Febmm:r I ')88 Software Productivity Tools

of the identifier into an editing window . In this tracks changes to them. Source fiks are stOred in way, the user can view both the use and declara­ a CMS library in a compressed format as changes tion on the terminal screen , as in Figure 3. to the original version of the fi le. Users reserve a Users can perform more general queries using file from the CMS library ro make changes to it. the SCA FIND command. Using FIND, a user can After they are satisfied with their changes, they locate all the places where a variable is used. replace the file into the CMS library. Users may LSE disp lays the list of places and highlights the need tO get a fi le from CMS without intending to information for the first occurrence. Using single change it. This is called fe tching the fi le; a fi le keystrokes, the user can select an occurrence and that has been fe tched bur nor reserved cannot not have the editor access and display the corre­ be re placed . Two users may reserve the same file sponding source, as shown in Figure 4. at the same time. When the second replaces his LSE's integration with SCA has tremendously version, CMS will assist in merging the two sets of enhanced its va lue as a program development changes. CMS provides many features for organiz­ tool . Together these tools make programming ing a library, grouping sets of re lated fi les and tasks on a large software system much easier, related ve rsions of fi les, maintaining va riants of since users do not need to rely on their memory fi les, and inquiring into the status and history of or large cross-reference listings to locate declara­ fi les in the library . tions and uses of identifiers. The editor accesses CMS has become an important pan of the daily the appropriate fi les and finds the source line of routine for many VMS users. LSE provides com­ interest without the user even having to know the plete access to all CMS commands from within fi le name. By making it easier for developers to the editor. LSE can also invoke CMS for the user. understand a software system, LSE with SCA For example, when LSE needs to access a source speeds the development of new code and helps file as part of the SCA GOTO DECLARAT ION func­ developers make changes ro existing code more tion described above, LSE can invoke CMS to reliable. access the source fi le and bring it into the editor to display the specified declaration. Special LSE VA X DECjCMS - Code Ma nagement commands provide for reserving and re placing System fi les while performing appropriate ediror win­ The VAX DECjCMS tool stares project source dow and fi le management, eliminating interme­ fi les project, manages access to those files, and diate steps for the user.

code_vector_li�it = 259;

TYPE �ode_va lue = �in_code .. max_code; code_vector_ length = 0 .. code_vector_ li�it; code_vector = ARRAY [i . . code_vector_ ) i�it] or code_va lue;

{ A trans lation table specifies, for each input character, what output character it shou ld be trans lated to ldel_code �eans simpl� de let e) . When a string of input codes a II have the co�pr·ess f I a� set , on l4 the . . I ·- VAR table : tr·ans_tab I e I;

VAR code, rep l ace_code : code_va lue; i 1 .. code_vector_ li�it; co�press : BOOLEAN;

PROCEDURE si�na Ldup I icate ( code : code_ va lue I; VAR .. . . ·.t.... I I

143 lines read fro� file SYS$COHHON :[SYSHLP . EXAHPLES .SCAJBUILDTABLE .PAS; 1 59 l in es read fro� fi le SYS$COHHON :[SYSHLP. EXAHPLES .SCAJ TYPES.PAS; 1

Fig ure 3 Editor Display with Declaration and Us e of code_va lue

Digital Technical journal 3 7 No. () Ft'bruarr• I ')88 Language-Sensifil,e Edit or

The integration of LSE and CMS is an added programming languages. including OATATRIEVE, convenience for LSE users that streamlines the a data management tool ; DOCUMENT, a docu­ usage of the two tools, and so enhances produc­ mentation markup language; and COOL, a com­ tivity. mon clara dictionary language.

Summary Fu ture Directions The combination of templates, on-line help, and The hardware environment for software develop­ interfaces with compi lers makes LSE an effective ment is shifting from timesharing systems to and easy-ro-use program editor. Users can mne workstations. This shift will open some opportu­ the templates provided with LS E fo r their own nities for L'iE, making it possibk for L<; E design­ environ ment or define templates, help, and a ers tO consider including capabil ities that require compiler interface for languages that Digital does more hardware resources than were typically not support. available in the past. The current text-editor Through integration with SCA and CMS, LSE orientation does nor preclude enhancements that expands its capabilities from a source-code edi­ require parsing the source, such as pretty-print­ tor 10 a high-productivity user interface to the ing, cursor movement by syntactic element, and source code for a software system. elision . The display capabilities of a workstation Since May 1985 when version 1 of the editor could support pretty-printing, using bold and shipped, LSE has continued to evolve as a compo­ italic fonts to improve the readability of source nent of the VAX set, a set of productivity rools code. The availability of the mouse, icons, graph­ for software development. The VAX Language­ ics, etc .. allows for many enhancements to the Sensitive Editor is now playing an important role user interface. both as an editor designed to support wri ting The software that supports program develop­ software and as a hub for the integration of soft­ ment will also continue to evolve. Ad ditional ware development wols. lan-guages and tools can be expected to include This paper describes the current release of LSE ancl SCA support, and there will be opportu­ VA.'( LSE, version 2.1 7 LSE supports 15 languages nities for LSE to grow through integration with including Ad a. BASIC, C, COBOL, FORTRAN, other components of the programming support Pascal, PL/1. In addition LSE supports some non- environment.

S�111bo l Class Hodu l e \Line T�pe of Occurrence nll-lr!•JD· .BUILD_ TABLE\-47 VAR (variable) dec l aration �·····l! BUILD_TABLE\74 wr· i te r· efer·ence BUILD_TABLE\76 r· ead r· efer·ence BUILD_TABLE\77 r· ead r·efer·ence BUILD_TABLE\96 write reference BUILD_TABLE\97 r· ead r·efer·ence BUILD_TABLE\99 r· ead r·efer·ence BUILD_TABLE\100 read r·efer·ence (;luear t riND CODE for•ard rep l ace_code := repl_vector [ 1J; rOR i := 1 TO orig_ len DO BEG IN �ode := orig_vector(il; IF table[codel . trans_va lue <> undef_code THEN signa l _duplicate (code>; tab\e[codel . trans_va lue .- code; END; ..

37 occurrences found 13 s�111 bo ls, 1 name ) 143 lines read fro111 file SYS$COHHON:£S YSHLP . EXAHPLES .SCAJBUILDTABLE.PAS; 1

F(P, ure 4 Editor Display with Selected "write reference" to code

38 Digital Te chnical journal No. (i Fe bruarr• 1988 Software Productivity Tools

References 5. R. Stallman, "EMACS: The Extensible, Cus­

I. T. Te itelbaum, T. Reps, "The Cornell Pro­ tomizable, Self-Documenting Display Edi­ gram Synthesizer: A Syntax-Directed Pro­ tor," Proceedings of the ACM SIGPLAN­ gramming Environment," Communications SJGOA Sy mposium on Te xt Ma nipulation of the ACM, vol . 24, no. 9 (September Qune 1981): 147-156. 1981): 563-573. 6. S. Wood, "Z-The 95% Program Editor," 2. M. Zelkowitz, "A Small Contribution to Edit­ Proceedings of the ACM SJGPLAN-SIGOA ing with a Syntax Directed Editor," Proceed­ Sy mposium on Te xt Ma nipulation Qune ings of the ACM SJGSOFT SJGPLAN Sy mpo­ 1981): 1-7. sium fo r Practical Software Development 7. Guide to VA X Language-Sensitive Editor Environments (May 1984): 1-6. and VA X Source Code Analyzer (Maynard: 3. J. Horgan, D. Moore, "Techniques for Digital Equipment Corporation, Order No. Improving Language-Based Edirors," Pro­ M-FY24B-TK. April 1987). ceedings of the ACM SJGSOFT SIGPLAN 8. VA X DECjCMS Reference Manual (May­ Sy mposium fo r Practical Software Develop­ nard: Digital Equipment Corporation, Order ment Environments (May 1984): 7-14. No . M-L372B-TE, November 1984). 4. VA X EDT Reference Ma nual (Maynard: Digital Equipment Corporation , Order No. AA-Z300A-TE, September 1984).

Digital Te chnical journal 39 No. 6 Februar)• I 'J88 Stephen R. Greenwood I

VAX SCAN: Rule-based Te xt Processing Software

The VAX SCANproduct simplifies for programmers the building of software that recognizes complex text pattems. The product achieves this simplifi­ cation by uniting powerful text-based pattem-matching capabilities with a procedural language that integrates these capabilities into the VAX/ VMS environment. In typical applications of the product, programmer time to design, code, debug, and maintain progmms is greatly reduced, contribut­ ing to increased software productivity. The short development time, low cost, and high reliability of the VAX SCAN product itselfis attributable to the procedures and toolsavailable in Digital's engineering environment.

The text processing capabilities typically pro­ Principles of the Language vided by high-level languages are quite primi ­ The text processing capabilities of the SCAN. lan­ tive , especially in contrast to the syntax diagrams guage are based on the simple substitution used in describing programming languages. paradigm of finding a specified pauern and High-level languages normally include opera­ replacing it with a string of text, much like a sub­ tions ro move , compare, concatenate, search, and stitution command in an editor. SCAN differs perhaps sort text strings of fixed length. Syntax from most editors, howevn, in that the search diagrams. more formally known in computing pattern can be as complex as a grammar for a pro­ literarure as grammars, pn:sent a very nice model gramming language. The replacement text can be for very complex text patterns. generated by an arbitrarily complex sequence of Programmers throughout the computing indus­ procedural statements. The actual SCAN lan­ try understand both primitive text operations guage construct that supports this substitution and the more complex patterns represented paradigm is called a macro. by syntax diagrams. However, few programmers The syntax of a macro is given as fo llows: use syntax diagrams to express input to a pro­ MACRO . . . gram, because of the complexity involved in ma cro_name attribute building the software needed to recognize such { pa ttern } ; patterns. macro_body MACRO ; The VA,'( SCAN product solves the complex­ END ity of using syntax diagrams by making text Jn a macro, the pat tern specifies the text that pauern recognizers available to the general VAX the macro is ro search for and march. The rext programming community. The SCAN language marched by the pattern is then replaced by text allows a user to express language grammars generated by the macro_body. Figure 1 is an in a form very similar to the BNF-like syntax example of a macro that searches for rime values diag�ams that appear in VA XjVMS documenta­ and replaces them with a string. tion . The product builds a recogn izer, in the In this example the pattern describes a time form of an object module, that matches text value as an integer, fol lowed by a colon , fo llowed described by the grammar. In addition , the by a second integer, fo llowed optionally by product supports the VAX Common Language another colon and a third integer. Whenever this Environment, so that VAX SCAN recognizers pattern is marched. the VAX SCAN product can be easily integrated into existing programs. replaces that occurrence of the pattern with the

40 Digital Tec hnical jo urnal No. (j F

text generated by the macro body. The algorithm • The method by which text matched by the pat­ for generating the replacement text increments a tern can beviewed by the macro body static variable named count and specifies the • The interplay between the procedural code re placement text with a special SCAN statement and text scanning called the Al'\ISWER statement. The first time the macro is invoked, the replacement text will Origin of the Language be the string "time : 1 ". The second invocation of the macro produces the replacement text Few languages are designed from scratch; SCA..l'\1

"time : 2". The TRIGGER keyword determines is no exception. The text processing paradigm the type of macro, described later in the section employed by the SCAN language is an outgrowth Invoking Macros. of techniques dev eloped for parsing computer The body of the macro in this example is writ­ languages using context-free grammars. These ten using the procedural portion of the SCAN techniques permit the syntax of a language to be language. This portion not oo.ly serves as a means described by a grammar. Efficient algorithms arc of creating replacement text, but also allows then used to check that an input stream of char­ the language to achieve int egration with the aCters conforms to the syntax specified by the ' VAX jVMS environ ment. In that respect, SCAN is grammar. similar to PASCAL. The MAC RO language developed at Sperry Uni­ The procedural portion of the SCAN lan­ vac Corporation for their l l 00 Series computers guage includes procedures, fu nctions, state­ rook this concept and appl ied it to string substi­ ments, and data structu res that a programmer tution and, in fa ct, developed the notion of can use ro interface with procedures written macros that are found in the SCAN language . 2 The in other VAXjVMS languages. Procedures that PA SCAL language and the VA.XjVMS architecture can either call or be ca lled by routines written also influenced the design of the SCAN language. in other languages can encapsu late SCAN's SCAN's data structures and data types are based text processing capabilities. This encapsula­ on concepts in PASCAL. The desire for SCAN ro tion permits the VA XjVMS user ro conveniently integrate we ll with the rest of the VAXjVMS envi­ access the unique capabi lities of the SCAN ron ment mandated that it be convenient to call language . SCAN programs from other VA.X jVMS languages. Macros are the central construct of the SCAN language. Some additional considerations must be Invoking Macros discussed , however, to fu lly understand the lan­ Several principles influenced the rule s used by guage : the SCAN language to invoke macros . Since a pattern might appear numerous times in a fi le of • The origin of the language text, the first principle was to apply a macro each e irs pattern is found so that the macro can • The invocation of macros tim make multiple transformations, just like an editor • The specification of patterns substitution command . Second , it was important

MACRO find_t ime TR IGGER { integer ':' integer [ '·' integer ] };

DECLARE count : STAT IC INTEGER ;

count = count + 1;

ANSWER 'time : ' , STR ING< count >;

END MACRO ;

Fig ure 1 Example of a Macro

Digital Te chnical journal 4 1 No. 6 Februarv 1988 VA X SCAN: Rule-based Text Processing Software

to be able to specify many concurrent substitu­ The fi le file_to_be_scanned .dat, in this tions. Therefore, when a VAX SCAN program is example (specified in the START SCAN state­ compiled, linked, and then run, it behaves much ment) is searched for a_pattern, b_pattern, like a batch (rather than an interactive) substitu­ and c_pattern (specified in trigger macro pat­

tion command. The third principle in designing terns) . The file f i 1 e_ t o_be_crea ted . da t (also macro invocation rules was that patterns may specified in the START SCAN statement) is cre­ be of varying degrees of complexity. To manage ated by the program. Its contents will be the orig­

ve ry complex patterns, the patterns have to be inal file, file _to_be_scanned .dat, including separable into pans, much like a program can be the substitutions performed by the macros. separated into multiple procedures. Syntax macros permit patterns to be defined in Two types of macros are defined to support terms of other patterns. That is, within one pat­ these principles: syntax macros, and trigger tern a programmer can request that a pattern macros. These names reflect the two different described by a separate syntax macro be recog­ macros distinguished by the attributes SYNTAX nized. This concept of composing a pattern from and TRIGGER within a SCAN macro declaration. more elementary patterns is the basis of formal Trigger macros provide search and replace language theory. 1 This topic is explored in semantics. A program may contain any number greater depth in the next section. of trigger macros. Upon executing a START Sp ecifyingPa tterns SCAN statement, a program begins scanning the input stream of text specified by that state­ The design of patterns is influenced by the fol­ lowing three sources: ment for the patterns specified by the trigger macros. Figure 2 illustrates a series of trigger 1. The BNF-style syntax diagrams customers macros. find in Digital's manuals

MACRO find_pattern_a TR IGGER { a_pattern };

ANSWER repl acemen t_text ;

END MACRO ;

MACRO find_pattern_b TR IGGER { b_pattern }; ANSWER replacem ent_text ;

END MACRO ;

MACRO find_pattern_c TR IGGER { c_pattern };

ANSWER replacemen t_text ;

END MACRO ;

PROCEDURE main- procedure MA IN; START SCAN INPUT FILE 'file _to_be_scanned .dat '

OUTPUT FILE 'file_to_be_c reated .da t';

END PROCEDURE ;

Figure 2 Tr igger Macro Example

4 2 Digital Technical jo urnal No. 6 February 1988 Software Productivity Tools

2. Compiler theory for context-free grammars The token ide ntifier illustrates the pattern for an identifier in the SCAN language . The 3. The original MAC RO language implemented token definition uses two sets, alpha and other, by Sperry Univac Corporation to simplify the specification of its pattern. All three sources recognize a two-level The definitions of these sets appear in the exam­ approach for specifying patterns. The lower level ple as we ll. recognizes simple constructs, such as numbers, In macros, higher level patterns are defined keywords, names, and punctuation . The higher using the same operators that are used in roken level arranges the lower level constructs into declarations. Unlike a lower level pattern, statements and groups of statements. however, the operands of a macro pattern are This two-leve l approach has several advan­ rokens and other macros, rather than characters tages. Abstracting primitives into lower level and sets. Therefore, tokens are the building patterns results in a more uniform use of primi­ blocks of a macro pattern. Referencing a macro tives . That, in turn, makes the overall pattern within a macro pattern provides a subroutine­ easier tO remember, which is of great benefit like capability for patterns. The placement of a tO a programmer trying to design a language macro name in a macro pattern indicates that that he is trying ro recognize using the VAX the pattern of that macro should be recognized SCAN product. Compiler theory also states that at the point of reference. lower level patterns can be recognized very Defining the pattern of one macro in terms efficiently if they conform tO a set of rules, such of other macros adds significant power ro as regu lar expressions. 1 SCAN's patterns. This power, illustrated in Fig­ To take advantage of those features, the SCAN ure 4, shows the syntax for a SCAN token decla­ language provides a two-level description of a ration. The pattern found in a roken declara­ pattern. The lower level pattern, called a token, tion is equivalent to patterns that can be groups characters. The higher level pattern that described using regular expressions. Token appears in a macro is composed of tOkens. patterns need to express the precedence of Figure 3 illustrates several SCAN rokens. three operators: alternation, concatenation, and The syntax for the patterns comes largely from repetition . In addition, a token pattern sup­ the conventions used in Digital's software ports nested subpatterns that are either required manuals 5 Curly braces surround a sequence or optional. that is required, and square brackets surround This example utilizes macros to provide an optional sequence. An ellipsis indicates • Levels of abstraction that the prior sequence can occur multiple times, and a ve rtical bar separates alternative • Sharing of patterns choices. Thus in the Figure 3 example, • Recursion morse_code_let ter describes a required sequence of " . " or "_" characters. This Levels of abstraction refers to the process of required sequence can occur one or more times. building a hierarchy of concepts in which each

TOKEN mo rse_code_letter { { '_' : } . . . } ;

TOKEN begin_keyword { 'BE GIN' };

' ' SET alpha < ' a ' .. z OR ' A' .. 'Z' >;

SET other < '0' .. '9' OR '$' OR '-' > ; TOKEN identifier { alpha [ alpha : other ] ... };

Figure 3 To ken Examples

Digital Technical jo urnal 4 3 No. (, Februarv I !)88 VA X SCAN: Rule-based Text Processing Software

level of the hierarchy is built on the next lower sharing patterns by means of macros. Much as a level. The SCAN language justifies two levels subromine is a ve hicle fo r sharing code in a of patterns, tokens and macros, based on this FORTRAN program, macros are a vehicle for principle. In Figure 4, the principle is employed sharing a pattern in SCAN . fu rther. To k en_dec laration is defined in terms Recursion - defining a pattern in terms of of token_pat tern, which is defined in terms of itself - is very useful in the description of many token_operand. Each macro describes a level patterns, especially when patterns can be nested of abstraction that makes a complex pattern as in the case of a token pattern. In Figure 4, both simpler to write and to understand. In this token_pattern is defined in terms of particular example, the levels expose the prece­ to k en_pa t tern because token patterns can be dence of the three token pattern operators. arbitrarily nested using curly braces and square T_pa t3 and token_pattern are examples of brackets.

TOKEN token_keywo rd .... pa tterns for the tokens TOKEN ide ntifier .... have been om itt ed TOKEN character_string .... TOKEN I , • I .... TOKEN I I I •••• TOKEN I [ I • • • • ) TOKEN I I • • • •

TOKEN I { I • • • •

TOKEN I } I 0 0 0 0

I • I TOKEN ' . . .. TOKEN ' ...' ....

! syntax for token dec larat ion

MACRO token_dec larat ion SYNTAX { token_keywo rd ident ifier '{' token_pattern '}' ';' };

syntax for alternation

MACRO token_pattern SYNTAX { t_pa t3 [ 'I' Lpa t3 l ... } ;

! syntax for conca tenat ion

MACRO t_pa t3 SYNTAX {t_pa t2 ...};

! syntax for repet ition

MACRO t_pa t2 SYNTAX { t_p at 1 [ ' ...' J};

! syntax for optional and req uired pa tterns

MACRO t_pat 1 SYNTAX token_operand '[' token_pattern 'l'

' { ' token_pattern '}' };

! syntax for operands

MACRO token_operand SYNTAX { character_string ide ntifier };

Figure 4 Sy ntax Macro Example

44 Digital Tecbnicaljournal No. 6 February 1988 Software Productivity Tools

Viewing Ma tched Te xt in the Macro Interplay between Procedural Code Body and Te xt Scanning Macro patterns provided a simple mechanism The SCAN language is an interesting amalga­ for describing complex patterns. Macro bodies mation of a rule-based language and a proce­ provided a powerful mechanism for creating dural language. The text scanning capabilities of replacement text. For a complete solution, the the SCAN language are rule-based. Tokens and SCAN language needed a robust means so that a macros describe the rules for recognizing pat­ macro body could view the text matched by the terns. These descriptions have no concept of pattern. flow of control. Macro bodies, on the other hand, A solution to this problem was provided in are algorithms with a very definite concept of the original MACRO language. That solution flow of control. The interaction of the rule-based consisted of inserting variables in the pattern tO and procedural parts of SCAN isa key princi pie of capture the text matched by a segment of the pat· the language. tern. See Figure 5 for an example. A SCAN program starts executing in procedural

Hour, minute, and second are variables mode at the start of its main procedure. That inserted to capture matched text. Hour is main procedure may be a SCAN main procedure assigned the text matched by the first integer or one written in another VA.XfVMS language. token. Minute is assigned the text matched The rule-based technique of pattern matching by the second integer roken. Second holds the begins with the execution of a STARTSCAN state­ text matched by the third integer token, which ment. The START SCAN statement specifies the may be the null string if the optional pattern input stream to be scanned by the macros and the £ I : I integer lis not matched. output stream to hold the transformed input A pattern can, in fa ct, contain an arbitrarynum­ stream. The main procedure in Figure 6 is a SCAN ber of variables, each of which can capture the maio procedure named ma i n_p roc. The START text. line number, and column position of one or SCAN statement commences the search for the pat­ more tokens matched by the pattern. tern described by the macro translate_morse.

CONSTANT h 1hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh 1j

CONSTANT m 1mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 1;

CONSTANT s 1ssssssssssssssssssssssssssssssssssssssssssssssss1;

rep lace a time such as 5:45:01 with h:mm:ss where number of h1s, m1 s and 515 corresponds to the num ber

of digits in the time

MACRO find_time TR IGGER

hour : integer 1:1 minute: integer 1 : 1 second : integer l };

)] ANSWER h[ 1 ..LENG TH( hour , 1:1; ) ANSWER m[ 1 ..LENG TH( minute ] ; <> IF second 11 THEN

I , I ANSWER . ' h [ 1 ..LE NGTH( second J l; END IF;

END MACRO ;

Figure 5 Va riables fo r Capturing Matched Te xt

DlgitaJ TechnicaJJournaJ 4 5 No. 6 Fe bruary 1988 VA X SCAN: Rule-based Te xt Processing Software

Each time a macro matches its specified pat­ Within its problem domain, the VAX SCAN tern , procedure mode is entered for the duration product increases programmer productivity of the execution of the macro body. When the by dramatically simplifying the solution to a macro body completes execution , the text problem. Consider the fol lowing program frag­ replacing the matched text wdl have been gener­ ment: ated and can be substituted in the output stream for the matched text. TOKEN space { { ' ' I s'ht' } ... }; Eventually, the input stream will bt· exhausted MACRO TR IGGER }; and the output stream completes . At this point, compress { space ANSWER ' ; execution continues with the statement fo llow­ ' END MACRO ; ing the START SCAN that initiated pattern match­ ing, and the program returns to procedural PROCEDURE ma in_proc MA IN; mode. START SCAN

INPUT FILE 'in pu t_logical' SoftwareProd uctivityBene fi ts of OUTPUT FILE 'output_logical '; Us ing VAX SCAN END PROCEDU RE ; Increasing software productivity depends on reducing the cost of implementing software This short sequence expresses the concepts of while maintaining a high degree of reliability. opening a fi le, scanning it for arbitrary-length

MODULE mor se_code ;

DECLARE let ter_count , error_count : INTEGER ; EXTERNAL PROCEDURE

morse_to_ascii DYNAM IC STRING, FIXED STR ING( 1) ) OF BOOLEAN ;

. TOKEN mo rse_let ter . , . , -, } . . . } ; { ,

MACRO trans late_morse TR IGGER { do ts : mor se_letter };

DECLARE char : FIXED STR ING< 1);

IF morse_to_as cii( dots, char ) THEN

ANSWER char ; ELSE

er ror_count � error_count • 1; END IF;

letter_count s letter_count • 1;

END MACRO ;• trans late_morse "/ ;

PROCEDURE ma i n_proc MA IN;

letter_count • 0; error_count = 0; START SCAN

INPUT STRING ' . . . _ . __.. _ I

OUTPUT FILE 'sysSou tput' ;

WR ITE letter_coun t, error_coun t;

END PROCEDURE ;• ma in_proc "/ ;

END MODULE ;• mor se_code */;

Figure 6 A Complete SCAN Program

46 Digital Te chnical journal No. 6 February I 988 Software Productivity Tools

sequences of contiguous blanks and tabs (s'ht' is Another typical use is to build translators. the SCAN notation for a horizontal tab) , re placing Translators make modifications to parts of a file the matched sequence with a single blank. and and leave the rest of the fi le unchanged. During placing a copy of the modified input fi le in an the development of the SCAN language, several output file. changes were made to its syntax. For example, The sequence is at least one order of magni­ parentheses were changed to braces in token and tude shorter than an equivalent algorithm in PA S­ macro patterns, and fi les became explicitly CAL. In general, using SCAN decreases the times declared rather than implicitly declared objects. to design, code, debug, and maintain programs. Rather than change all the programs in the test The domain of programs well suited for imple­ system manually, we wrote a translator to do the mentation using the VAX SCAN product is some­ job. The program consists of 188 lines of code, what difficult to assess. In this section, several 60 blank lines, and 7 comment tines. Like the types of applications are surveyed to give an idea previous example, the program converts all files of the range of applications and the cost of imple­ that match a specified fi le specification and menting each. reports which fi les were modified. A typical use of the VAX SCAN product is to A third example that merges the concepts of an create a tool that will extract info rmation from extractor and a translator is a program that reads a set of fi les. An example of an extractor is a BASICsource fi les that may contain record decla­ program to read a VAX SCAN source fi le and rations. For each BASIC record declaration en­ report the numbers of blank lines, the lines countered, the SCAN program outputs an equiva­ containing just comments, and the lines contain­ lent VAX Common Data Dictionary declaration. ing code. A version of such an extractor was This program consists of 207 lines of code and 71 written to gather statistics for this paper. The lines of comments and took approximately one program prompts for a fi le specification, which workday to write and debug. Sample input and out­ may contain wild-card characters, and then scans put for this record translator is given in Figure 7. all the fi les that match the fi le specification. A language with a limited problem domain has The program lists the statistics for each fi le the potential to decrease programmer productiv­ matched and the totals fo r all fi les scanned. The ity, because such a language is unlikely to be a extractor consists of 50 lines of code, 25 blank programmer's main implementation language. lines, and 3 lines of comments; the entire extrac­ Thus a programmer has a tendency to fo rget the tor program took approximately 30 minutes to details of the language, which in turn decreases create and debug. his efficiency.

Input Output

200 RECORD examp le examp le STRUCTURE . BYTE a,b a DATATYPE SIGNED BYTE .

INTEGE R WORD cCS> b DATATYPE SIGNED BYTE . GROUP nes ted_group c DATATYPE SIGNED WORD ARRAY 0:5. HFLOAT d( 10,10) nes t ed_group STRUCTURE .

STRING e•S d DATATYPE H_FLOATING ARRAY 0 : 10 0 : 10.

REAL f e DATATYPE TEXT SIZE 5. END GROUP f DATATYPE F_FLOAT ING. END RECORD END nes ted_group STRUCTURE .

END examp le STRUCTURE. 300 EHD

Figure 7 Sample Input and Output fo r Record Translator

Digital Technicaljournal 47 No. 6 February 1988 VA X SCAN: Rule- based Text Processing Software

VAX SCAN attempts to minimize this problem • Phase 0-Requirements gathering by • Phase 1 - Design of the product • Basing pattern matching constructs on syntax diagrams used in Digital's documentation • Phase 2-Implementation of the product

• Providing a VA,'( Language-Sensitive Editor interface for creating and compiling SCAN • Phase 3 - Field testing of the product programs • Phase 4 - Maintenance of the product • Ad ding support for the VAXjVMS Debugger, including features to monitor pattern match­ Each phase has a set of inputs and outputs . The ing ou tput of Phase 0 is a product-requirements doc­

• Providing simple integration with procedures ument that becomes an input to Phase I. The out­ written in other VAXjVMS languages put of Phase 1 includes a description of the product at a level that can be documented, a • Basing the syntax on a welJ-known language , design of the product to a level at which the PASCAL implementation costs can be estimated, and a • Providing an extensive on-line help facility development plan describing Phase 2 as a set of In summary, the benefits of using the VAX schedu led tasks. SCAN product are twofold. First, for a large class These procedures and policies greatly reduce of problems, SCAN drastically reduces the cost of the effort required to manage a project by provid­ designing, implementing, and maintaining the ing a common model for describing a project. solution to those problems. Second, the product The procedures provide a common framework contains many ease-of-use features to minimize and terminology to address issues of scheduling, the overall cost of software development. cost, and task completion. A project manager, his team members, and people from other sup­ Leveraging the VAX SCAN porting groups, such as field test administration , Implementation all measure their efforts in terms of these com­ The initial version of the VA X SCAN product took mon parameters. Since the definitions of tasks are approximately three developer-years to produce. specified as standards, time is not wasted dis­ This period includes the time to design the lan­ cussing differences of opinions about these defi­ guage, design and implement the VAX SCAN com­ nitions. As a result, the different groups that pi ler and the run-time library, and internally test develop, field test, and manufacture the product the product. The initial version was ready tO be can work rather independently. The phase pro­ field tested by customers at the end of the three­ cess details at what times those groups need to developer-year effort. coordinate their activities and the manner in Producing an optimizing compiler and run­ which they will do so . time library in only three deve loper-years is a Many of the documents that are part of the significant accompl ishment. Three factors in phase process have markup language templates . Digita l's engineering environment contributed Each template outlines the contents of the docu­ greatly to making this level of software produc­ ment and includes a series of questions and tivity possible. checklists of the items to be considered during the preparation of the docu ment. The questions • Digital's Software Development Policies and and checklists are a distillation of the successes Procedures and fa ilures of previous software products; the • The VAXjVMS software environment use of these questions and checklists helps tO • The VAXjVMS tool set reduce the number of unantici pared problems that often plague software development. Digital's Software Development Digital's Software Development Policies and Po licies and Procedures Procedures provided a platform on which the Digital's Software Development Policies and Pro­ VAX SCAN team built their product. By fo llow­ cedures define the life cycle of a standard Digital ing these guidelines, less time was spent man­ 4 software product. 5 This life cyc le is divided into aging the project, leaving more time for the five phases. design and development of the fi nal product.

48 Digital Technicaljountal No. 6 February 1988 Software Productivity Tools

VAX/VMS Software Environment • Command line interpretation At Digital, software engineers have always strived • Internalmemory management to achieve a high degree of integration among VAX:jVMS products. For example, command line • Tools for debugging a compiler parsing for most products is provided by a com­ mon command line utility. The use of the VAX During the design phase (Phase I) of the VAX RMS product as the standard IjO package is SCAN project, the team created a prototype for another example. VA X languages foster integra­ the product using the VCG to generate code for tion by supporting parts of an application being SCAN concepts that were new to the VCG. The written in diffe rent languages. To meet this level success of this prototype effort showed that using of integration, Digital has evolved standards that the VCG concept was the correct solution to describe reducing the development cost of the VA X SCAN

• Calling sequence product. The final results were impressive . By using the • Format of data types VCG, the team cut the cost of implementing the SCAN compiler by at least a factor of three. At the • Format of descriptors same time, the VAX SCAN compiler generates • Alignment of record fields high-performance machine code to move strings and perform arithmetic operations that rivals the • Processing of exceptions performance of any of the other VAX compilers.

• Interface to the debugger The productivity benefits gained by using the common code generator are far reaching. Bug • Management of dynamic storage rates in the code generator are remarkably low, because the code generator is tested by four dif­ • Structure of files ferent languages. In addition, as new demands For a product like VAX SCAN, each standard are made for compilers to support tools such as represents a design problem that had already the VA X Language-Sensitive Editor, the work been solved. The VA X SCAN product simply had needed to support the tool is divided in two: to conform to each existing standard. the language-specific work, and the common The desire for an integrated language environ­ work needed by all the compilers supported by ment naturally results in shared code among VMS the VCG. Frequently, this duality reduces the products. Initially, this sharing existed only at amount of work the VAX SCAN product must run time in a common math library and a com­ do to support such tools. Finally, each VCG mon record management system. Eventually, improvement is actually an improvement for four however, engineers recognized that large sec­ languages. Thus, for example, a better register tions of code were being duplicated over and allocation algorithm benefits all four languages over again in each compiler. Therefore, the next because it will be added to the common portion logical step was to develop a common compiler of each compiler. with language-specific sections to handle lan­ The original vision of VAXjVMS products being guage-specific tasks such as parsing. On the well integrated had a handsome payback for the VA XjVMS system, this common compiler is the VAX SCAN product. First, it resulted in a set of VA X Code Generator (VCG) . standards that defined many of SCAN's external At the time SCAN was being designed, the VCG and internal interfaces. Second, integration fos­ supported the VAX PL/I and the VAX C compil­ tered code sharing, which greatly reduced the ers. Moreover, work was underway to support the cost of implementing the VAX SCAN software. VAX Ada compiler. The VCG included VA XjVMS To ol Set • A global and peephole optimizer Building software systems without the proper • A comprehensive code generator tools can be an inefficient and difficult process. Several of the tools available in the VA XJVMS • Object module generation environment are topics of other articles in this 4'6'7'11 • Listing and error-message facilities issue of the Digital Te chnical jo urnal.

Digital Technical jo urnal 49 No. 6 February 1988 VA X SCAN: Rule-based Text Processing Software

These tools, like the standards mentioned in the Summary previous section, provided guidance to the VA X The VA X SCAN project is regarded as a success at SCAN team by suggesting ways of solving prob­ Digital for two rather diverse reasons. First, the lems. For example, the VAX Code Management product itself is a good balance of text processing System software describes a method for coordi­ features and integration with the VA XjVMS envi­ nating changes to the source code of the com­ ronment. The result for a VAX SCAN user is much piler and the run-time system. The VAX DEC/Test greater programming productivity. Manager software describes a means of testing the The second reason for success is attributable product. In general, evaluating and choosing a to the cost effectiveness of the project. The tool that exists is simpler and cheaper than engineering costs of the product were low, the designing one for the specific needs of a project. product was timely, and the quality was excel­ A list of the major tools used during the devel­ lent. opment ofthe VA X SCAN product follows: The VAX SCAN product is a fine example of good ideas implemented by using sound software • The VAX Code Generator engineering principles and supported by effec­

• An IALR(1) Parsing System developed by the tive tools. VA X Ada team References • The VA XjVMS Debugger 1. A. Aho, R. Sethi, and ]. Ullman, Compilers,

• The VAX RMS (Record Management System) Principles, Te chniques and To ols (Read­ software ing: Ad dison-Wesley Publishing Co., 1986). 2. Sp erry Un ivac Series 1100 MACRO Pro­ • The VMS Run-time Library grammer Reference (Roseville: Sperry

• The VAX Language-Sensitive Editor Univac Corporation, Issue: UP-8336 Revi­ sion 3). • The VMS Message Utility 3. VA X SCAN V1. 1 Documentation Kit (May­ • The VAXDE CjTest Manager software nard: Digital Equipment Corporation, Order No . QL495-GZ-1.1, 1986). • The VAX DECjMMS (Module Management System) software 4. B. Beander, "VAXjVMS Software Develop­ ment Environment," Digital Te chnical jo ur­ • The VA X DECjCMS (Code Management nal (February 1988, this issue) : 10-19 . System) software 5. Software Development Policies and Proce­ • The VAX Performance and Coverage Analyzer dures (Maynard: Digital Equipment Corpo­ ration, 1981). • The VAX BLISS Compiler and MACRO assembler 6. G. Lupton, "Language-Sensitive Editor," Digital Te chnical jo urnal (February 1 988, Of the three-developer-year effort required this issue) : 28-39 . to implement the initial version of the VAX SCAN product, 8 developer-months were spent 7. L. Ziman, M. Dickau, "Project Management in the design of the language, compiler, and of the VAX DEC/Test Manager Software run-time system. The remaining 28 developer­ Ve rsion 2.0," Digital Te chnical jo urnal months were spent implementing and testing the (February 1988, this issue) : 110-1 16 . product. At the conclusion of the implementation 8. P. Gilbert, "Development of the VAX NOTES phase, the system consisted of 75,000 lines of System," Digital Te chnicaljournal (Febru­ source: 60,000 in the compiler, and 15,000 in ary 1988, this issue) : 117-124. the run-time library. Forty-four percent of the lines were actual code, 20 percent were blank lines, and 36 percent were comments. The test system exercised over 90 percent of the code paths in the compiler.

50 Digital Technical journal No. 6 February 1988 RobertA. Conti I

Software Productivity Fe atures Provided by the Ada Language and the VAX Ada Compiler

The Ada language provides a number of features that can increase soft· ware development productivity. Many of these, such as packages, tasks, and exceptions are not present in conventional programming languages (such as C, FORTRAN, and Pascal). Others, such as strong typing ruks, range-checking, and portability, are provided by some conventional lan­ guages, but not all. Beyond the inherentAda language features, Digital's Ada compiler for the VMS operating sy stem provides additional features that enhance productivity. Examples are automatic inlining, portability checking, and a comprehensive program library manager. This paper introduces the major productivity features of both the Ada language and Digital's Ada compiler, anddescribes some of the benefits that result.

Ada -:s Contributio n to Productivity Response time For the purposes of this paper, software produc­ Reuse of software components tivity will be defined tO be the total profit gener­ ated by a software product divided by the rota! Ta i lorabi lity development costs (nowadays, mostly labor) Time to market required to design, deve .lop, test, maintain, and enhance the product over its entire life cycle. User expectations This definition of software productivity is one lt should be apparent that the traditional use of that the manager of a commercial software busi­ "lines of code per day" to define software pro­ ness might use. By including both profit and ductivity is incomplete in comparison with the expense, this definition also includes the effe cts definition used in this paper. of attributes that are associated with software The U.S. Government developed the Ada pro­ quality, attributes such as gramming language to help decrease the life­ Compatibility cycle costs of its computer programs. (Profit was not a fa ctor.) Ada's fe atures are intended to Ease of implementation make software easier tO design , read, and mod­ ify, as we ll as tO be more reliable and portable Ease of use between computer systems. In short, the fea­ Execution time tures are intended either tO reduce expenses or increase the quality of the software . Both of Future extensibility these effects make software development more Maintainability productive according to our definition of pro­ ductivity. Commercial enterprises shou ld also be Memory space able to achieve improved software productivity by using Ad a. Portability of applications In March 1985, Digital released its VAX Ada Reliability compiler product fo r computers running the

Digital Technicaljournal 51 No. 6 February 1988 Software Productivity Fe atures Provided by the Ada Language

VMS operating system. This product includes a the view it presents tO its clients. The term number of additional features that reduce costs clients here refers tO all the software that relies or improve software quality. upon the features provided by the interface. Although we have not made actual measure­ Good programmers have learned to informally ments of the productivity increases that wou ld discipline themselves to recognize the distinc­ result from using Ada in general, and VAX Ada in tion between interface and implementation, and particular, our experience in developing soft­ to design interfaces that are independent of an ware in other languages indicates that certain implementation. The Ada language formalizes fe atures are quite beneficial. This paper intro­ this design practice by allowing a programmer duces those features. We encourage the reader tO code the interface (the specification) sepa­ to reflect upon their collective impact. rately from the implementation (the body) for Familiarity with a conventional program­ program units such as procedures and functions. ming language, such as FORTRAN, C, Pascal, The language requires this separation for other or BLISS, is assumed. (References 1, 2, and 3 program units such as packages and tasks (to be are textbooks containing detailed information described later) . on programming in the Ada language . Refer­ The following example illustrates this separa­ ' ence 4 is the more formal and technical lan­ tion: guage standard.) The specification defines Inherent Productivity Fe atures of the what the cal ler can rely upon . Ada Language procedure SORT (X : in out WIDGETS> ; The Ada language has inherent compiler-inde­ pendent features that offer great promise for The body contains the current improving productivity. These fe atures can be implementation. categorized as either "new" (those that wou ld is programmer) or "improved" (those that may Dec larations go here. exist in other languages but have been improved begin in Ada). Details of the current sort

algorithm go here. Ne w Features and Th eir Benefi ts end ; The new features of the Ada language support several modern software engineering concepts The fo llowing benefits accrue from formaJiz. that can improve productivity. ing this separation of the specification from the body: Fo rmal Separation of Sp ecification and Body • Improving the implementation is easier. Many software interfa ces are designed with too Client software is less likely to depend on much knowledge of the current implementation details of the first implementation; instead, it details. In some cases, the interfaces are even depends on just the interface. produced as a side-effect of doing the imple­ mentation . This common error of blurring the • The client software is portable to other distinction between interface and implementa­ targets, because changes in implementation tion often results in haphazardly designed soft­ details are more likely to be hidden from ware with poor quality. The software is often clients rather than entangled in the interfaces. difficult to extend (because it depends unneces­ • Consistent user views are maintained. The sarily on some arbitrary detail of the first imple­ interfaces tend to be more logical because the mentation), and its performance is hard to design can be done before the implementa· improve (because the implementation cannot be tion details are considered. replaced by one with diffe rent arbitrary details) . Moreover, the software is often too closely cou­ • The software tends tO be more decoupled pled to functionally unrelated software, is not from unrelated software . (It tends tO be more portable to other targets, and is inconsistent in "modular.")

52 Digital Tecbnlcaljournal No. 6 February I 988 Software Productivity Tools

Pa ckages them easier for a user to fi nd and comprehend. Programmers often use the term "software pack­ The col location of related implementation soft­ ages" informally to describe collections of ware in the package body allows a maintainer tO related types, declarations, and operations. Con­ better understand the ramifications of a potential ventional languages provide no way ro bind such change. related software together. The Ada language formalizes the concept of Support fo r Abstract Data Ty pes package. An Ada package has a specification, Abstract data types are a relatively recent com­ which contains what the client software sees, puter science concept. An abstract data type rep­ and a body, which contains the package's cur­ resents a set of abstract objects having a we ll­ rent implementation. The specification contains defined set of applicable operations. A client can types, objects, and subprogram specifications operate upon an object only with the operations that define the interface for the package's provided. Information about the internal struc­ clients. The body supplies the bodies for the ture of the object is hidden from the client. subprogram spe cifications and anything else The benefits of this information hiding are needed to implement them. The specification that the programmer of the client software is and the body can be compiled separately, so that presented with a simpler conceptual model, interfaces can be developed and checked long one containing only the information that is before the implementation is coded. essential for a client to know. Furthermore, The fol lowing example shows a package client software cannot apply arbitrary opera­ specification and body for a window manager: tions to the object and thereby become depen­ dent on accidental internal det ails that might pac kage WIHDOW_MAHAGER is need to change in a future implementation. This section wou ld contain the Information hiding makes implementation of specification of da ta, the object easier, because hiding clarifies which procedure� , end functions that information current clients depend on and represen t or opera te upon which information can be changed without windows , end can be used by affecting them. client �of twere. Ada formalizes the concept of information hid­

The specification for a ing by allowing an implementor to declare "pri­

procedure that creates vate types" in a package specification. Al though window� . clients can declare objects of a private type, only procedure CREATE_W J HDOW the implementation code in the package body CW : out WIHDOW> ; can operate on the det ailed internals of such objects. For example, if the object represents a end ; window on the terminal, the client may only be allowed ro request that it be shrunk, etc. How­ pac k age body WIHDQW_MAHAGER is ever, the implementation may need to privately This section corre�ponds to the associate the window with a fi le and can store current implementation of the the file name within the window object, hidden window mana ger . from the client.

The body of a procedure that Figure 1 illustrates the two views of an

creates window� . abstract data type.

procedure CREATE_W I HDOW Ada's support of abstract data types should

CW : out WIHDOW> is enhance an impl ementor's ability to extend

begin applications in the future, as well as to make software more reliable.

end ; Ta sks end ; Most operating systems provide constructs to Packages promise easier use and modification support concurrent execution, even on unipro­ of software. Related objects and operations tend cessors. But these constructs are typically very to be grouped in the same package, thus making low level and difficult to use, and they are cer·

Digital Technicaljo urnal 53 No. () Fe bruary I 988 Software Productivity Features Provided by the Ada Language

pac kage WINDOW_MANAGER is

-- The client knows the object only by the operations -- that are provided here .

type WINDOW is private;

procedure SHR I NK_W J NDOW CW WINDOW ; P PERCENTAGE >; end ; pac kage body WINDOW_MANAGER is

-- The implementation knows the ob ject by its internal -- 5tructure. In this case the window is really a record. type WINDOW_ IMPLEMENTATION is

record F : FILE_TYPE ; end ;

end ;

Figure 1 Tw o Views of an Abstract Data Ty pe tainly not portable. The Ada language is one of tion. Implementations that support this feature the few widely available programming languages (such as VAX Ada) provide a powerful reason to that has built-in constructs for concurrency use tasks - even on a uniprocessor - to easily (called tasks) . Ada's tasks are both easy to use and obtain greater throughput and responsiveness portable; the language standard requires that all from an otherwise IjO bound program. On multi­ implementations support tasks in a specified processors, users expect that an Ada implementa­ manner. tion will assign different tasks to diffe rent pro­ AnAda task is like a procedure that executes in cessors for a program speedup. (VAX Ada does parallel with other parts of the program. A task not currently have this feature.) startS executing as soon as the first statement of Using tasks written in Ada rather than the con­ its declaring unit executes; tasks need not be currency primitives provided by the operating started explicitly. Like packages and subpro­ system can lead to better productivity because grams, a task has a specification and a body. • Tasks are portable (less rework) The following example illustrates the essential syntax of a task: • Tasks are easier to understand (reduced

ta!lk T i!l maintenance) Thi5 5ection wou ld dec lare • Ta sks are easier to code (faster time to market) the entrie!l that client !loftware can cal l. • Using tasks on a uniprocessor results in free end ; performance improvements when multipro­

ta!lk body T i!l cessor support becomes available begin Like other Ada constructs, tasks also improve Th i!l !lection wou ld contain the the overall thought process (even on a uniproces­ 5tep5 that the ta5k sor) . Programmers soon stop "thinking serially." execute!! in parallel wi th other ta5k!l. With tasks able to do asynchronous IjO, it becomes unthinkable to lock up the user's key­ end ; board while a program does work that could be Although not required by the Ada language done in the background. As a result, products specification, the Ada marketplace highly values using tasks are more likely to be responsive to the ability of an Ada compiler to execute other user inputs. This benefit is important for any tasks while one task is waiting for an 1/0 comple- product with a user interface.

54 Digital Technicaljournal No. 6 Fe bruary 1988 Software Productivity Tools

Separation of Representation fr om Ty pe vides a way tO force data representations and star­ In many programming languages, integer vari­ age layouts. ables are intimately bound tO the attributes of a Separating the types from their representations particular machine. For example, integer vari­ achieves portability between machines (provid­ ables are often bound tO the number of bits ing of, of course, that a machine type can be needed to represent some integer machine type . found that can satisfy the required range) . Error The Ada language separates the inherent prop­ checking is enhanced because the compiler auto­ erties of a variable's type from the underlying matically checks that the value of the variable machine types. For example, the declaration stays within its declared range. Exception Ha ndling type PLANET_NUMBER is range 1 . . 9; In the conventional languages without built-in SPACECRAFT_LOCAT ION : PLANET_NUMBER ; exception handling, a programmer must manage says only that type PLANET_NUMBER needs a status variables that indicate whether or not a call range of I to 9; the corresponding machine type to a procedure has failed. In addition, those lan­ is neither apparent nor important. The compiler, guages provide no easy means for automatically not the programmer, chooses which hardware passing error notifications to the calling routine data type will actually be used (for example, or for cleanly specifying recoveryactions . 8, 16, or 32 bits). This general concept - in In contrast, the Ada language has built-in fea­ which the most a programmer need specify are tures for handling exceptions. A programmer can the range and precision, and the implementation decide which error conditions should be defined, chooses the detailed representation - is present when they should be signified, and how they are for all Ada types. For cases in which detailed to be handled. Moreover, all these features are representations are important, the language pro- portable. Figure 2 shows how this feature works.

-- The error condition is dec lared .

LDST_THE_L INE : exception;

procedure GET_DATA is begin

if STATUS /• NORMAL

then

rai se LOST_THE_L INE ; -- Th is stateme nt signifies that the error has occurred .

end if; end ; procedure TEST_COMM_L INE is begin

Th is section calls routines that can raise except ions .

GET_ DAT A; exception

-- Th is sect ion specifies all the error handl ing code for this procedure.

when LDST_THE_L INE => NOTIFY_REPAIR_STATION;

raise; Th is statement passes the

- - current error to the cal ler. end ;

Figure 2 Example of Ada's Features fo r Exception Ha ndling

Digital Tecbnicaljourual 55 No. 6 Fe bruary 1988 Software Productivity Fea tures Provided by the Ada Language

Because error handling is easily programmed lete interface specifications. This feature saves in the Ada language, error-handling code is easier the time that wou ld otherwise be spent tracking to read, and the constructed software is likely tO down obscure run-time errors. be more reliable. The presence of a program library also creates the opportunity for Ada implementations tO Dy namic Me mory provide many other additional features and bene­ Another feature often absent from conventional fits. Some features that become possible, many of languages is the allocation and deallocation of which are now common among implementations dynamic memory. Although many operating sys­ in the industry, are tems provide this feature, most do not provide it • A simplified means of compiling and linking in a portable form . the entire program Ad a, however, has a bu ilt-in dynamic memory feature that all Ada implementations support: • A simplified means of recompiling program pointer variables can be declared, and they can units after another unit on which they depend have dynamic objects assigned tO them. Figure 3 has changed describes how pointer variables can be used. • Query functions to answer questions about the Ada's dynamic memory feature increases program as a whole portability. In addition, because Ada's pointers • Ways of using multiple libraries together to are typed (can on ly point tO objects of specified realistically match project needs type), two other benefits accrue: It makes their • Optimizations that take into account more use less error-prone, and it enables a compiler to than one procedure (interprocedural opti­ exploit the additional knowledge it has about the mizations) objects, such as their maximum and minimum • Subsystems, that is, the ability to restrict client size, for optimization. software tO use only certain allowed interfaces

Th e Program Library The more important productivity benefits In most conventional languages, each procedure likely tO accrue are faster development, and less is compiled independently of other procedures. time spent tracking down obscure errors. Thus it is possible to have a set of software rou­ tines that is inconsistent. For example, callers Overloading could assume certain conditions about the inter­ Most programming languages require a unique face, yet those conditions may have changed. name for each program unit declared. The Ada The Ada language addresses this problem by language, however, allows any number of pro­ requiring a program library to hold a consistent gram units to have the same name, provided that copy of all program units. The program library the units have different interfa ces, called signa­ manager makes it impossible to link a program tures. This concept, called overloading, allows that has internal inconsistencies caused by obso- procedures that have the same logical effect but

type PTR is access HODE ; Th is sta teme nt dec lares a pointer

type .

X PTR ; This statement dec lares a pointer

var iable.

X new NODE ; -- This sta tement dynamically

allocates an object and as signs

-- a po inter to it.

Figure 3 Ada Pointer Va riable

56 Digital Te cbnlca/]ournol No. 6 February 1988 Software Productivity Tools

that operate on diffe rent data types to have the dures are needed for sorting integers and one­ same name. For example, a package defining dimensional arrays, even though a programmer both vectors and matrices could have two proce­ may want to use the same sorting algorithm for dures called CREATE and two called DELETE. both. There is no need to construct artificially different In contrast, the Ada language allows the defi­ names to differentiate between the procedures. nition of a generic form of package, procedure, Overloading helps implement the principle of or fu nction . A generic program unit is indepen­ orthogonality, which means that each operation dent of the type of data on which the program applying to one object type can also apply to any unit operates. This fe ature allows algorithms to other object type, whenever that is meaningfuL be coded in their purest abstract form. Overloading is simply a generalization of the After defining a generic program unit, a pro­ common language feature allowing arithmetic grammer can then create an executable program operations to be applied to both integers and real unit by specifying the actual types upon which numbers. Ada allows a programmer to define the the generic unit is to operate. Creating an exe­ meanings of most built-in operators for any pro­ cutable instance of a generic unit is called grammer-defined data types; for example, one "instantiating" the generic unit. When instantiat­ can define an addition operation for one's own ing the generic, one can also specifyas parame­ matrix type. ters any procedures, functions, or tasks that need to be called by the unit. For example, a generic Generics sort package may need to pass the fu nction "less Many languages force programmers to recode than" for the particular type to be sorted, such utility operations for each type of data on which as a matrix. they will operate. For example, different proce- Figure 4 illustrates the use of generics.

gener ic

Dec lare that an arbitrary type is to be specified as the actual type to be sorted . type ELEMEHT_TYPE is private;

-- Since nothing is assumed about the type , the

-- comparison function mu st be passed in as a param eter .

��o�i th function "<" CL: ELEMENT_TYPE , R : ELEMEtH_TYPE>

return BOOLEAN ;

pac kage QU I CKSORT is

-- The sort package interface ��o�ould be ��o�r itten here .

end ;

package body QU I CKSORT is

The gener ic impl emen tation ��o�ou ld be ��o�r itten in this section; it ��o�ould use the passed-in function "<" .

end ;

The follo ��o�ing statements create an instance of the QU I CKSORT pac kage

for the type BOXCAR ��o�h ich is def ined in package TRAINS . The generated pac kage is no��o� capable of sorting BOXCARS using the guicksort method .

��o�ith QUICKSORT ;

package QS_BOXCAR is ne��o� QU I CKSORT CTR AIHS.BOXCAR , TRAINS."<" >;

Figure 4 Example Us e of Generics in Ada

Digital Technical journal 57 No. 6 February I 988 Software Productivity Features Provided by the Ada Language

Generics allow the easy reuse of software, thus Ad a also provides the subtype construct, a way avoiding the inevitable errors caused by recoding to name part of the full range of values of a type. an algorithm. The promise of generics is that one Ad a's rules for mixing variables of different can truly code an algorithm just once . types within the same expression are stronger Improved Fea tures and Th eir Benefi ts than the rules in other typed languages, such as Pascal. The only implicit type conversions Ada improves upon a number of features that are allowed in Ad a are between numeric literals and commonly provided by other languages . In gen­ compatible numeric types. No implicit conver­ eral, these features increase the amount of sions are allowed between values of different checking done at compile time, and thus save types. Ty pe conversions can be done , but pro­ debugging time and make modifications easier to grammers must explicitly write them. Every accomplish. compiler is required to report an error if an Strong Ty ping Rules attempt is made to combine different types with­ Like several other languages, Ada allows a pro­ out conversion. grammer to define not just variables but types of The main benefit of these typing rules is variables. A type is simply a template for a set of that, when used properly, they help with error values that share some properties. For example, detection. An Ada compiler will detect many the type named INTEGER describes a set of dis­ errors during compilation that might not even crete, signed values with a certain range and per­ be checked for in other languages, or that mits certain well-defined operations, such as might not immediately manifest themselves even addition and subtraction. (A type does not have a at run-time . For example, typing can prevent size; objects have that property.) the accidental truncation of a real number to The Ada language also allows a programmer to an integer when rounding is required. Simi­ define a derived type, which is a new type whose larly, typing can prevent the accidental addition properties are derived from a parent type. of a variable in feet to one in yards, or the addi­ Although a derived type has the same properties tion of an employee's ID number to his or her as its parent type, its values, like the values of any salary. other type, have different meanings and cannot Figure 5 illustrates several forms of type defini­ be mixed freely with the values of other types. tion and use.

-- De fine a mach ine- i ndependent type and a subtype . type PHYS I CAL_NUMBER i5 digits 8 range -1.0E35 .. 1.0E35 ; subtype POSITIVE_PHYS I CAL_NUMBER is

PHYSI CAL_NUMBER range 0.0.. PH YS I CAL_NUMBER 'LAST ;

-- De fine new types der ived from the above . type FORCE is new PHYS ICAL-NUMBER; type MASS is new POSITIVE_PHYSI CAL_NUMB ER; type ACCELERATION is new PHYS I CAL_NUMBER ;

-- Dec lare var iables of each type .

F : FORCE ; M : MASS ; A : ACCELERAT ION;

FORCE and MASS cannot be mixed in the same expression

wi thout an explicit conversion bac k to the base type ,

PHYS I CAL_NUMBER. Ada compi lers mu st diagnose the fo llo wing

illegal expression.

F :• M;

-- The follo wing statement is lega l because there are

-- explicit conversions .

F :• FORCE • PHYS ICAL_NUMBE R > ;

Figure 5 Example of Forms of Ty pe Definition

58 Digital Tecbnica/]ournal No. 6 Fe bruary 1988 Software Productivity Tools

The productivity benefits of Ada's typing rules suite) . These validation tests not only ensure are reduced error tracking (and, hence, more portability of Ad a code but also fo rce compilers productive use of development and maintenance to translate the whole language exactly as time) and more readable and reliable programs. required. (A compiler must be validated once a year and must pass all new validation tests.) Range Checking These tests ensure that legal programs are trans­ In addition to providing strong typing ru les, the lated correctly and that all required error report­ Ada language requires that range checking be a ing, both compile-time and run-time , is accom­ default behavior (with most compilers permit­ pl ished . ting its suppression). Range checking involves either compile-time or run-time checks for com­ ProductivityFe atures Provided by puted values that exceed the legal range for a VAX Ada type. VAX Ada provides additional productivity fea­ In many languages, calculations can overflow tures related to optimization, program library without being detected. Not only are such errors support, and smooth interaction with other often difficult to locate, but the erroneous results VAX languages and the underlying VMS environ­ could cause an unpredictable and possibly disas­ ment. (Detailed product documentation is pro­ trous outcome. For example, on one fateful day vided in references 6, 7, and 8. Reference 9 for a bank in New York, a number of real-time describes how VAX Ada fits in with the VMS financial transactions overflowed the 16-bit vari­ environment, and reference 10 describes some of able; millions of dollars were lost as a conse­ the design decisions that were made in develop­ quence. If the program had been written in Ada, ing VAX Ada.) built-in checking could have detected the error the moment the first transaction overflowed. Inlining The VAX Ada compiler implements the language­ Pa rameter-passing Checks defined pragma INLINE. (In Ada, a pragma is a Many languages do not detect the passing of the compiler directive.) The compiler thus replaces wrong number of actual arguments (either more calls to subprograms with inline code expansions or fewer) , or the accidental interchange of argu­ (unless the subprogram uses a feature like ments with different types. These kinds of prob­ tasks that prevent these expansions) . In addition lems often lead to obscure run-time errors that to honoring such explicit requests from the are very difficult to track down. programmer, the compiler will automatically Ada implementations, however, are required to replace certain calls with inline code; the com­ detect and report such errors the moment the piler uses a heuristic to decide if inlining results program unit is compiled. These checks immedi­ in more efficient execution than the call. (The ately save debugging time and eliminate the dele­ heuristic considers both space and execution terious effect that such errors have on product time.) quality. Inlining allows programs to run faster, and its use simplifies the conceptual design process. Va lidation and Po rtability Call overhead is no longer a concern, and pro­ Ada, unlike many languages, has strict rules on grammers can define the most logical interfaces. what implementations are allowed to do . An implementation is allowed neither to extend the ImportjExport Pragmas language it compiles nor to compile only a part The VA X Ada compiler defines several pragmas to of that language. As a result, all Ada compilers match the Ada language to the VA X Calling Stan­ translate the same language, yielding increased dard in an optimum fashion. For example, the portability between computer systems. Pro­ pragma IMPORT_PROCEDURE allows a call grammers need not learn the specifics of an to a procedure written in any language. This implementation in addition to the language. pragma permits a programmer to specifyparame­ Before an Ada compiler can be called vali­ ter-passing mechanisms (reference, value, and dated, it must pass a series of validation tests descriptor) , the procedure's external name, and (currently over 4,000, but the number has other VA X-specific attributes that are not part of increased with each release of the validation test the Ada language.

Digital Technical Jo urnal 5 9 No. 6 February I 988 Software Productivity Features Provided by the Ada Language

These pragmas allow the mixing of Ada sub­ in a task. To support VMS software interrupts programs with existing programs written in in a .like manner, the VAX Ada compiler pro­ other languages so that the benefits of Ada can vides an implementation-defined attribute called be obtained for new code, and the benefits of AST_ENTR Y. This attribute causes a software reusing existing code can also be rea lized. interrupt to generate a call ro a task entry point. This feature simplifies the interfacing of Ada Po rtability Checks programs to the VMS environment. VA X Ada can perform portability checks while compiling a program. These checks inform a Summary programmer of the uses of potentially non­ The Ada features we have discussed make this portable features. Such features include imple­ language an exce llent choice for general-pur­ mentation-defined pragmas and other features pose programming. Ada's principal productivi ty that the Ada language permits to allow tailoring benefits are realized in software that is more tO the specific computer system. These checks extensible, portable, maintainable, reliable, and allow a programmer to manage the trade-off reusable. The VAX Ada compiler adds further between portability and access to implementa­ features that enhance productivity. Wh ile each tion-dependent features. feature taken separately may not seem that sig­ nificant, the combined benefit of all of them Program Library Fu nctions should be quite significantly improved software As mentioned earlier, Ada's unique concept of a productivity (as defined earlier in terms of program library provides many benefits. The profit and life-cycle costs) relative to the use of VA X Ada program library provides all of those other languages. benefits, as well as some others. Acknowledgment • Any program units made obsolete by revising The author wishes to thank Barbara Rising a particular unit can be automatically recom­ Bishop for greatly improving the content and piled. readability of this paper. • The compilation order for the program units of a program can be determined by simply References naming the main program. 1. ]. Barnes, Programming in Ada (Reading: • An entire program can be bu ilt automatically Addison-Wesley Publishing Company, by specifying just the name of the main pro­ 1984). gram. 2. G. Booch, Software Components with

• Ada program units can be imported from Ada: Structures, To ols and Subsystems ot her program libraries, or from other lan­ (Menlo Park: The Benjamin/Cummings guages . Publishing Company, Inc., 1987).

Asynchronous IjO Op erations 3. G. Booch, Software Engineering with Ada The VAX Ada run-time library performs asyn­ (Menlo Park: The Benjamin/Cummings chronous inputjoutput for aJI predefined Ada Publishing Company, Inc , 1987). ljO operations. As a result, a programmer can 4. Ada Programming Language , ANSijMIL­ use tasks ro do computation and I/0 in parallel STD-181SA -1983 (United States Depart­ and obtain greater throughput and more respon­ ment of Defense, U.S. Government Printing sive user interfaces. Office, 17 February 1983).

Support fo r Asynchronous Op erations 5. In the examples throughout this paper, The VMS operating system defines the asyn­ lowercase is used to distinguish Ada chronous system trap (AST) construct for deal­ reserved words from programmer-defined ing with asynchronous events. An AST is really a identifiers (shown in uppercase) . Note that software interrupt. Ada allows hardware inter­ a comment in the language is identified by rupts to be mapped into a call ro an entry point using a leading double hyphen (--) .

60 Digital Te cb11icaljournal No. 6 February 1988 Software Productivity Tools

6. Developing Ada Programs on VA X/ VMS (Maynard: Digital Equipment Corporation , Order No. M-EF86A-TE, 1985).

7. VA X Ada Programmer's Run-Time Refe r­ ence Manual (Maynard : Digital Equipment Corporation, Order No. M-EF88A-TE, 1985).

8 . VA X Ada Language Reference Ma nual (Maynard: Digital Equipment Corporation, Order No. M-EG29A-TE, 1985).

9. C. Mitchell, "Engineering VAX Ada for a Multi-Language Programming Environ­ ment," Proceedings of the ACM SIGSOFT/ SIGPLAN Software Engineering Sy mpo­ sium on Practical Software Development Environments, ACM SIGPLAN No tices , vol . 22, no. 1 oanuary 1987) : 49-58.

I 0. R. Conti, "Critical Run-Time Design Trade­ offs in an Ada Implementation," Proceed­ ings of the jo int Ada Conference, Fift h Na tional Conference on Ada Te chnology and Washington Ada Sy mposium (March 1987): 486-495.

Digilal Tecbnicaljournal 61 No. 6 FebrumJ' 1988 Brian A. Axtell William H. CliJford, ]r. jeffr eyS. Saltz

Programmer Productivity Aspects of th e VAX GKS and VAX PHIGS Products

The recent availability of high-level, device-independent standards fo r computer graphics programming has made the graphics programmer's task easier and Ja r less time-consuming. Graphics programs, once major undertakings, can now be produced quickly and once written can be easily transported to other graphics devices and host systems. The VAX GKSand VAX PHIGS products are implementations of two of the major graphics standards. These products are based on a common architecture, the Base Graphics Architecture, consisting of five layers. The architecture was designed to incur minimum overhead in accessing high-petformance devices, allow the reuse of many code modules, andprovi de easy extension of the products.

Computer Graphics Standards By the mid-1970s, however, falling hardware Interactive computer graphics is the synergistic prices and the availability of better proprietary union of computer graphics output with an graphics interfaces allowed computer graphics assortment of input techniques to facilitate oper­ use to become fairly widespread. With this expan­ ator feedback and controL Interactive graphics sion came the realization that a standard graphics has long been recognized as the most effective interface was needed that was both high level and available mechanism for manjmachine interac­ device independent. Enough experience existed tion. The presentation of information in a visual by then tO attempt to design such a system . form takes advantage of the superb pattern recog­ The Core Graphics System, developed under nition capabilities of humans. We ll designed the sponsorship of the Association for Computing input techniques enable much more natural and Machinery, was the first significant device­ 1 efficient interaction than is possible through key­ independent graphics interface. The concepts board input alone. embodied in Core profoundly influenced subse­ Throughout the early years of computing and quent graphics standards. Core never became an well into the I 960s, the high cost of hardware and official standard , however, because too many the even higher cost of sofrware production in­ nonconforming implementations became avail­ hibited the widespread use of interactive compu­ able before standardization could be achieved. ter graphics. The expense of writing software was Moreover, the experience gained from these early due in large part to the graphics interfaces of the implementations of Core showed that a better period, which were typically device dependent interface could be designed. and at a very low functional level. Because of the The Graphical Kernel System, or GKS, is a ben­ low level of these interfaces, the application had eficiary of the experience gained from Core and to do a lot of work to create even a sim pie image . was adopted both as an ANSI and ISO standard 2 thus making graphics programming diffi cult and in 1985 It is the first official ANSI/ISO graphics time-consuming. The device-dependent nature of standard. Digital's implementation of GKS for these interfaces often meant that programs had to the VMS operating system, called VAX GKS. be largely rewritten for each diffe rent device on has been available since I 984 and is now in its which graphics output was to be produced. third release.

62 Digital Te chnical jo urnal No. 6 February 1988 Software Productivity Tools

The Programmer's Hierarchical Interactive to the Workstation Manager section for more Graphics System, or PHIGS, is being developed as details.) both an ANSI and an ISO standard. 3 PHIGS has a • VAX GKS or VA X PHIGS workstation handlers diffe rent architecture and different design goals are activated at run time, not directly linked from GKS; it is compatible with GKS, however, with the application. Therefore an application wherever this compatibility is technically fe asi· can operate any supported device without ble. The first release of VAX PHIGS became avail· re linking, thus minimizing the link time and able in January 1988. executable program size of an application. The Role of VAX GKS and VAX PRIGS This al lows the application programmer to Products in SoftwareProductivity spend more time doing productive debugging.

The VAX GKS and VAX PHIGS products are high­ • Both VA X GKS and VAX PHIGS provide exten­ leve l functional interfaces that make graphics pro­ ded input and output fu nctionality. Additional gramming easier and more straightforward. A output pri mitives implement common objects competent programmer should be able to (for example, circles and ellipses in VAX develop graphics software after a short period of G KS) , thus saving the programmer from hav­ study. Moreover, the use of these systems eases ing to build them from lower level pr imitives. the difficulty of finding trained graphics applica­ Extended output functionality (for example, tions programmers. Many colleges and universi­ lighting, shading, and depth cueing in VAX ties teach GKS as part of their computer-graphics PHIGS) provides services not easily layered on curricula; there is evidence that they will teach top oft he basic mode I. PHIGS as we ll. Each standard is built on the model of an • Both VAX GKS and VAX PHIGS integrate abstract graphics workstation with standardized cleanly into a windowing environment. VA X input and output capabilities. That is, each stan­ GKS and VAX PHIGS enable the creation of dard defines an idealized device for acquiring applications with the same "look and feel" as input from the operator and for generating graph­ other applications written using low-level ics output . To a large extent, the job of the GKS or windowing commands. The application pro­ PHIGS implementor is to provide a layer that grammer is thus freed from concerns about the makes a particular real device behave like an windowing system, and the user sees a consis­ ideal device. We call such a layer a "workstation tent windowing interface. handler." If the abstract workstation model of • Support for many devices is available through GKS or PHIGS can be implemented quite directly both VAX GKS and VAX PHIGS. VAX GKS is on a rea l device, such a device is called GKS-Iike 4 supported on VWS workstations and on or PHIGS-like. ReGIS, HPGL, POSTSCRIPT, and Sixel devices. This device-independent approach means PHIGS is supported on these devices as well that a program written to the GKS or PH IGS as on the VA Xstation 8000 high-performance interface can, within limits, run unchanged on workstation. any supported device. Moreover, if an appli­ cation is written to one of the standard language VAX GKS and VAX PRIGS as bindings (i.e., uses standard fu nction names and Complementary Graphics Standards parameterization), then the application can eas­ GKS and PHIGS are designed to satisfy diffe rent ily be ported to any other GKS or PHIGS host sys­ needs. GKS is a two-dimensional interactive view­ tem supporting that language binding. ing system. That is, GKS provides a mechanism Although the basic fu nctionality of both GKS by which images, described as collections of two­ and PHJGS is strictly defined, Digital's implemen­ dimensional output primitives , can be displayed. tations fu rther increase programmer productivity In addition, GKS provides a variety of operator in several ways. input methods. Although GKS can be used to • Both VAX GKS and VAX PHIGS provide for very display graphs and charts and other two-dimen­ easy support of additional devices. Any user sional information, it can also be used as the can add support for a new device by simply imaging stage of a higher level system. Such a sys­ writing a few functions and building a table tem first does the necessary processing to convert defining the capabilities of the device. (Refer its objects (for example, objects defined in three-

Digital Te chnicalJo urnal 63 No. 6 Fe bruary 1988 Programmer Productivity Aspects of the VAX GKS and VAX PHIGS Products

dimensional space) into GKS two-dimensional PHIGS: to be able to describe and implement the output primitives for display. application model (the concept or system being GKS commands for setting attributes of pri­ modeled by the application) and to allow the effi­ mitives and for generating output primitives can cient revision of that implementation and its cor­ be either aggregated into collections called responding image. "segments" or executed immediately without Since typical application models are multi­ being retained in a segment. The set of defined level, PHIGS structures can be organized hierar­ segments can be used later to rebuild the chically. The relationship between levels of the image on a display, thus the application docs not hierarchy may represent geometric relation­ need to reissue those commands. GKS segments ships (for example, the positional relationship are not revisable; once defined, the contents of between the components of an articulated arm) a segment cannot be modified. However, each or other logical relationships as dictated by appli­ segment has several attributes (for example, a cation requirements. (One can think of GKS seg­ segment transformation and a segment visibility ments as providing a single level of hierarchy.) control) that can be changed . In PHIGS, for example, one can define a "wheel" PHIGS is a three-dimensional interactive mod­ structure (a collection ofcommands that describes eling system. Like GKS, PHIGS offers operaror something having the appearance of a wheel) and input techniques; unlike GKS, however, PHIGS a "body" structure . One can then define a "car" primitives are defined in a three-dimensional structure as an assemblage consisting of four coordinate system rather than a two-dimensional instances (displayed images) of the wheel and system. PHIGS also does not have immediate exe­ one instance of the body structure. Each instance cution of output commands. Instead, all com­ has a geometric relationship to the car (that is, it mands that set attributes of output primit ives, has some position with respect to the car) . The generate output primitives, or define other car can be moved around as a unit, with the rela­ aspects of the graphics database are aggregated tive positions of the wheels and body maintained into revisable collections called "structures." by virrue of the geometric relationships imposed This approach is motivated by two of the goals of on them. This concept is illustrated in Figure 1.

Wheel and 11re data courtesy ol Bngham Young University. Car bOdy data copyright Evans and Sutherland Computer Corporation.

Figure 1 PHJ GS Structure Hierarchy

64 Digital Tech nicaljourna/ No. 6 Fe bruary 1988 Software Productivity Tools

GKS and PHIGS share a common set of output primitives, although GKS allows their definition APPLICATION APPLICATION only in two dimensions, whereas PHIGS allows it LAYER in three. These primitives are lines, symbols ("markers"), polygons, text, and pixel arrays. GKS and PHIGS also share a common input model. This model defines six input device types BINDING that are abstractions of common physical input BINDING LAYER devices, such as pointing devices, dials, button boxes, and keyboards. GKS was designed as a very general interfa ce on top of which many different classes of applica­ KERNEL tions can be built. In fa ct, it is possible to build a KERNEL layer on top of GKS that emulates PHIGS. PHIGS LAYER is also a very general interfa ce, and the range of applications that can make use of PHIGS' model­ ing and three-dimensional capabilities is large WORKSTATION WORKSTATION and diverse. However, PHIGS is a bit more spe­ HANDLER MANAGER cialized than GKS, and building some styles of graphics applications on top of PHlGS is difficult if the application model cannot be made to fit PHIGS' modeling facilities. The benefits of incorporating modeling into WORKSTATION LAYER PH IGS are twofold. First, the application writer has a we ll developed, standard modeling facility avai lable. Second, it becomes economically feasi­ ble for hardware vendors to provide direct hard­ DEVICE LAYER ware support for PHIGS modeling because the modeling facility is standardized. Such devices can provide very high performance. Figure 2 Layers of the Base Graphics Architecture The VAX PHIGS and VAX GKS products have a common architecture called the Base Graphics

Architecture, described next. • VAX GKS and VAX PHIGS products had to con­ form to their respective standards. Base Graphics Architecture Digital's Base Graphics Architecture was • Since performance is critical in graphics sys­ designed as a general framework for implement­ tems, the architecture had to allow access to ing graphics systems. It takes a layered approach any high-perfo rmance hardware features of a and consists of five components: device. Moreover, the system had to incur min­ • Application layer imal overhead in using those performance fea­

• Language binding layer tures.

• Kernel layer • Adding support fo r new devices had to be rela­ • Workstation handler layer tive ly easy. • Device layer

This approach is shown in Figure 2. Following • Wherever possible, components of each a discussion of the design goals of the Base implementation of the architecture had to be Graphics Architecture, each of these components interchangeable and reusable. is described . • The architecture had to provide a mechanism Design Goals for customers and third-party vendors to write The design team set a number of goals for the graphics handlers for their own devices and tO design of the Base Graphics Architecture. integrate them into the system.

Digital Tecbnical]ournal 65 No. 6 Februmy 1988 Programmer Productivity Aspects of the VAX GKS and VAX PHIGS Products

• The architecture had to be extensible enough Th e Ke rnel Layer to allow for changes in both graphics hard· The kernel is the portion of the architecture that ware technology and graphics standards. manages and controls the device-independent operations of the graphics system. Its main func­ Each layer of the Base Graphics Architecture is tion is to act as a router, directing commands to described below. the appropriate workstation (multiple worksta· The Application Layer tions can be simultaneously active in both GKS and PHIGS) and to serve as a collection point for The application layer is not a part of the graphics input events generated by the input devices. The system but is, rather, a user of its services. The kernel also maintains all information about the application programmer defines and has total state of the system as a whole and is capable of control of the application layer. In reality, the responding to inquiries about system state and application layer is typically another series of facilities. For example, an application can layers of increasing fu nctionality, not the monoli· inquire about the types of devices available or thic component depicted in Figure 2. The inter· the number of active workstations. Furthermore, face seen by the application layer is called a the kernel is responsible for reporting errors back language binding and is supported by the bind· to the application. ing layer. Another responsibility of the kernel is to acti· vate the workstation handlers. These components Th e Binding Layer of the workstation layer are not linked directly A language binding is a functional interface to the with the higher levels of the system, but instead capabilities of the graphics systems. A binding are built as shareable images. When the services layer consists of several bindings, but a typical of a handler are first needed, the kernel activates application uses only one language binding. Each the handler through a VMS library routine. binding is oriented toward some particular lan­ The advantages of dynamically activating a guage or calling convention. The language bind· workstation handler, rather than linking some or ings of the binding layer fall into rwo categories: all handlers directly with the application, are as follows: • Standard bindings for single languages. These

are language bindings developed by the same • A user-supplied handler can be incorporated organizations that developed the graphics without the need to link it (that is, using the standards. For example, the VAX GKS binding VAX linker) directly with the application and layer includes a FORTRAN language bind· kernel. It is only necessary to define several ing that conforms to an ANSI/ISO standard. logical names that indicate the file name and Such a single-language binding is designed to entry-point table symbol name for the particu­ be consistent with the capabilities of the par­ lar workstation type. (An entry point table is a ticular language. Programs written to such structure similar to a VMS transfer vector.) standardized bindings are portable to any implementation of the same binding for that • Link time is substantially reduced because an graphics standard. application is only linked against the language binding interface, which is itself a shareable • VAX calling convention bindings . Each bind­ image. ing layer provides a VAX run-time library (RTL) binding, called the GKSS and PHIGS S • The amounts required by the application of bindings for VAX GKS and VAX PHIGS, respec­ both disk space and virtual address space are tively. These bindings conform not to inter­ significantly reduced. national standards but rather to the VAX fVMS calling standard. Therefore, VAX GKS and The VAX GKS and VAX PHIGS kernels are opti· VAX PHIGS can be accessed through these mized for the most common fu nctions. They bindings from any language that conforms to incorporate various caching schemes and "hot this calling standard . paths" to accelerate performance for expected configurations and call sequences. Therefore, The language binding layer is typically a very for many functions, the kernel merely has to per­ thin shell over the kernel layer. fo rm one or rwo tests and then call the next layer.

66 Digital Teclmical Jo urnal No. 6 February 1988 Software Productivity Tools

Th e Wo rkstation Layer calls the "open workstation" function of the The collection of workstation handlers that con­ workstation manager. The workstation manager, stitutes the workstation layer is responsible for in turn, activates the appropriate device-handler implementing the workstation abstraction of its shareable image, again through a VMS library particular graphics standard . For any graphics sys­ routine. tem based on the Base Graphics Architecture, the The main job of the workstation manager is to workstation handler interface must be defined at span the semantic distance between the worksta­ a very high level to allow access to the high-per­ tion handler and the device handler interfaces. formance features of a device. The exact nature of that job differs depending on The fu nctions of the workstation handler inter­ the abstract workstation implemented by the face for VAX GKS, for example, are basically one­ workstation manager. However, a typical worksta­ to-one with GKS functions that deal with the tion manager does the following tasks: input, output, and workstation state. The work­ • Maintains state information on behalf of the station handler interface for VAX PHIGS has a low-level device similar relation to the PH IGS functions. Thus the implementor of a workstation handler can take • Performs necessary geometric transformations advantage of the capabilities of those devices that closely match the workstation abstraction. • Simulates functionality not available through a Although the workstation hand ler interface thus particular device handler includes many entry points, the implementation • Performs data management of aggregated out­ of each function should be re latively straightfor­ put primit ives and attributes (for exa mple, ward for devices that closely match the worksta­ GKS segments or PHIGS structures) tion abstraction . However, most current devices do not have • Responds to inquiries about workstation state an architecture that closely matches the work­ and available facilities station handler interface. Very few devices, for example, could be considered GKS-like or In reality , the design of the device handler PHIGS-like (though this situation is slowly interface requires that each workstation manager changing) . The job of writing a workstation implement its particular workstation abstraction handler for a low-leve l device is indeed an for a range of abstract low-level devices. That is, a arduous one . To minimize the effort needed to device handler need not implement the entire interface such a device, an abstract graphics low-level abstraction. The workstation manager is device at a much lower level has been defined . expected to simulate those functions not sup­ Therefore , each implementation of the Base plied by the device handler. In fact, the only Graphics Architecture needs a workstation han­ mandatory output function for the device handler dler that implements that implementation's is a fu nction which draws a series of connected high-level workstation abstraction for this low­ lines. If necessary, the workstation manager will level abstract graphics device. simulate all other output primitives in terms of VAX GKS, for example, has such a special that single primitive. Similarly, most of the input workstation handler that implements the GKS fu nctionality of the device handler is optional; workstation abstraction for the low-level device. the workstation manager will simulate the miss­ VAX PHIGS also has one for the PHIGS abstrac­ ing functionality. tion . This special workstation handler is called a For example, both VAX GKS and VAX PHIGS workstation manager; an implementation of the can generate polygons whose interiors can low-level abstraction is called a device handler, be fi lled with a solid color or with various as shown in Figure 2. crosshatched patterns. The implementor of a device handler may choose to support this primi­ Wo rkstation Ma nager tive directly if the device for which the device To the kernel, the workstation manager is just handler is being written has this capability. If another workstation handler. It is activated the the device does not provide this capability, the same wayand is accessed through the same inter­ device handler can have the workstation manager face as other workstation handlers. After activat­ simulate fi lled polygons using the line-drawing ing the workstation manager image, the kernel primitive of the device handler.

Digital Tecbnical journal 67 No. 6 Fe bruary 1988 Programmer Productivity Aspects of the VAX GKS and VA X PHIGS Products

Device Ha ndler stalling it in place of the previous version. Relink­ A single device handler interface is common to ing the application program, kernel, and worksta­ all implementations of the Base Graphics Archi­ tion manager is not required . tecture. As a result, PHIGS supported several dozen devices immediately when the develop­ Device Layer ment of the PHIGS workstation manager was com­ The device layer, the lowest layer in our architec­ plete . Those supported devices were the ones for ture, marks the lower boundary of the Base which device handlers had previously been Graphics Architecture. This layer consists of the developed in support of VAX GKS. various devices made available to the application The device handler interface defines 27 func­ by the higher levels of the architecture. The tions, a device description table, and an entry­ interface to this layer is device dependent. point table (a transfer vecror) . The workstation manager consults both the device description Integration with Windowing Sy stems table and the entry-point table to determine what The Base Graphics Architecture was designed so functionality is available through the device han­ that a windowing system could be treated as just dler and what must be simulated. another device type within the Base Graphics Because of the adaptability of the workstation Arc hitecture . The model for supporting window­ manager, a new device can be added by wri ting ing systems realizes each instance of a "worksta­ just seven device handler functions and then tion" (the abstract device, not a specific device, building the device description and entry-point such as a VA Xstation 11/GPX workstation) as a tables. Such a minimal implementation will not separate window on the device's display. Thus provide optimal performance for most devices, multiple GKS and PHIGS workstations can be but wi II allow them to be put into service active on the same device under the control of quickly. Over time, the implementor of a device one or more applications. For example, "work­ handler can add more device handler functions station" windows for both PHIGS and GKS and a tO take advantage of the capabilities of the dev­ third window created through the graphics com­ ice. As it becomes available, each new version of mands of the native windowing system can all co­ the handler is placed into service simply by in- exist on the same display, as shown in Figure 3.

Figure 3 Th ree Workstation Windows in One Screen Disp lay

68 Digital Technicaljournal No. 6 Fe bruary 1988 Software Productivity Tools

The method by which a windowing system is implementation of these primitives as extensions supported (whether through a workstation han­ can also take advantage of any support provided dler or a device handler) logically depends upon by the underlying device (for example, if the the level of graphics support provided by the device has a circle primitive). VAX GKS also pro­ windowing system . For example, the VMS Win­ vides control over such things as line join and cap dowing System (VWS) is supported through a styles, as well as primitive "writing modes" (for device handler because VWS is neither PRIGS­ example, replace, complement, negate) . like nor GKS-like enough to warrant writing a fu ll 4 workstation handler. However, if three-dimen­ VA X PHIGS Extensions sional and other higher level capabilities exist in In November 1986, an ad hoc working group a windowing system (for exa mple, the proposed representing some twenty companies and univer­ 5 6 X3 D-PEX extension to the X Window System ), sities, including Digital, was formed tO propose then it might best be supported with a worksta­ and develop extensions tO PHIGS in the area of tion handler. lighting, shading, depth cueing, back-face sur­ The implementor of the handler can use any face processing, and curve and surface represen­ tool kit that the windowing system provides tO tation . The set of functionality formulated by this create windows and tO perform certain classes of group is called PHlGS + . input operations. For example, typical tool kits Wh ile PHIGS+ is not an official standards provide menuing capabilities that can be used to effort, a baseline document has been made ava il­ support the CHOICE input type defined by able tO the members of the ANSI PHIGS commit­ 7 PHIGS and GKS. When the tool kit is used for all tee for comment. It is the intent of the ad hoc possible windowing operations, all windows have PHIGS + working group that a revised PHIGS + the same appearance to the user or application draft be made ava ilable to the official standards programmer, even when they are generated by bodies when the document is complete. different graphics standards. The net effect is a VAX PHIGS includes extensions in most of the graphics standard operating within the window­ areas being addressed by the PHIGS+ group. ing environment. VAX PHIGS supports depth cueing, back-face sur­ fa ce processing, several different types of lights, Extensibilityin the Base Graphics various surface rendering effects (methods for Architecture simulating shiny or matte surfaces) , and an advanced output primitive . The Base Graphics Architecture includes a mech­ These extensions are defined in the VAX PHIGS anism that allows an implementation to provide workstation handler interface. Where possible, extensions in a manner conforming to standards. these extensions are also simulated by the PHIGS Such extensions can define additional output workstation manager. Therefore , within the lim­ primitives and provide extended control capabil­ its of a particular device, these extensions are ities. The VAX PHIGS and VAX GKS products use available on all devices supported through the these mechanisms tO provide extensions sup­ device handler interface . Using these extensions ported by Digital. In addition, the design of the effectively, howeve r, is possible only on a device Base Graphics Architecture enables the imple­ that can simultaneously display a reasonably mentors of workstation or device handlers to add large number of colors or shades of gray. Cur­ their own extensions. The special capabilities of rently the PH IGS workstation manager requires a particular device, not otherwise accessible that a device be able to simultaneously display through the standard functionality, can be made 64 colors in order tO simulate these extensions. available in this way. Summary VA X GKS Extensions The VAX GKS and VAX PHIGS products are VAX GKS has extended output primitives for gen­ extended implementations of the existing GKS erating various unfilled and fi lled circles, circu­ and proposed PHIGS computer graphics stan­ lar arcs, ellipses, elliptical arcs, and rectangles . dards, both of which are high level and device These are frequently needed primitives which an independent. Both PHIGS and GKS make com­ application programmmer would otherwise have puter graphics programming far less complex to generate using standard GKS primitives . The than in the past. Moreover, they allow program

Digital TecbnlcaiJournal 69 No. 6 Fe bruary 1988 Programmer Productivity Aspects of the VA X GKS and VA X PHIGS Products

portability among diffe rent graphics devices and References different host systems. These qualities can lead to 1. Computer Graphics, vol . 13, no. 3 (August greatly increased application programmer pro­ 1979). ductivity. Both VAX GKS and VAX PHIGS are based on a 2. Standard ISO 794 2 for Information Process­ single architecture designed by Digital. This ing Systems, "Graphical Kernel System architecture allows the efficient utilization of (GKS)," International Standards Organiza­ high-performance devices, the reuse of large por­ tion (1985). tions of code during implementation, flexibility 3 Draft Standard ISO 9592 for Information in the approach taken tO support a particular Processing Systems, "Programmers Hierar­ device, and access tO the unique capabilities of a chical Interactive Graphics System device . This approach has boosted the produc­ (PHIGS) ," International Standards Organi­ tivity of the VAX GKS and VAX PHIGS implemen­ zation ( 1987). tation teams , and is expected to minimize the work required of third parties to add device sup­ 4. Micro VM S Wo rkstation Graphics Program­ port . ming Guide (Maynard: Digital Equipment Corporation, Order No. M- G1 1 0B-TN, Acknowledgments 1985). The authors wish to acknowledge the tremendous 5. PEX Protocol Sp ecification, Ve rsion 3. 00 contributions made by the VAX GKS and VAX (Boston: Massachusetts Institute of Te chnol­ PHIGS development teams: Dwight Brown, ogy, 1987). George Chaltas, Kenney Chan, Ke ith Comeford, Glenn Davison , Jeff Ford, Garry Poege l, and Moh­ 6. X Wi ndow Sy stem Protocol, Ve rsion 11, Fung Shen. This effort could not have been made Project Athena (Boston: Massachusetts without continuing guidance from John Institute of Technology, 1 987). McConnell. Finally, we wish to thank Joy Ki n­ 7. PHIGS + Functional Description, Revision near for her constant support of our efforts. 2.0, report issued by the ad hoc PHIGS + Committee (1987).

70 Digital Tecbnical]ournal No. 6 Fe bruary 1988 LewisLasher I

The VAX RALLY Sy stem - A Relational Fo urth-generation Language

1be VAX RALLY sys tem, a fo nns-based fo urth-generation language, is designed to simplify the production of interactive database applications. 1bedesigners of this system sought a balance between ease of use andflex­ ibility in the development of the object-based defi nition system. The defi ni­ tion system allows commonly anticipated fe atures to be implemented by nonprocedural means, and otherfe atures to be implemented bymeans of escapes to otherlanguages. The run-time environment allows many users non-interfering, concurrent readjwrite access to the same data. The rep­ resentation of an application by a set of objects allowed the definitionsy s­ tem to be implemented as a RALLY application. This use of RALLYfo r its own user interface gave the designers a fast and effective means to make product improvements.

The VAX RALLY system is a fo rms-based , founh­ RdbjVMS databases. The definition system con­ generation language ( 4GL), or application gener­ sists primarily of the following: ator, for database applications. Like other • An object-based, nonprocedural set of tools fou nh-generation languages, it increases the pro­ • ductivity of application definers by providing A procedural language whose syntax is adap­ them with high-level constructs that simplify ted from the Pascal language application development. • Interfaces to programs written in traditional, This paper discusses the design principles third-generation programming languages (3GL) that have contributed to RALLY's usability by The run-time environment provides the fo llow­ application definers and to the efficient develop­ ing: ment of the RALLY definition system itself. In • Vinual multitasking in a single VMS process particular, attention is given to the design trade­ offs inherent in 4GLs; the design of an object­ • Flow control within and among tasks based environment; and the use of the same • Screen painting data model for application data as for the applica­ • Capture and validation of input from the user tion itself. of an application

ProductDesign Overview • Data manipulation operations to and from Ve rsion 1.0 of the VAX RALLY system was databases developed over a three-year period ( 1983- The users of VA X RALLY fall into two classes 1986) as a combined effon by Digital Equip­ corresponding to its two major divisions: applica­ ment Corporation and Foundation Compu­ tion definers who use the definition system to ter Systems, Inc., the original authors of the build applications, and end users who use these "ALLY" 4GL product. Digital is currently applications. However, because the definition developing later versions of the VAX RALLY system itself uses a RALLY application for its system. forms and menus, both classes of users, and VAX RALLY is an application definition system indeed both of the major divisions of RALLY, are and run-time environ ment for applications using affected by the same design decisions.

Digital Technicaljo urnal 71 No. 6 February 1988 Th e VAX RA LLY Sy stem - A Relational Fo urth-generation Language

Trade-offs between Ease of Us e RALLY includes an integrated procedural lan­ and Po wer guage called ADL (Application Development Lan­ A recurring trade-off in the design of RALLY, as in guage) . ADL can be used for application features, 4GLs generally, involves the tension between the such as arithmetic formulae, that are easier to goals of ease of use and power, or flexibility. If describe using a language than by fi lling our the 4GL is roo complicated, then users may fi nd forms. A definer typically uses an ADL procedure it simpler to continue using traditional 3GL pro­ ro specify the formula for a computed field, ro gramming methods. If the 4GL is nor sufficiently define the conditions for validating data opera­ powerful to meet the users' needs, users may tions, to manipulate data outside of a form or have no choice but to resort to 3GL program­ report, or to alter flow control . ming. Lastly, a RALLY application can be integrated programs wrirren in rradirional program­ RALLY accommodates both these goals in sev­ wirh eral ways. First, RALLY is specifically optimized ming languages. AJrhough this does nor con­ rribure directly tO the productivity improve­ for the development and execution of a particular ments realizable by using a 4GL, ir expands the class, albe it a large class, of applications, namely, range of applications that can use RALLY interact ive database applications. Second, RALLY provides the application definer with a small set of carefully chosen objects and the ability to Kinds of App lications Generated combine these simple objects into complex com­ The word "application" can refer to practically binations. Finally, RALLY provides four different, any effect achievable by traditional, 3GL pro­ partially overlapping approaches ro application gramming languages. The goal of RALLY is more development: the builder rools, the editing envi­ focused: ro simplify the production of interac­ ron ment, an integrated procedural language, and tive database applications. Within the narrower the ability to interface with programs written in domain of these data processing applications that traditional programming languages. In effect, the RALLY generates, it is possible ro predict what designers solved the problem of having ro trade fea tures are most likely ro be useful. The concep­ off either ease of use or flexibility by resolving tual basis for both the definition system and the the problem in, not one, but several aspects of run-time system was largely governed by such the RALLY sofrware . predictions about the types of applications RALLY The bu ilder rools are a greatly simplified sub­ wou ld generate. set of the features available in RALLY as a whole. The VAX RALLY product, like other 4GLs,sim­ Few choices are given to the application definer, plifies application development by providing the extensive defaulting is used, and many RALLY application definer with a small set of high­ concepts are simplified or omitted alrogether. level constructs and the tools with which to com­ By using the builder tools, a novice can learn bine them. Because RALLY focuses on database the most important RALLY concepts immediate.ly, applications, many of RALLY's constructs are data­ build a usable application in a short time, and base operations: reading, inserting, deleting, and defer learning other RALLY concepts until updating records; and committing and rol ling needed. back transactions. The major components of such The editing environment is the largest part of applications are fo rms through which end users RALLY. It is almost entirely fo rms-based. An appli­ enter data to be written to the database, and cation definer fi lls in blanks on forms to specify reports on which end users see data that has been rhe options to employ or rhe connections to make read from the database. Finally, although the pri­ between RALLY objects. Using these forms, the mary focus of RALLY is on interactive applica­ definer has no need to learn or recall a language tions, RALLY is also designed to be able ro pro­ syntax . Although the set of features RALLY offe rs cess records in batch. is nor simple, the entire set is organized through RALLY's menus, and each form is labeled . Because The Object-based Definition Sy stem each form presents a manageable amount of func­ RALLY conceives of an application as an inter­ tionality, it can be documented using on-line connected network of objects. Each object has interactive help messages specific to each form attributes that represent characteristics of the or to each field on a form. application.

72 Digital Te chnicaljou rnal No. 6 february 1988 Software Productivity Tools

Virtually an entire VAX RALLY application can applications; DSDs are invisible auxiliaries to be defined by filling out forms to specify the forms or reports. Ye t there are two strong reasons attributes of and connections between objects. for the existence of the DSD as a separate object: The application definer gives each object a name data independence, and reusability. that is used when connecting it to another object. Data independence means the isolation of an Because the application definition system con­ application from unnecessary dependence upon trols the storage and retrieval of objects, it always the details of data storage. Most characteristics of makes available to the application definer a list of a database application, and most characteristics all the objects at the definer's disposal. These of a form or report, are unconcerned with such lists, called lists of values, improve productivity details. RALLY isolates these storage details, such because the definer need not rely on memory or as the type of database and the names of relations written notes. Because the definer can specify and databases, in the DSD object. The form/ connections by moving the cursor to the list of report object handles the user-visible features of values and pointing at the appropriate name, the application, such as how data is formatted without having to type the name manually, the and, in the case of forms, validated. definer is encouraged to give objects long, Reusability refers ro the ability ro make several descriptive names. uses out of information that is defined only once, The major typesof objects that may be defined thus avoiding redundant and time-consuming in a RALLY application are tasks, menus, form/ work. The same DSD, describing the same source reports, data source definitions, ADL procedures, of data, may connect to several different forms external program links, number formats, and and reports. Conversely, a formjreport can be date formats. Formjreports and data source defi­ reconnected at different times to different DSDs, nitions, which are discussed in more detail later allowing a form or report with the same fe atures in this paper, contain subobjects such as fields. to operate on a different set of data. The design of these objects was governed by An additional reason to separate out DSDs as principles of modularity. When a set of charac­ objects in their own right is the design goal of teristics is likely to be used together, those char­ supporting both interactive and batch process­ acteristics belong in the same object. When a ing. Interactive data processing occurs in RALLY subset of these characteristics is likely to be in form/reports; batch processing occurs in ADL changed while the other characteristics remain procedures. For both types of data processing, an unchanged, that subset may belong in a separate application definer must specify the source of object. These decisions are heavily dependent data, restrictions on record selection, and a lock­ upon knowledge of how the characteristics are ing strategy. The information that must be speci­ viewed and used by users, in this case, by appli­ fied for both interactive and batch processing is cation definers. contained in the common object, the DSD. The most significant design decisions in the set The existence of DSDs as separate objects con­ of RALLY object types were tributes tO the goal of simplifying ADL syntax. A small set of high-level primitive functions serves • The separation of data source definitions from for all access methods. forms and reports Fo rm and Report Fu nctions in a Single • The unification of forms and reports into a sin­ Object gle object type The VAX RALLY system treats both forms and The fol lowing sections describe the data reports as a single object, called a formjreport. source definition and formjreport objects, and Despite the common practice, even in this paper, the data groups which are the basic structure of of referring tO forms and reports as distinct phe­ formjreports. nomena, they share essential characteristics for the display and formatting of data records with Data Source Definitions accompanying text. Providing a single construct The data source definition (DSD) , unlike the simplifies the concepts that a definer must learn. formjreport or menu, is not an object with an Moreover the formjreport is an inclusive, not dis­ obvious justification for existence. Forms, junctive, generalization of the characteristics of reports, and menus are visible components of both forms and reports. Not only the conceptual

Digital Tecbnicaljournal 73 No. 6 Fe bruary 1988 The VA X RALL Y ,�ystem - A Relational Fo urth-generation Language

definition of the formjreport object bur also sources, possibly in different databases or in dif­ each instance can include the union of the sets of ferent kinds of databases and fi les. More specifi­ form characteristics and report characteristics. cally, groups in a formjreport form a hierarchy Because a single RAlLY formjreport object can that reflects the re lationship between diffe rent handle the entire set of interactive data opera­ data streams. Each group can have one or more tions, the definer can attain considerable produc­ children groups . In such a parentjchild relation­ tivity after mastering a sma II set of concepts. ship, the data in the child group is related to and A formjreport by default reads records from dependent on a record in the parent group; a one or more data sources, disp lays them to the field or set of fields in each child record must be user, and performs data manipulation operations equal to the corresponding ficld(s) in the parent in accordance with the end user's actions, writing record . In re lational database terms, this simu­ data out to the data source(s) . By default, an lates a join between the data in the parent and end user may browse through the records dis­ child groups. The application definer, simply by played, modify or delete them, insert new defining a parent/child relationship, achieves the records, commit or roll back the database trans­ foJlowing effects: actions, perform queries tO view a subset of the • When records are read for the child group, an data, and perform these same data operations on implicit restriction is added to read only the subset shown as a result of the query. records for which the corresponding fields Each capability can be removed outright from a match. given fo rm/report or restricted conditionally. For example, to make a formjreport behave like a tra­ • When records are inserted into the child ditional data entry form, the definer may elimi­ group, RALLY automatically fills in the fields nate the capabilities of reading existing records to match those in the parent record. and querying. • When records are deleted in the parent group, Data Groups - Building Blocks of RAlLY can, at the option of the definer, delete RALLYFo rmjreport Structure all records in the child group. preserving the integrity of the database. Traditional forms processing software confines its function to the collection of data, leaving the pro­ This ability tO organize related data is crucial grammer to write the collected data to a fi le or in the development of applications that use database. The RAlLY designers recognized that, relational databases. Data normalization forces once collected, data is most commonly written logically related data to be separated into multi­ out to a data source. Consequently RALLY pro­ ple relations to avoid repetition or excessive vides as a standard option the combined function­ fu nctional dependencies within a single record. ality of collecting data and writing it to a file. To display repeated data in its proper context and RAlLY also allows for "pure forms" that do not to display dependent descriptive data, a single automatically write out data. form or report often relies on data from several The most significant design feature within relations. A r:ypical example is an order entry RALLY form/reports is the data group. This struc­ form. Repeating data for rhe line items in the ture is specifically designed for forms or reports order are stored in a relation separate from the connected to a data source, such as a database or order header data. The order header data contains a file with normalized data. a reference to the customer, but descriptive The data group itself does not contain the defi­ information abour the customer (such as name nition of the file or the relation in a database that and address) must be looked up from a separate stores the data for the group. Rather, the DSD relation. Similarly, line items refer to products, object discussed earlier contains this informa­ but the descriptive information (product name tion. Therefore aspects of the user's interaction and price) are looked up in yet another relation. with the data (formatting of output and restric­ A hierarchy of data groups in a RALLY fo rm; tions on input) are separated from details of how report corresponds directly to the relationships and where data is stored, which may differ in among these relations. character between diffe rent types of data sources. At each level in the hierarchy of groups. the Several data groups can be combined within a definer can allow or restrict insertion, dele­ formjreport to support access to several data tion, and update of records. The definer can also

74 Digital Technical journal No. 6 February I 'J88 Software Productivity Tools

define additional fields such as computed fields The VAX RALLY system offers the definer two and aggregates. RALLY produces an instance of levels of escape: an integrated procedural lan­ such fields for each record in the data group. For guage, AOL, that runs within RALLY; and the abil­ example, an aggregate field in a parent group ity to call traditional programs that run on the will produce a set of subtotals, one for each VAX system, independent of RALLY Both ADL record in that group. procedures and calls to external programs latch Control break reports are also implemented in on to a RALLY application at various "hooks," RALLY with data groups. The fields on whose val­ called action sites. ues the control break is based are placed into a separate data group above the rest of the data. A'> Action Sites with form/reports based on simulated joins, each A number of action sites are available at various level of control break can have aggregates and levels in the application. formatting attributes defined in the data group. A simple example is a computed field that has Nonrepeating fields, such as grand totals, are an action site for a procedure which supplies the owned by a special group called the main group. formula for the computation. In addition , action The main group sits atop the hierarchy, own ing sites can be invoked before and after the user the top-level data groups. This group can also be moves the cursor to each field or changes the used for "pure forms" whose data is not automati­ value of each field; before and after insertions, cally written out to a data source. deletions, and updates in each data group; before A special kind of data group, called a list-of­ and after commits, rollbacks, queries, and invo­ values group, offers a simple, nonprocedural cation of the formjreport as a whole; and at the method fo r ensuring referential integrity. From explicit request of the end user. Ac tion sites that the point of view of the end user, the list of occur before an event generally have the ability values assists in supplying a value for a particular to prevent that event from taking place. For field. The end user uses the RALLY command example, the before-deletion action site, under LIST_OF_VALUES (typically using a function conditions specified by the definer, can forbid key) to move the cursor from the field to the list. the user conditionally from deleting records in The user then moves the cursor to the desired a particular data group. By using external pro­ value and uses the RALLY command SELECT grams or AOL procedures, the definer could VALUE to copy the value to the field. The applica­ call upon a system service to determine a user's tion definer has the option to restrict the user to login-identification, read records from an autho­ selecting a value that appears in the list of values. rization file, andjor call a RALLY menu to allow The implementation of a list of values is simple the user to reconfirm, before proceeding with and consistent with the definition of other the deletion. groups: a OSO describes the data that will appear Because RALLY has direct control over the in the list, and the data group describes the fo r­ "action stack" that governs the flow control, matting of the data on the screen. Because the action sites can be put to very powerful use. At list-of-values data is independent of the other any action site, the definer can call another data on the formjreport, the list-of-values group RALLY action (for example, formjreport, menu, is neither a parent nor a child of the other data ADL procedure, or external program), spawn a groups, but is realized as an independent sibling RALLY task, return to an existing task, "unwind" owned by the main group. the action stack, or invoke a RALLY command (for example, COMMIT) . Escapes to Procedural Programming RALLY tries to anticipate the features that will be ADL Procedures required in applications and to provide the Although external programs can do things ADL definer with the option to include those features. procedures cannot, their effect on the RALLY Fields, data groups, and formjreports as a whole application is limited to their ability to write are replete with options. But this is not enough. their output parameters into RALLY fields or No collection of options will meet the require­ variables. ments of all applications. Therefore, RALLY ADL provides a convenient way to define com­ allows the application definer to escape from the puted fields without having to link to an exter­ nonprocedural confines we provide. nal program. Although there is some overlap

Digital Tecbnicaljourna/ 75 No. 6 Fe bruary 1988 The VAX RA LLY Sy stem - A Relational Fo urth-generation Language

between the abilities of external programs and external program links, and ADL procedures. ADL procedures, certain operations are better Most of these manipulations are done in terms of su ited to ADL procedures. An ADL procedure can data operations: creating, deleting, and modify­ directly read and write fields and variables in the ing information in a "record" that represents application; indicate that a validation has fa iled, each object. preventing a data operation from go ing fort h; The implementation of the definition system invoke RALLY formjreports, menus, error mes­ uses RALLY fo r its own forms, treating the sages, or he lp messages; unwind the RALLY exe­ application definer's objects as data. Thus the cution stack ro a specified point; and manipulate definition system is simply an example of an RALLY tasks. In addition, ADL can read, query, application built with RALLY, although probably and write data through a set of built-in functions much larger than the typical RALLY appli­ that closely parallel the operations permitted in cation. However, the definition system also formjreports. As with formjreport groups, the includes tools, such as editors, and has an specific definition of the location of the data is access method specifica lly designed for effi ­ isolated in the DSD object. ciently storing application objects in fi les. Moreover, given the choice of using either an The code that supports this access method ADL procedure or an external program, a definer can also be called directly by the definition will find the ADL procedure significantly fa ster to system code, or by ADL procedures in the implement. To incorporate an external program definition system application. into a RALLY application, the definer must leave The definition system uses its data about the RALLY, edit the text of the program, compile and application to assist the application definer. link the program, and return to RALLY. To usc an Whenever the application definer has the oppor­ ADL procedure, the definer can invoke the ADL tunity to connect one RALLY object to another ed itor and compiler from within the definition (for example, at action sires) , the definition system menus, and test the results without leav­ syst<.:m displays a list of values showing all ing RALLY the <.:xisting objects of the appropriate type . The syntax of ADL is a good example of how Whenever the application definer attempts RALLY accommodates the definer's need fo r both to delete an object, the definition system simplicity and power. checks for references from other objects; if it A.DLderiv es its syntax from Pascal in order to finds any such references, it warns the definer provide local variables, parameters with call by and displays a report that lists the referenc­ reference, cond itionals, and loops. The syntax is ing objects. Again, standard fo rmjreport features relaxed for cases that do not use all these fe a­ are used in connection with the specialized tures: access method. An interesting aspect of the way objects are • If an A.DL procedure does not use parameters, handled as data is the way object names are the PROCEDURE statement may be omitted . handled. The definer regards the object's name • If an ADL procedure does not use local vari­ as the primary key that uniquely identifies ables, the BEGIN and END enclosing the pro­ the object. However, to encourage the devel­ cedure body may be omitted . opment of mnemonic names, RALLY allows the defi ner to rename objects. Therefore, As a result, an ADL procedure that specifies the RALLY internally identifies objects not by name formula for computed fields - the most com­ but by an internal identifier nor displayed to the mon use of ADL - can be written as a single Pas­ definer. From RALLY's internal point of view, cal statement. For example: the name is just another attri bute of each

FORM_REPDRT . COMPUTED_F I ELD object that can be changed at wi ll. From the

FORM _REPORT - I HPUT _F I ELD_ 1 definer's point of view, renaming an object

• FORM_REPORT • I HPUT_F I ELD_2 ; automatically renames all references to that object Implementation of the The definition system uses a specialized form Definition System of escape to 3GL programs to support nonstan­ The definer of a RALLY application man ipulates dard formjreporrs. Called the 3GL access objects such as formjreports, menus, DSDs, method, this technique allows the definition sys-

76 Digital Technical journal No. 6 February 1988 Software Productivity Tools

tern to present information in tabular form even fields that are subject to cross-field validation when the underlying data is not stored as a imposed either in the database or by the RALLY sequence of records. For example, the definition application. system uses the 3GL access method to display an RdbjVMS record selection expression on a series Transaction Management of tabular formjreports: sorting, restrictions, and in Fo rmjreports projections. Each "3GL DSD" is implemented by RALLY uses RdbjVMS transaction and locking a single routine that can be called with one of mechanisms tO extend TEAMDATA 's straightfor­ several function codes. These fu nctions corre­ ward data table metaphor from a single-user tO a spond tO the data operations that are supported in multiuser environment. Moreover, the designers form;reports and in ADL procedures: get first wanted tO allow many users to access the same record , get next record, insert, delete, update, data concurrently with minimal interference commit, and rollback. Each such routine, in between users, and to do so with reasonably effi­ effect, implements its own access method, sup­ cient performance. plying data and effectuating data operations. The The following brief review of the RdbjVMS definition system can use the "access method" as transaction and locking mechanisms will help in any other data source, for example, to supply the explaining RALLY's implementation of shared­ data for a list of values. write access in form;reports. RdbjVMS provides essentially two types of TheRun-t ime Environment transactions: read-only and readjwrite. A read-only transaction, as its name implies, Mapp ing Us er Actions permits only reading operations. It gives a "snap­ to Database Op erations shot" of the state of the database as it was when The typical end user of a VAX RALLY application the transaction started; later changes by other is not skilled in database concepts. Therefore , to users are not seen in a read-only transaction. A aid this user, database operations should happen read-only transaction does not take out any locks in a natural correspondence to the end user's on the database and is affected only by those rela­ actions. tion-level locks taken by other transactions with The basic concepts that RALLY presents tO an "exclusive " access. end user are very similar tO those presented by A readjwrite transaction must be used to write the VAX TEAMDATA software, a data management to an RdbjVMS database. In addition to various tool specifically designed for exclusive use by degrees of locking of relations, a readjwrite unsophisticated end users. Specifically, the "data transaction locks individual records as it operates table" metaphor by which TEAMDATA operates is on them. If a readjwrite transaction reserves a very similar to the mechanisms that RALLY uses. A relation for shared-write access, many transac­ data table evokes the classical form of a table to tions may read a given record, but only one trans­ represent a relation. Rows in the table represent action may write to a particular record. The records; columns represent fields in the relation. mechanisms Rdb uses to ensure this are called The VAX RALLY run-time environment includes "read locks" and "write locks." As a readjwrite built-in commands with which the end user transaction reads a record , it takes a read lock manipulates records or navigates between fields on that record. For so long as this transaction in a form/report. Function keys have been pre­ holds that lock, no other transaction is allowed defined for the commands most commonly used. to delete or modify that record. However, other To delete a record, the user moves the cursor to readjwrite transactions may read the record, a field in the record and invokes the DELETE taking their own read locks on the same record. RECORD command (typically by pressing the Read-only transactions are unaffected. When a Remove key) . To modify a record, the user moves readjwrite transaction writes to a record, a write the cursor to the desired field in the desired lock is taken on that record. For so long as record and types the new value. However, this transaction holds that lock, no other read/ the update is not communicated to the data write transaction may read that record or write to source until the user moves the cursor off the it. Both read locks and write locks are held until record. Besides minimizing the cost of repeated the readjwrite transaction is terminated by a updates, this allows the user to change several commit or rollback.

Digital Tecbnicaljournal 77 No. 6 February 1988 The VA X RALLY System - A Rel ational Fo urth-generation Language

RALLY's simple, elegant "data table" model, if action . taking a write lock on the record . Note implemented simply by usinga single readjwritc that RALLY can still read and display rhe record RdbjVMS transaction, wou ld thwart the goal of for other users despite the write lock, because noninterfering, simultaneous, multiuser access. the other users are reading from their own read­ Recall that a readjwrite transaction takes a read only transactions. Finally, when the current lock as a side effect of reading each record . The user performs a commit, whether explicitly by mere displaying of a record in the table, even using the COMMIT command, implicitly by without the user attempting to modify it, wou ld means of a positive exit from a formjreport, or immediately interfere with other users' access to as a result of the application definer's design , that record . Al though several user s cou ld each the read-write transaction is committed and all read the record, their read locks would prevent Jocks are released. any user from writing to the record. This would Advantages of Us ing RALLYfo r Its make the shared-write access virtually unusable. Own Us er Interface To decrease contention among users, RALLY The use of RALLY to implement the user interface implements shared-write access using two Rdb/ for the RALLY definition system has resulted in VMS transactions: a read-only transaction for dis­ several advantages. Despite some early bootstrap­ playing records in a data table fa shion, and a ping difficulties, the use of RALLY within itself readjwrite transaction that is used sparingly as needed when the user performs data update has noticeably improved the quality of the product. Any time we change the user interface operations. fo r the definition system, we simultaneously Another difficulty is caused, however, by the exercise the definition system as we ll as the use of the read-only transaction to display exist­ run-time system. The definition system has prof­ ing data for the user's perusal. Because the read­ ited from the ease with which we have been able only transaction supplies a "snapshot" of the data to incorporate into it the same features that as it was when the transaction started, it is possi­ are easy to develop in applications, notably, vali­ ble for the displayed data to lag behind the actual dation, lists of values of valid choices, and flex­ state of the database. Other users may have writ­ ible flow control . As part of the ongoing develop­ ten and committed changes to the record in the ment work on future versions of the VAX RALLY meantime. product, we have been able to experiment To alleviate this problem of stale data display readily with the user interface for the definition while avoiding the overhead of repeatedly read­ system, for example: ing from the database to check for updated records, RALLY employs a compromise. RALLY • We have implemented prototypes of theme nu checks for and reports discrepancies only at the structure of the definition system in which point where a user attempts to modify or delete a menus and forms have been changed to reflect record . After RALLY warns the user that the better the re lationship between the various record has undergone changes since it was read attributes of each object. The time to imple­ from the read-only transaction, RALLY redisplays ment these changes has been negligible, the record with its current data. RALLY does this allowing the development group to spend by reading the record from the readjwrite trans­ appropriate amounts of time evaluating design action. This read operation, called a select for alternatives, rather than on implementation update (SFU), is done as soon as the end user details. changes a single field in a record. This action by • We are studying a change in the way an appli­ the user is the earliest indication RALLY has that a cation definer specifies the location of RALLY user's interest in a record is more than that of pas­ objects. In the current version, an application sive observation. Reading from the readjwrite definer specifies for each object the start row, transaction serves two purposes: it allows RALLY end row, start column, and end column. Under to alert the current user to any interim changes in the proposal being studied, the definer wou ld the record, and by taking a read lock on the specifythe start row and column, and the size record , it prevents other users from making any in rows and columns. This change was easily fu rther changes to the record. When the current prororyped without changing the way RALLY user moves the cursor off the current record, stores the information. We introduced com­ RALLY writes the changes tO the readjwrire trans- puted fields fo r the size information and made

78 Digital Technicaljournal No. 6 February 1'}88 Software Productivity Tools

the end coord inates nondisplayed fields. Only of lists of va lues to translate keywords into code two ADL procedures were needed, despite the numbers. fact that this change affected numerous forms Summary • We are experimenting, by means of the 3GL To make application definers more productive, access method, with formjrepons that display the VAX RALLY system is designed to be at once data about the application in a nonstandard easy to use and powerful. VAX RALLY achieves fashion. For example, we have designed a these goals in several ways . First, it offers a small formjreport that would list the location infor­ set of concepts that address those application fea­ mation about all the fields, groups, and text tures commonly needed by application definers. objects in a formjreport . The 3GL routine to Second, RALLY gives the definer ways to combine su pply the data for this formjreport adds small pieces and ways to move in and out of the spaces to the begi nning of each object's name nonprocedural environment of the definition sys­ so that indentation reflects the depth in the tem. Finally, the designers of this object-based hierarchy of formjreport groups. system delineated objects based on knowledge of • We have prototyped a way to streamline the how the objects are likely to be used. means by which an application definer works The VAX RALLY product's unified form; on related objects in a RALLY application. In reports, comprising combinations of data groups the current version, the application definer connected to data sources, provide application works on one object at a time, returning to the definers the functionality most needed for inter­ menu tree each time to select a diffe rent active database applications. object. The prototype took advantage of The representation of an application by a set of RALLY's "local function" feature, by which an connected objects allows programming to be application definer can give the user the abil­ treated as a data processing application that ity to ca ll a RALLY action at will by pressing a manipulates those objects. In particular, this rep­ key. This fe ature wou ld allow an application resentation has allowed RALLY to implement the definer to press a key to edit an object named user interface for the application definition sys­ on the current screen. For example, if an tem as a RALLY application. application definer were editing a menu and The use of RALLY formjreports as the basis for were to move the cursor to the name of the the RALLY definition system has resulted in formjreport that is called as a choice from that several significant advantages, both in anticipat­ menu, RALLY wou ld suspend its editing of the ing the needs of users and in increasing our own menu and allow editing of the formjreport. productivity and flexibility in developing VAX RALLY. The speed with which such changes can be made has allowed us to compress several cycles General References of design, implementation, testing, and reaction E. Horowitz, A. Kemper, and B. Narasimhan, "A into the time ordinarily taken to complete a sin­ Survey of Application Generators," IEEE Soft ­ gle cycle. The ability to respond substantively to wa re Qanuary 1985) : 40-53. user feedback is a major contribution to our efforts to improve VAX RALLY 's user interface. j. Martin, Fo urth-Generation Languages (Engle­ Also , our experience with the definition sys­ wood Cliffs : Prentice-Hall, 1985). tem, one of the largest applications ever built C. Date, An Introduction to Database Systems , using RALLY, has given us valuable insight in Third Ed ition (Reading: Ad dison-Wesley, 1981). evaluating RALLY and proposing new features. VA X RdbjVMS Guide to Data Ma mpulation Lists of va lues is an example of a fe ature infl u­ (Maynard: Digital Equipment Corporation, enced by the use of RALLY by the definition sys­ Order No . AA-N036B-TE, 1985) . tem. Several features were added to lists of values fo r the benefit of the definition system to make VA X RALLY Dialog Us er's Guide (Maynard: Dig­ the fe ature more useful for applications gener­ ital Equipment Corporation, Order No . AA­ ally. These fe arures include the ability to validate GX89A-TE, 1986) . the user's typed input against a list of values, the VA X RA LLY ADL Us er's Guide (Maynard : Digital ability for variables in the application to affect Equipment Corporation, Order No . AA-GX90A­ the set of records in the list of values, and the use TE, 1986).

Digital Te chnical jo urnal 79 No. 6 February 1988 Linda E. Benson Michael Gianatassio,jr. Karen L. McKeen

V1X and VA L U- Software Productivity To ols fo r Distributed Applications Development

Digital's VA X V1Xproduct is a distributed infonnation-retrieval tool th at operates in conjunction with another tool, the VA X V1XAp plication Link Utilities, or VA LU. These products enhance software productivity by providing components that work together to aUow the development and integration of applications in distributed, heterogeneous environments. V1X and VA LU provide the means fo r creating information services, providing network access to either centralized or distributed infonna­ tion, and building external applications through basic tools and program­ ming interfaces. The development of distributed applications with V1X and VA LU requires little or no knowledge of the underlying network.

The designs of VTX and VALU center on a dis­ • Even if information could be reached, it was tributed open architecture using the client/ not we ll organized for ease of access. server model. This open architecture allows VfX and VALU applications tO be integrated with oth­ • At the decentralized locations, information ers available through Digital's networking envi­ was usually maintained in diffe rent ways by ron ment. The architecture enables these two those people most fa miliar with it. products to provide a simplified development environment for applications. Within this envi­ These problems describe the information situ­ ronment, a developer can create distribured arion in most corporate business environments. applications that allow geographically dispersed Therefore, the challenge for the VTX design­ users to access information stared in geographi­ ers was ro determine how a corporation handles caJly dispersed locations connected by a com­ information flowing between different locations. purer network. This flexibility allows the base Note that information in this case could be any­ services of a distributed information retrieval thing from policies and procedures manuals, to system tO be extended into a more robust dis­ job postings and travel schedules, to CAD/CAM tributed system. In such a system, a deve loper drawings and technical documentation. A major can integrate applications with other software goal of the designers was that a minimum of spe­ products or external computer systems. cial learning should be required by people accessing the information; browsing through it The V1XPro ject Goals should be as simple as using a telephone. This goal caused the VTX architects tO exam­ The designers of the VTX product had to address a unique set of problems re lated to information ine various systems with these characteris­ tics, including public videotex systems, that access through a computer network. The central addressed the problems listed earlier. A main problem was how to efficiently distribute infor­ feature of public videotex was that basic naviga­ mation on -line tO a large group of people. The tion through the on-line information system was chief aspects of this problem were the fol lowing: simple for users. They could easily locate infor­ • There were neither means to access informa­ mation and then rely on the system to quickly tion stored in dispersed locations nor easy and easily access that information. Public video­ ways tO alert potential users about it. tex systems were also distributed systems: users

80 Digital Te chnicaljo urnal No. G Fe bruary 1988 Software Productivity Tools

were geographically dispersed, and the informa­ success as a prototype, the development of the tion accessed was located in multiple informa­ product began, following the goals described tion "stores" in a computer network. above. Somewhat later in the development Since public videotex was a well-accepted sys­ phase, the team added goals aimed at making the tem that was also easy to use, the architects product extensible, thus taking advantage of the chose it as the basic model on which to build open architecture. Extensible in this case simply the VTX product. meant the ability tO enlarge the tool-kit nature of Figure 1 presents the architects' view of a sim­ the product so that an application developer plified model of information flow within a cor­ could expand the system by interfacing with poration. This model would be refined as project other products and environments. As a result, goals were clarified. designers identified some growth areas for these products that took advantage of their flexible open architecture. These additional goals led to the concept for the VALU product. Although VTX would provide the base services for a distributed infor­ mation retrieval system, those services had to be expanded tO interact with other applica­ tions. VALU was conceived as a tool kit that would allow application developers to enhance applications built with the basic system pro­ vided by VTX.

• Enhanced tools and environments for the information providers

• Tools for acquiring and incorporating infor­ mation into the VTX-based system

• Integration with non-VTX applications Figure 1 Information Flow Model through interfaces that require no knowledge of the underlying network by the application developer Although ease-of-use was a primary goal for an information retrieval system, other major goals Building the Base V1X Sy stem included the following: This section discusses the characteristics of a • Information access must be fast. distributed architecture, how it facilitates appli­ cation development and integration, and the • The product must accommodate access from a methods for building upon it. variety of desktop systems and terminals from different manufacturers. Characteristics of a Distributed

• The product should be protOcol neutral so Architecture that information, regardless of its format or Distributed connotes dispersion, spreading out presentation-level content, could be stored in and placing things in different places. A dis­ a single data store. tributed application comprises two or more application components, separated from each • The product must suppon a distributed envi­ other, but working together tO form the applica­ ronment in which the information, as we ll as tion. An application component is a self-con­ its users and information providers, might all tained program that executes independently of be geographically dispersed. other application components. Guided by these goals, the designers built a Application components may reside on differ­ prototype that was then tested by users within ent CPUs or on the same CPU. In either case, Digital to determine its ease-of-use factors, per­ these components need some means to commu­ formance, and general acceptability. Based on its nicate with each other. The communication

Digital Tecbnical]ournal 81 No. 6 February 1988 VTX and VA LU To ols fo r Distributed Applications Development means chosen may vary depending on the local­ SERVER ity of other application components. For exam­ CLIENT SERVER COMPONENT - COMPONENT r-- COMPONENT ple, components residing on the same CPU may (BROKER) communicate through shared memory, whereas those on different CPUs may communicate over Figure 2 ClientjServerjBroker Relationships a network. The design of an application com­ ponent is independent of the locality of other components. At run time, the software support­ Application protocols define how functions ing the system can sense any difference in locali­ that are specific to the application are distri­ ties and choose the appropriate communication buted across its components and the rules for means. component interaction. Although the applica­ Since the locations of application components tion protocol is independent of the communica­ are transparent to the design of the application, tion means, the protocol may require certain these components may be distributed across characteristics, for example, full-duplex com­ heterogeneous, or mixed-vendor, environments. munications. For example, DECnet software extends its con­ Application protocols must be invisible to the nectivity to heterogeneous environments through application developer. In the VTX and VA LU SNA and X.25 networks. Therefore, the compo­ products, callable application protocol libraries nents of a VTXjVALU system may also be dis­ are implemented to increase the productivity of tributed across these environments. the application developer by

ClientjServer Relationship • Creating a single library that supports the The components of a distributed application application protocol and is shared by all have a clientjserver relationship. As consumers application components requiring support for of resources, clients initiate requests to servers; the protocol as providers of resources, servers respond to • Having the application developer learn a sim­ requests from clients. ple, higher level call interface rather than all Clients and servers generally interact accord­ the details of the application protocol ing to a requestjresponse protocol. Since requests and responses may be formulated over • Defining a clear interface for integrating multiple messages, a "token" is used to regulate applications into a distributed environment whose turn it is to communicate. The applica­ tion component that possesses the token has the • Insulating the application from changes in the right to communicate. The completion of com­ application protocol munication is signaled by the passing of the • Resolving incompatibilities between applica­ token, either explicitly by flags within the mes­ tion protocol versions in application compo­ sage or implicitly by the message type. nents Servers may communicate with other servers. The server that initiates the communication then • Insulating the application developer from the becomes a client to the other server. If a server communication means used between applica­ communicates with another server on behalf of tion components its client, that server is called a broker. Figure 2 illustrates this broker relationship. Brokering • Facilitating the development of applications facilitates application integration and allows that can be accessed by simultaneous users clients transparent access to any application available throughout the network. How the VA X Ho w the Architecture Achieves the VALU product integrates applications through Project Goals brokering services will be discussed later. Fast information access in a distributed environ­ Regardless of which communication means is ment is achieved by making VTX available over used for application components, a set of ru les networks with the DECnet architecture. This in the form of protocol messages must be architecture extends connectivity to multiven­ defined to specify how functions are distributed. dor environments, thus achieving the goal of These rules are called the application protocol. accessing dispersed information. For example,

82 Digital Tecbnicaljournal No. 6 February 1988 Software Productivity Tools

the DECnetjSNA Gateway and the packet-switch­ over the network. Their fou ndation is based on ing interface products (X.25) provide access to the distributed architecture illustrated in Fig­ a great variety of non-Digital environments. ure 3. The architecture allows transparent access to The store of information referred to earlier is heterogeneous environments through the broker­ contained in the vrx info rmation base, a hierar­ ing capability of servers. Users can navigate chical system of pages called an infobase. The transparently to other vrx servers or to applica­ infobase contains presentation information that tions that have been integrated intO the vrx users can navigate through using menus and key­ environment. Applications integrated into that words. On-line updates are allowed because the environment can be developed independently of infobase is shared among the infobase, update, whatever input devices the users have. There­ and VISTA servers. fore, application deve lopers can make their The terminal control program (TCP) is applications available immediately to any user responsible for presentation management and on the network having the standard vrx client. parsing users' requests according to the specific No additional software needs to be installed on input devices being used. The TCP maps those the client systems. requests w specific vrx function requests, The features mentioned above provide fu ll which are then sent to the information server. support fo r processing and storing data in a truly The TCP and the information server communi­ distributed, heterogeneous fashion. By allowing cate through the DECnet software using an transparent navigation to servers and applica­ application protocol called the videotex access tions, the architecture can retrieve data stored in protocol (YAP). any format from any point throughout the net­ Infobase servers communicate with other work. infobase servers or with applications on behalf of the TCPs. This communication is transparent Components of the V1XjVALU to the TCPs, thereby providing transparent Product Set access to them. All communication between The vrx and VALU product set includes a col­ infobase servers and applications is through the lection of application components interacting YAP application protocol .

INFORMATION LU6.2 I TCP APPLICATION SERVER l ..., .. 1 r � Ir l VAP INFORMATION VAP I VAS DECnet I TCP I APPLICATION SERVER APPLICATION f { l ---·,.. I JAMS X.2S I TCP ELK APPLICATIO APPLICATION / ...... _ � , � i l ••• • I ---· � !'--.. _..,. INFOBASE '-- _..,. 1

RMS RMS VUP I UPDATE VISTA 1 VIP I CLIENT I CLIENT 1 VISTA VUP UPDATE f-- UPDATE I SERVER SERVER l CLIENTl VISTA 1 VIP CLIENT 1 VUP I RUSL I CLIENT I I KEY

RMS - RECORD MANAGEMENT SERVICES

Figure 3 V7X and VA LU Product Architecture

Digital Tecbnicaljournal 83 No. 6 February 1988 V7X and VA LU To ols fo r Distributed Applications Development

The external link kit (ELK) interfa ce is an essary capabilities that allow application devel­ application protocol library used by pro­ opers to easily build interfaces to other applica­ grammers to integrate applications into the VTX tions. environment through the VAP protOcol. The VTX application service (VAS) is a dis­ VTX as an Application tributed application integration tool . VAS allows Development To ol access to applications on other computer sys­ The VAX VTX base product provides the tools tems through both DECnet and non-Digital net­ for quickly and easily building a distributed works. The productivity gains of developing and information application. The open architecture integrating applications into the distributed allows an application developer to easily extend environment of VTX using VAS is discussed later. the application by adding new applications and The VTX infobase structure tool and assistor support for heterogeneous environments as the (VISTA) is a distributed application development requirements change. These extensions are dis­ tool that displays a graphical view of an infobase cussed later in the section Application Integra­ to assist a user in creating and managing an tion Using VAS. infobase. VISTA clients are information providers The basic VTX components allow an informa­ who interact with a VISTA server tO perform tion provider tO create an infobase. The infobase those tasks. Al l communication between VISTA is the application; the information provider is clients and servers is by means of the DECnet the application developer. Policies and proce­ network using an application protocol called the dures manuals, sales and competitive informa­ videotex information provider (VIP) . Using tion articles, reference manuals, jobs books, VISTA yields productivity gains that are dis­ training schedules and course descriptions, cussed in the next section. CAD/CAM drawings, and newswire stories are Update clients are information providers who examples of information that can be organized, interact with an update server tO create and maintained, and delivered as VTX information maintain an infobase. The update server uses a applications. command-oriented interface. Al l communication With the simple information-based applica­ between update clients and servers is through tion, VTX relieves an information provider from the DECnet network by means of an application having to know specific details of the underlying protocol called the videotex update protocol communications and the terminals. Thus, the (VUP) . information providers can direct their attention Application developers can also use RUSL to the content of the information and can more (remote update server link) , an application pro­ easily design the structure by which an info rma­ tocol library used to create infobase manage­ tion user accesses the infobase . ment tools as well as to allow the updating or populating of an infobase from a program. The VIS TA RUSL application protocol library was bu ilt to VISTA is a tool to increase the productivity of the support the VUP protocol . The VTX update information provider when creating a dis­ client was built using this protOcol library. tributed information system using VTX. VISTA helps a naive information provider to become To ols Provided fo r Application productive very quickly and allows an experi­ Development enced information provider to remain produc­ This section discusses two tOols provided by the tive. VISTA provides a simple graphical interface VTX and VALU product set to enhance the pro­ for the naive user, yet also has a command-line ductivity of application developers building dis­ interface for the more experienced user. The tributed information systems. VTX information productivity of creating applications increases systems can be grouped into two classes: those because VISTA is easy to use. simply providing information tO users, and those VISTA uses the clientjserver model to allow interfacing with other applications to enhance one VISTA server tO maintain the actual VTX and expand on the information itself. The first infobase files. One or more information pro­ tOol, VISTA, addresses the needs of application viders can access that VISTA server through developers building the first type of information the VISTA user-interface program (VISTA client) . system. The second tool, VAS, provides the nee- Building on the fou ndation of a distributed

84 Digital TechnicaiJournal No. 6 February 1988 Software Productivity Tools

architecture allows information providers to be infobase. In an application development envi­ dispersed geographically. VISTA can coordinate ronment, the VISTA options allow an information multiple information providers working on an provider tO easily specify all the necessary infor­ infobase by al lowing them to reserve portions of mation for each page. The information provider it for updating. can invoke an ed itor to supply the text for any VISTA uses a simple graphical interface that page without leaving the environment. VISTA allows an information provider to quickly design builds the text of menus according to a default the layout of a VfX infobase. The picture dis­ style and allows that style tobe modified for any played represents the hierarchical nature of the menu page . Its simple forms interface allows the infobase, much like an information provider information provider to specify additional infor­ wou ld imagine the infobase menu structure to mation for any page. This page information is be . VISTA improves and enhances the infobase grouped into categories of similar items, each development process by allowing the info rma­ with its own form. Figure 5 shows a VISTA form. tion provider to design the menu structure right Once the information provider has created the on the terminal instead of constructing it first on menu structure, VISTA handles the process of paper. Figure 4 shows a sample of a VISTA building that infobase from the picture . Each screen with a menu structure. page in the menu structure is converted from An information provider using VISTA builds a graphical format to infobase format through the VTX infobase by selecting options from the strip VISTA server. VISTA handles the underlying com­ menu appearing at the bottom of the screen. The plexities of page generation, such as page num­ large work area above the options menu displays bering and the association between a menu page the current state of the infobase as the informa­ and its choice pages. tion provider continues to work. The single box Once the info rmation provider has initially at the top of the work area depicts a current developed an information application using menu. Any number of boxes drawn just below VISTA , the tool can continue to be used to the current menu show the menu choices from enhance, extend, and maintain the application . the main menu. Any of those menu choices may themselves be menus. The menu structure of the Extending the VTX Application VTX infobase can be mod ified through simple Using the VA X VAI. .U product, the information add, delete, and cut-and-paste operations. provider can extend the basic application into a Each box at the top and across the center of more interactive one by using the VAI..U tools for the screen represents a particular page in the two-way access to the infobase. Not only can the

VAX VTX VI STA V3 . 0

ENGINEERING JUL . 1987 SCHEDUL ES MANUFACTUR ING AUG . 1987 COURSE DESC . MARKET ING SEP . 1987 SEMINARS SALES VIDEOTAPES FINANCE

ADD CHO ICE SELECT DEL CHD 1 cE G u I T__ ___, HELP I __ I I I I I ._I _ I Use the arrow keys to selec t an option , then press RETURN .

Figure 4 Sample VIS TA Menu Structure

Digital Technical jo urnal 8 5 No. o February I Y88 VTX and VA LU To ols fo r Distrihuted Applications Development

V3 .0

General Page Control Information

General Information For :

Page number : __ Page type: MEN_U _ Closed user group: 25 Coding: ASC I I --- Charge value : 0 Screen width : � Creation da te: Expiration da te: 01-JAN-1988 Clear : y Dire ct: y Log : E Save : y More : E User da ta: More Be low Press FIND or PF 1-L to select a value from the Irs! of supplied va lues

Figure 5 Sample VIS TA Fo rm application distribute information to users, it The VAS language consists of eight verbs used can also collect information from them, process [0 it, and return the results. Using VALU allows an application developer ro define the flow of con­ • Display information from the infobase of the trol for infobase access by a user. VA LU also server (optionally merging data from VAS) , allows simple connections to external applica­ collect user responses, and make flow-control tions that can provide information to the system. decisions based on user requests Some examples of external programs are on-line • Declare local and global variables ordering and registration systems. The next section describes the VAS component • Manipulate the contents of variables of VALU. VAS is a powerfu l application develop­ ment and integration tool that provides the fu nc­ • Pass variables and srare information to exter­ tionality ro connect the VTX system with appli­ nal applications (or to local user-written rou­ cations on DECnet, SNA (LU6.2), and X.2'5 tines that have been dynamically loaded in networks. VAS was bu ilt on ELK, the application the VAS image) and receive responses protocol library that is provided with VALU . • Make flow-control decisions based on Application Integration Us ing VA S responses from user input, external applica­ VAS is a flow-comrol and integration layer be· tions, and user-written routines tween the VTXjVALU environment and external applications. Figure 6 illustrates how applica­ • Log the contents of variables tions are integrated using VA S. VAS simplifies application development and Interaction with External Applications int egration by providing a fourth-generation lan­ VAS interacts with external applications over guage for using YAP . The VAS language is spe· communication channels using its own request/ cializcd ro provide the fu nctions of YAP and co response protocol . A request contains current faci.l itatc the integration of external applica­ stare information about a user, for example, tions. VAS applications are organized into scripts which transaction the request was made from, called transaction definitions. Application flow the contents of variables, what operation is control can occur by transaction definitions being requested, and time-stamp information. A transferring processing ro other transaction defi­ response contains updated variables from the nitions. processing of the application.

86 Digital Technicaljo urnal No. 6 Februar)' 1')88 Software Productivity Tools

.._IBM SNA- APPLICATION COMPONENT

TCP VAP VTX VAP SERVER ELK VAS -������ -APPLICATION COMPONENT

- X.25-- APPLICATION COMPONENT

Figure 6 Application Integration through VA S

VAS transaction definitions specify the names tem need to conform on ly ro rhe request/ of the communication channels over which response protocol . requests to applications are made. A single VAS transaction may interact with multiple applica­ VA S and the X. 25 Environment tions over various communication channels. The VAS can communicate wirh any packet-mode VAS operator dynamically associates communica­ dara terminal equipment (DTE) that is connec­ tion channel names to specific communication ted to a packer-switching data network (PSDN) types and specific applications. VAS has built-in by using the VAX PS I product. The PSDN pro­ su pport for communicating with applications vides task-to-task communication between any over the DECnet, SNA (LU6.2), and X.2'5 net­ two computers connected to an X. 2 5 network. works . The same request/response protocol is Therefore , the environment is heterogeneous used over all communication types. Transaction in narure. The VAS developer needs no knowl­ definitions arc written ind<:: pendently of the edge of the PSDN, the VAX PSI product, or the communication channels used ; the application remote DTE being accessed . Applications writ­ developer req uires no knowledge about the net­ ten on the remote DTE need to conform only to work. the rcquestjresponse protocol. Communication channels are shared by users and may have multiple outstand ing requests; VA S and DEC net Applica tions however, each user can have only one outstand­ VAS can communicate with other DECoct appli­ ing request . VAS manages the sending and cations using either transparent or nontranspar­ receiving of all requests. These acttVIttes ent task-tO-task communication . Once again, the include suspending the execution of a transac­ VAS application developer needs no knowledge tion definition, timing requests, receiving a of the DECnet software. External applications request and identifying which user's transaction written on the remote system need to conform definition ro resume, and extracting the contents only to the requestjresponse protocol. of a request into local variables. Using the concepts of the request/response Ha ndling Simultaneous Us ers protocol over communication channe ls, a VAS The VAS developer writes transaction definitions appl ication deve loper can build an application as if they were synchronous and for a single user. that uses a consistent programming interface to After VAS compiles and loads the transactions, communicate with a variety of heterogeneous they become available for simultaneous users. environments. Let us examine some of the VAS interleaves user activity by suspending users details that VAS handles for the application whose transactions arc currently performing deve loper. asynchronous activities, for example, waiting fo r the TCP to respond to rhe last page displayed, or VA S and SNA (L U6. 2) VAS uses the DECnetjSNA Gateway and the VMS VAX IBM APPCjLU6.2 products to communicate wirh rhe SNA environmenr, as illustrated in Figure 7. CICS TRANSACTION The application developer using VAS requires CICS TRANSACTION no knowledge of the DECnerjSNA Gateway and VMS APPCjLU6.2 products, or of the II3M envi­ ronment. The CICS transactions on rhe 113M sys- Figure 7 VA S in tegration Us ing SNA (L U6. 2)

Digital Te chnical journal H7 No. o Fe/Jruarv I 988 VTX and VA LU To ols fo r Distributed Applications Development

wamng for an application to respond to a start using the updated transaction definitions. request. In other words, while one user is wait­ When there arc no active users on the older ing for some sort of IjO to complete, VAS pro­ transaction definitions, they are automatically cesses another user's request. unloaded . Since VAS automatically handles simultaneous users, simultaneous requests may be generated Sample VA S Application to the same application. However, developing This section contains a sample VA.'I application applications tO service simultaneous users can that displays a form page to the user and then be complex. Therefore, VAS has built-in fea­ sends a data block (using the REQUEST step) to tures that allow single-user, synchronous appli­ pass two fields with initial values (specified by cations to service simultaneous users. In that the DFLD variables) to a CICS transaction for way the burden of developing a simultaneous­ processing. Upon return ing from the CICS trans­ user application is removed from the appli­ action, the VAS application directs that page 102 cation developer. These activities are all accom­ be displayed from the VTX infobase to the user. plished through the subchannel feature of the This sample VAS application could be parr of an communication channel. The VAS operator can interactive banking application that gathers data start multiple copies of the same application on from the user and then val idates it (for example, a single communication channel. When requests a user's bank account number and access code) are made over the channel, VAS allocates a copy before allowing access to the system. of the application that is not currently being used . If all subchannels are busy, VAS holds the TRANSACT ION first_trans/ENTRY request until a subchannel becomes available. ='accesscode'

The subchannel feature creates a pool of identi­ BEG IN cal applications which can be distributed across DISPLAY '10 25' all communication types. From the VAS transac­ REQUEST checkcode sna_chan1 tion definition , this pool of applications acts like BEG IN a single communication channel. Figure 8 illus­ RFLD- 1 DFLD 1/LENGTH=6 trates these subchannel capabilities. RFLD-2 DFLD2/LENGTH= 15

END

Creating High Availability Computing EXIT/PAGE•'10 2'

Environments END VAS fu nctions are managed without interrupting any active users. Such functions include starting The channel to the SNA gateway has been and stopping communication channels, opening established with a command of the fo llowing and closing log fi les. loading new transaction format: definitions, modifying the contents of global variables, and changing the association between VAS>START CHANNEL sna_chan1/SNA• channel names and communication types. CGWY =sna_gwy , The VAS application developer can make ACC=cics_access, updates to transaction definitions and load them TPN=cics_trans 1) dynamically. Users accessing the transactions before they were modified will continue to usc G\X'Y is the name of a DECnctjSNA Gateway, the older version of the transaction definitions. ACC is the access name on rhar gateway that New users who connect ro VAS will immediately allows access to CICS, and TPN is the name of the IBM host transaction tO be invoked .

r---- APPLICATION A Sample Distributed Application 1----- APPLICATION A Us ing V1Xj VALU CHANNEL A VAS t--=----'-'-=-:__-+---- APPLICATION A The fo llowing example ill.ustraces an applica­ 1----- APPLICATION A tion that was built using the VAX VTX and VAX '----- APPLICATION A VAL U products to handle document publishing with an integrated free-cexc search product. This Fig ure 8 VA S Suhchannels example highlights some capabilities of these

88 Digital Te chnical journal No. 6 Februarv I ')88 Software Productivity Tools

distributed products and illustrates the benefits A fu rther extension to this application gives of building applications utilizing the distributed the user the option of mai ling the current article base system. being viewed. Figure 9 gives a view of the com­ The primary functions of this appl ication ponents of this distributed document publishing provide system. This example demonstrates some of the most • A periodical called Sales Update on-line for important productivity benefits to an applica­ corporate-wide distribution tion developer. • The capability ro supply users with a free-text • A single application program utilizing VTX search feature using a third-parry application and VAL U can be accessed by a network of called BAS IS users without regard ro the asynchronous

• The capability to supply users with a hard­ environment and the need ro support a large copy fo rmatted ve rsion of any article in the number of users. periodical through an integrated mail inter­ • An information retrieval-only application can face easily be extended ro interface with external In this application the screens of information _�:-roducts without changing the user interface are formatted by a preprocessing application or disrupting the information retrieval "ser­ that creates the VTX infobase. This same appli­ vice." cation also provides data to the BAS IS database • The applications provide VTX-like access that and supplies files formatted for hardcopy out­ is consistent with the information retrieval put. access to provide extended capabilities to the The information user is presented with a users. This extension allows the application menu of categories of Sales Update articles, to free the users of the system from having to along with an option ro search through them for learn a new interface to the newly integrated a part icu lar text string. If the user selects a product. In fact, the integration to another search option, the VALU application will pass the product may be virtually transparent to users. string ro the BASIS application, which returns ro • The application can support both softcopy VAL U one or more article IDs that match the (on-line) and hardcopy distribution from a search criteria. VALU then creates a menu single source fi le. dynamically with associated title strings to help identify the articles located. Subsequently, the Summary user chooses a menu selection and the resultant Working together, the VA X VTX and VAX VALU article is displayed. products provide a rich set of software produc-

SEARCH SEARCH CRITERIA CRITERIA VTX VALU BASIS - RESULTANT RETURNED MENU ARTICLE - OPTIONS ID NUMBERS

..______, ,__ TEXT FILES FOR HARDCOPY DISTRIBUTION VTX BASIS INFOBASE DATA BASE

FORMATTED SCREENS TEXT PAGE NUMBERS OF ARTICLE INFORMATION SUBJECTS ARTICLES

Figure 9 Distributed Document Publishing Application

Digital Technical jo urnal 89 No. 6 Februarv I YBB VTX and VA LU To ols fo r Distributed Applications Development

tivity tools and programming inrerfaces. These VA X VA LU Documentation Kit (Digital Equip­ products enable an application developer ro con­ ment Corporation , Order No QL035-GZ-V2.0, struct bmh centralized and distributed appli­ 1986). cations used in both homogeneous and heteroge­ J Morency, D. Porter, R. Pitkin, D. Oran, "The neous environments. Because of the range and DECnetjSNA Gateway Product - A Case Study in flexibility of these tools, the resu lting systems Cross Vendor Networking," Digital Te chnical can differ sign ificantly according tO the function jo urnal (September 1986): 35-53. sets utilized and configurations selected . For example, with these products, a simple informa­ P Beck, ). Krycka, "The DECnet-VAX Product ­ tion retrieval or transaction-based system can first An Integrated Approach ro Networking," Digital be built and then evolve inco a more complex sys­ Te chnicaljournal (September 1986): 88-99. tem based on the concepts of the VTXfVAl.U open distribmed architecture. The ability to expand a system is essential; with the capabilities provided through VTX and VAl.U, this evol utionary system model is easily achieved. Genera/ References

VA X VTX Documentation Kit (Maynard: Digital Equipment Corporation, Order No . QL031 -GZ­ V3.0, 1987).

90 Digital Te chnical jo urnal No. 6 Februarv I ')88 RonaldF. Brender Bevin R. Brett CharlesZ. Mitchell

Pragmatics in the Development of VAX Ada

The software toolsand techniques (p ragmatics) used daily bythe VAX Ada developers significantly contributed to increases in product performance and developer productivity. Approximately 500, 000 lines of code were writtenfo r this project. Of particular interest in this project's development is the automation of the coding process, instrumentation of the compiler, built-in consistency checking within the compiler (selfchecking), and the use of selfdescribing data structures. This paper gives examples of how these toolsand techniques were used in the development of the compiler. However, these toolsand techniques can be applied to a widerange of soft· ware development effo rts.

Software engineering literature to a great degree niques that would minimize the time spent on focuses on design and implementation method­ routine or duplicate activities and wou ld maxi­ ologies, as well as on tools to go with them. Little mize the amount of time available for interesting is said , however, about day-to-day tools and tech­ technical problems. Thus, we balanced the value niques that can also significantly impact the pro­ of each tool and automation technique against ductivity and effectiveness of a development the time it would take to build the tool or team. develop the technique and use it. We wanted ro The development of VAX Ada involved writing spend most of our time developing VAX Ada. approximately 500,000 lines of BLISS source These are some examples of activities we auto­ code. This paper discusses some of the tools mated: and techniques that have been important over • Production of error-message information held the course of that development. The tools and in common between the compiler and the user techniques fa ll roughly into the following documentation categories: • The task of creating and entering tests into the • Automation VAXAd a test system • Instrumentation • Compiler bui lds and check-in procedures • Self-checking • The process of managing multiple versions of • Self-description the compiler

We would like to suggest that many of these • Some debugging tasks tools and techniques could be useful ro any soft­ • Key algorithms within the compiler ware developer and could be applied to any pro­ ject of significant size . The first two examples are described in more detail in the following sections. The last example Automation is described at the end of this paper in the section During the development of VAX Ada, we wrote Self-description. support code ro automate various aspects of the coding process and tO help with day-to-day devel­ Production of Error-Message opment activities. Our interest was never in rools Information or automation techniques for their own sake. VAX Ada has close to I ,000 distinct error mes­ Instead , we were interested in tools and tech- sages. One of the VAX Ad a user manuals lists all

Digital Tecbnicaljournal 91 No. 6 Fe bruary 1988 Pragmatics in the Development of VA X Ada

the error messages in an appendix. The usual make sure that these complex, and important, method for producing error messages and docu­ tasks would be done routinely and accurately. menting them is to create two source files: one For example, we developed source file to be maintained by the compiler • Command procedures tO help create a test that developers and processed by the VMS message followed project conventions compiler, and one source fi le to be maintained by the writer and processed by the documentation • Command procedures to automatically insert a processor. rest in the test system This method quickly became difficult for us to • The ability to mark comments within a test as manage. Messages were continually added as the keywords, and then automatically read the key­ VAX Ad a compiler was being developed - even words comments and enter them as attributes during the final stages of the development cycle. of the test in the rest system Because documentation is written and reviewed concurrently with compiler development, the • Support for all major classes of tests (com­ writer and editor of the user manual needed to piler, project library manager, debugging sup­ have a matching set of messages in the documen­ port, etc.), so that no major areas of testing tation for draft reviews and fi nal production . The were neglected writer and editor also needed tO be able to sug­ gest wording modifications as messages were Instrumentation written, including wording modifications to the The richness of the Ada language presents a num­ messages added late in the development cycle. ber of challenges tO compiler writers. One of To keep the two sets of sources synchronized, these challenges is tO achieve good compiler per­ we chose to automate. We wrote a processor that formance. We fo und that instrumenting the com­ accepts a superset of the language accepted by piler tO measure its use of resources was an the VMS message compiler as input. The addi­ important factOr in developing a high-perfor­ tional language constructs allowed us to write mance product. one source file containing all the messages and We used general-purpose tools such as the VMS any descriptive text appropriate for the docu­ Debugger and the VAX Performance and Cover­ mentation. Both the developers and the writer age Analyzer extensively during the development were allowed access to the file. We then used of VAX Ada. These tools were also very useful in the processor to produce two output files: one improving performance. However, our special­ file containing all the messages and the coding ized instrumentation he! ped us analyze the required for the message processor; and one file behavior of the compiler at a level relevant to the containing all the messages, any descriptive text, development strategies we were using; we could and the formatting constructs needed for the doc­ then better understand how these strategies were umentation processor. working. Many performance problems wou ld Our processor saved us from having tO review never have been identified had it not been for our two different sources at already busy times in the specialized instrumentation. As a result of our development cycle. This approach also allowed positive experiences, virtually every component the user manual appendix to be generated imme­ in the compiler that manages a resource is instru­ diately for each new version of the compiler. mented to gather detailed performance statistics. The fo llowing sections show how instrumenta­ Creating and Entering Te st-System tion is used tO Te sts • Provide data for design decisions affecting An important task tOo often neglected is the compiler performance preservation of tests written during development for later use by developers and maintainers. We • Regulate the behavior of the compiler observed during the development of VAX Ad a that • Provide information on the behavior of the the number of tests added to our test system compiler during maintenance and debugging varied inversely with the difficulty of adding them. Our objective in automating the tasks asso­ Although some of this instrumentation code is ciated with adding tests to our test system was to present in the production version of the compiler

92 Digital Te chntcaljournal No. 6 Febmary 1988 Software Productivity Tools

that is shipped to customers, most of it is condi­ lation unit first. The entire tree representation tionally compiled into only the debugging ver­ and code-generation information for a subpro­ sion of the compiler. gram is no longer needed once the code has been generated, and can be freed before the code is Instrumentation as a Performance generated for the next subprogram. Thus, the Design Aid memory is available for reuse by the next subpro­ As the Ada language itself was being developed , gram. This approach reduced the amount of we began to research the novel aspects of imple­ memory required to compile a typical Ada unit menting an Ada compiler and developed a bread­ by a factor of rwo or more. This improvement, board compiler as a vehicle for our research. combined with the tree modifications mentioned Because the breadboard compiler had been previously, made it possible for us tO meet our instrumented extensively, we used it to collect compiler performance goals with respect to vir­ data to guide the design of the eventual product. tual memory usage. A number of design decisions were made as a result of the data collected during the research period. The role of instrumentation with respect Instrumentation to Regulate Compiler to the compiler's use of virtual memory is exam­ Behavior ined in particular in this section. We also used instrumentation data gathered dur­ One major resource problem in the breadboard ing a compilation to actually modify the overall compiler was the vast amount of virtual memory flow of the compiler, and thus improve the com­ required to compile some representative Ada piler's performance. In particular, the compiler programs. The amount was often an order of mag­ uses instrumentation data to modify its behavior nitude more than was acceptable to meet our according to the availabiliry of memory. This compiler performance goals. Our instrumenta­ kind of optimization is often seen in computer tion data revealed that the tree structure used tO operating systems and in general manufacturing internally represent the Ada code occupied most processes, but rarely seen in software tools such of the memory. Therefore , a finer analysis of the as compilers. This section describes the use of data was performed based on frequency of occur­ instrumentation data to diagnose and solve a pag­ rence and size of the individual kinds of tree ing-rate problem we detected during the devel­ nodes. opment of the compiler. For example, we instrumented the tree node The VAX Ada compiler consists of a number of creation routine to count the number of nodes of phases that process the internal tree representa­ each kind that were created. The counts and the tion of Ada source code in a series of tree traver­ number of bytes occupied by each node of a par­ sals, or walks. Walks in the semantics phase mod­ ticular kind were displayed in the compilation ify the tree representation tO reflect the semantic listing file. meaning of the Ada code. Later walks, prior to An analysis of the data showed that re lative ly optimization and code generation, add code-gen­ few kinds of nodes occupied most of the space eration information to the tree. and suggested a number of improvements in the Each of these walks is instrumented to show design of the tree tO reduce the frequency of such the amount of CPU time, elapsed time, page nodes. The combined effect of these improve­ fa ults, and 1/0 operations involved. An analysis ments halved the memory required for represent­ of this information during the development of ing the tree for typical Ada units. VAX Ada showed that a very large number of page Further instrumentation showed that the addi­ faults often occurred for typical program units. tion of code-generation information to the tree Even with larger than normal working sets, the representation substantially increased the tree paging rate was high enough to significantly size. These measurements suggested that memory increase the load on the system, thus affecting usage could be decreased by recycling memory as overall system performance and responsiveness soon as possible. An "inside-out" code-generation for all users. Comparison of the paging rates with scheme was devised for our version 1.0 compiler. the same data for other parts of the compiler, With this approach, object code is generated for and against the totals for the whole compilation, the most deeply nested subprograms in a compi- showed that a very large proportion of the page

Digital Technicaljournal 93 No. 6 February 1988 Pragmatics in the Development of VAX Ada

faults occurred during the walks that added elapsed time and system performance. The exact code-generation information. numbers vary according tO the actual VAX hard­ The trouble with any "static" solution to this ware configuration and Ada code being com­ problem is that page faults are a property of the piled. However, figures for the code-generation amount of physical memory available tO the com­ phases were often halved, resulting in 30 percent piler. The amount of physical memory varies or more overall improvement for the whole com­ based on both the VAX hardware configuration pilation . and the use of that hardware by other VMS pro­ This dynamic measurement of working set, vir­ cesses ru nning concurrently with the process tual memory, and tree size and the subsequent executing the compiler. tuning of the selection of sets tO the process's In an effort to solve this problem, we measured available resources means that all resources - the size of the tree for typical Ada subprograms. large or small - were fu lly exploited. This tech­ We found the tree size to be significantly smaller nique is applicable for enhancing the perfor­ than the size of the code doing the individual tree mance of any compute-bound programs that also wa lks. Furthermore, the code for the tree walks use significant amountS of virtual memory. was larger than typical VMS working sets . Thus, the code for each walk was paged out by the sub­ Instrumentation as a Debugging and sequent walk and then paged back in again for Maintenance Aid the next subprogram. We concluded that the high In addition tO using instrumentation to obtain paging rates were caused by our inside-out code­ resource measurements, we have used it to debug generation approach, which was designed to min­ the compiler. We have also found it to be a useful imize the use of virtual memory. maintenance aid. To reduce the paging of the code, we chained Instrumentation data is read by calling one of a together the trees for sets of subprograms and did number of routines either from the VMS Debug­ each walk across all the elements of the set ger or from code triggered by an event. (Events before applying the subsequent walk to any of are special places in the compiler code.) The them. This approach is contrary tO the earlier routine displays the instrumentation data on the goal of reducing memory usage by doing one sub­ terminal (so the programmer can see it right program completely before doing the next one. away) and in the listing fi le (for post-mortem However, in this context the earlier goal is more examination) . The debugger or event-driven rou­ accurately stated as "keeping the memory usage tines are capable of producing human-readable to within the amount of memory that is avail­ listings of large and complex data structures. able." The listings help simplifythe task of debugging As a result of our observations, we also made the compiler, as it can be very time-consuming the compiler "self-correcting" in a release fol­ to examine directly a very complex data struc­ lowing version 1.0. We instrumented the com­ ture, such as a tree, with a general-purpose piler tO measure the amount of virtual memory tool like the VMS Debugger. (An example listing available, the amount of physical memory avail­ appears at the end of this paper in the section able, and the pre-code-generation size of each Self-description.) subprogram 's trees. In addition , very conserva­ Each event is specified in the compiler code by tive heuristics estimate the additional memory a DEILEVENT macro. This macro takes one or required for the code-generation information for more parameters. The first parameter is the name each su bprogram. Together, the measurements of the event, and subsequent parameters specify and heuristics are used by the compiler tO build additional code that causes instrumentation data the largest possible set of subprograms that do tO be displayed. not present a danger of exceeding the available An event will not occur unless its name has virtual memory. Furthermore, the sets are chosen been given either on the command line that so that the code for the largest phase plus the size invoked the compiler or via a simple interpreter of all the trees for the subprograms in the set are that is linked into the compiler. The interpreter less than the size of the wo rking set extent of the displays event names and allows breakpoints to VMS process. be set or canceled on particular events. For exam­ This modification successfully lowered the ple, the Ada compiler implements a sophisticated paging rates of the compiler, hence improving syntax error recovery scheme that attempts a

94 Digital Technical]ournal No. 6 Fe bruary 1988 Software Productivity Tools

large variety of local corrections when an error is Detecting an internal error - often well detected. When the parser makes an unexpected before the error leads to a compiler crash or, correction, events in the recoverycode can be set worse , bad code is generated - is the primary to gather the data to determine why. Events in the advantage of an assertion. Assertions often point recovery code are set by the setting of break­ out errors that otherwise wou ld not be noticed points on all events whose names start with during internal testing. PAILRECOVERY. The result is an informative dis­ Assertions also help in analyzing failures, as play at the start of error recovery, and another dis­ they provide a very good point at which to start play as each kind of recovery is attempted . The a search for the cause of an error. In a com­ displays can then be used to determine the reason plex, multiphase compiler, a bug in an early for the particular recovery chosen. phase can result in a compiler crash in a much The info rmation obtained by setting an event later phase or in the generation of bad code. gives precise information that is needed to deter­ In many cases, the relationship between symp­ mine why the compiler code made a particular toms reported by a user and the actual problem decision, as opposed to the more general infor­ can be very remote and obscure. For example, mation given by the VMS Debugger. Often the approximately half of all performance failures time saved in analyzing each problem exceeds reported by users of the VAX Ada compiler trigger the amount of time required initially to put the some kind of assertion failure when compiled events into the code. Furthermore, such events using the debugging ve rsion of the compiler. As are still in place for the benefit of future develop­ a result, many problems that might have re­ ers who need to make enhancements or debug quired days to fix in the absence of assertion other problems. checks have been fixed very quickly because we knew where to look for the problem. Although Self-checking we have no statistics, we have no doubt that asser­ tions have saved an enormous number of mainte­ As mentioned previously, the VAX Ada product nance hours. contains approximately 500,000 lines of BLISS Assertions also he lp in day-tO-day development, source code. Of these lines, approximately 5 per­ debugging, and project management. Simple cent are concerned with consistency checking inspection of the assertion fa ilure message is (self-checking) of some kind. This is not very enough to know who should be the first to look at much incremental code in terms of overall devel­ the problem, and the person assigned to investi­ opment cost, yet the reliability and productivity gate the problem has a good idea of where to benefits have been enormous. look. Assertions are also useful when the code is The following sections examine some of the enhanced, as new code is checked against the consistency checks we incorporated in the VAX assumptions made by the original programmer. Ada compiler for use by developers and main­ We implemented assertion checks using a tainers. We look at the use of assertions in the series of BLISS macros. (Although BLISS macros code, at the use of special macros to mark unim­ were used to implement the checks, similar plemented features, and also at how we used effects can be achieved with subprograms in self-checking to track down memory-manage­ other languages, such as VAX Ada, if the compiler ment errors. evaluates static, constant expressions at compile­ time and supportS inline expansion of subpro­ Assertions in the Source Code gram calls.) These macros are listed in Table 1. An old idea in software engineering is to include Each macro takes two or more parameters. assertions in the source code. In its simplest The first parameter is the assertion (expression) form, an assertion is a simple expression whose tO be checked. The second is a text string to be va lue should be true at a given point in the code. displayed in the diagnostic produced when the If the assertion is fa lse, then something is wrong assertion is false. By convention, this text string and execution should be aborted . Although the includes the name of the routine in which the idea of assertions is not new, we believe that their failure occurred in order to simplify assigning value is underestimated and that assertions are initial responsibility (blame) for the failure. Any too often neglected in developing large software additional parameters are interpreted as addi­ applications. tional code to be executed if the assertion fails;

Digital Tecbnicaljournal 95 No. 6 Fe bruary 1988 Prag matics in the Development of VA X Ada

Ta ble 1 Assertion Macros and Their Effects

Effect in Effect in Macro Name Debugging Compiler Production Compiler

DEB__ASSERT If assertion is false, thengi ve a diagnostic None message and enter VMS De bugger.

DEB_WA RN If assertion is false, then give a diagnostic None message and continue.

DIAG__ASSERT Same as DEB__ASSERT. If assertion is false, then abort compiler.

typically these parameters are used to display Ma rking Code Pa ths fo r additional information related to the fa ilure. Un implemented Fea tures Numerous assertions in the source code can We adopted a rule during the development of have a negative effect on performance. For exam­ VAX Ada that the software at each intermediate ple, the consistency checks in the Ad a compi ler base level had to be robust. We req uired that the increase compilation time by about 50 percent. compiler diagnose the use of an unimplemented Thus, if assertions are tO be included in the fi nal feature rather than crash or generate bad code. product, developers will naturally hesitate to use This fo rm of self-checking was implemen­ them freely. We addressed this problem by caus­ ted by the two macros called DJAG__N\'1 and ing the DEB-ASSERT and DEB_WA RN macros ro DJAG__]\ryLSTOP. These macros are called with a be cond itionally compiled. The assertion checks text string that identifies the particular fe ature are made only in a debugging version of the com­ that has not been implemented. The execution of pi.lerthat is used for internal testing. The macros either results in a "nor-yet-implemented " diag­ are compiled as "no operations" in the produc­ nostic. DJAG_NY! is used in situations where tion version of the Ada compiler and thus have no processing can continue after the diagnostic. impact on performance. DJAG__Nl' LSTOP is used w indicate that the On the other hand , it is desirable in some situa­ compiler should be aborted after reporting the tions to retain the self-check in a production ver­ problem since there is no easy way tO recover sion but cause a fa ilure to be have differently than gracefu lly. it does in the debugging version of the compiler. These macros proved to be a good clerical The DlAG__ASSERT macro addresses this situa­ device for keeping track of work remaining. In tion . DlAG__ASSERT behaves in the same manner addition , our approach - never leave a hole - as the DEB-ASSERT macro in a debugging version contributed gr eatly tO the reliability of rhe of the compiler; however, DJAG--ASSERTaborts a product. Robustness was the norm throughout production version of the VAX Ada compiler. development rather than a last-minute, clean-up (The abort reports fa ilure of an internal consis­ activity. Over a long development effort, it is easy tency check and requests that the user submit a tO put off writing a particular code path for problem report.) another day and even easier to forget about it as These three assertions DEB_ASSERT, the days and months pass. DEB_WAR :'\1.and DJAG__ASSERT - are the most com mon form of consistency checking used in Tr acking Me mory-Management Errors the VAX Ada compiler. More general kinds of The last approach to se lf-checking we discuss in checking are provided, for example, by special­ this section is the use of special consistency ized analysis routines and even com plete traver­ checks ro he lp track down some obscure mem­ sals over the in-memory tree. ory-management errors in the compiler. Memory-

96 Digital Tech nical jo urnal No. 6 February 1988 Softwar:-e Pr:-oductivity Tools

management problems can be very difficult to • Simplify creation of some kinds of instrumen­ diagnose because, for example, large programs tation often operate correctly for a long time after a rou­ • Provide sophisticated debugging aids tine writes to the wrong location in memory. The error-tracking progression that we Each node in the tree contains an eight-bit field describe here occurred during the development named the KIND field. This field contains a value of the initial version of the compiler. In each of indicating the kind of information represented in the three problems in the progression, the intro­ that node. More than 230 kinds of nodes are used duction of a new check led immediately to the throughout the compiler. (There are many kinds discovery of additional cases where the same because the tree is used not only for statements error was occurring but, fo r whatever reasons, no and expressions, as is common in many compil­ negative consequences had yet been observed. ers, but also for declarations, in place of a more Each of these cases was a bug that wou ld eventu­ traditional symbol table. Indeed, except for com­ ally have been triggered, requiring many hours of me nts, the entire unit being compiled is repre­ a deve loper's time to debug. Finding the errors as sented by a single tree.) a result of one of these checks was fa r less expen­ The KIND field is located at the same offset in sive in terms of development time than fi nding every node; given the address of a node, it is easy them one at a time as each situation arose. to determine its kind and thus the information The first problem we discovered in the pro­ ava ilable in that node. gression was that a block of memory was being Moreover, the KIND field can be used to access freed as expected, but the block size specified in a "node property table" in the compiler that con­ the tree was larger than the amount of memory tains a description of the fields in each kind of originally allocated for the block. To guard node, including the fields' types, offsets, and so against this behavior, we allocated (in the debug­ on . Because each node describes itself in its ging version of the compiler) an extra longword KIND field and because the KIND field can be for each request and used it to remember the used to access the node property table, we refer allocated size. This procedure allowed us to to the compiler tree as a "self-describing" data check the deallocation requests. structure . Later, we discovered that a routine was The source-code definition of the tree repre­ attempting to deallocate a block of storage back sentation can be thought of as essentially a vari­ to a zone (subheap) other than the zone from ant-record type, where the kind value is a tag which the block was allocated. We coped with that discriminates among the variants. The actual this behavior by changing the extra longword to run-time description of the tree representation contain the Exclusive Or of the allocation size goes beyond the level of detail that can and the address of the zone control block. be expressed even in a strongly typed language , Still later, we discovered that storage was being such as Ad a. For example, the description read after it had been deallocated. To cope with distinguishes between four kinds of pointer this behavior, we changed the deallocation pro­ fields - all of which are simply pointers to other cedure so that it overwrote the deallocated stor­ nodes in the tree from a data-type point of view . age with all one bits. The one bits allowed us to However, it is the presence of the variant-record distinguish unallocated storage ones from newly definition itself as pan of the compiler that is allocated storage, which is generally initialized unusual and leads to valuable implementation to all zero bits. techniques. Self-description Automation of Key Algorithms The primary data structure used throughout the Several parts of the compiler use the node prop­ com piler is a tree representation of the unit erty table as a major part of their operation. For being compiled. This representation was made example, the part of the compiler that reads and self-describing in order to writes the tree representation to disk, called the compilation library component, is driven almost • Automate key algorithms in the compiler completely from the node property table. As a • Simplify creation of internal consistency result, we can easily add, delete, or change a checks (self-checking) field, introduce new node kinds, and so on. After

Digital Technicaljournal 9 7 No. G February 1988 Pragmatics in the Development of VAX Ada

a change is made, all that is needed is to recom­ fixed position in every kind of node.) If a node is pile the few BLISS modules that create the node encountered that already has the bit set, then a property table, link a new compiler, and con­ cycle has been detected (and a bug exposed). tinue development. The compi l ation library code This check is performed at least twice, sometimes docs not need to be recompiled, let alone modi­ three times, during a compilation when the fied, to reflect the change; it adapts automati­ debugging version of the VAX Ada compiler is cally. used. Similar considerations apply to other pans of the compiler. In particular, the compiler has an Instrumentation through the algorithm for copying trees that is fundamental No de Property Ta ble to the implementation of generic instantiation, The node property table also provides a valuable inline expansion of subprogram calls, and default tool for implementing certain kinds of instru­ parameter evaluation. This algorithm is also heav­ mentation . Statistics on kinds and amount of stor­ ily driven in part by the node property table. age by kind are readily calculated using simple Many utility routines also make good use of tree wa lks like the one described for self-check­ the node property table, for example tO create a ing in the preceding seCLion. node of a given kind - given the code for the kind - the required size is obtained from the Enhanced Debugging node property table, and each field of the new node is properly initialized as appropriate for Finally, the node property table provides the that type of field. basis for the variety of debugging display routines that were written as part of the project. As Self checking Based on the No de described in the section Instrumentation, these Property Ta ble routines go well beyond what could be achieved We have described some kinds of self-checking by even the best general-purpose debugger, earlier in this paper; it is also interesting to see including the VMS Debugger. Rather than show that some self-checking is based on the node the tree as a series of independent nodes, we can property table. First, we must back up and be a display the tree as the nested data structure it little more precise in our vocabulary. really is. Extraneous information, such as the Al though we talk of the "tree," this is really addresses of nodes that are not legitimately something of a misnomer. The tree is really a pointed to by attribute pointers, can be sup­ general directed graph. However, there is a sub­ pressed. Certain kinds of nodes that are actually set of the pointer fields that, in fact, does deter­ "private" in the Ada sense can be displayed in mine a spanning tree - a set of paths that a natural manner by display routines created spreads from the root (the COMPILUNIT as part of the implementation of these abstract kind of node) and reaches every node exactly node kinds. once (and without cycles) . Pointer fields that Figure I illustrates one such display for a sim­ define the spanning tree are called "son ple example program. It is not necessary to pointers," whereas all other pointer fields arc understand this display in detail; rather, compare called "attribute pointers." (Son pointers are the kind of display one could get from node-by­ one of the several kinds of pointers alluded to node displays versus the highly annotated and earlier; there are three kinds of attribute interpretive example shown. A display tool such pointers.) as the one that produced this example clearly is Because the "tree-ness" of the program repre­ application specific and could be produced only sentation is so important ro the correct operation as part of the project in which it is used. More of the compiler, one of the most important self­ importantly, even a project-specific roo! such as checks the compiler makes is tO ensure that the this would not be practical without the run-time tree really is a tree. This self-check is accom­ self-description of the data structures in use. plished by a routine that starts at the root (the COMPIL_UNIT node) and uses the node property Summary table to visit every node in the unit. Every son During the development of VAX Ada , we relied on pointer is followed. As each new node is encoun­ established design and implementation method­ tered, a bit is set. (This bit is reserved at the same ologies, and we made extensive use of VMS tools

98 Digital Tecbnical]ournal No. 6 February 1988 Software Productivity Tools

1 procedure FOO is 2 X : I HTEGER : • 0; 3 begin 4 if X > 0 then X : • X - 1; end if; 5 end ;

00880380 : FOO - K_PROC_BODLDECL >: < >: _CORRES_BLOCK K_80DY < 008AD9 F4 : K_DECLS< @2 :4 008AD898 :' X - K_OBJ_VAR IABLE < HOH_COHST , AHA_USED @2 :4 INTEGER - K_REFER 0 - K_ I HTEGER_VAL < CTC_VAL , STAT IC ®2 :22 SL > COHF_BEG_SEQ _08J_FLAG1 OCCURS_ J H_HAME _08J_FLAG2 008ADDEO: _SYMTAB _PRAG_REP_CC > _DECLSJLAGS ( (void) ) : _COHT IHUE 00000000 _CL_VIS>

K_ ! F_STMT < @4 :4 K_BI HARY_OP < HOH_COHST @4 :9 GL IHT X - K_REFER < HOH_CQHST 008AD898 : @2 : 4 > 0-K_ I HTEGER_VAL 00000230 : I HTEGER @STAHDARD/4 _81 HO I HG 00000140 : BOOLEAN @STAHDARD/2 _RES_TYP > ( K_ASS I GH_STMT < @4 :20 X - K_REFER < HOH_COHST 008AD898 : ®2 :4> K_B I HARY_OP < HOH_COHST ®4 :25 BIHARY_M IHUS_ IHT X-K_REFER < HOH_COHST 008AD898 : @2 :4> 1 - K_ I HTEGER_V AL < CTG_VAL , STAT IC @4 :27 SL > 00000230 : INTEGER @STAHDARD/4 _8 IHDIHG 00000230 : IHTEGER @STAHDARD/4 _RES_TYP > _ASS IGH_FLAGS > _LAST_LOCATOR

_IF_FLAGS > > (void 1 ist ) ( (void ) ) : HO_EXCP_PART _BODY_FLAGS _PRAG_REP_CC FF752988 : _ZOHE @3 :0 _BEG IH_LOCATOR @5 :0 _LAST_LOCATOR 0088CA 1C: _S I GARGS TIME _SAVED_OPT_STATE TIME _LOCAL_OPT_STATE _LOCAL_SUPP_CC >

DST _HAS_SEG , DST_HAS_ZEM , EX I M_ALLOWED , I S_ELABORATED' , I S_GBL_V I 5, IS_L IB_UH IT, ME CH_F IXED _PROC_FLAGS _PRAG_REP_CC 0088CA 1C: _SYMTAB < >: _FULF ILLS < >: _STATUS_OBJ > >

Figure I Example Tree Disp lay in the VA X Ada Compiler

Digital Techtllcal]ournaJ 99 No. 6 Fe brumy 1988 Pragmatics in the Development of VA X Ada

(VMS Debugger, VAX Performance and Coverage roles in our day-ro-day practices, from early Analyzer, and so on) . However, we also used design phases through field test. We continue to some relatively simple internal tools and accom­ use these tools and techniques in the ongoing panying development philosophies to help us maintenance and evolution of VA X Ada. We hope increase our productivity, improve the reliability that our successful experience wi th these prag­ of our product, and decrease maintenance costs. matics on the VPu"\. Ada project will help promote Automation, instrumentation, self-checking, wider interest in the use of such ideas on other and internal self-description all played major software projects.

100 Digital Tecbnicaljournal No. 6 February 1988 Steven]. Grass I

Development of a Graphical Program Generator

To develop an unprecedented graphical-interfa ce product fo r generation of COBOL applications, project engineers explored a new development approach. During the advanced development phase, typ es of generators were researched, a prototype was built, and product goals were outlined. In the product development phase, the major com ponents of the VAX COBOL GENERATOR software - the data dictionary, the work-file system, and the graphical display - were designed andcoded. In addi­ tion, developers integrated existing components into the generator. Te sting, design documentation, and proj ect review were alsopart of this phase. This development approach, combined with the use of several development tools, proved to be productive and resulted in a stable and reliable product.

The VAX COBOL GENERATOR software is a skeleton was built that contained most of the fourth-generation language approach to the cre­ core fu nctionality of the generator. Once the ation of commercial applications. Using this gen­ skeleton was completed, additional function­ erator, the programmer draws a picture resem­ ality could easily be added. The skeleron con­ bling a flowchart of the final application rather sisted of the work-fi le system, some screen than use an editor to write lines of code. This interface routines, the driver fo r the main screen picture produces VAX COBOL code, which can editor, and the driver for the code generator. be compiled, linked, run, and debugged. All Each member of the ream was responsible for maintenance is performed at the graphical level. one of these components of the skeleton. The This paper describes the development of the second base level marked the enhancements ro VAX COBOL GENERATOR software from initial the skeleton and the definition of system func­ concept to product shipment, a process that tionality in design documents. rook approximately three years. Because this was the first pro duct containing a graphical Advanced Development interface developed at Digital, many unique pro­ A program generator was so unlike other prod­ ject development problems were encountered. ucts being designed at Digital at that rime that This paper discusses how the project team it was necessary to spend the first project solved these problems in the research, develop­ phase on advanced development work. During ment, document, test, and project review stages. this time, research could be performed, ideas The project can be divided into rwo major could be excha nged between the developers, phases. The first phase involved advanced devel­ and breadboards and prototypes could be cre­ opment and lasted one year. During this phase, a ated. This advanced development phase indeed prototype was developed and demonstrated to proved ro be worthwhile and a significant step various management groups . toward the prod uct's success. In particular, the The second phase was prod uct development development of the prototype provided a way to and lasted two years. This product development communicate concepts to each other and to man­ phase was divided into two major base levels, agement. Remarkably, the prototype, although that is, milestones at which specified capabili­ crude, incorporated all the underlying concepts ties are complete. At the first base level, a contained in the final product.

Digital Te chnical journal 101 No. o Fe bruary 1988 Development of a Graphical Program Generator

From June 1983 to June 1984, two developers many of the lower level routines still need to be worked on the advanced development of the VAX edited by hand. Once having edited the gener­ COBOL GENERATOR software. This section dis­ ated program, the developer can no longer cusses that work, including the creation and enhance the product using the generator since demonstrations of the protOtype. the hand-coded changes would be lost . There­ fore, after a program has been developed, all Defining Product Sp ecifications and program maintenance must be performed at the Beginning Research code level. The development time saved using The specifications for the proposed product the program generator is only with reference to were open ended. The one , general product the program development phase, nor the pro­ requirement was that the program generator gram maintenance phase. would generate VAX COBOL code . At that time, The VAX COBOL GENERATOR team saw an the kind of generator ro build, the interface ro opportunity to close the gap between program use, and other aspects of the product were not development and maintenance. They decided to yet understOod or specified . produce a program generator that wou ld create The first two tasks facing the developers, the entire program. Software development gains therefore , were to determine what a program in terms of developer time saved would then generatOr was and what currently existed in the extend beyond development phase to encompass marketplace. They learned that although much the maintenance phase as we ll. had been accomplished in the development of The team also saw another opportunity. Most fo urth-generation languages, not all product existing generators are restricted in the types of approaches had been fully explored. They saw applications that can be generated . Although the advantage in creating a type of generator that generators cou ld relatively easily create typical not only increased programming efficiency but commercial applications, complex applications also was simple and interesting to use. were more difficult to create. The generator They learned there are two types of genera­ team decided to build an open-ended product tors: application generators and program genera­ that stressed flexibility in the level of program tors. Each has its own advantages and serves a complexity. distinct market. Finally, most of the interfaces of the genera­ An application generatOr processes commands tars we studied were menu-driven. Users were interpretively. It does not produce source code. required to repeat continually the same steps in The application generatOr has a close relation­ order to create the application. The generatOr ship with its application's database and is used team fe lt that a user-friendly human interface mainly for relatively simple data retrieval and wou ld be a more expedient tOol and more report generation. Where execution speed is not appealing to users as we ll. critical, programmers use application genera­ The decision to produce a graphical interface tors for quick development turnaround. DATA­ for the generatOr was one of the first the devel­ TRIEVE, RALLY, and Cognos' PowerHouse are opment team made. In 1983 the VT200-series examples of application generarors. terminals were beginning to be shipped, and A program generator, on the other hand, pro­ graphics workstations were starting to be devel­ duces source code. Program generators can ge n­ oped . It was evident to the generator team that erate anything from BAS IC to Ada program code. graphical terminals would become integral to Because the code produced can be compiled, program development work. execution speed of the created application is Moreover, human-factors research and the faster than the execution speed of a similar developers' own experience with graphical application produced by an application genera­ wo rkstations confirmed the decision . During tor. A program generator does, however, rake researc h, the developers spent a great deal of longer tO develop the application. time using the revolutionary Xerox Star worksta­ Most program generators produce about 70 tion, which had been introduced in the early percent of the fi nal application. However, the 1980s applications produced are skeletal and have to This workstation demonstrated the power that be ed ited after development. The generated pro­ windowing and icons can give to software devel­ grams consist of the high-level structure, but opment. The developers fe lt that they could

102 Digital Te cbtlicaljournal No. 6 February 1988 Software Productivity Tools

expand the concepts of the Xerox Star (later fu r­ ten in the VAX COBOL language. The fi nal ther demonstrated in Apple's Macintosh com­ product wou ld be written in the preferred puter) ro the area of program generation. The development language, VAX BLISS. Also, it was icons could represent the various data and pro­ decided that a graphical software package such cedural entities in typical programs, and the as the Graphical Ke rnel System (GKS) would not windows could be used ro define these entities. be used for the protorype . The strengths of GKS The flexibility of a graphical workstation were terminal independence, easier mainte­ allowed this to happen. nance, and better performance, none of which In summary, research into types of generators were goals of the protorype . In addition, the and user interfa ces helped to determine the fo l­ time taken to learn GKS wou ld cause a needless lowing product goa ls prior to the development slowdown in the protorype development. of the prototype: Instead, ReGIS escape sequences would be out­ put. This decision w output ReGIS directly • The generator wou ld produce an entire VAX instead of using GKS wou ld speed the effort to COBOL program, thus extending use of the obtain a wo rking protorype. generator through the maintenance phase. As it turned out, many major functions of a program generator were completely defined • The generator wou ld have the flexibility to create complex as well as simple types of within the prowrype. The method of fo rm, applications. report, and file definition, as well as the con­ cepts of procedural and data flow could all be • The generator interface would be a graphical visualized using the prororype, which even did interface; it wo uld be easier and faster to use program generation. The prototype was so com­ than conventional ediwrs. plete that a member of Digital's Management Information Systems (MIS) department used it to Project Va lue of the Pro totyp e develop applications to be used within his ln the early research stages, the product ideas department. The prototype demonstrated how formulated by developers were so unlike any these high-level graphical concepts could be previously developed products that the ideas translated into the generation of source pro­ were diffi cult to explain and demonstrate. Icons, grams. for example, were not in general use at that time; and the ideas of boxes and lines represent­ Product Sp ecification Approval ing operations and control flow were entirely Demonstrations of the prototype were given well new concepts and were difficult to grasp. over one hundred times tO all levels of manage­ Attempts to draw the ideas on paper fa iled since ment, project leaders, some customers, and Digi­ drawings could not show the fa cile action of the tal 's Research and Development Committee . inte rface that the developers visualized. Devel­ Reaction was positive. It was agreed that the opers decided they needed to construct a proto­ product was ready for development. Unfortu­ type. nately, reviewers had little with which to com­ Creating a prototype before specifications pare the product and were therefore unable tO were clearly defined was a risk the deve lopers offer the constructive criticism the developers wanted to take . The simple prowtype proved were seeking. Feedback on the product would them right. It was invaluable for communicating be gained through the more painful process of project concepts among the developers and to experience. management. Their first task was to decide what fu nctions w Product Development build into the prowtype. A prowtype is written The product development phase of the VAX w demonstrate ideas, without regard to mainte­ COBOL GENERATOR software started in june nance or the performance of any programs pro­ 1984 and ended in September 1986. This two­ duced. Therefore, the developers knew that the year period began with the first written design prOtotype code wou ld have to be discarded specifications and closed with delivery of the when the product development began. In order product. Product development included design , to ensure that no code wou ld be reused, it was coding, and testing phases. No more than fou r decided that the entire prototype wou ld be writ- software engineers were assigned to the project

Digital Technicaljournal 103 No. 6 February 1988 Development of a Graphical Program Generator

at any one time. An additional three engineers A node graphically represents data and opera­ were assigned to write the graphical package tions to be performed on that data. There are (described later in this paper). The final eight node types, and they fa ll into two cate­ product had over 140 ,000 lines of source code, gories: procedural and data. Procedural nodes or nearly 3000 lines per developer per month represent fu nctional tasks to be performed in the during the coding phase. This achievement was program or represcm strucmre in the program. considerable when compared tO 198';, when Examples of procedural nodes are those that 6'50lin es of code was the average number writ­ represent the movemem of data between two ten per developer per month. 1 The productivity data nodes, a sorting operation, or the manipula­ of the VAX COBOL GENERATOR team was due in tion of a menu . Data nodes represent data to be large part to Digital's software development accessed in the program. Examples of data nodes 2 environment tools Some of these rools are are forms, files, and reports. described later in this paper. The connections between the nodes represent This section gives a brief overview of the flow in the program. Procedural connections COBOL generatOr product. Fol lowing this show procedural flow; data connections show overview are descriptions of the components and data flow. systems selected for and designed in the The programmer creates nodes on each level product's implementation. The major compo­ of an application and connects them to show nents of the generator are the data dictionary, procedural and data flow. Editors, pop-up forms, the work-file system, and the graphical display and pop-up menus prompt the programmer for system. Also described in this section is the detail about the nodes and connections. From reuse in the generatOr of previously developed this information, the VAX COBOL GENERATOR components. software creates a VAX COBOL program. Figure 1 shows an example of a VAX COBOL Product Fu nctions Overview GENERATOR screen. The procedural type nodes Programs are graphically described within the READ-INFO and SHOW-ERROR are shown as are VAX COBOL GENERATOR product by a combina­ the form node EMP-FORM and fi le node EMP­ tion of nodes and connections between these FILE. The data connections are shown as dashed nodes. After the nodes and connections arc cre­ lines, and the direction of the connections indi­ ated and defined, the V�'( COBOL GENERATOR cates that data is to be read from the form and software can create a VAX COBOL source pro­ written into the file. The procedural connection gram which can be compiled. is shown as a solid line that indicates control

· · f . • •\' di:U l�l ·.li '··· r

' ' I · [] ....

"

Figure I VA X COBOL GENERA TOR

I 04 Digital Technical journal No. 6 February I '}88 Software Productivity Tools

flow is to go from the READ-INFO data move­ tions. Deve lopers. however, needed a method ment node to the SHOW-ERROR procedure that cou ld in addition understand entities node. defined by the user within the generator, such as The generator helps the novice user via sev­ reports and user-defined procedures. Therefore, eral fu nctions. At any time, the programmer can the COD was used within the generator so that use the Help key to obtain context-sensitive users cou ld optionally share record definitions; information about an operation. The pro­ another method, the generator library, was grammer can also use the HELP command or devised so they could share other components of choose help from the menu. Easy-to-understand the generator. error messages also guide the programmer The generator library lets the user share any through the design process. form, fi le, report, local storage, procedure , or The VAX COBOL GENERATOR software field definitions . If a user of the generator wants enforces top-down programming. The pro­ to share a program component, this component grammer begins program definition at the top can be either defined directly in the library or level of the program. Group nodes, which repre­ stored in the library from an application. Any sent much more complex operations at a lower other program developer wishing to reuse that Level, create structure within the generated pro­ component in another program can simply refer­ gram. Editing a group node moves the pro­ ence it. grammer down one level in the program, thus Each time a new node or field definition is breaking the program into smaller, more modu­ created within an application, the generator per­ lar pieces. fo rms a search through all known generator Other VAX COBOL GENERATOR fe atures libraries. If a match is fou nd, the user is given an include data dictionaries and libraries for the option of referencing the component from the storage and reuse of common data and proce­ library. If he chooses to reference the compo­ dures, an RdbjVMS interface, a complete jour­ nent, the previously defined component is then naling capability, and a method to escape into read into the application and is thus reused. The an editor where user-defined COBOL code can internal representation stored in a library fi le is be entered into the generated program. Also identical to its counterpart defined in the appli­ available are program documentation facilities cation . Because of this, no conversion is required that include a map with a breakdown structure to be performed by the generator when a com­ of the program. These facilities also permit the ponent is referenced, and the internals of the addition of user-written documentation to parts generator do not need to know whether the com­ of the program. ponent is referenced from a library or is defined within the application. The Data DictionaryAs set The data dictionary utility proved to be one of A main component of the VAX COBOL GENERA­ the strongest assets of the product for two main TOR is its data dictionary. Al though this compo­ reasons. First, because components can be nent was not included in the prototype, the reused in many applications, users' programs developers found in later research that nearly all can, for example, all have the same interface. fourth-generation languages, no matter how Each user does not have to redefine the compo­ primitive, contain a data dictionary. As Digital's nents for his particular application . Second, mul­ MIS department pointed out, a successful tiple users can simultaneously develop diffe rent product must contain a depository for reusable pieces of the same program. Each user can programming. Developers therefore devised a define components in a different library, then method by which users of the VAX COBOL GEN­ one user can integrate these components into ERATOR product could easily define data and the application. procedural entities in a central location. Users The one restriction is that only the generatOr could then share these entities within one or can read from or write tO the library structure, more programs developed using the generator. because the library structure is defined by the The VAX Common Data Dictionary (CDD) was VAX COBOL GENERATOR software. Forms can the current Digital standard for sharing data be shared among COBOL programs developed by among the layered products. CDD is excellent as the generator, but not with any other language a standard for sharing record-structure defini- or tool .

Digital Te chnicaljo urnal 105 No. 6 February 1988 Development of a Graphical Program Generator

The Wo rk-file Sys tem pie caJis to GKS routines to create the square Another key component in the VAX COBOL GEN­ containing the node, draw the text, and draw the ERATO R software is the work-file system struc­ icon representing the node type. In addition , ture, a key component in any software product. It calls would have to be made to determine if all or was important to the product's success to give part of the node would be visible, to determine users the ability to quickly access the generator's which font to use for the drawing of the text, to database on disk and to manipulate records in determine the select area for the node, and others memory. The developers were looking for an as well . Instead, one higher level routine per­ easy-to-use, efficient interface when they forms all these fu nctions. decided on the format of the VAX COBOL GENER­ Another product being developed, the VAX AT OR database file and the associated manipula­ Software Project Manager program, also needed a tion routines. Instead of developing a new file graphical interface with the same capabilities our structure, the deve lopers decided that the data­ product required. Consequently, the group of base wou ld be an RMS indexed file. They could three engineers, mentioned earlier, developed then use standard VAX RMSfi le manipulation rou ­ the high-level graphical manipulation routines tines rather than write new routines. Time saved layered above the VAX GKS software . These rou­ during development was considerable. Most tines wou ld be completely terminal indepen­ work-file systems take months to develop; the dent . Any product using these routines wou ld run generator's system was performing within two on a terminal, where the cursor is manipulated weeks. Run-time performance, thought at first to by arrow keys, or on a workstation , where the cur­ be a possible problem, was acceptable. sor is manipulated by a mouse . By sharing the After the work-file records were read into same human-interface routines, both the genera­ memory, the routines used for manipulation were tor and the VAX Software Project Manager pro­ patterned directly after their RMS fi le counter­ gram have the same appearance and interface . parts. Each record contained a key, so records Consequently, users who learn one product's were read by key and then read sequentially. interface can more easily learn the other. Records defining a node were logically grouped Because developers needed to focus only on together since their keys were alike. developing the high-level screen interface. they were able to spend more time writing the genera­ Th e Graphical Disp lay Sy stem tor internals as opposed to developing the graph­ Another key component in the VAX COBOL GEN­ ical display system internals. ERAT OR is the graphical display system. The VAX GKS program is Digital's implementation of the Us e of Existing Components ISO (IS 7942) and AN SI (ANS X3. 124-1985) As noted earlier, the implementation language for GKS standard for two-dimensional, device-inde­ the VAX COBOL GENERATOR software is the VAX pendent graphics. The VAX GKS program had just BLISS language . Because Digital's software prod­ been released when development of version 1 .0 ucts had been written in BLISS, any components of the VAX COBOL GENERATOR software began 3 written for these existing products could easily Because it appeared to contain all the graphical be integrated into the generator. Time to market primitives and terminal independence for which was important, so the time-saving use of any the generator team had been looking, it was cho­ already written software was encouraged. Conse­ sen to be the graphical system for the generator. quently, during the design stage, developers Early in the development cycle, however, it decided to reuse some of these components. became apparent that a higher level graphical In addition to saving schedule time, the use interface was needed. GKS provided the required of these existing components meant greater functionality, but the routines were too low level producr stability. The components had been used for the developers' purposes . By layering a higher within Digital's products and therefore had been level interface above GKS, the generator's inter­ tested by customers for years, and the compo­ nals would be simpler and, therefore, easier ro nents wou ld be tested again after integration in develop and maintain. the generator. After the integration had been Using GKS calls, a node representation on the completed, the reused pieces contained fewer screen would be extremely complex to con­ errors than any other components within the struct. The generator wou ld have to make multi· generator.

106 Digital Tecb nicaljournal No. 6 February 1988 Software Productivity Tools

This section describes the components that interface routines saved a great deal of develop­ were used or adapted within the generator. ment time.

Fo rms Editor Design Documentsfo r Project Us e The definition of a form node required the use of The developers wrote a design document for a forms editor to define the layout of the form on each major piece of functionality needed for ver­ the screen. Digital had two forms products at that sion 1.0 of the VAX COBOL GENERATOR soft­ time: VAX FMS (Forms Management System) and ware. These documents included the following: VAX TOMS (Terminal Data Management System) • The reason for the new functionality programs. Not only did each contain a forms edi­ ror, but the key definitions within each were • A high-level description of the fu nctionality identical. The VAX COBOL GENERATOR develop­ • Details of the fu nctionality, including each ers decided to use this established set of key routine to be written or enhanced sequences. The similarity between the FMS and TOMS key­ • A test strategy pads is not a coincidence. Developers of TOMS • Resource and time estimates for design, cod­ modified the FMS sources for their TOMS ing, and testing of the new functionality product. The generator developers also decided to modify the FMS sources. Only two changes These documents had three purposes. The first were needed: one was to change the field purpose was to have the team review the pro­ attributes such that they were particular to the posed implementation of the new functionality. generator, such as autoterminate; and the other Any discrepancies in the graphical interface was to include an interface to the generator's data design or internal implementation were fou nd dictionary. Routines were written to convert early, before actual coding began. The second between the forms editor's internal data struc­ purpose was to give the project leader a reliable tures and those of the generator. estimate of coding time for determining future Within a very short time - one week - the schedules. The third purpose was ro help the forms editor had been integrated and was work­ individual writing the documentation keep cur­ ing within the generator. Writing a forms editor rent on all new features. With this info rmation, from scratch wou ld have taken much longer, per­ the writer could allocate the appropriate time haps six to eight months. needed to write about new features.

File Sys tem Product Te sting - Internaland A set of routines was needed to access the External product's various fi les, such as the generator Three diffe rent types of testing were performed database, generator libraries, journalfi le, and any during development of the VAX COBOL GENERA­ COBOL files that wou ld be generated. TOR software: To perform the standard set of operations on • Testing by the developers these files, the generator deve lopers chose the fi le I/0 system developed for the VAX TPU (Text • Human-factors testing, by the Software Usabil­ Processing Utility) software . It contained the ity Engineering Group standard open, close, read, and write routines in • Field testing, at Digital's internal sites and at a modular, easy-to-integrate form. As with the external (customer) field test sites forms editor, integration was si mple and stability was high. The standard internal test method for software at Digital is the VAX DEC/Test Manager (DTM) • CDD Interfa ce Routines software . DTM can test generated data and text The VAX COBOL compiler already contained rou­ screens. DTM can test the generator database fi le, tines that read in records defined within the VAX any libraries, and any generated COBOL source Common Data Dictionary software . The VAX code. A large DTM test set consisting of approxi­ COBOL GENERATOR developers converted these mately two hundred tests was run at period ic routines so that they recognize generator data intervals and at base levels. Te sts were required structures. Ag ain, the use of previously written to be written for each new piece of functionality.

Digitfd Te chnical jo untal 107 No. 6 Fe bruary I ')88 Development of a Graphical Program Generator

However, DTM has no capability for testing The second phase began in February 1986 and graphical screens. In fact, the developers discov­ marked the beginning of serious field test. The ered that there was no product available that VAX COBOL GENERATOR software, with nearly could test graphical and asynchronous interfaces. all functionality, was installed in 14 customer Any previous graphical product was tested inter­ sites and 200 Digital sites. Excellent feedback actively by the users, and this method was also was given from all sites . One customer site was used for the generator. able to generate a 1 00,000-line program during At each base level , the VAX Performance and this five-month period. Coverage Analyzer (PCA) software was run on the This site notified the developers immediately entire test set to determine test coverage. If PCA whenever a problem occurred; developers could determined that a large piece of code was not then fix errors in the product before it was being tested by the test set, one of the developers released to the public. wrote a new test for the code. The third and fi nal phase of field test began in The second type of testing was human-factors July 1986. All bugs found in the earlier tests had testing, performed by the Software Usability been fi xed, and all final fu nctionality was 5 Engineering Group. Early in the development included in the product. This phase was used to effort, test subjects (programmers) unfamiliar ensure that no major bugs remained in the gener­ with the VAX COBOL GENERATOR software were ator. asked to perform si mple tasks using the product. In summary, all three types of testing con­ The test report listed and prioritized all problems tributed to the stability of the final product. the subjects encountered. As a result of this Many bugs were found and corrected, the docu­ report, changes were made to both the human mentation was simplified, and the human inter­ interface and to the documentation. face was improved as a result of the various types For example, one problem, which accounted of testing. for the largest percentage - 19 percent - of the total error time, involved field termination Project Te st Communications in the editor used for the definition of record The unique and efficient communication structures. In this editor, the Return key was medium for internal test queries and responses 6 used to create a new field's definition. Users was VAX NOTES conferences The VAX NOTES were mistakenly using this key when attempt­ network communications product functions as an ing tO move from one attribute to another for a electronic blackboard and lets users conduct on­ field's definition. As a result, new field defini­ line conferences or meetings. Using the VAX tions were being created inadvertently. The NOTES program, any user on Digital's engineer­ generator team changed the definition of the ing network could make suggestions and ask Return key as a result of the human-factors questions during development, and report prob­ testing, and selected another key for the creation lems during field test . Because the developers of new fields. mon itored the NOTES conference continually, Other changes were implemented as a result of users were given feedback quickly. Ad ditionally, this testing. Better labeling was devised within the VAX NOTES program allowed the number of the generator, and better, easier-to-understand test sites to expand . Developers had more time to documentation and context-sensitive help mes­ work with more than the usual number of sites sages were written. because the program not only facilitated commu­ The third type of testing was field test. This nications between sites but also maintained a testing was divided into three phases. The first record of any communications. phase began in September 1985 and included an For communications outside the corporation unlimited number of Digita l sites and a limited during field test, a hotline was installed. Devel­ number of customer sites. Practica lly all func­ opers took turns answering the calls. The imme­ tionality was included in this release except fo r diate feedback allowed for quick problem resolu­ report and son nodes. Sites were asked to test tion, contributing to a successful field test. Also, how well the VAX COBOL GENERATOR software an on-line Quality Assurance Report (QAR) sys­ fit into their development environments. Unfor­ tem was avai lable to all field test sites. The sites tunately, fe edback from external sites was very could log on to the machine containing the sys­ limited in this phase . tem and comment on bugs or make suggestions.

108 Digital Te chnicaljournal No. 6 Fe bruary 1988 Software Productivity Tools

Developers wou ld periodically scan the QAR Conclusion database. The development process of the VAX COBOL GENERATOR software was successful. An entirely Project Management To ols new process for the generation of programs was devised, prototyped, written, and tested. More­ During VAX COBOL GENERATOR development, over, the process was completed in an amazingly developers kept resource status and task records short period of a little over three years. on paper. Group meetings were the only formal Three main factors contributed to the high pro­ mechanism for the exchange of status informa­ ductivity of the VAX COBOL GENERATOR team tion . A tool was used for scheduling, but and stability of the final product: it did not provide the task tracking capabili­ ties that were needed. Simultaneous with the • An early prototype, through which ideas could development of the generator, the VAX Software be presented for debate Project Manager program was being developed. • The reuse of existing technology, so time was Early versions of this tool were used at the not spent doing work that had already been end of the generator's development cycle, and performed ts benefits were readily apparent. Before the • Va rious forms tool was available, the scheduling done on of testing, where the technologi­ paper contained errors. The VAX Software cal and human interface designs could be Project Manager program automated the tested process. The scheduling estimates made with Acknowledgments this cool were found to be much more reliable I would like to thank the developers who worked than those done manually. Moreover, tasks are graphically displayed, making it easy to visual­ on the VAX COBOL GENERATOR software , including David Tarbay, Leo Treg ize a schedule and tO test various schedule giari, John scenarios. Ronan, Deb Bourquard, Andy D'Amore, Bill Foun­ tas, and Rich Phillips. Also , I would like to acknowledge the contribution made by the mem­ Project Review Meeting bers of the graphical display team, Vick Ben­ Immediately upon release of the VAX COBOL nison, )ay Bolgatz, and)eff Orthober. GENERATOR software, a project review meet­ ing was held. Al l developers presented their References views on how tO improve the process for build­ 1. H. Davis, "Measuring the Programmer's Pro­ ing this product. Although the development ductivity," Electronic Engineering Man­ process had gone well, many valid points were ager (February 1985): 44-48. raised at this meeting. For example, the newer mem-bers of the team had found much of the 2. B. Beander, "VAXjVMS Software Develop­ code difficult tO understand . The code wou ld ment Environment," Digital Te chnicaljour­ therefore be harder tO maint ain. As a result of nal (February 1988, this issue) : 10-19. this discussion, plans were put in place tO 3. B. Axtell, W. Clifford , J. Saltz, "Programmer develop coding standards for future versions of Productivity Aspects of the VAX GKS and the product. Another point concerned infor­ VA X PHIGS Products," Digital Te chnical mally made design decisions that went jo urnal (February I 988, this issue) : 62-70 . unrecorded and were often lost. h was decided 4. L. Ziman, M. Dickau, "Project Management at the meeting tO create a NOTES conference of the VAX DECjTest Manager Software where discussions and decisions internal tO Version 2.0," Digital Te chnical jo urnal the group could be logged. Using this confer­ (February 1 988, this issue) : 1 1 0-1 16. ence, future developers of the generatOr could easily reference the earlier decision­ 5. M. Good, "Software Usability Engineering," making process. Digital Te chnical jo urnal (February 1988, Team members found the me eting to be an this issue) : 125-133. extremely valuable exercise and wi ll hold such 6. P. Gilbert, "Development of the VAX NOTES meetings at the conclusion of all fu ture product System," Digital Te chnical]ournal (Febru­ releases. ary 1988, this issue) : I 17-124.

Digital Technicaljo urnal 109 No. 6 Fe bruary 1988 Linda Ziman MartinDickau I

Project Management of the VAX DEC/Test Manager Software Ve rsion 2. 0

To produce a camp/ex, major softwarever sion in less than one-year's time, the DECjT est Manager team became the first at Digital to use all available VMS productivitytools. As part of their strategy to meet a shortened sched­ ule and at the same time maintain quality, the team chose an iterative development app roach. Throughout the phases of requirements analysis, specification and design, and implementation, the team took advantage of the many software toolsavailable to streamline every aspect of the project frmn source code management to performance analysis. The tools used include VA X NOTES conjerencing, the VA X Language-SensitiveEditor, VA X DECjCMS software, and the VA X Performanceand Coverage Analyzer.

As software development has increased in com­ tions are stored in a DEC/Test Manager-controlled plexity, software engineers have sought products directory, called a "library," and are easily that help to increase their productivity as well as accessed, tailored, and managed via DECjTest 1 the quality of the software they produce. At the Manager commands. same time, market pressure to deliver more soft­ The core of a DECjTest Manager test descrip­ ware products has increased. Software productiv­ tion is a user-supplied script which the DECjTest ity tools are essential elements in the engineering Manager executes when the test is run. Use of a effort to meet the market need for the same or script and the environmental and processing con­ greater software functionality delivered in a trol specified in a test description ensures that shorter amount of time. It is not uncommon for only changes in the software being tested can the development cycle of a major version of a soft­ alter the results of a test run. ware product to be longer than one year in dura­ After test execution is completed , the DEC/ tion. The VAX DEC/Test Manager team was able Test Manager automatically compares the results to deliver a major version to market in less than of the run with a set of benchmark results. seven months. This paper describes the develop­ The comparison statistics are available through a ment methodology and the productivity tools simple command, and the DECjTest Manager used at various software life-cycle stages to provides a set of fu nctionality and commands achieve this shortened time to market. for locating and examining the results of those tests that indicated some type of change from VAX DECjTest Manager expected behavior. All file management is han­ ProductOverview dled automatically. The VAX DEC/Test Manager software is a produc­ The VAX DECjTest Manager software version I . 0 tivity tool that automates the regression testing of was mainly a batch testing system. Pressure came software during the development and mainte­ both from the market and from internal Digital nance phases. Regression testing ensures that engineering groups to produce a follow-on ver­ modifications made to the software do not affect sion that wou ld do more, one that would test the previously tested execution of the program. graphical applications on character-cell termi­ The DECjTest Manager allows users to describe nals, such as a VTS2 or VT100 terminal. tests as a set of files, execution environmental From the outset of the project, the develop­ aspects, and processing options. These descrip- ment team realized the difficulty of the design

110 Digital Tecbnicaljournal No. 6 Fe bruary 1988 Software Productivity Tools

problem. Ad ding to the difficulties of develop­ The first phase in iterative development is ment would be the lack of a commercially avail­ product requirements analysis, which we discuss able product with which to compare functionality next. for such a roo!. Requirements Analysis Ph ase Project Setup One of the most difficult tasks of software engi­ The DEC/Test Manager version 2.0 development neering is forming a clear statement of the prob­ team consisted of three junior engineers, a princi­ lems the software must solve. pal engineer, and a project leader. This team was Traditionally, on small, less rigorously struc­ to be responsible for producing the new func­ tured projects, programmers may interview users tionality and fo r maintaining the previously about their needs. On large-scale projects, such released versions of the software. as those done by Digital Software Engineering, At the beginning of the project, the team mem­ more formal requirements analysis is undertaken bers recognized they would need to improve early in the project's life cycle. However, no mat­ prod uctivity in order to deliver a product in the ter how formal the analysis process, the quality time required. The work involved to produce ver­ of the resulting problem statement strongly sion 2.0 would be as complex as it had been for depends on the information gathered during that the major ve rsion 1.0. Version 1.0 had taken process. 18 months tO produce, but market pressure dic­ The DEC/Test Manager version 2.0 team tated a second version be delivered in 10 months wanted to decrease the time normally taken to or Jess. gather this information without decreasing the Consequently, the DEC/Test Manager team was quality of the requirements analysis phase and one of the first at Digital ro use all of the VMS pro­ without reducing the number of target markets ductivity tools currently available. The team's contacted for information. use of these roots is discussed in the sections Also to be considered in gathering require­ below. The tools consisted of the following soft­ ments information is whether the proposed ware: product is new or a revised ve rsion of an exist­ ing product. For a new product, requirements • VAX Language-Sensitive Editor are often based on the abilities of similar prod­ • VAX DECjCMS (Code Management System) ucts or simply on programmers' and potential users' wishes. The requirements list for a new • VAX DECjTest Manager version of an existing product, however, can • VAX DECjMMS (Module Management System) be an ex-panded and more precise list. Mem­

• VAX Performance and Coverage Analyzer bers of a user community can make suggestions based on their judgments of the deficiencies • Problem Tracking or QARsystem of a product. The problem facing developers • VAX NOTES is finding a mechanism by which ro obtain this feedback in a short period of time from these • Digital Standard Runoff users. The project team also decided to use an itera­ The team chose not one but several ways to tive development approach. This approach dif­ obtain responses from widely dispersed and fers from the development method of complet­ varied groups of users. These methods are the ing all requirements of one stage, or phase, in the subject of the balance of this section. software life cycle before proceeding to the next. 2 Instead, an iterative process allows prob­ VA X NO TES Conferencing lems to be detected, fixed, and quickly incorpo­ The DEC/Test Manager version 2.0 team used a rated at any stage. Consequently, errors made in variety of methods to quickly gather feedback on the design are caught and corrected long before version 1.0 from groups in such diverse locations the field test. Additionally, this approach allows as japan, England, and California. To obtain input the corrected software tO be made available to from groups in similarly widespread locales, the the people who had detected the original prob­ developers of version 1.0 of the DECjTest Man­ lems. These users can then further evaluate and ager had spent a great deal of time in the require­ test the software. ments gathering stage.

Digital Technicaljournai 111 No. 6 Fe bruary 1988 Project Management of the VAX DECjTest Manager Software Versio n 2. 0

Significant time was saved through rhe use of Developer as Us er VAX: NOTES software. NOTES is a computer The team itsdf rook advantage of the DEC/Te st conferencing tool which rbc DEC/Test Mana­ Ma nager environment to gather requirements ger ream used to set up a forum for discussing input and to analyze the input. The ream fe lt that 5 its producr AJ rhough NOTES at that time software developers who use their own software existed only as several prototype tools, it pro­ products produce higher qualiry softwa re . vided a workable fo rum for the entire internal AJ rhough softwa re developers are already inti­ user base . Nor only was the team able to gather mately fa miliar with the internal technical requests and ideas from the internal base, bur derai ls of their products. if they are not also the it also made available preliminary specifications users , they lack external perspective. and gathered fe edback before any prototyping External perspective is the perspective of the was starred. customer or software user. A software developer NOTES alone, however, was inadequate for who has this perspective can more easi ly under­ gathering all the fe edback needed. At that stand and identify with customers' fe edback. The rime, only a small portion of the user com munity DECjTe st Manager had always been used ro rest act ively participated in the NOTES conference . itself; therefore, the developers themselves were Moreover, any fe edback from just one source users of the prod uct, were fa miliar with the tool 's might be skewed . To correct for this possibility, inadequacies, and knew where improvements a more direct form of input gathering was could be made. As users, they were a.lso able to also implemented. The DEC/Test Manager ve r­ screen requirements and specification ideas fo r sions I .0 and I .1 insta llation command proce­ usefu l ness before go ing to the user base for more dures had been made to send VAX Mail notifi­ suggestions. cation of any Digital internal site instal lation to the DEC/Te st Manager development account. Software Quality Management These notifications provided the team with a Quality assurance groups are nor common in fa irly complete list of the version 1.0 installed Digital Softwa re Engineering because Digital's base. The ream used the list as a distribution and management believes each software developer interest list for sending users questionnaires and and each deve lopment grou p must be respon­ requirements-input requests. These requests sible fo r the quality of irs own product. serve d to draw more users into the requirements­ AJ though this is indeed the case , the Software analysis process and to raise the quality of the Quality Management (SQM) Group performs requirements analysis itself. a unique, cross-product quality assurance fu nc­ tion. The VMS SQ M Grou p, part of the VMS Operating System Group, works with the various Quality Assurance Reports product groups to ensure that all VMS pro­ Most Digital software engineering groups keep ducts fo llow certain conventions for instal­ track of all bug reports and suggestions in lability and inter-operability The SQM Grou p Quality Assurance Report (QAR) databases . therefore obtains a set of tests suites fo r each These databases are used ro record problems, product and, on a frequent basis, rests how assign problems to specific developers , ana­ well products work with each other on the VMS lyze quality statistics, and generally ensure that operating system . problems do not go unnoticed . Many groups, Through this process, the SQM Group pro­ including the DEC/Test Manage r project team, vided va luable input to the requirements phase allow direct access to their QAR databases by of the DECjTesr Manager development cycle. aU users, both external fie ld test sires and inter­ SQM required all product groups ro submit nal users. Indeed, a QAR database had been regression rest subsets that could be executed used to gather fe edback about versions 1 .0 and under the DEC/Test Manager software. This 1. 1 from external field test sites and was request led nor just to requirements from the always ava ilable to internal users fo r reporting groups responsible for applications that run on bugs or submitting suggestions. Therefore, the the VMS operating system, but also to require­ DEC/Test Manager team was able to cull ments from the SQM rest administrators who product requirements from the interna l QAR needed test-control functionality to make their system root . task easier.

112 Digital Te chnical jo urnal No. 6 February I Y88 Software Productivity Tools

In summary, through the combined use of to be up to date on the current thinking for all these tools and improved information-gathering version 2.0 components but also assured the doc­ processes, the DEC/Test Manager team was able ument was continually reviewed , which helped to move into the specification stage within the the team complete the specification faster. first weeks of the project. The time savings was significant since several months was the generally Concurrent Prototyp ing and expected timeframe at this stage for a project of Ma intenance this scope and complexity. A conflict presented itself at this point in the development process. Each developer wanted to Sp ecification and Design Phase test various creative solutions, and each devel­ During the specification and design stage, devel­ oper needed to access any module in the system opers define what the system will do, how it will to quickly build prototypes. If such access were 1 be used, and how it will be implemented. This allowed, it was likely that too many people stage is often the longest in the development life would access the same module at the same time, cycle. Again the team sought ways to decreasethe causing skew. This problem was solved by the time required to complete the phase. They use of the VAX DECjCMS (Code Management Sys­ started by streamlining how the specification tem) software . itself would be documented. The specifications were made available at the Since the DECjTest Manager team did not know same time as a prototype version, which acted as what method they were going to use to test a "living" specification . The prototype included graphical applications on character cell termi­ some of the major interface changes, such as inte­ nals, they wanted to explore a number of differ­ gration with DECjCMS. Users were able to try out ent design possibilities. They also wanted to the proposed interfaces and comment on what ensure that as they refined their ideas the product they liked or did not like. In some cases, the specification document would be continually developers made available multiple solutions updated to reflect current specifications. They that provided the same underlying functionality did not want to write the specification at the end to determine which possible solution suited the of the phase when there would be a higher possi­ users best. This approach enabled the team to bility of inadvertently leaving out details. As a make decisions on functional ity based on users' result, they developed procedures that would feedback early in the development process. The allow for continual update of the specification developers avoided the much longer process of and also enable them to prototype a number of first developing one solution, waiting for feed­ alternative implementations. back, issuing a modified solution, and continuing Next the version 2.0 functionality was divided the steps until results are attained. into major components, and each component was Complicating the prototype effort was the assigned to a developer for specification. A lan­ need to continue to maintain the released ver­ guage-sensitive editor, VAX LSE, template was sion 1.1. DECjCMS met the developers' need to written both for specifications and designs to control multiple simultaneous development enforce a convention for information. As first threads. DECjCMS maintains "elements," which drafts of specifications were completed, the team are all of the versions of a single file stored as met to review the specifications and suggest deltas from the original version. Each particular changes. The changes were incorporated into the version ofthe element file is called a "generation." documents, and the next pass of the complete A generation from which another generation is specification was built. (The main specification derived is called an "ancestor" generation. The document was a Digital Standard Runoff proce­ trail from a generation through all of its ancestors dure with Include files for each of the compo­ back to the first generation of the element is nent files the individual developers wrote. It was called a "line of descent." Al l elements have a a simple matter of processing to generate the lat­ main line of descent, which consists of genera­ est specification, and as a result, processing was tion 1, followed by generation 2, and so on. In done frequently during this stage.) addition, DECjCMS allows lines of descent to Finally, each new specification was made avail­ branch off from the main line into what are able in a public directory for internal review. called "variant" lines. These variant lines of This availability not only enabled the entire team descent exist in parallel to each other and to the

Digital Technicaljo urnal 113 No. 6 February 1988 Project Management of the VA X DECjTest Manager Software Version 2. 0

main line. Changes made to one variant, fo r A single executable ve rsion was built from the example, are not reflected in the other lines of merged protorypes and made available through­ descent unless the changes are explicitly merged ou t Digital on the DECnet network. Notification across lines. of the availabiliry was posted in NOTES and was Before any prototype work was begun, the sent by VAX Mail directly to all known installed DECjTest Manager team agreed that all nonpro­ sites. duction code would be replaced into the group A single executable image was used because code library as variants, leaving the main line for the team fe lt that a user was more likely to try the maintenance and production version 2.0 devel­ various prototypes if only one image had to be opment. used. CMS classes - sets containing one specific Feedback on the protorypes was gathered from generation of each elemenr in the set - were NOTES and Mai I. The team directly polled used to represent complete prototypes, such as installed sites, asking about the specific ques­ CMS_INTEGRATION and RESUBMIT. A developer tions the protarypes had been designed tO wo rking on a particular prototype could then answer. retrieve modules by telling CMS that he wanted During the design phase, thousands of CMS the latest generation on the same line of descent merges were completed without problem. With­ as the generation in the particular class: out this merge capability, it wou ld have been impossible to allow the developers access to all CMS FETCH module.bl i II II modules at the same time. Without concurrent /GENERATION•cms_ integration• access tO all modules, the developers could not These classes were also used with DECjMMS have built as many prototypes as quickly as they (Module Management System) software to per­ did. Further, without these early protorypes, the form builds. DECjMMS works from a description participation of internal users wou ld not have of objects to be built and their dependencies on been as high , and version 2.0 then would not other objects. The tOol searches CMS libraries for have been bu ilt in the required timeframe. components and makes use of CMS classes if told to do so. DECjT est Manager and Performance AJI individual protarypes were also included and Coverage Analyzer Integratio n in a more global class , PROTOTYPE, to allow Testing was begun early in the development cycle the team to construct a single, executable proto­ to find and fi x problems as early as possible. type version containing all current prototype The existing DEC/Te st Manager test set was run efforts. In some cases, the same module had against the prototype image to check for regres­ been mod ified differently for several of the indi­ sions in nonprotatype areas of the code. Some vidual prototypes, and these modifications had of the more comprehensive protorypes tOuched to be reconciled for the combined prorotype. many modules; therefore , it was important DECjCMS RESERVE/MERGE fu nctionality was of to ensure preexisting fu nctionaliry remained major assistance. This procedure combines two unchanged . In addition, some new tests were generations from diffe rent lines of descent, auto­ written for the prototypes, the purpose of which matically including independent changes and was not tO test the protarype for correctness. flagging different changes to the same region of Rather these tests could be run with the VAX Per­ the fi le (called merge "confl icts") for human formance and Coverage Analyzer (PCA) tO evalu­ resolution . ate the relative performance of different proto­ Once CMS performed its merge and the merge types and also to help tune the design. conflicts were resolved, the team used CMS's abil­ The DEC/Te st Manager and the PCA can be ity tO compare a fi le against generations stored in used together in an integrated fashion. This inte­ a CMS library. This comparison was made to gration allows programmers to use good-cover­ ensure the merged fi le contained code that made age regression test suites as performance tests, sense and that was expected tO be there. This merely by changing PCA collection from cover­ step was crucial to avoid accidental loss of code age information tO performance statistics. For during the conflict-resolution process and tO the DEC/Te st Manager version 2.0 development, ensure that the automatically merged code sec­ each developer was required not only tO create tions were still valid BLISS. and run the regression test suite but also to do

114 Digital Technical jo urnal No. 6 Fe bruary 1988 Software Productivity Tools

code-path coverage analysis on his code before it development, a procedure was run first to was checked back into the master source library. determine if code was checked back into the The team was striving to have 90 percent of the master CMS source library, and second to start code paths covered, and PCA allowed us to check a system build if it was warranted . As a result how we were doing throughout development. of this procedure, all recent code was always The same tests were used to gather data on the available to developers as they were writing poorly performing commands so developers new modules. Problems were found early rather could identify which routinesjmodules needed than at a later, larger integration at the end of to be looked into. a base lcve I. While analyzing the prototype performance, Also, as part of the build procedure, an auto­ the developers ran PCA over the rest of the code matic test set execution was done. Just as the to identify the sources of several known perfor­ incremental integrations had done, these test sets mance problems in version I. I. The results from kept the team aware of problems with the system. this performance analysis led to the redesign of Moreover, the test sets were always available so several key modules. The redesign produced no implementors could easily schedule time for test user-visible fu nctional changes but significantly review. improved the performance of the commands We were able to detect and fix bugs early, most often used. rather than have bugs mushroom into larger prob­ lems as more code was added. Problems were Implementation usually identified and solved while the code was Once the feedback from the prototypes and still fresh in the developers' minds; therefore, design reviews was incorporated into the designs time for bug fixes was reduced. and the specification and design phase was The use of the DECjTest Manager to test itself closed, implementation of DECjTest Manager during development was also beneficial as an version 2.0 was begun. early problem-detection system. In addition, a The effort was approached in three diffe rent number of early versions of the software were ways . First, some of the prototype code was good released to internal Digital users. These users enough to be kept, with only a little time spent to helped to identify problems early, during the make it production quality in the handling of implementation phase, when problems are error cases. Then, the variant generations in the often easiest and least expensive to fix and CMS library were merged back onto the main line have the least impact on the project schedule. of descent - now the version 2.0 development Because these early ve rsions were used and stream. refined for weeks internally, serious problems Second , the deve lopers took as much relevant never reached the customer field test sites. existing code as possible from other Digital For example, internal user feedback indicated projects. This existing code was modified and that some changes would be required in the reused in the DECjTest Manager version 2.0. new functionality. This valuable feedback caused Modification of the existing code for the Test a complete interface change; however, the work Manager environment proceeded far more was completed before external field test was quickly than writing new code. Included among begun. the code the team reused was the pseudoterminal During implementation, NOTES and a QARsys­ driver from an internal tool called PTYCON-32. tem were used for problem reporting and track­ Also adopted and upgraded to handle VT 200- ing. This reporting enabled the developers to series terminals was the EDT group's terminal associate problems with sections of code, to simulator fo r building in-memory screen pictures report progress, and to associate it with a test for VTS 2 and VT I 00 terminals. in the DECjTest Manager library. A project Third, new code was written for all remaining rule req uired that all bug fixes have a test in functionality. No new code was considered the library and a PCA coverage analysis per­ complete until a code review was held. The formed before the bug was considered fixed. As DECjTe st Manager team then adhered to the a resu lt of enforcing this procedure, the DEC/ software engineering principle of frequent, Test Manager team achieved a code coverage of small integrations rather than one large integra­ 87 percent, finding two thirds of the problems tion. Therefore, every evening during active themselves.

Digital Technicaijournai liS No. 6 February 1988 Project Management of the VA X DECjTest Manager Software Version 2. 0

Conclusion References

The DECjTest Manager team was able to produce 1. B. Beandcr, "VAXjVMS Software Develop­ 30,000 lines of prototype code (thrown away) ment Environment," Digital Te chnicaljour­ and 60,000 lines of tested, production-quality nal (February l 988, this issue) : l 0-19. code in under seven months as a result of several key factOrs: creative use of project procedures 2. A. Duncan and T Harris, "Software Produc­ and tools, team commitment w these procedures, tivity Measurements," Digital Te chnical and the use of the developing product by the jo urnal (February 1988, this issue) : 20-27 . team as well as many internal users. It has been 3. P. Gilbert, "Development of the VAX NOTES two years since DECjTest Manager version 2.0 System," Digital Te chnicaljournal (Febru­ became available to customers, and fewer than ary 1988, this issue) : 117-124. six unique problems have been reported by the customer base.

116 Digital Technicaljo urnal No. 6 February 1988 Peter D Gilbert I

Development of th e VAX NO TE S Sy stem

The VAX NO TES computer conferencing system is a communications tool fo r on-line discussions. This paper discusses the innovative strategies devised bythe VAX NO TES team fo r the development of this system, includ­ ing the decisions to builda prototype, to perfo nnhuman fa ctors engineer­ ing, and to include a technical writer early in the development cy cle. Also described in this paper are several keyproduct fe atures, with emphasis on their effe ct on product performance and extensibility: the multitasking, multithreaded server; the user interface; the underlying storage medium; and the callableinterf ace.

Computer conferencing is a sofrware tool for its roots in the PI.ATO system developed at the ongoing discussions among individuals. Users University of Illinois. The K-NOTES program was can asynchronously read and write messages in written as a test of the newly added indexed the conference at times suitable to themselves. file support in the VAX PL/1 language. VMS soft­ The computer conference provides an organized ware engineers also used K-NOTES to log VMS structure and a permanent record of users' design changes that might be of interest to oth­ messages, or discussions. Computer conferenc­ ers. K-NOTES was a crude but effective tool for ing is a viable substitute for conventional meet­ communication and became popular with other ings, offering conspicuous savings in space, time, engineering groups within Digital. Indeed, it was and money. often mistaken for part of the VMS operating The VA X NOTES system is a computer confer­ system. encing tool designed for use on the VA XfVMS Sofrware developers using PDP-I I systems also operating system. The development effort was wanted access to computer conferences. A soft­ successful, meeting its requirements and sched­ ware deve loper wrote the NOTES- 11 program as ule and incorporating several innovations. The a "midnight project" - an unfunded project VAX NOTES system is used by Digital's hardware developed outside normal working hours. Origi­ and sofrware engineers for structured project nally developed for the RSTS/E operating system, communication and has found similar favor with NOTES- II was later adapted to run on the RSX, customers. VMS, and P;os operating systems. Many of the The success of the VAX NOTES system is largely enhancements suggested for K-NOTES were due to the design and development strategies incorporated into NOTES- II, including support used in its creation, and to the context in which it for multiple time zones and a per-user record of was developed. This paper discusses the rationale conferences (later called the "notebook" ). behind these decisions and their effect on the As Digital's engineering nerwork grew to hun­ product. This retrospective may be useful to dreds of computers connected by DECnet soft­ other sofrware developers. ware, the number of computer conferences First, we briefly discuss the origins of com­ abounded. Most products had a conference fo r puter conferencing at Digital. Digital users to ask questions, report possible problems, and suggest enhancements. There A BriefHisto ryof NO TES at Digital were also conferences on a dive rsity of other sub­ Early in I980, a Digital engineer wrote a com­ jects of general and personal interest, such as puter conferencing program, K-NOTES, that had security, smoking policy, and jobs.

Digital Te chnicaljou null 117 No. 6 February I 988 Development of the VA X NOTES Sy stem

VAX NO TES Project Decisions Our development tools included VAX DEC/ In late I 984, VAX NOTES became a fu nded pro­ CMS (Code .Management System) software for ject with two developers and a one-year develop­ source control and VAX DECjMMS (Module Man­ ment schedule. agement System) software for doing incremental We looked at several competitive conferencing builds. In addition, the NOTES utHity itself systems. Nearly all of these systems were oriented became an important development tool . toward hardcopy terminals. Most included an Because of limited resources and time, we integrated electronic mail system or personal made no systematic test suite. We correctly messaging capability, and most had good basic assumed that an internalfi eld test with thousands documentation. We fo und that in many of these of users wou ld provide an incredible amount of systems, navigation through the messages in a testing. We discuss our test results in the section conference was difficult. Field Test at the end of this paper. None of the systems provided good support over a computer network, none integrated well Development Decisions with the underlying system, and none offered a This section describes three elements of the deve­ callable set of routines for accessing conferences. lopment strategy used for the VAX NOTES pro­ Based on our study, we decided our product ject. Each ofthese - the prototype, usability test­ should offer users the fol lowing: ing, and early inclusion of a technical writer -

• A screen-oriented terminal interface is a somewhat novel approach and led to the pro­ ject's success. • An interface to the VMS Mail utility

• An easily conceptualized structure with sim­ Reasonsfo r Building a Prototyp e ple navigation (A comb-like structure of topics Several of our product decisions conspired tO and replies was chosen.) make a protOtype very desirable. First, our un­ • A distribu ted conferencing system that makes tried uses of the new VAX TPU utility imposed 1'2 efficient use of DECnet capabil ities some risks. Second , we had decided to perform human-factors testing; our short development • Simple, introductory documentation for non­ cyc le required that some form of the product be technical and novice users available early for this testing. Finally, a proto­ A serve r-based technology layered on DECnet type wou ld give the deve lopers more experience software was clearly needed to efficiently sup­ with serve r-based applications and with TPU. port distributed conferencing. In short, we expected to learn from the proto­ During the planning of the project, we also type. (In addition to meeting our project needs, determined that the product shou ld be extensi­ the prototype could be used within Digital ble, which is discussed in the section Extensible to provide better access to remote conferences Design through the Callable Interface . In add i­ while the VA.'( NOTES system was being devel­ tion we decided to use the VAX TPU (Text Pro­ oped .) cessing Utility) software to implement the user We only required that the protOtype wou ld interfa ce, and VAX RMS (Record Management support core fe atures . The prototype wou ld use Services) software for the underlying storage the same conference srorage fo rmat as was used mechanism. Discussion of the use of a server, the by NOTES- II. Therefore, fe atures that required user interface, the srorage features, and the changes in the storage format (such as keywords callable interface can be found in the section and membership lists) were necessarily absent Design of Main Features. from the prototype. Also, some rarely used opera­ These product decisions were coupled with tions were left unimplemented in the prototype decisions abom how tO structure the develop­ since NOTES- I 1 could be used instead. Indeed, ment process. We decided tO build a protOtype. the prototype was incapable of even creating a Also, we worked with the Software Usability new conference. Engineering (SUE) Group to perform human-fac­ We decided to include in the prototype one 3 tOrs testing. Finally, we included a technical additional fe ature called the notebook. The writer early in the developmem cycle. These notebook is a per-user record of the conferences decisions are discussed in the next section. in which the user participates and of the notes

118 Digital Technical journal No. (, February 1988 Software Productivity Tools

that have been read in those conferences. From on a benchmark task in their first 30 minutes of our experience with the notebook feature in using VAX NOTES; 8 to 10 successes was consid­ NOTES- 11, we anticipated difficulties in provid­ ered the best case. The learning rate was mea­ ing a smooth integration berween the notebook sured by comparing the performance on a second and the rest of the VAX NOTES interface. The benchmark with the first, where the performance notebook fe ature was included in the prototype was measured as a work speed re lative to a prac­ so that human-factors tests could help us resolve ticed expert performing the same tasks. The goal any problems. for initial satisfaction was fairly positive, 1.0 on a scale from -3.0 to 3.0 Us ability Engineering of NOTES In the first VAX NOTES tests, users exceeded We enlisted Digital's SUE Group tO help evaluate our best-case level for initial use, and so we and improve the VAXNOTES system's ease of use. adjusted our goal. The learning rate was accept­ The first task was to create a measurable defini­ able, but the evaluation score fe ll just short of our tion of usabiliry. This definition allowed us to goal. We then made changes tO the interface as a measure and improve the VAX NOTES user inter­ resu lt of the SUE Group's impact analysis. With face. The SUE group helped us identify our goals these changes, VAX NOTES eventually met or and offered guidance to help us reach them. The exceeded our usability goals. usability engineering approach tO sofrware devel­ opment is described in the paper "Software Inclusion of a Te chnical Writer 3 Usability Engineering," this issue. We were able to include a technical writer on the The VAX NOTES usabiliry definition included development team shortly after the protorype measurements for 10 attributes. These are was completed, rather than after implementa­ tion as is often the case. The writer's review and • Initial use exposition of the planned interface was concur­ rent with (and sometimes preceded) its imple­ • Learning rate mentation. The writer's most important role and • Infrequent use one of our goals was to ensure that the system cou ld be clearly documented for nontechnical • Compatibiliry with NOTES- 1 1 users.

• Compatibiliry with other competitive confer­ As a member of the development team, the encing systems writer made technical changes tO simplify fea­ tures that were difficult to explain, made the set • Initial evaluation of commands more self-consistent, and suggested other improvements to the implementors. • Casual evaluation Because the technical writer was involved in • Mastery evaluation the process early, problems with either the code or the documentation could be resolved early, • Error recovery and a combination of both code and documenta­ tion changes could be used in the solution. • Fear of feeling foolish The early involvement of a technical writer also The first three attributes were measured by user gave us high-qualiry documentation (and on-line performance on NOTES benchmark tasks. For he lp) for field test and some of the human-factors these, the metrics were the number of errors and studies, thereby allowing the whole system to be the number of successful interactions made by evaluated. the users in 30 minutes. The compatibility and evaluation attributes were measured with atti­ Design of Main Fe atures tude questionnaires containing a semantic differ­ In this section we discuss the reasons for includ­ ential. The final rwo attributes were measured ing a server, creating the user interface with with unique questionnaires. TPU, selecting the VAX RMS sofrware as a stor­ The testing centered on the initial use, learn­ age medium, and designing a callable interface ing rate , and initial evaluation attributes. that is extensible. We also discuss how these For initial performance, the goal was for initial features were implemented and what trade-offs users to have three or four successfu l interactions were made.

Digital Tecbnical]ourna/ 119 No . 6 Fe bruary 1988 Development of the VAX NO TES Sy stem

Us e of a Server requests; the code removes a control block from The K-NOTES and NOTES-1 1 tools relied on RMS the queue, performs the operation , and sends and the DECnet software to transparently access back the results. [f there are more results than conferences on remote systems. Although the can be returned in a single packet, a continuation VAX RMS software provides efficient access to control block is enqueued, and later activated by individual records in a remote file, most NOTES receipt of a "send me more" request from the operations require multiple RMS operations. For user. example, to list a directory of the notes in a con­ The prototype processed each request either ference would require one RMS operation per to completion or until the results filled the note. This performance is efficient when the con­ server's response buffer. A disadvantage with this ference is stored on the same system that the user approach is that some requests may take a consid­ is on. However, when the conference is several erable amount of time before returning any slow network links away from the user, the results. For example, a request to search for a round-trip delays for every RMS record operation string within a note could cause significant are seen as frustrating delays at the user interface. delays for other users of the server. This problem The advantages of using a NOTES server running was solved by making the server multithreaded. on the system that hosts the conference were The mu ltithreaded VA X NOTES server can clear. The server would switch from one task to another even before the first task has completed. The implementation • Perform many local RMS operations without a proved to be far easier than we had anticipated, network delay and the multithreaded server simplified the han­

• Support sophisticated requests more effi ­ dling of users' "send me more" requests. Each ciently than RMS "thread of execution" has a separate stack and set of registers. Whenever the server does an I/0 • Send back only the information that the user operation, it does so without waiting and tells the requested "scheduler" to find and work on another request. When the r;o completes, the corresponding • Allow DECnet software to send larger, hence, request block is again enqueued for processing. fewer network packets with fewer transmis­ Because I/0 is invariably done for a request, sions, resulting in a tenfold improvement in there is no need for a sophisticated scheduler. the time taken to satisfy users' requests The "scheduler" saves the registers on that A multitasking server can handle requests from thread's stack, searches for a thread that can pro­ one or more users. Because more requests can be ceed, restores irs registers, and continues the handled, fewer server processes, hence fewer execution of that new thread. Since the VAX resources, are needed on the system hosting con­ NOTES system does relatively little processing fe rences. Also, VAX RMS software pools buffers, between ljO operations, no user is able to signifi­ which offers another advantage when several cantly affect the response time seen by other users are reading the same conference: a user's users. request may be satisfied from a buffer that had been read for a diffe rent user's earlier request . Th e VA XTPU Programmable Editor Multitasking gives the server process some­ VAXTPU software is a programming language and thing to do between an individual user's requests. interpreter that is shipped with the VMS operat­ For example, while a user is reading a note, the ing system. TPU is especially suited for writing server can process other requests. The multitask­ text editOrs and has been used to write the ing usually does not have much effect on the human-engineered EVE editOr, the ACL (Access response time seen by the users. Control List) editor, and VAX LSE (Language-Sen­ The prototype server employed multitasking. sitive EditOr) . The VAX NOTES server used both mu ltitasking Although the VAX NOTES system is not an edi­ and multithreading. tor, we wanted an interactive screen-oriented When the multitasking server receives a user interface . We expected that VA XTPU would request, it builds a control block tO contain the provide a good language in which to write the request and queues it onto a list. The syn­ interface, since the functions for handling the chronous code is in a loop that processes the screen, key definitions, strings, and text manipu-

120 Digital Te chnicaljo urnal No. 6 Fe brumy 1988 Software Productivity Tools

lation are built in to the TPU language. The The use of the VAX TPU software was not with­ VAX TPU utility also provides a callable interface, out pitfalls. The VAX NOTES team made the most so that it can call and be called by programs writ­ sophisticated use of TPU software to date and dis­ ten in other languages. By using these features of covered several interesting bugs in the TPU soft­ the VAX TPU language, we could economically ware. Because the development groups were in produce a desirable user interface. close proximity to each other, little time was lost However, TPU's callable nature had not yet wa iting for fixes or workarounds. But not all of been used in anything as complicated as the VAX the problems were resolved; for example, TPU's NOTES product, and we were unsure whether its controi-C handling was designed for simple edi­ performance as an interpretive language wou ld tors, not complex applications, and controi-C adversely affect the perceived performance. The handling remained a weak point in the first prototype helped resolve this issue early, and the release of VAX NOTES. performance proved to be far better than we needed. VA X RMSSo ftware fo r Storage The decision to use TPU had a considerable and We had several good reasons for choosing the unexpected payoff: because of its interactive and VAX RMS software for our underlying storage interpretive nature, it was possible to greatly medium. reduce the time spent in the compile-link-run • VAX RMS software is bundled with the VMS cycle. After the initial investment in getting operating system. NOTES and TPU "talking" together, typical devel­ opment ofthe TPU code followed this procedure: • The code is robust.

• Run the VAX NOTES utility. • VAX RMS performs we ll on large indexed fi les. (We wa nted an entire conference to be held • Read the TPU code into a text buffer (by a within a single file.) command issued at the NOTES prompt) . • RMS supportS concurrent, coordinated access • Edit the code. and updating of records.

• Select and recompile the modified procedures • We were familiar with the VAX RMS software, in the code . This was done with only a few therefore we wou ld not need to spend time keystrokes. Developers had defined a key to gaining experience with it. mean "compile the selected code ." Although these reasons were compelling, we • Exit the buffer; return to the NOTES prompt. considered some alternatives because of the fo l­ lowing issues. The NOTES software's view of the • Te st the change . info rmation in a conference wou ld be more akin This procedure continued until the developer to a true database than to an indexed fi le. There­ was happy with the changes. Then the buffer fore, the use of indexed fi les would require that containing the TPU code wou ld be written out to development time be spent implementing a a file, and the developer could either exit the makeshift database atop indexed files. Also, the NOTES utility or proceed with other changes. VAX NOTES system would be enhanced if it and The power of this approach resulted in an esti­ other tools used a relational or object database . mated development-time savings of four person­ Despite these considerations, we decided in months. fa vor of RMS based on the benefits of using RMS We discovered, however, a disadvantage with listed above, our interest in not adding licensing this evolutionary approach. The resulting TPU costs for customers, and a tight schedule. code tended to be poorly commented because We did, however, decide to insulate the higher the code wou ld be working before there was any functions in NOTES from RMS with a callable need for comments. We solved this problem with interface. This permits a fu ture release of the an occasional "revolution": we reviewed the VAX NOTES system tO use a diffe rent underlying code as a whole, improved the organization, storage medium without affecting code that uses made it more systematic, and added the com­ the callable interface. ments necessary to understand and maintain the The VAX NOTES system stOres data in the RMS code. records with a type-length-value (TLV) encod-

Digital Te chnicaljournal 121 No. 6 Fe bruary 1988 Development of the VAX NOTES System

ing called Digital Data Interchange Syntax (Two interfaces used within Digital show the (DDIS) . Each datum in a record identifies itself value of this approach. One is designed for off­ by a "type" code and is fol lowed by the length line - batch - reading and writing of notes; and actual value of the datum. For example, the the other is an interactive interface imple­ "header" information in each note in a confer­ mented with another programmable editor - ence may include the author's name, the date the EMACS.) note was written, a title, and a list of keywords. A • Have the flexibility to be extended to handle sample note header with TLV encoding fol lows: a variety of possible (and now unforeseen)

{author} 5 "BYRNE" future enhancements

{date} 8 "22-JAN-1979 10:30:27" • Provide a we ll-defi ned interface between the {title} 15 "Crys tallography" two halves of VA X NOTES: the user interface and the fi le access routines The {author}, {date}, and {title} are suitably encoded, and 8 bytes are used to represent the • Map easily to remote procedure calls or net­ date (as on VMS) . work packet transfers The advantages of TLV encoding are that it is • Be reentrant, that is, the "context" of an opera­ succinct and evolutionary. Ad ditional informa­ tion must be specified in each call, and opera­ tion (such as coauthors or an edit history) can tions done with different contexts should be added without changing the format or invali­ be independent of each other (This indepen­ dating any existing data. Changes are made by dence also proved to be an advantage for simply creating a new type code for the new implementation of the multithreaded, multi­ information, making minor changes within the tasking NOTES server.) callable routines, and providing the new informa­

tion to the callable interface. • Be capable of becoming a supported interface for use by customers Extensible Design through the Callable Interfa ce The callable interfa ce proved quite successful in meeting these goals. Computer-conferencing has a practically unlim­ Al l the routines in the callable interface have ited range of possible uses and usefu l exten­ 4 the same format: sions. We could not foresee all possible future enhancements, or provide even a tenth of the 5tatus • NOTESSobject_o p eratlon many requested features because of our develop­ (context , inp uts, outputs) ment schedule. Resigning ourselves to the fact The ob ject is either NOTEFILE, CLASS, that we could not know what our code would ENTRY, KEYWORD, NOTE, PROFILE, or USER. eventually become, we resolved to provide The operations BEGIN, END, ADD, DELETE, extensibility that could support a variety of GET, and MODIFY are common to most of these future directions. objects. A few additional routines handle lists One of the design decisions allowing for this contained within an object, for example, extensibility is described in this section. NOTES S NOTE_GET_TEXT gets the next line of The design of the callable interface was a sig­ text from the note most recently accessed (with nificant aspect of the VAX NOTES system devel­ the specified context by NOTESSNOTE_GET. opment. This interface provides all the opera­ The context parameter is an uninter­ tions that access conferences and notebooks. The preted value defined by the NOTES facility. On callable interface must meet several goals; it a NOTESSobjecLBEGIN call, NOTES stores a must nonzero value in the context parameter. That context is passed to other routines for access • Allow the underlying srorage medium to be changed in a future implementation; for exam­ ro that kind of object. A call to NOTES SobjecL.END ple, from RMS fi les ro object databases frees the resources used to maintain the context and zeroes the value. • Al low other user interfaces to be added with­ The inpu ts and outputs are item lists. An out publishing (and compromising) the item list is a list of one or more item descriptors, underlying storage format each of which specifies an item code. The item

122 Digital Technical journal No. 6 February I '.>88 Software Productivity Tools

list is terminated by an item code of 0. Figure 1 interface), the storage formats are actually the depicts the structure of a single item descriptor. same. Because the formats are the same, the vari­ ety in the code is reduced, and the inner routines can be reused (and stressed) in several differ ent ways, making them more robust. This similar­ BUFFER LENGTH ITEM CODE ity offers another advantage. A future release I could easily allow personal notes or annotations BUFFER ADDRESS to be stored in the user's notebook, through RETURN LENGTH ADDRESS the use of the existing NOTESS NOTE_operation routines. Access to remote conferences is easily effected. Figure 1 Structure of an Item Descriptor An operation code (for the routine) , the context, the inputs, and the requested outputs are "lin­ The item code specifies the item of informa­ earized" into a TLV format (DDIS) and sent to the tion that the caller is specifying (through the NOTES server on the remote system. The routine input !I parameter) or requesting (via the is called, and the returned status and outputs are ou tput !I parameter) . The buffer length and "linearized" and sent back. This can be viewed as buffer address describe the buffer (for inputs or a set of specialized Remote Procedure Calls outputs), and the return length address is the (RPC) . address of a location into which NOTES will The inputs to NOTESSNOTE_GET may also write the actual length of the requested informa­ include "hints" to indicate, for example, tion (for outputs) . whether the text of the note is desired. (The To see how this works, we consider the calls user may want to read the text or may simply needed to read notes 1 through 5 from a local want a directory of the notes.) If the operation conference . The name of a file is specified is one that may be repeuuve (such as as an input to NOTESSNOTEFILLBEGIN, NOTESSNOTE_GET) , the server makes multiple which opens the file and initializes the note­ calls to that routine so that it can buffer and send fi le context. The notefile context is passed back larger packets of information. There are as an input to NOTESS NOTE_BEGIN, and this some complications in the handling of signaled establishes the note context. A call to exceptions and buffering, and in how the server NOTESSNOTE_GET requests notes 1 through 5. val idates the context, but they had little effect on (The string "1-5" is passed as an input.) Then the overall design. NOTESS NOTE_GET_TEXT is repeatedly called until a "no more text" status is returned. The pro­ Field Te st cess repeats: another call to NOTESSNOTE_GET The internal field test was impressive. Within an is made (specifying a "continue" item code) . hour of making the VA X NOTES system available Eventually NOTESSNOTE_GET returns the status within Digital, it was installed on four conti­ "no more notes." nents. Besides providing a popular tool on hun­ At any time, the same note context could dreds of systems on Digital's engineering net­ be used to read other notes by omitting the work, we also provided the medium - a VAX continue item code. This would cancel reading NOTES conference - by which users could eas­ of the current stream of notes (I through 5, ily report problems and make suggestions. We in the example) before handling the new were deluged. request. This cancellation can be avoided by cre­ These reports directly contributed to the qual­ ating another note context, with another call ity and success of the VAX NOTES system. to NOTES S NOTLBEG IN. The resources used to maintain the note context are freed by Acknowledgments a call to NOTESS NOTE_END, and similarly Thanks are due the editors and reviewers of this with NOTESSNOTEFILE_END, which deallocates paper, and the many people whose interest memoryand closes the fi le. in computer conferencing influenced the VAX There is no separate call for opening a note­ NOTES system, especially Mark Goodrich, Len book. Although conferences and notebooks Kawell, Valerie Rogers, Benn Schreiber, and contain different kinds of information (in our Tom Spine.

Digital Technical journal 123 No. 6 Fe brumy 1988 Development of the VA X NO TES Sy stem

References

1. DECnet Digital Ne twork Architecture (Phase IV) General Description (Bedford : Digital Equipment Corporation, Order No. AA- 149A-TC, 1982).

2. P. Beck and ]. Krycka, "The DECnet-VAX

Product - An Integrated Approach to Net­ working," Digital Te chnicaljournal (Sep­ tember 1986) : 88-99.

3. M. Good , "Software Usability Engineering ," Digital Te chnical jo umal (February 1 988, this issue) : 125- 133 .

4. S. Hil tz, The Ne twork Na tion: Hu man Communication via Computer (Reading: Ad dison-Wesley, 1978).

124 Digital Tecbnicaljournal No. 6 Fe bruary I ')88 Michael D. Good I

Software Us ability Engineering

Us ability is an increasingly important competitive issue in the software industry. Software usability engineering is a structured approach to building software systems that meet the needs of users in various envi­ ronments with varying levels of computer experience. This app roach emphasizes observation of people using software systems to learn what people want and need from software systems. The three principal activi­ ties of software usability engineering are on-site observations of and interviews with system users, usability sp ecification development, and evolutionary delivery of the system. These activities are parallel steps in the development cycle.

Computer system designers have not always The Software UsabilityEngineering adopted a user-centered perspective on software Process design. Instead, many designers resolved design The role of engineering is to apply scientific questions about the human-computer interface knowledge to produce working systems that are by using introspective criteria such as personal economically devised and fu lfill specific needs. preference or conceptual appeal. Our software usability group has adapted engi­ This introspective approach to user-interface neering techniques to the design of user inter­ design might produce a usable system when faces. To understand user needs, engineers must software engineers represent actual users. How­ observe people while they are actually using ever, computer systems today are being built computer systems and collect data from them on for a wide range of people whose needs often system usability. Observation and data collection have little in common with the needs of system can be approached in the following ways: designers. In response to market demand for systems that • Visiting people while they use computers in satisfy a growing and varied user community, the workplace usability is becoming an increasingly important competitive issue. Designers are striving to cre­ • Inviting people to test prototypes or partici­ ate computer systems that people can use easily, pate in usability evaluations at the engineering quickly, and enjoyably. Indicative of this trend is site increased membership since 1982 in profes­ • Soliciting feedback on early versions of sys­ sional groups such as the Association for Comput­ tems under development ing Machinery's Special Interest Group on Com­ puter-Human Interaction (ACM SIGCHI) and the • Providing users with instrumented systems Computer Systems Group of the Human Factors that record usage statistics Society. Digital's Software Usability Engineering Group Our group uses these methods to gather infor­ believes that engineers must learn about the mation directly from users, not through second­ needs and preferences of actual users and should hand reports. We use these methods to study the build systems to accommodate them. With an usability of current versions of our products, understanding of customer environments, an competitive systems, prototypes of new systems, awareness of technological possibilities, and and manual paper-based systems. imagination, we have produced many ideas for Our software usability engineering process products that meet users' needs. evolves as we use it in product development. As

Digital Tecbnlcal]ournal 125 No. 6 Fe bruary 1988 Software Usability Engineering

of 1987. the process consists of three principal their work, about the details of their system inter­ activities: faces, and about their perception of various aspects of the system. The user and the engineer • Visiting customers to understand their needs. work together to reveal how the user experiences By understanding a customer's current experi­ the system as it is being used. These visits with ence with a system, we gain insight into our users are the best way for engineers to learn opportunities to engineer new and better sys­ about users' experiences with the system. tems. We collect data on users' experiences Ideally, the number of interviews conducted primarily through contextual interviews, that per product depends on how much data is being is, interviews conducted while users perform generated in each succeeding interview. The their work. interview process stops when new interviews no • Developing an operational usability speci­ longer reveal much new usability data. In prac­ fication for the system. We base the system tice, resource and time limitations may stop the specification on our understanding of users' interview process before this point. In any event, needs, competitive analysis, and the resources our approach is to stan with a small number of needed to produce the system. This specifica­ interviews (four or less) with people in various tion is a measurable definition of usability that jobs. We use these interviews to determine how is shared by all members of the project team. many and what type of users will be most useful for uncovering new usability data. • Ad opting an evolutionary delivery approach to system development. Developers start by Information Gained in Field Studies building a small subset of the system and then Contextual interviews reveal users' ongoing "growing" the system throughout the develop­ experience of a system. Other types of inter­ ment process. We continue to study users as views, which are not conducted while the user the system evolves. Evolutionary delivery is an works, reveal users' summary experience, that is, effective method for coping with changing experience as perceived after the fact. Data on requirements - a fundamental aspect of the ongoing experience provides a richer source of development process. ideas for interface design than data on summary These three development activities are parallel, experience. not sequential. We do not view user-interface For example, data collected from field studies design as a separate and initial pan of the devel­ has revealed the importance of interface transpar­ opment process but as an ongoing process in sys­ ency to users. A transparent interface allows the tem development. user to focus on the task rather than on the use of These usabilityengineering techniques apply the interface. Our understanding of transparency ro most software development environments and as a fundamental usability concept comes from an are most effective in improving software usability analysis of data on ongoing experience. when applied together. However, designers who Some interface techniques can help keep the use any single technique can improve a system's user in the flowof work, thus increasing interface usability. Our group has used this process in the transparency. One example can be drawn from a development of several of Digital's software prod­ workstation application for desktop publishing. ucts, including the EVE text editOr and VAXTPU Pop-up menus that appear at the current pointer (Text Processing Utility) software, VA X NOTES location create a flow of interaction that reduces software , MicroVMS workstation, VA X Software mouse movement and minimizes disruption to the Project Manager, VA X COBOL Generator soft­ user's task. Users do not have to move their eyes ware, VAX Language-Sensitive Editor, and VAX and hands to a static menu area to issue com­ DEC/CMS (Code Management System) software . mands, making this an effective interface feature for experienced users. Visiting Customers to Understand We will consider using pop-up menus in TheirNe eds new workstation software applications when we Data colJected at the user's workplace provides believe their use will keep the user in the flow of insight into what users need in both new and work. modified systems. During interviews of users We have developed our understanding of trans­ actually working with their systems, we ask about parency by observing people using a variety of

126 Digital Tecbnicaljournal No. 6 Fe bruary 1988 Software Productivity Tools

applications in diffe rent jobs. Transparency is an Whenever possible, we videotape interviews. aspect of usability that we find across many dif­ If users are unwilling to have their work video­ ferent contexts. In developing new products, it is taped, we audiotape the session while the second also important to consider the diversity of envi­ team member takes detailed notes to supplement ronments in which people will use the system. the taped information. The two team members Different users in different contexts have diffe r­ meet after the interview to reconstruct an accu­ ent usability needs. Some important aspects of rate record of events. user's context are Even without any taping or note-taking, engi­ neers can learn a great deal from user visits. • Type of work being performed Although the detail from the interview may not be remembered, the understanding gained dur­ • Physical wo rkplace environment ing the interview is still a valuable source of insight. • Interaction with other software systems Develop ing an Op erational Us ability • Social situation Sp ecification Studying users provides a rich, holistic under­ • Organizational culture standing of how people experience software systems. However, each person will have his All these aspects influence the usability of a or her own interpretation of user experience system for each individual. As with other prod­ as it relates to usability. Similarly, a team of ucts, software systems are used in the field in people working on a project wiJI find that each ways not anticipated by the designers. member has a diffe rent understanding of what Because the context in which a system is "usabi lity" means for that product. Keeping used is so important, we interview a variety of these understandings private and unarticu­ users who use particular productS to perform lated can have two undesirable results. First, different tasks. We look for common elements of team members work toward different and some­ usability for groups of people, as well as the times mutually exclusive goals. Second, the distinctive elements of usability for individual team does not have a shared criterion for what users. it means to succeed or fail in meeting users' needs. 2 Conducting Contextual Interviews Our group constructs shared, measurable Interviewers bring a focus, or background, 1 to definitions of usability in the form of operational their visitS with users. The focus determines what usability specifications. These specifications are is revealed and what remains hidden during a an extension of Deming's idea of operational defi­ 3 visit. The engineer needs to enter an interview nitions. We based our usability specifications on with a focus appropriate to achieve his goals. For the s stem auribute specifications described by l 5 example, in some visits an engineer may need to Gilb and Bennett. A usability specification, look for new product ideas; in others, the engi­ described in the following section, includes a list neer may need ideas to improve an existing of usability auributes crucial for product suc­ product. cess. Each attribute is associated with a measur­ To avoid losing data, interviewers should not ing method and a range of values that indicates try to extensively analyze their data during the success and failure. session. We use two-person teams, where one team member concentrates on the interview and Constructing a Us ability Sp ecification the second member records the data. Contextual The development of the VAX NOTES conferenc­ interviews rapidly generate large amounts of ing system provides an example of a usability 6 data. The data derives from an understanding of a specification Ta ble 1 is a summary of the usabil­ user's experience of a system, as shared by a user ity specification for the first version of the VAX and an interviewer. To generate such data, inter­ NOTES system. Five items are defined for each viewers need tO concentrate on their relation­ attribute: the measuring technique, the metric, ships with users and understand what users do the worst-case level, the planned level, and the during the session. best-case level.

Digital Tecbnical]ournal 127 No. 6 February 1988 Software Usability Engineering

Ta ble 1 Summary Usability Specification for VAX NOTES Version 1.0

Worst- Best- Usability Measuring Case Planned Case AHribute Technique Metric Level Level Level

Initial NOTES Number of 1-2 3-4 8-10 use benchmark successful task interactions in 30 minutes

Initial Attitude Evaluation 50 67 83 evaluation questionnaire score (0 to 1 00)

Error Critical- Percent 10% 50% 100% recovery incident incidents analysis "covered"

The measuring technique defines the method For error recovery, the metric was the percent­ used ro measure the attribute. Details of the age of incidents reported with the prototype measuring technique (not shown in Table 1) systems that would be "covered" (i.e., elimi­ accompany the brief description in the summary nated) by changes made in version 1 .0 of the VAX table. There are many diffe rent techniques fo r NOTES system. measuring usability attributes. We have usually The worst-case and planned levels define a measured usability attributes by asking users to range from fa ilure to meet minimum accept­ perform a standardized task in a laborarory set­ able requirements to meeting the sp ecification ting. We can then use this task as a benchmark for in fu ll . This range is an extension of Deming's comparing usability attribute levels of different single criterion value, which determines success systems. or failure. It is easier to specifya range of values In the VAX NOTES case, we chose ro measure than a single value for success and failure. initial use with a 14-item benchmark task that an Providing a range of values for several attributes expert VAX NOTES user could finish in three also makes it easier to manage trade-offs in levels minutes. Initial users were Digital employees of quality of different attributes. who had experience with the VMS operating sys­ The best-case leve l provides usefu I manage­ tem and the Digital Command Language but not ment information by estimating the state-of-the­ with conferencing systems. The users completed art level for an attribute. The best case is an their initial evaluations using 1 0-item Likert-style estimate of the best that could be achieved with questionnaires after they finished the benchmark this attribute, given enough resources. task. Error recovery was measured by a critical­ For the initial use of VAX NOTES software, incident analysis. In the analysis, we used ques­ we defined the planned leve l as experiencing tionnaires and interviews to collect information 3 or 4 successfu l interactions in the first half about costly errors (critical incidents) made by hour of use. We considered 1 or 2 successfu l users of the prorotype versions of the VAX NOTES interactions to be the minimum acceptable level, software . and 8 to 10 successful interactions to be the best The metric specifies how an attribute is that could be expected. In practice the actual expressed as a measurable quantity. Ta ble 1 leve l was 13 successful interactions, suggesting shows the definitions of the metrics in the VAX that we set the levels for this attribute too con­ NOTES specification. For the initial-use attri­ servatively. bute, the metric was the number of successful The planned level for initial evaluation (67) interactions in the first 30 minutes of the bench­ was fairly positive. Users' neutral feelings were mark task. For the initial-evaluation attribute, acceptable but negative feelings were not, so we we scored the questionnaire on a scale ranging set the worst case at 50. We set the best case at from 0 (strongly negative) ro 100 (strongly posi­ 83, which represented the highest scores we had tive) , with 50 representing a neutral evaluation. seen so fa r when using this questionnaire with

128 Digital Technicaljournal No. 6 February 1988 Software Productivity Tools

other products. The actual tested value was 67, Adop ting EvolutionaryDelivery matching the planned level. Changing requirements pose a challenge in user­ We planned an error-recovery level that could interface design as they do elsewhere in software cover 50 percent of the reported critical inci­ development. Brooks refers to changeability as dents. The worst-case level was set at a fairly low one of the essential difficulties of software engi­ 1 0 percent, whereas the best case wou ld be to neering - a problem that is part of the nature of 8 cover all of the reported critical incidents. In software engineering and that will not go away. practice, 72 percent of the critical incidents Evolutionary delivery exploits, · rather than were covered, exceeding the planned level. ignores, the changeable nature of software 4 Many usability specifications provide further requirements. This technique has been referred 8 detail by including "now" levels and references. to as incremental development and as iterative 9 Now levels represent current levels for an attri­ design. We believe that "iterative design" is usu­ bute, either for the current version of the product ally a redundant term in software design. Unless or for competitive products. References can be otherwise mandated by external sources, most 10 used to add more detail, such as describing how software design is already an iterative process. the levels were chosen, and to document the The waterfall model and similar models of soft­ usability specification. ware design are useful for managing project User needs and expectations are shaped in deliverables, but they do not describe what hap­ part by the marketplace; therefore competitive pens in software design and development. Evolu­ analyses can provide important data for usabil­ tionary delivery takes for granted the iterative ity specifications. We have constructed usability nature of the design process, rather than treating specifications that compare the system under iteration as an aberration from textbook methods. development to either the current market leader, Evolutionary delivery is the process of deliver­ the product with the most highly acclaimed user ing software in small, incremental stages. An ini­ interface in the market, or both. We can also tial prototype subset of the software is built and compare the systems by measuring usability on tested. New features are added and existing fea­ appropriate benchmark tasks. tures refined with successive versions of the sys­ tem. The prototype evolves into the finished Limitations of Us ability Sp ecifications product. Constructing a usability specification helps build Evolutionary delivery helps to build the project a shared understanding of usability among the team's shared understanding of the system's user­ diverse people working on a development project. interface design. Contemporary direct-manipula­ However, to achieve a shared understanding, trade­ tion user interfaces are too rich, dynamic, and offs have to be made. Usability specifications rep­ complex to be understood from paper specifica­ resent a constricted and incomplete definition of tions. Even simpler terminal-based interfaces are usability. The analytic definition of usability is too involved to be understood completely with­ necessarilyless completethananindividual'sholis­ out being seen in action. Early delivery of subset tic understanding based on observing people use systems helps everyone on the development team 7 systems Nonetheless, we deliberately trade off understand the system being designed, making it the holistic understanding for the analytic defini­ easier to build a shared vision of the final system. tion because the latter economically focuses our Early, incremental deliveries also demonstrate efforts on essential elements of product usability. project progress in a concrete form. Demonstrat­ If engineers do not understand the needs of ing improvements to the system at the user­ users before creating a specification, they risk interface level can be an important factor in developing a specification that does not reflect maintaining managerial support for a project and users' needs. As a result, the product that meets continuing availability of resources. its specification might still be unusable or com­ The techniques used to improve system usa­ mercially unsuccessful. Development teams must bility during the stages of evolutionary delivery continually evaluate usability specifications dur­ include the following: ing the development process and make the

changes necessary to reflect current information • Building and testing early prototypes on users' needs. This approach is part of evolu­ tionary delivery, described next. • Collecting user fe edback during early field test.

Digital Te chnicaljo urnal 129 No. 6 Fe bruary 1988 Software Usability Engineering

• Instrumenting a system to collecr usage data users tO complete a standardized task, where that task is the only one that can be completed • Analyzing the impact of design solutions using the prototype system. Special equipment These general-purpose techniques can be used can make it easier to conduct these rests and to independently of an overa ll usability engineer­ col lect more complete data, but is not necessary. ing process. They are described in the fo llowing For example, videOtaped records can help in sections, some with examples from the evolu­ later analyses, but as with user visits, we can 9 '' tionary delivery of the EVE text ediwr · learn much without them. For many years we tested prototypes in spare Building and Te sting Prototyp es offices, developers' offices, or users' offices. Our The first step in an evolutionary delivery pro­ group now rests most prototypes in ou r usability cess is building and resting protOtypes. These engineering laboratory, which is equipped with 12 prototypes effectively test for ease of learning computer hardware and software, a one-way mir­ and can provide the germinal product. Prowtyp· ror, and videotaping equipment. The laboratory ing also helps identify potential interface prob· resources provide greater opportunity for rou· lcms while still very early in the development tine testing and elaborate data collection. cycle. Collecting Us er Fe edback during Early From the point of view of usability engineer­ Field Te st ing, the first protOtype subset produced should facilitate usability testing. This typically means The earlier a system can be delivered to a group that the system of users fo r field test, the sooner valuable infor­ mation wi ll be available tO designers. User data • Includes only simple versions of the most collected in the field is usually a richer source important and most frequently used features of information than laboratory data collected of the product under controlled conditions. Field data takes into account the context in which the system is • Is able tO complete a simple benchmark usetl. task that the designer will use for a prelimi­ We use "field rest " to describe any version of nary evaluation of the system's usability software distributed to a group of people for use atrri bu tes in their work. This definition includes the distri­ bution of early subset versions as well as the • Is useful only for limited testing, not for nor· mal work later versions commonly referred tO as field-test software. Early field testing often begins by giv­ If the first prototype is actually useful fo r nor­ ing a usable subset system to users who under­ mal wo rk, it is probably a larger portion of the stand the status of the product and agree to use project than needs to be delivered at this stage. and evaluate it. The first prototype of the EVE text editor User visits, described previously, are a good was available three weeks after development way tO collect fi eld-test data. Another way to col­ bc::ga n. This prototype tested only the keypad lect user feedback is by electronic communica­ interfa ce. At that point, we had neither imple­ tion . Digi ta l's developers frequently use this mented nor fu lly designed rhe command-line effective method by making early field-test ver· features. To rest ease of learning, seven new sions available on Digital's private world-wide computer users used EVE in informal labora­ DECnet network and by encouraging user fe ed­ tory sessions. They performed a standard text· back through el.ecrronic mail or a VAX NOTES <:diring task. The rests showed that the keypad conference. interface was basically sound ; only minor Designers of the EVE text editOr and VAX TPU changes tO the basic EVE keypad commands software relied on user feedback by means of were required. This prototype was the first electronic communication throughout the devel­ of I 5 versions of EVE that users tested over opment cycle. Preliminary versions of the EVE 21 months. ed itor were available for daily work six months Because prototypes are nor suitable for daily before external fi eld test began. Overall, we use , they must be tested in controlled condi· received 362 suggestions from 75 diffe rent tions. For example, the test might involve asking users. We implemented 212 (or 59 percent) of

130 Digital Te chnical journal No. 6 February I �88 Software Productivity Tools

these suggestions for the version of EVE shipped and the command set. During internal field test, with the VAX/VMS operating system version 4.2. we collected command frequency data from a We received 225 (or 62 percent) of the sugges­ small set of EVE users to refine the command set. tions before field test began. More of these sug­ We also used command transition data as the gestions were implemented than suggestions basis for the arrangement of the arrow keys on received later: 65 percent of the suggestions the LK20 1 keyboard into an inverted-T shape. received during internal field test were imple­ Usage data from an experimental text editor mented compared to 48 percent of the sugges­ showed that the transition from the down-arrow tions received during external field test. key to the left-arrow key occurred more than Although contextual interviews provide more twice as often as any other transition between 11" 3 information than users' reports of summary arrow keys. 1 The inverted-T arrangement also experience, the summary experience data is still allows three fingers of the user's hand to rest on valuable. The two methods complement each the three most frequently used arrow keys, with other. The on-site interviews provide details of an easy reach up to the up-arrow key . users' ongoing experiences in the context of sys­ Collectors of usage data must be concerned tem use; on the other hand, electronic mail, con­ about user privacy and system performance. ferencing, and problem reports provide sum­ Users should know about the nature of the data mary experience data from a wider range of collection and be informed when data is being users than engineers could interview. collected. They should also have the option of Early field testing is especially important for using a system that has not been instrumented collecting data on experienced users. Experi­ and does not collect usage data. enced users, as well as new or infrequent users, To inform users that data is being collected, must find systems easy to use. Early field testing designers can modify the instrumented version is an excellent way to develop a test popula­ of the system so that a notification message is tion of experienced users before a product is displayed each time this version is invoked. released. By the time later field test versions Users are thus reminded that all actions are are available, these experienced users will be a being recorded. To minimize performance prob­ valuable source of data on longer-term usability lems on instrumented versions, engineers can issues. design the logging system so that any necessary delays occur at the start and finish of an applica­ Instrumenting the System to Collect tion, not at random intervals while the applica­ Us age Data tion is being used. Knowing how frequently and in what order peo­ ple use a system's functions helps engineers with Analyzing the Impact of Design low-level design decisions. For example, engi­ Solutions neers can use usage data to order functions on Designers make an impact analysis of user data menus, putting less frequently used commands collected during evolutionary delivery to esti­ on less accessible menus. Our group has col­ mate the effectiveness of design techniques in 15 lected and analyzed usage data for text editors meeting product goals. In usability engineer­ and operating systems, and compared this with ing, design techniques are usually ideas devel­ 13 14 data collected by other groups. · oped after watching people use computer sys­ We collect usage data by asking people to use tems. Estimating the effectiveness of a set of an instrumented version of a functioning system, design techniques for meeting a set of usability either an existing product or a field-test version. attributes helps to economically focus engineer­ We collect the most complete data by recording ing effort on key issues. and time-stamping each individual user action. Impact analysis tables contain percentage esti­ Keeping frequency counts of user actions also mates of the contribution of each technique to provides useful usage data, but does not include the planned levels for each usability attribute. data on transitions between actions or time spent Impact analysis tables list product attributes and with different functions. proposed design techniques in a matrix. Each For the EVE editor, we used command fre­ entry in the table estimates the percentage that quency data from five different text editors to this technique will contribute toward meeting guide the initial design of the keypad interface the planned level of this attribute.

Digital Technicaljournal 131 No. 6 Februmy I 988 Software Usability Engineering

Our software usability group creates impact Chauncey Wilson, Dennis Wixon, and Bill Zim­ analysis estimates in several ways, such as ana­ mer. Dorey Olmer helped edit this paper. lyzing the videotapes made during user visits. With laboratOry tests, we have derived estimates References from the time actually spenr as a result of inter­ I. T. Winograd and F. Flores, Understanding face roblems encountered on a benchmark ' p Computers and Cognition: A New Fo un­ task. Impact analysis data can also be pre­ dation fo r Design (Norwood : Ablex, sented graphically using Pareto charts. 17 1986).

Conclusion 2. ). Whiteside, "Usabi lity Engineering," Unix Our group applies usability engineering in the Review , vol . 4, no. 6 Qune 1986): 22-37. development of many new software products 3. W. Deming, Quality, Productivity, and within Digital. Software usability engineering Competitive Position (Cambridge: MIT techniques can be used by any group of engi­ Center for Advanced Engineering Study, neers that designs interactive software. No spe­ 1982). cial equipment or prior experience is necessary to start applying these techniques, although 4. T. Gilb, "Design By Objectives," Unpub­ equipment and experience can improve the lished manuscript avai.lable from the author results. at Box 102, N-1411 Kolbotn, Norway As we have gained experience with usability (1981). engineering, we have moved from laboratOry 5. ). Bennett, "Managing to Meet Usability tests tO field visits as the main source of usabi l­ Requirements: Establishing and Meeting ity data. We find that field-test data provides a Software Development Goals," Visual Dis­ richer source of ideas for user interface design. play Te rminals , eds. ). Bennett, D. Case, ]. laboratory testing is still valuable, however, Sandelin, and M. Smith (Englewood Cliffs : especially for testing early protOtypes. We are Prentice-Hall, 1984): 161-184. now bringing some contextual interview tech­ niques tO our laboratory tests, interviewing users 6. P. Gilbert, "Development of the VAX as they perform a task rather than observing NOTES System," Digital Te chnical jo urnal them as they work on their own. For more ad­ (February 1 988, this issue) : 117- 1 24. vanced prototypes, we may ask users tO use the system with their own work, which they bring 7. H. Dreyfu s and S. Dreyfus, Mind over with them to the laboratory. Controlled labora­ Machine (New Yo rk: The Free Press, tory experimentation techniques are still usefu l 1986). for deciding some important design issues, such 8. F. Brooks, Jr., "No Silver Bullet: Essence and as choosing screen fonts for an application. Accidents of Software Engineering," IEEE A user-oriented approach to software design Computer, 20, no. 4 (April 1987): 10-19 requires a commitment to understanding and meeting users' needs through observation of 9. M. Good , "The Iterative Design of a New people using systems. Software usability engi­ Text Editor," Proceedings of the Human neering techniques, applied in whole or in part, Fa ctors Society 29th Annual Meeting , vol . can produce computer systems that enrich 1 (1985): 571-574 . human experience. 10. B. Curtis, et al., "On Building Software Pro­ Acknowledgments cess Models Under the Lamppost," Proceed­ ings of the IEEE 9th International Co nfe r­ The techniques described here were deve loped ence on Software Engineering (1987): in a group effort by present and past members of 96-I 03. the Software Usability Engineering Group, including Mark Bramhall, Alana Brassard, Jim 11. M. Good, "The Use of Logging Data in the Burrows , Elisa del Galdo, Charles Frean, Design of a New Te xt Editor," Proceedings Kenneth Gaylin, Karen Holtzblatt, Sandy )ones, of the CHI '85 Human Fa ctors in Comput­ Thomas Spine, Eliot Ta rlin, John Wh iteside, ing Systems (1985): 93-97.

132 Digital Tecbnlcaljournal No. 6 Fe bruary 1988 Software Productivity Tools

12. M. Good , J. Wh iteside, D. Wixon and S. Jones, "Building a User-Derived Interface," Com munications of the ACM, 27 (OctO­ ber 1984 ): 1032-104 3.

13. J. Whiteside, et at ., "How Do People Really Use Te xt Editors?" SIGOA Ne wsletter, 3 Oune 1982): 29-4 0.

14. D. Wixon and M. Bramhall, "How Operat­ ing Systems Are Used: A Comparison of VMS and UNIX," Proceedings of the Human Fa ctors Society 29th Annual Meeting, vol. I (1985): 245-249

15. T. Gilb, "The 'Impact Analysis Ta ble' Applied tO Human Factors Design," Human- Computer Interaction-INTER­

ACT '84 , ed. B. Shackel (Amsterdam: North-Holland, 1985): 65 5-659.

16. M. Good , et al., "User-Derived Impact Analysis as a Tool for Usability Engineer­ ing," Proceedings of the CHI '86 Hu man Fa ctors in Computing Sys tems ( 1986): 24 1-246.

17. K. Ishikawa, Guide to Quality Control, second revised ed. (Tokyo : As ian Productiv­ ity Organization , 1982).

Digital Te chnical journal 133 No. 6 Fe bruary 1988 r r• .

ISBN l-55558-005-X

Printed in USA EY-8259E-DP Copyright © February 1988 Digital Equipment Corporation