Improving the Quality of Quality Determination

Pro cesses

Leon J Osterweil

Department of

University of Massachusetts

Amherst MA

USA

Abstract

This pap er suggests a systematic orderly pro cessbased approach to stating ob jec

tives and knowing if and when they have b een achieved We suggest that qualityinsoftware is a complex

multifaceted arrayofcharacteristics and that it is imp ortant to establish sp ecic ob jectives along various

software quality dimensions as requirements for software quality assurance determination pro cesses We

prop ose that pro cess technology b e used to design co de execute evaluate and migrate pro cesses that

are demonstrably eectiveinachieving required software pro duct quality ob jectives Recently there have

b een numerous highly visible eorts to co dify the assessmentofsoftware pro cesses and to use assessment

results to improve them In this pap er we argue that these eorts function as testplans for software pro

cesses We b orrow some of the notions prop osed in these eorts and indicate how they can b e used to

construct a discipline of measuring and evaluating howwell pro cesses can b e exp ected to deliver sp ecic

knowledge ab out software pro duct qualities Welooktowards the gradual but eventual establishmentof

an orderly discipline of software quality demonstration pro cess development that should ultimately sup

port a marketplace in which denitive knowledge ab out the nature of software pro ducts can b e b ought

and sold

Keywords

software pro cess software quality pro cess improvement integrated and analysis

INTRODUCTION

As the use of computers b ecomes more p ervasive in our civilization and so ciety it b ecomes increasingly

imp ortant to be sure that the software used to direct and control them is of high quality Computer

systems now supp ort virtually all infrastructural areas of so cietyThey are essential in banking They

supp ort key medical functions They facilitate educational pro cesses They are used to help design roads

and buildings and to co ordinate their construction They are the backb one of our communications systems

They are essential to such national security functions as defense and intelligence Computer systems are

now increasingly integral to the practice of science to many forms of the arts and to the humanities

and so cial sciences Indeed it is increasingly clear that computer systems are themselves critical so cietal

infrastructure

All of this underscores the imp ortance of b eing sure that computer systems do their jobs satisfactorally

Unfortunately this is evidently often not the case Computer systems routinely fail in use Often these

failures are merely annoying as when a bill for arrives Sometimes the results are amusing as when

a form letter is comically misaddressed But computer failures can b e serious as when a New York bank

This work was supp orted in part by the Air Force Materiel Command Rome Lab oratory and the

Advanced Research Pro jects Agency under Contract FC

had to b orrowseveral billions of dollars b ecause its computer failed to reconcile its b o oks at the end of a

business dayFailures can cause serious injury or death as when a computer system administered lethal

doses of radiation to patients It is easy to see that more serious failures for example in defense systems

can p otentially cause large scale death and destruction In the past most failures have b een attributable

to software rather than hardware Thus it is clear that assuring that computer software is of high quality

and will not fail is an issue of very great imp ortance

Assessing the qualityof software and assuring that it will not fail is dicult and complex Muchof

the complexity is b ecause there are many dimensions of quality and software can fail in many dierent

ways Functional correctness is the most obviously imp ortant quality Certainly we rely up on software

to compute the right answers But other quality attributes are also imp ortant and may sometimes b e

of more imp ortant than functional characteristics For example sp eed is critically imp ortant esp ecially

in interactions with human users and in realtime monitoring Robustness is also usually quite imp or

tant but is often overlo oked Computer systems must react resp onsibly when they receive incorrect and

unexp ected inputs They must avoid certain behaviors and must not fail catastrophicallyOther qual

itycharacteristics that software is generally exp ected to demonstrate are reliability comprehensibility

safety userfriendliness and mo diability The list could go on It is also worth noting that these charac

teristics may conict with each other Thus execution sp eed is often sacriced to achieve reliability and

robustness for example

It is a real challenge to determine the extenttowhich a piece of software demonstrates a particular

qualityFor example it is very hard to determine if a piece of software can ever p ossibly fail to compute

correct functional values Even a mo destsized piece of software may compute dozens of functions and

p erhaps dierent sets of functions for dierent inputs The inputs to any function may come from a

potentially innite space of p ossibilities Testing merely samples from that space How do es one determine

when the sampling has b een suciently representative Similar problems arise in testing software sp eed

Do es one determine worst case p erformance or average case If one is interested in the average case

then over what sample do es one average Assessing the readability mo diability and userfriendliness of

software p oses serious challenges of still dierent kinds

APPROACHES TO SOFTWARE QUALITY

Testing

Postdevelopment testing is the traditional way to determine and assure qualityThereareobvious advan

tages to exercising the completed software with sp ecimen input data Program b ehavior can b e observed

in the actual deployment environment to enable study of interactions with runtime supp ort and op

erating systems Program instrumentation can be used to supp ort arbitrarily intensive scrutiny of the

programs execution Thus failures can b e identied and studied in minute detail and evolutions of values

of variables can b e studied Flowofcontrol details can b e observed recorded and analyzed

But dynamic testing has serious drawbacks as well Instrumentation inevitably p erturbs the programs

execution Thus the instrumentation may cause new failures or may mask and distort previous failure

phenomena A more fundamental problem with dynamic testing is that it can demonstrate the presence

of faults leading to failures but it is generally not able to demonstrate their absence Early in testing

failures are exp ected and testing leads to fault correction But later it is increasingly hop ed that failures

will not b e detected and faults are no longer present As determining the absence of faults b ecomes the

key goal dynamic testing b ecomes increasingly inappropriate

When dynamic testing is used to show the absence of faults it is essential to select test data sets

that thoroughly exercise the program An extensive literature do cuments many approaches to selecting

test data How Some emphasize systematic sampling of the programs input data space blackbox

or requirementsbased testing eg see RAO These approaches tend to emphasize mo des in which

actual users tend to use the program Their weakness is that they pay less attention to less frequently

used mo des such as failure recovery that may still b e quite imp ortant They also fail to concentrate on

implementation structures that may b e errorprone Thus a second ma jor test data selection approachis

socalled white b ox testing in which test cases are chosen to assure that implementation structures are

thoroughly exercised HowRT Some combination of these approaches seems indicated

Static Analysis

Static analysis complements dynamic testing in its abilitytoshow the absence of certain classes of faults

It has the additional advantage of not requiring execution of the program and hence the selection of

test data sets Static analyzers work on mo dels of the program and its p ossible executions The most

familiar static analyzers are incorp orated into compilers Compiler syntax analyzers build and examine

parse trees and determine the presence or absence of syntactic faults More ambitious compilers and

similar standalone analysis to ols determine the presence or absence of certain semantic faults eg the

PFORTFortran Analyzer Ryd and the Lint C analyzer

More p owerful static analyzers build increasingly sophisticated program mo dels and employ increas

ingly p owerful mathematics to demonstrate the absence of wider classes of faults Thus dataow analyzers

OF DC construct annotated owgraphs to represent program execution b ehaviors and then com

pare them to regular expressions that describ e sequences of events that represent desired behaviors

Exp erience suggests that these mo dels help analysts study such qualities as safety and robustness but

are less helpful in studying functional prop erties Formal verication AGB Bjo uses such for

malisms as predicate calculus to represent program functional behaviors Inference techniques such as

symb olic execution Kin Cla then develop mo dels of program functionality that are then compared

to desired functional b ehavior sp ecications to determine the presence or absence of functional faults

While these analyses can demonstrate the absence of some classes of faults it is imp ortant to note

that they cannot determine the absence of all kinds of faults it is also imp ortant to note that these

analyzers check mo dels of execution b ehavior rather than actual execution b ehavior and may therefore

yield inaccurate or incomplete results

Development Approaches

While testing and analysis of nished co de remains the most p opular approach to assuring software qual

ity there is considerable sentiment that it is more eective to build qualityinto softwareasitisbeing

develop ed This philosophy is consistent with practices in industries such as automobile manufacturing

Building qualityinto software entails emb edding quality assurance steps and practices into the develop

ment pro cess There is now a rough consensus ab out the overall strategy for doing this For example

it is now agreed that nished software should contain a sp ecication of its requirements various design

representations testcases and their corresp onding test results co de and do cumentation It is also agreed

that dierent of these artifacts need more attention during dierent development stages Thus building

qualityinto the pro duct entails assuring that each new artifact is prop erly develop ed and integrated with

the others Thus for example as design sp ecications are built they should b e shown to b e resp onsive

to stated requirements as co de is built it should be shown to correctly implement the corresp onding

designs after co ding is complete a testing pro cess should demonstrate that test case execution results

are consistentwith requirements This seems to b e a useful general prescription for developing quality

software but it lack the details of precisely how and when to do all of this

Integration of These Approaches

While it is reasonable to build qualityinto software as it is b eing develop ed it also seems essential to

evaluate the software after development to b e sure that the desired quality has b een achieved It seems

clear that some sort of combination of testing the nished co de and analyzing mo dels of it for faults is

needed Many also b elieve that this sort of combination is also what is needed to assess qualityonan

ongoing basis during development As indicated ab ove it seems agreed that integrating these approaches

is desirable but there is no agreement ab out the details of howtodothis

THE ROLE OF SOFTWARE PROCESS

Software pro cess work supp orts building qualityinto software byfocusingonhowsoftware is built and

assessed The software pro cess community regards software as a complex intricately interconnected in

formation aggregate that has b een built by executing a pro cess that should b e orderly and systematic

Asoftware pro duct can b e considered to b e of high quality only when its constituent artifacts have b een

shown to b e consistent with each other and within themselves in a varietyofways Evidence of these

forms of consistency is captured by other software artifacts generally rep orts from such quality assurance

activities as analysis and testing Thus high quality software pro ducts are necessarily large and complex

and the pro cesses bywhich these complex structures are assembled and assessed must b e appropriately

complex as well Unfortunately software pro ducts are generally develop ed using informal ad ho c pro cesses

that usually reside in the heads of those who execute and sup ervise them As a consequence software

pro jects are often p o orly co ordinated inadequately sup ervised and ineectiveindelivering pro ducts of

demonstrably high quality

Making the pro cess of developing software explicit can address many of these problems Indeed it seems

that increasing the rigor and sp ecicity of the denitions of software pro cesses should lead to increased

eectiveness in developing software pro ducts whose quality is b oth assured and demonstrable Carefully

and explicitly dened software pro cesses can sp ecify exactly how and when the various comp onents of a

software pro duct are to b e pro duced analyzed and tested thereby supp orting demonstration of pro duct

quality

More careful and explicit software pro cess denition also facilitates software understanding develop

ment team communication and pro ject control But in this pap er we emphasize that sup erior grasp of

the pro cess of developing software leads to sup erior quality in the pro ducts develop ed We indicate how

this results from increased abilitytoplanquality instilling activities in advance to sp ecify how they are

to b e done to secure supp ort esp ecially by computers for executing these activities and to verify that

they have actually b een done

Software Pro cesses as Software

The key to seeing how software pro cess rigor and formality eects improved software pro duct quality seems

to us to be an appreciation of the nature of software pro cesses as software themselves Earlier Ost

we suggested that pro cesses and applications share manykey characteristics Software pro cesses aim to

develop a complex information aggregate the completed software pro duct much as application software

aims to develop complex information structures Moreover just as applications sup ervise p erformance of

their work through the vehicle of an abstract computing device consisting of hardware and software

software pro cesses sup ervise the p erformance of software development on an abstract device comp osed of

hardware software p eople and organizations Figure suggests these parallels diagrammatically Other

comp elling parallels are put forth in OstOst

Our conclusion is that software pro cesses can and should b e dened precisely and rigorously using pro

Figure Software pro cesses are a typ e of software The software pro cess description is typ e of co de

whose execution results in the development of an application software pro duct

gramming languages Exp erimentation with a range of pro cess programming languages Sut KBS

Kat supp orts this premise Among the demonstrated b enets of this are provision of a blueprint

for integrating to ols to supp ort software development indication of howdevelopment tasks can b e paral

lelized acquisition of deep er insights into software development as an activity and co ordination of human

software workers with supp ort technologies

Here we are particularly interested in programming pro cesses that lead to the developmentofsoftware

of demonstrably high quality Earlier we suggested that such quality is b est achieved by co ordinating

testing and analysis with development throughout the stages of the development lifecycle We indicated

that there is agreement ab out the broad outlines of this approach but not ab out the details Pro cess

programming provides a vehicle for b eing precise ab out these details What is less clear however is the

rationales for driving the development of software pro duct qualityassuring pro cess co de

The Sp ectrum of Software Pro cess Artifact Typ es

Thinking of software pro cesses as software helps determine a rationale for integrating qualityassuring

to ols and technologies If a software pro cess is software then it must consist of more than just co de

As noted ab ove qualitysoftware consists of a varietyoftyp es of artifacts such as requirements design

and testcases as well of demonstrations of their consistencyThus quality software pro cesses must also

consist of nonco de artifacts and demonstrations of consistency as well Indeed much software pro cess

technology fo cuses on using mo deling and design MABFGea to create blueprints for pro cess

co de implementation But it is requirements sp ecications that are p erhaps the key typ e of software

artifact Requirements sp ecify what characteristics completed software must have and are used to make

decisions ab out how pro duct software is to b e architected and designed They are also used to sp ecify

the criteria against which completed software co de is to be tested These inculde functionality sp eed

accuracy and safety

Software pro cesses also have requirements to whichtheymust adhere For example they have timing

requirements that sp ecify the amount of time allowed for completion of the pro ject They often have

robustness requirements that sp ecify pro cess resistance to such unanticipated stresses as loss of key

p ersonnel They also have functional requirements that enumerate the sorts of artifacts to b e pro duced

what their characteristics should b e and the ways in whichtheymustbeshown to b e consistent with

each other

Software pro cesses also undergo testing evaluation and evolution Whenever a software pro cess is used

the outcome is evaluated Changes are made the pro cess fails to achieveits requirements Often these

failures are recognized while the pro cess is still executing but often the recognition arises from a static

analysis of the pro cess In any case pro cess evolution and improvement are to b e exp ected The view of

software pro cesses as complete software pro ducts is represented by the diagram in Figure

The Capability Maturity Mo del

The most active and visible work in software pro cess is that centering on the Capability MaturityModel

CMM at the US Department of Defense Institute SEI at Carnegie Mellon

University under the direction of Watts Humphrey Hum Hum The CMM is a set of ve normative

mo dels of software development augmented by elab orate evaluation instruments and institutions that

help determine whichoftheve mo dels most closely describ es the pro cess actually used byasoftware

development organization The rst normative mo del describ es a development pro cess that is essentially

chaotic pro ducing pro ducts of p o orly determined quality according to an unreliable schedule without

the b enet of careful planning budgeting estimation or commitment Organizations that t this mo del

b est are classied as Level Organizations and are said to op erate at Level

The second normative mo del describ es organizations that can repro duce their activities and results

on successive pro jects These organizations generally have thought ab out their development pro cess

but have not dened it carefully Level Organizations have dened their pro cess and have used it

consistently to pro duce software in predictable ways and according to predictable schedules At level

organizations are measuring and analyzing their pro cesses and at level organizations are using

pro cesses that automatically measure themselves and use the measurements to automatically improve

themselves SEI evaluations have shown that most software development organizations op erate at level

although the numb ers and p ercentages of organizations op erating at higher levels have b een inching up

CMM evaluations are generally carried out by teams of human assessors who make essentially qual

itative judgements ab out how assessed organizations op erate These judgements are guided by SEI as

Figure Software pro cess software is more than just software pro cess co de As software itself software

pro cesses are also comp osed of pro cess requirements pro cess design as well as pro cess analyses and

execution histories

sessment guidelines and frameworks that strongly emphasize analysis of do cumented evidence of how

organizations have p erformed in the past Thus CMM level assessment is a strong function of the do c

umentation and artifacts pro duced by previous executions of an organizations pro cess The use of the

CMM to improve organizational pro cess is shown diagrammatically in Figure

A clear overriding goal of the CMM is to instill in organizations a consciousness of the imp ortance of

pro cess and a desire to advance to higher levels of pro cess maturityThus the CMMcentered movement

is often referred to as the Pro cess Improvementmovement The US military increasingly sp ecies that it

will pro cure custombuilt software only from organizations that demonstrably op erate at higher levels of

Figure The CMM is used to study a software development organizations pro cess CMM evalua

tions are based on inferences ab out the nature of the pro cess drawn from observations of past pro cess

executions

maturity generally at level or ab ove This has strongly stimulated software development organizations

to improve their pro cesses

SEI success in p opularizing the CMM has interested other organizations in software pro cess assess

ment Thus for example the International Standards Organization ISO is working on ISO an

international standard for software development The Europ ean Software Institute ESI is interested in

this activityaswell as are software engineering standards b o dies around the world The SPICE pro ject

is attempting to harmonize this growing set of pro cess assessment activities and organizations

Hop e that SPICE and other eorts can harmonize these activities seems be based largely on the

fact that all center on the notion of Key Pro cess Areas KPAs A Key Pro cess Area is an area of

software development endeavor that is particularly critical KPAs contain collections of practices called

key practices that contribute to success in the KPA Some KPAs fo cus on such practices as planning

estimation and training The current CMM version identies some KPAs and indicates whichKPA

is crucial for advancement to which next levels Dierent pro cess assessment schemes handle KPAs

dierently But there seems to b e a clear consensus that the essence of assessmentofsoftware pro cesses

is scrutinyofhow organizations attackvarious KPAs

One of the more controversial asp ects of the CMM is that it points strongly to certain KPAs as

being key in moving organizations to the next highest level In essence the CMM uses KPAs to rivet

the attention of improvementconscious organizations on a narrow set of pro cess areas often to the near

exclusion of other areas Thus the KPAs most crucial to improvement from lower CMM levels and

tend to receive the most attention It seems widely assumed that improvementfromtheselower levels

results in higher software pro duct quality While this may b e the case it is not b ecause there are large

numbers of KPAs that fo cus on this as an issue Indeed any improvements in software quality result

indirectly from other pro cess issues

The CMM as a Software Pro cess Test Plan

Earlier we noted that software pro cesses like other kinds of software evolve when they fail to adequately

address their requirements Wesaw that the CMM supp orts this sort of pro cess improvementby serving as

a guide in assessing software pro cesses We further noted that CMMbased assessments examine artifacts

for evidence of sucient comp etence in KPAs This evidence is generally in the form of records and

artifacts pro duced by past executions of the organizations pro cess

Thus we see that the CMM functions essentially as a test plan for software pro cesses It mandates

certain kinds of b ehaviors as b eing required if the pro cess is to b e assessed at a higher level and accepts

as evidence artifacts pro duced as outputs from pro cess executions

While the CMM has had an undeniably p ositive impact in fo cusing attention on the need to improve

software pro cesses we can now see some aws in this approach We noted ab ove for example that

quality is best built into software rather than b eing tested in at the end Yet paradoxically we see

that CMMbased pro cess improvement inherently attempts to improve pro cess quality through p ost ho c

testing of the pro cess One problem with p ost ho c testing is that once the test plan is known software

development tends to b e skewed to address key testing criteria In this case since the CMM is known in

advance software pro cess improvement eorts tend to b e skewed to address those KPAs that are keys

in supp orting organizational advancement to the next higher level

Of greatest concern to us however is the direction in which the CMM tends to skew improvement

eorts Careful study of the CMM and its KPAs suggests that the key goal of the CMM and the key

dimension along which pro cess improvement is encouraged is pro cess predictability and not pro duct

qualityWithKPAs fo cusing on prediction estimation and planning the currentversion of the CMM

incentivizes and rewards organizations for b ecoming increasingly accurate in predicting cost and schedule

As the CMM is pro duced by the DoDsp onsored SEI this is neither surprising nor inappropriate Further

suggestions that pro duct quality improvement is strongly correlated with organizational predictability

seem both plausible and statistically supp orted Nevertheless it seems imp ortant that organizations

whose primary concern is for software pro duct quality realize that their primary ob jective is not the

same as the primary ob jective of the current CMMbased pro cess improvementmovement In the next

section we outline a prop osal that should interest organizations whose primary goal is assuring software

pro duct quality

THE IMPROVEMENT OF SOFTWARE QUALITY INSTILLING

PROCESSES

We b elieve that the key to systematic improvement in the ability of pro cesses to build in and demonstrate

quality in software pro ducts is to sp ecify precise software quality demonstration requirements for soft

ware pro cesses and to use a CMMlike approach to incentivize improvement along this software pro cess

dimension

Software Pro duct Quality Requirements

Earlier we indicated that quality in software is a manyfaceted manydimensional characteristic We

describ ed failures in functionality eciency safety and robustness as examples of ways that software

could demonstrate lack of quality Later we suggested that software should b e viewed as the pro duct of

asoftware development pro cess From this viewp ointwe see functionality eciency safety robustness

etc as software pro duct requirementtyp es

Thus we can view the demonstration of quality in software as b eing the pro cess of demonstrating

the adherence of pro duct software to sp ecied quality requirements This explains why dierentsoftware

pro ducts require dierent sorts of quality demonstrations It also argues strongly for the imp ortance of

identifying what those quality attributes must be in advance of the development of the pro duct soft

ware Being precise ab out sp ecic quality requirements is most imp ortant but it is also quite dicult It

seems at rst glance quite easy to sp ecify that the software pro duct must never fail to compute required

functions that it must never fail to execute suciently rapidly and that it must never violate safety

conditions On the other hand simply sp ecifying these conditions leaves unanswered the more imp ortant

and challenging question of what evidence that the software meets these requirements is to be taken

as acceptable Wehave already noted that virtually all programs can b e sub jected to virtually limitless

amounts of dynamic testing At what p oint should testing activities b e considered adequate How thor

oughly should eachof the many functions of a program b e exercised How precisely should timing be

measured What pro cedures should b e followed in order to assess a software systems userfriendliness

These and similar questions seem to us to b e pro cessrelated They suggest that it is more fruitful to

consider quality determination to b e a pro cess and to consider what the requirements for this pro cess

are Consideration of the requirements for the software pro duct quality determination pro cess seems to

lead to more useful conclusions ab out how to p erform software testing and analysis

Software Pro duct Quality Determination Pro cesses

We now consider software pro duct quality determination pro cesses to be typ es of software We b egin

by considering their requirements This transforms the more traditional question What qualities must

pro duct software have to the more pragmatically useful question What demonstrations of software

quality will be acceptable as sucient evidence that pro duct software has the desired qualities OR

b etter yet What do I wish to learn ab out my program and how will I knowwhenI have learned it

Remarkably many software testing and evaluation pro cesses b egin and end without any clear enunciation

of even such primitive goals Small wonder that so many testing activities end unsatisfactorally Without

sp ecied requirements for this pro cess it is imp ossible to determine whether it has succeeded or failed

Once quality determination pro cess requirements have b een sp ecied it is p ossible to design the pro cess

of making the determinations Presumably the determinations are to be made by a skillful integration

of suchtechniques as dynamic testing and static analysis applied to the software artifacts pro duced as

by the development pro cess Dierent requirements will lead to dierent congurations and integrations

This is reassuring as it explains the observed phenomenon of widely diering approaches to software

quality determination employed in actual practice This diversity should b e exp ected as a consequence of

dierent requirements

Quality determination should then execute in parallel with pro duct development Evidence suggests

that concentrating quality determination towards the b eginning of the development pro cess is most

eective but exp erience shows that most development activities will concentrate quality determination

towards the end In any case it is most imp ortant that the quality determination pro cess itself be

evaluated as it executes While it is hop ed that sp ecifying requirements for the pro cess and designing it

carefully will lead to more successful quality determination pro cesses it must b e exp ected that they may

still fail to meet their requirements and will require mo dication evolution and improvement The need

for such mo dication and the exact nature of the improvements needed cannot b e determined without

careful evaluation of the pro cess itself

An Example

The preceding discussion is claried by an example Supp ose an aircraft navigation system must be

built Central to the navigation system is software resp onsible for reading lo cation altitude and sp eed

data from sensors comparing that data to routing vectors and sending realtime instructions to ight

control systems to determine the course of the plane Clearly this software has functional requirements

In addition however it has stringent p erformance requirements needed to assure that mo dications in

direction are timely There are safety requirements to assure that the software never misdirects the plane

disastrously There are also user interface requirements accuracy requirements and others

These requirements must b e used to establish baselines against whichtoevaluate the qualityofthe

software to be built It is clearly necessary but denitely not sucient to simply require that the

eventual software system will adhere to these requirements Some of these requirements will b e of greater

imp ortance than others Thus assurances that they have b een met will b e of greater consequence and

condence of the eectiveness and accuracy of the pro cesses for assuring them will need to b e higher It

seems more direct and straightforward to underscore and op erationalize these dierences in imp ortance

and priorityby elab orating these basic pro duct quality requirements into quality determination pro cess

requirements

Thus for example functions that compute the lo cation of the plane accurately may be considered

more imp ortant than functions that up date co ckpit displays accurately In that case greater assurance of

the eectiveness of the quality determination pro cess would b e warranted for the former functions than

for the latter This might b e made quite clear and concrete by indicating dierent degrees of assurance

of the absence of faults in the computation of these dierent functions In the case of less imp ortant

functions a testing regimen requiring a certain more mo dest level of coverage might b e mandated Once

the testing level has b een established testing testing coverage should b e measured on an ongoing basis

and testing should continue until it has b een reached at which p oint it should terminate For more critical

functions however it would seem reasonable to require more assurance and to resp ond by designing a

quality determination pro cess that also included more rigorous forms of analysis of software functionality

These attempts at more rigor should likewise b e monitored and failure to complete them would require

more urgent resp onse

Similarly considering the requirements for the pro cess of determining acceptable p erformance leads

to imp ortant sharp ening of the basic pro duct p erformance requirements It may be that new p osition

and direction information must be determined every millisecond without fail Or it may be that it is

p ermissible to b e late with this information o ccasionally In the latter case it is imp ortant to dene just

what is meantby o ccasionally and it might suce to resp ond by designing a dynamic testing pro cess

that measures the p ercentage of cases in which results are computed late The former case implies that the

quality determination pro cess must deliver ironclad assurances of timeliness In this former case testing

may b e a reasonable way to start o but some sort of static analysis activitywould b e necessary later on

Further the requirement for ironclad assurance dictates that the pro cess of enunciating and completing

analytic pro ofs would have to go to completion Failure of the quality determination pro cess to do so

would have to b e detected as part of the ongoing evaluation of the pro cess and would require decisive

resp onse either mo dication of the pro cess or mo dication of the requirements for ironclad assurance

Sp ecic safety prop erties must b e stated preciselyOnceenunciated the designer of the quality deter

mination pro cess would have to devise eective demonstrations that they have b een satised As in the

preceding discussion this pro cess will require ongoing evaluation to assure that it do es not fail to deliver

the ironclad assurances required It is doubtful that dynamic testing can ever deliver such assurances

but static dataow analysis often can For example safe op eration of the plane might require that newly

computed p osition and direction data never be be overwritten b efore b eing used to compute new heading

data This can b e readily transformed into an event sequence sp ecication that can then b e the basis for

static dataow analysis If attempts to complete this analysis fail then either analysis eorts must be

strengthened or the safety requirement reconsidered

In all of the ab ove cases we see that the need to op erationalize the pro cess of determining pro duct

quality and to continually compare success in doing so to sp ecied pro cess requirements has led to the

ability to b e more sp ecic ab out how to design the pro cess This in turn leads to the abilitytobemore

sp ecic in the details of which testing and analysis approaches to use and howto integrate them An

imp ortant consequence of this approach is that it makes very clear the fact that one should neither exp ect

nor seek the emergence of one standard testing and analysis pro cess for all software Dierentsoftware

systems have dierent pro duct requirements These in turn give rise to dierent quality determination

pro cesses and they give rise to dierences in the sp ecics of howtointegrate testing and analysis into

the development lifecycle

On the other hand all such quality determination pro cesses share the need for continuous improvement

We b elieve that a CMMlike approach can b e quite useful in guiding improvement eorts

Software Pro duct Quality KPAs

The foregoing discussion has shown that one should exp ect quality determination pro cesses to evolveover

time Quality determination pro cesses have uncertain outcomes b ecause of uncertainty ab out the inherent

quality of the software b eing assessed unfamiliarity with how to state requirements p erform design and

enco de such pro cesses eectively and b ecause of the immaturity of current testing and analysis to ols

Thus quality determination pro cesses must be the sub jects of continuous improvement This suggests

considering using a CMMlike approach to do this

Indeed we b elieve that quality determination pro cesses should progress from b eing chaotic to rep eat

able to dened to measured and ultimately to self improving Thus the basic structure of ve normative

mo dels seems applicable here Further the use of Key Pro cess Areas KPAs would seem to b e appro

priate here as well It seems imp ortant to reect on what these KPAs might b e Certainly planning for

software pro duct quality determination as a separate and separable activitywould have to b e one such

key pro cess Measuring testedness observing analysis failures reviewing pro duct quality determination

progress and developing pro cess architectures and designs all seem to b e likely candidates for KPAs as

well This seems to b e an imp ortant area for further investigation

Finallywecannow return to our earlier observation that the directions in which the current CMM is

driving improvement are not directly towards pro duct qualityAsKPAs seem to b e the vehicles for setting

such directions it now seems that identifying KPAs sp ecic to pro duct quality determination mightbe

just the needed vehicle Current discussions of the inclusion of software pro duct qualityKPAs in the next

version of the CMM is encouraging But organizations fo cused on software pro duct qualityimprovement

need not wait for CMM revisions They should be able to begin identifying and institutionalizing key

practices immediately Sup erior quality in their pro ducts should ensue

REFERENCES

AGB AL Ambler DI Go o d JC Browne WF Burger RM Cohen CG Ho ch and RE Wells

Gypsy A Language for Sp ecication and Implementation of Veriable Programs In Proceedings of

ACM Conference on Language Design for Reliable Software pages March Published as

SIGPLAN Notices VolNo

BFG Sergio Bandinelli Alfonso Fuggetta and Sandro Grigolli Pro cess Mo deling inthelarge with

SLANG In Proceedings of the Second International Conference on the Software Process Berlin

Germany pages February

Bjo D Bjorner On the Use of Formal Metho ds in Software Development In Proceedings of the Ninth

International Conference on Software Engineering pages March Published by IEEE

Computer So ciety Press

Cla Lori A Clarke A System to Generate Test Data and Symb olically Execute Programs IEEE

Transactions on Software Engineering SE September

DC Matthew Dwyer and Lori Clarke Data Flow Analysis for Verifying Prop erties of Concurrent

Programs In ACM SIGSOFT Software Engineering Notes Proceedings of the Second ACM Sigsoft

Symposium on Foundations of Software Engineering pages Decemb er

ea Richard J Mayer et al IDEF Family of Metho ds for Concurrent Engineering and Business Re

engineering Applications Technical rep ort Knowledge Based Systems Inc

How WE Howden Intro duction to the Theory of Testing In E Miller and WE Howden editors

Tutorial SoftwareTesting and Validation Techniques pages IEEE New York

How WE Howden Functional Program Testing and Analysis McGrawHill Series in Software Engi

neering and Technology

Hum Watts S Humphrey Characterizing the Software Pro cess A MaturityFramework IEEE Soft

ware pages March

Hum Watts S Humphrey Managing the Software Process SEI Series in Software Engineering

AddisonWesley August

Kat Takuya Katayama A Hierarchical and Functional Software Pro cess Description and its Enaction

In Proceedings of the Eleventh International Conference of Software Engineering pages

KBS Gail E Kaiser Naser S Barghouti and Michael H Sokolsky Exp erience with Pro cess Mo delling

in the MARVEL Software Development Environment Kernel In Bruce Shriver editor Proceedings

of the TwentyThird Annual Hawaii International Conference on System Sciences Kona Hawaii

volume I I pages January

Kin James C King A New Approach to Program Testing In Proceedings of the International Con

ferenceonReliable Software pages

MA Carlo Montangero and Vincenzo Ambriola OIKOS Constructing Pro cessCentered SDEs In

Anthony Finkelstein Je Kramer and Bashar Nuseib eh editors Software Process Model ling and

Technologychapter pages John Wiley Sons Inc Taunton Somerset England

OF Leon J Osterweil and LD Fosdick DAVE A Validation Error Detection and Do cumentation

System for Fortran Programs SoftwarePractice and Experience Octob er

Ost Leon J Osterweil A Pro cessOb ject Centered View of Software Environment Architecture In

R Conradi Didriksen T and D Wanvik editors AdvancedProgramming Environments pages

Trondheim SpringerVerlag

Ost Leon J Osterweil Software Pro cesses are Software Too In Proceedings of the Ninth International

Conference of Software Engineering pages Monterey CA March

RAO Debra J Richardson Stephanie Leif Aha and T Owen OMalley Sp ecicationbased Test

Oracles for Reactive Systems In Proceedings of the Fourteenth International Conference on Software

Engineering Melbourne Australia pages May

RT Debra J Richardson and MC Thompson A Formal Framework for Test Data Selection Criteria

Technical Rep ort Computer and Department UniversityofMassachusetts

at Amherst Amherst MA Novemb er

Ryd BG Ryder The PFORTVerier SoftwarePractice and Experience

Sut Sutton Jr Stanley M and Heimbigner Dennis and Osterweil Leon J APPLA A Language

for SoftwarePro cess Programming ACM Transactions on Software Engineering and Methodology July