Improving the Quality of Software Quality Determination Pro cesses Leon J Osterweil Department of Computer Science University of Massachusetts Amherst MA USA Abstract This pap er suggests a systematic orderly pro cessbased approach to stating software quality ob jec tives and knowing if and when they have b een achieved We suggest that qualityinsoftware is a complex multifaceted arrayofcharacteristics and that it is imp ortant to establish sp ecic ob jectives along various software quality dimensions as requirements for software quality assurance determination pro cesses We prop ose that pro cess technology b e used to design co de execute evaluate and migrate pro cesses that are demonstrably eectiveinachieving required software pro duct quality ob jectives Recently there have b een numerous highly visible eorts to co dify the assessmentofsoftware pro cesses and to use assessment results to improve them In this pap er we argue that these eorts function as testplans for software pro cesses We b orrow some of the notions prop osed in these eorts and indicate how they can b e used to construct a discipline of measuring and evaluating howwell pro cesses can b e exp ected to deliver sp ecic knowledge ab out software pro duct qualities Welooktowards the gradual but eventual establishmentof an orderly discipline of software quality demonstration pro cess development that should ultimately sup port a marketplace in which denitive knowledge ab out the nature of software pro ducts can b e b ought and sold Keywords software pro cess software quality pro cess improvement integrated software testing and analysis INTRODUCTION As the use of computers b ecomes more p ervasive in our civilization and so ciety it b ecomes increasingly imp ortant to be sure that the software used to direct and control them is of high quality Computer systems now supp ort virtually all infrastructural areas of so cietyThey are essential in banking They supp ort key medical functions They facilitate educational pro cesses They are used to help design roads and buildings and to co ordinate their construction They are the backb one of our communications systems They are essential to such national security functions as defense and intelligence Computer systems are now increasingly integral to the practice of science to many forms of the arts and to the humanities and so cial sciences Indeed it is increasingly clear that computer systems are themselves critical so cietal infrastructure All of this underscores the imp ortance of b eing sure that computer systems do their jobs satisfactorally Unfortunately this is evidently often not the case Computer systems routinely fail in use Often these failures are merely annoying as when a bill for arrives Sometimes the results are amusing as when a form letter is comically misaddressed But computer failures can b e serious as when a New York bank This work was supp orted in part by the Air Force Materiel Command Rome Lab oratory and the Advanced Research Pro jects Agency under Contract FC had to b orrowseveral billions of dollars b ecause its computer failed to reconcile its b o oks at the end of a business dayFailures can cause serious injury or death as when a computer system administered lethal doses of radiation to patients It is easy to see that more serious failures for example in defense systems can p otentially cause large scale death and destruction In the past most failures have b een attributable to software rather than hardware Thus it is clear that assuring that computer software is of high quality and will not fail is an issue of very great imp ortance Assessing the qualityof software and assuring that it will not fail is dicult and complex Muchof the complexity is b ecause there are many dimensions of quality and software can fail in many dierent ways Functional correctness is the most obviously imp ortant quality Certainly we rely up on software to compute the right answers But other quality attributes are also imp ortant and may sometimes b e of more imp ortant than functional characteristics For example sp eed is critically imp ortant esp ecially in interactions with human users and in realtime monitoring Robustness is also usually quite imp or tant but is often overlo oked Computer systems must react resp onsibly when they receive incorrect and unexp ected inputs They must avoid certain behaviors and must not fail catastrophicallyOther qual itycharacteristics that software is generally exp ected to demonstrate are reliability comprehensibility safety userfriendliness and mo diability The list could go on It is also worth noting that these charac teristics may conict with each other Thus execution sp eed is often sacriced to achieve reliability and robustness for example It is a real challenge to determine the extenttowhich a piece of software demonstrates a particular qualityFor example it is very hard to determine if a piece of software can ever p ossibly fail to compute correct functional values Even a mo destsized piece of software may compute dozens of functions and p erhaps dierent sets of functions for dierent inputs The inputs to any function may come from a potentially innite space of p ossibilities Testing merely samples from that space How do es one determine when the sampling has b een suciently representative Similar problems arise in testing software sp eed Do es one determine worst case p erformance or average case If one is interested in the average case then over what sample do es one average Assessing the readability mo diability and userfriendliness of software p oses serious challenges of still dierent kinds APPROACHES TO SOFTWARE QUALITY Testing Postdevelopment testing is the traditional way to determine and assure qualityThereareobvious advan tages to exercising the completed software with sp ecimen input data Program b ehavior can b e observed in the actual deployment environment to enable study of interactions with runtime supp ort and op erating systems Program instrumentation can be used to supp ort arbitrarily intensive scrutiny of the programs execution Thus failures can b e identied and studied in minute detail and evolutions of values of variables can b e studied Flowofcontrol details can b e observed recorded and analyzed But dynamic testing has serious drawbacks as well Instrumentation inevitably p erturbs the programs execution Thus the instrumentation may cause new failures or may mask and distort previous failure phenomena A more fundamental problem with dynamic testing is that it can demonstrate the presence of faults leading to failures but it is generally not able to demonstrate their absence Early in testing failures are exp ected and testing leads to fault correction But later it is increasingly hop ed that failures will not b e detected and faults are no longer present As determining the absence of faults b ecomes the key goal dynamic testing b ecomes increasingly inappropriate When dynamic testing is used to show the absence of faults it is essential to select test data sets that thoroughly exercise the program An extensive literature do cuments many approaches to selecting test data How Some emphasize systematic sampling of the programs input data space blackbox or requirementsbased testing eg see RAO These approaches tend to emphasize mo des in which actual users tend to use the program Their weakness is that they pay less attention to less frequently used mo des such as failure recovery that may still b e quite imp ortant They also fail to concentrate on implementation structures that may b e errorprone Thus a second ma jor test data selection approachis socalled white b ox testing in which test cases are chosen to assure that implementation structures are thoroughly exercised HowRT Some combination of these approaches seems indicated Static Analysis Static analysis complements dynamic testing in its abilitytoshow the absence of certain classes of faults It has the additional advantage of not requiring execution of the program and hence the selection of test data sets Static analyzers work on mo dels of the program and its p ossible executions The most familiar static analyzers are incorp orated into compilers Compiler syntax analyzers build and examine parse trees and determine the presence or absence of syntactic faults More ambitious compilers and similar standalone analysis to ols determine the presence or absence of certain semantic faults eg the PFORTFortran Analyzer Ryd and the Lint C analyzer More p owerful static analyzers build increasingly sophisticated program mo dels and employ increas ingly p owerful mathematics to demonstrate the absence of wider classes of faults Thus dataow analyzers OF DC construct annotated owgraphs to represent program execution b ehaviors and then com pare them to regular expressions that describ e sequences of events that represent desired behaviors Exp erience suggests that these mo dels help analysts study such qualities as safety and robustness but are less helpful in studying functional prop erties Formal verication AGB Bjo uses such for malisms as predicate calculus to represent program functional behaviors Inference techniques such as symb olic execution Kin Cla then develop mo dels of program functionality that are then compared to desired functional b ehavior sp ecications to determine the presence or absence of functional faults While these analyses can demonstrate the absence of some classes of faults it is imp ortant to note that they cannot determine the absence of all kinds of faults it is also