SOFTWARETESTINGTOOLS PenttiPohjolainen DepartmentofComputer ScienceandApplied Mathematics UniversityofKuopio March2002 2 CONTENTS 1Introduction ............................................................................................. 5 2References............................................................................................... 10 3DivisionoftheTools .............................................................................. 11 3.1TestDesignTools.........................................................................................................11 3.1.1TestCaseTools ...........................................................................................................11 3.1.2DatabaseTools.............................................................................................................12 3.1.3DataGenerators...........................................................................................................13 3.1.4GeneralTestDesign ...................................................................................................14 3.2GUITestDrivers...........................................................................................................16 3.3LoadandPerformanceTestingTools.................................................................25 3.4TestManagementTools ............................................................................................32 3.4.1CORBA .........................................................................................................................32 3.4.2C/C++.............................................................................................................................33 3.4.3Others.............................................................................................................................36 3.5TestImplementationTools ......................................................................................48 3.5.1Java .................................................................................................................................48 3.5.2C/C++.............................................................................................................................51 3.5.3Others.............................................................................................................................53 3.6TestEvaluationTools.................................................................................................58 3.6.1Java .................................................................................................................................58 3.6.2C/C++.............................................................................................................................60 3.6.3Others.............................................................................................................................65 3.7StaticAnalysisTools ..................................................................................................69 3.7.1Java .................................................................................................................................69 3.7.2C/C++.............................................................................................................................71 3.7.3Others.............................................................................................................................75 3 4 1Introduction Thisworkstartedfromthesubjectofmyprograduthesis“Thenewestmethodsandtoolsfor softwaretesting”.Afteralongsearchtherewerenearly600(sixhundred)toolsfound.Iamsure, thattherearetoolsmuchmorethanthesenowoccurred.Averygoodaidtomewasthelistin Internet(www.testingfaqs.org),whichBrianMarickmadefamousandisnowmaintainedbyDanny Faught.OthersourceshavebeenInternetoverall,thebrochuresoftheproductsandtheliterature. Becausetheamountofthetoolswassolarge,Ihadtorestrictthemandtotakeuponlythemost interestingonce.Thedivisiontothemaingroupswas:Design,GUI(GraphicalUserInterface), LoadandPerformance,Management,Implementation,Evaluation,StaticAnalysisandoutsideof inspection:DefectTracking,WebSitesandMiscellaneous. Thelimitsbetweenthegroupsareoutoffocus,becausetherearemanytools,whichcanbelongto severalclasses(SeeFigure1.). Design StaticAnalysis Implementation Management GUI Evaluation Loadand Performance Figure1.TheDivisionofthetools 5 Ashortdescriptionofthemaingroups: TestDesignTools Toolsthathelpyoudecidewhattestsneedtobeexecuted.Testdataandtestcase generators. Total15tools. GUITestDrivers Toolsthatautomateexecutionoftestsforproductswithgraphicaluserinterfaces. Client/servertestautomationtools,includingloadtesters,alsogohere. Total28tools. LoadandPerformanceTools Toolsthatspecializeinputtingaheavyloadonsystems(especiallyclient-server systems).ThesetoolsareoftenalsoGUItestdrivers. Total21tools. TestManagementTools Toolsthatautomateexecutionoftestsforproductswithoutgraphicaluserinterfaces. Alsotoolsthathelpyouworkwithlargetestsuites. Total43tools. TestImplementationTools Miscellaneoustoolsthathelpyouimplementtests.Forexample,toolsthat automaticallygeneratestubroutinesgohere,asdotoolsthatattempttomakefailures moreobvious(assertiongenerators,etc.) Total27tools. TestEvaluationTools Toolsthathelpyouevaluatethequalityofyourtests.Codecoveragetoolsgohere. Total31tools. StaticAnalysisTools Toolsthatanalyseprogramswithoutrunningthem.Metricstoolsfallinthiscategory. Total33tools. Totalsum198tools. InPlugIT-projectinterestingtoolsareatleast:Regression,requirement,component,integration, object-oriented,coverage,testcaseandusecase. Searchingwiththesewordstherecent198toolstheresultswere: RegressionTools Regressiontestingtoolsareusedtotestsoftwareaftermodification.Dividingin groupsasabove(oneortwotypicalexamplespergrouparepresented)therewere: Design:None GUI:AutoTesterforWindows(No3)isspecificallymeanttosupportprojectteams inautomatingregressiontesting.Others4,6,12,15and27. LoadandPerformance:TeleprocessingNetworkSimulator(No20)canbeusedto automateregressiontesting.Others10and12. Management:TestManager(No2)providesaninteractivedevelopmentenvironment forworkingwithregressiontestsuites.OTF–OnObjectTestingFramework(No18) isatoolforSmalltalkobjects,inwhichregressiontestingisautomaticwithfull loggingofresults.Others1,4,5,10,14,16,27,28,29,37and38. Implementation:Junit(No5)isaregressiontestingframeworkusedbydevelopers whoimplementunittestsinJava.Others1,15and18. 6 Evaluation:Logiscope(No26)identifieseffectivenonregressionifprogramfiles havebeenmodified. StatisticAnalysis:ParaSoftJtest(No6)automaticallyperformsregressiontestingof Javacode. Total28tools. RequirementTools Requirement-basedorrequirementdefinitionrelatedtools. Design:Caliber-RBT(No1)isatestcasedesignsystemforrequirement-based testing.Others6and15. GUI:Panorama-2(No8)isatoolforrequirementanalysis.Another17. LoadandPerformance:SilkRealizer(No19)isatoolthatenablesusersdevelopeand deploysystemleveltestingsimulatingrealworldeventstoassurethatapplications willmeettherequirements. Management:AutoAdviser(No13)providesfromrequirementsthroughproductiona centralrepositoryfororganizingandmanagingbusinessrequirements,testsand associatedfiles.Others18,21,35and37. Implementation:None Evaluation:PanoramaC/C++(No15)isarequirementandtestcaseanalysistool. StaticAnalysis:None Total12tools. ComponentTools Tools,whichhavesomerelationshipswithcomponent-programming. Design:Inferno’s(No2)capabilitiesincludecreationofanintuitivelibraryof reusablecomponentsthatsupportshared-scriptsanddata-driven-scripts. GUI:None LoadandPerformance:None Management:SilkPilot(No1)letsyoutestthebehaviourofdistributedobjectswithin yourapplication’sservercomponents.AutoAdviser(No13)consolidatesyourtest librarycomponentsandprovidestestteammemberswithaccesstothosecomponents. Others2,32,40and42. Implementation:AssertMateforJava(No2)isasystemthataidsJavaengineersuse assertionstoensureproperimplementationofcomponentinterfaces.Another1. Evaluation:QC/Coverage(No16)helpsusersbyidentifyingcodecomponentsthat havenotbeenadequatelytested. StaticAnalysis:WhiteBox(No33)provideinsightintothecomplexityofdifferent componentsofthesoftware. Total11tools. IntegrationTools Toolsusedwithintegrationtesting. Design:ObjectPlanner(No13)allowssoftwaredevelopersandmanagerstocalculate theapproximatetimeschedulestoperformunitandintegrationtesting.Another15. GUI:SolutionEvaluationTool(No14)isusableintestingtheintegrationofnew applications. LoadandPerformance:None Management:Cantata(No3)isasolutionforunitandintegrationtestingofCand C++code.Others5,6,14,25,32and40. Implementation:Autotest(No14)isanautomatictestharnesstoolforintegration testing.Another15. Evaluation:Cantata++(No7)isaneffectivetoolforC++integrationtesting.Others 18and27. 7 StaticAnalysis:None. Total15tools. Object-orientedTools Toolsusedspeciallywithobject-orientedsystems.AllJavaandC++toolsfall automaticallyinthiscategoryalthoughthesearchhasnotfoundthemwithkeyword object. Design:T-VECTestGenerationSystem(No15)isintegratedinanenvironmentto supportthedevelopmentandmanagementofstructuredorobject-oriented requirementsspecifications.Another12. GUI:VermontHighTestPlus(No23)includesobject-levelrecord/playbackofall WindowsandVBcontrols.Others4,8,16,17,18and27. LoadandPerformance:None Management:TOOTSIE(No42)isatotalobject-orientedtestingsupport environment.Others1,2and18. Implementation:ObjectTester(No11)isasoftwaretooltoautomatethegenerationof C++unittestscripts.Others1and2. Evaluation:TotalMetricforJava(No1)isasoftwaremetricstooltocalculateand displayobject-orientedmetricsfortheJavalanguage. StaticAnalysis:ObjectDetail(No12)isasoftwaretooltohelpautomatethemetrics