Jtest Tutorial JtestTutorial Tutorial Tutorial Welcome to the Jtest Tutorial. This tutorial walks you through how to per- form common Jtest tasks using example files. Please note that although the four types of tests (static analysis, white-box testing, black-box testing, and regression testing) are dis- cussed separately, Jtest can perform all of these tests with just one click of the Start button. This tutorial contains the following lessons: • “Lesson 1 - Performing White-Box Testing” on page 2 • “Lesson 2 - Performing Static Analysis” on page 13 • “Lesson 3 - Performing Black-Box Testing” on page 23 • “Lesson 4 - Performing Regression Testing” on page 51 • “Lesson 5 - Testing a Set of Classes” on page 54 • “Lesson 6 - Using User-Defined Stubs” on page 59 • “Lesson 7- Using JUnit Test Classes with Jtest” on page 69 • “Lesson 8- Using Jtest in a Multi-User Environment” on page 79 1 Lesson 1 - Performing White-Box Testing Lesson 1 - Performing White-Box Testing White-box testing checks that the class is structurally sound. It doesn't test that the class behaves according to the specification, but instead Tutorial ensures that the class doesn't crash and that it behaves correctly when passed unexpected input. White-box testing involves looking at the class code and trying to find out if there are any possible class usages that will make the class crash (in Java this is equivalent to throwing an uncaught runtime exception). Jtest completely automates white-box testing by automatically generating and executing test cases designed to fully test the class. During the test, Jtest executes the class using a symbolic virtual machine, then searches for uncaught runtime exceptions. For each uncaught runtime exception that is detected, Jtest reports an error and provides the stack trace and the calling sequence that led to the problem. This lesson contains the following sections: • “Performing White-Box Testing: Overview” on page 2 • “Performing White-Box Testing: Example” on page 3 • “Viewing Automatically Generated Test Cases” on page 6 • “Suppressing Exceptions” on page 7 • “Viewing a Text or HTML Format Report” on page 7 • “Setting an Object to a Certain State” on page 8 Performing White-Box Testing: Overview To perform white-box testing, just tell Jtest what class or set of classes to test then click the Start button. If you test a project, results will be displayed in the Class Name> Errors Found> Uncaught Runtime Exceptions branch of the Project Testing UI's Results panel. 2 Lesson 1 - Performing White-Box Testing If you test a single class, results will be displayed in the Uncaught Runt- ime Exceptions branch of the Class Testing UI’s Errors Found panel. Tutorial Performing White-Box Testing: Example 1. Go to Jtest’s Class Testing UI. (This UI opens by default when you launch Jtest). 2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test. 3. Browse to Simple.class (in <jtest_install_dir>/examples/eval) using the Browse button in the Class Name panel. 4. Click the Start button in the tool bar. Jtest will perform static and dynamic analysis on the class. A dialog box will open to notify you when testing is complete. Information on test progress will be displayed in the Test Progress panel. Problems found will be reported in the Errors Found panel. 3 Lesson 1 - Performing White-Box Testing Tutorial You will see two types of errors reported in the Errors Found panel. For this example, we're going to focus on the uncaught runtime exception found. The Errors Found panel will list the following uncaught runtime exception: This error message reveals that there is some input for which the class will throw an uncaught runtime exception at runtime. This could cause the application running this class to crash. To see a stack trace like the one the Java virtual machine would give if this uncaught runtime exception were thrown, expand this branch. 4 Lesson 1 - Performing White-Box Testing To see an example usage of this class that would lead to the reported uncaught runtime exception, expand the Test Case Input branch. Tutorial The error shows us that the "startsWith" method is implemented incor- rectly. The method should return false for the argument "" and "0" instead of throwing a runtime exception. If the error is not fixed, any application using this class will eventually crash or give incorrect results. To view the source code of the class (with the problematic line of the stack trace highlighted), double-click the node containing the exception's file/line information. 5 Lesson 1 - Performing White-Box Testing Tutorial Viewing Automatically Generated Test Cases To see a selection of the test cases that Jtest automatically generated for this test, click the View button in the tool bar. The View Test Cases win- dow will open. In the View Test Cases window, expand the Automatic Test Cases branch. This will display a list of this class’s methods. Click any method with a number greater than zero by it, and open the Test Cases branch, or press Control and click your right mouse button, then choose Expand Children from the shortcut menu that opens. The inputs that Jtest gener- ated for each method are now displayed. 6 Lesson 1 - Performing White-Box Testing Suppressing Exceptions You can tell Jtest to suppress uncaught runtime exceptions that you do Tutorial not expect to occur or that you are not concerned with by adding Design by Contract tags to your code. To have Jtest suppress errors for inputs that you do not expect to occur, use the @pre tag to specify what inputs are permissible. To have Jtest suppress expected exceptions, use the @exception tag to specify what exceptions you want Jtest to ignore. For example, if you wanted to suppress reports of an expected Negative- ArraySizeException that occurs when a negative index is used as an index to an array, you might enter the following comment above the appropriate method: /** @exception java.lang.NegativeArraySizeException */ This not only tells Jtest to ignore the exception, but it also makes code easier to understand and maintain. If you use the @pre tag to indicate valid method inputs, and then use ParaSoft’s Jcontract to check Design by Contract contracts at runtime, you will automatically be alerted to instances where the system passes this method any unexpected inputs. Viewing a Text or HTML Format Report Jtest automatically creates a report for each test. You can view this report by clicking the Report button. By default, this report is formatted in text (ASCII) format. If you would like Jtest to generate HTML reports (e.g., if you want to post the report on your development intranet), choose Prefer- 7 Lesson 1 - Performing White-Box Testing ences> Configuration Options> Report Format> HTML before you click the Report button.. Tutorial By default, the report file will be saved in <jtest_install_dir>/u/user- name/results. To prompt Jtest to save reports to a different location, mod- ify the Common Parameters> Directories> Results parameter at the Global, Class, or Project level. Setting an Object to a Certain State In some cases, you may want to set up an initial state prior to testing a class. For example, suppose that a class is used as a global object con- 8 Lesson 1 - Performing White-Box Testing taining static member variables accessible by any other project within the application. When Jtest tests an object that uses this static member vari- able, a NullPointerException will result because the variable has not been Tutorial set. This problem can be solved by giving Jtest initialization code. For example, the file ExecGlobal.java (in <tutorial_install_dir>/lesson1/ExecGlobal.java) contains a static variable that is assigned by its ancestor Exec, which is defined in the Exec.java file (also in <tutorial_install_dir>/lesson1/Exec.java). In this example, the instantiation on the Exec object is performed by the Exec object's "main" method, which is defined in the Exec.java file. When Jtest tests the ExecTest object's doTest method, it does not know that a certain state is assumed by the doTest method. Thus, initialization needs to be per- formed. To see why initialization is necessary: 1. Go to Jtest’s Class Testing UI. (This UI opens by default when you launch Jtest). 2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test. 3. Use the Browse button in the Class Name panel to choose ExecTest.class (in <tutorial_install_dir>/lesson1/ExecTest.class) 4. Test the class by clicking the Start button in the Class Testing UI tool bar. A NullPointerException is reported because the assumed state is not available. You will need to perform static initialization on the ExecTest class to ensure the objects referenced by this class are in the assumed 9 Lesson 1 - Performing White-Box Testing state. Tutorial To perform the necessary initialization: 1. Click the Class button to open the Class Test Parameters win- dow. 2. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> Common. 3. Double-click the Static Class Initialization node. 10 Lesson 1 - Performing White-Box Testing Tutorial The Static Class Initialization window will open. 4. Enter the following initialization code in the Static Class Initializa- tion window. new Exec (); This will create an instance of the "Exec" object before the class "ExecTest" is tested. 11 Lesson 1 - Performing White-Box Testing Tutorial 5.