USER'S GUIDE Version 6.4 New editions of this guide incorporate all material added or changed since the previous edition. Update packages may be used between editions. The manual printing date changes when a new edition is printed. The contents and format of this manual are subject to change without notice. Generated: 10/1/2017, 8:42 PM Rev: cb0043d Part Number: User's Guide for VectorCAST/Manage v.6.4 VectorCAST is a trademark of Vector Software, Inc. © Copyright 2017, Vector Software, Inc. All rights reserved. No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any informational storage and retrieval system, without written permission from the copyright owner. U.S. Government Restricted Rights This computer software and related documentation are provided with Restricted Rights. Use, duplication or disclosure by the Government is subject to restrictions as set forth in the governing Rights in Technical Data and Computer Software clause of DFARS 252.227-7015 (June 1995) and DFARS 227.7202-3(b). Manufacturer is Vector Software, Inc. East Greenwich RI 02818, USA. Vector Software reserves the right to make changes in specifications and other information contained in this document without prior notice. Contact Vector Software to determine whether such changes have been made. Third-Party copyright notices are contained in the file: 3rdPartyLicenses.txt, located in the VectorCAST installation directory. 2 Table of Contents INTRODUCTION 6 VectorCAST/Manage Overview 7 The VectorCAST/Manage Project 9 WORK FLOW SCENARIOS 12 Using Manage in Your Organization 13 Single User Setting Up an Initial Manage Project 14 Create An Empty Project 14 Customize the Configuration 14 Create a New Environment 15 Add Test Cases 16 Sharing a Manage Project With Your Team 19 Put a Project Under Source Code Management 19 Add New Unit Test Environment 19 Create Automatic Regression Test Script 20 Performing Change-Based Testing 22 Modify Source Code 22 Perform Incremental Build/Execute 22 Testing with Multiple Configurations 24 Add a New Compiler Configuration 24 Build / Execute the New Compiler 24 Add a New Environment to the Configuration 25 Sharing Tests and Results Between Multiple Users 27 Import Results from Master Project 27 Fix Test Failures Locally 28 Commit Changes to Source Code Management 30 Using Imported Results With Change-Based Testing 32 Import Results 32 Make Modifications 32 Perform Incremental Rebuild 33 Sharing Testing Trends Using Reports and Graphs 34 Store Results and Generate Report 34 Modify Graph Options 34 Building a Manage Project From Existing Environments 37 Create a New Project 37 Managing Configuration Options for Multiple Environments 40 Set Up Common Configuration Options 40 Clear Elevated Configuration Options 41 3 Add a New Test Suite Node and Reconfigure Options 42 Creating a Compiler Node Using an Existing .CFG File 44 Load the .CFG File 44 Open the Configuration Editor 44 Change-Based Testing with VectorCAST/QA 46 Using the Python Script 46 Incremental Build and Execute 49 VECTORCAST/MANAGE TOOL REFERENCE 52 The Manage Project Structure 53 The project.vcm File 53 Project Directory 53 Environment Directory 53 Using the Manage Interface 55 Understanding The Management Tree 55 Understanding the Files Tab 61 The Project Editor 70 The Messages Window 70 Using the Jobs Window and Job Monitor 73 The Jobs Window 73 Opening the Job Monitor 74 Job Status Viewer 75 Manage Status Viewer 77 Project-Related Tasks 80 Creating an Empty Manage Project 80 Setting Default Options for Your Project 80 Creating a New Unit Test Environment 82 Creating a Manage Project From Existing VectorCAST Environments 84 Opening a Manage Project 87 Option Factoring in the Project Wizard 88 Building and Executing Tests 88 Configuration-Based Test Cases 90 Target Setup and Teardown 93 Opening a Test Environment 95 Automating Test Script Maintenance 96 Storing Results 101 Using the Project Update Editor 102 Editing and Modifying Configurations 105 Creating Specialized Environment Configurations 110 Adding a Compiler 113 Add an Environment to the Project Tree 115 Disabling/Enabling an Environment 117 Renaming an Environment 118 Using Monitored Environments in a Manage Project 119 Understanding the Project Lock 120 Integrating VectorCAST/Manage With Your SCM System 122 4 Understanding Workspaces 122 Working With Imported Results 124 Manage Reports 129 Trend Reporting 129 Summary Report 134 Code Coverage Summary 139 Original Source Coverage Report 142 Manage Data Summary 146 Change-Based Testing 151 VectorCAST/QA - System Test Automation 152 The Python Configuration Script 152 Add a Test Case Using Python Configuration Script 153 Add a Manual Test 154 Incremental Build and Execute 156 Interactive Execution of Tests 158 Component Coverage 160 Change Impact Report 164 Python API 167 Manage Python Global Functions 167 Manage Python Classes 167 Exception Errors 176 Example Python Script 179 VectorCAST/Manage and Jenkins Integration 181 COMMAND LINE INTERFACE 182 Introduction 183 Running a Script 183 Running a Manage Script With Multiple Commands 184 The Manage Tutorial Revisited 186 Creating the Project 186 Creating an Environment Group 187 Creating Source, Platform, Compiler and Test Suite Nodes 188 Importing Test Environments 188 Adding an Environment Group to a Test Suite 190 Building and Executing the Project 190 Storing, Listing, and Deleting Results 191 Generating Reports 192 Duplicating a Test Suite 194 Setting Coverage Type on a Test Suite 195 Example Script (.bat) 197 INDEX 199 5 Introduction VECTORCAST/MANAGE OVERVIEW 7 VECTORCAST/MANAGE OVERVIEW VectorCAST/Manage is an extension of the VectorCAST family of unit and integration testing tools. VectorCAST/Manage allows tests and test results to be shared across the enterprise, providing a single point-of-control for all unit- and integration-test activities. At-a-glance logs, summary reports, and color- coded pass/fail criteria highlight the status of each test within the regression suite. Within a Manage Project, Test Environments can be grouped into larger “Environment Groups” and “Test Suites.” Environments can be members of multiple Environment Groups, and Environment Groups can be assigned to multiple Test Suites. This enables users to structure their VectorCAST/Manage project to match the architecture of their application. For example, the application sub-systems will map onto Manage Test Suites or Environment Groups, and individual source files will map onto Manage Test Environments. Because Environment Groups and Test Suites can be easily duplicated, the same tests can be run using various source baselines, on different host platforms, or with a different compilers. The integrated SQL database and graphing facility in VectorCAST/Manage enables users to view historical trend data for an individual software component, or any group of software components. This makes it easy to analyze regression trends across the software-testing life cycle. VECTORCAST/MANAGE OVERVIEW 8 The Manage Status Panel and HTML reports enable users to view the current status of each test case. Data is automatically recorded for build status and duration, test execution status and duration, and code coverage achieved. Using the integrated Python interpreter, additional comparisons can be added for each component and a column for the comparisons added to the report. For example, a user might want to compare the test execution time to some threshold or might want to run a static analysis tool such as “Lint” on the source files in the Manage project. The Test Execution Module allows tests to be distributed across multiple physical or virtual servers, with all results from the remotely executed tests incorporated in the VectorCAST/Manage Database. The VectorCAST/Manage project allows multiple users simultaneous access to run tests and to view status and results. The entire development team can use VectorCAST/Manage. l Program Managers use the high-level reports and graphs to track testing progress. l QA Managers use the status panel and reports to track release readiness. l QA Engineers control multiple test configurations and test scheduling. l Software Engineers create new tests, and identify and fix bugs. l Software Architects control the over-all design of the Manage project as well as controlling the peer review and change control process. l Engineers use the tool to easily design test campaigns and monitor release readiness. THE VECTORCAST/MANAGE PROJECT 9 THE VECTORCAST/MANAGE PROJECT A VectorCAST/Manage project is the set of all files, directories and test components managed and controlled by VectorCAST/Manage. A Manage project is made up of the project file (yourprojectname.vcm) and the project directory. The project file is an XML file and is typically kept under source code control. The project directory is the directory in which Manage stores the files to build and execute environments and the SQL historical database. It is named the same as the project. A Manage project can be created from a set of existing VectorCAST Test Environments, using the New Project Wizard, or as an empty project. The interface of a typical VectorCAST/Manage project is shown below. THE VECTORCAST/MANAGE PROJECT 10 The project tree for VectorCAST/Manage is displayed in the upper left hand corner of the application window. It depicts a hierarchical view of the components of the project as a set of nodes on a tree. The following is a list of Project Tree nodes from the root or top-most level to the leaves or bottom-most level: l The root node is the overarching container that contains all the other nodes within a project. There is a single root node for each project. Its name reflects the name of the project. l The source node represents all of the code for a given baseline to be tested. l The platform node represents the platform of the data sources for the environments under that node.