T17 Concurrent Session Thursday 05/08/2008 3:00 PM
The Promise of Model-Based Testing
Presented by:
Mahesh Velliyur Maveric Testing Solutions
Presented at: STAREAST Software Testing Analysis & Review May 5-9, 2008; Orlando, FL, USA
330 Corporate Way, Suite 300, Orange Park, FL 32043 888-268-8770 904-278-0524 [email protected] www.sqe.com
Mahesh Velliyur Executive Director and Co-Founder, Maveric Systems Ltd.
Mahesh Velliyur is an Executive Director and co-founder of Maveric Systems. He has over 20 years of experience in the IT industry in the area of IT Strategy, architecture, design and implementation of IT solutions to the private and government sector.
After serving premier consulting and technology organizations, Mahesh started his first entrepreneurial venture in 1992. He has wide ranging experience in delivering complex development and testing solutions to clients in Europe and US through innovative models of delivery.
At Maveric, apart from being the architect of Maveric's proprietary testing frameworks and methodologies, Mahesh heads delivery operations of Maveric across the various delivery centers. He takes accountability for all offshore delivery in Maveric.
With Maveric rapidly becoming multi-locational in its delivery, Mahesh’s primary challenge is ensuring that clients perceive the same level of responsiveness and capability in Maveric teams across various delivery locations. Mahesh and his team also ensure that we are constantly innovating and improving our testing methods and processes to significantly enhance the quality of our client’s applications.
Mahesh holds the belief that test automation will be the one area in testing where we will see the most rapid and radical changes in the coming years. He has therefore created a dedicated Centers of Excellence for Test Automation in Maveric Systems. The automation CoE invests in creating proprietary, re-usable and tool independent automation frameworks to support Maveric’s clients in the design, execution and management of tests. The CoE also focuses on evaluating best of breed automation tools across the product lifecycle to deliver cost effective and maintainable solutions to our clients.
Mahesh holds an MBA from XLRI, India and a Bachelors in Engineering from the Indian Institute of Technology, Madras. Promise of Model Based Testing
New Jersey Mumbai Chennai Bangalore Dubai London New Jersey Mumbai Chennai Bangalore Dubai London New Jersey Mumbai Chennai Bangalore Dubai London
Maveric Systems confidential Coverage
Background
Automation Landscape
Test design – A perspective
Model based testing – A positive influence
UAT Challenges – Promise of MBT
Design process overview – Retail banking Background
Software Testing - Relatively nascent as an independent industry
Key drivers – over past 10 years
Early years - need for objectivity / independence
Mid Years – domain / vertical competencies, regression & repeatability
What now ? What now?
Productivity improvement across the testing lifecycle,
achieved through
Process improvement initiatives (Alignment to TMM, TPI etc)
Automation initiatives Automation Landscape
Test Defect Test Planning Test Design Test Automation Test Closure Execution Mgmt.
Scoping & estimation tools Test automation tools Takes up to 50-65% of time in Environment preparation tools Testing Test Construction – simulation tools, integration stubs Most significantly impacts the quality Test management tools of testing Defect management tools Currently very limited automation Data generation tools SOA testing tools A Model based design Configuration & Version control tools Build & release management tools approach is the focus of
this presentation Test Design – A Perspective
Good design takes up to 50-65% of time in the testing life cycle
Difficult to standardize a structured approach to design
No scientific method to ensure comprehensiveness – no optimization on the
number of test conditions
Increased dependency on the Individual’s expertise and experience.
Process is not documented; therefore not Auditable.
Warrants a high understanding of the domain among all team members or high
dependence on business users Model Based Testing – in the right direction
Key Benefits Model • An independent basis for based verification testing Boundary Pre/Post • Objective coverage Value
assessment Equivalence Transition Application partitioning Model • Scientific mechanism to of design definition achieve desired focus on Functional techniques Pair-wise specific areas Data-flow Rule of ‘n’ • Ability for the business user to relate to process Design pack and outputs generation
Maveric has been leading extensive research over the last two years in adoption of Model based Testing for UAT UAT Design Automation – A Wish List
A generic model pertaining to the domain; allowing user customisation DOMAIN
Automated generation of test cases Principal component using a repertoire of testing principles Ancillary components applied on the above model resulting in
Coverage predictability
Duplication avoidance / reduction Greater Test Data accuracy Testing Test Better Test results (Oracles) Principles Management identification Boundary Value, Run-plan, Functional Better execution planning and Equivalence Partitioning Decomposition, prioritisation etc. Control Reports etc.
Focus is on “what to test” and “how much to test” UAT Challenges
Coverage Predictability
Duplication avoidance / reduction
Ensuring Greater Data Accuracy
Identification of Expected results
Execution Planning / Prioritisation
Needs a solution that uses and extends Model Based Testing UAT Challenges - Coverage Predictability
Through well defined business models encompassing
a. Definition of Product Groups, Products and Lifecycle Events
For example, the following might be relevant in the Retail Banking area: UAT Challenges - Coverage Predictability
Through well defined business models encompassing b. Product / Transaction rules Definition
For example, a loan cannot be sanctioned when the Nature of Borrower is ‘Joint – Greater than Maximum’ Attributes of Relevance Business Rules
Type of Interest Loan Nature of Interest Curre Interest Statu Borrower Base Product Borrowers Rate ncy Base s s Definition Attribute Value Individual Home Single Rack rate Yen Actual Sanctioned Balance Amount Nature of Borrowers Joint – Greater than Fail Maximum Corporate Auto Joint – Less than Negotiate USD Expected Disbursed Maximum d Rate Balance Amount Personal Joint – Equal to Preferred GBP Outstandin Maximum Rate g Principal
Joint – Greater AED than Maximum
Euro
Type of Loan Nature of Interest Base Product Def ID Interest Rate Currency Interest Base Status Borrowers Product Borrowers Definition
LN01 Individual Home Joint – Greater Rack rate Yen Actual Sanctioned FAIL than Maximum Balance Amount UAT Challenges - Coverage Predictability
Through well defined business models encompassing
c. Definition of business flow related rules
These would include negative rules related o sequencing of transactions as well as definition of commonly used business flows
For e.g.: In Loans, an institution may permit assignment of loans only after the drawdown is complete .
Assignment Timing :
Value Status Before Setup FAIL After Setup, Pre-Drawal FAIL Post-Drawal, before repayment PASS Post Drawal and repayment FAIL Post closure FAIL UAT Challenges - Coverage Predictability
Through well defined business models encompassing
d. Capturing Intuitive intelligence - For example, you may wish to test each Loan Product for each type of borrower based on past experience even though the same is not technically necessary Attributes of Relevance Distribution Rules
Type of Loan Product Nature of Borrowers Interest Rate Currency Borrowers Type of Borrowers Loan Product Individual Home Single Rack rate Yen 1 2 Corporate Auto Joint – Less than Maximum Negotiated Rate USD
Personal Joint – Equal to Maximum Preferred Rate GBP
Joint – Greater than Maximum AED
Product Nature of Borrowers Currency Interest Rate Status Type of Borrowers Loan Product Definitions LN 01 Individual Auto Single Yen Rack rate LN 02 Individual Home Joint – Less than USD Negotiated Rate Maximum LN 03 Individual Personal Joint – Equal to GBP Preferred Rate Maximum CLEAR LN 04 Corporate Auto Single AED Rack rate LN 05 Corporate Home Joint – Less than Euro Negotiated Rate Maximum LN 06 Corporate Personal Joint – Equal to Yen Preferred Rate Maximum UAT Challenges - Coverage Predictability
Through Consistent application of testing Principles
a. Equivalence partitioning
For example, the following set presents the Equivalence partitioning of the Tenor in Loan setup; optimised to cover partitioning across other attributes as well:
b. Boundary value analysis
For example, the Amount boundary for a cash deposit in a Savings Account is Rs. (10,000 - 15,000) UAT Challenges – Coverage Predictability
By Enabling Tracking at various levels Functional Decomposition Reports enable the users in reviewing outputs generated at various levels - helps in tracking changes in the outputs at various stages Business Scenarios
Control Reports UAT Challenges – Duplication Avoidance
Application of Rule of N Duplication occurs when the product level attribute values are independently considered for testing without an attempt to combine them. The following example explains how this can be restricted at Product definition – Using Rule of N we can cover all 25 attribute values in 5 conditions Product setup definitions
Below is the Product Matrix (Test Conditions) without the duplication of the definitions UAT Challenges – Duplication Avoidance
Application of Orthogonal Array The following example explains how this can be restricted at a Transaction level. Using Orthogonal Array (pairwise) we can combine the 4 attributes with 10 attribute values into 10 conditions ensuring all pair combinations are covered
Transaction: Drawdown definitions
Below is the Transaction Matrix implementing a pairwise algorithm superimposing business rules
Drawdown ID Drawdown Method Drawdown Status Drawdown Mode Drawdown Currency Status Drawdown 1 Single Partial Draft USD Pass Drawdown 2 Multiple Full Draft GBP Pass Drawdown 3 Multiple Partial Account Transfer USD Pass Drawdown 4 Single Full Account Transfer GBP Pass Drawdown 5 Single Partial Cheque INR Fail Drawdown 6 Multiple Full Cheque USD Pass Drawdown 7 Multiple Full Draft INR Fail Drawdown 8 ~Single Partial Account Transfer GBP Pass Drawdown 9 ~Multiple ~Partial Account Transfer INR Fail Drawdown 10 ~Single ~Full Cheque GBP Pass UAT Challenges – Duplication Avoidance
Reducing set-up activities by combining product and lifecycle conditions Duplication occurs due to set-up requirements necessary to execute particular conditions that occur in latter stage lifecycle processes. Following example to test a condition relating to prepayment of a loan, it would be necessary to setup a loan and make a drawdown. Here the pre-payment is attached to an earlier Loan set-up and drawdown that have already been created for an earlier condition Work Flow - Loans Scenario to test the Prepayment after Drawdown Scenario Status PASS Scenario ID SCE01 Product Setup ID LOAN 5 Transaction ID DRAWDOWN 2 PREPAYMENT 1 PRODUCT SETUP DETAILS DRAWDOWN DETAILS EXPECTED RESULTS Contract Referance number M33100 Drawdown Method Multiple Loan-products Home Value date of the Transaction 10-May-07 Credit a/c number SB1499 Drawdown Status Full Debit a/c number CA1999 Loan currency USD Amount 3000 Currency USD Dradown Mode Account Transfer Pre payment Charges Reasons charges Applicant type Individual Mode of By cash Payment Drawdown Currency USD GL Entries Type of Customer Bank Customer Profit GL NA Drawdown Amount 50000 Cash Credit GL 3000 Segments Salaried PREPAYMENT DETAILS Disbursement Mode Banker's Cheque Prepayment Amount Full Minimum term from Repayment Mode SI-Direct Debit Qualifying Criteria Opening - equivalent to EI(Interestl) + Equivalent to minimum Repayment Method Prepayment Basis Bullet(Principal) percentage
Tenor (in months) Between Min and Max
Loan Amount 100000 UAT Challenges – Duplication Avoidance
Separation of Expected results from Rules Duplication occurs in case of Fees and Charges, where one has to develop separate set of scenarios, without an attempt to integrate them with the existing scenarios. Following example on optimization at a module level Fee setup definitions
Fee rule UAT Challenges – Duplication Avoidance
Separation of Expected results from Rules
This happens across modules as well – For example in case of GL Validations and Batch Processing validations, when they are not getting integrated with actual scenarios. UAT Challenges – Greater Test Data Accuracy
Through pre-definition of critical attributes
Attributes critical to the product and transaction definition may be defined up-front so that the Rule of N / Pairwise / Orthogonal array
techniques may be applied on them. Data values for them can therefore be automatically generated as part of the test conditions as
shown below. Product setup definitions
Below is the Product Matrix (Test Conditions) with values filled in for each attribute UAT Challenges – Greater Test Data Accuracy
Through parameterisation of variable values
Certain attributes require to take values based on user specific definitions. For example, the ‘Tenor’ attribute needs definitions of
Minimum and Maximum values. Such parameters may be pre-identified for each domain for structuring necessary user input
Product setup definitions
Below is the Product Matrix (Test Conditions) with specific values filled in for the ‘Tenor’ attribute
8
75
30
12 (min)
60 (max) UAT Challenges – Greater Test Data Accuracy
Through mapping of the scenarios to the Application Under Test
This is to address screen flows, nomenclature changes to attributes and additional attributes that may be required for the product / transaction definition
Mandatory fields
Data Dependencies Dependency not filled for scenario 3 Optional fields
Controls reflects Red System generated fields color as one of the dependencies is not Data Dependencies filled for scenario 3 UAT Challenges – Expected Results
By associating multiple test oracles to each condition During the conduct of test execution of the scenarios, each test condition might warrant verification of multiple expected results at different locations and triggers. For example -
Scenario to test the Scenario to test the Scenario to test the Loan Closure with WORK FLOW MATRIX - LOANS Loan closure with Early Full Settlement Prepayment qualifying Prepayment criteria is not met. Scenario ID SCE01 SCE02 SCE03 Scenarios marked in Red are meant for Scenario Status Pass Pass Fail testing the Negative Product Definition ID LN01 LN02 LN03 scenarios Loan-products Home Personal Home Loan currency USD USD INR Repayment method EI (Principal + EI (Principal) Bullet Interest) Transaction Definition ID DRAW02 DRAW02 DRAW01 PREPAY01 REP01 Loan Setup Screen List of attributes…..
Prepayment Screen List of Attributes …….
Expected Results Online Validation Status Pass Pass Fail Value date of the Transaction 10-May-07 10-Jun-07 10-Jul-07 On EOD of Value Date – Fees & Charges Report Charges Amount 3000 750 1500 Currency USD USD USD Reasons Pre payment charges Early Full Settlement Processing Charges On EOD of Value Date – GL Report of relevant Accounts Debit A/C SB1499 / Nil SB1500 / Nil SB1501 / Nil Credit A/C CA1999 / 3000 CA2000 / 750 CA2501 / 1500 UAT Challenges - Execution Planning & Prioritization
By minimizing Logical Days Schedule the tests in a way that the number of logical days required to execute the tests would be minimized. This would need a mechanism to determine appropriate transaction dates for each condition
Creating transactions 3 days Value dated 2 days Batch processing 5 days Frequencies 4 days Holiday 1 day Checking the Holiday transaction 1 day Total Logical days – 16 days Run Plan with negative scenarios sequenced first
Prioritizing of Test Conditions: 1. Negative Conditions relating to – Modules, Holiday treatment, Value dated transactions & Frequencies. 2. Positive Conditions relating to – Modules, Value dated transactions, Batch processing & Frequencies UAT Challenges – Execution Planning & Prioritization
By Appropriate Categorization Categorizing of the Test conditions helps in providing clarity with respect to Test execution. During execution the defects can also be categorized with respect to Base functionality, Interfaces or Enhancements, thus aiding in prioritization during regression testing .
Run Plan with all scenarios of Base functionality, Enhancements & Interfaces
Module level Enhancements
Base Functionality
Interfaces (Inter Module and Intra Module) Design Process Overview – Retail Banking
Phase Process Components Outputs
Review Interact with Boundary Definition Questionnaire baseline business administration documents users
Product matrix Matrix Definition Product Maintenance Product rules Transaction maintenance Transaction rules definition definition rules definition matrix matrix
Scenario generation Scenarios Fees & Charges Process Flow Link Rules definition definition definition Workflow matrix
Run Plan Generation Batch Processing Calendar Run Plan with rules definition definition dates & values
Analytics Reporting and Control Reports Coverage Analytics & metrics Analytics Next Steps
Visit us at the Stall OR Contact us at [email protected] [email protected]