Statistical Analysis Based Service Catalogue
Abstract
3. Introduction and Scope:
The testing industry is looking for innovative ways to optimize testing efforts and cost. One of the ways is through usage of tools and accelerators. Output Based Service Catalogue allows the customer to select only the services which he is interested in and pay only for the services consumed. This helps in transforming the organisation culture away from an FTE based to Consumption based model. The engagement model also gets standardised between Business & IT which significantly reduces lead times. Every service identified as part of catalogue will have standardised inputs, activities, outputs, and approvals mechanism. There will be predetermined service price, which helps the customer insulated from any effort fluctuations. Payment will be based on milestones which are linked to successful delivery of the expected outputs and not on effort spent by resources.
Online Estimation tool developed for the same purpose provides the estimates at a click of a button for the services requested by business. “R” statistical analysis tool is mapped to database from where it fetches the effort at sub tasks level from historical data and provides the range of effort required for any of the services selected from Catalogue.
Business benefits:
Transforms organisation culture away from an FTE based to Service Output based model / consumption based model, with clear ownership on both the requestor of the service & provider of services. Payment linked to successful outcome
Predictable quality, Service price & Outcome through standardized Input, Output & tasks
Engagement model standardisation between Business & IT significantly reduces lead times associated with testing request to estimation approvals
Service standardisation to achieve common understanding and measure of success in delivering testing services
Complete predictability of service cost and service quality standards
Continuous optimisation of services committed
Huge reduction in time, effort & cost to scope new service engagements
Accurate estimate through use of historical data and statistical analysis tool
Delivery risks transferred to service provider thereby driving the right behaviours from the partner organisation
Research/ Study content:
Many Tier1 companies are moving towards the service catalogue model due to the following business drivers:
Need “One-Stop Shopping” for business users
To establish standardized service with high quality outcome at a predictable price
Require optimized Service Delivery through reduction in operational time, effort & cost Better utilization of resources to effectively meet the demand from business users
Effort to provide the service should be derived automatically through effective use of historical data
1.1 Need for New Models Outsourcing is no more being looked at as just a cost advantage. Gone are the days when it was looked at as “Outsource and forget.” Companies are ready to invest time in working closely with outsourcing vendor to get a better control over management, get in-depth insights into supplier disruptions and thereby lower the risks. Companies are also willing to spend more provided they see a clear value. Trust and transparency play a crucial role in a successful outsourcing partnership.
Downturn in the global markets has forced many companies to relook at their outsourcing strategy. Traditionally, companies have been working on Time and Material Model wherein what was counted was the number of resources working on a project. But as outsourcing is getting more and more integral to the overall business strategy, companies are increasingly adopting for output based pricing in which billing is proportionate to the actual outcome delivered such as defects fixed, test cases designed, test cases executed, support incidents acted upon, impact on sales outcome and so on. This is enabling them to seek ROI on every dollar spent.
The earlier trend of time and material (T&M) based pricing caused many vendors to ignore productivity. Interestingly, low productivity meant higher revenue at times since the vendor could ask for more headcount to meet deadlines. With fixed price contracts and output based pricing, this is no longer the case.
Services companies now focus on increasing employee work efficiency and productivity more than ever. This visibility triggers a self-improvement cycle thereby increasing the work output. Work can be delivered with the existing team in lesser time. In the long term, a performance oriented culture benefits every company and project – whether your business model is based on T&M, output or fixed price
The Output Based service Catalogue (OBSC) will be a model for client to request and consume Services from a defined catalogue of fixed price service components.
The aim of the OBSC is to evolve the service engagement model from being an FTE- consumption model to a service output consumption model. This will include the establishment of a priced catalogue of standardised services against which client can request services to be delivered by the Service Provider.
The OBSC is intended to reduce the effort and time needed to scope new service engagements, and to further standardise the way in which Services are provided.
OBSC Model helps align clients’ and vendor’s stakes and their interests. It encourages both client and vendor to work together towards the common goal of improving efficiency and bringing in predictability
From a Client perspective, this allows them to convert fixed cost to variable cost, reduce the total cost of ownership and more importantly provides greater flexibility
From vendor perspective, it provides opportunity to monetize IP/solutions/productivity gains, challenges them to be more innovative in their solutions and service delivery
1.2 Why Output Based Testing Today, success is increasingly enjoyed by companies that partner with specialized companies providing intermediate services - to deliver end products to customers. Such collaborations achieve superior performance by exploiting the economies of specialization
that accrue to focused partners, by sharing the benefits of lower costs and by encouraging innovation. Migrating to the value partnership model requires understanding of how to work within a collaborative network setting and digitizing much of what we do into an outcome. Once a business process has been digitized into an outcome, and carved out from its traditional business context, the process of outcome based delivery model begins.
In terms of testing and validation services, output based models entails providing a spectrum of testing services such as Functional testing, Non Functional, Non Production Environment and test data management. The goal is to implement a testing model that is incentive compatible, and are value maximizing for both parties. While there are multiple models for pricing in testing arrangements, Let us look at the context and the key components of Output Based service Catalogue (OBSC):
There are four key variable components that will influence the sizing and pricing of a Service under an OBSC engagement model.
1. Project Profile – The project variables relating to the size and complexity of each project that consumes the Services.
2. Services Profile – The macro set of Services that are consumed by a project.
3. Service Components – The individual Service Components that form part of the overall service profile.
4. Activities and Services Performed – The detailed Services that are to be performed as part of the Service Components.
1.3 The business value of a Service Catalogue To ensure quality, timely and cost efficient request fulfillment, providers must establish a consistent source of available IT services through a Service Catalogue. Essentially, implementing a successful Service Catalogue will deliver benefits in three main areas.
Increased customer satisfaction by establishing realistic expectations
When making a decision about a service, a customer will set their expectations based on answers to some basic questions: What the service is and how they can avail it? How I can request for the services & for what type of applications? How do I use the service and whom do I reach out to for any support? Are there other additional components that I might find useful? How much does it cost? Are there options to reduce the cost, or could I pay more for an enhanced service? When can I expect delivery & what format?
An actionable Service Catalogue delivers value by enabling Line of business to set service provision expectations as well as determine how services should be consumed. A well designed, actionable Service Catalogue will also modify the service consumption behaviour
of customers to enable business to realize the efficiencies and cost savings of streamlined service provision.
Reduced costs, time and increased efficiency
Request management tends to be an inefficient and slow process in many organizations. A single, consistent Service Catalogue can deliver immediate value by ensuring control over “what can be ordered and who can order it”. There will be a set of 24 services from which the services can be ordered. The key thing to be noted is that the scope of the activity need to fixed and finalized and should not change over a period of time or once the work has started. As the effort required will be automatically provided by the online estimation tool (statistically analyzed data), the overall effort in understanding the scope and estimating the effort required to meet the deliverables are completely eliminated.
Enhanced reporting and metrics
A Service Catalogue offers the business a clear picture of “demand and financial management” around their IT organization and its service delivery: Monitor numbers of requests and fulfilment by service as well as LOB Milestone and SLA driven metrics reported on a regular basis Application/LOB wise Productivity
1.4 Structure of Output Based service Catalogue (OBSC)
The delivery services are split into 4 categories.
11 Functional Testing Services (Static Testing, Test Planning, ST, SIT, Automation, Virtualisation, UAT, UAT Support, PVT, Business Rules, Data Migration Testing);
7 Non-Functional Testing Services (PBA, Design Static Test, Performance Testing, PE, F&R, Operational Readiness*, Accessibility Testing*);
2 Non-Production Environments Management (Environments Management & Coordination, Environment Build & Deployment);
5 Test Data Management Services (Consulting, Data Profiling, Test Data Provisioning, Test Data Privatisation, Test Data Archive and Purge).
Sub-set of Engagement and Delivery Services with pre-determine service size & price will make OBSC.
1.5 How Each Service is defined?
1.6 Grouping of Services to move to OBSC
1.7 Service Catalogue Model-Evolution Journey
Stage 1: Establish Service Catalogue, Operating Model Transition to Consumption Based Model
Capgemini will work with client to establish a Service Catalogue Council (SCC) with participants from both organisations
Establish core testing Service Catalogue and make it available via online portal for the pilot phase
Setup the roadmap for the Service Catalogue in terms of processes and tools for changes/optimisation to the current offerings as well as addition of future services
Plan and manage organisation change, stakeholder impacts and gather support for the adaption of consumption based model
Stage 2: Pilot, Stabilise and Expand
Plan and conduct Pilot projects with testing engagement and service delivery via the Service Catalogue
During the stabilise stage, the SCC will prioritise based on the feedback and lessons learned from the pilot phase, focusing on enabling a consumption based service model including operations, governance and reporting. The SCC will make recommendations to the baseline Service Catalogue roadmap to further improve the Service Catalogue
The SCC will continuously monitor and evaluate business user feedback, conduct trend analysis of services on the Service Catalogue and plan for future services
The SCC will also aim to improve the processes and tools for the Service Catalogue to enable reporting and governance both for test services fulfilment but also for the realisation of business benefits through using the SC and consumption based model
Stage 3: Optimise/Steady State
The optimise stage will be an on-going activity and will extend the focus of activity from addressing known priorities to proactively defining the next levels of process efficiencies, extending automation further into the testing services and driving higher levels of centralisation, virtualisation and innovation
Service Catalogue estimates and costs will reduce over time with improved efficiencies
1.8 Pre determined Service Size and Price Example for a Project
The effort required to perform any activity at application level and for any particular phase is captured from the online estimation tool. The effort data received is after the statistical analysis done on the historical data for that application at phase level. This helps in elimination of the standard process of estimating the effort manually once the request is received from business on the portal. The detailed steps involved in analysis of the historical data are highlighted in subsequent sections.
Note: Not all services can be offered as Pre- determined Fixed Price Services from the catalogue due to varying degrees of maturity, complexity of the project & delivery service requirements. This is applicable to only projects where the scope is fixed & standard outcome is expected.
Below is an example of Project level estimate using service catalogue
1.9 Comparison to Traditional Delivery Models
Client Perspective
Traditional Models Output Based Models
Pricing Under-utilization of resources Client pays for the unit of work when work load reduces; delivered; don’t need to “create work” to especially if there is frequent keep people. change in work load Overheads High client management Reduced client management overhead overhead Scope Requires multiple Change The model is flexible enough to Change Requests and SOW changes in accommodate majority of scope changes case of scope without the need of Change Requests as change the changes can be raised as a output to be delivered Cost Doesn’t include the break-up of Allows client to see a detailed break-up Breakup cost at the level of deliverables of cost at the level of specific deliverables. Gives better predictability in IT spend since cost for demand is transparent to the client. Staffing / Ideal for scenarios where client Ideal for scenarios where client expects a Work expects to retain resources for fluctuation; “pay on results” and not on Allocation their knowledge, even at a cost’ FTEs or pre-agreed models
Service Provider Perspective Traditional Models Output Based Models Staffing/ Less control on staffing as it is Flexibility in staffing & resource Operational usually decided by the management. Can result in better Efficiencies customer Utilization of staff by resource pooling across projects
Operating Staffing and rates are Delinking of value proposition with Margin transparent; limited influence effort, and introduction of value pricing on margins usually results in higher margins
Margin Difficult to improve margins; Progressively better margins by Improvement improved productivity can investing in automation and use of IP lead to reduced revenues tools
Risk Risk is with the client Risk is with service provider
Estimation Usually client controls the Some of the challenges are – Ambiguity staffing and estimation in selecting the services by business, understanding of the model, Unavailability of historical Statistical Data
4. Process involved in Statistical Data Analysis and Base lining
Various steps involved
Data Gathering Data Cleaning Model Building Model Deployment
2.1 Data Gathering: Getting the required data regarding Planned Effort, Actual Efforts, Work Order Details, Type of Work Order, Test Case Progression and Regression, Test Automation, Environment B&D, UAT, Performance Engineering, Services Virtualized, Defect details, Defect Severity, Risks and Issues and so on from various sources or any standard test management tool.
2.2 Data Cleaning: Cleansing the Data or Data treatment will be done for all the independent variables selected. All null values, Outliers will be addressed as part of this task so that we have good set of data for our analysis purpose
2.3 Model Development and Validation Model Development & Model Validation involves multiple steps. The details of each step are highlighted below.
1.1) Process flow
1.2) Input Data
1.3) Data Processing & Treatments
1.4) Variable Selection
1.5) Modeling Exercise
1.6) Amendments to Predictions
1.7) Accuracy
Documentation of model development and Mapping of Variables will follow the above steps.
2.3.1 Process Flow
2.3.2 Input Data for a Phase
For our analysis purpose let us assume we have take System Testing phase of the application. All the key parameters from the Master data are collected from different sources like:
Work order details on Actual Effort Test Cases details (Progression & Regression) Defects details including category & Aging Issues and risks data Start date, End Date RTS Data Dump-includes Test Designing and Test Execution Efforts only for each Application at Work Order Level.
2.3.3 Data Processing and Treatment
Data for all the above mentioned parameters will be collated & analyzed. Treatment of data was done for incomplete, Null and outliers.
S.NO Data Cleaning comment
1 Removed Rows where Application Name is not applicable
2 Removed Rows where Difference between End date and Start Date is negative
3 Removed Rows where Test Cases-Progression or Regression is Not Applicable
2.3.4 Variable Selection “R” statistical analysis tool was used for our analysis. Random forest and Shadow variable comparison (Boruta algorithm) was used to select the list of variables which showed significant variation in the dependent variable as against the independent variables.
2.3.5 Modelling Exercise
When Tried to segregate the Application Names on Actual Effort using Decision Tree, received following pattern:
Complete Regression Model with Significant variables:
Application Name Estimate Std. Error t value Pr(>|t|)
Application A * -1.16E+03 5.82E+02 -1.993 0.04713
Application B *** -1.06E+02 6.46E+02 -0.164 0.86986
Application C ** -2.93E+01 6.06E+02 -0.048 0.96145
Application N * 7.64E+02 5.57E+02 1.371 0.17136
Where * represent the significance of the application during analysis & the Pr (probability value) used to identify the level of significance.
2.3.6 Amendments to prediction Based on the Observed outcomes from the regression models and the accuracy levels, various permutations and combinations will be tried and the one with higher accuracy level will be finalized.
2.3.7 Accuracy
Various models were tested for different variable selection for all the Significant Applications:
Options Variables Stability OBS APP comments
Application Name + WO Type + 1 63.1% 11637 867 Low Accuracy WODuration
Application Name + WO Type + WO Duration +Cycles+Progression Moderate 2 73.7% 11332 627 +Cycles+Regression + Issues + Accuracy Risks
Application Name + WO Type + WO 3 Duration +Progression +Regression 89.6% 11332 627 High Accuracy +Productivity + Regression
2.4 Model Deployment: SQL code development- Deployment of Model on SQL/other platform, UI Development and maintenance will follow once the analysis is completed. Now the system is ready to generate the effort estimates for all the services which are part of Service Catalogue at a click of a button.
5. Online Estimation Tool
The outcome from the “R” statistical tool gives the details of all the significant applications and the details related to their productivity and effort required for any particular kind of services that have been defined. This data is loaded into SQL database and linked to the online estimation tool.
The tool provides the option to select the Service Type, Activity Type, Application Name, and Service Catalogue Item from the drop down list available in the estimation tool. Once these fields are updated with the necessary selections, the tool gives back the effort details required to perform that particular service as part of Service Catalogue. In this way the manual way of estimating the effort will be avoided.
Below is a snapshot of the online estimation tool.
6. Conclusion
From the above data presented in various sections we have seen how the historical data for different services can be used and fed into the statistical tool “R” to get the effort details at each application and service level.
Hence we can conclude that by use of Service Catalogue, all the following benefits can be achieved.
Huge reduction in time, effort & cost to scope new service engagements
Moving away from an FTE based to Service Output based model or consumption based model & Payment linked to successful outcome & not on FTE
Predictable quality, Service price & Outcome through standardized Input, Output & tasks
Complete predictability of service cost and service quality standards
Accurate estimate through use of historical data and statistical analysis tool
Delivery risks transferred to service provider thereby driving the right behaviours from the partner organisation
References & Appendix
Service Catalogue Exhibit from Account
Results / Outcome from R Statistical Analysis tool used for applications within the account
Service Catalogue models related information from Google
Author Biography
Raghu is an experienced Testing Service Professional with 17+ years of experience and working as Director with Capgemini from last 3+ years. He holds a Doctorate in Business Administration from IIBMS, PGDBA from Symbiosis and Bachelor of Engineering degree from BMS College of Engineering Bangalore University.
Raghu is a certified Project Management Professional (PMP) from PMI, Certified Scrum Master (CSM), Six Sigma Certified - Green Belt, Capgemini Level 2 Certified Engagement Manager.
Raghu is part of testing services for a client within financial services. He is heading the process transformation and innovation team within the account. Raghu has worked on implementing the Output based Service Catalogue Model within the financial account at Capgemini and has used “R” Statistical Analysis tool for predictive Analysis.
THANK YOU!