<p> COTS Systems Integration Services Tools</p><p>COTS Systems Integration: Proposal Evaluation Guide</p><p>EI Toolkit Reference Document Updated: Version 2.1 August 2011 Table of Contents</p><p>1 INTRODUCTION...... 1 2 PREPARATION...... 2</p><p>2.1 RECEIVE AND INVENTORY SUBMITTALS...... 2 2.2 DEFINE EVALUATION TEAM...... 2 2.3 DEFINING EVALUATION CRITERIA...... 2 2.3.1 Weighting Evaluation Criteria...... 2 2.3.2 Changing the Evaluation Criteria...... 2 2.4 DOCUMENTS...... 2 3 EVALUATION APPROACH...... 4</p><p>3.1 OVERVIEW...... 4 3.2 EVALUATION PROCESS...... 4 3.2.1 Familiarization...... 4 3.2.2 Requested Information...... 4 3.2.3 Documenting Proposal Ambiguities, Deficiencies & Inadequacies...... 4 3.2.4 Identifying Strengths, Weaknesses & Risks...... 4 3.2.5 Communicating with Vendors...... 5 3.2.6 Stages of Review...... 5 3.2.7 Reference Checks...... 5 3.3 EVALUATION CONSENSUS...... 6 4 USING THE EVALUATION MATRIX TOOL...... 7</p><p>4.1 SCORING CRITERIA...... 7 4.2 SCORING...... 7 5 PROPOSAL EVALUATION GUIDE...... 8</p><p>5.1 ADHERENCE TO RFP INSTRUCTIONS...... 8 5.2 COMPANY INFORMATION...... 8 5.3 PROJECT UNDERSTANDING...... 9 5.4 FUNCTIONAL & NON-FUNCTIONAL REQUIREMENTS...... 9 5.5 IMPLEMENTATION APPROACH/METHODOLOGY...... 10 5.6 PROJECT ORGANIZATION...... 12 5.7 EXPECTED BUYER DUTIES...... 13 5.8 PROJECT MANAGEMENT APPROACH...... 13 5.9 TRAINING & KNOWLEDGE TRANSFER...... 15 5.10 POST IMPLEMENTATION APPROACH...... 15 5.11 PAST PERFORMANCE AND RELATED ENGAGEMENTS...... 16 5.12 ORAL PRESENTATIONS...... 16 5.13 RISK IDENTIFICATION...... 16 5.14 FEE SUMMARY...... 17 5.15 PROFESSIONAL ARRANGEMENTS...... 17</p><p>COTS Systems Integration Services Tools | www.esi.mil 2 1 INTRODUCTION The systems integrator proposal evaluation process includes examining each proposal, in detail, against the evaluation factors, sub-factors and criteria set forth in the RFP and the evaluation matrix, and assigning a rating with supporting narratives. The proposal evaluation process assesses the proposal and the vendor’s ability to perform. The principal purposes of the evaluation process are:  Determine which proposals are acceptable  Provide a sound basis for making an informed and reasoned vendor selection by identifying: o A clear picture of the issues considered during the evaluation by identifying areas of concern and uncertainty as well as those, which provide substantial assurance of a successful outcome o List the strengths, weaknesses and risks of the proposed approaches o Formalize the evaluation process by quantifying scores  Ensuring the RFP was valid if: o An adequate number of qualified proposals were received o The proposed fees are relatively close in range</p><p>COTS Systems Integration Services Tools | www.esi.mil 3 2 PREPARATION 2.1 Receive and Inventory Submittals Prior to beginning evaluation of the proposals received take an inventory of what you have received with regard to each vendor to ensure you have all of the submitted documents. 2.2 Define Evaluation Team The evaluation team is typically made up of high level project personnel, Project Manager and Project Executive Officer, and 2 – 3 other key Business and Technical Owners whose functions will be most impacted. If the evaluation team is too large it becomes difficult operate efficiently and effectively. Three to five as a core team usually works well. Be sure to identify any conflicts of interest prior to beginning the evaluation process. Typically, the Project Executive Office or the Project Manager will lead the evaluation effort. The Lead and other core team members will be involved through the entire evaluation process and should be involved in reviewing all proposals. The core team may draw upon other key resources in an advisory capacity, but it is the core team member responsibility to rate the proposal using the advisors input.</p><p>2.3 Defining Evaluation Criteria The criteria used to evaluate each vendor should correspond with the criteria stated in the RFP as well as any best practice attributes of any methodologies, approaches, plans, logs, sample documents requested in the RFP. There should be up to three levels of criteria, high-level factor, sub-factor and specific attributes of a sub-factor. The reviewing team should agree upon the evaluation criteria, prior to beginning proposal evaluations.</p><p>Appendix A of this document is a sample Evaluation Matrix formulated after the SI RFP Template. 2.3.1 Ranking Evaluation Criteria Criteria should be ranked at the highest factor level. The higher the ranking the more important it is to the success of the project. The reviewing team should agree in advance to the rank to assign to each high-level factor: High Medium, or Low importance. 2.3.2 Changing the Evaluation Criteria Once the final evaluation criterion has been agreed upon by the team, it should only be changed with approval of the team lead or consensus or the team. This will ensure all reviewers are using the same evaluation criteria. Evaluations must be consistent across reviewers and vendors. 2.4 Documents Reviewers should have all the following documents at hand to begin the review process:  Original RFP  All vendor proposals  Evaluation Matrix  SI Proposal Review Guide & Checklist</p><p>COTS Systems Integration Services Tools | www.esi.mil 4 3 EVALUATION APPROACH 3.1 Overview Reviewers must examine each proposal individually in detail to measure it against the evaluation factors and sub- factors. Reviewers assign a rating and document the basis for the rating. This is the core of the evaluation process. </p><p>3.2 Evaluation Process 3.2.1 Familiarization Prior to receiving the proposals each member of the proposal review team should become familiar with the RFP process, the requested information, the evaluation criteria, evaluation process and the rating system. 3.2.2 Requested Information Each proposal should be checked for its compliance with all RFP requests. Assessing the requests should be a matter of the proposal either meeting or not meeting each request; any proposal not fully meeting every one of the requests may be considered for rejection without further consideration. If no vendor proposals meet all requests then you may want to consider re-thinking the RFP. Appendix A is a checklist, which can be used for this process. It currently correlates with the RFP requests contained in the System Integration RFP Template. 3.2.3 Documenting Proposal Ambiguities, Deficiencies & Inadequacies The purpose of this documentation is to prepare to go back to the vendor for additional information or clarification. Reviewers must document problems in evaluating a proposal due to:  Ambiguous language  Unclear meaning  Failed to respond to the RFP instructions  Not enough information was provided to evaluate the proposed approach, etc.  Information contains obvious flaws, error or omission that increase the risk of unsuccessful performance 3.2.4 Identifying Strengths, Weaknesses & Risks Reviewers must identify and document, as part of the evaluation rating, strengths, weaknesses and risks. Numerical scores and ratings must be supported with specific details as to why the proposal is weak or risky. The strengths, weakness and risks form a large part of the selection rational. However, that fact that a proposal is deficient, weak or at risk in some areas does not mean that it should be excluded from further consideration. 3.2.5 Communicating with Vendors One individual should be selected to coordinate communications with the vendors. All issues documented, by the reviewers as stated in Section 4.2.3, should be consolidated, by vendor, and submitted to the vendors at one time for response. The time allowed for response should be consistent for all vendors and commensurate with the defined schedule of events, regardless of the number of issues a vendor may have.</p><p>COTS Systems Integration Services Tools | www.esi.mil 5 3.2.6 Stages of Review There are typically three stages of proposal review:  Initial Evaluation  Oral Presentation Evaluation  Final Evaluation Each level assesses the vendor’s submitted information against the evaluation criteria as described in Section 5 of this document. However, as each team members goes through the evaluation process, oral presentations, obtains further information and has questions / concerns answered or alleviated scores may fluctuate, issues or risks may become apparent resulting in a change in a particular rating and reason for score. 3.2.6.1 Initial Evaluation At this stage each proposal requirement is reviewed individually against the evaluation criteria and best practices. After the reviewer familiarized him/herself with each proposal, then comparisons can be made across proposals, this process may result in rating changes. In terms of the project organization, resumes can be reviewed objectively against the criteria and best practices however, ratings may change based on meeting individuals during the oral presentations in terms of subjective criteria like fit with the style and culture of your organization. During this stage, the reviewer may want to add, delete or change evaluation criteria. This should be done using the evaluation criteria change process described in Section 3 of this document. When initial scoring of proposals is complete, all responsive vendors are considered capable of performing the work. Note: The oral presentation evaluation section of the matrix is not completed at this time. 3.2.7 Reference Checks Reference checks are a valuable part of the decision making process and are usually handled by the Evaluation Team Lead. During reference checks, obtain information on how the vendor performed for the company, major issues or problem that occurred and how they were resolved, contractor’s ability, reliability, professionalism and similarities of projects. Reference checks should be done prior to the oral presentations so that reviewers will have the opportunity to question the vendor on any unfavorable areas of the reference check. Avoid relying solely on a reference check; request clarification from the vendor. 3.2.7.1 Oral Presentation Evaluation At this stage, vendor’s slides, presentation and team skills & chemistry are rated against the oral presentation criteria. The initial scores of vendors, although a good indicator of their capabilities, must not be considered as completely reliable or an absolute indicator of the vendors ability to do the work. It is not unusual for vendors extremely skilled in preparing high scoring written responses to do poorly in oral presentations. It is always a good practice, no matter what the scoring spread, to interview at least the three top-ranked firms. Endeavor to keep the process competitive for as long as possible. 3.2.7.2 Final Evaluations During this stage, all reviewers should revisit their ratings and comments to ensure all factors and sub-factors were rated using all available information, such as written proposals, oral presentations, oral presentation slides, answered questions, additional obtained information from your own research, etc., as well as comparing each vendor against the others.</p><p>COTS Systems Integration Services Tools | www.esi.mil 6 Any changes, updates or modifications to your evaluation criteria ratings or comments should be finalized. This stage will result in your Final Vendor Evaluation Matrix.</p><p>3.3 Evaluation Consensus The final rating of each proposal should be assigned by a consensus of the reviewers. A consensus requires a meeting of the minds on individual assessments, deficiencies, strengths, weaknesses and risks. In cases where the reviewers are unable to reach agreement, without unreasonable delay to the process, the Project Executive Officer has the authority to make the decision.</p><p>COTS Systems Integration Services Tools | www.esi.mil 7 4 USING THE EVALUATION MATRIX TOOL The evaluation Tool consists of two MS Excel spreadsheets within one workbook, the Definitions page and the Evaluation Criteria Checklist. 4.1 Definitions The below definitions are used for ranking the relative importance of each RFP area.  High: Factor is of the highest relative importance and should be given more consideration that others in making the final evaluation/assessment.</p><p>Score Level Description 3 Excellent SI has fully addressed and comforms to best practices 2 Strong SI has adequately addressed this factor. Minor deviations from best practices. A critical issue raises doubt as to SI qualification with this factor. Factor is 1 Cause for Concern inadequately address. Large deviations from best practices  Medium: Factor is of moderate importance when compared to others.  Low: Factor is low when compared to others and should be considered the least when making the final evaluation/assessment. These rankings should be used to evaluate the elements of the RFP against each other. They will be subjective assessments that your team should arrive at on a consensus basis. </p><p>In addition, the below definitions are used to grade each detailed RFP response:  A – Excellent: SI has fully addressed and conforms to best practices.  B – Strong: SI has adequately addressed this factor. Minor deviations from best practices.  C – Concerns: A minor issue raises doubt as to SI qualification with this factor. Factor is not fully addressed. Moderate deviations from best practices.  F – Does not Meet: A critical issue raises doubt as to SI qualification with this factor. Factor is poorly addressed. Large deviations from best practices.</p><p>4.2 Assessment Process Completing the Evaluation Criteria Checklist:  Read the Definitions sheet and agree on descriptions provided or adjust for your project.  On the Checklist sheet, make sure there is a column for each vendor. Add new columns as needed. Note that each section of sub factors rolls up to a factor.  As a team, agree on the importance ranking for each factor.  Using the checklist discuss the responses and materials provided by each vendor for each factor and sub factor. Reach consensus on a grade for each.  Grades are given at the sub-factor level and assessed at the end for an overall grade for that particular factor. For example, if 10 out of 11 sub-factors are graded as “A” and the 11 th is given a “B”, it is likely the overall assessment will be an “A”.</p><p>COTS Systems Integration Services Tools | www.esi.mil 8 When all vendor sub factor and overall grades have been agreed upon, review the grade results against the importance ranking and combine all grades to agree upon an overall summary ranking for each vendor (1, 2, 3, etc.) Remember, this process is subjective in nature and will require group discussion to reach consensus. The information in the next section provides additional guidance to assist in this process.</p><p>COTS Systems Integration Services Tools | www.esi.mil 9 5 PROPOSAL EVALUATION GUIDE This section should be used as a high level guide while reviewing vendor submitted information. For more detailed evaluation comparison review the corresponding Best Practice Toolkit section. This will contain detailed best practice descriptions and sample tools and templates to be used for comparison. The format of this section follows the format of the Vendor Evaluation Criteria Checklist. 5.1 Adherence to RFP Instructions</p><p>Evaluation Factor Evaluation Guide</p><p>Timeliness Was the proposal submitted within the specified time period? Were there any deficiencies, inadequacies and ambiguities? Completeness Was the proposal well laid out? Does the response address the content of the RFP or is it a Overall quality and ‘canned’ response dealing primarily in generalizations? professionalism Does the response specifically address what was requested or were there counter offers to what the RFP called for? Overall responsiveness Did the vendor understand what was being asked of them? </p><p>5.2 Company Information</p><p>Evaluation Factor Evaluation Guide Financial viability Did they submit financial statements? Assess strength of financials by using industry standard ratios. Organization structure Research various financial web sites for additional information, i.e. CNBC.com or NASDAQ.com. Assess company’s litigation Overall exposure to history and status. government sector Does this company have a history in this sector? Does the servicing department or organization have the Service resources to support this project? department/organization Does the firm limit its work to specialized areas? Does the structure, size and industry principal area of specialization match the area covered by the experience RFP? Does the company information indicate the firm can handle the project size, scope and complexity?</p><p>5.3 Project Understanding</p><p>Evaluation Factor Evaluation Guide Overall understanding of Does the proposal clearly depict a thorough understanding of project objectives the project and its objectives as well as the overall vision of the Understanding of the business company? requirements Does the vendor understand the business requirements and how they will support the project objectives and company vision? Understanding of end-state vision and problem statement Is the vendor’s end-state vision equated to yours?</p><p>COTS Systems Integration Services Tools | www.esi.mil 10 5.4 Functional & Non-Functional Requirements</p><p>Evaluation Factor Evaluation Guide Completeness of vendor Is it clear that the vendor understood all of the functional response requirements? Extent of proposed Did the vendor respond to all functional requirements? What enhancements was the extent of system enhancements both as an individual vendor and compared to the other bidders? Extent of BPR required Can the vendor identify, at a high-level, where BPR should be performed? Vendor ability to meet / handle Are there a significant number of requirements that the vendor requirements does not believe they can handle? Does the vendor demonstrate a clear understanding of DoD information assurance requirements?</p><p>5.5 Implementation Approach/Methodology</p><p>Evaluation Factor Evaluation Guide Does the approach make sense in general? Does the approach make sense for this project? Does this approach support the goals of the project and your company? Does the approach fit with the culture of your organization? Is the methodology easy to follow? Is this approach been proven successful on comparable projects? Implementation methodology/approach detail provided Application to this project Implementation methodology includes:  Road map  Phases descriptions with key tasks, deliverables, tools and templates to be used, exit criteria, key decision points, milestones and approximate durations</p><p>Are phases clearly defined? Is it clear when a phase ends/begins? Is it clear what work will be performed during Phases each phase? Does each phase result in clearly defined deliverables? Are the resultant deliverables sufficient to support this project? Do the deliverables support the goals of the project? Deliverables, Milestones & Key Are milestones clearly defined and do they occur with a decision points reasonable frequency? Have key decision points been defined? Are the deliverables measurable? Does the vendor have sufficient tools and templates available to Tools & Templates facilitate efficient and effective work and deliverable creation?</p><p>COTS Systems Integration Services Tools | www.esi.mil 11 Is this methodology specific to the SW being implemented? Does it model, if not follow exactly, the SW vendors Software package specific recommended methodology? Is this methodology proven for this SW package? Does work / project plan reflect the implementation methodology? Are all deliverables and milestones present? Do the high-level tasks make sense for the work to be done? Does the plan reflect a reasonable timeline? Are the task durations reasonable? Has this plan been successfully used on comparable projects? High-level project plan included in Appendix C High-level work plan Project plan includes, at least:  Major activities  Durations  Dependencies  Milestones  Phases  Timeline Is the strategy clearly defined? Does it make sense for the Roll-out strategy culture of your company? Is the timeline reasonable? Does this plan support the goals and vision of your company? Is the Data Strategy is clearly defined and is it reasonable for your company? Data Strategy includes: Data Strategy  Major activities/tasks  Tools and techniques to be used  Deliverables & milestones  Roles & responsibilities Is the Change management approach detailed, clearly defined and makes sense for your company? CM approach includes:  Methodology  Activities / tasks Change Management  Timelines Approach  Deliverables  Milestones  Roles & responsibilities  Method of measuring and mitigating resistance to change  Lessons learned  Communication plan details</p><p>COTS Systems Integration Services Tools | www.esi.mil 12 Is the IA Management Plan approach detailed, clearly defined, and demonstrates a complete knowledge of current DoD IA policies, information protect issues, and service/agency unique requirements? IA approach includes: Information Assurance  Methodology Management  Activities / tasks  Timelines  Deliverables  Milestones  Roles & responsibilities</p><p>Is the Business process re-engineering process/methodology are clearly defined? Roles & responsibilities clearly defined BPR Appendix A, of the RFP, includes vendor response to suggested BPR by functional requirement The recommended BPR effort makes sense and is reasonable for your company.</p><p>5.6 Project Organization</p><p>Evaluation Factor Evaluation Guide Has a Project organization chart has been provided. Does it contain the appropriate representation of skills. Are the resources named? Organization chart includes: Organization Chart  All expected project roles (vendor, subs, buyer & software vendor)  Reporting & communication lines</p><p>Are roles & responsibilities are clearly articulated for each Roles & Responsibilities: SI, participating organization? Do the responsibilities make sense Subs, SW Vendor, Buyer and are achievable by your resources? Are resource plan for vendor, buyer, subs and software vendor provided? Resource Plans: SI, Subs, SW Resource plan includes: Vendor, Buyer  Number of dedicated resources  Percentage required Is there a mapping to roles indicated on organization char? Do resumes indicate what project role the resource is being Resumes mapped to roles presented for?</p><p>COTS Systems Integration Services Tools | www.esi.mil 13 Are all key personnel from vendor, subs and software vendor Proposed project team & identified by name? resumes review Does Appendix E, of the RFP, includes a completed resume template for each named resource  Level of education and experience are commensurate On site work schedules with the role and responsibilities the resource will be assigned to  Resource has proven past experience within this Travel distance to your industry and/or has proven experience on a comparable company project(s) and can bring a desired commercial perspective  Key personnel have successfully completed the requirement for and received certifications from Good mix of required appropriate industry organizations relevant to their government experience & proposed roles (e.g., Certified Information System commercial experience Security Professional (CISSP)). Resource schedules and locations are a good fit with your company, i.e. work days start / end times, overtime, etc.</p><p>5.7 Expected Buyer Duties Are the duties the vendor expects you to perform before, during or after the project reasonable, appropriate and achievable for your organization? Are the expected duties adequately described?</p><p>5.8 Project Management Approach</p><p>Evaluation Factor Evaluation Guide Existing PM methodology / Does the vendor have an existing methodology? Is the approach approach is proven on comparable projects? Does the vendor’s methodology include tools & templates Tools & templates available for efficient & effective management of the project? Are the deliverables clearly defined and are they sufficient for PM Deliverables managing a project of this size, scope and complexity? Is status reporting is done weekly at the team level. Are executive status reporting done at least monthly? Are sample status reports were provided as part of Appendix D Status Reporting of the proposal? Are status reports are commensurate with best practices as compared to the Best Practices Toolkit?</p><p>COTS Systems Integration Services Tools | www.esi.mil 14 Is a risk management approach clearly defined and feasible for managing risk at your company? Has the approach been proven on comparable projects? Risk management plans includes process for:  Identifying risks  Assessing impact of risks  Mitigating risks  Tracking risks progress Are sample Risk Mitigation Plans included in Appendix D, of the RFP? Risk Management Risk Mitigation Plan includes, at least:  Date risk identified  Overall risk statement  Risk description  Area of impact  Impact statement  Risk owner  Mitigation actions  Due date  Status Has an Issues management approach/plan been clearly detailed? Has it been proven on comparable projects? Issues management plan includes:  Issue identification  Issue escalation procedures  Issue resolution/closure procedures  Sign-off required An issue log is included in Appendix D, of the proposal? Issue log includes, at least:  Issue initiator & contact information Issues Management  Date/time opened  Associated document, if applicable  Summary description  Impact/criticality  Type  Issue owner  Current status  Completion date  Resolution, development request, software vendor support request </p><p>COTS Systems Integration Services Tools | www.esi.mil 15 Has a configuration management/change control approach and plan been provided in detail? Does it include:  Process for managing, investigating tracking and closing change requests  Process for assessing impact to cost, schedule and resources Configuration  Managing multiple version / configuration changes Management/Change Control  A diagram/flowchart of the change control process including approvals and sign-off  Roles, responsibilities & authority  Use of Change Control Board Does appendix D include a sample configuration management/change control process & procedures, log and change request form?</p><p>5.9 Training & Knowledge Transfer</p><p>Evaluation Factor Evaluation Guide Has a training and knowledge transfer approach been detailed? Have all phases and deliverables have been stated? Is the approach feasible for use in your company? Approach includes, at least:  All impacted roles  Process to determine training needs  Training strategy Training & Knowledge Transfer  Method and schedule to deliver training  Timeline  Metrics, tools and processes used to ensure knowledge transfer has occurred  Documentation provided  Milestone and deliverables  Roles & responsibilities</p><p>5.10 Post Implementation Approach</p><p>Evaluation Factor Evaluation Guide Is the post implementation support detailed and sufficient for Strategy your company and has it been proven on comparable projects? Help Desk Support plan includes, at least:  Help desk Configuration Management  Configuration management  Infrastructure support Infrastructure Support  System/database support  Staffing levels Information Assurance  Roles and Responsibilities</p><p>System / Data Base Support</p><p>COTS Systems Integration Services Tools | www.esi.mil 16 5.11 Past Performance and Related Engagements</p><p>Evaluation Factor Evaluation Guide Are full details of past performance provided? Details include: Past Performance  Experience with SW vendor  Management cohesiveness  Experience with integrating this system with multiple legacy  Management of comparable project in size, scope and complexity  Experience leading change management and BPR efforts Related Engagements  Experience with information assurance with a comparable project in size, scope, and complexity Is the related engagement table completed in its entirety?  Engagements are clearly relevant Has performance in delivering quality services meeting technical, management cost and schedule requirements been described?</p><p>5.12 Oral Presentations</p><p>Evaluation Factor Evaluation Guide Key project leadership and team members should have been Style & Chemistry present. Were roles and responsibilities clear across team members? Will project leadership be able to effectively communicate with company leadership? Do you have Format followed the RFP confidence in the management team? Do you have confidence in the individual team members? Do the individuals act as a Information presented did not team? Is team leadership apparent? conflict with the written Was the material presented clear concise and relevant? proposal</p><p>Project team presentation</p><p>Ability to sufficiently answer questions</p><p>5.13 Risk Identification Vendor has identified real and relevant risks with the project, their approach, buyer constraints, etc. and has detailed proposed mitigation actions, impact and timeline for those risks. A vendor who says there is no risk present or has minimized potential risks should be scored poorly as a failure or inability to identify potential risk will impact the success of the project.</p><p>COTS Systems Integration Services Tools | www.esi.mil 17 5.14 Fee Summary For fixed price contracts, evaluation normally can be as simple as a comparison of the offered prices to ensure the price is fair and reasonable. Expenses should be commensurate with industry standards at the time the project is bid. The proposed fee should be evaluated to determine whether the proposed fee of the effort is complete, reasonable and/or realistic and to ensure all requirements are reflected in the fee proposal. Congruence between written proposal, oral presentation and fee shall be utilized to determine realism of the proposed fee. Unrealistic or inconsistencies between written proposal and the fee proposal should be considered a major project risk. Are there comparable differences in what is to be delivered?</p><p>5.15 Professional Arrangements Vendor’s who do not agree to use the standard fixed fee contract and/or who take a material exception to any of the terms and conditions, for which negotiation is not an option should not be retained for future consideration.</p><p>COTS Systems Integration Services Tools | www.esi.mil 18</p>