Product and Technical Demonstrations ----- Two Case Studies
Total Page:16
File Type:pdf, Size:1020Kb
Product and Technical Demonstrations ----- Two Case Studies PIL Webinar 36 Feb. 15, 2018 1 Three Ways to Receive Information Written Video Oral (includes Product or Capability Demonstrations) 2 FAR Guidance on Each Method (see FAR 15.102) Written Video Oral “Shall”— clearly allowed, “May”— • Reps & Certs but no guidance is • Capability given • Past Performance • Signed Offer • Work Plans or Sheet Approaches • Exceptions to • Staffing Resources T&Cs • Transition Plans 3 More on Oral Presentations (see FAR 15.102) Streamline the Selection Process Lower B&P Costs Opportunity for Dialogue At any Time May Include During the Demonstrations Process and Tests 4 Product or Capability Demonstrations If you were buying a new car, would you rather— • read a hundred page report, or • take the car for a test ride? In some cases, it might make sense to do both – but still, the test ride (product or technical demonstration) is vitally important in a purchase decision... 5 Case Study 1: Customs and Border Protection (CBP) Non-Intrusive Inspection (NII) Density Meters 6 What is a Density Meter? (Think of a stud finder!) • When inspecting a car tire, the meter provides a reading of the density of that tire. • If the reading is much different than what would be expected, then the officer could infer that there was likely narcotics, etc. smuggled in the tires. 7 What is a Density Meter? Lets watch! Customs and Border Protection Officers catch some traffickers at the San Ysidro Port of Entry 8 Density Meter History • CBP began buying these in the 1990s. • Legacy products reaching end of life needed to be replaced. • Solicitation released in FY15. • During evaluation a critical flaw was noticed in the SOW describing the required capabilities. • Solicitation then cancelled in FY15. • This solicitation released in FY17. • $16.5M • Single-award IDIQ • Fixed unit pricing with volume discounts. 9 Density Meter – Old vs. New Requirements OLD NEW* Inspection Threshold: The Density Meter shall be capable of detecting a change in • Scan Rate: The unit shall be able to density 99% of the time through at least 6 mm of steel at a rate of 2 inches/sec with a scan at least 4 readings per second. maximum time to establish baseline reading of 4 seconds. Accuracy of the classification rate shall not be less than 99% (see equation below). • Depth of Reading: The system will be Objective: The Density Meter shall be capable of detecting a change in density 99% of the time through at least 8 mm of steel at a rate of 4 inches/sec with a maximum time capable of detecting anomalies through to establish baseline reading of 2 seconds. Accuracy of the classification rate shall not no less than 8 mm of steel. be less than 99% (see equation below). • Scan Accuracy: The system shall provide consistent readings within +/- 2% for an anomaly placed behind 8 mm Accuracy (ACC) = The degree to which the indications of a change in density are correct. of steel. True Positive (TP) = Indication of a change in density when there is one. True Negative (TN) = No indication of a change in density when there is not one. • Operation Modes The system shall False Positive (FP) = Indication of a change in density when there is none, “false alarm”. support at least 2 modes of operation: False Negative (FN) = No indication of a change in density when there is one; does not alarm over a change in density. – Inspect: The system shall probe the surface Sample Rate: The Density Meter shall sample an object at least 4 readings per second. and provide the user with an indication of Operational Modes: The Density Meter shall support at least two (2) modes of changes in density. operation: Calibrate: Each density meter must be able Inspect: The Density Meter shall probe through a surface and provide the operator with – an indication (visual & audible) of changes (increase & decrease) in density. to self-calibrate to maintain consistent Calibrate: Each Density Meter must be able to self-calibrate, when activated by the detection response throughout its lifecycle. operator in the field, to maintain consistent response throughout its lifecycle. Calibration fixture will be provided with *In Test Plan and downloadable each system. 10 Density Meter – Evaluation Factors Phase 1 Factor 1: Technical Performance and Approach • Oral Presentation and Capability Demonstration • Performance Evaluation (Government Testing) Advisory (Soft, Voluntary) Down Select based on Factor 1 Phase 2 Factor 2: Management Approach (written) Factor 3: Past Performance (written) Factor 4: Price (written) Trade-Off Selection Decision based on Factors 1, 2, 3, and 4 11 How are we going to do it? Factor 1 – Technical Performance and Approach This factor will be addressed during Phase 1, at the oral presentation and capability demonstration for Phase 1A and at the performance evaluation for Phase 1B. This factor covers sections 3.2 through 3.3 of the Statement of Work as indicators, which must be addressed during the oral presentation and capability demonstration. The indicators marked with an asterisk (*) are the minimum demonstration requirements. (See downloadable Density Meter TET Orientation Presentation) For evaluation factor 1, the Government will consider the offeror’s proposed approaches and results of the performance evaluation to arrive at a confidence rating of the offeror’s technical performance and approach to meeting the procurement’s objectives. RATINGS FOR FACTORS 1 THROUGH 3 Rating Definition High Confidence The Government has high confidence that the offeror understands the requirement, proposes a sound approach, and will be successful in performing the contract with little or no Government intervention. Some Confidence The Government has some confidence that the offeror understands the requirement, proposes a sound approach, and will be successful in performing the contract with some Government intervention. Low Confidence The Government has low confidence that the offeror understands the requirement, proposes a sound approach, or will be successful in performing the contract even with Government intervention. 12 Density Meter – Performance Evaluation ---- What we actually did! After the oral presentations and capability demonstrations, we took the devices into our warehouse and performed several tests (see Attachment B – Test Plan). 13 Density Meter – Why are we doing it this way? We are using oral presentations in this evaluation for the value of having an interactive dialogue with offerors key personnel and for the opportunity to better understand the offeror’s proposed technical solutions. Likewise, demonstrations or other challenges—in this case, Performance Evaluation Testing—can be powerful and effective tools that can save time and reveal true capabilities of offerors. 14 Density Meter – Performance Evaluation RFP, Page 46, Phase 1B – Performance Evaluation “After the offeror’s oral presentation and capability demonstration, but starting no later than the next business day, the Government will conduct a Performance Evaluation of the offered Density Meter device at CBP’s test facility, covering the Inspection requirement in Section 3.2.1 of the SOW (see evaluation Factor 1--Technical Performance and Approach below). For this phase, vendors must bring an operational device of the exact type or types proposed for evaluation and provide the shipping return address for the device. The device will be shipped to the specified address at the end of the performance evaluation. The evaluation will consist of one hundred fifty (150) scans to evaluate the offered device’s performance on several factors. The Government will conduct three trials of the offered device: one at the threshold inspection level, one at the threshold penetration level with a faster inspection rate, and one at a higher, objective level. The results of the trials will be considered in evaluation Factor 1—Technical Performance and Approach. The Performance Evaluation does not replace or otherwise remove the requirement for the device to pass Acceptance Testing following contract award in accordance with the processes and procedures outlined in Section 3.6 of the SOW. Although government personnel will conduct the evaluation tasks, vendors shall ensure that one of their presenters can instruct the government test personnel on the operation of their offered device along with providing the operator’s manual." 15 Density Meter – Why Do a Technical Demonstration? • Historically the written responses did not fully reflect the operational capabilities of the system. • Based on market research, we knew we were setting a high standard of performance. In fact, we reduced the scan speed of the threshold requirement to ensure a competitive environment. • The demonstration was seen as a way to streamline the evaluation of the proposal, which was successful. We decided to do testing to provide us with a greater confidence in the actual performance capabilities of the system. It also allowed us to get the devices in the hands of end-users for feedback. • We were able to make a clear distinction between the vendors as a result of the demonstration and testing. • In fact, the vendor who likely would have been graded the most favorable in the purely paper-based evaluation from 2 years ago performed worse than other evaluated products. 16 Case Study 2: Office of the Chief Procurement Officer (OCPO) Electronic Contract Filing System (ECFS) 17 What is an ECFS? • Exactly what it sounds like! • The ECFS will ultimately replace the traditional paper file and will support workflow, document management and records management. • There is been an increase in demand for electronic storage. Accompanying this increase in demand, a host of ECFS has entered the federal market. 18 ECFS History • First ECFS Pilot in March of 2013. • Established to support Continuity of Operations (COOP) and DHS' expanding Telework initiatives for the procurement community to still access contract files remotely. • The Pilot was built using SharePoint and it was determined that it was not sufficient and did not adequately meet the procurement communities needs.