Analytic and Operational Support for the Engagement of the Public and Patients REQUEST for PROPOSAL RFP # PCO-AOSEPP2021 February 8, 2021

Analytic and Operational Support for the Engagement of the Public and Patients REQUEST FOR PROPOSAL RFP # PCO-AOSEPP2021 February 8, 2021

KEY DATES

Request for Proposal Released February 8, 2021

Pre-Proposal Conference February 18, 2021

Written Questions Due February 24, 2021

Deadline for Proposals April 14, 2021

Projected IDIQ Award Date May 21, 2021

Projected Task Order Award Date(s) On a rolling basis, following IDIQ Awards

Projected IDIQ Start Date August 16, 2021

PCORI RFP # PCO-AOSEPP2021 1 About PCORI

In 2010, Congress authorized the establishment of PCORI as an independent nonprofit organization incorporated in the District of Columbia. The authorizing law establishes PCORI’s purpose, significant parts of its governance structure (including the appointment of PCORI’s Board of Governors and Methodology Committee), and key responsibilities and requirements of PCORI. The authorizing law was amended in legislation that was signed into law on December 20, 2019. PCORI is charged with helping patients, clinicians, purchasers, and policy makers make better-informed health decisions by “advancing the quality and relevance of evidence about how to prevent, diagnose, treat, monitor, and manage diseases, disorders, and other health conditions.” It does this by funding research guided by patients, caregivers, and the broader healthcare community, which results in high- integrity, evidence-based information. PCORI’s authorizing law, as amended by the reauthorizing law, directs PCORI-funded research to capture, as appropriate, the full range of outcomes data, including, as appropriate, potential burdens and economic impacts of the utilization of health care services. PCORI’s strong patient-centered orientation directs attention to individual and system differences that may influence research strategies and outcomes. PCORI is charged with producing useful, relevant clinical evidence through the funding of new research and the analysis and synthesis of existing research. PCORI is committed to transparency and a rigorous stakeholder-driven process that emphasizes patient engagement. PCORI uses a variety of forums and public comment periods to obtain public input to enhance its work.

Patient-Centered Outcomes Research Institute 1828 L St., NW, Suite 900 Washington, DC 20036 Phone: (202) 827-7700 Fax: (202) 355-9558 Email: [email protected]

PCORI RFP # PCO-AOSEPP2021 2 Opportunity Snapshot

The Patient-Centered Outcomes Research Institute (PCORI) seeks Contractors to enter into indefinite- delivery, indefinite-quantity (IDIQ) agreements to provide programmatic and analytic support for the Engagement Department. The Contractors shall provide the following types of professional services/support:

• Science of Engagement

• Economic Resource Services

• Environmental Scans, Literature Reviews, and Issue Briefs

• Education and Training

• Facilitation and Support of Meetings and Community Engagement

• Public Comment Proposals should be received no later than 5:00 p.m. Eastern Time on April 14, 2021. This will be a multiple-award procurement opportunity. The Contractors will operate at the direction of and receive guidance from PCORI.

PCORI RFP # PCO-AOSEPP2021 3 How to Proceed

1. Review the Requirements Examine all sections of the RFP and learn about what makes PCORI’s research different.1

2. Consider the Evaluation Criteria Consider the organization’s eligibility requirements and PCORI’s specific requirements to see whether your organization, your interests, and your capabilities fit the requirements listed in this RFP. Check the PCORI website for any modifications or amendments up to the submission deadline.

3. Develop Your Proposal Develop your response to the requirements within this RFP.

4. Follow Submission Guidelines See the Submission Guidelines section of this document.

5. Submit Your Proposal Proposals are due by 5:00 p.m. Eastern Time on April 14, 2021.

1 Available at pcori.org/research-we-support/pcor

PCORI RFP # PCO-AOSEPP2021 4 Contents

About PCORI ...... 2

Opportunity Snapshot ...... 3

How to Proceed ...... 4

Background ...... 9

Statement of Work ...... 13

IDIQ Requirements ...... 21 Pre-Proposal Conference ...... 21 Period of Performance ...... 21 Type of Agreement ...... 21 Award without Communications or Clarifications ...... 24 Best and Final Offers ...... 24 Withdrawal or Modification of Proposals ...... 24 Late Submissions ...... 24 Retention of Proposals ...... 24 Documentation Requirements ...... 24 Basis of Compensation to the Contractor ...... 24 Post-Award Debriefing ...... 25

Post-Award Information ...... 26 Place of Performance ...... 26 Hours of Service ...... 26 Invoice Content ...... 26 Billing and Payment Procedures ...... 27 Post-Award Project Implementation Information ...... 27

Submission Guidelines ...... 28 Volume 1: Administrative – Page Limit: None ...... 28 Volume 2: Technical Solution – Page Limit: 35 ...... 29 Volume 3: Past Performance – Page Limit: 5 ...... 29 Volume 4: Price Proposal – Page Limit: 4 ...... 29

PCORI RFP # PCO-AOSEPP2021 5 Task Order Request #1: Support Services for the Engagement in Health Research Literature Explorer ...... 30 Background ...... 30 Project Requirements ...... 31 Deliverables/Delivery Schedule ...... 41 Period of Performance ...... 45 Type of Task Order ...... 45 Task Order Evaluation Criteria ...... 45 Volume 1: Technical Solution – Page Limit: 35 ...... 45 Volume 2: Price Proposal – Page Limit: 4 ...... 45

Task Order Request #2: Uptake of PCORI-Funded Research Results in Medicare Policy and Coverage ...... 47 Background ...... 47 Project Requirements ...... 49 Objective 1: Conduct a Landscape Review ...... 49 Objective 2: Curate a List of Policy Sources and Databases ...... 51 Objective 3: Develop Recommendations for Tracking Uptake ...... 51 Specific Requirements/Tasks ...... 51 Deliverables/Delivery Schedule ...... 54 Period of Performance ...... 56 Type of Task Order ...... 56 Task Order Evaluation Criteria ...... 56 Volume 1: Technical Solution – Page Limit: 35 ...... 57 Volume 2: Price Proposal – Page Limit: 4 ...... 57

Task Order Request #3: Update of the PCORI Engagement Rubric ...... 58 Background ...... 58 Project Requirements ...... 59 Specific Requirements/Tasks ...... 60 Deliverables/Delivery Schedule ...... 63 Period of Performance ...... 65 Type of Task Order ...... 65 Task Order Evaluation Criteria ...... 65 Volume 1: Technical Solution – Page Limit: 35 ...... 66 Volume 2: Price Proposal – Page Limit: 4 ...... 66

PCORI RFP # PCO-AOSEPP2021 6 Task Order #3 Attachment A: Overview of PCORI Engagement Evidence Sources and Engagement Resources ...... 67

Task Order Request #4: Studying the Factors and Contexts for Engaging Diverse Research Partners ...... 68 Background ...... 68 Project Requirements ...... 69 Specific Requirements/Tasks ...... 70 Demonstrate Commitment to Diversity, Equity, and Inclusion ...... 78 Deliverables/Delivery Schedule ...... 79 Period of Performance ...... 82 Type of Task Order ...... 82 Task Order Evaluation Criteria ...... 82 Volume 1: Technical Solution – Page Limit: 28 ...... 82 Volume 2: Price Proposal – Page Limit: 4 ...... 82

Task Order Request #5: Resource Center to Support PCORI Activities Related to the Capture of Patient-Centered Cost and Economic Impact Data ...... 83 Background ...... 83 Project Requirements ...... 83 Specific Requirements/Tasks ...... 84 Deliverables/Delivery Schedule ...... 86 Project Lead Authority ...... 87 Project Management ...... 88 Period of Performance ...... 88 Type of Task Order ...... 89 Task Order Evaluation Criteria ...... 89 Volume 1: Technical Solution – Page Limit: 35 ...... 89 Volume 2: Price Proposal – Page Limit: 4 ...... 89

Appendix A: Proposal Evaluation Criteria ...... 90 Technical Approach ...... 91 Project Management Plan ...... 91 Organization and Staffing ...... 92 Past Performance ...... 92

PCORI RFP # PCO-AOSEPP2021 7 Appendix B: Past Performance Questionnaire ...... 94

Appendix C: Post-Award Project Implementation Information ...... 98 Intellectual Property ...... 98 Independent Contractor/Agency ...... 98 Confidentiality and Non-Disclosure ...... 98 Indemnification ...... 99 Insurance ...... 99 Use of Names or Marks; Publicity ...... 99 Conflicts of Interest ...... 99

PCORI RFP # PCO-AOSEPP2021 8 Background

Since our establishment, PCORI has been committed to meaningful involvement by patients, caregivers, clinicians, and other healthcare stakeholders in all our activities, as well as throughout the research we fund. Bringing together all healthcare stakeholders—with patients at the center—to help set research priorities and evaluate applications is our formula for ensuring we fund and conduct the most relevant research possible. We believe that including patients and other stakeholders in the research process, from topic selection through dissemination and implementation of results, will lead to trustworthy and usable information likely to be taken up in practice. We have three engagement goals:

• Build a patient-centered outcomes research (PCOR) community.

• Engage the PCOR community in research.

• Promote dissemination and implementation of PCOR research findings. Although staff throughout PCORI are committed to authentic involvement of patients and other stakeholders, our Engagement and Dissemination and Implementation programs focus specifically on such engagement. These programs reach out to patients and other stakeholders with events and workshops, trainings, funding opportunities, and an Ambassador program, as well as helping stakeholders participate in research topic generation and selection, review of research funding applications, and dissemination and implementation of research findings. We also guide researchers on ways to include patients and other stakeholders in their projects. Additionally, one of PCORI’s strategic goals is to influence clinical and healthcare research funded by others to be more patient-centered. PCORI aims to demonstrate that a patient-centered, stakeholder- driven approach to clinical comparative effectiveness research (CER) has value to other funders of healthcare research by improving the quality, relevance, and usability of study findings. PCORI’s ability to achieve this goal depends, in part, on having scientific and non-scientific (i.e., lay) persons who are well- trained in PCOR concepts and techniques. As PCORI, its portfolio of research and research support awards, the science of engagement, and the engagement of our unique public and patient stakeholders evolve, PCORI requires a diverse group of vendors to provide support that will enhance our ability to achieve key goals. The estimated annual spend under the current Analytical and Operational Support IDIQ vehicle is $3.1 million. Contractors selected through this IDIQ agreement will directly manage the project, interface with key PCORI program staff members, ensure adherence to the approved budget, meet established deadlines, and provide recommendations for PCORI follow-up actions, as appropriate. The Contractors selected through this IDIQ agreement must be able to do the following:

• Demonstrate a thorough understanding of PCORI’s mission and goals.

PCORI RFP # PCO-AOSEPP2021 9 • Demonstrate a thorough understanding of PCOR, CER, and patient and stakeholder engagement principles.

• Demonstrate a commitment to diversity, equity, and inclusion (DEI).

• Demonstrate capacity to provide programmatic and analytic support for PCORI in the following areas: o Science of Engagement ▪ Determine effective patient and stakeholder engagement strategies and their effect on the planning, conduct, dissemination, implementation, uptake, and impact of PCORI-funded research. ▪ Analyze and code award portfolios. ▪ Collect and analyze quantitative and qualitative data, including primary (Contractor collected) and secondary data (PCORI reporting data). ▪ Develop content for resources and products that summarize engagement findings and/or provide evidence-based guidance for the practice of engagement. ▪ Provide maintenance and support of Engagement Tool and Resource Repository. ▪ Assess and facilitate DEI in engagement. o Economic Resource Services Provide support for PCORI activities related to the capture of patient-centered cost and economic impact data as part of PCORI-funded research projects, and related PCORI initiatives. This component includes the following: ▪ Provide technical support to PCORI in considering approaches to cost data collection, sources of cost data, costing methods, or cost analysis to inform PCORI’s work in operationalizing its mandate to collect data on cost and economic impacts of interventions. ▪ Provide subject matter support for convenings of stakeholders or technical experts to identify challenges, explore opportunities, or recommend approaches for addressing questions related to cost data collection or cost analysis. ▪ Provide support for modeling of the health and economic impacts associated with the uptake of PCORI research findings. This activity requires using data from PCORI-funded research and other sources to project health outcomes, changes in resource use (increase or decrease), and estimated cost impacts from specific perspectives. o Environmental Scans, Literature Reviews, and Issue Briefs ▪ Plan and conduct reviews of the existing peer-reviewed, gray, and other literature on clinical, engagement, and health topics for various stakeholder audiences.

PCORI RFP # PCO-AOSEPP2021 10 ▪ Provide support for PCORI’s Engagement in Health Research Literature Explorer. o Education and Training ▪ Develop and maintain scientific and layperson education/training on engagement and CER/PCOR. ▪ Develop training and other education products across a variety of media. ▪ Develop and facilitate trainings and learning tools about engagement for PCORI staff. ▪ Develop, refine, and revise education and training materials for patients and other stakeholders. ▪ Formally evaluate engagement trainings developed for a variety of stakeholder audiences. o Facilitation and Support of Meetings and Community Engagement ▪ Provide content and facilitation support for multi-stakeholder, multi- organizational, and community-specific meetings. ▪ Develop pre- and post-meeting materials, including but not limited to exploratory landscape reviews on key topics, meeting summaries, white papers, synthesis of input, and application to programmatic priorities, as appropriate. ▪ Develop and use innovative methods for stakeholder input, topic generation, and prioritization, including deliberation with underrepresented communities and stakeholder groups. ▪ Employ various methods to engage stakeholder communities effectively. o Public Comment ▪ Review and analyze comments received from the public in response to PCORI calls for public comments. ▪ Develop recommendations and draft revisions of documents posted for formal public comment. PCORI encourages the Offerors to reflect an understanding of the attributes mentioned above in their proposals. All Offerors should demonstrate the following:

• Understanding of CER/PCOR, as well as patient- and community-engaged research principles and methods

• Experience and expertise implementing a clear plan for engaging patients and other stakeholders in projects across all domains addressed in their proposals

• Experience advancing methods and strategies to support patient and stakeholder engagement in research

PCORI RFP # PCO-AOSEPP2021 11 • Experience with and commitment to DEI

• Experience and expertise designing and trialing strategies to address specific challenges in ensuring patient and stakeholder engagement in research

• Understanding of, and meaningful experience working with, different PCORI stakeholder groups involved in research

• Experience and expertise in writing in plain language on technical content

• Experience in graphic design and science and data visualization

• American Medical Association (AMA) Manual of Style proficiency

• Experience and expertise in providing Section 508 compliance services

• Experience with and access to language translation services

• Experience and expertise in licensing, copyright, and digital rights management

PCORI RFP # PCO-AOSEPP2021 12 Statement of Work

Through this RFP, PCORI announces its intention to contract for Analytic and Operational Support for the Engagement of the Public and Patients. PCORI envisions that through this RFP it will contract with multiple awardees whose roles will include the following activities:

• Science of Engagement • Economic Resource Services • Environmental Scans, Literature Reviews, and Issue Briefs • Education and Training • Facilitation and Support of Meetings and Community Engagement • Public Comment PCORI welcomes Offerors to demonstrate the capacity to provide professional services in multiple domains; however, this is not required to qualify for this IDIQ agreement. Only those Offerors qualified in the domain associated with a specific Task Order, as determined by PCORI, will be able to compete for that Task Order. PCORI may consider consortia of Contractors to compete on Task Orders under this IDIQ agreement. PCORI seeks Contractors to enter into agreements to complete work in the following areas: Science of Engagement Provide support for developing a body of evidence around engagement approaches proven effective through PCORI’s funding portfolios and other resources nationally and internationally. PCORI also seeks to develop evidence for the design and trial of strategies to improve or address challenges in engagement. By “engagement in research,” PCORI refers to the meaningful involvement of patients, caregivers, clinicians, and other healthcare stakeholders throughout the research process—from topic selection through design and conduct of research to dissemination and implementation of results. PCORI believes that such engagement can influence research to be more patient-centered, useful, and trustworthy and ultimately lead to greater use and uptake of research results by the public and patients. PCORI seeks to better understand what engagement approaches work best at different stages of the research process and for whom they work, as well as what impact engagement has on a study from the participant and researcher perspectives. PCORI also seeks to develop evidence for DEI in engagement, including assessing and facilitating engagement with populations that have been structurally marginalized and/or historically underrepresented in research and examining the relationships between engagement and studies that align with these community needs. The ultimate objective of these activities is to synthesize this information and make it available as a resource or series of resources for individuals and organizations implementing engagement. The Contractor(s) selected through this IDIQ agreement to compete on Task Orders under this domain must be able to do the following:

PCORI RFP # PCO-AOSEPP2021 13 • Provide support for the evaluation of tools, guides, and resources designed to support engagement in PCOR and CER; and develop, recommend, and apply innovative approaches to engagement in research. o Demonstrate the ability to identify and critically appraise domestic and international resources on the science of engagement. o Demonstrate experience with identifying evidence gaps and developing innovative approaches to patient and stakeholder engagement.

• Provide support for evaluation, analysis, and identification of what approaches to engagement work best at different stages of the research process and for whom they work, as well as what effect engagement has on a study from different perspectives. o Demonstrate the ability to conceptualize, design, and execute investigative or evaluative projects using robust and appropriate methods, including the ability to identify and use innovative methodological approaches. o Demonstrate the ability to find or develop and apply conceptual models or frameworks as the basis of new projects. o Demonstrate familiarity with a range of quantitative, qualitative, or mixed method data collection approaches and the capacity to identify appropriate data sources, plan and conduct appropriate data collection strategies, and incorporate multiple perspectives into the interpretation of results. o Demonstrate the ability to analyze existing administrative and survey data for trends in research partnerships that resulted in effective engagement strategies, meaningful scientific outcomes, or both. o Demonstrate the ability to develop a protocol, work plan, and timeline to develop and implement the necessary framework to acquire relevant information. o Demonstrate the ability to synthesize findings into a variety of media, including presentations, white papers, frameworks, and tools to share findings within and outside of PCORI to a variety of audiences. o Demonstrate the ability to identify gaps in PCORI’s current evaluation tools and data to determine what additional information needs to be collected to answer PCORI’s questions about effective engagement strategies and effects of engagement. o Demonstrate content expertise for key topics and populations of interest, including DEI in research processes, and the range of stakeholders engaged with PCORI (e.g., patients, caregivers, clinicians, health systems, policy makers, payers, purchasers).

• Develop content for resources and products that summarize and convey engagement findings from broad portfolios of PCORI-funded projects, as well as evidence-based guidance for the practice of engagement.

PCORI RFP # PCO-AOSEPP2021 14 o Develop content for tools, guides, and resources for patients, the public, and researchers partnering on CER/PCOR. The resources developed would integrate evidence- and practice-based knowledge about engagement derived from multiple sources including PCORI research and analyses, PCORI-funded research awards, and external sources. o Demonstrate ability to plan strategically in the development and release of integrated products and resources. o Demonstrate ability to partner with communications and marketing agencies and in- house teams.

• Provide support for portfolio coding and analyses for numerous programmatic areas (e.g., the Engagement Awards program). o Demonstrate the ability to develop methods for identifying and synthesizing engagement processes, activities, outcomes, and key themes from the interim and final progress reports. o Demonstrate the ability to synthesize and interpret findings and produce report(s) on promising practices and common challenges, and uncover potential gaps/future areas of opportunity for patient and stakeholder engagement. o Demonstrate the ability to complete abstraction and assessment of select documents related to the Engagement Award according to established taxonomy, including providing suggestions for recommended additions and/or updates.

• Provide support and maintenance for the PCORI Engagement Tool and Resource Repository. o Demonstrate the ability to identify and critically appraise domestic and international resources on the practice of engagement within health research. o Demonstrate the capacity to identify and assemble groups of related tools and resources, or gaps in tools and resources, to package and diffuse to targeted stakeholder groups. o Demonstrate experience describing engagement tool and resource types and potential uses. Economic Resource Services Provide support for PCORI activities related to the capture of patient-centered cost and economic impact data as part of PCORI-funded research projects, and related PCORI initiatives. This component includes the following overall tasks:

• Provide support for convening stakeholders or technical experts to identify challenges, explore opportunities, or recommend approaches for addressing questions related to cost data collection or cost analysis.

PCORI RFP # PCO-AOSEPP2021 15 • Provide support for modeling of the health and economic impact associated with the uptake of PCORI research findings. This activity requires using data from PCORI-funded research and other sources to project health outcomes, changes in resource use (increase or decrease), and estimated cost impacts. The assessment shall analyze the impact on patients, families, payers, or other stakeholders, as specified by PCORI in each Task Order. Contractor(s) selected through this IDIQ agreement to compete on Task Orders under this domain must be able to do the following:

• Demonstrate capacity to conduct literature reviews and develop reports, briefs, and white papers on state of practice or best practices regarding specific methodological questions in cost data collection and cost analysis, drawing on expertise in economics, cost analysis, and accounting. o For illustrative purposes, such topics may include, but are not limited to, capture of patient out-of-pocket costs or non-monetary patient impacts, productivity costs or enhancements, approaches to costing healthcare interventions, and uses of administrative data.

• Demonstrate capacity to assume a “think tank” role to propose areas where methodological advancement would support PCORI’s goals with respect to capturing economic burdens, and, upon PCORI agreement, undertake such work.

• Demonstrate the ability to gather and synthesize input and advice from diverse stakeholders on cost data collection, measurement, and analysis.

• Demonstrate the ability to provide subject matter support for convenings of stakeholders or technical experts, including the ability to refine questions related to cost data collection and measurement, structure meetings or workgroup sessions that will productively address such questions, and moderate these convenings. o As an illustrative example, such convenings might focus on, but are not limited to, identification of gaps in information available to patients and healthcare system decision makers, areas where cost data collection or measurement can contribute, or provision of technical expertise to inform questions related to cost data collection or measurement.

• Demonstrate the ability to develop reports and briefs for professional and lay audiences on methods and promising practices.

• Demonstrate experience developing simulation models projecting health outcomes and costs, ranging from simple low-tech approaches to more complex and intensive models.

• Demonstrate the ability to specify health outcomes to include in a model of regional- and national-level impact, assuming study findings are implemented nationally.

• Demonstrate the ability to specify all relevant measures of healthcare utilization that would meaningfully affect costs for various stakeholder communities.

• Demonstrate familiarity with a range of data sources and the capacity to identify the appropriate data sources that capture the outcomes and impacts described.

PCORI RFP # PCO-AOSEPP2021 16 • Demonstrate the ability to access and use the identified data to derive estimates required for the modeling activity.

• Demonstrate experience with developing methods for assigning costs to outcomes, including specific datasets/data sources to be used to assign costs for budget modeling.

• Demonstrate experience with synthesizing findings and providing reports that place a strong emphasis on the visual representation of data and conclusions, including charts and graphs.

• Demonstrate access to and ability to consult with a range of clinical subject matter experts to help construct models representing realistic care processes across a wide range of clinical conditions.

• Demonstrate understanding of the state of the art of cost data collection and analysis, including familiarity with microcosting approaches as well as use of relevant data sources, and be able to oversee such projects or tasks as identifying strategic approaches to capturing specific costs or developing a catalog of cost data sources from a variety of perspectives.

• Demonstrate access, as necessary or helpful, to other stakeholder experts and expertise, including, but not limited to, payers/insurers, purchasers/employers, and patients.

• Demonstrate the ability to conduct rigorous analyses of the impact on costs and outcomes of implementing research findings into practice. Examples of impact analyses can be found at www.pcori.org/impact. Environmental Scans, Literature Reviews, and Issue Briefs Provide support for generating environmental scans of research findings, literature reviews, and issue briefs based on synthesis of the relevant scientific literature. Products presenting findings from these activities may include white papers, policy briefs, a curated repository of pertinent literature, or other summary materials. The Contractor may be asked to summarize and analyze literature to determine promising practices or to describe where gaps exist in the available evidence (including engagement) in language appropriate to PCORI’s diverse public and patient stakeholder audiences.

Contractor(s) selected through this IDIQ agreement to compete on Task Orders under this domain must be able to do the following:

• Demonstrate the capacity to develop a strategy for conducting a literature review or environmental scan, identifying search terms, databases, and other information sources, including the ability to articulate and assess trade-offs for different solutions.

• Demonstrate experience with conducting clear and concise literature reviews, spanning years of research and findings from multiple study types, and environmental scans, spanning a variety of relevant sources.

• Demonstrate the ability to analyze and synthesize a large body of evidence and identify and succinctly describe the key takeaways.

PCORI RFP # PCO-AOSEPP2021 17 • Demonstrate achievement in preparing formatted, publication-ready reports appropriate for a variety of audiences including, but not limited to, PCORI staff, researchers, patients, purchasers/employers, and payers/insurers.

• Demonstrate proficiency with reference management software, such as EndNote.

• Demonstrate an understanding of qualitative and quantitative methods (research design, and data collection, management, and analysis techniques).

• Demonstrate the ability to support the refinement and maintenance of the Engagement in Health Research Literature Explorer, a curated online collection of peer-reviewed literature on engagement, by categorizing content to meet the needs of diverse end-users. Note: Contractors are not sought to, and will not, conduct systematic reviews, evidence syntheses, or meta-analyses under this domain.

Education and Training Develop and provide scientific and layperson education or training to ensure that all stakeholders have the knowledge and skills necessary to participate in and contribute as research partners in CER/PCOR. Learners may include, but are not limited to, researchers, patients, caregivers, purchasers, payers, clinicians, and policy makers.

Contractor(s) selected through this IDIQ agreement to compete on Task Orders under this domain must be able to do the following:

• Demonstrate experience and expertise in training diverse scientific or lay audiences using evidence-based and innovative practices in adult learning.

• Demonstrate capacity to perform training needs assessments.

• Demonstrate capacity to design, develop, implement, and maintain training programs for multi- stakeholder research partnerships, including efficient use of learning and communication technologies for geographically dispersed audiences.

• Demonstrate experience with the development of new tools and interventions to support participation in research and/or engagement in care and uptake in findings.

• Demonstrate capacity to plan and execute comprehensive training evaluations that include perspectives from an appropriate range of learners.

• Demonstrate the ability to generate a broad reach and uptake of resources throughout the healthcare community.

• Demonstrate the ability to develop workforce trainings on how to assess and support project- level engagement. Facilitation and Support of Meetings and Community Engagement Provide support for gathering healthcare patient and stakeholder communities and contributing to all aspects of systematic information collection from those stakeholder communities to ensure that PCORI-

PCORI RFP # PCO-AOSEPP2021 18 relevant information is obtained. The information to be collected may include, but is not limited to, opinions of these patient and stakeholder communities regarding their research goals and agenda, dissemination and implementation of evidence, and recommendations from the communities for future directions for PCORI, as well as analysis of community perceptions of PCORI and CER/PCOR.

Contractor(s) selected through this IDIQ agreement to compete on Task Orders under this domain must be able to do the following:

• Demonstrate capacity for convening two or more stakeholder communities, having demonstrated experience and expertise in working with one or more of them.

• Demonstrate an understanding of the appropriate method (including roundtables, regional meetings, deliberation, town halls, etc.) and rationale for convening specific individuals and organizations either by community or by combining multiple communities.

• Demonstrate capacity to conduct convenings using multiple modes of gathering participants, including innovative ways of engaging stakeholders in virtual contexts.

• Demonstrate ability to guide discussions with stakeholder communities and collect and analyze all appropriate data.

• Demonstrate ability to use analyzed data to provide recommendations.

• Demonstrate ability to provide meeting summaries (including white papers, executive summaries, etc.).

• Demonstrate ability to engage, and previous success with engaging, diverse stakeholder audiences and groups in health research priority-setting and related activities. o Demonstrate ability to work with communities to identify priorities, assess existing knowledge and familiarity with health topics and research, and communicate principles and examples of patient-centered outcomes research. o Demonstrate mastery of various methods of engaging stakeholder communities. o Demonstrate ability to build trust among diverse communities. o Demonstrate mastery of health research topics and methods. o Demonstrate knowledge of community assets and barriers to engaging in research. o Demonstrate ability to document and reflect in a variety of accessible formats the perspectives and feedback from diverse communities. Public Comment Provide services related to the solicitation, recording, analysis, synthesis, tracking, and reporting on comments collected from the general public on a diverse scope of matters. Public comment projects may take the form of supporting PCORI in less-formal venues, such as seeking feedback on planned or proposed activities during community or multi-stakeholder convenings, or through formal mechanisms, such as those that would emulate a regulatory comment period provided by a unit of government. (Please

PCORI RFP # PCO-AOSEPP2021 19 note: PCORI is not a unit of government.) Public comment opportunities may cover matters of PCORI policy, research methods, evidence syntheses, or other reports. We anticipate that Offerors competing under this domain will demonstrate strong skills in qualitative analysis and access to clinical, methodological, and subject matter experts, as necessary, for robust support to PCORI.

Contractor(s) selected through this IDIQ agreement to compete on Task Orders under this domain must be able to do the following:

• Demonstrate experience and expertise in collecting comments from the public in writing and verbally.

• Demonstrate experience applying appropriate coding methods and developing codebooks for curation and analysis of public comments.

• Demonstrate ability to analyze and report on public comments received.

• Demonstrate ability to develop disposition table(s) of comments received.

• Demonstrate expertise in synthesizing across the body of public comments to provide informative and accessible public-facing summaries.

• Demonstrate ability to develop recommendation memoranda.

PCORI RFP # PCO-AOSEPP2021 20 IDIQ Requirements

Pre-Proposal Conference

A virtual Pre-Proposal Conference will be scheduled with all interested Offerors to discuss the scope of the RFP and provide an opportunity to:

• Review the RFP requirements; • Facilitate a clear understanding of PCORI’s objectives; and • Promote competition. The Pre-Proposal Conference will occur on February 18, 2021, and will last approximately 90 minutes. PCORI will provide each interested Offeror with further instructions regarding the date, time, webinar/conferencing information, and maximum allotment of attendees given limitations. Offerors will need to notify PCORI of their interest by 5:00 p.m. Eastern Time on February 17, 2021, by sending an email to [email protected].

Period of Performance

The period of performance for the IDIQ agreement will be three (3) years from the agreement effective date. The projected IDIQ agreement award date is May 21, 2021.

Type of Agreement

PCORI expects to award multiple IDIQ agreements as a result of this RFP. The agreement type provides an estimate of the work required to be performed over the contract period. The actual quantities of services ordered under the agreement may be significantly less or possibly more than identified in the deliverables. Contractors must be able to perform all activities described in the IDIQ agreement and applicable Task Order(s). PCORI encourages the use of Contractor teaming arrangements and/or subcontractors to ensure that project teams have the necessary expertise to accomplish the work. Task Orders All work under this agreement will be ordered by the issuance of individual Task Orders. Five (5) Task Order requests for proposal are included as part of this solicitation. Task Orders may be issued on a firm fixed price, time and materials, or cost reimbursable basis. All Task Orders exceeding the micro-purchase threshold ($150,000) will be subject to competition among the IDIQ agreement holders. The IDIQ agreement holders will be issued Task Orders under this agreement describing the specific task(s) to be performed, the required deliverables, and authorized direct labor hours, travel, supplies, materials, other direct costs, and total costs. No work effort is authorized under this agreement except as expressly described in the Task Orders. The Contractor shall ensure that adequate resources are dedicated to satisfy the requirements of Task Orders, including, but not limited to, furnishing the necessary personnel, material, and services as required. In addition, inherent in providing these goods and services, the Contractor shall provide the

PCORI RFP # PCO-AOSEPP2021 21 supervision and management effort necessary for efficient and effective administration and control of work performed under the Task Order. For time and materials Task Orders: The Contractor will be required to provide supporting documentation for such rates, which may include a cost element breakdown of each Loaded Hourly Labor Rate (including profit) in accordance with the Contractor’s cost accounting system, as well as any other supporting information the Procurement Department deems necessary. For time and materials and cost reimbursable Task Orders: The Contractor will be required to provide technical status reports that describe, at a minimum, the technical progress per the requirements of the Task Order. The frequency (e.g., monthly) of when these reports are due will be outlined in the Task Order request. The Contractor will also be required to provide a final report, which shall consist of (at a minimum) the work performed and results obtained for the entire Task Order period of performance. The report shall be in sufficient detail to describe comprehensively the results achieved, and it should be submitted before the last day of the Task Order performance period. A technical status report will not be required for the period when the final report is due. Task Order Requests for Proposal PCORI shall issue Task Orders under this IDIQ agreement. Each individual Task Order request may be firm fixed price, time and materials, or cost reimbursable, or any combination of the three. Task Order requests may include performance incentives (negative and positive), as applicable to the performance success of the Task Order. All Task Order requests will incorporate all terms and conditions of the IDIQ agreement. In addition, the proposed Task Order request will include the following to the extent applicable to individual Task Orders:

• A Statement of Work or other statement describing the work to be performed, the deliverables, the period of performance, the points of contact, inspection, and acceptance of deliverables.

• The type of Task Order.

• The proposal due date/time and the method of delivery for proposals.

• Specific instructions on what to include in the proposal submission. This may include, but is not limited to, written responses summarizing technical and price approaches.

• Evaluation factors and their relative order of importance.

• Any other pertinent information deemed appropriate. Technical proposals will be evaluated against the criteria set forth in each Task Order request for proposal; the criteria in each Task Order request for proposal may be specifically tailored to each individual requirement. The technical proposal should address each criterion and contain information necessary for satisfactory performance of the required services. First Five Task Order Requests As part of this RFP, PCORI will issue the first five Task Orders in conjunction with the award of multiple- award IDIQ agreements. The first five Task Orders will be awarded along with the underlying IDIQ agreement. PCORI will award the IDIQ agreement and the first five Task Orders resulting from this RFP to

PCORI RFP # PCO-AOSEPP2021 22 the responsible Offerors whose offer conforming to the RFP will be most advantageous to PCORI. Information for these five Task Orders is included in this RFP. IDIQ Agreement Evaluation Criteria IDIQ proposals will be evaluated using the following criteria and evaluation factor rates. Specific details regarding the proposal evaluation process, rating factor descriptions, and other general requirements are included in Appendix A to this RFP.

Category Weight of Rating Factor

Technical Approach 35%

Project Management Plan 15%

Organization and Staffing 20%

Past Performance 30%

Price Proposal PCORI expects Offerors to provide two sets of pricing in their price proposals.

The overall base pricing will support the labor categories in the overarching IDIQ agreement.

Prices will be proposed based on established catalog or commercial market prices. Offerors who are General Services Administration (GSA) Schedule holders are encouraged to offer pricing that is competitive with their currently awarded Schedule rates.

For cost reimbursable and time and materials Task Orders, the ceiling amounts will be established in the execution of each Task Order.

When evaluating the price proposal, the Task Order pricing will be used to evaluate price reasonableness. A determination shall be made as to the reasonableness and the completeness of both price proposals. PCORI shall use their price analysis to determine whether the proposed prices are fair and reasonable. Price will not receive an adjectival rating.

The Offeror shall submit functional responsibilities and labor qualifications for each labor category under this RFP. Functional responsibilities shall include the principal duties to be performed by the labor category. Labor qualifications shall include experience and/or education requirements for all labor categories. In addition, the Offeror shall provide a narrative outlining the rationale for the functional responsibilities and qualifications proposed for each category. The proposed labor qualifications will be incorporated into the resulting contract, thereby becoming the standards for use during performance of the effort. These labor categories shall be referenced in responding to Task Order requests in which the requirements align with their functional responsibilities.

PCORI RFP # PCO-AOSEPP2021 23 Award without Communications or Clarifications

PCORI reserves the right to award without communications or clarifications. Therefore, it is in the best interest of each Offeror to include its most favorable terms in its initial submission. However, PCORI reserves the right to request communications or clarifications.

Best and Final Offers

Subsequent to receiving the original proposals, PCORI reserves the right to notify all technically acceptable Offerors within the competitive range and to provide them an opportunity to submit written best and final offers (BAFOs) for the agreement at a designated date and time. BAFOs shall be subject to the late submissions provision and the withdrawal or modification of proposals provision of this RFP. After receipt of a BAFO, no discussions shall be reopened unless PCORI determines that it is clearly in PCORI’s best interest to do so (i.e., it is clear that information available at that time is inadequate to reasonably justify Contractor selection and award based on the BAFOs received). If discussions are reopened, PCORI shall issue an additional request for BAFOs to all technically acceptable Offerors still within the competitive range.

Withdrawal or Modification of Proposals

An Offeror may modify or withdraw its proposal upon written notice via email to [email protected] not later than the closing date and time for receipt of proposals.

Late Submissions

Late proposals, requests for modification, or requests for withdrawal shall not be considered unless a late modification of a successful proposal makes terms more favorable for PCORI.

Retention of Proposals

All proposal documents shall become the property of PCORI, be retained by PCORI, and not be returned to the Offerors.

Documentation Requirements

The Contractor may be required to provide documentation to support its legal ability to operate in the United States.

Basis of Compensation to the Contractor

PCORI expects to award an agreement for the technical and cost proposal that is proposed, negotiated with PCORI during the BAFO process, and listed in the agreement executed between the organizations. Any Contractor quality issues that result in the re-drafting of work or increased labor required to meet deliverables during the performance of the contract are the financial responsibility of the Contractor, and re-work will be done at the Contractor’s expense.

PCORI RFP # PCO-AOSEPP2021 24 Post-Award Debriefing

PCORI’s Procurement Department will provide a timely notification to all unsuccessful Offerors once an award has been made. All unsuccessful Offerors may request a post-award debriefing by providing a written request to [email protected] within three (3) business days after receiving the award notification. If a debriefing is given to the requesting Offerors, PCORI will work with the unsuccessful Offeror to schedule a reasonable date and time. PCORI will provide verbal or written debriefs.

PCORI RFP # PCO-AOSEPP2021 25 Post-Award Information

Place of Performance

The Contractor is required to provide the facilities necessary to fulfill requirements under the agreement. The Contractor shall choose its staff or acquire the necessary personnel support and provide suitable work facilities.

Hours of Service

The Contractor shall be available Monday through Friday between 8:30 a.m. and 5:30 p.m. Eastern Time. PCORI has regular observance of federal holidays: New Year’s Day; Birthday of Martin Luther King, Jr.; Washington’s Birthday; Memorial Day; Independence Day; Labor Day; Veterans Day; Thanksgiving Day; and Christmas Day. The Contractor must comply with PCORI’s rules and regulations, all required forms, and any changes in procedures. The Contractor will be informed of any such changes and updates, as necessary, by PCORI’s Procurement Department. Upon the request of employees or other persons with disabilities participating in official business, the Contractor must arrange necessary and reasonable accommodations for the impaired individual(s) per Section 508 compliance.

Invoice Content

The Contractor shall invoice on a monthly basis. The invoice shall include the period of performance covered by the invoice and the level of effort for each task performed. All hours and costs shall be reported by division, task, and labor category and shall be provided for the current billing month and in total from project inception to date. If teaming or subcontracting is proposed, one consolidated invoice from the primary Contractor shall be submitted in accordance with other terms and conditions of the agreement. The invoice must be signed by an individual authorized by the Contractor. The Contractor shall provide the invoice data in spreadsheet form with the following detailed information (the listing shall include separate columns and totals for the current invoice period and the project to date, identified by division and task):

• Contractor’s name • Contractor’s address • Purpose of the invoice • Date the invoice was submitted • Contractor labor category • Labor rate • Details of the goods or services provided (for goods, please indicate the quantity) or description of the deliverable • Travel expenses (if applicable and will need to be provided with receipts)

PCORI RFP # PCO-AOSEPP2021 26 • Other direct costs (if applicable and will need to be provided with receipts) • Monthly and total cumulative hours worked • Total value of the invoice

The Contractor must submit the final invoice within 30 days after the agreement termination date. The final invoice must be marked “FINAL” and shall be submitted to PCORI. The Contractor agrees and understands that an incomplete invoice will be rejected and returned without payment.

Billing and Payment Procedures

PCORI prefers electronic invoicing. Invoices shall be provided to the Finance Department and to the Project Lead on a monthly basis. Billing address: PCORI Attn: Finance Department 1828 L St NW Suite 900 Washington, DC 20036 Email: [email protected]

Post-Award Project Implementation Information

Additional specific details regarding post-award project implementation and other general requirements are included in Appendix C to this RFP.

PCORI RFP # PCO-AOSEPP2021 27 Submission Guidelines

Submissions should be organized in four separate volumes: (i) Administrative, (ii) Technical Solution, (iii) Past Performance, and (iv) Price Proposal, sent as separate files. All text should be Arial or Times New Roman font, no less than 11 point, with one-inch margins and single spaced. Graphics and tables may be included. We accept MS Word, MS Excel, and Adobe PDF formats. The Offeror is advised that its offer, if accepted by PCORI, will form a binding agreement. Therefore, Offerors should take care to properly address the requirements set forth in the RFP. These instructions set forth the requirements for preparation and submission of proposals and establish the format and specific content of each offer. This is to assure completeness and suitability for evaluation purposes and the cohesiveness of the resulting agreement. Be sure that the proposal explains how your firm will fully satisfy all performance requirements of the intended agreement. Submit a complete proposal; proposals that are not complete may be determined unresponsive and rejected. Proposals should be sent electronically to [email protected]. Proposals not received by the specified due date and time will not be considered. Offerors shall submit proposals in response to this solicitation electronically as follows.

Volume 1: Administrative – Page Limit: None

Each submission must include a cover letter with the following information:

• Offeror’s name and mailing address • Reference to the solicitation (RFP) number (PCO-AOSEPP2021) • Technical and contract points of contact (name, phone number, and email address) • Business type, size status, or type of organization or institution (e.g., other than small business, small business, small–disadvantaged small business, historically black colleges and universities [HBCUs], HUBZone, nonprofit organization, etc.) • Dun & Bradstreet Number (DUNS) • Federal tax ID number (EIN, TIN, SS) • Affirmation that the proposal is valid for at least 60 days • A statement specifying the extent of agreement with all terms, conditions, and provisions included in the solicitation and agreement to furnish any or all items upon which prices are offered at the price set opposite each item • Acknowledgment of any amendments by reference

PCORI RFP # PCO-AOSEPP2021 28 Volume 2: Technical Solution – Page Limit: 35

Volume 3: Past Performance – Page Limit: 5

All past performance must be for comparable ongoing projects or those completed within the last three (3) years. Offerors should have relevant non-PCORI references complete the attached Past Performance Questionnaire contained in Appendix B of this RFP; references should submit completed questionnaires directly to the RFP email address, [email protected], by 5:00 p.m. Eastern Time on April 14, 2021.

Volume 4: Price Proposal – Page Limit: 4

The written price proposal shall contain the following:

• Offerors will demonstrate how they arrived at their proposed pricing by detailing labor categories, labor rates, and travel costs. • Proposed pricing must include estimated direct expenses including all subcontractor labor (if applicable), travel costs, and other direct expenses. Include burden as applicable, specifying burden rates and burden calculations.

PCORI RFP # PCO-AOSEPP2021 29 Task Order Request #1: Support Services for the Engagement in Health Research Literature Explorer

Competition for this Task Order is limited to those Offerors qualifying under the “Environmental Scans, Literature Reviews, and Issue Briefs” domain of the PCORI IDIQ agreement for Analytic and Operational Support for the Engagement of the Public and Patients.

Background

To support the conduct and study of engaged research, PCORI launched the Engagement in Health Research Literature Explorer (“Literature Explorer”) in 2017 to make scientific literature about engagement in health research more accessible to researchers, stakeholder partners, and others who want to learn about engagement. The Literature Explorer addresses a gap in other literature search engines, such as PubMed/Medline, that have not developed standardized search terms for stakeholder engagement in research. PCORI developed a tailored literature searching strategy that is updated annually to produce a searchable, cataloged collection of more than 1,500 peer-reviewed publication citations made available to the public on PCORI’s website.

As the field of engaged research is rapidly growing and information about the conduct and impact of engagement is in high demand, PCORI seeks Contractor support to revisit the goals and functionality of this resource to ensure that the collection of literature is supporting those who want to learn from the peer-reviewed literature. Through this RFP, PCORI also seeks to synthesize the publications included in the Literature Explorer to produce valuable findings for the field. The current tags and filters in the Literature Explorer were developed to organize the existing literature base broadly. Still, PCORI recognizes there may be an opportunity to add specificity to the search functionality through additional content tags to help end-users better connect to information that is of greatest interest to them. PCORI is also looking for opportunities to synthesize and learn from the peer-reviewed literature included in this resource to provide more guidance to the field on best practices and to identify other opportunities to advance PCORI’s study of the science of engagement.

Since its creation, the Literature Explorer has remained a frequently visited page on the PCORI website. PCORI encourages health researchers, particularly PCORI awardees, partners, and research staff, to use this resource when developing engagement methods and plans for their project proposal. PCORI staff circulate periodic updates to highlight notable articles on engagement to staff. The Literature Explorer has also been used by several external research teams to source articles for original research (e.g., Jull et al. [2019] BMC Med Res Methodol and Harrison et al. [2019] Health Expect). Thus, the Literature Explorer has not only cataloged existing articles but has influenced the development of new frameworks, models, and scholarship surrounding patient and stakeholder engagement in health research.

The Literature Explorer includes articles sourced from PubMed/Medline, articles identified in a review of topically relevant peer-reviewed journals that are not indexed in PubMed, and articles that are recommended for inclusion by PCORI staff or other end-users. To maintain the relevance of the Literature

PCORI RFP # PCO-AOSEPP2021 30 Explorer, a vendor team has provided ongoing support of this resource since its creation to ensure that scientific articles are searched, screened, and coded for inclusion on a frequent basis.

The Literature Explorer currently provides articles organized by examples of engagement practices, evaluations of the impacts of engagement, models and frameworks for engagement, and editorials and commentaries that share perspectives on the inclusion of patients and other stakeholders in health research. It also includes tags (i.e., filters) for the types of stakeholders involved in the research and at which phases to aid in the search and retrieval of relevant information.

PCORI is currently working with a Contractor to gather feedback on the existing content and functionality of the Literature Explorer from a small but diverse group of end-users. Through this evaluation, PCORI seeks to learn more about what our stakeholders want and need to know about research engagement and how the Literature Explorer can be better tailored and maintained to meet these needs. Recommendations from this evaluation and future assessments will be used to inform the future scale, format, and features of the resource.

Project Requirements

Through this Task Order issued within the engagement IDIQ domain “Environmental Scans, Literature Reviews, and Issue Briefs,” PCORI seeks analytic support to (1) conduct a scoping literature review to describe the content of the peer-reviewed literature on engagement within PCORI’s Engagement in Health Research Literature Explorer; (2) develop and execute strategies to improve the usefulness and use of the Literature Explorer for a variety of stakeholder end-users; and (3) provide ongoing administrative support for the screening, coding, and maintenance of the online Literature Explorer resource.

The Contractor will maintain the current processes for the screening, coding, and maintenance of the Literature Explorer while simultaneously conducting a scoping literature review of the articles included in the resource. Following the completion of the scoping review and the generation of recommendations to improve the usefulness and use of the Literature Explorer, the Contractor will propose and implement updated processes to maintain the Literature Explorer throughout the remainder of the period of performance.

The Contractor will work closely with PCORI staff and a subcommittee of PCORI’s Advisory Panel on Patient Engagement (PEAP) to receive strategic direction throughout the period of performance.

PCORI anticipates that additional related tasks will become available beyond the initial period of performance to continue administrative support of the Literature Explorer, conduct further analysis of literature captured in the Literature Explorer related to PCORI’s needs and priorities, and develop dissemination materials including a manuscript or other outputs from the activities in this scope of work.

Conduct a Scoping Literature Review

• The Contractor will conduct a scoping literature review of the articles included in PCORI’s Literature Explorer to summarize the breadth and depth of information about engagement in health research available. Findings from the scoping review will be used to identify important gaps

PCORI RFP # PCO-AOSEPP2021 31 in the peer-reviewed literature, implement improvements to enhance the usefulness and use of the Literature Explorer, and identify opportunities for future literature synthesis on research engagement topics. • The engagement topics included in the review will be proposed by the Contractor and refined with input from PCORI but should include topics related to how to do engagement well, the difference engagement makes, and key factors about the engagement and research contexts, such as: o The difference engagement makes: ▪ In the research process ▪ For research quality and use ▪ For researchers and institutions ▪ For partners engaged in the research o How to do engagement well: ▪ Resourcing engagement ▪ Strategies for establishing partnerships ▪ Practices related to facilitating DEI in engagement ▪ Training research team members ▪ Engagement successes and challenges ▪ Barriers and facilitators for the conduct of engagement ▪ Strategies for maintaining consistent participation ▪ Methods and metrics for evaluating engagement within projects ▪ Strategies for sustaining partnerships o Engagement and research contexts: ▪ Format or structure through which engagement is facilitated (e.g., centralized advisory body) ▪ Engagement activities and methods ▪ Experiences of those who are engaged ▪ Types of engagement practices (e.g., input, consultation, collaboration or shared leadership) ▪ Demographic characteristics of engaged partners (e.g., racial/ethnic background, rural/urban locations, age) ▪ Type of health research in which engagement is occurring (e.g., health conditions being studied, intervention types) ▪ Characteristics of study subjects (e.g., target populations [inclusion of racial/ethnic minorities]) ▪ Characteristics of study setting (e.g., rural, urban) • There may be 2,000 articles included in the Literature Explorer by the time of this analysis. Due to the quantity of included articles, PCORI may provide guidance to prioritize or focus the review for some article types (e.g., articles that provide evidence for the science of engagement), but the scoping review will be expected to include all available articles in the Literature Explorer. • Upon completion of the scoping literature review, PCORI may consider funding for additional related activities in an extended future period of performance. Such activities may include co-

PCORI RFP # PCO-AOSEPP2021 32 production of a manuscript reporting findings in the peer-reviewed literature, or requests for additional literature synthesis to inform PCORI’s guidance and best practices for engagement.

Improve the Usefulness and Use of the Literature Explorer

• The Contractor will generate and implement recommendations to increase the usefulness and use of the Literature Explorer by PCORI staff, PCORI awardees, PCORI stakeholders, health service researchers, other funders, and those working in adjacent fields with an interest in engagement. • Using findings from the scoping literature review, past user-testing, reviews of other literature repositories, and discussions with PCORI staff, the Contractor will develop recommendations to improve the usefulness and use of the Literature Explorer, including: o Reconceptualized goals and scope for the Literature Explorer relative to the current needs of the field and the state of the literature; o Enhanced format and functionality (e.g., new filters) relative to needs of the resources and considering other bibliographic tools; o Revisions to article inclusion/exclusion criteria; o New or revised written descriptions of the information available on the Literature Explorer website; o New or improved website functions (e.g., data export); o Multi-media materials to promote use of the site (e.g., quarterly email digest; brief video clips for lay users); and o Promotional strategy. • Recommendations will be finalized in collaboration with PCORI, including Communications staff and associated PCORI web contractors, who will support the implementation of online changes to the Literature Explorer. • The Contractor will create any necessary materials to support the implementation of approved recommendations.

Maintain the Literature Explorer

• On an ongoing basis, the Contractor will search, screen, and code relevant articles for inclusion in the Literature Explorer to produce quarterly updates to the online resource. This work will occur throughout the period of performance. • The Contractor will maintain detailed administrative records and an updated project management protocol to reflect all aspects of the review process. • As necessary, the Contractor will identify and implement opportunities to refine and streamline existing processes to produce quarterly updates to the Literature Explorer website. • This Task Order may have future opportunities beyond the contract period to continue administrative support for the Literature Explorer.

Specific Requirements/Tasks

The Contractor must perform the following tasks:

PCORI RFP # PCO-AOSEPP2021 33 Objective 1 – Conduct a scoping literature review to describe the types and quantity of information about multi-stakeholder engagement in health research published in the peer-reviewed literature. Task 1.1 Develop a scoping literature review protocol

The Contractor will develop a protocol to guide the conduct of the scoping literature review. The purpose of the protocol is to allow PCORI and key stakeholders to provide input on the methods to be used in the review and ensure the rigor of the proposed work. The protocol should include the key research questions and their prioritization, a detailed description of data extraction and entry methods, including definitions for each attribute to be analyzed, draft and final abstraction forms as relevant, staff roles, and analysis methods to be used in the review. If necessary, the protocol should describe how the scoping review will prioritize or focus on the most important articles (e.g., science of engagement articles). The protocol should also describe how data from the literature review can be translated into new or revised content tags for filtering within the online Literature Explorer. The Contractor will facilitate input from PCORI staff and a subcommittee of the PEAP in the development of the protocol, revise the protocol in response to PCORI’s input, and submit a final protocol.

Deliverables:

• One (1) or two (2) versions of a draft protocol and a final version in response to comments from PCORI and a subcommittee of the PEAP on the draft protocol. • Draft slides and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with a subcommittee of the PEAP to solicit feedback on the draft protocol. • A presentation to a subcommittee of PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the draft protocol.

Task 1.2 Conduct a scoping literature review

The Contractor will conduct a scoping review of all publications included in PCORI’s Engagement in Health Research Literature Explorer to catalog the types and quantity of information available in each publication according to the approved literature review protocol. As outlined in Project Management Task B, the Contractor will plan and lead biweekly project status calls (i.e., every two weeks) with key PCORI staff to discuss the status, observations, analytic decisions, and challenges experienced in the conduct of the scoping literature review.

Deliverables:

• See Project Management Task B. • Final analytic files (e.g., extraction tables) produced in the conduct of the literature review.

Task 1.3 Present the findings to PCORI

PCORI RFP # PCO-AOSEPP2021 34 The Contractor will present a synthesis of the findings from the scoping literature review to key PCORI staff and a subcommittee of the PEAP. PCORI will review the draft slide deck and provide comments. The Contractor will revise the draft slide deck in response to PCORI’s input and submit a final slide deck.

Deliverables:

• A draft slide deck of the findings and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with key PCORI staff and a subcommittee of the PEAP to solicit feedback on the literature review findings. • A presentation to PCORI staff. • A presentation to a subcommittee of PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the literature review findings.

Task 1.4 Develop a report of the analysis methods, findings, and recommendations

The Contractor will develop a report with full project documentation that will be used to share findings with PCORI staff and leadership to support internal conversations about knowledge gaps that PCORI could fill through further evaluation efforts, and to inform the potential creation of a scientific manuscript with findings from the scoping literature review. The report should include an executive summary, literature review methods, findings, synthesis of the current state of the literature including notable gaps, and recommendations for areas for further literature synthesis to identify best practices for engagement. The Contractor will develop an outline, one (1) or two (2) versions of a draft report, and a final version of the report in response to PCORI’s input.

Deliverables:

• An outline of the components to be included in the final analytic report. • One (1) or two (2) versions of a draft report from the scoping literature review and a final version of the report in response to comments from PCORI and additional input gathered under Task 1.3.

Objective 2 – Develop and execute strategies to improve the usefulness and use of the Engagement in Health Research Literature Explorer for a variety of stakeholder end- users. Task 2.1 Generate recommendations to improve the usefulness and use of the Literature Explorer

The Contractor will generate recommendations to improve the usefulness (i.e., value) and use (i.e., utilization) of the Literature Explorer by considering the need for this resource from a multi-stakeholder perspective, and using findings from the scoping literature review, findings from the stakeholder feedback activity conducted in 2020, and an exploration into the content and functionality of other literature repositories.

PCORI RFP # PCO-AOSEPP2021 35 The Contractor will propose revised goals/purpose(s) for the Literature Explorer and consider how to meet those needs through revised content and functionality. Recommendations should consider the types of literature included in the resource (i.e., inclusion/exclusion criteria), the tags or filters available to facilitate a content search of the included publications, improvements for any written descriptions or explanatory material on the website (e.g., supplemental materials), an implementation plan to update and maintain the Literature Explorer, and potential strategies to promote the resource widely. The recommendations should include the rationale and an assessment of the resources, time, and feasibility of implementing and maintaining changes to the Literature Explorer. PCORI will consider the trade-offs between different approaches to improve the usefulness and use of the resource when approving a final set of recommendations.

The Contractor will create a draft and final slide deck with the proposed recommendations to facilitate a meeting with a subcommittee of the PEAP to solicit strategic feedback. The Contractor will revise the recommendations in response to PCORI and PEAP input and submit a final memo of recommendations.

Deliverables:

• A draft memo outlining revised goals, suggested improvements with an implementation plan, and promotional strategies for the Literature Explorer and a final version in response to PCORI and PEAP input. • A draft slide deck and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with key PCORI staff and a subcommittee of the PEAP to solicit feedback on the draft recommendations to improve the usefulness of the Literature Explorer. • A presentation to PCORI staff. • A presentation to a subcommittee of PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the recommendations to improve the usefulness and use of the Literature Explorer.

Task 2.2 Implement revisions to improve the usefulness of the Literature Explorer

The Contractor will work with PCORI staff, including the PCORI Communications Team, to implement agreed-upon revisions to improve the usefulness of the Literature Explorer. Activities include updating existing article coding as necessary with information from the scoping literature review and producing materials that will facilitate updates to the website, such as data spreadsheets and text about the tool to include on the web page. In addition, the Contractor will update PCORI’s existing management protocol, which PCORI will share at the start of the project, to outline revised processes required to maintain the Literature Explorer.

Deliverables:

• MS Excel spreadsheet with new or revised coding content for included articles for the PCORI Communications Team to use to update the Literature Explorer website. • Draft written website content (e.g., web copy, revised supplemental materials) and final written website content in response to PCORI input.

PCORI RFP # PCO-AOSEPP2021 36 • An updated management protocol outlining revised processes required to maintain the Literature Explorer.

Task 2.3 Promote the use of Literature Explorer

The Contractor will work with PCORI staff to implement strategies outlined and approved in Task 2.1 to promote the use of the Literature Explorer within PCORI and the broader research community.

Deliverables:

• Draft written or digital promotional content (e.g., quarterly email digest of recent articles, web copy, etc.) and final written or digital promotional content in response to PCORI input.

Objective 3 – Provide ongoing maintenance for the Engagement in Health Research Literature Explorer by searching, screening, and tagging relevant articles to include in the online resource. The Contractor will update the Literature Explorer on a quarterly basis for the duration of the Task Order period of performance, first according to an existing management protocol, then later according to a revised protocol developed under Task 2.2. As noted above, specific tasks under Objectives 1 and 2 will occur concurrently with the ongoing maintenance of the Literature Explorer. This overlap may require articles that are added to the Literature Explorer during the scoping literature review and subsequent generation of recommended changes to the Literature Explorer to be re-reviewed according to a revised management protocol.

The tasks and deliverables outlined below reflect current processes for the maintenance of the Literature Explorer. PCORI staff will share additional detail about the existing processes and deliverables upon award. PCORI and the Contractor may choose to modify specific deliverables to accommodate management protocol revisions implemented during the period of performance to enable the Contractor to independently manage the Literature Explorer with appropriate quality control measures.

Task 3.1 Identify articles for screening

The Contractor will identify articles for screening through multiple sources, including through an established search string in PubMed/Medline (including newly identified articles and a proportion of articles from a backlog of publications to be reviewed), publication in relevant journals not indexed in Medline (e.g., Journal of Patient-Reported Outcomes), PCORI’s awardee publication tracking efforts, and recommendations made by PCORI staff. The Contractor will also de-duplicate articles and input bibliographic information into the appropriate tab in the Master Workbook for title/abstract screening and/or full-text review.

Deliverables:

• MS Excel spreadsheet of articles (including duplicates) identified by source for title/abstract and full-text review

Task 3.2 Screen articles for inclusion

PCORI RFP # PCO-AOSEPP2021 37 The Contractor will source articles as necessary and conduct a single-reviewer title/abstract and full-text screening for articles identified from PubMed/Medline and the specific journals noted in the project protocol using rigorous inclusion/exclusion criteria. The Contractor will also maintain a spreadsheet with documentation of screening decisions.

Deliverables:

• MS Excel spreadsheet reporting all articles excluded to date at the title/abstract review with bibliographic information, codes applied for title/abstract exclusion criteria, and notes describing any unusual issues or difficult coding decisions.

Task 3.3 Tag included articles and extract publication data

The Contractor will apply content tags for included articles. Tags should identify articles that may be of high interest to PCORI, and the Contractor should document the rationale for these tags based on guidance developed in the management protocol.

The Contractor will request input from the PCORI project officer, as needed, to resolve outstanding tagging and abstraction questions before submitting monthly deliverables.

Deliverables:

• MS Excel spreadsheet reporting all included articles to date with bibliographic information, content tags, a tag and rationale for notable articles, and notes describing any unusual issues or difficult coding decisions. • PRISMA Flow Diagram summarizing the articles sourced, screened, included, and excluded in the Literature Explorer to date.

Task 3.4 Facilitate updates to the Literature Explorer website

The Contractor will submit a spreadsheet of changes to implement in the online Literature Explorer database formatted for uploading to the PCORI website as described in the management protocol. The changes to the website database will include new article entries and revisions to existing article coding that result from re-coding efforts or to correct errors identified in a monthly spot check of article entries (see below). The Contractor will spot check a sample of changes implemented to the website to ensure accuracy.

Deliverables:

• MS Excel spreadsheet of updates for the Literature Explorer website.

Task 3.5 Revise management protocol

Throughout the period of performance, the Contractor will develop and implement strong processes to ensure the timeliness and quality of the activities to maintain the Literature Explorer. PCORI will share an

PCORI RFP # PCO-AOSEPP2021 38 existing management protocol and provide guidance on the activities for the maintenance of the Literature Explorer while the scoping literature review is being conducted.

The Contractor will refine and maintain a management protocol for the maintenance of the Explorer on a quarterly basis. The management protocol should contain clear and accurate documentation for all Contractor activities related to the maintenance of the Literature Explorer, including a record of key decisions and notable changes to the project that occur over time.

Deliverables:

• A revised management protocol for the Literature Explorer.

Task 3.6 Review search strategy

On an annual basis, the Contractor will propose revisions to the Literature Explorer search strategy including but not limited to the PubMed/Medline search string based upon analysis of relevant articles not captured by the current search strategy, updated MeSH terms, and other analyses as appropriate (e.g., reviewing keywords for included articles). Upon consultation with PCORI staff, Contractor shall test and implement search string revisions. Once the revised search string has been tested and confirmed, the Contractor will update and finalize a search string and strategy revision memo, ensuring that the final document reports on any critical decision or discussion points worth noting for the historical record.

Deliverables:

• Draft search string and strategy revision memo and a final version in response to PCORI input.

Project Management The Contractor will employ strong project management practices throughout the period of performance to ensure project quality and manage the project timeline, deliverables schedule, and agreement budget. The Contractor will maintain shared files and project documentation in a Box.com folder established by PCORI. The Contractor will proactively seek input from PCORI staff as needed to ensure the successful planning and execution of the project. The Contractor must provide PCORI staff at least five (5) business days to provide input on project deliverables.

Task A Conduct project kickoff call

The Contractor will schedule and coordinate a project kickoff call at a date and time convenient for the PCORI and Contractor teams. The call will provide an introduction between the Contractor personnel and PCORI personnel who will be involved with the Task Order and an opportunity to discuss technical, management, and reporting procedures. The Contractor shall provide the following 48 hours prior to the kickoff call:

• Introduction of personnel; • Roles and responsibilities; • Overview of project tasks, including schedule and deliverable dates;

PCORI RFP # PCO-AOSEPP2021 39 • Quality control procedures; • Reporting requirements (e.g., period status reports); • Initial deliverables; and • Additional issues of concern.

Deliverables:

• A kickoff call agenda and meeting materials within 48 hours prior to the call. • A kickoff call with PCORI staff. • A call summary within three (3) business days of completion of the call that includes all elements covered in the kickoff call and listed above.

Task B Biweekly project status calls

The Contractor will plan and lead biweekly project status calls (i.e., every two weeks) with key PCORI staff. Topics for discussion should include:

• Progress to date on tasks/sub-tasks, submitted/upcoming deliverables, and any points of interest from the preceding two weeks. • Status, observations, analytic decisions, and challenges experienced in the conduct of the scoping literature review. • Questions and/or possible adjustments to discuss with PCORI as they arise. • Discussion of draft submissions and clarification of feedback provided.

Deliverables:

• A biweekly status call agenda 24 hours prior to the call. • A biweekly status call with PCORI staff. • A call summary of notable discussions and key decisions within three (3) business days of completion of each biweekly status call.

Task C Monthly Status Report

The Contractor will submit monthly status reports throughout the period of performance, including:

• Activities during reporting period, by task, to include ongoing activities, new activities, activities completed, deliverables submitted for that period, and progress to date on all above-mentioned activities. Start each section with a brief description of the task. • Problems and corrective actions taken. Also include issues or concerns and proposed resolutions to address them, including risk mitigation plans. • Personnel gains, losses, and status. • PCORI actions required. • Schedule (showing major tasks, milestones, and deliverables; planned and actual start and completion dates for each).

PCORI RFP # PCO-AOSEPP2021 40 • Financial status including cumulative invoiced costs total to date, including hours and cost by labor categories (including planned vs. actual rates), hours billed by individual (including planned vs. actual hours), cost to completion, and variance. Also include subcontractor labor and other subcontractor costs (including planned vs. actual hours).

Deliverables:

• Monthly status report

Deliverables/Delivery Schedule

All deliverables must be completed and include, but are not limited to, the following:

Task Deliverable Quantity and Due Date

Media

OBJECTIVE 1: Conduct a scoping literature review to describe the types and quantity of information about multi-stakeholder engagement in health research published in the peer-reviewed literature. 1.1 1 or 2 versions of a draft literature review 1 MS Word Within five (5) weeks protocol document of Task Order Effective Date

1.1 Draft slides that, when finalized, the Contractor 1 MS Within five (5) weeks will use to facilitate a remote meeting with a PowerPoint of Task Order Effective subcommittee of the PEAP to solicit feedback on slide deck Date the draft protocol 1.1 Final slides in response to PCORI input that the 1 MS One (1) week after Contractor will use to facilitate a remote meeting PowerPoint receiving PCORI’s with a subcommittee of the PEAP to solicit slide deck comments feedback on the draft protocol 1.1 A presentation to a subcommittee of PCORI’s -- To be determined PEAP 1.1 Written summary of input from key PCORI staff 1 MS Word Three (3) business and a subcommittee of the PEAP on the draft document days after input protocol meeting 1.1 Final literature review protocol in response to 1 MS Word Two (2) weeks after comments from PCORI and the PEAP on the draft document receiving PCORI’s final protocol comments 1.2 Final analytic files produced in the conduct of the To be Within ten (10) literature review determined months of Task Order Effective Date

PCORI RFP # PCO-AOSEPP2021 41 1.3 A draft slide deck of the findings that, when 1 MS Within nine (9) finalized, the Contractor will use to facilitate a PowerPoint months of Task Order remote meeting with key PCORI staff and a slide deck Effective Date subcommittee of the PEAP to solicit feedback on the literature review findings 1.3 A final slide deck of the findings in response to 1 MS One (1) week after PCORI input that the Contractor will use to PowerPoint receiving PCORI’s facilitate a remote meeting with key PCORI staff slide deck comments and a subcommittee of the PEAP to solicit feedback on the literature review findings 1.3 A presentation to PCORI staff -- To be determined 1.3 A presentation to a subcommittee of PCORI’s -- To be determined PEAP 1.3 A written summary of input from key PCORI staff 1 MS Word Three (3) business and a subcommittee of the PEAP on the literature document days after input review findings meeting 1.4 An outline of the components to be included in 1 MS Word Within seven (7) the final analytic report document months of Task Order Effective Date 1.4 1 or 2 versions of a draft report from the scoping 1 MS Word Within eight (8) to literature review document nine (9) months of Task Order Effective Date 1.4 A final version of the scoping literature review 1 MS Word Later of (i) ten (10) report in response to comments from PCORI and document months after Task additional input gathered under Task 1.3 Order Effective Date or (ii) two (2) weeks after receiving PCORI’s final comments OBJECTIVE 2: Develop and execute strategies to improve the usefulness and use of the Engagement in Health Research Literature Explorer for a variety of stakeholder end-users. 2.1 A draft memo outlining revised goals, suggested 1 MS Word Within 11 months of improvements with an implementation plan, and document Task Order Effective promotional strategies for the Literature Explorer Date 2.1 A draft slide deck that, when finalized, the 1 MS Within 11 months of Contractor will use to facilitate a remote meeting PowerPoint Task Order Effective with key PCORI staff and a subcommittee of the slide deck Date PEAP to solicit feedback on the draft recommendations to improve the usefulness of the Literature Explorer

PCORI RFP # PCO-AOSEPP2021 42 2.1 A final slide deck in response to PCORI input that 1 MS One (1) week after the Contractor will use to facilitate a remote PowerPoint receiving PCORI’s meeting with key PCORI staff and a subcommittee slide deck comments of the PEAP to solicit feedback on the draft recommendations to improve the usefulness of the Literature Explorer 2.1 A presentation to PCORI staff -- To be determined 2.1 A presentation to a subcommittee of PCORI’s -- To be determined PEAP 2.1 A written summary of input from key PCORI staff 1 MS Word Three (3) business and a subcommittee of the PEAP on the document days after input recommendations to improve the usefulness and meeting use of the Literature Explorer 2.1 A final memo in response to PCORI and PEAP 1 MS Word Two (2) weeks after input outlining revised goals, suggested document receiving PCORI’s final improvements with an implementation plan, and comments promotional strategies for the Literature Explorer 2.2 New or revised coding content for included 1 MS Excel Within 12 months of articles for the PCORI Communications Team to spreadsheet Task Order Effective use to update the Literature Explorer website Date 2.2 Draft written website content (e.g., web copy, 1 MS Word Within 12 months of revised supplemental materials) document Task Order Effective Date 2.2 Final written website content (e.g., web copy, 1 MS Word One (1) week after revised supplemental materials) in response to document receiving PCORI’s PCORI input input 2.2 An updated management protocol outlining 1 MS Word Within 13 months of revised processes required to maintain the document Task Order Effective Literature Explorer Date 2.3 Draft written or digital promotional content (e.g., 1 MS Word Within 13 months of quarterly email digest of recent, notable articles) document Task Order Effective Date 2.3 Final written or digital promotional content (e.g., 1 MS Word One (1) week after quarterly email digest of recent, notable articles) document receiving PCORI’s in response to PCORI input input OBJECTIVE 3: Provide ongoing maintenance for the Engagement in Health Research Literature Explorer by searching, screening, and tagging relevant articles to include in the online resource. 3.1 Articles (including duplicates) identified for 1 MS Excel Quarterly after Task screening by source for title/abstract and full-text workbook Order Effective Date review

PCORI RFP # PCO-AOSEPP2021 43 3.2 Articles excluded to date at the title/abstract 1 MS Excel Quarterly after Task review with bibliographic information, codes workbook Order Effective Date applied for title/abstract exclusion criteria, and notes describing any unusual issues or difficult coding decisions 3.3 Articles included to date with bibliographic 1 MS Excel Quarterly after Task information, content tags, a tag and rationale for workbook Order Effective Date notable articles, and notes describing any unusual issues or difficult coding decisions 3.3 PRISMA Flow Diagram summarizing the articles 1 MS Word Quarterly after Task sourced, screened, included, and excluded in the document Order Effective Date Literature Explorer to date 3.4 Coding updates for the Literature Explorer 1 MS Excel Quarterly after Task website workbook Order Effective Date 3.5 A revised management protocol for the Literature 1 MS Word Quarterly after Task Explorer document Order Effective Date 3.6 Draft search string and strategy revision memo 1 MS Word Within 12 months of document Task Order Effective Date 3.6 Final search string and strategy revision memo in 1 MS Word Within 13 months of response to PCORI input document Task Order Effective Date Project Management Materials Task A kickoff call agenda and meeting materials 1 MS Word 48 hours prior to the A document kickoff call

1 MS PowerPoint slide deck Task A kickoff call with PCORI staff – To be determined A Task Kickoff call meeting summary 1 MS Word Within three (3) A document business days

following the kickoff

call Task Biweekly project status call meeting agenda 1 MS Word 24 hours prior to each B document or biweekly call calendar invitation Task Biweekly project status call summary 1 MS Word Within three (3) B document business days

PCORI RFP # PCO-AOSEPP2021 44 following each biweekly call Task Monthly status report 1 MS Word Due on the 1st of C document every month

Period of Performance

The period of performance for Task Order #1, Support Services for the Engagement in Health Research Literature Explorer, will be 18 months from the Task Order Effective Date of September 1, 2021.

Type of Task Order

PCORI expects to award one (1) time and materials Task Order. This agreement type provides an estimate of the work required to be performed over the agreement period. Contractors must be able to perform all activities described in the Task Order. PCORI encourages the use of teaming arrangements to ensure that project teams have the necessary expertise to accomplish the work.

Task Order Evaluation Criteria

Task Order proposals will be evaluated using the following criteria and evaluation factor rates. Specific details regarding the proposal evaluation process, rating factor descriptions, and other general requirements are included in Appendix A to this RFP.

Category Weight of Rating Factor

Technical Approach 35%

Project Management Plan 15%

Organization and Staffing 20%

Past Performance 30%

Volume 1: Technical Solution – Page Limit: 35

Volume 2: Price Proposal – Page Limit: 4

The written price proposal shall contain the following:

• Offerors will demonstrate how they arrived at their proposed pricing by detailing labor categories, labor rates, and travel costs.

PCORI RFP # PCO-AOSEPP2021 45 • Proposed pricing must include estimated direct expenses including all subcontractor labor (if applicable), travel costs, and other direct expenses. Include burden as applicable, specifying burden rates and burden calculations.

PCORI RFP # PCO-AOSEPP2021 46 Task Order Request #2: Uptake of PCORI-Funded Research Results in Medicare Policy and Coverage

Background

PCORI’s mission is to produce and promote high-integrity, evidence-based information that comes from research guided by patients, caregivers, and the broader healthcare community to help patients, clinicians, policy makers, and other stakeholders make informed health decisions. This is reflected in PCORI’s strategic plan, which specifies that one of PCORI’s three strategic goals is to speed the uptake and implementation of results from PCORI’s funded research.

Guided by the PCORI Evaluation Framework, PCORI assesses the organization’s progress on this goal. Evaluation efforts so far include monitoring and analyzing awardee publications and tracking the uptake or use of results, such as in clinical decision tools, in practice guidelines, and in public and payer policies. This Task Order is focused on tracking use of PCORI-funded research in payer policies specifically. As of December 2019, we had identified 45 citations (see slide 13) of PCORI-funded research in a range of policy documents, including reports from government agencies, international organizations, and think tanks. As PCORI’s portfolio of completed research studies continues to grow, and as policy makers have more time to use the evidence, we anticipate more citations in the future.

PCORI RFP # PCO-AOSEPP2021 47 Figure 1: Metrics for Tracking Uptake (this Task Order focuses on payer policies)

There are many challenges to tracking use of PCORI’s research in policy outlets, especially in a manner that is both systematic and sustainable (e.g., balancing time/resources required with the sheer volume of possible information). One specific challenge is that there is no existing sampling frame of all relevant policy documents. Moreover, not all policy documents are easily found or searchable for citations. For example, with private payers in particular, there is little publicly available information on the specific evidence used to support decisions.

Today, most citations we have identified in policy sources have come from the Altmetric Policy Tracker.2 This resource allows us to systematically capture bibliometric citations to PCORI-funded research; however, we are limited to the policy outlets and authors captured within Altmetric’s tool. Outside of this tool, we have been able to manually track some examples of policy uptake on an ad hoc basis. Typically, these are examples shared via word-of-mouth.

This Task Order will support PCORI’s efforts to track the uptake of PCORI-funded research results into Medicare policy and coverage. We envision this Task Order as the first in a series, with each subsequent Task Order focusing on a different aspect of public or private payer policy or other types of policies. Future topics may include clinical policy (e.g., professional societies; facility/health system associations);

2 How does Altmetric track policy documents? https://help.altmetric.com/support/solutions/articles/6000129069- how-does-altmetric-track-policy-documents-

PCORI RFP # PCO-AOSEPP2021 48 Medicaid and the Children’s Health Insurance Program (CHIP); purchaser/employer policy; and other public policies at the local, city, and state level.

We have three broad objectives for this Task Order looking at uptake in Medicare policy:

1. Better understand how public payers (with an emphasis on Medicare) use, consult, or consider health services research in policy development. 2. Obtain a curated list of policy sources relevant to Medicare that can be added to automated citation tracking tools such as Altmetric and Overton.io.3 3. Obtain recommendations for new or enhanced ways to track the use/citation of research in Medicare policy documents.

For this work, we are interested in both enacted policy (e.g., coverage decisions) as well as proposed policies and policy recommendations, such as work by think tanks, advisory groups, and others. The following list of organizations are examples of how we have operationalized “Medicare policy,” but the list is by no means exhaustive:

• Centers for Medicare & Medicaid Services (CMS) • Medicare Coverage Database (MCD) • Medicare Evidence Development & Coverage Advisory Committee (MEDCAC) • Medicare Payment Advisory Commission (MedPAC) • Medicare Administrative Contractors (MACs) making Local Coverage Determinations (LCDs) • Federal advisory groups (e.g., Advisory Committee on Immunization Practices) • Think tanks (e.g., Bipartisan Policy Center, Brookings Institution, American Enterprise Institute, Commonwealth Fund) and other groups that make Medicare policy recommendations

Project Requirements

PCORI seeks a Contractor with health policy expertise to lead a landscape review on public payer (particularly Medicare) policy makers’ use of health research; compile a list of policy documents and/or databases that can be used for citation tracking; and provide recommendations on approaches to tracking potential uptake. Findings from the landscape review, the list of documents, and recommendations shall be compiled and submitted in a final report.

Objective 1: Conduct a Landscape Review The objective of the landscape review is to better understand how public payers use and consider health research in coverage decisions and other policy matters. Under this task, the Contractor shall review information (published research, websites, gray literature, etc.) on the use/consultation/uptake of health research evidence in public payer policy and synthesize key findings. Because of Medicare’s role as the largest public payer, and its unique role influencing wider payer policy, the landscape review should

3 We received additional guidance from Altmetric that policy documents/sources must have a PDF. Ideally, each document will also have its own landing page. This is an example from the World Health Organization of a landing page that can be added to the Altmetric Policy Explorer.

PCORI RFP # PCO-AOSEPP2021 49 emphasize Medicare. However, we are also interested in understanding public payers more generally and would like to include groups such as the Medicaid and CHIP Payment and Access Commission (MACPAC) as part of this review. From this, we hope to better understand channels through which PCORI-funded research may be used and how those channels might be documented.

The landscape review should be guided by these themes and questions, as well as any others based on the Contractor’s expertise and that may arise during the review process.

• Evidence consultation process: We want to understand the processes through which health research evidence is consulted by key decision makers and, if possible, the extent to which these processes result in citations that we can track. Some guiding questions include: o To what extent is health research cited in policy coverage? ▪ What kinds of documents are most typically cited (e.g., systematic reviews, individual papers/results, policy analyses, reports, other)? ▪ Are some payers more likely to cite research than others? How does Medicare compare? Are there differences between LCDs and National Coverage Determinations (NCDs) in this respect? o What is the process through which Medicare Administrative Contractors, MEDCAC, MedPAC, MACPAC, and others consult/review/consider healthcare research? ▪ How often does this happen? ▪ If not through citations, how else do decision makers within these groups demonstrate that decisions are well supported? o What key organizations advise on public payer policy or make policy recommendations? ▪ Some organizations of interest include MEDCAC, MedPAC, and MACPAC; however, other advisory councils may be relevant as well. ▪ What is the relative scope and responsibility of the different organizations? • Medicare Coverage Database (MCD): We want to understand how to best and most responsibly use the MCD as a source. Provide an overview of the MCD, including notes on: o What information is available in the MCD, and why? What are the caveats? For example, how should we interpret the quality and completeness of information within the MCD? o Where else are coverage decisions communicated? o What is the difference in terms of processes, decision makers, and documentation between LCDs and NCDs? Who are the key decision makers? o How often are coverage decisions updated or changed? o For tracking uptake, which documents in the MCD should we prioritize and why? • Type of policy documents and other policy authors: We are interested in a wide range of policy documents, including technical reports, policy analyses, and other types of policy recommendations that extend beyond coverage decisions. o Other than coverage determinations, what types of policy documents (including policy recommendations from external organizations) are most relevant for public payer and Medicare policy? o Which documents are most likely to cite research?

PCORI RFP # PCO-AOSEPP2021 50 Objective 2: Curate a List of Policy Sources and Databases The Contractor should develop and implement a strategy to curate a comprehensive list of Medicare- relevant policy sources, including but not limited to websites, individual documents, and repositories or databases, that can be routinely searched for citation of PCORI-funded research. This list is meant to facilitate efficient, effective searching for future citation of PCORI research through automated tools, and/or a more manual process, as necessary, for key reports. The Contractor should provide guidance on which sources to prioritize and why. The prioritization should take into consideration the types of research that PCORI funds and the research areas that have the most potential for uptake. PCORI will work with the Contractor to share information about the portfolio of funded research to inform this exercise.

Automated citation tracking tools such as Altmetric and Overton.io rely on web-scrapers to search for references to specific DOIs, PMIDs, and other document identifiers. Both tools require eligible websites to be publicly available and typically to have PDF versions of key documents. We will share the final list of policy sources with these groups with the aim of adding new sources to their policy repositories.

Objective 3: Develop Recommendations for Tracking Uptake The Contractor will provide PCORI with recommendations to help elucidate the potential for tracking the uptake or use of evidence in the Medicare context, including in actual coverage decisions as well as in proposed legislation and policy recommendations. Based on the Contractor’s expertise and findings from the landscape review, the Contractor will summarize the strengths and weaknesses of alternative approaches to systematically track uptake of PCORI-funded research into Medicare policy and recommend the best ways. We are particularly interested in expanding our approaches beyond tracking citations. We are also interested in recommendations to make the citation tracking process more efficient and effective.

Specific Requirements/Tasks

The Contractor must perform the following tasks:

Task 1.1 – Kickoff Call & Project Management The Contractor will schedule and coordinate a project kickoff call at a date and time convenient for PCORI and Contractor teams. The call will provide an introduction between the Contractor personnel and PCORI personnel who will be involved with the Task Order and an opportunity to discuss technical, management, and reporting procedures. The Contractor shall provide the following at the kickoff call:

• Introduction of personnel • Roles and responsibilities • Overview of project tasks, including schedule and deliverable dates • Quality control procedures • Additional issues of concern

The Contractor will also schedule and coordinate monthly project update calls with PCORI staff. This will be an opportunity to provide status updates and discuss challenges.

PCORI RFP # PCO-AOSEPP2021 51 Deliverables:

• A kickoff call agenda and meeting materials • Written summary of kickoff call that includes all elements covered in the call and listed above • Monthly progress call agenda and meeting materials, including a brief memo on the project status, observations, and challenges • Written summary of monthly call Task 1.2 – Protocol The Contractor will develop a protocol to guide the conduct of the landscape review (Task 1.3) and the curation of a list of policy sources (Task 1.4). The protocol ensures that the Contractor and PCORI staff have a shared understanding of the process and desired deliverables. The protocol should include:

• Key research questions, including additional or alternative questions proposed by the Contractor • Detailed description of search strategy, including search strings and databases for both tasks o For the list of policy documents, this search strategy should take into consideration PCORI’s national research agenda and the types of studies that PCORI funds. • Detailed description of the methods, screening processes, and inclusion/exclusion criteria for both tasks • Proposed template for the final list of policy sources (Task 1.4) • Plan for sequencing—the Contractor shall propose whether the landscape review and curation of a list of policy sources shall be conducted sequentially or simultaneously

The Contractor will facilitate input from PCORI staff, revise the protocol in response to PCORI’s input, and submit a final protocol.

Deliverables:

• Draft protocol for PCORI review and comment • Final protocol Task 1.3 – Conduct Landscape Review (Objective 1) The Contractor shall review information on the uptake/use of health research evidence in public payer policy, with an emphasis on Medicare policy and coverage. The Contractor should respond to the guiding questions described in the protocol, as well as any others that arise during the review.

Deliverables:

• Draft synthesis of methods and key findings, including comment on the current state of the literature and notable gaps in available information • See Task 1.6 – Final Report for additional final deliverables Task 1.4 – Curate List of Policy Sources (Objective 2) The Contractor shall identify key policy outlets or authors, publications, and/or databases relevant to Medicare policy, including enacted policy and coverage determinations, proposed policy, and policy recommendations. The final list should take into consideration PCORI’s priority areas for research funding,

PCORI RFP # PCO-AOSEPP2021 52 which involve both systems-level studies and individual-level studies that look at specific treatment options.

The list should include the following elements, as well as any others recommended by the Contractor, particularly key elements identified in the recommendations (Task 1.5):

• Source name • Source category (e.g., think tank, government agency) • URL • Type of policy document (e.g., coverage, policy recommendations, government policy report) • Relevance to Medicare • Prioritization or rank • Rationale for inclusion and prioritization • Limitations • Compatibility with citation tools such as Altmetric or Overton.io (i.e., public availability and PDF format) • Description for access requirements (for those not publicly available in PDF form) • Comment on timeliness of resources (e.g., is this resource updated with new information? How often? When should we seek to refresh the information from this source?)

The final list of sources will be shared with citation tracking groups such as Altmetric and Overton.io; however, the Contractor should also include highly relevant sources that do not meet the criteria for those tools. We are also interested in reviewing “close calls”—documents that were seriously considered for inclusion in the final list but ultimately excluded. Examples of these documents should be provided in a separate tab of the final list with a note on why the source was excluded.

Deliverables: • Draft synthesis of methods and key findings o Key findings should include overarching observations from the review—such as themes about the types of documents, frequency with which sources include cited reference lists, trends among excluded sources, etc.—and any takeaways from the process that could inform the next Task Order in this series. • Draft curated list of policy sources o As applicable, this should include sources that were strong candidates for inclusion but were ultimately excluded. • See Task 1.6 – Final Report for additional final deliverables Task 1.5 – Develop Recommendations (Objective 3) Based on the landscape review, identification of policy sources, and the Contractor’s expertise, the Contractor shall recommend approaches for systematically and efficiently tracking the uptake, use, or consultation of PCORI-funded research in Medicare coverage decisions and related policy documents. As applicable, recommendations should propose new approaches (i.e., outside of citation tracking) and/or enhanced approaches to make the citation tracking process more efficient and effective. The recommendations should summarize the strengths and weaknesses of available options. The

PCORI RFP # PCO-AOSEPP2021 53 recommendations should also provide guidance on which policy sources (from Task 1.4) to prioritize and a recommended timetable for refreshing the list of sources to ensure the information remains current.

Deliverables: • Draft recommendations • See Task 1.6 – Final Report for additional final deliverables Task 1.6 – Final Report The Contractor shall develop a final report summarizing the approach, key findings, and recommendations. The report will serve as an internal reference guide on the Medicare policy landscape for our ongoing work to track the uptake and use of PCORI-funded research results. Some elements of the report (noted below) will be shared with other internal or external stakeholders and should operate as standalone documents in those instances. The report should include:

• Executive summary (standalone document—will be shared internally with PCORI staff) • Recommendations for PCORI to expand and enhance tracking of use of PCORI-funded research in policy documents • Landscape review: o Synthesis of methods and key findings, including comment on the current state of the literature and notable gaps in available information o EndNote citation file of included articles and documents for PCORI’s EndNote library • Policy sources: o Description of the methods and main findings from identifying and curating a list of policy sources o Excel file of the final list of included sources (standalone file—to be shared with citation tracking organizations)

Deliverables: • Outline of the final report for PCORI review and comment • Complete draft of the final report for PCORI review and comment o Separate draft components should be shared as they are available; see delivery schedule for more details. • Final report

Deliverables/Delivery Schedule

Prior to commencing work, the Contractor shall develop and submit to PCORI a project schedule that includes project tasks and activities, milestones, deliverables, and PCORI’s review and approval of draft and final deliverables. All deliverables must be completed, and they include but are not limited to the following, with each deliverable to be provided to PCORI:

Task 1.1 – Kickoff Call & Project Management Deliverable Deliverable Description Due Date

PCORI RFP # PCO-AOSEPP2021 54 1(a) Kickoff call agenda 48 hours prior to the kickoff call, which should take place within 2 weeks of the Task Order Effective Date 1(b) Kickoff call summary, including a recap Within 3 business days of the completion of of all elements covered in the call the kickoff call 1(c) Monthly progress call with PCORI staff, Agenda and memo provided no later than 2 including a brief written memo on the business days in advance of the progress call project status, observations, and challenges 1(d) Monthly progress call summary Within 1 week of the completion of the monthly progress call Task 1.2 – Protocol Deliverable Deliverable Description Due Date 2(a) Draft protocol for PCORI review and Within 4 weeks of Task Order Effective Date comment 2(b) Final protocol for PCORI approval Within 6 weeks of Task Order Effective Date Task 1.3 – Landscape Review Deliverable Deliverable Description Due Date 3(a) Draft synthesis of methods and key Draft of synthesis should be provided as soon findings for PCORI review and comment as it is ready, but no later than week 26 (draft final report) 3(b) Final synthesis of methods and key Deliverable included in final report findings (included in final report) 3(c) EndNote library of all included sources Deliverable included in final report Task 1.4 – Curate List of Policy Sources Deliverable Deliverable Description Due Date 4(a) Draft synthesis of methods and key Draft of synthesis should be provided as soon findings for PCORI review and comment as it is ready, but no later than week 26 (draft final report) 4(b) Final synthesis of methods and key Deliverable included in final report findings (included in final report) 4(c) Draft list of included sources for PCORI Draft list should be provided no later than review and comment. As applicable, a week 26 (draft final report) second tab should list sources that were “close calls” for inclusion in the final list but were ultimately excluded. 4(d) Final list of included (and nearly Deliverable included in final report included) sources (appendix to final report; Excel file)

PCORI RFP # PCO-AOSEPP2021 55 Task 1.5 – Develop Recommendations Deliverable Deliverable Description Due Date 5(a) Draft recommendations for PCORI Draft of recommendations should be review and comment provided as soon as it is ready, but no later than week 26 (draft final report) 5(b) Final recommendations should be Deliverable included in final report included as a section in the final report Task 1.6 – Final Report Deliverable Deliverable Description Due Date 6(a) Outline Within 20 weeks of Task Order Effective Date 6(b) Draft of final report for PCORI review Within 26 weeks of Task Order Effective Date and comment. The draft will include all sections and appendices described above. Sections can be delivered as soon as they are ready for review, but no later than within 26 weeks of Task Order Effective Date. The Executive Summary should operate as a standalone document to be shared with other PCORI staff. 6(c) Final version of final report for PCORI Within 28 weeks of Task Order Effective Date approval, responding to feedback provided during the review of the draft final report

Period of Performance

The period of performance for Task Order #2, Uptake of PCORI-Funded Research Results in Medicare Policy and Coverage, will be seven (7) months from the Task Order Effective Date.

Type of Task Order

PCORI expects to award one (1) fixed price Task Order. This agreement type provides an estimate of the work required to be performed over the agreement period. Contractors must be able to perform all activities described in the Task Order. PCORI encourages the use of teaming arrangements to ensure that project teams have the necessary expertise to accomplish the work.

Task Order Evaluation Criteria

PCORI RFP # PCO-AOSEPP2021 56 Category Weight of Rating Factor

Technical Approach 35%

Project Management Plan 15%

Organization and Staffing 20%

Past Performance 30%

Volume 1: Technical Solution – Page Limit: 35

• Technical Approach: Systems, Methods, and Approach (Suggested Page Limit: 15) • Organization and Staffing (Suggested Page Limit: 5) • Project Management Plan (Suggested Page Limit: 5) • Optional: Additional past performance of work related to Medicare policy and/or IDIQ domain (“Environmental Scans, Literature Reviews, and Issue Briefs”) (Suggested Page Limit: 10)

Volume 2: Price Proposal – Page Limit: 4

The written price proposal shall contain the following:

PCORI RFP # PCO-AOSEPP2021 57 Task Order Request #3: Update of the PCORI Engagement Rubric

Competition for this Task Order is limited to those Offerors qualifying under the “Science of Engagement” domain of the PCORI IDIQ agreement for Analytic and Operational Support for the Engagement of the Public and Patients.

Background

One of PCORI’s strategic goals is to influence clinical and healthcare research funded by others to be more patient-centered. PCORI aims to demonstrate that a patient-centered, stakeholder-driven approach to clinical research has value to healthcare research by improving the quality, relevance, and usability of study findings. PCORI’s ability to achieve this goal depends, in part, on having scientists, patients, and other stakeholders who are well-trained in PCOR and prepared to work collaboratively on a PCOR research team.

Engaging patients and other stakeholders as partners in research projects can take many forms. However, many researchers, patients, and other stakeholders lack clarity about when and how to engage as partners in planning, conducting, and disseminating research. To address the need for guidance on creating meaningful stakeholder partnerships in patient-centered clinical CER, PCORI’s Engagement Rubric was created in 2014 in collaboration with the PEAP.

The Engagement Rubric is grounded in a set of principles that lay a foundation for equitable partnerships (reciprocal relationships; co-learning; partnerships; and transparency, honesty, and trust), and provides real-world examples drawn from PCORI’s research portfolio of ways to engage stakeholder partners throughout each phase of a research study. Notably, given the lack of firm evidence about varied engagement methods at the time of its development, the Engagement Rubric was intentionally nonprescriptive. However, it has been utilized to provide guidance to applicants for PCORI funding about potential engagement approaches to consider. Since the development of the Engagement Rubric, PCORI has developed a number of other resources and guidance documents to inform engagement practice, which can be found on the PCORI Engagement Resources page, and in this Task Order #3 Attachment A.

In light of PCORI’s reauthorization and the expansion of the evidence around engagement methods and practice, including several PCORI analyses and a qualitative research study, PCORI is poised to update the Engagement Rubric. Using findings from evaluations and analyses of our research portfolio, a qualitative study of PCORI projects’ engagement, practice-based knowledge acquired over PCORI’s first ten years, and the robust literature that has proliferated in the last decade, the updated Engagement Rubric will incorporate evidence-based guidance for meaningful engagement, broaden the spectrum of research partner involvement, and more explicitly address the needs of different stakeholder audiences.

More specifically, to reflect PCORI’s key priorities for advancing engagement both within PCORI and in the field, the Engagement Rubric will:

• Support the use of appropriate, effective engagement practices • Encourage strong collaborations and sustainable partnerships

PCORI RFP # PCO-AOSEPP2021 58 • Make engagement more accessible, easier, and more efficient • Improve DEI in engagement • Diffuse learnings about engagement, its barriers, and its facilitators

Project Requirements

The Contractor shall provide the following types of services/support under this Task Order: Through the issuance of this Task Order request for proposal, PCORI announces its intention to contract for an update of the PCORI Engagement Rubric. PCORI expects to award one (1) Task Order to the successful Offeror. The Offeror will be expected to demonstrate the capability to provide professional service in support of the following activities and tasks: 1. Development and completion of an environmental scan. These activities could include, but are not limited to, development of search strategy; collaboration with PCORI on setting criteria for the scan; review and synthesis of the peer-reviewed and gray literature on engagement policies, requirements, and guidance; conducting key informant interviews; and writing a report. 2. Coordination of strategic planning with PCORI staff. These activities could include, but are not limited to, gathering and synthesis of relevant PCORI evidence on engagement; designing a planning process; producing materials for planning meetings; facilitation of strategic planning discussions with PCORI staff and potentially other advisors to identify an approach to the Engagement Rubric update; production of meeting minutes; and synthesis of discussions into an actionable strategic plan and roadmap for developing content for the Engagement Rubric update. 3. Plan and conduct a convening meeting to garner additional stakeholder input. These activities could include, but are not limited to, identification and recruitment of stakeholder attendees with relevant expertise; preparing materials for the meeting; preparing attendees to effectively participate in the meeting; facilitating large group discussion as well as breakout sessions; conducting the meeting remotely or in person; and drafting meeting minutes and recommendations. 4. Production of content for the Engagement Rubric. These activities should be informed by the previous three tasks and could include, but are not limited to, synthesis of evidence and practice- based knowledge on engagement, DEI, and other relevant topics; translation of evidence into practical guidance; and development of content to potentially be used in a number of media (e.g., web, print, video, audio, or other formats). Production of the final Engagement Rubric for public display will be completed under a separate opportunity. 5. Create presentations and publications regarding Engagement Rubric development. These activities could include, but are not limited to, understanding the landscape of engagement in research (informed by the environmental scan) and summarizing the overall work completed in collaboration with PCORI of updating the Engagement Rubric; articulating priorities for various stakeholders (e.g., funders, research organizations, patients, and other stakeholders) to advance engagement in research; and highlighting connections to other areas, such as healthcare design and delivery, in the findings on engagement in research.

The Contractor will operate at the direction of and receive guidance from PCORI.

PCORI RFP # PCO-AOSEPP2021 59 Specific Requirements/Tasks

The Contractor must perform the following tasks:

Task 1: Environmental Scan of Engagement Policies, Requirements, and Guidance One of PCORI’s priorities is to better understand organizational policies and findings regarding engagement outside of PCORI. As part of this work, PCORI seeks an environmental scan to inform the Engagement Rubric update and consider recommendations for possible future PCORI activities. The Contractor will conduct an extensive environmental scan of existing guidance and policies related to engagement, including among current and past PCORI awardees, other funders and government agencies, and other sources identified in collaboration with PCORI staff. As part of this environmental scan, the Contractor will (i) summarize relevant organizational policies or requirements for engagement; (ii) develop a list of proposed key informants based on a set of criteria developed with PCORI; (iii) develop key informant interview protocol; (iv) conduct up to 30 key informant interviews and up to four (4) focus groups if appropriate; and (v) submit a report on the findings. The Contractor will present a summary of the environmental scan to the PEAP or a PEAP working group during one of its meetings.

Deliverables:

• One (1) or two (2) versions of draft protocol(s) and a final version in response to comments from PCORI. • Draft slides and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with a subcommittee of the PEAP to solicit feedback on the draft approach. • A presentation to a subcommittee of PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the draft protocol. • A final environmental scan report. • A draft slide deck of the findings and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with key PCORI staff and a subcommittee of the PEAP to solicit feedback on the environmental scan findings. • A presentation to PCORI staff. • A presentation to PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the environmental scan findings.

Task 2: Strategic Plan for Engagement Rubric Update Given that the PCORI Engagement Rubric serves many purposes—including providing guidance to PCORI awardees and presenting evidence and real-world examples of engagement—the Contractor will facilitate a strategic planning process with PCORI staff. As part of this strategic planning process, the Contractor will (i) review PCORI-generated evidence about engagement and further synthesize where needed (aided by PCORI’s documentation and classification of past evidence; see Task Order #3 Attachment A); (ii) summarize applicable evidence from the environmental scan; (iii) integrate findings from PCORI’s work in DEI, including integration of principles for DEI developed by the PEAP; (iv) in addition to regular project

PCORI RFP # PCO-AOSEPP2021 60 meetings, facilitate seven (7) to ten (10) planning meetings with PCORI staff and other advisors as indicated; and (v) in collaboration with PCORI, develop a strategic plan and roadmap to develop updated Engagement Rubric content that includes both the ultimate format for the Engagement Rubric as well as the anticipated content.

Deliverables:

• One (1) or two (2) versions of a draft strategic plan and a final version in response to comments from PCORI and the PEAP on the draft plan. • Draft slides and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with a subcommittee of the PEAP to solicit feedback on the draft protocol. • A presentation to PCORI staff. • A presentation to PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the draft protocol. • A final strategic plan report. • A draft slide deck of the findings and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with key PCORI staff and the Engagement, Dissemination, and Implementation Committee (EDIC) to solicit feedback on the strategic plan. • A written summary of input from key PCORI staff and a subcommittee of the EDIC on the strategic plan. • Integration of input into final strategic plan.

Task 3: Plan and Conduct a Convening Meeting to Garner Additional Stakeholder Input PCORI is a multi-stakeholder environment, and we engage stakeholders in all aspects of our work. To elicit additional stakeholder input and perspectives on the update of the Engagement Rubric, the Contractor will plan and conduct a convening meeting of up to 50 organizations that utilizes both remote technology and an in-person component. In the event that circumstances permit a remote format only, the plan will adjust accordingly. As part of the meeting planning process, the Contractor will (i) develop an agenda with PCORI input; (ii) curate an invitation list of key stakeholder representatives with PCORI input; (iii) conduct the meeting (remote or in person, as appropriate) in collaboration with PCORI; (iv) develop a meeting summary; and (v) integrate outcomes from the meeting into the strategic plan for the Engagement Rubric update. PCORI will provide meeting facilities and any necessary travel support for attendees for an in- person meeting.

Deliverables: • One (1) or two (2) versions of a draft agenda and a final version in response to comments from PCORI on the draft agenda. • Draft slides and a final version in response to PCORI input for the Contractor to facilitate remote and/or in-person meeting components, as appropriate. • A written summary of input from stakeholders in attendance at the meeting. • A final meeting summary. • Integration of input into final strategic plan.

PCORI RFP # PCO-AOSEPP2021 61 Task 4: Engagement Rubric Content Based on the strategic plan and the results of the environmental scan and convening, the Contractor will produce content for the updated Engagement Rubric. This will involve (i) synthesis of evidence and practice-based knowledge on engagement approaches and impact; DEI; and other relevant topics; (ii) translation of evidence into practical guidance; (iii) identification of real-world examples; (iv) development of content to potentially be used in a number of media (e.g., web, print, video, audio, or other formats); and (v) collaboration with a contractor, as necessary, that will design and format the ultimate Engagement Rubric product under a separate Task Order.

Deliverables:

• One (1) or two (2) versions of draft content and a final version in response to comments from PCORI and the PEAP on the draft content. • Draft slides and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with the PEAP to solicit feedback on the draft content. • A presentation to PCORI staff. • A presentation to PCORI’s PEAP. • A written summary of input from key PCORI staff and a subcommittee of the PEAP on the draft content. • A draft slide deck and a final version in response to PCORI input that the Contractor will use to facilitate a remote meeting with key PCORI staff and the EDIC to solicit feedback on the strategic plan. • A written summary of input from key PCORI staff and a subcommittee of the EDIC on the strategic plan. • Final Engagement Rubric content with all input integrated.

Task 5: Create Presentations and Publications Regarding Rubric Development PCORI is interested in contributing to and helping shape national and international discussions on the role and influence of engagement in clinical and care delivery research; the update of the Engagement Rubric will be key to informing the discussions. The Contractor will work with PCORI staff to (i) create products that inform the engagement field, and (ii) facilitate communication of the Engagement Rubric update process with lay and scientific audiences.

As such, PCORI seeks to communicate the process of updating the Engagement Rubric more broadly, including through academic publications. In support of this, the Contractor will co-develop with PCORI at least two (2), and up to four (4), manuscripts and/or reports. Topics for these manuscripts or reports will be subject to approval by PCORI and may include, but are not limited to, (i) describing the process to update the Engagement Rubric, including outlining goals and principles; (ii) summarizing the overall engagement findings included in the Engagement Rubric; (iii) articulating priorities for various stakeholders (e.g., funders, research organizations); and (iv) making recommendations to the field for future activities. The Contractor will submit a proposed publication and dissemination plan for approval by PCORI prior to submitting any manuscript or report for any publication to ensure, among other things,

PCORI RFP # PCO-AOSEPP2021 62 that the proposed publication and dissemination plan will not preclude use and dissemination by PCORI and the public. The Contractor will produce short lay summaries of the manuscripts and/or reports or other recommended communication vehicles for public dissemination, as well as PowerPoint presentations for PCORI staff use.

Deliverables/Delivery Schedule

All deliverables must be completed and include, but are not limited to, the following:

• Kickoff meeting, agenda, minutes, and action items • Project plan • Monthly project and financial status reports (for all work performed under the agreement by Task Order) • Environmental scan of engagement policies, requirements, and guidance • Strategic plan for an updated Engagement Rubric • Convening meeting • Engagement Rubric content • Publications and presentations regarding Engagement Rubric development

Each deliverable shall be provided to PCORI.

Deliverable Description Quantity and Date Completed Media 1 Kickoff meeting Electronic file Within 2 weeks of the Task Order Effective Date 2 Revised project plan Electronic file Within 4 weeks of Task Order Effective Date 2a PCORI review (as applicable) – 10 business days from submission of the project plan to PCORI 2b Resubmit project plan with PCORI Electronic file 5 business days from comments addressed receipt of PCORI comments 3 Calls with PCORI – Biweekly (i.e., every two weeks), beginning after kickoff meeting 4 Monthly project and financial status – Monthly, beginning reports after kickoff meeting 5 Task 1: Environmental Scan Approach Electronic file Within 2 months of Task Order Effective Date 5a PCORI review (as applicable) – 10 business days from submission to PCORI

PCORI RFP # PCO-AOSEPP2021 63 5b Resubmit final Environmental Scan Electronic file 5 business days from Approach with PCORI comments receipt of PCORI addressed comments 5c PCORI review and approval (as – Within 5 business days applicable) of submission to PCORI 5d Task 1: Final Environmental Scan report Electronic file Within 4 months of Task Order Effective Date 5e PCORI review (as applicable) – Within 10 business days of submission to PCORI 6 Task 2: Strategic Plan for Engagement Electronic file Within 6 months of Rubric Task Order Effective Date 6a PCORI review (as applicable) – 10 business days from submission to PCORI 6b Resubmit strategic plan with PCORI Electronic file 5 business days from comments addressed receipt of PCORI comments 7 Task 3: Convening Meeting Agenda Electronic file Within 6 months of Task Order Effective Date 7a PCORI review (as applicable) – 10 business days from submission to PCORI 7b Resubmit convening meeting agenda with Electronic file 5 business days from PCORI comments addressed receipt of PCORI comments 7c Submit draft attendee list Electronic file Within 6 months of Task Order Effective Date 7d PCORI review (as applicable) – 10 business days from submission to PCORI 7e Resubmit attendee list with PCORI Electronic file 5 business days from comments addressed receipt of PCORI comments 7f Convening meeting – Within 8 months of Task Order Effective Date 7g Draft meeting summary Electronic file Within 9 months of Task Order Effective Date 7h PCORI review (as applicable) – 10 business days from submission to PCORI 7i Resubmit meeting summary with PCORI Electronic file 5 business days from comments addressed receipt of PCORI comments

PCORI RFP # PCO-AOSEPP2021 64 8 Task 4: Engagement Rubric Content Electronic file Within 10 months of Task Order Effective Date 8a PCORI review (as applicable) – 20 business days from submission to PCORI 8b Resubmit Engagement Rubric content Electronic file 10 business days from with PCORI comments addressed for receipt of PCORI approval comments 8c PCORI review and approval – Within 10 business days from receipt 8d Submit final Engagement Rubric content Electronic file Within 12 months of Task Order Effective Date 9 Task 5: Manuscripts, lay summaries, Electronic files Within 15 months of and/or PowerPoint presentation(s) Task Order Effective summarizing the Engagement Rubric, as Date determined 9a PCORI review (as applicable) – 20 business days from submission to PCORI 9b Resubmit with PCORI comments Electronic file 10 business days from addressed for approval receipt of PCORI comments 9c PCORI review and approval Electronic file Within 10 business days from receipt 9d Submit final manuscripts, lay summaries, Electronic files Within 18 months of and/or PowerPoint presentation(s) Task Order Effective Date

Period of Performance

The period of performance for Task Order #3, Update of the PCORI Engagement Rubric, will be 18 months from the Task Order Effective Date.

Type of Task Order

PCORI expects to award one (1) cost reimbursable Task Order. This agreement type provides an estimate of the work required to be performed over the agreement period. Contractors must be able to perform all activities described in the Task Order. PCORI encourages the use of teaming arrangements to ensure that project teams have the necessary expertise to accomplish the work.

Task Order Evaluation Criteria

PCORI RFP # PCO-AOSEPP2021 65 Category Weight of Rating Factor

Technical Approach 35%

Project Management Plan 15%

Organization and Staffing 20%

Past Performance 30%

Volume 1: Technical Solution – Page Limit: 35

Volume 2: Price Proposal – Page Limit: 4

The written price proposal shall contain the following:

PCORI RFP # PCO-AOSEPP2021 66 Task Order #3 Attachment A: Overview of PCORI Engagement Evidence Sources and Engagement Resources

Sources for PCORI’s evidence on engagement include:

• PCORI evidence catalog and synthesis effort • Database of engagement examples from PCORI portfolio • Updated Engagement Plan data from PCORI projects • Literature review on engagement of underrepresented communities (forthcoming) • Synthesis of Health Literature Explorer (forthcoming) • Engagement measures landscape review and gap analysis (forthcoming) • Findings and recommendations on clinician engagement in PCOR • Engagement Awards and Pipeline to Proposals portfolio analyses

Products, guidance, and resources to inform engagement practice:

• Research Fundamentals learning package • Building Effective Multi-Stakeholder Research Teams learning package (forthcoming) • Engagement Tool and Resource Repository • Engagement in Health Research Literature Explorer • Engagement in Data Analysis Guidebook • PCORI Compensation Framework • PCORI Engagement Plan • PCORI Engagement Rubric • PCORI in Practice Webinars

PCORI RFP # PCO-AOSEPP2021 67 Task Order Request #4: Studying the Factors and Contexts for Engaging Diverse Research Partners

Background

PCORI is committed to stakeholder engagement in health research. By “engagement in health research,” we refer to the meaningful involvement of patients, caregivers, clinicians, and other healthcare stakeholders throughout the research process—from topic selection and study design through the conduct of the research and dissemination and uptake of results. We know that engaging diverse patients and other stakeholders throughout the research process affects the acceptability, feasibility, quality, and relevance of patient-centered outcomes research, which has the potential of producing findings that are more trustworthy and more readily adopted into practice.4

PCORI’s ability to address health inequities through research is predicated on the inclusion of diverse research partners throughout the research process. However, we know there is variability in the level or intensity at which research partners with diverse backgrounds and perspectives are engaged in our research. Challenges in reaching and engaging diverse research partners limit the inclusion of critical perspectives needed to ensure that PCORI studies address issues with the greatest potential impact on underrepresented and underserved populations. Moreover, omission of these perspectives may preclude the use of strategies that would enhance recruitment and retention of historically understudied populations.

PCORI is seeking a Contractor to perform a set of tasks to expand our knowledge of the factors that influence the interest and ability of members of underrepresented racial and ethnic groups to work in collaboration with research teams. Although many different groups are underrepresented and underserved in health research as partners and collaborators, this project specifically focuses on individuals from underserved racial and ethnic groups because of the disproportionate effect of the COVID-19 pandemic on these groups and recognition of the role that systemic racism has in determining COVID-19-related health outcomes.

Research teams have cited difficulty in identifying diverse research partners—particularly those who are Black, Indigenous, or people of color (BIPOC)—and teams may lack skills in navigating cultural differences. Additionally, group-level factors (e.g., cultural values, knowledge, history) and system-level factors (e.g.,

4 Sheridan S, Schrandt S, Forsythe L; Advisory Panel on Patient Engagement: Hilliard TS, Paez KA. The PCORI Engagement Rubric: promising practices for partnering in research. Ann Fam Med. 2017;15(2):165-170.

PCORI RFP # PCO-AOSEPP2021 68 economic inequities) exist that encourage or discourage individuals from underrepresented racial and ethnic groups from entering into research partnerships.

While the focus of this work is on underrepresented racial and ethnic groups, this does not diminish PCORI’s interest in addressing issues that hinder the involvement of other underrepresented groups in research partnerships, for example, individuals living in rural communities. We hope this study will serve as a model for research about other underrepresented communities. The scarcity of engagement of underrepresented racial and ethnic groups as research partners is a widespread issue, not unique to PCORI; it is cited widely in the literature on community service and volunteerism.5

Thus, the primary purpose of this Task Order is to inform effective engagement of underrepresented racial and ethnic research partners in health research. This work will help PCORI fill a gap in our knowledge base and allow us to develop actionable strategies and products that could better support the access and involvement of individuals from underrepresented racial and ethnic groups in research partnerships.

This study will include a literature scan of publications and materials that address the barriers, facilitators, and methods for engagement of diverse racial and ethnic groups as research partners, including effective tools, resources, and approaches. Additionally, primary data will be collected to understand the lived experiences of underrepresented racial and ethnic research partners in projects funded by PCORI and other funders.

The urgency of our research questions necessitates that this important work be completed in a timely manner to ensure rapid implementation. PCORI emphasizes engagement and transparency with the public throughout the course of this project. The Contractor is expected to be committed to and intentional about engaging a Technical Expert Panel (TEP) to gain feedback, working with PCORI and the PEAP,6 and disseminating findings as they emerge (e.g., in PCORI blog posts, peer-reviewed manuscripts, convenings, reports, and other products). PCORI staff, advisory panels (e.g., PEAP), and committees (e.g., PCORI’s Methodology Committee) may use the knowledge generated by this work to inform future programs, guidance, standards, or project requirements.

The Contractor will operate at the direction of and receive guidance from PCORI. PCORI encourages Offerors to demonstrate an understanding of the issues mentioned above in their proposals.

Project Requirements

The objective of this Task Order is to identify what has encouraged or discouraged involvement in research partnerships among individuals from underrepresented racial and ethnic groups, through a scan of

5 Johnson KJ, Lee SH. Factors associated with volunteering among racial/ethnic groups: findings from the California Health Interview Survey. Res Aging. 2017;39(5):575-596. doi: 10.1177/0164027515618243. 6 PCORI’s Advisory Panel on Patient Engagement (PEAP) is instrumental in helping PCORI ensure the highest patient engagement standards and a culture of patient-centeredness in all aspects of our work.

PCORI RFP # PCO-AOSEPP2021 69 relevant literature and a research study that investigates the quality of research partnership experiences and perspectives that may influence engagement decisions. Research Questions:

1. What are the individual-, organizational-, or system-level contexts and factors that influence the involvement (or non-involvement) of individuals from underrepresented racial and ethnic groups in research partnerships? 2. What are the experiences, levels of involvement, perceived inclusiveness, and perceived level of inclusion of individuals from underrepresented racial and ethnic groups in research partnerships? 3. What tools, resources, approaches, and specific practices effectively build diverse, equitable, and inclusive (patient-centered outcomes) research partnerships? How well do they align with the needs identified and what are the gaps in known practices or available tools, resources, or approaches? Key activities to be completed by the Contractor under this Task Order include:

TASK 1. PROJECT MANAGEMENT Task 1.1. Conduct project kickoff call Task 1.2. Progress meetings and reports throughout the project TASK 2. PROJECT DESIGN Task 2.1. Establish and engage a Technical Expert Panel (TEP) Task 2.2. Design project plan for literature scan and research TASK 3. LITERATURE SCAN Task 3.1. Develop a protocol for conducting the literature scan Task 3.2. Conduct literature scan TASK 4. RESEARCH Task 4.1. Develop a protocol for conducting the research Task 4.2. Conduct the research TASK 5. COMMUNICATION AND DISSEMINATION Task 5.1. Create a plan for communication and dissemination Task 5.2. Share project progress and findings with PCORI, the PEAP, and other designated entities throughout the project Task 5.3. Develop a final report of the analytic methods and synthesis of findings on how to support meaningful engagement of individuals from underrepresented racial and ethnic groups PCORI expects that Tasks 1 through 5 will be completed within fifteen (15) months. Offerors are encouraged to propose shorter solutions for Tasks 1 through 5. Manuscripts will be initiated within the first fifteen (15) months of the contract, but do not have to be completed and published in that time frame.

Specific Requirements/Tasks

The Contractor must perform the following tasks:

PCORI RFP # PCO-AOSEPP2021 70 Task 1: Project Management Task 1.1: Conduct project kickoff call

The Contractor shall work with PCORI to schedule and coordinate the kickoff meeting via a video conference call. The meeting will provide an introduction between the Contractor personnel and PCORI personnel who will be involved with the agreement. PCORI also will discuss the roles and responsibilities of the Project Leads during the kickoff meeting. The meeting will provide the opportunity to discuss technical, management, and reporting procedures. At a minimum, the attendees shall include key Contractor personnel and PCORI’s Procurement Department and Project Leads. The Contractor shall provide the following at the kickoff meeting:

Introduction of personnel:

• Points of contact • Project management plan, including: o Prioritization of Contractor activities o Quality surveillance o Roles and responsibilities o Overview of project tasks o Schedule • Reporting requirements • Any initial deliverables • Additional issues of concern Deliverables:

• Kickoff meeting materials—shared at least one (1) business day prior to the meeting o Provide a meeting agenda outlining the objectives and discussion points for the meeting. o Include electronic copies of the pertinent materials for PCORI staff review (e.g., slides, documents). • Kickoff meeting minutes with action items—sent to Project Leads within three (3) business days o Include a list of the participants of the meeting, a summary of the meeting, and main discussion points. o Document changes or adjustments to the project. o Include vendor and PCORI action items (with expected due dates).

Task 1.2: Progress meetings and reports throughout the project

The Contractor shall work with PCORI to schedule biweekly project status calls to share progress to date on tasks/sub-tasks; submitted/upcoming deliverables; questions and/or possible adjustments to discuss with PCORI as they arise; discussion of draft submissions and clarification of feedback provided; and any points of interest from the preceding two weeks. The agenda and supplemental materials for each call will be provided to PCORI at least one (1) business day prior to the meeting.

The Contractor will submit monthly status reports to include:

PCORI RFP # PCO-AOSEPP2021 71 • Activities during reporting period, by task, to include: o Ongoing activities, new activities, and activities completed. o Deliverables submitted for that period. o Progress to date on all above-mentioned activities. o A brief description of the task at the start of each section. • Problems and corrective actions taken. Also include issues or concerns and proposed resolutions to address them, to include risk mitigation plans. • Personnel gains, losses, and status. • PCORI actions required. • Schedule showing major tasks, milestones, and deliverables; include planned and actual start and completion dates for each. • Financial status including: o Cumulative invoiced costs total to date including hours and cost by labor categories (including planned vs. actual rates). o Hours billed by individual (including planned vs. actual hours), cost to completion, and variance. o Subcontractor labor and other subcontractor costs (including planned vs. actual hours).

Deliverables:

• Biweekly status calls (for each meeting): o Provide a meeting agenda—shared at least one (1) business day prior to the meeting. o Include electronic copies of the pertinent materials for PCORI staff review (e.g., slides, documents). o Provide meeting minutes and other pertinent follow-up materials—sent to Project Leads within three (3) business days. • Monthly status report submitted to the Project Leads by the 10th of each month

Task 2: Project Design Task 2.1: Establish and engage a Technical Expert Panel (TEP)

The Contractor will establish a multi-stakeholder TEP to provide input and feedback throughout the project. TEP members may include members of PCORI Advisory Panels and should include individuals not otherwise connected to PCORI. Offerors should develop an Engagement Plan for working with the TEP throughout the conduct of the project and for incorporating input from the PEAP (see Task 5). In their proposals, Offerors should recommend the size and stakeholder makeup of the expert panel and the payment structure. PCORI will review and approve the list of suggested invitees before formal invitations are extended to join the group. PCORI may make suggestions of invitees.

Deliverables:

PCORI RFP # PCO-AOSEPP2021 72 • Draft and final list of TEP members (including professional affiliation, title, and contact information) and the role that each member will play throughout the project, submitted for review by PCORI staff. • Draft and final Engagement Plan for when and how to engage the TEP, including how the plan will be implemented.

Task 2.2: Design project plan for literature scan and research

The Contractor will design and submit a project plan for both the literature scan and research components of the project. The Contractor will recommend whether the literature scan and research should be done with some overlap or simultaneously.

The project plan should address PCORI’s goal of understanding how to improve the DEI of individuals from underrepresented racial and ethnic groups as research partners in patient-centered outcomes research. The plan should address the following research questions:

Research Questions:

1. What are the individual, organizational-, or system-level contexts and factors that influence the involvement (or non-involvement) of individuals from underrepresented racial and ethnic groups in research partnerships? 2. What are the experiences, levels of involvement, perceived inclusiveness, and perceived level of inclusion of individuals from underrepresented racial and ethnic groups in research partnerships? 3. What tools, resources, approaches, and specific practices effectively build diverse, equitable, and inclusive (patient-centered outcomes) research partnerships? How well do they align with the needs identified and what are the gaps in known practices or available tools, resources, or approaches? The final analysis should synthesize across both components to answer the questions above and distinguish between factors that are important across and within underrepresented groups broadly, factors that particularly affect individuals from underrepresented racial and ethnic groups, and those that are specific to certain underrepresented racial and ethnic groups.

This study is not intended to be confined to the experiences of individuals who have partnered with researchers on patient-centered outcomes research but may include individuals who have been (or wanted to be) involved in other types of research that employs engagement methods.

Mixed methods approaches to answering these questions are anticipated.

Deliverables:

• Draft of a comprehensive project plan. • Final comprehensive research plan addressing comments from PCORI on the draft plan.

Task 3: Literature Scan Task 3.1: Develop a protocol for conducting the literature scan

PCORI RFP # PCO-AOSEPP2021 73 The Contractor will develop the detailed plan to guide the literature scan and synthesis. The plan should include a detailed description of the methods to be used for searching, screening, and synthesizing the literature. PCORI will review the draft plan and provide comments.

The literature scan should include gray and peer-reviewed literature that discusses the challenges and successes that research teams and underrepresented racial and ethnic research partners experience in engaging in diverse, equitable, and inclusive partnerships. The literature scan should also explore connections between the impact of diverse research partnerships on the recruitment and retention of diverse research participants and other indicators of relevant or high-impact research. Literature findings are not limited to any type of method and should include literature on both individual and organizational factors and strategies. Studies may include those that utilize historical, sociological, social anthropological, and ethnographic methods, and the search should learn from fields related to engagement in health research (e.g., community-based participatory research, engagement in health care, etc.). The scan must also include tools, resources, and approaches used to build diverse, equitable, and inclusive research partnerships. The Contractor should provide a general assessment of the overall quality and effectiveness of these tools relative to the identified needs, although a specific assessment of each tool is beyond the scope of this project.

The Contractor will also develop an analysis plan to interpret and synthesize available findings. The Contractor will revise the plan in response to comments from PCORI and submit a final plan.

Deliverables:

• Draft of the comprehensive literature scan research plan. • Final version of the comprehensive literature scan research plan addressing comments from PCORI on the draft plan.

Task 3.2: Conduct literature scan

The Contractor will scan relevant literature to answer the research questions noted above. The Contractor will synthesize the findings for internal and external use. The Contractor will provide a draft table of findings and/or a memo of initial analytic learnings, which will be shared with PCORI staff.

The Contractor will conduct a review meeting with the Project Leads and other PCORI staff to discuss preliminary findings. PCORI will work with the Contractor to determine who should be included in the meeting and when the meeting should be scheduled.

Deliverables:

• Preliminary tables of findings and/or memo of initial analytic learnings to discuss with PCORI. • Meeting with PCORI staff to discuss preliminary findings. • Meeting minutes within three (3) business days after the meeting. • Annotated EndNote library, shared with the Project Lead, containing all the pertinent literature included in the scan.

PCORI RFP # PCO-AOSEPP2021 74 Task 4: Research Task 4.1: Develop a protocol for conducting the research

PCORI seeks a deep understanding of the experiences of individuals from historically underrepresented perspectives who have participated as partners on research projects or have wished to but found they were unable to do so. In other words, we are interested in direct experience in participating on research teams as well as the experiences of those who may have encountered substantial barriers that do not allow them to be engaged in research partnerships. The experiences of partnering in research can be in PCORI-funded work, as well as in work with other funders or in other types of experiences, which may also provide important insights. The Contractor will develop the research plan, which should include qualitative or mixed methods to guide this portion of the project. The research plan should include a detailed description of the methods to be used, the approach that will be used to identify individuals who can provide insight into their experiences (i.e., sampling), what types of data will be collected, and how those data will be analyzed.

The sampling strategy should include:

• Both people who have been partners on PCORI-funded projects and people who have been partners on projects not funded by PCORI • Individuals from underrepresented racial and ethnic groups with a variety of backgrounds and experiences • Methods that do not rely on PCORI to provide specific lists of names, although PCORI will collaborate with the Contractor to develop a plan for whom to include in the study and how to reach them, including considering options for outreach (e.g., PCORI website, newsletters) and for communication with affiliated groups (e.g., Ambassador program7) • Methods that do not rely on Principal Investigators of research projects to nominate or select research partners as potential participants

PCORI will review the draft protocol and provide comments. The Contractor will revise the protocol in response to comments from PCORI, the TEP, and/or advisory panels and will submit a final research protocol.

While the Offeror must propose the ideal mix of methods for this project, the following examples of approaches may be helpful in understanding the potential scope of this work:

• Brief survey(s) of organizations, projects, or individuals • In-depth interviews or focus groups with people leading the work, community groups, and participants • Participant observation

7 PCORI’s Ambassador Program includes individuals from the stakeholder community interested in advancing patient-centered outcomes research (PCOR).

PCORI RFP # PCO-AOSEPP2021 75 • Case studies of specific approaches, contexts, or settings • Analysis of secondary qualitative data from organization or project websites

Deliverables:

• Draft and final of comprehensive research protocol. • Draft and final interview guide(s), survey(s), or other data collection instrument(s) as relevant to the proposed solution. • Final version of comprehensive research protocol addressing comments from PCORI on the draft research plan.

Task 4.2: Conduct the research

The Contractor will obtain ethical approval from the appropriate Institutional Review Board and then conduct the primary data collection, analyze the data, and interpret the findings from the research study. The Contractor will synthesize the findings for internal and external use.

Deliverables:

• Regular updates about the study process (e.g., recruitment, data collection) during biweekly project status calls while the study is being conducted. • As applicable based on the methods used for the study: o Descriptive statistics about the sample, draft results tables, and an analytic memo to share preliminary findings for each research question. o Interim qualitative analysis, coding structure, memos, and emergent themes as appropriate. • Meetings with PCORI staff to discuss preliminary findings. • Meeting minutes provided within three (3) business days after the meeting.

Task 5: Communication and Dissemination Task 5.1: Create a plan for communication and dissemination

The Contractor will develop a plan for bidirectional communication throughout the course of this project. The document should include plans to communicate the project and research plans to, and solicit input from, (i) PCORI, (ii) the PEAP, and (iii) other entities as designated by PCORI. Offerors may suggest other mechanisms or groups to engage throughout the course of the project to promote collaboration and incorporation of diverse perspectives to guide this work. See Task 5.2 for guidance on items to include in the plan.

Deliverables:

• Draft of a comprehensive communication plan. • Final comprehensive communication plan addressing comments from PCORI on the draft plan.

PCORI RFP # PCO-AOSEPP2021 76 Task 5.2: Share project progress and findings with PCORI, the PEAP, and other designated entities throughout the project

The Contractor will communicate project progress and findings throughout the course of this project to (i) PCORI, (ii) the PEAP, and (iii) other designated entities, as described in the finalized communication plan.

For PCORI staff: In addition to regular communication and work with the PCORI project team, the Contractor will give two (2) or three (3) presentations regarding the project plans and findings to PCORI staff. PCORI staff will work with the Contractor to identify the appropriate venues, frequency, meeting time(s), and audience for the meetings, and PCORI will facilitate the meetings.

For PCORI’s PEAP: The Contractor will discuss with PCORI’s PEAP (or a PEAP subcommittee, e.g., DEI Subcommittee, Patient-Centered Approaches Subcommittee) their (i) research plan, (ii) findings and implications, and (iii) final report. The PEAP or a PEAP subcommittee should be allowed to provide the Contractor with targeted feedback to inform the research design and react to early findings. At minimum, the PEAP or a PEAP subcommittee should be engaged for feedback on the project plan, interpretation of preliminary findings, and final report. PCORI staff will work with the Contractor to identify the appropriate PEAP meeting at which the presentation will occur and will facilitate the meeting. The Contractor will develop and submit a draft PowerPoint slide deck for the presentations. PCORI will review the draft slide deck and provide comments, and the Contractor will revise the slide deck in response to input from PCORI and submit the final slide deck.

For other designated entities: PCORI will work with the Contractor to identify other organizations and/or groups who would be interested in this work. The Contractors will co-create products with PCORI for public dissemination throughout the project to share progress and get feedback at select time points. The following are projected deliverables at approximate milestones of the project:

• Blog on the project plans, • Manuscript and blog about literature scan results, and/or • Manuscript and blog about research study results.

Deliverables:

• Present the (i) project plan and goals, (ii) preliminary findings and implications, and (iii) final report to PCORI, the PEAP, and/or other designated entities. o Outline of presentations for each meeting. o At least one (1) comprehensive draft of the slide deck for each meeting. o Final comprehensive slide deck for each meeting addressing comments provided by PCORI on the draft slide deck. o Presentation of the comprehensive slide deck to each of the identified groups. o Meeting minutes provided within three (3) business days after each convening. • Co-produce PCORI blog post(s) about the (i) project plan and goals, and (ii) findings and implications.

PCORI RFP # PCO-AOSEPP2021 77 o Outline of blog post(s). o Draft of blog post(s). o Final version of blog post(s). • Co-produce a presentation for PCORI staff about the findings and implications. o Draft of presentation for PCORI staff. o Final version of presentation for PCORI staff. • Co-produce at least one (1) manuscript of publishable quality and submit for peer review. o Outline of manuscript(s). o Drafts of manuscript(s). o Final version of manuscript(s).

Task 5.3: Develop a final report of the analytic methods and synthesis of findings on how to support meaningful engagement of individuals from underrepresented racial and ethnic groups

The Contractor will develop a final report of the project, with a goal of synthesizing learnings across tasks. The report should include an executive summary, and the body of the report should include a summary of the literature scan and research study methods and findings, and the implications of the findings, developed in collaboration with the TEP, that may inform new practices, programs, and/or policies to facilitate engagement of individuals from underrepresented racial and ethnic groups in meaningful research partnerships.

Findings and implications should be reviewed and discussed with PCORI staff and the PEAP. PCORI may use the knowledge generated by this work to inform future programs, guidance, standards, or project requirements.

PCORI will review the draft report and provide comments. The Contractor will revise the report in response to input provided by PCORI and submit a final report to be used by PCORI and the TEP to develop actionable knowledge.

Deliverables:

• Final report: o Outline of final report. o At least one (1) draft of the final report. o Final version of the final report.

Demonstrate Commitment to Diversity, Equity, and Inclusion

PCORI is committed to DEI as an important part of its mission and deems these principles central to the programs, projects, and activities it considers and supports. PCORI seeks to improve its efforts and offerings aimed at increasing the participation and contributions of individuals from groups that have been historically underrepresented and underserved, individuals with disabilities, and individuals from economically disadvantaged backgrounds. Offerors should not only be able to articulate their commitment to DEI principles and how they would successfully perform under a resultant agreement with

PCORI RFP # PCO-AOSEPP2021 78 PCORI, but also provide specific examples of how this commitment has been demonstrated in similar work performed in the past, including relevant outcomes. Examples may include, but are not limited to, improved stakeholder engagement, enhanced employee satisfaction, policy changes, or other results.

Deliverables/Delivery Schedule

For general guidance:

• All agendas and supplemental materials for meetings with the PCORI Project Team should be provided at least one (1) business day ahead of the meeting. • All agendas and supplemental materials for meetings with the PCORI staff (beyond the Program Team) should be provided at least five (5) business days ahead of the meeting. • All agendas and supplemental materials for meetings with the PCORI PEAP or other external groups should be provided at least seven (7) business days ahead of the meeting. • All meeting minutes for any meeting conducted should be provided no later than three (3) business days following the meeting. • Monthly status reports are always due on the 10th day of each month. Should the 10th fall on a weekend or holiday, the monthly status report is due on the closest business day before or after the 10th. • PCORI will review and provide feedback on one (1) or two (2) drafts for all written deliverables and presentations, based on need and quality of drafts. • The Contractor is expected to return revisions or final versions of deliverables within one (1) week after receiving feedback from PCORI. • PCORI will return feedback on deliverables about seven (7) business days after receiving drafts from the Contractor. • The Contractor will co-produce at least one (1) manuscript based on the literature scan and/or empirical research findings.

All deliverables must be completed, and they include but are not limited to the following, with each deliverable to be provided to PCORI. The following schedule does not include all drafts but estimates the approximate final due date for each deliverable, which accounts for at least one (1) draft and time for PCORI to provide feedback. We invite Offerors to suggest appropriate adjustments while maintaining the overall timelines of the project.

Task Description Quantity/Media Final Due Date Agenda and materials for 1 MS Word document 1 business day prior to any All meeting with the PCORI Project submitted electronically meeting Team AND/OR

PCORI RFP # PCO-AOSEPP2021 79 Electronic files attached to the meeting invitation 1 MS Word document submitted electronically Agenda and materials for 5 business days prior to All AND/OR meeting with the PCORI Staff kickoff meeting Electronic files attached to the meeting invitation 1 MS Word document Agenda and materials for submitted electronically 7 business days prior to All meeting with the PCORI PEAP or AND/OR kickoff meeting other external groups Electronic files attached to the meeting invitation 1 MS Word document 3 business days after any All Meeting minutes submitted electronically meeting 10th day of each month Task 1 MS Word document Monthly status reports during the performance 1.2 submitted electronically period Task Draft and final list of Technical 1 MS Word document 3 weeks after the kickoff 2.1 Expert Panel submitted electronically meeting Task Draft and final engagement plan 1 MS Word document 3 weeks after the kickoff 2.1 for Technical Expert Panel submitted electronically meeting Task 1 MS Word document 5 weeks after the kickoff Draft and final project plan 2.2 submitted electronically meeting Task Draft and final communication 1 MS Word document 2 weeks after project plan is 5.1 and dissemination plan submitted electronically approved Task Outline, draft, and final blog 1 MS Word document 3 weeks after project plan is 5.2 post of project plan submitted electronically approved Outline, draft(s), and final slide 1 MS PowerPoint slide Task 4 weeks after project plan is deck of project plan for deck submitted 5.2 approved presentation to PEAP electronically 1 MS PowerPoint slide Task Presentation of project plan to deck submitted TBD 5.2 PEAP electronically Task Draft and final literature scan 1 MS Word document 4 weeks after PEAP meeting 3.1 protocol submitted electronically Task Preliminary tables and findings 12 weeks after final literature 1 MS Word document OR 3.2 from literature scan scan protocol is approved

PCORI RFP # PCO-AOSEPP2021 80 1 MS PowerPoint slide deck submitted electronically Task 12 weeks after final literature Annotated EndNote library 1 EndNote library 3.2 scan protocol is approved 1 MS PowerPoint slide Task Presentation of preliminary At next closest biweekly deck submitted 3.2 findings to PCORI Project Team status call meeting electronically Task 1 MS Word document 10 weeks after final literature Draft and final research protocol 4.1 submitted electronically scan protocol is approved 1 MS Word document OR Task Preliminary findings from 1 MS PowerPoint slide 12 weeks after final research 4.2 research deck submitted protocol is approved electronically 1 MS PowerPoint slide Task Presentation of preliminary At next closest biweekly deck submitted 4.2 findings to PCORI Project Team status call meeting electronically Task Outline, draft(s), and final 1 MS Word document TBD—on literature scan 5.2 version of at least one (1) submitted electronically and/or research findings manuscript 4 to 5 weeks after presenting Outline, draft(s), and final slide 1 MS PowerPoint slide Task preliminary findings from deck for presentation of findings deck submitted 5.2 research to PCORI Project and implications to PEAP electronically Team 1 MS PowerPoint slide Task Presentation of findings and deck submitted TBD 5.2 implications to PEAP electronically Draft(s) and final slide deck of 1 MS PowerPoint slide 2 to 3 weeks after presenting Task findings and implications for deck submitted findings and implications to 5.2 presentation to PCORI staff electronically PEAP 1 MS PowerPoint slide Task Presentation of findings and deck submitted TBD 5.2 implications to PCORI staff electronically Task Outline, draft, and final blog 1 MS Word document 3 weeks after presentation to 5.2 post of findings and implications submitted electronically PCORI staff Outline, draft(s), and final Task 1 MS Word document 5 weeks after presentation of version of final report to PCORI 5.3 submitted electronically findings to PCORI staff staff

PCORI RFP # PCO-AOSEPP2021 81 Period of Performance

PCORI expects that Tasks 1 through 5 will be completed within fifteen (15) months. Offerors are encouraged to propose shorter solutions for Tasks 1 through 5. Manuscripts will be initiated within the first fifteen (15) months of the contract, but do not have to be completed and published in that time frame.

Type of Task Order

Task Order Evaluation Criteria

Category Weight of Rating Factor

Technical Approach 35%

Project Management Plan 15%

Organization and Staffing 15%

Past Performance 35%

Volume 1: Technical Solution – Page Limit: 28

Volume 2: Price Proposal – Page Limit: 4

The written price proposal shall contain the following:

PCORI RFP # PCO-AOSEPP2021 82 Task Order Request #5: Resource Center to Support PCORI Activities Related to the Capture of Patient-Centered Cost and Economic Impact Data

Competition for this Task Order is limited to those Offerors qualifying under the “Economic Resource Services” domain of the PCORI IDIQ agreement for Analytic and Operational Support for the Engagement of the Public and Patients.

Background

Congress created PCORI in 2010 as a nonprofit, nongovernmental organization and reauthorized PCORI in December 2019 for an additional ten years. As part of that reauthorization, PCORI is directed to consider the “full range of outcomes data” in research it funds.

Consideration of full range of outcomes data.

Research shall be designed, as appropriate, to take into account and capture the full range of clinical and patient-centered outcomes relevant to, and that meet the needs of, patients, clinicians, purchasers, and policy-makers in making informed health decisions. In addition to the relative health outcomes and clinical effectiveness, clinical and patient-centered outcomes shall include the potential burdens and economic impacts of the utilization of medical treatments, items, and services on different stakeholders and decision-makers respectively. These potential burdens and economic impacts include medical out-of-pocket costs, including health plan benefit and formulary design, non-medical costs to the patient and family, including caregiving, effects on future costs of care, workplace productivity and absenteeism, and healthcare utilization.

As part of PCORI’s implementation of this authority, it seeks to establish a Resource Center to support PCORI activities, both internal- and external-facing, related to the capture of patient-centered cost and economic impact data as part of its research and engagement projects. Internal-facing activities may include, for example, summarizing the state of practice regarding specific methodological questions pertaining to cost data collection, and outlining alternative responses or mechanisms for responding to methodological questions that arise regarding PCORI’s efforts in this area. External-facing activities may include convening stakeholders under PCORI’s banner to, for example, identify gaps in available information where PCORI’s support of cost data collection or improvements in methods have potential to further the goals of a more patient-centered and efficient health system.

Project Requirements

The Resource Center will undertake activities in response to specific questions and priorities as requested by PCORI staff, including acting as a resource to support related Methodology Committee activities and needs. In addition, through internal expertise and through outreach to appropriate stakeholders, the Resource Center will identify methodologic areas that require elaboration, assuming a “think tank” role to identify areas where advancement may support PCORI’s mission, and, upon direction from PCORI,

PCORI RFP # PCO-AOSEPP2021 83 undertake such work. The Resource Center should also have the capacity to facilitate and support multi- stakeholder meetings and community engagement activities.

PCORI will establish a process for determining and prioritizing Resource Center tasks, which will reflect PCORI’s priorities and its role as a trusted entity across patient and stakeholder communities. Tasks may include:

• Provide technical support to PCORI in considering approaches to cost data collection, sources of costs data, costing methods, or cost analysis to inform PCORI’s work in operationalizing its mandate to collect data on cost and economic impacts of interventions. • Provide subject matter expertise and support for stakeholder or technical expert convenings to identify challenges, explore opportunities, or recommend approaches for addressing questions related to cost data collection or cost analysis. • Based on input from convenings and engagement, publish landscape reviews and white papers on various topics, such as “ensuring that costs and value assessments include relevant patient data” or “determining economic impact and value from patient perspectives” or “considering costs and value related to new innovations from the patient perspective.” • Identify opportunities where further investment—in either engagement or research—can advance or support patient and stakeholder evidentiary needs related to cost and value. PCORI encourages Offerors to demonstrate an understanding of the attributes mentioned above in their proposals.

Specific Requirements/Tasks

Contractor(s) who are selected for this Task Order must be able to do the following:

• Demonstrate capacity to conduct literature reviews and develop reports, briefs, and white papers on the state of practice or best practices regarding specific methodological questions in cost data collection and cost analysis, drawing on expertise in economics, cost analysis, and accounting. o For illustrative purposes, such topics may include, but are not limited to, capture of patient out-of-pocket costs or non-monetary patient impacts; productivity costs or enhancements; approaches to costing healthcare interventions; and uses of administrative data.

• Demonstrate capacity to assume a “think tank” role to propose areas where methodological advancement would support PCORI’s goals with respect to capturing economic burdens and, upon PCORI agreement, undertaking such work. (The Offeror should have adequate resources for this task, rather than relying on technical expert input.)

• Demonstrate the ability to gather and synthesize input and advice from diverse stakeholders on cost data collection, measurement, and analysis.

• Demonstrate the ability to provide subject matter support for convenings of stakeholders or technical experts, including the ability to refine questions related to cost data collection and

PCORI RFP # PCO-AOSEPP2021 84 measurement, structure meetings or workgroup sessions that will productively address such questions, and moderate these convenings. o As examples, such convenings might focus on (i) identification of gaps in information available to patients that would be a productive focus of PCORI’s cost data collection, (ii) identification of such gaps in information available to healthcare system decision makers, or (iii) technical input regarding questions related to cost data collection or measurement.

• Demonstrate the ability to develop reports and briefs for professional and lay audiences on methods and promising practices.

• Demonstrate experience developing simulation models projecting health outcomes and costs, ranging from simple low-tech approaches to more complex and intensive models.

• Demonstrate the ability to specify health outcomes to include in a model of regional- and national-level impact, assuming study findings are implemented nationally.

• Demonstrate the ability to specify all relevant measures of healthcare utilization that would meaningfully affect costs for various stakeholder communities.

• Demonstrate familiarity with a range of data sources and the capacity to identify the appropriate data sources that capture the outcomes and impacts described.

• Demonstrate the ability to access and use the identified data to derive estimates required for the modeling activity.

• Demonstrate experience with developing methods for assigning costs to outcomes, including specific datasets/data sources to be used to assign costs for budget modeling.

• Demonstrate experience with synthesizing findings and providing reports that place a strong emphasis on the visual representation of data and conclusions, including charts and graphs.

• Demonstrate access, as necessary or helpful, to other stakeholder experts and expertise, including, but not limited to, payers/insurers, purchasers/employers, and patients.

• Demonstrate AMA Manual of Style proficiency.

• Demonstrate experience and expertise in providing Section 508 compliance services.

• Demonstrate experience with and commitment to DEI.

PCORI RFP # PCO-AOSEPP2021 85 Deliverables/Delivery Schedule

Deliverable Description Due Date (Year 1)

1 Kickoff call with PCORI • Kickoff call within five (5) business days of Task Order Effective Date • Emailed agenda one (1) day prior to the kickoff call • Kickoff call minutes within two (2) business days of the kickoff call 2 Draft work plan Within ten (10) business days of the completion of the kickoff call

3 Kickoff meeting with PCORI Within four (4) weeks of the Task Order Effective Date

4 Final work plan Within six (6) weeks of the Task Order Effective Date

5 Recurring biweekly check-in calls with First call within eight (8) weeks of the Task PCORI and status report Order Effective Date and every following two (2) weeks until end of the Task Order Term

6 Presentation for updates to PCORI’s Within eight (8) weeks of the Task Order Board of Governors Effective Date

7 Interim progress report Due thirteen (13) weeks from Task Order Effective Date

8 Interim progress report Due twenty-six (26) weeks from Task Order Effective Date

9 Presentation for updates to PCORI’s Within twenty-six (26) weeks of Task Order Board of Governors Effective Date

10 Interim progress report Due thirty-nine (39) weeks from Task Order Effective Date

PCORI RFP # PCO-AOSEPP2021 86 11 Interim progress report Due fifty-two (52) weeks from Task Order Effective Date

12 Presentation for updates to PCORI’s Within fifty-two (52) weeks of Task Order Board of Governors Effective Date

13 Provide support for convening #1 TBD

14 White paper #1 TBD

Deliverable Description Due Date (Year 2)

15 Interim progress report Due sixty-five (65) weeks from Task Order Effective Date

16 Interim progress report Due seventy-eight (78) weeks from Task Order Effective Date

17 Presentation for updates to PCORI’s Within seventy-eight (78) weeks of Task Order Board of Governors Effective Date

18 Interim progress report Due ninety-one (91) weeks from Task Order Effective Date

19 Final progress report Due one hundred five (105) weeks from Task Order Effective Date

20 Presentation for updates to PCORI’s Within one hundred five (105) weeks of Task Board of Governors Order Effective Date

21 Provide support for convening #2 TBD

22 White paper #2 TBD

Project Lead Authority

Performance of work under this Task Order must be subject to the technical direction of the Project Lead or a representative designated in writing. The term “technical direction” includes, without limitation, direction to the Contractor that directs or redirects the labor effort, shifts the work between work areas or locations, fills in details, and otherwise serves to ensure that tasks outlined in the work statement are accomplished satisfactorily.

• Technical direction must be within the scope of the specification(s)/work statement. • Technical direction may be oral or in writing.

PCORI RFP # PCO-AOSEPP2021 87 The Project Lead does not have authority to issue technical direction that:

• Constitutes a change of assignment or additional work outside the specification(s)/statement of work; • In any manner causes an increase or decrease in the Task Order’s price, or the time required for Task Order performance; • Changes any of the terms, conditions, or specification(s)/work statement of the Task Order; or • Interferes with the Contractor’s right to perform under the terms and conditions of the Task Order.

Project Management

The Contractor must perform the following tasks:

• Identify points of contact and kickoff call. On the Task Order Effective Date, the Contractor shall provide PCORI staff with “Resource Center” staff contact information for responsible contacts for the center. Within five (5) business days of Task Order Effective Date, the Contractor shall schedule a call with PCORI staff for a 1- to 2-hour call to discuss the goals, objectives, and tasks of the Task Order. The Contractor shall provide minutes within two (2) business days of the call. • Draft work plan. Within ten (10) business days following the kickoff call, the Contractor shall submit a work plan, subject to PCORI review and approval, detailing their approach to managing the scope of work, any changes from the contract proposal along with rationale for changes, and a detailed timeline. • Kickoff meeting. Within four (4) weeks of Task Order Effective Date, the Contractor shall schedule a half-day remote meeting, including the Project Director and key program staff for the Contractor, with PCORI program staff and PCORI Procurement Department staff. The meeting will be for the Contractor to introduce Resource Center project staff and to present a detailed plan of their goals for the center and their approach to implementing the work plan, for PCORI input. • Final work plan. Within six (6) weeks of Task Order Effective Date, the Contractor shall submit a final work plan, updated in response to PCORI input on the draft plan and the kickoff meeting. • Biweekly check-in calls. The Contractor shall schedule and participate in biweekly calls with PCORI staff. The biweekly calls shall facilitate discussions on contract tasks, barriers or problems affecting Task Order implementation, future tasks, and any administrative issues relevant to routine performance duties. The Contractor shall provide a preliminary call agenda one (1) business day in advance of the call and provide summary minutes and action items within two (2) business days after the call. • Interim progress reports. The Contractor shall submit quarterly progress reports detailing project activities and milestones accomplished in the prior quarter. Period of Performance

The period of performance for Task Order #5, Resource Center to Support PCORI Activities Related to the Capture of Patient-Centered Cost and Economic Impact Data, will be two (2) years from the Task Order Effective Date with one-year options to extend by PCORI.

PCORI RFP # PCO-AOSEPP2021 88 Type of Task Order

Task Order Evaluation Criteria

Category Weight of Rating Factor

Technical Approach 40%

Project Management Plan 30%

Past Performance 30%

Volume 1: Technical Solution – Page Limit: 35

• Technical Approach (Suggested Page Limit: 20) • Organization and Staffing (Suggested Page Limit: 5) • Project Management Plan (Suggested Page Limit: 5) • Past Performance (Suggested Page Limit: 5)

Volume 2: Price Proposal – Page Limit: 4

The written price proposal shall contain the following: Offerors will demonstrate how they arrived at their proposed pricing by detailing labor categories, labor rates, and travel costs. Proposed pricing must include estimated direct expenses including all subcontractor labor (if applicable), travel costs, and other direct expenses. Include burden as applicable, specifying burden rates and burden calculations.

PCORI RFP # PCO-AOSEPP2021 89 Appendix A: Proposal Evaluation Criteria

The Basis of Award will be Best Value. An IDIQ agreement and Task Order(s) will be awarded to the Offeror(s) whose quote, conforming to the RFP, represents the best value to PCORI. When determining best value, the Technical Solution and Past Performance, when combined, are significantly more important than cost or price. Proposals that are unrealistic in terms of technical or schedule commitments, or unrealistically high or low in terms of price/cost, may be deemed to be reflective of an inherent lack of technical competence or indicative of a failure to comprehend the complexity and risks of the proposed work and may be grounds for rejection of the proposal. Cost/price is not the most important evaluation factor, but its degree of importance will increase commensurate with the degree of equality among different Offerors’ Technical Solution and Past Performance proposals. PCORI will award multiple IDIQ agreements and Task Order(s) as a result of this RFP. PCORI will award the IDIQ agreement and Task Order(s) resulting from this RFP to the responsible Offeror(s) whose proposal(s), conforming to the RFP, will be most advantageous to PCORI. An evaluation of each Offeror’s price proposal will be made to determine if it is realistic for the work to be performed, reflects a clear understanding of the requirements, and is consistent with various elements of the technical proposal. PCORI will evaluate the reasonableness of the price of each offer in relation to the Offeror’s relative quality. Reasonableness will be determined considering other competitive prices received and a comparison to the Independent Cost Estimate.

Whether a price is too high or too low may be considered in the context of comparison to other price proposals as well as to the Independent Cost Estimate, or to any other information the Procurement Department deems useful in its discretion. The price reasonableness evaluation will result in a determination of a fair and reasonable price to PCORI. An award will be made to the Offeror that proposes the best value, with the technical approach being most important. Proposals will be evaluated using generally the following criteria, with specific weights for each rating factor provided in the respective IDIQ/Task Order sections of this RFP: