Assessment of Prevention, Diagnosis, and Treatment Options (APDTO)

Assessment of Prevention, Diagnosis, and Treatment Options (APDTO) PFA Cycle 1, 2017 Offline Reviewer Critique Template

The APDTO program is seeking applications for comparative clinical effectiveness research designed to provide information that will inform critical decisions that face patients and caregivers, clinicians, policy makers, and healthcare system leaders.

Important Reminders:  Before beginning your first critique, read the APDTO PCORI Funding Announcement. Key aspects of the funding announcement can also be found in the reference table at the end of this document.  Use this offline template to draft a written critique for each criterion you have been assigned to review. The offline reviewer critique template begins on page 3.  Please use the scoring rubric on page 2 to ensure that your comments and numeric scores align. This helps other reviewers and Program staff to better understand your scores and to use that information in preparing for the in-person meeting.  Once you have completed your written critique based on the review criteria, double-check to make sure that you have addressed the key points listed on page 10.  Once you have finished drafting your entire critique, please copy/paste your comments into PCORI Online.

1 Assessment of Prevention, Diagnosis, and Treatment Options (APDTO) PFA Cycle 1, 2017 Offline Reviewer Critique Template

Reference: PCORI Scoring Chart – Criterion Scores

Definitions of strength and weakness modifiers:  Major Strength: An attribute that is likely to lead to improvements in healthcare and/or outcomes  Moderate Strength: An attribute that would probably lead to improvements in healthcare and/or outcomes  Minor Strength: An attribute that could lead to improvements in healthcare and/or outcomes  Minor Weakness: An easily addressable weakness that does not substantially lessen the impact of the study’s results on healthcare and/or outcomes  Moderate Weakness: A weakness that would lessen impact of the study’s results on healthcare and/or outcomes

 Major Weakness: A weakness that would seriously limit impact of the study’s results on healthcare and/or outcomes

Please use this template only as a placeholder for your critiques. Once you have finished your critique, please copy/paste your comments into the PCORI Online Critique Form. (To access the Online Review form for each assigned application, navigate to the Reviewer Dashboard in PCORI Online.)

Keep comments for each criterion to no more than 5000 characters, including spaces. Comments for each criterion should list out strengths and weaknesses using a bulleted format. Comments for the Protection of Human Subjects and Overall Narrative sections should be written in paragraph form.

Request ID:

Criterion 1. Potential for the study to fill critical gaps in evidence (Not required for Patient and Stakeholder Reviewers) The proposal addresses the following questions:  Does the application convincingly describe the clinical burden?  Does the application identify a critical gap in current knowledge as noted in systematic reviews, guideline development efforts, or previous research prioritizations?  Does the application identify a critical gap in current knowledge, evidenced by inconsistency in clinical practice and decision making?  Would research findings from the study have the potential to fill these evidence gaps?

Strengths:   

Weaknesses:   

Criterion 1 Score:

Criterion 2. Potential for the study findings to be adopted into clinical practice and improve delivery of care (All reviewers) The application should describe how evidence generated from this study could be adopted into clinical practice and delivery of care by others. The application should also address the following questions:  Does the application identify who will make the decision (i.e., the decision maker) or use (i.e., the end-user) the study findings (not the intervention) this study produces, such as local and national stakeholders?  Does the application identify potential end-users of study findings—such as local and national stakeholders—and describe strategies to engage these end-users?  Does the application provide information that supports a demand for this kind of a study from end-users?  Would this study’s research findings have the potential to inform decision making for key stakeholders? If so, provide an example. How likely is it that positive findings could be reproduced by others, resulting in improvements in practice and patient outcomes? Identify the potential barriers that could hinder adoption of the intervention by others.  Does the application describe a plan for how study findings will be disseminated beyond publication in peer-review journals and at national conferences?

Criterion 2 Score:

Criterion 3: Scientific merit (research design, analysis, and outcomes) (Not required for Patient and Stakeholder Reviewers) The application should show sufficient technical merit in the research design to ensure that the study goals will be met. The application should also address the following questions:  Does the application describe a clear conceptual framework anchored in background literature which informs the design, key variables, and relationship between interventions and outcomes being tested?  Does the Research Plan describe rigorous methods that demonstrate adherence to PCORI Methodology Standards?  Is the overall study design justified?  Are the patient population and study setting appropriate for the proposed research question?  Does the application provide justification that the outcome measures are validated and appropriate for the population?  Are each of the comparators (e.g., active intervention arm and comparator arm) described clearly and well-justified? If “usual care” is one of the arms, is it adequately justified and will it be sufficiently measured?  Are the sample sizes and power estimates appropriate? Is the study design (e.g., cluster randomized design, randomized controlled trial, or observational study) accounted for and is the anticipated effect size adequately justified?  Is the study plan feasible? Is the project timeline realistic, including specific scientific and engagement milestones? Is the strategy for recruiting participants feasible? Are assumptions about participant attrition realistic, and are plans to address patient or site attrition adequate?

Criterion 3 Score:

Criterion 4: Investigator(s) and environment (Not required for Patient and Stakeholder Reviewers) This criterion should assess the appropriateness (e.g., qualifications and experience) of the investigator(s)/team and the environment’s capacity (e.g., resources, facilities, and equipment) to support the proposed project. It should not be an assessment of the institution’s quality. The application should also address the following questions:  How well-qualified are the PIs, collaborators, and other researchers to conduct the proposed activities? Is there evidence of sufficient clinical or statistical expertise (if applicable)?  Does the investigator or co-investigator have demonstrated experience conducting projects of a similar size, scope, and complexity?  If the project is collaborative or dual-PI, do the investigators have complementary and integrated expertise? Are the leadership, governance, and organizational structures appropriate for the project? o (Dual-PI Option Only) Does the Leadership Plan adequately describe and justify PI roles and areas of responsibility?  Is the level of effort for each team member appropriate for successfully conducting the proposed work?  Does the application describe adequate availability of and access to facilities and resources (including patient populations, samples, and collaborative arrangements) to carry out the proposed research?  Is the institutional support appropriate for the proposed research?

Criterion 4 Score:

Criterion 5: Patient-centeredness (All reviewers) The application should demonstrate that the study focuses on improving patient-centered outcomes and employs a patient-centered research design (i.e., a design informed or endorsed by patients). (Note: The study can be patient-centered even if the end-user is not the patient, as long as patients will benefit from information.) The proposal should also address the following questions:  Does the application include a thorough description about which outcomes (both benefits and harms) are important to patients, and are those outcomes included in the study plan?  Does the application provide information that indicates that closing the evidence gap is important to patients and other stakeholders?  Are the interventions being compared in the study available to patients now, and are they the best options for comparison (including whether they would be chosen by patients and their healthcare providers for managing the condition being studied)?

Criterion 5 Score:

Criterion 6: Patient and stakeholder engagement (All reviewers) The application should demonstrate the engagement of relevant patients and other stakeholders (e.g., patients, caregivers, clinicians, policy makers, hospitals and health systems, payers [insurance], purchasers [business], industry, researchers, and training institutions) in the conduct of the study. Quality of engagement should be evaluated based on scope, form, and frequency of patient and stakeholder involvement throughout entire research process. The application should also address the following questions:  Does the application provide a well-justified description of how the research team incorporates stakeholder involvement? Does the study include the right individuals (e.g., researchers, patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders) to ensure that the projects will be carried out successfully?  Does the application show evidence of active engagement among scientists, patients, and other stakeholders throughout the research process (e.g., formulating questions, identifying outcomes, monitoring the study, disseminating, and implementing)? Is the frequency and level of patient and stakeholder involvement sufficient to support the study goals?  Is the proposed Engagement Plan appropriate and tailored to the study?  Are the roles and the decision-making authority of all study partners described clearly?  Are the organizational structure and resources appropriate to engage patients and stakeholder throughout the project? Strengths:   

Criterion 6 Score:

Protection of Human Subjects: (Not required for Patient and Stakeholder Reviewers)

Does the application have acceptable risks and/or adequate protections for human subjects? (Yes/No)

Please provide comments related to human subjects protections, if any.

Overall Score:

Overall Narrative: Please provide your overall narrative here. The narrative should be written in paragraph form and provide a summary of strengths and weaknesses that drove the overall score.

Reference: Key Points

Once you have completed your written critique based on the review criteria, double-check to make sure that you have addressed these key points:

Criterion 2  Does the proposed research study the benefits and harms of interventions delivered in real-world settings, and will the study provide practical information to help patients make informed healthcare decisions? Criterion 3  Does the proposed research compare at least two viable alternatives, each supported by efficacy data? If efficacy data is not sufficient, then are the interventions in common use?  Does the proposed research have sufficient statistical power to detect clinically meaningful effects? Are proposed sample sizes justified? Are assumptions used in power calculations explained? Criterion 5  Does the proposed research include health outcomes that are meaningful to the patient population, their caregivers, and family members under study, and that are likely to guide their decisions? All Criteria (as appropriate for each type of reviewer):  Does the proposed research adhere to all relevant PCORI Methodology Standards? (Note relevant Review Criteria for addressing each standard.) o Identify a true gap in evidence (RQ-1; Crit. 1) o Provide a rationale and define “usual care,” if it is proposed (RQ-5; Crit. 2 & 3) o Show that the outcomes chosen are relevant to patients and end users (RQ-6; Crit. 4) o Show that study participants represent target populations and are properly recruited, enrolled, and retained (PC-2; Crit. 3) o Describe rigorous data analysis plans (IR-1, IR-3; Crit. 3) o Describe their plan and statistical methods for handling missing data in their project (MD-1, MD-2; Crit. 3) o Address heterogeneity of treatment effects and state the aims of HTE analysis (HT-1; Crit. 3)

Notify your MRO if the application proposes to:  Measure cost-effectiveness or conduct a formal cost-effectiveness analysis as a primary aim.  Develop, test, and/or validate individual decision aids.

Reference: APDTO PFA Summary

What is APDTO interested in? The APDTO program is seeking applications for comparative effectiveness research designed to provide evidence-based information to inform critical decisions that face patients and caregivers, clinicians, policy makers, and healthcare system leaders. APDTO must fund applications that compare the effectiveness of at least 2 viable alternatives that have established efficacy.

PCORI seeks to fund investigator-initiated research that:  Compares the effectiveness of two or more strategies for prevention, treatment, screening, diagnosis, or management that are known to be efficacious but have not been adequately compared in previous studies. It may be appropriate to include as a comparator a Please notify your MRO as generally accepted practice that occurs with insufficient soon as possible if you evidence of efficacy or effectiveness. In cases where a believe any of your comparator does not have sufficient evidence of efficacy, assigned applications do documentation of the prevalence of use is necessary. not include a comparator.  PCORI is particularly interested in studies that are Your MRO will discuss any conducted in typical clinical populations and that address identified issues with the full range of relevant patient-centered outcomes program staff and provide (PCOs) you with further  Addresses a high-priority evidence gap, as identified by instructions. prior systematic reviews, clearly defined gaps in clinical guidelines, or other credible evidence reviews  Among compared groups, investigates various factors that account for variation in treatment outcomes, with attention paid to demographic, biological, clinical, social, economic, or geographic factors, comorbidities, and other factors that may influence those outcomes; strategies may focus on patient populations with a single condition or involve patients with a range of conditions.

What is APDTO not interested in? PCORI cannot fund applications that measure cost effectiveness. If an application meets  Research cannot conduct a formal cost-effectiveness analysis in any of these the form of dollar-cost per quality-adjusted life year to conditions, please compare two or more alternatives. Note that the consideration notify your MRO as of costs borne by the patient, such as copayments, deductibles, soon as possible. Your and other out-of-pocket costs, may be appropriate secondary MRO will discuss all outcomes. identified issues with  Research cannot measure the relative costs of care of two or program staff and more alternative approaches as the primary criterion for provide you with choosing the preferred alternative. further instructions.

PCORI cannot fund applications that develop practice guidelines, coverage recommendations, payment or policy recommendations. PCORI cannot fund research whose findings will include: • Practice guidelines • Coverage recommendations • Payment or policy recommendations • Creation of clinical practice guidelines or care pathways • Establishing efficacy for a new clinical strategy • Pharmacodynamics • Study of the natural history of disease • Fundamental science or study of biological mechanisms

APDTO applications should not focus on the following categories:  Instrument development  Developing, testing and validating new decision aids/tools or clinical prognostication tools  Pilot studies intended to inform larger efforts

 Comparisons of patient characteristics rather than clinical strategy options  Studies comparing interventions for which the primary focus in the role of community health workers or patient navigators