THE STANDARDS OF NORMAL SCIENCE: PSEUDOSCIENCE IN HEALTH TECHNOLOGY ASSESSMENT Recent Current Issue Reports have made the case that the methods espoused by groups such as the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), the Academy of Managed Care Pharmacy (AMCP), and the Institute for Clinical and Economic Review (ICER) to support modeled claims for cost-effectiveness fail the standards of normal science. This follows from their focus on the construction of imaginary lifetime simulations where value claims are driven by imaginary, incremental cost-per-QALY calculations set against arbitrary willingness-to- pay thresholds. This approach to technology assessment should be rejected. It adds nothing to our discovery of new facts because it fails the demarcation test between science and pseudoscience. Here’s why. The Pseudoscience Position approach is “to define value on the basis of ISPOR reaffirmed its commitment to the microeconomic principles, recognizing that construction of imaginary worlds in a 2018 value is best defined as what individuals (or Special Task Force Report.1 This task force was others acting on their behalf) would be willing to convened to “review relevant perspectives and pay to acquire more health care or other goods appropriate approaches and methods to support and services.” This is best achieved, the Report the definition and use of high-quality value maintains, by informing resource allocation frameworks”; presumably “high value” imaginary decisions by “approximating” the value of constructs, although no criteria are suggested interventions in incremental cost per quality that may distinguish “high value” from “low adjusted life year (QALY) term, capturing length value” imaginary worlds, or how a metric might and quality of life in an imaginary reference case be applied to inform those who are willing to construct. utilize imaginary worlds in formulary decisions. 2 This approximate value of interventions is Expressing a concern that in attempting to achieved through the construction of lifetime simplify the problem of value, competing value imaginary worlds. A hypothetical population is frameworks could end up making ad hoc tracked, on the basis of a series of assumptions assumptions and simplifications not supported detailing the structure of the simulation, the by theory or evidence, the Report takes the choice of a therapy and a comparator, the position that frameworks that do not capture the assumed transition of the target patient group full costs and benefits of treatment may distort through the various stages of disease over their decisions, although “full” is not defined outside lifetimes as determined by selected clinical of broad reference case guidelines. ISPOR’s response parameters, and the time spent in each disease state. Each stage of the disease is Patient Access & Affordability Project | Current Issues – Report #3 assigned a utility score extracted from the with their respective mottos Provando e literature, and the result is the creation of Riprovando – “prove and again prove,” and average imaginary lifetime QALYs for the drug nullius in verba – “take no man’s word for it.”3 and its comparator. Incremental lifetime costs per QALY are estimated and matched to baseline By the early 20th century, standards for empirical “willingness to pay” thresholds. Thus is the assessment were put on a sound methodological approximate – yet imaginary – “value” of a new basis by Sir Karl Popper (1902-1994) in his product assessed compared, for example, to the advocacy of a process of ‘conjecture and standard of care. However, ICER takes these refutation.4 5 Hypotheses or claims must be comparisons and uses them to make capable of falsification; indeed, they should be recommendations for price discounting and framed in such a way that makes falsification access. The imaginary and arbitrary construct is likely. Falsification of our hypotheses forces us not intended to provide credible, evaluable, and to reconsider our models and the assumptions replicable claims – the construct is incapable of built into those models. This leads to the obvious providing such information. Yet ICER claims it is point that claims or models should not be the “state of the art” in health technology judged on the realism or reasonableness of assessment. Judged against the standards of assumptions, or on whether the model normal science, the ICER construct qualifies as represents – for a public advocacy research nothing more than pseudoscience. group such as ICER – their belief in lifetime comparative cost-per-QALY outcomes in a The Standards of Normal Science future hypothetical reality. The requirement for testable hypotheses in the evaluation and provisional acceptance of claims Although Popper’s view on what demarcates made for products and devices is so fundamental science from pseudoscience is now seen as an as to be unexceptional. For hundreds of years, oversimplification involving more than just the scientists and other thinkers have accepted as a criteria of falsification, the demarcation problem 6 baseline requirement that if a research agenda is remains. Certainly, there are different ways of to advance – if there is to be an accumulation of doing science, but what all scientific inquiry has knowledge – there must be a process of in common is the “construction of empirically discovering new facts. verifiable theories and hypotheses.” Empirical testability is the one major characteristic Indeed, as early as the 16th century, Leonardo da distinguishing science from pseudoscience; Vinci (1452 – 1519), in notes that appeared theories must be tested against data. The posthumously for his Treatise on Painting development of pharmaceutical products and (published in 1641), clearly anticipated the the evidence standards required by the Food and standards for the scientific method, which were Drug Administration (FDA) for product widely embraced a century later in rejecting evaluation and marketing approval is driven by thought experiments that fail the test of stringent adherence to the scientific method. experience. However, paradoxically, once a product is launched, to support its claims about cost- By the 1660s, the scientific method, following the effectiveness, ICER’s modeled pricing and access seminal contributions of Bacon, Galileo, recommendations abandon the scientific Huygens, and Boyle, had been clearly articulated method. by associations such as the Academia del Cimento in Florence (1657) and the Royal Society The rejection of a research program that meets in England (founded 1660; Royal Charter 1662) the standards of normal science by groups such as ICER is best exemplified by the latest version 2 of the Canadian health technology guidelines, interpretation of ISPOR and ICER’s continued, which state, in part: “Economic evaluations are unqualified acceptance of the reference case as designed to inform decisions. As such, they are its core business model, as a sociological distinct from conventional research activities, phenomenon. The ICER reference case that which are designed to test hypotheses.” 7 While constructs evidence to support pricing and this position puts modeled health technology affordability decrees can be characterized as an assessment in the category of pseudoscience, it analog of gene pool propagation “by leaping is also what may be described as a relativist from body to body via sperm or eggs.”8 position. Human beings are good at imitation. The Rather than subscribing to the position that the reference case meme appears to be adept in its standards of normal science are the only infectivity, supported by an organizational standards to apply in health care decisions and infrastructure to defend it against competing value claims, the relativist believes that all views, ensuring survival through supporting perspectives are equally valid. Health care propagation, longevity, acceptability, and decisions are to be understood sociologically. No copying fidelity. The widespread adoption and one body of evidence is superior to another. propagation of this meme in seen with literally Results of a lifetime modeled simulation are on thousands of imaginary world technology an equal basis with those of a pivotal Phase 3 assessments published. ISPOR’s continuing randomized clinical trial. For the relativist, the enthusiastic efforts at transmission and infection success of a scientific research program – in this ensure a high copying fidelity with few willing to case one built on hypothetical models and reject the “faith and mysteries” of their parents’ simulations – rests not on its ability to generate belief system.9 new knowledge but on its ability to mobilize the support of the community. Basing decisions on Driven by Assumption models and simulations underpins the Knowledge is provisional and permanently so; consensus view that evidence is constructed, we cannot, at any stage, prove that what we never discovered. This camp would have you “know” is true. Attempting to believe or justify believe that instead of discovery, science is about our belief in a theory is logically impossible. 11 rhetoric, persuasion and authority. Truth is However, by empirical assessment, we can try consensus. and demonstrate our preference for one theory over another and apply it to the best of our The Health Technology Assessment knowledge. Meme If truth is consensus, how is this consensus, Constructing imaginary worlds that were never resting upon the construction of imaginary intended to