A Bridge for the 'Leap of Faith' with Computational Models
Total Page：16
File Type：pdf, Size：1020Kb
Office of Science and Engineering Laboratories Excellence in Regulatory Science A Bridge for the ‘Leap of Faith’ with Computational Models Pras Pathmanathan, PhD, Richard Gray, PhD, Leonardo Angelone, PhD and Tina Morrison, PhD Model as used in BACKGROUND APPLICABILITY Reality being modelled 1. Describe COU context-of-use R-COU Provide details on both the ‘reality’ to be modelled and the model that will simulate In silico methods are currently being developed to augment in vitro and in Regardless of whether direct or indirect validation is employed, there is a M-COU ‘reality’. For example: vivo evaluation methods for pharmaceutical products, medical devices fundamental question that remains regarding the applicability of the R-COU: Estimate the worst-case temperature change in the tissue around the implant 1 and biological products . A grand promise lies in reducing the number and computational model and the validation results to the COU. As shown in ? that might occur during MRI across a wide-array of parameters… size of clinical trials by augmenting them with data from in silico trials, and Table 1, the main VVUQ methods address the model’s predictive M-COU: Solve an electromechanical model and a thermal model5 using a human body realising precision medicine through simulation-based diagnosis, therapy, capability when compared to experimental results, as well as numerical anatomical model6 containing the device, to determine the maximum temperature 2 and clinical guidance . accuracy, the robustness of the QOI to perturbations, and identification change under simulated MRI conditions for a range of parameters that closely represent and quantification of uncertainty. They do not address the relevance of Despite decades of research, however, progress in translating Image courtesy of Maria Iacono the clinical setting… computational models to clinical care has been limited3.Onemajor the VVUQ evidence for applying the computational model to the COU. In order to truly determine the credibility of the predictions from the challenge is demonstrating the reliability of predictions from in silico SET-UP 2. Sources of validation evidence Primary computational model for the COU, one needs to determine how Subject matter General approaches. Below, we describe how the current practice of developing a Describe the different types of validation results and sources of evidence. Denote one as validation validation applicable the supporting scientific evidence and the computational model expertise model, performing validation, and then modifying the model in some way the primary validation evidence. results results to apply it to a new setting, without ever explicitly addressing whether the are for the proposed COU. Our goal is to address the following question: 1. Primary validation evidence: Phantom validation experiments using the new device. Other predictions remain credible given the modifications, can lead to a ‘leap of 2. Animal validation experiments using a previous version of the device. Q-App: If there is agreement between the outputs from the model and Historical data COU-based Established faith’ being required to trust the results. Therefore, as a ‘bridge’ for the 3. All historical validation evidence regarding similar models with different devices. experiment in the validation setting(s), can we (or: why can we) be validation theory leap of faith, we provide a framework that formalises the process for 4. Literature regarding electromagnetic simulations using human anatomical models results scrutinizing the available scientific evidence so that one can make confident in the model predictions for the context of use? an informed decision. Model as used in Answering Q-App requires careful consideration of the computational Validation comparator 3. Describe primary validation evidence validation simulations model, the COU, and the available scientific evidence. Usually, a R-V Provide details on both the validation setting and the model for the validation setting. M-V subjective decision is made using sound scientific judgment, based on all R-V: A saline-filled phantom implanted with the new device was placed inside an RF CURRENT VVUQ METHODS Directly the available evidence and utilising subject matter expertise. Currently, coil; temperature changes were measured… comparable Current practice for demonstrating credibility relies on verification, however, there is no formal or systematic method for accomplishing this, M-V: A geometrical model of the phantom was used with the embedded device with validation, sensitivity analysis, and uncertainty quantification4.The especially for modelling scenarios that rely on indirect validation. one RF coil. The EM/thermal model was used to compute temperature change… questions these methods address are provided in Table 1. For brevity, we refer to the collective methods as VVUQ. THE LEAP OF FAITH Images courtesy Elena Lucano The overall goal of VVUQ is to evaluate the credibility of the 4. Describe model aspects common between M-V and M-COU computational model, the belief in its predictive capability, for a specific The figure below illustrates our interpretation of the process to determine The model that was ‘validated’ using the primary validation evidence will likely vary from the model that is used for the context of use. However, context of use (COU), which is the specific role and scope of the credibility. Different stages of the assessment process are represented there will be many aspects of the model that remain the same. Therefore, describe the aspects of the model that are equivalent between M-COU computational model and simulation results used to inform a decision, by arrows, where the length of the arrow loosely represents the and M-V: confidence gained from that stage in the VVUQ methodology. Maxwell’s equations are solved Equations of thermal model Parameters for EM properties of the device …. 5. Describe M DESCRIPTIONS 6. Describe R R-COU M-COU Describe the aspects of the model that Describe the relevant differences between are different between M-V and M-COU R-V and R-COU Phantom → virtual human Phantom → human (both sexes, range Path of device lead of ages and BMIs) ΔR ΔM … Path of device lead Table 1. Questions that can be asked when evaluating the credibility of the computational model for a context of use MR system present in R-COU, only RF R-V M-V coil in R-V … DIRECT & INDIRECT VALIDATION 7. ∆R in light of common aspects between M-V and M-COU (Table on the right) VVUQ methods have been mainly motivated and developed by the If the validation comparison is deemed adequate, can we be confident that the model engineering and physical sciences communities, such as aerospace, aspects common between M-V and M-COU, from step 4, are appropriate for the COU? automotive, structural, and nuclear industries, and have been vital to the Differences between R-V and R-COU, from step 6, may mean that this is not the case. success of computational modelling in these sectors. Part of this success Therefore, for each aspect from step 4, provide rationale and/or evidence for why it is is due to the ability to perform ‘direct validation’. appropriate that that aspect remains the same by considering each of the relevant differences presented in step 6. The table to the right will help you walk through this. Validation is the process of determining the degree to which a model or For each difference from R, answer the following question: “with this difference in mind, simulation accurately represents the real world from the perspective of the OVERVIEW : APPLICABILITY FRAMEWORK FRAMEWORK APPLICABILITY : OVERVIEW is it acceptable to keep this aspect of the computational model the same for the COU?” intended use of the model or simulation. Applications for which direct validation is possible (e.g., engineering By direct validation, we mean a validation study that includes a carefully applications) are illustrated on the left, where good agreement between 8. Reasons to support model credibility given ∆M designed comparator that closely matches the setting of the COU. model and experiment in the validation provides high confidence in the model for the COU. For applications that rely on indirect validation (e.g., For each modification in ∆M, explain why the COU predictions can be trusted given each modification, keeping in mind the COU In the engineering and physical science industries, direct validation is many medical applications) good validation results may not be sufficient Q) Why can predictions be trusted given the modification: phantom → virtual human? often feasible. For example, in aviation, new airplanes are simulated to generate trust in the model for the COU (middle). If applicability is not Q) Why can predictions be trusted when the path of the conductor lead is altered? before they are constructed because direct validation of the carefully addressed at this point, a ‘leap of faith’ (red arrow) is made by ASSESSMENT Q) .. computational model is possible using smaller-scaled airplanes that are the modeller, and may be required of any scientist reviewing the Careful arguments are required, referring to all supporting evidence rigorously tested in a wind-tunnel. simulation results. We believe that a paradigm shift is necessary to ensure that applicability 9. M-COU in light of ∆R (Table on the left) In contrast, for computational models in medicine, direct validation is is explicitly and rigorously addressed. To this end, we developed an Consider M-COU in the context of differences between R-V and R-COU. often not possible. Possible reasons are ethical, technological and/or applicability framework for systematically and comprehensively financial. For such models, ‘indirect validation’ is performed. assessing the applicability of a computational model and its validation 10. Discuss the overall applicability of the computational model for the COU evidence for a specific COU This requires careful consideration of the specific questions raised in Steps 7-9, and how well they were addressed using the supporting evidence By indirect validation, we mean a validation study that uses a or subject matter expertise.