Office of Science and Engineering Laboratories Excellence in Regulatory Science A Bridge for the ‘Leap of Faith’ with Computational Models Pras Pathmanathan, PhD, Richard Gray, PhD, Leonardo Angelone, PhD and Tina Morrison, PhD

Model as used in BACKGROUND APPLICABILITY Reality being modelled 1. Describe COU context-of-use R-COU Provide details on both the ‘reality’ to be modelled and the model that will simulate In silico methods are currently being developed to augment in vitro and in Regardless of whether direct or indirect validation is employed, there is a M-COU ‘reality’. For example: vivo evaluation methods for pharmaceutical products, medical devices fundamental question that remains regarding the applicability of the R-COU: Estimate the worst-case temperature change in the tissue around the implant 1 and biological products . A grand promise lies in reducing the number and computational model and the validation results to the COU. As shown in ? that might occur during MRI across a wide-array of parameters… size of clinical trials by augmenting them with data from in silico trials, and Table 1, the main VVUQ methods address the model’s predictive M-COU: Solve an electromechanical model and a thermal model5 using a human body realising precision medicine through simulation-based diagnosis, therapy, capability when compared to experimental results, as well as numerical anatomical model6 containing the device, to determine the maximum temperature 2 and clinical guidance . accuracy, the robustness of the QOI to perturbations, and identification change under simulated MRI conditions for a range of parameters that closely represent and quantification of uncertainty. They do not address the relevance of Despite decades of research, however, progress in translating Image courtesy of Maria Iacono the clinical setting… computational models to clinical care has been limited3.Onemajor the VVUQ evidence for applying the computational model to the COU. In order to truly determine the credibility of the predictions from the challenge is demonstrating the reliability of predictions from in silico SET-UP 2. Sources of validation evidence Primary computational model for the COU, one needs to determine how Subject matter General approaches. Below, we describe how the current practice of developing a Describe the different types of validation results and sources of evidence. Denote one as validation validation applicable the supporting scientific evidence and the computational model expertise model, performing validation, and then modifying the model in some way the primary validation evidence. results results to apply it to a new setting, without ever explicitly addressing whether the are for the proposed COU. Our goal is to address the following question: 1. Primary validation evidence: Phantom validation experiments using the new device. Other predictions remain credible given the modifications, can lead to a ‘leap of 2. Animal validation experiments using a previous version of the device. Q-App: If there is agreement between the outputs from the model and Historical data COU-based Established faith’ being required to trust the results. Therefore, as a ‘bridge’ for the 3. All historical validation evidence regarding similar models with different devices. experiment in the validation setting(s), can we (or: why can we) be validation theory leap of faith, we provide a framework that formalises the process for 4. Literature regarding electromagnetic simulations using human anatomical models results scrutinizing the available scientific evidence so that one can make confident in the model predictions for the context of use? an informed decision. Model as used in Answering Q-App requires careful consideration of the computational Validation comparator 3. Describe primary validation evidence validation simulations model, the COU, and the available scientific evidence. Usually, a R-V Provide details on both the validation setting and the model for the validation setting. M-V subjective decision is made using sound scientific judgment, based on all R-V: A saline-filled phantom implanted with the new device was placed inside an RF CURRENT VVUQ METHODS Directly the available evidence and utilising subject matter expertise. Currently, coil; temperature changes were measured… comparable Current practice for demonstrating credibility relies on verification, however, there is no formal or systematic method for accomplishing this, M-V: A geometrical model of the phantom was used with the embedded device with validation, sensitivity analysis, and uncertainty quantification4.The especially for modelling scenarios that rely on indirect validation. one RF coil. The EM/thermal model was used to compute temperature change… questions these methods address are provided in Table 1. For brevity, we refer to the collective methods as VVUQ. THE LEAP OF FAITH Images courtesy Elena Lucano The overall goal of VVUQ is to evaluate the credibility of the 4. Describe model aspects common between M-V and M-COU computational model, the belief in its predictive capability, for a specific The figure below illustrates our interpretation of the process to determine The model that was ‘validated’ using the primary validation evidence will likely vary from the model that is used for the context of use. However, context of use (COU), which is the specific role and scope of the credibility. Different stages of the assessment process are represented there will be many aspects of the model that remain the same. Therefore, describe the aspects of the model that are equivalent between M-COU computational model and simulation results used to inform a decision, by arrows, where the length of the arrow loosely represents the and M-V: confidence gained from that stage in the VVUQ methodology.  Maxwell’s equations are solved  Equations of thermal model  Parameters for EM properties of the device  ….

5. Describe M DESCRIPTIONS 6. Describe R R-COU M-COU Describe the aspects of the model that Describe the relevant differences between are different between M-V and M-COU R-V and R-COU  Phantom → virtual human  Phantom → human (both sexes, range  Path of device lead of ages and BMIs) ΔR ΔM  …  Path of device lead Table 1. Questions that can be asked when evaluating the credibility of the computational model for a context of use  MR system present in R-COU, only RF R-V M-V coil in R-V  … DIRECT & INDIRECT VALIDATION 7. ∆R in light of common aspects between M-V and M-COU (Table on the right) VVUQ methods have been mainly motivated and developed by the If the validation comparison is deemed adequate, can we be confident that the model engineering and physical sciences communities, such as aerospace, aspects common between M-V and M-COU, from step 4, are appropriate for the COU? automotive, structural, and nuclear industries, and have been vital to the Differences between R-V and R-COU, from step 6, may mean that this is not the case. success of computational modelling in these sectors. Part of this success Therefore, for each aspect from step 4, provide rationale and/or evidence for why it is is due to the ability to perform ‘direct validation’. appropriate that that aspect remains the same by considering each of the relevant differences presented in step 6. The table to the right will help you walk through this. Validation is the process of determining the degree to which a model or For each difference from R, answer the following question: “with this difference in mind, simulation accurately represents the real world from the perspective of the OVERVIEW : APPLICABILITY FRAMEWORK FRAMEWORK APPLICABILITY : OVERVIEW is it acceptable to keep this aspect of the computational model the same for the COU?” intended use of the model or simulation. Applications for which direct validation is possible (e.g., engineering By direct validation, we mean a validation study that includes a carefully applications) are illustrated on the left, where good agreement between 8. Reasons to support model credibility given ∆M designed comparator that closely matches the setting of the COU. model and experiment in the validation provides high confidence in the model for the COU. For applications that rely on indirect validation (e.g., For each modification in ∆M, explain why the COU predictions can be trusted given each modification, keeping in mind the COU In the engineering and physical science industries, direct validation is many medical applications) good validation results may not be sufficient Q) Why can predictions be trusted given the modification: phantom → virtual human? often feasible. For example, in aviation, new airplanes are simulated to generate trust in the model for the COU (middle). If applicability is not Q) Why can predictions be trusted when the path of the conductor lead is altered? before they are constructed because direct validation of the carefully addressed at this point, a ‘leap of faith’ (red arrow) is made by ASSESSMENT Q) .. computational model is possible using smaller-scaled airplanes that are the modeller, and may be required of any scientist reviewing the Careful arguments are required, referring to all supporting evidence rigorously tested in a wind-tunnel. simulation results. We believe that a paradigm shift is necessary to ensure that applicability 9. M-COU in light of ∆R (Table on the left) In contrast, for computational models in medicine, direct validation is is explicitly and rigorously addressed. To this end, we developed an Consider M-COU in the context of differences between R-V and R-COU. often not possible. Possible reasons are ethical, technological and/or applicability framework for systematically and comprehensively financial. For such models, ‘indirect validation’ is performed. assessing the applicability of a computational model and its validation 10. Discuss the overall applicability of the computational model for the COU evidence for a specific COU This requires careful consideration of the specific questions raised in Steps 7-9, and how well they were addressed using the supporting evidence By indirect validation, we mean a validation study that uses a or subject matter expertise. It should now be clear how the applicability framework provides a methodological approach to break down the comparator with significant differences to the setting of the COU. The framework enables the practitioner to systematically convert the question of applicability into a series of tractable questions to inform the use of the computational model for the COU. broad question Q-App into more specific tractable questions for For example, computational simulations used for microgravity space-flight evaluating the trustworthiness of a model, which can be answered by Key takeaways missions are often validated in settings that mimic low gravity. Often, referencing the multiple sources of scientific evidence or utilising subject  Assessing the applicability of a computational model and validation evidence is essential for avoiding ‘leaps of faith’ many types of indirect validation experiments are performed and collected matter expertise. This is illustrated in the right of the above figure, where • Especially true for models that rely on ‘indirect validation’ together as evidence to support credibility. For models with clinical COUs the proposed framework for assessing applicability uses the same  The essence of assessing applicability lies in the careful, detailed description of the reality and model components, followed by where direct validation is often not possible, the types of indirect supporting evidence to build upon the credibility provided by the validation systematic discussion of: validation may include experiments involving animal, cadavers, in vitro evidence. Our approach reduces the possibility of a real or perceived leap • relevant differences between validation and COU for both model (∆M) and reality (∆R) specimens, bench-top systems, or phantoms. Furthermore, these of faith by making explicit the potential limitations of applying a settings might also involve observations of a quantity of interest (QOI) • if the model is appropriate given the differences ∆R computational model to a COU, and using sound scientific judgment to • why the ‘validated’ computational model can be trusted given the changes ∆M that is related to but different from the QOI in the COU. support or address each one.  Framework could be invaluable in the design of validation experiments

SELECTED REFERENCES: 1. Advancing regulatory science at FDA: a strategic plan. US FDA; 2011; 2. Viceconti et al, in silico Clinical Trials: How Computer Simulation will Transform the Biomedical Industry, 2016; 3. Winslow et al, Computational medicine: translating models to clinical care. Sci Transl Med 2012; 4. Oberkampf WL, Roy CJ: Verification and validation in scientific computing. Cambridge University Press; 2010; 5. Angelone et al, Analysis of the role of lead resistivity in specific absorption rate for deep brain stimulator leads at 3T MRI. IEEE Transactions on Medical Imaging 2010; 6. Christ et al, The Virtual Family-development of surface-based anatomical models of two adults and two children for dosimetric simulations. Physics in Medicine and Biology 2010