Cardioid: Whole Human Heart Modeling and Simulation

a collaboration between IBM and Lawrence Livermore National Laboratory

Dr. Frederick Streitz Director, HPC Innovation Center Lawrence Livermore National Laboratory

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344

LLNL-PRES-603112 Partnership Overview

• LLNL partnering with U.S. industry to promote U.S. competitiveness • Creates Livermore Valley Open Campus and the HPC Innovation Center • IBM partnering with LLNL to form Deep Computing Solutions in the HPCIC

• Working together, IBM and LLNL create Cardioid code: aimed at accelerating cures for heart disease and aiding in drug screening and the development of new medical devices and patient-specific therapies. • Seeking partners for application and commercialization of Cardioid, as well as other projects Partnering with Lawrence Livermore National Laboratory

• Livermore Valley Open Campus: research park environment with collaborative space • Ready access for all partners to world-renowned facilities and resources • HPCIC delivers computing solutions with access to computing and expertise HPCIC • Coming Soon: Institute for Translational Biomedicine • 25 year plan to develop 2.5M sq.ft. hosting 3,000 researchers HPCIC and Deep Computing Solutions

• Innovation ecosystem focused on HPC solutions for industry problems Partnering with industry to develop, prove, • HPCIC offers access to world-class and deploy HPC solutions computing resources and 1000+ computational experts • IBM adds experience with industrial-grade software solutions and deep bench of scientists and engineers • Together, LLNL and IBM bring an unmatched combination of capability, experience and resources Top Supercomputer vs Top Killer

Sequoia 20 petaFLOPS 98,304 nodes Near-cellular resolution, real-time simulation 1,572,864 processors of electrophysiology of the human heart Sudden Cardiac Arrest and Arrhythmia

• Sudden Cardiac Arrest (SCA) is a leading cause of death in the United States ~ 325,000 deaths/year (around 18 during this talk!) • SCA results from irregular and chaotic electrical disturbances that are yet to be fully understood in mechanistic and quantitative detail • Diverse drugs (see examples below) can promote arrhythmia and SCA – even antiarrhythmics • Pharmaceutical companies and the FDA struggle to understand this cardiotoxicity, and numerous high profile drugs have been withdrawn at high cost

Antiarrhythmics Antimicrobials Antidepressant Antipsychotics Others Withdrawn Almokalant Ciprofloxacin Amitriptyline Chlorpromazide Alosetron Astemizole Amiodarone Clarithromycin Desipramine Droperidol Arsenic Cisapride Azimilide Erythromycin Doxepin Haloperidol Dolasetron Grepafloxacin Dofetilide Gatilfloxacin Fluoxetine Pimozide Methadone Rofecoxib Ibutilide Irtraconazole Imipramine Quetiapine Sumatriptan Sertindole Procainamide Ketoconazole Sertraline Risperidone Zolmitriptan Sibutramine Quinidine Levofloxacin Venlafaxine Sultopride Terfenadine Sematilide Moxifloxacin Thioridazine Terodiline Sotalol Ziprasidone Difficulty Predicting Arrhythmia Risk

FDA Statement regarding azithromycin (Zithromax) and the risk of cardiovascular death

• Azithromycin is widely prescribed (55.3 million/year in the U.S.) • Initial studies suggested drug is safe • Recent studies show 47 deaths per million courses • Danger revealed only after administration across large and diverse population Cardiac modeling can help

• Complexity of measured ECG makes identification of mechanisms difficult • Some ECG features (long QT syndrome) linked to arrhythmia risk- but not specific • Current electrophysiology models suffer from lack of scale and resolution F. H. Netter. Thieme. 1990 Supercomputing to the rescue! First: Create accurate model of real heart from actual data

Visible Human Project® Create mesh model from actual heart data Generate cardiac fibers

Introduce M-cell islands Then: “a simple matter of coding”

Reaction cell model of heart

ten Tusscher electrophysiological model (2006) 1 m = 1+ e(Vm+29)/7 dg Reumann, Gurev Rice, 2009 = (m(V )- g)t -1(V ) dt m m

(25-Vm )/10 (Vm+30)/10 -1 1+e +80 / (1+ e ) t = 2 562e-(Vm+27) /240 +31 Finally: Tune code to hardware

• Use low level SPI for halo data exchange of data between • Use of vector intrinsics and custom divides tasks (DMA) • Movement of conventional integer operations to floating point • Assigned “diffusion work” and “reaction work” to different units to exploit the SIMD units cores • L2 on node thread barriers • Transformed the potassium equation to remove serialization • Partitioned cells over tasks using an upper bound on time rather • Expensive 1D functions in reaction model expressed with than equal time metric rational approximates • Application managed threads • Single precision weights to reduce L2 bandwidth requirements • SIMDized diffusion stencil implementation for diffusion stencil • Zero flux boundary conditions approximated by method with no • Conventional unrolling of loops over cells to exploit SIMD units global solve • Sorting by cell type to improve vectorization • High performance I/O is aware of BG/Q network topology • Sub-sorting of cells to increase sequential/vector load and • Low overhead in-situ performance monitors storing of data. • Assignment of threads to diffusion/reaction dependent on • log function from libm replaced with custom inlined functions domain characteristics • On the fly assembly of code to optimize data movement at • Co-scheduled threads for improved dual issue runtime • Multiple diffusion implementations to obtain optimal • Memory layout tuned to improve cache performance performance for various domains • No explicit network barrier • Remote & local copies separated to improve bandwidth utilization Finally: Tune code to hardware

• Use low level SPI for halo data exchange of data between • Use of vector intrinsics and custom divides tasks (DMA) • Movement of conventional integer operations to floating point • Assigned “diffusion work” and “reaction work” to different units to exploit the SIMD units cores • L2 on node thread barriers • Transformed the potassium equation to remove serialization • Partitioned cells over tasks using an upper bound on time rather • Expensive 1D functions in reaction model expressed with than equal time metric rational approximatesAchieve 60% of• Application theoretical managed threads • Single precision weights to reduce L2 bandwidth requirements • SIMDized diffusion stencil implementation for diffusion stencil • Zero flux boundary conditions approximated by method with no • Conventional unrolling of loops over cells to exploit SIMD units global solve • Sorting by cell type topeak improve vectorization performance • High performance on I/O is aware of BG/Q network topology • Sub-sorting of cells to increase sequential/vector load and • Low overhead in-situ performance monitors storing of data. • Assignment of threads to diffusion/reaction dependent on • log function from libmSequoia replaced with custom inlined supercomputer functions domain characteristics • On the fly assembly of code to optimize data movement at • Co-scheduled threads for improved dual issue runtime • Multiple diffusion implementations to obtain optimal • Memory layout tuned to improve cache performance performance for various domains • No explicit network barrier • Remote & local copies separated to improve bandwidth utilization Deliver: electrophysiology of human heart in exquisite detail

ArrhythmicSlow-motion behavior simulation in heart of heartwedge beat Real-time simulation of heart beat Deliver: reentrant activation in presence of d-sotalol Arrhythmic behavior in heart wedge

with drug without drug Thousand-fold advancement in the state-of-the-art

0.1 mm resolution 370 million tissue cells

Cardioid Analysis of model ECG from simulation can Previous state of the art be compared against actual ECG patterns Model Development Strategy

1. Start with electrophysiological activity to study arrhythmia, sudden cardiac arrest and pharmacological effects of drugs on heart 2. Add mechanical model and vascular flow to study dynamics of drug delivery to heart tissue 3. Add fluid dynamics model to study blood flow, pooling, clotting and murmurs Working as team to deliver results

IBM Team LLNL Team John Jeremy Rice Arthur A. Mirin John A. Gunnels David F. Richards James N. Glosli Viatcheslav Gurev Erik W. Draeger Changhoan Kim Bor Chan John Magerlein Jean-luc Fattebert Matthias Reumann William D. Krauss Hui-Fang Wen Tomas Oppelstrup

Partner Opportunities

• For cardiology community: exploit/customize/optimize Cardioid for specific purposes, starting from current state of code • Drug screening against adverse effects • Drug discovery/design • Optimum drug dosage • Cardiac ablation for arrhythmia • Cardiac device placement and settings • For biomedical community: partner in the creation and/or development of other solutions

Summary/Key Points

• Lawrence Livermore National Laboratory is open for business! • Cardioid demonstrates power of high performance computing for solutions • Now seeking partners from across biomedical industry to sponsor development of Cardioid for commercialization or to pursue other proprietary work • Pharmaceuticals Contact:

• Medical devices Frederick Streitz James Sexton Director, HPCIC Director, DCS • Health care delivery [email protected] [email protected] m