NEURON + Threads Simulations on Multicore Desktops

Home , Brian (software), GENESIS (software)

The NEURON Simulation Environment Didactic Presentations

NEURON + Threads

Simulations on multicore desktops.

Thread style in NEURON

Join run() { while (t < tstop) { multithread_job(step) plot() } }

void* step(NrnThread* nt) { ... nt−>id ... }

We never use. Condition Wait

multithread_job(run)

run(NrnThread* nt) { while(t < tstop) { step(nt) barrier() if (nt−>id == 0) { plot() } barrier() } } Reminiscent of MPI

Fixed step: t −> t + dt

STRBUG setup triangreduce bksub update cond solve gates

SBUGT

SUGTB

ST R B UG

Global var dt y’ = f() dy’/dy 27 ||Vector operations

Ideal cache efficiency ......

CPU 1

...... cache line

8 or 16 doubles

......

CPU 2 Runtime (s) Cache Efficiency Threads Off On 1 4.92 0.45 2 1.14 0.23 10000 passive compartments 4 core 3GHz x86_64 4 0.29 0.12 8 0.23 0.09

$ mkthreadsafe NEURON { SUFFIX CAlM95 USEION ca READ cai,cao WRITE ica RANGE gbar,ica GLOBAL minf,tau } Translating CAlM95.mod into CAlM95.c Notice: Assignment to the GLOBAL variable, "minf", is not thread safe Notice: Assignment to the GLOBAL variable, "tau", is not thread safe Force THREADSAFE? [y][n]: n

DERIVATIVE state { rate(v) m’ = (minf − m)/tau }

PROCEDURE rate(v (mV)) { LOCAL a a = alp(v) tau = 1/(tfa*(a + bet(v))) minf = tfa*a*tau }

Force THREADSAFE? [y][n]: n y

NEURON { THREADSAFE SUFFIX CAlM95 USEION ca READ cai,cao WRITE ica RANGE gbar,ica GLOBAL minf,tau }

$ mkthreadsafe NEURON { POINT_PROCESS GABAa POINTER pre ... } VERBATIM return 0; ENDVERBATIM Translating gabaa.mod into gabaa.c Notice: Use of POINTER is not thread safe. Notice: VERBATIM blocks are not thread safe Notice: Assignment to the GLOBAL variable, "Rtau", is not thread safe Notice: Assignment to the GLOBAL variable, "Rinf", is not thread safe Force THREADSAFE? [y][n]: n

If thread results differ, a good way to diagnose the cause is to use prcellstate.hoc

$ nrngui mosinit.hoc ../prcellstate.hoc

load_file("prcellstate.hoc")

// serial model finitialize(−70) prtop(0) // constructs cs0.0.1 (5MB)

//switch to 4 threads finitialize(−70) prtop(1) // constructs cs1.0.1

Parallel Computation "Faster" is the only reason

But... greater programming complexity new kinds of bugs ...and not much help for fixing them.

Can the day or week of user effort be recovered?

8192 processor EPFL IBM BlueGene 1 hour at 700MHz 3 months at 3GHz

Parallel Computation

A simulation run takes about a second want to do 1000’s of them, varying a dozen or so parameters.

A simulation run takes hours. want to spread the problem over several machines.

Parallel Computation

A simulation run takes hours. want to spread the problem over several machines. Network Subnets on different machines Cells communicate by: logical spike events with significant axonal, synaptic delay. postsynaptic conductance depends continuously on presynaptic voltage. gap junctions

Parallel Computation

A simulation run takes hours. want to spread the problem over several machines. Single cells portions of the tree cable equation on different machines.

PreCell PostCell PostSyn

PreSyn

PreCell PostCell PostSyn NetCon

PreSyn

nc = new NetCon(PreSyn, PostSyn)

CPU 2 PreCell CPU 4 PostCell PostSyn NetCon

PreSyn

CPU 2 PreCell CPU 4 PostCell PostSyn

PreSyn

pc = new ParallelContext()

CPU 2 PreCell CPU 4 PostCell PostSyn

gid = 7 PreSyn gid = 9

Every spike source (cell) must have a global id number.

CPU 0 CPU 3 CPU 4 pc.id 0 pc.id 3pc.id 4 pc.nhost 5 pc.nhost 5 pc.nhost 5 ncell 14 ... ncell 14ncell 14 gid gid gid 0 3 4 5 8 9 10 13

An efficient way to distribute: for (gid = pc.id; gid < ncell; gid += pc.nhost) { pc.set_gid2node(gid, pc.id) ... } body executed only ncell/nhost times, not ncell.

CPU 2 PreCell CPU 4 PostCell PostSyn

gid = 7 PreSyn gid = 9

Create cell only where the gid exists.

if (pc.gid_exists(7)) { PreCell = new Cell() }

CPU 2 PreCell CPU 4 PostCell PostSyn

gid = 7 PreSyn

Associate gid with spike source.

nc = new NetCon(PreSyn, nil) pc.cell(7, nc)

CPU 2 PreCell CPU 4 PostCell PostSyn NetCon

7 gid = 7 PreSyn

Create NetCon on CPU where target exists. nc = pc.gid_connect(7, PostSyn)

Run using the idiom

pc.set_maxstep(10) stdinit() pc.psolve(tstop)

pc.set_maxstep() uses MPI_Allreduce to determine minimum delay.

any spike here must be delivered here

minimum delay exchange exchange

CPU 2 PreCell CPU 4 PostCell PostSyn NetCon

7 gid = 7 PreSyn

0 0 1 2 3 4 5

−40 2.875 (ms)

−80

0 2 4 6

CPU 2 PreCell CPU 4 PostCell PostSyn NetCon

7 gid = 7 PreSyn

n 1 gid 7 t 2.875 gid −−− t −−− t

0 2 4 6

Page 204 Copyright © 1998-2016 N.T. Carnevale and M.L. Hines, all rights reserved The NEURON Simulation Environment Didactic Presentations CPU 2 PreCell CPU 4 PostCell PostSyn NetCon . . 7 . n 0 gid = 7 cpu 1 PreSyn gid −−− t −−− gid −−− t −−− n 1 n 1 cpu 2 gid 7 MPI_Allgather gid 7 t 2.875 t 2.875 gid −−− gid −−− t −−− t t −−− n 0 cpu 3 0 2 4 6 . . .

CPU 2 PreCell CPU 4 PostCell PostSyn NetCon . . 7 . n 0 gid = 7 cpu 1 PreSyn gid −−− t −−− gid −−− t −−− n 1 cpu 2 gid 7 t 2.875 gid −−− t t −−− n 0 cpu 3 0 2 4 6 . . .

MPI_ISend − Two Phase, Two Subinterval Artificial Spiking Net Allgather DCMF_Multicast − Two Phase, Two Subinterval Blue Gene/P Record−Replay − One Subinterval Computation Time (includes queue) Argonne National Lab Strong Scaling 32 32 2M Cells 1/4M Cells 16 1k Conn/cell 16 10k Conn/cell

8 8 Runtime (sec) Runtime Runtime (sec) Runtime 4 4

2 2

1 1

0.5 0.5 8 16 32 64 128 8 16 32 64 128 K processors K processors Weak Scaling 30 30

20 20

1k Conn/cell 10k Conn/cell

Runtime (sec) Runtime (sec) Runtime

10 10

2M cells 32M cells 1/4M cells 4M cells 0 0 8 16 32 64 128 8 16 32 64 128 K processors K processors

256 point process artificial cell spike compression + bin queue 64 + self events not on queue

sec

16 262144

4 65536 runtime computation ideal

1 32 128 512 2048 8192 #CPU Each cell fires randomly every 10 to 20 ms. tstop = 200(ms) 65K cells, 1000 random connections per cell delay = 1(ms) 256K cells, 10,000 random connections per cell weight = 0

Gap Junction Specification Continuous Voltage Exchange pc.source_var(&source_var, sgid) pc.target_var(&target_var, sgid) pc.source_var(&s1.v(x1), 1) s1.v(x1) sgid pc.target_var(&g2.vgap, 1) 1 g2.vgap

s1 { g1 = new HalfGap(x1) } s2 { g2 = new HalfGap(x2) } g1.vgap sgid pc.target_var(&g1.vgap, 2) 2 s2.v(x2) pc.source_var(&s2.v(x2), 2) gap.mod NEURON { ASSIGNED { POINT_PROCESS HalfGap v (millivolt) ELECTRODE_CURRENT i vgap (millivolt) RANGE r, i, vgap i (nanoamp) } } PARAMETER { r = 1e9 (megohm) } CURRENT { i = (vgap − v)/r }

2068 2.4 GHz Opteron Processors 140

70 1024 0 0 50 100 150 200

Traub et. al. (2005) J. Neurophysiol 93: 2194 A single column thalamocortical network model 256 exhibiting gamma oscillations, sleep spindles and epileptogenic bursts. Run time

64 Ideal run time Spike exchange time (s) Mean, max, min Computation time 16 8516 Mean, max, min variable transfer time 5954

3560 cells 14 types 4 3500 gap junctions 5,596,810 equations 73,465 spikes 1,122,520 connections 19,844,187 delivered 1

25 50 100 200 400 800 #CPU

1024 80 Traub 60 #Cells 356 Cells 40

256 20

0 s 2000 3000 4000 5000 Complexity 64

Runtime 4058 pieces 26.1 Computation time 18.7 16 Whole cell balance 13.2 Multisplit, No Gap Junctions Multisplit, With Gap Junctions 4 32 128 512 2048 CPUs

d a d a d a d a d a b d a d a b d a d a b d a b d a b d a b d a b d a b d a b d a b d a b d a b d a b d a b d b d a b d a b d a b d b d a b d b d a b d b d a b d b d a b d b d a b d

569

#CPU Runtime (s)

1 54.8

3 22.2 19.5 18.3 564 expected ideal

467

0 0 500 1000 1500

−40

−80 De Schutter & Bower (1994) J. Neurophysiol 71:375 Ported to NEURON by Jenny Davie, Volker Steuber, and Arnd Roth.

200 16 Pieces 4 CPU 150 #comp

100

0 0 4 8 12 16 Piece

401 395 400 404 400

#comp

300

Time (s) 200 CPU Computation Exchange 0 13.82 0.56 Runtime(s) 16 pieces, 1 cpu 55.0 1 13.35 1.03 100 wholecell, 1 cpu 56.2 2 13.47 0.90 16 pieces, 4 cpu 14.4 0 1 2 3 4 3 13.56 0.82 CPU

200 200 16694 Pieces

150 10000 cells 150 # #

100 100

50 50

0 0 0 10000 20000 30000 40000 50000 0 10000 Complexity Complexity

2048 Runtime, Whole Cell Runtime, Multisplit 1024 Average Computation time

512 s

256

128

64 1024 2048 4096 8192 CPUs

Results must be independent of Number of processors Distribution of cells

What about RANDOM? Reproducible Independent Restartable

Associate a random stream with a cell.

Use cryptographic transformation of several integers. run number stream number (cell gid) stream pick index

10000

8000 Cell ID out.spk (58,858 lines) 6000 3.975 8050 3.975 8621 4000 ... 999.125 4632

2000

0 0 200 400 600 800 1000 t (ms)

Line# spike(ms) gid 40 8548: 107.25 1 18501: 488.625 1 mV soma 27276: 594.25 1 51470: 913.475 1 0 0 200 400 600 800 1000 ms

cAD_1 −40

−80 −40

−50

40 tmgExSyn −60 8 tmgInhSyn 39 distinct sources

−70 570 590 ms 610 630

Debugging

1) GID and time of first spike difference. 2) All spikes delivered to synapses of that Cell? 3) When and what is the first state difference?

The NEURON Simulation Environment Slide 1

Initialization, broadly speaking:

We want to get the same result every time we click on Init & Run, no matter what we did before

Note: this presentation explicitly omits details of initialization of ionic concentrations and equilibrium potentials

The NEURON Simulation Environment Slide 2

Initialization should assign values at t = 0 for membrane potential gating states ionic concentrations chemical kinetic states voltage across capacitors in linear circuits internal states of op amps random number generators and properly configure event queues vector record and play counters

The NEURON Simulation Environment Slide 3

NEURON’s finitialize() " sets t = 0 " clears event queue " sets up internal data structures that depend on topology and geometry " initializes Vector.play controller " delivers events whose delivery time is 0 " if finitialize was called with v_init argument, sets v in all compartments to v_init " calls INITIAL block of every inserted mechanism in every segment " if extracellular is used, sets vext to 0 " initializes ions; calculates equilibrium potentials if necessary " initializes mechanisms that WRITE ion concentrations; recalcs equilib potentials as needed " calls all other INITIAL blocks " initializes LinearMechanism states " calls INITIAL blocks inside NET_RECEIVE blocks; if this spawns network events, delivers any whose delay is 0 to their target NET_RECEIVE blocks " if fixed time step integrator is used, calls all BREAKPOINT blocks " initializes adaptive integrator (if being used) " intializes any cvode.record and vector.record recordings

The NEURON Simulation Environment Slide 4

Default initialization: the standard run library nrn/share/nrn/lib/hoc/stdrun.hoc (MSWin: c:\nrn\lib\hoc\stdrun.hoc)

stdinit() Called when you click on Init or Init & Run in the RunControl or enter a new value for v_init in the Init button’s field editor

proc stdinit() { realtime=0 // "run time" in seconds startsw() // initialize run time stopwatch setdt() init() initPlot() }

The NEURON Simulation Environment Slide 5

init() Most customizations are made here

proc init() { finitialize(v_init) // User−specified customizations go here. // If this invalidates the initialization of // variable dt integration and vector recording, // uncomment the following code. /* if (cvode.active()) { cvode.re_init() } else { fcurrent() } frecord_init() */ }

The NEURON Simulation Environment Slide 6

INITIAL blocks in NMODL

HH−like mechanisms

PROCEDURE rates(v(mv)) { minf = alpha(v)/(alpha(v) + beta(v)) . . . } . . . INITIAL { rates(v) m = minf . . . }

The NEURON Simulation Environment Slide 7

Kinetic schemes

INITIAL { SOLVE scheme METHOD steadystate } e.g. NEURON { USEION k READ ek WRITE ik } STATE { c1 c2 o } INITIAL { SOLVE scheme METHOD steadystate } BREAKPOINT { SOLVE scheme METHOD sparse ik = gbar*o*(v − ek) } KINETIC scheme { rates(v) : calculate the 4 k rates. ~ c1 <−> c2 (k12, k21) ~ c2 <−> o ( k2o, ko2) }

The NEURON Simulation Environment Slide 8

Default initialization of STATEs

Use state0, e.g. PARAMETER { state0 = 1 } or alternative syntax STATE { state START 1 } It’s best to be explicit INITIAL { m = m0 h = h0 } To make them visible from hoc NEURON { GLOBAL m0 RANGE h0 }

The NEURON Simulation Environment Slide 9

Typical custom initializations

Steady state unperturbed system system under constant voltage or current clamp

Defined starting point on a trajectory of an oscillating or chaotic system

Adjust parameters to meet some condition

How? Use a custom init() procedure. Load after the standard library, so it won’t be overwritten.

The NEURON Simulation Environment Slide 10

Initializing to steady state "Travel into the past," take large steps with implicit Euler, then return to the present. proc init() { local dtsav, temp finitialize(v_init) t = −1e10 dtsav = dt dt = 1e9 // if cvode is on, turn it off to do large fixed step temp = cvode.active() if (temp!=0) { cvode.active(0) } while (t<−1e9) { fadvance() } // restore cvode if necessary if (temp!=0) { cvode.active(1) } dt = dtsav t = 0 if (cvode.active()) { cvode.re_init() } else { fcurrent() } frecord_init() }

The NEURON Simulation Environment Slide 11

Initializing to a desired state Especially useful for oscillating or chaotic models.

Run a "warmup simulation," then save all states objref svstate, f svstate = new SaveState() svstate.save()

If desired, write state info to a file for future use f = new File("states.dat") svstate.fwrite(f) To read from a file objref svstate, f svstate = new SaveState() f = new File("states.dat") svstate.fread(f)

The NEURON Simulation Environment Slide 12

Initializing to a desired state continued

A custom init() that restores saved states proc init() { finitialize(v_init) svstate.restore() t = 0 // t is one of the "states" if (cvode.active()) { cvode.re_init() } else { fcurrent() } frecord_init() }

The NEURON Simulation Environment Slide 13

Initializing to a particular resting potential

One approach: adjust the leakage equilibrium potential so that leakage current balances the other ionic currents when the cell is at the desired resting potential Example: for a single compartment model with hh, proc init() { finitialize(v_init) el_hh = (ina + ik + gl_hh*v)/gl_hh if (cvode.active()) { cvode.re_init() } else { fcurrent() } frecord_init() }

The NEURON Simulation Environment Slide 14

Alternative strategy: add a mechanism that injects a constant current to balance the other currents. Example: NEURON { SUFFIX constant NONSPECIFIC_CURRENT i RANGE i, ic } UNITS { (mA) = (milliamp) } PARAMETER { ic = 0 (mA/cm2) } ASSIGNED { i (mA/cm2) } BREAKPOINT { i = ic } This needs a different custom init()

The NEURON Simulation Environment Slide 15

Custom init() to use with constant current mechanism: proc init() { finitialize(−65) ic_constant = −(ina + ik + il_hh) if (cvode.active()) { cvode.re_init() } else { fcurrent() } frecord_init() }

Neuroscience Gateway: Enabling Computational Neuroscience using High Performance Computing

Amit Majumdar, Subhashini Sivagnanam, Kenneth Yoshimoto San Diego Supercomputer Center, UCSD Ted Carnevale, Michael Hines Yale School of Medicine Angus Silver, Padraig Gleeson University College London

US NSF Awards: DBI #1458840, #1458495, #1146949; #1146830 UK BBSRC Award: BB/N005236/1

2016 Summer NEURON Course HHMI | Janelia Research Campus, Ashburn, Virginia

Motivation for NSG

Advances in Computational Neuroscience: • Increase in Modeling papers (J. of Neurophysiology, J. of Neuroscience) e.g. NEURON - used in >1600 publications as of June, 2015 • New Journals (Neural Computation, J. of Computational Neuroscience) • Trend in research proposals to NSF, NIH – Modeling a big part • Evolution and refinement of computational neuroscience tools NEURON, MCell, GENESIS, MOOSE, NEST, PyNN, Brian etc

Advances in Cyberinfrastructure (CI): • High Performance Computing(HPC):10s–100s terafloppetaflopexascale • Faster Interconnects: Infiniband, proprietary (Cray, IBM etc.) • Next generation data technologies: Data analytics software, Hadoop • Software for complex workflows

HPC Challenges for Computational Neuroscience • Modeling projects start “small” and many are forced to stay “small” • Rapid growth in development of complex neuronal network models, parameter sweep estimations require HPC and CI • Only few neuroscientists, in the world, have access to large scale HPC • Barriers of entry to HPC • Write peer-reviewed proposals for computer time • Understand HPC machines, policies, complex OS/software • Install and benchmark complex tools on HPC resources • Understand and managing multiple remote authentication systems • Figuring out data transfer, management, storage issues

The Neuroscience Gateway (NSG)

1. Easy user interface – providing easy model upload, running of codes 2. Provide neuronal simulation tools – widely used by computational neuroscientists 3. Ability to easily get to the results, download results NSG catalyzes and democratizes computational neuroscience research for everybody including researchers and students from underrepresented minority institutions

How to get an account on NSG

Fill out form at https://www.nsgportal.org/reg/reg.php

• Account creation process is used to verify users • Users submit brief contact and technical information • Account is usually set up within 24 hrs. • Users added to NSG email list – very infrequent news post

Users View of NSG Screen after portal log in

Users View of NSG – continued Can clone task - reproducibility and data validation

Input/output Users View of NSG - continued • Zipped format Single file ; Save folders; Multiple data sets

Available software/tools on NSG

Set parameters Users View of NSG – continued

Monitor task progressUsers View of NSG – continued For debug send this #

Users View of NSG – continued Output available

Users View of NSG – continued Download output

Email Notification upon Completion

Usage • Around 350 registered users • 2013 : 187,000 supercomputer core hours (Comet equivalent SUs) • 2014 :600,000 SUs • 2015 : 1,844,000 SUs • 2016 (acquired) : 3,700,000 SUs

Machine Cores/n Memory/n Total Total flop Notes ode ode #cores Stampede 2.7 GHz 16 32 GB 102,400 9600 TF TACC Intel SandyBridge (with Intel MIC) Comet 2.5 GHz Intel 24 128 GB 47,616 2000 TF 1728 cores max Haswell job size, SDSC

Points to Remember •Any researcher, student from anywhere can use NSG - free •Each user starts out with 20K hours on the portal based on their research and needs •Further allocation can be made also based on needs •Input file upload must be in Zipped format (with a single top directory) • work in progress to support other formats •Automatic deletion of user files based on time length of inactivity •Bundling of serial jobs (currently only for the Brian tool) • Users requiring large individual allocation •We will work with you to write your XSEDE allocation proposal • Use NSG with your individual allocation • You can request new software, pipelines to be provided via NSG 17

NSG REST Interface

NSG user interface Browser HPC Programmatic access interface resource 1

NSG HPC resource 2

RESTful HPC web resource3 services

Summary • 2nd NSF grant NSG-R will allow programmatic access from community projects OSB, NIF, OpenWorm, ModelDB etc or from individual users. • Collaboration with researchers from UK, Japan, India, Germany, France • High School, undergradute student participation • Workshops at Society for Neuroscience annual meetings, Computational NeuroScience annual meetings, MSI institutes • NSG presented at NEURON summer course every year • Please cite us if you use NSG (http://www.nsgportal.org/citation.html)S Sivagnanam, A Majumdar, K Yoshimoto, V Astakhov, A Bandrowski, M. E. Martone, and N. T. Carnevale. Introducing the Neuroscience Gateway, IWSG, volume 993 of CEUR Workshop Proceedings, CEUR-WS.org, 2013 • Please let us know of your presentations, publications so that we can include in reports

Our many thanks to… CNS tool Developers • Michael Hines, Yale SDSC • Tom Morse, Yale • Mark Miller • Padraig Gleeson, UK • Mahidhar Tatineni • Angus Silver, UK • Jerry Greenberg • Marcel Stimberg, France • Nicole Wolter • Romain Brette, France • SDSC Cloud team • Dan Goodman, UK • Markus Diesmann, Germany • Jochen Eppler, Germany XSEDE • Andrew Davison, France • Nancy Wilkins-Diehr • Subhasis Ray, NIH • Terri Schwartz • Upi Bhalla, India • Marlon Pierce • Dave Beeman, Colorado • Suresh Marru • Bruce Fischi, Harvard • XSEDE Security team • Pramod Khumbar, EPFL

,NA=PEJC =J@ QOEJC 4.:754 IK@AHO

:OA DK?" 6UPDKJ" =J@%KN /:1 PK OLA?EBU( +EKHKCE?=H LNKLANPEAO##=J=PKIU" >EKLDUOE?O 1JOPNQIAJP=PEKJ##OECJ=H OKQN?AO =J@ NA?KN@EJC :OAN EJPANB=?A##L=N=IAPAN L=JAHO" CN=LDO 8EIQH=PEKJ ?KJPNKH##@P" POPKL" EJPACN=PEKJ IAPDK@ 0EJP( GAAL PDAOA OAL=N=PA BNKI A=?D KPDAN BKN I=TEIQI ?H=NEPU =J@ PK O=RA ABBKNP ;ANEBU( ,HKOA I=P?D PK ?KJ?ALPQ=H IK@AH) 4QIANE?=H =??QN=?U =@AMQ=PA) OL=PE=H CNE@" EJPACN=PEKJ PEIA OPAL KN ANNKN ?NEPANEKJ!

8LA?EBUEJC >EKHKCE?=H LNKLANPEAO

9KLKHKCU >N=J?DEJC L=PPANJ! /AKIAPNU @E=IAPAN" HAJCPD! =J@ +EKLDUOE?O IAI>N=JA ?=L=?EP=J?A" EKJ ?D=JJAHO" LQILO $ $ $ ! ,KJJA?PEKJO >APSAAJ ?AHHO OUJ=LOAO" C=L FQJ?PEKJO! $ $ $ a( a(+*#$(" !&)! *#a* 'a%!) )!()! $ $ $

+EKHKCE?=H LNKLANPEAO( PKLKHKCU

3=GA PDA LEA?AO OA?PEKJO! :L> 8LA?EBU PDA @AB=QHP OA?PEKJ :<<>KK *OOAI>HA PDA LEA?AO

.T=ILHA

%% F:D> LA> IB><>K :L> KHF:" :PHG" =>G=JBL>7)8

%% KI>?:MEL K>KK KHF:

%% :KK>F;E> LA>F G=JBL>7B8 &!" KHF: '! T

+EKHKCE?=H LNKLANPEAO( CAKIAPNU =J@ >EKLDUOE?O

,KIL=NPIAJP=HEV=PEKJ GK>@ /AKIAPNU 4" =B:F +EKLDUOE?=H LNKLANPEAO -AJOEPU IA?D=JEOIO( BGK>JL .T=ILHAO( EKJ ?D=JJAHO @EOPNE>QPA@ KRAN PDA ?AHH OQNB=?A" LQILO" EKJ =??QIQH=PEKJ" >QBBANO

KHF: S GK>@ / ' 4 / +& %% 7MF8 E>G@LA =B:F / +& %% 7MF8 =B:F>L>J BGK>JL AA %% 2H=@DBG#2MPE>Q GLK T

:PHG S GK>@ / (' %% H== KH : GH=> BK :L &$+ 4 / '&&& =B:F / ' BGK>JL AA T

?HJ B/&"( =>G=JBL>7B8 S GK>@ / + 4 / (&& =B:F &-'! / '&-) %% L:I>J BGK>JL I:K %% I:KKBN> F>F;J:G> T

?HJ:EE 5: / ,& %% 7HAF

7=JCA R=NE=>HAO ;=NU ?KJPEJQKQOHU EJ OL=?A =HKJC PDA HAJCPD KB = OA?PEKJ .T=ILHAO( N" HAO 6ANP=EJ PK =J AJPENA OA?PEKJ .T=ILHAO( 5: ?UPKLH=OIE? NAOEOPEREPU!" 4" GK>@ /HK>=H R=NE=>HAO 8=IA =?NKOO =HH OA?PEKJO .T=ILHAO( <>EKBMK " L =J@ =L BETA@ PEIA OPAL EJPACN=PEKJ!

1JOPNQIAJP=PEKJ 9DEO IK@AH JAA@O =J AHA?PNK@A =P PDA OKI= PK EJFA?P OPEIQH=PEJC ?QNNAJP$

.T=ILHAO KB "LKEJP LNK?AOOAO"( ?QNNAJP ?H=IL" RKHP=CA ?H=IL" OUJ=LOA

5>FA?P OUJP=T

H;CJ>? KLBF %% :LL: H? KHF: KHF: KLBF / G>O 31E:FI &$+!

KLBF$=>E / ' %% 7FK8 =>E:Q KLBF$=MJ / &$' %% 7FK8 =MJ:LBHG KLBF$:FI / ,& %% 7G08 :FIEBLM=>

8EIQH=PEKJ ?KJPNKH .T=ILHA &( IEJEI=HEOP BKN BETA@ @P OEIQH=PEKJO ?BGBLB:EBR> #,+! %% BGBLB:EBR> N" KL:L> N:JB:;E>K" LBF> LKLHI / + =L / &$&(+ IJH< KBFME:L> ! S IJBGL L" N &$+! %% KHF: BK =>?:MEL K> L . LKLHI! S ?:=N:G<> ! %% :=N:G<> KHEMLBHG ;Q =L %% ?MG HJ IEHL J>KMELK" >$@$ IJBGL L" N &$+! %% KL:L>F>GLK LH FH=>E I:J:F>L>JK T T .T=ILHA '( QOEJC PDA OP=J@=N@ NQJ OUOPAI N9BGBL / #,+ %% 6F :L L//& LKLHI / + %% 7FK8 KL>IK9I>J9FK / *& %% IHBGLK IEHLL>=%FK =L / &$&(+ %% 7FK8 BGL>@J:LBHG LBF> KL>I K>L=L ! %% >GKMJ>K LA:L : OAHE> GMF;>J H? =LK %% OBEE ?BL BGLH '%KL>IK9I>J9FK JMG ! %% BGBLB:EBR>" LA>G JMG KBFME:LBHG

6NKCN=I KNC=JEV=PEKJ

FH=>EKI><$AH< "RENPQ=H KNC=JEOI" PKLKHKCU" CAKIAPNU" >EKLDUOE?O JB@$K>K "RENPQ=H ATLANEIAJP=H NEC" ?H=ILO" CN=LDO" NQJ ?KJPNKH BGBL$AH< "=@IEJEOPN=PERA SN=LLAN" EH:=9?BE> "GJG@MB$AH<"! EH:=9?BE> "FH=>EKI><$AH<"! EH:=9?BE> "JB@$K>K"!

-ARAHKL%@A>QC "RENPQ=H KNC=JEOI" DK?" 6UPDKJ" 435-2" /:1 ,AHH+QEH@AN" ,D=JJJAH +QEH@AN" 4APSKNG +QEH@AN! EJ SD=PARAN ?KI>EJ=PEKJ 3K@AH ;EAS 1PAN=PERA ?U?HA KB EJ?NAIAJP=H NAREOEKJ =J@ PAOPEJC :OA 4.:7543=EJ3AJQ PK ?QOPKIEVA EJPANB=?A =PP=?D OUJ=LOAO =J@ AHA?PNK@AO OAP QL CN=LDO =J@ NQJ ?KJPNKH OLA?EBU EJPACN=PEKJ IAPDK@

inside Ca

KINETIC pmp { ~ cabulk <−> cam (1/tau, 1/tau) Pump ~ cam + pump <−> capump (k1, k2) ~ capump <−> cao + pump (k3, k4) ica_pmp = 2*FARADAY*(f_flux − b_flux)

~ cam << −(ica) : there is a problem here

COMPARTMENT width {cam} : volume has dimensions of (um) COMPARTMENT 1 {pump capump} : area is dimensionless COMPARTMENT 1(m) {cao cabulk} }

Declarations for capump.mod NEURON { SUFFIX capmp USEION ca READ cao, ica, cai WRITE cai, ica RANGE tau, width, cabulk, ica, pump0 }

UNITS { (um) = (micron) (molar) = (1/liter) (mM) = (millimolar) (uM) = (micromolar) (mA) = (milliamp) (mol) = (1) FARADAY = (faraday) (coulomb) }

Copyright © 1998-2016 N.T. Carnevale and M.L. Hines, all rights reserved Page 239 Didactic Presentations The NEURON Simulation Environment Declarations for capump.mod PARAMETER { ASSIGNED { width = 0.1 (um) cao (mM) : 10 tau = 1 (ms) cai (mM) : 1e−3 k1 = 5e8 (/mM−s) ica (mA/cm2) k2 = 0.25e6 (/s) ica_pmp (mA/cm2) k3 = 0.5e3 (/s) ica_pmp_last (mA/cm2) k4 = 5e0 (/mM−s) } cabulk = 0.1 (uM) pump0 = 3e−14 (mol/cm2) }

STATE { cam (uM) <1e−6> pump (mol/cm2) <1e−16> capump (mol/cm2) <1e−16> } Equations for capump.mod INITIAL { BREAKPOINT { ica = 0 ica_pmp = 0 SOLVE pmp METHOD sparse ica_pmp_last = 0 ica_pmp_last = ica_pmp SOLVE pmp STEADYSTATE sparse ica = ica_pmp } } KINETIC pmp { ~ cabulk <−> cam (width/tau, width/tau) ~ cam + pump <−> capump ((1e7)*k1, (1e10)*k2) ~ capump <−> cao + pump ((1e10)*k3, (1e10)*k4) ica_pmp = (1e−7)*2*FARADAY*(f_flux − b_flux) : ica_pmp_last vs ica_pmp needed because of STEADYSTATE ~ cam << (−(ica − ica_pmp_last)/(2*FARADAY)*(1e7)) CONSERVE pump + capump = (1e13)*pump0 COMPARTMENT width {cam} : volume has dimensions of um COMPARTMENT (1e13) {pump capump}: area is dimensionless COMPARTMENT 1(um) {cabulk} COMPARTMENT (1e3)*1(um) {cao} cai = (0.001)*cam }

// define a replacement for the stdrun.hoc version of proc init() { finitialize(v_init) fcurrent() } // that lets you escape from the tyranny // of the steady state initialization of cai. proc init() { local savtau // will initialize cai to cabulk savtau = tau_capmp tau_capmp = 1e−6 finitialize(v_init) tau_capmp = savtau fcurrent() if (cvode.active()) { cvode.re_init() } }

Graph[0] Move Text x 64 : 76 y -0.00075 : 0.00825 soma.cai( 0.5 ) NEURON Main Menu 0.0075 File Edit Build Tools Graph Vector Window Graph[0] Move Text x -10 : 110 y -0.0101 : 0.1111 soma.cai( 0.5 ) 0.1 0.005 soma(0 - 1) (Parameters) 0.08 soma(0 - 1) (Parameters) 0.0025 nseg = 1 0.06 L (um) 3.1831 0.04 0 Ra (ohm-cm) 35.4 65 67 69 71 73 75 diam (um) 10 0.02 cm (uF/cm2) 1 0 width_capmp (um) 0.1 0 20 40 60 80 100 tau_capmp (ms) 1e+05 Graph[1] Move Text x -10 : 110 y -0.0003 : 0.0033 cabulk_capmp (uM) 100 pump0_capmp(mol/cm2) 3e-14 0.003

soma.ica( 0.5 ) Insert/Remove Mechanisms 0.002 soma pas hh 0.001 capmp capmp2 0 0 20 40 60 80 100

Grapher

Plot Erase All

Indep Begin -4 Indep End 2 Steps 20

Independent Var x

X-expr x

Generator cabulk_capmp=10^x init()

Two kinds of parallel problems

A simulation run takes about a second.

Want to do 1000’s of them, varying a dozen or so parameters.

A simulation of a large network takes hours.

Want to spread the problem over several machines, each machine handling a subset of the neurons in the network Serial Parallel s = 0 s = 0 for i = 1, 10 { for i = 1, 10 { s += f(i) pc.submit("f", i) } } while (pc.working) { s += pc.retval } Goals Keep all the machines as busy as possible. If there is only one machine the parallel program should run as fast as the serial program. Things asked for earlier tend to get done earlier.

Assumptions Workstation cluster − 1, 3, 15, 100 machines. Wide variety of machine speeds. Sending a byte is much slower than executing a hoc statement.

Domain Very coarse grain parallelization. || loop

f|| loop f f || loop

g g g g g

NEURON’s style A bulletin board . . . on top of MPI.

Launching a parallel program objref pc pc = new ParallelContext()

// setup which is exactly // the same on every machine // i.e. declaration of all // functions, procedures, // setup of neurons

pc.runworker

// the master scatters tasks // onto the bulletin board // and gathers results

pc.done

Example.hoc objref pc pc = new ParallelContext()

func f() {local s, i s = 0 for i=1,100000 { s += $1 } return s }

{pc.runworker()}

runtime = startsw() s = 0 for i=1,10 { pc.submit("f", i) } while (pc.working()) { s += pc.retval } print "sum = ", s print "runtime ", startsw() − runtime {pc.done()} quit()

$ mpiexec −n 1 nrniv −mpi example.hoc numprocs=1 NEURON −− VERSION 7.2 (428:986821b56b98) 2010−03−17 ... sum = 5500000 runtime 0.079999924

$ mpiexec −n 4 nrniv −mpi example.hoc numprocs=4 NEURON −− VERSION 7.2 (428:986821b56b98) 2010−03−17 ... sum = 5500000 runtime 0.019999981

Context and Communication

NEURON

g(i) neuron context g context

f(j)

post Bulletin board

take look Bulletin board look_take

context("stmt") : stmt executed on every worker

Context and Communication

With Python

def f(arg1, arg2): ... return any_pickleable_object ... pc.submit(f, (arg1, arg2)) ... while pc.working(): r = pc.pyret()

>8EB?>"Y ZUURY MUX 4THR_YPY UM 8RLJZXPJHR CPNTHRPTN

;TV[Z HTK ZXHTYMLX PSVLKHTJLY

FURZHNL ZXHTYMLX XHZPU

(.6<59:8/+4 )(;79:8/+4

8RLJZXUZUTPJ ZXHTYMUXSHZPUT

RUN# (.6<59:8/+4 )(;79:8/+4 $ ( ( ( HRR HY M[TJZPUTY UM MXLW[LTJ_ HTK YVHJL

6RHYYPJHR 6HIRL DOLUX_ V(0) V(x) ;TMPTPZL J_RPTKLX PT ZOL YZLHK_ YZHZL0 0 x (#=$ 1 (#*$ L = ) = 1 VO_YPJHR KPYZHTJL 1 RLTNZO JUTYZHTZ 6RHYYPJHR LRLJZXUZUTPJ KPYZHTJL ) 1 RT ( #*$)(#=$ 1 =) YU HZZLT[HZPUT #(#=$ 1 (#*$)(#=$ 1 L) ;TZ[PZP\LR_ YPSVRL

@XUIRLSY

>L[XUTY HXL TUZ PTMPTPZL J_RPTKLXY( 4ZZLSVZLK MP^0 XLK[JL KLTKXPZPJ ZXLL ZU MPTPZL RLTNZO LW[P\HRLTZ J_RPTKLX ( # #=$ " JUYO % -3+990-+3 ) JUYO #%-3+990-+3 ) $

%-3+990-+3 " VO_YPJHR RLTNZO ) ) " = )

DOL IHK TL]Y HIU[Z ZOL LW[P\HRLTZ J_RPTKLX HVVXU^PSHZPUT

>LPZOLX PTZ[PZP\L TUX YPSVRL(

7LYZXU_Y YVHZPHR XLRHZPUTYOPVY HSUTN Y_THVZPJ PTV[ZY(

6RHYYPJHR LRLJZXUZUTPJ KPYZHTJL ) " = ) MUYZLXY JUTJLVZ[HR LXXUX I_ UIYJ[XPTN ZOL KPXLJZPUT'KLVLTKLTJL UM HZZLT[HZPUT PT MPTPZL YZX[JZ[XLY(

DOL NUUK TL]Y HIU[Z ZOL LW[P\HRLTZ J_RPTKLX HVVXU^PSHZPUT ;Z PYT"Z \HRPK $0./'024 "113,/2*.- %032) 7LTKXPZPJ LRLJZXPJHRR_ \HXPLY ]PKLR_ ZLXSPTHZPUTY LW[PKPYZHTZ MXUS YUSH 7PHSLZLXY J_RPTKXPJHR PXXLN[RHX 5XHTJO VUPTZY -), VU]LX X[RL TU -), -), .7 1 ..

DOL 8RLJZXUZUTPJ DXHTYMUXSHZPUT 4 ZXHTYMUXSHZPUT MXUS HTHZUSPJHR ZU LRLJZXUZUTPJ YVHJL ZOHZ

PY PTZ[PZP\L

PY LSVPXPJHRR_'IHYLK

SHQLY TU XLYZXPJZP\L HYY[SVZPUTY HIU[Z HTHZUS_

4THZUSPJHR YVHJL 8RLJZXUZUTPJ YVHJL SLHY[XL0 VO_YPJHR KPYZHTJL SLHY[XL0 RUN#HZZLT[HZPUT$

9U[TKHZPUT0 Z]U'VUXZ HTHR_YPY UM LRLJZXUZUT[Y :U] ]LRR KU YPNTHRY VXUVHNHZL3

V i V i V V A i j =V i /V j A j i =V j /V i

V j V j

( ( CPNTHR ZXHTYMLX PY KPXLJZPUT'KLVLTKLTZ0 #01 # 10

( $ $ ' 4ZZLT[HZPUT PKLTZPZPLY0 #01 1 # 10 # 01 " # 01

DOL 8RLJZXUZUTPJ DXHTYMUXSHZPUT 9[TJZPUTHR KLMPTPZPUT UM LRLJZXUZUTPJ KPYZHTJL # ! +.((&22'-3&2*.- YPSVRL& KPXLJZ XLRHZPUTYOPV ZU HZZLT[HZPUT ( ( ( ( KPXLJZPUT'KLVLTKLTZ0 %01 " RUN# #01 $& % 10 " RUN# #10 $& ( ( HTK PT NLTLXHR %01 % 10 PT HT PTMPTPZL J_RPTKLX& PY PKLTZPJHR ZU JRHYYPJHR LRLJZXUZUTPJ KPYZHTJL V V V HKKPZP\L U\LX H VHZO ]PZO JUTYZHTZ i j k KPXLJZPUT UM VXUVHNHZPUT L i j L j k ( ( ( #02 1 (0)(2 " #(0)(1$·#(1)(2$ " # 01 # 12 ∴ %02 1 % 01 + % 12

EYPTN ZOL 8RLJZXUZUTPJ DXHTYMUXSHZPUT 4Z H MXLW[LTJ_ UM PTZLXLYZ +( JUSV[ZL RUN#HZZLT[HZPUT$ ILZ]LLT H XLMLXLTJL VUPTZ HTK HRR UZOLX VUPTZY UM PTZLXLYZ ,( KPYVRH_ XLY[RZY NXHVOPJHRR_ #UVZPUTHR$

4 JUT\LTPLTZ XLMLXLTJL VUPTZ0 ZOL YUSH i i 6OHTNPTN ZOL XLMLXLTJL VUPTZ HMMLJZY UTR_ ZOL KPXLJZPUT UM YPNTHR MRU] UT ZOL KPXLJZ VHZO j j ILZ]LLT ZOL URK HTK TL] RUJHZPUTY(

DOL HZZLT[HZPUT PKLTZPZPLY NP\L [Y ZOL ZXHTYMUXS PKLTZPZPLY (05 " $ 6;: " ' 6;: HTK (6;: " $ 05 " ' 05

C_THVZPJ RUJHZPUT HTK Y_THVZPJ LMMPJHJ_

A0 GOHZ VXLKPJZY OU] @C@ HSVRPZ[KL HZ ZOL YUSH \HXPLY ]PZO Y_THVZPJ RUJHZPUT3 40 ;M Y_THVYLY HJZ RPQL \URZHNL YU[XJLY& ( #05 #\URZHNL HZZLT[HZPUT MXUS Y_THVYL ZU YUSH$ UX 29>5 '2 964+ #Y_THVYL ZU YUSH \URZHNL ZXHTYMLX XHZPU$

9XUS 9PN( + PT

(9>5 #:$ PY PTKLVLTKLTZ UM Y_THVZPJ RUJHZPUT HTK Y_THVYL ZU YUSH \URZHNL ZXHTYMLX XHZPU ( 29>5 '2 964+ 1 + ) #05 1 *- ) # *, % *-$ VXLKPJZY \HXPHZPUT UM YUSHZPJ @C@ HSVRPZ[KL ]PZO Y_THVZPJ RUJHZPUT(

CUSHZPJ @C@ VXLKPJZLK I_ CUSHZPJ @C@ NLTLXHZLK I_ \URZHNL ZXHTYMLX XHZPU JUTK[JZHTJL'JOHTNL Y_THVYL 29>5 '2 964+ 9XUS 9PN( . PT 5 '2 964+ MHPRY ZU VXLKPJZ ZOL XLRHZPUTYOPV ILZ]LLT YUSHZPJ @C@ HSVRPZ[KL HTK Y_THVZPJ RUJHZPUT( ,( C_THVYLY KU TUZ HJZ RPQL \URZHNL YU[XJLY( A+0 GOHZ KU Y_THVYLY HJZ RPQL3 A,0 GOHZ ]U[RK IL H ILZZLX VXLKPJZUX UM ZOL XLRHZPUTYOPV ILZ]LLT YUSHZPJ @C@ HTK Y_THVZPJ RUJHZPUT3

9XUS 9PN( + PT

$9>5 #:$ PY PTKLVLTKLTZ UM Y_THVZPJ RUJHZPUT HTK

ZXHTYMLX PSVLKHTJL *- VXLKPJZY ZOL \HXPHZPUT UM YUSHZPJ @C@ HSVRPZ[KL ]PZO Y_THVZPJ RUJHZPUT( =LZ"Y ZX_ PZ ( ( (

CUSHZPJ @C@ VXLKPJZLK I_ CUSHZPJ @C@ NLTLXHZLK I_ ZXHTYMLX PSVLKHTJL * JUTK[JZHTJL'JOHTNL Y_THVYL -

9XUS 9PN( . PT UXSHRP`LK ZXHTYMLX PSVLKHTJL * VXLKPJZY ZOL XLRHZPUTYOPV - ILZ]LLT YUSHZPJ @C@ HSVRPZ[KL HTK Y_THVZPJ RUJHZPUT( ,( C_THVYLY HJZ RPQL J[XXLTZ YU[XJLY(

9XUS 9PN( + PT

CUSH ZU Y_THVYL \URZHNL ZXHTYMLX XHZPU 2964+ '2 9>5 PY PKLTZPJHR ZU TUXSHRP`LK ZXHTYMLX PSVLKHTJL * ( - 964+ @XUUM0 2964+ '2 9>5 1 *- ! #*+ + * -$ 1 *- ! * & 964+ I[Z *& PY ZOL SH^PS[S ZXHTYMLX * ILZ]LLT HT_ RUJHZPUT HTK ZOL YUSH( DOLXLMUXL 2 1 * 964+ '2 9>5 -

8B? 1CH?;L -CL=OCN ,OCF>?L

/IL CHA GI>?FM NB;N B;P? FCH?;L =CL=OCN ?F?G?HNM ;H> G;S ;FMI CHPIFP? H?OLIHM -CL=OCN ?F?G?HNM CH=FO>? ALIOH># =OLL?HN PIFN;A? MIOL=?# 6# -# IJ ;GJ 5IN?HNC;F ;JJFC=;NCIHM CH=FO>? ?@@?=NM ;H> =IGJ?HM;NCIH I@ ?F?=NLI>? 6 - NQI$?F?=NLI>? PIFN;A? =F;GJ IBGC= ;H> HIHFCH?;L A;J DOH=NCIHM

'% ,LCHA OJ ; 1CH?;L -CL=OCN ,OCF>?L

3.9643 2;CH 2?HO & ,OCF> & 1CH?;L -CL=OCN

8B? 1CH?;L -CL=OCN ,OCF>?L

QCL? L?MCMNIL =;J;=CNIL CH>O=NIL PIFN;A? MIOL=? =OLL?HN MIOL=? ALIOH> IJ?L;NCIH;F ;GJFC@C?L CHNL;=?FFOF;L HI>? CHNL;$ ;H> ?RNL;=?FFOF;L HI>?M

+LL;HA?) MJ;QH =IGJIH?HNM -FC=E IH J;F?NN? ;H> >L;A IHNI =;HP;M

+LL;HA?) =IHH?=N =IGJIH?HNM

-FC=E ;H> >L;A NI IP?LF;J L?> =CL=F?M

,F;=E MKO;L? CM "MIF>?L DICHN"

5OFF ;J;LN NI

-FC=E ;H> >L;A NI H?Q FI=;NCIH

-FC=E IH ; F;

% % % NI =B;HA? CNM H;G?

-FC=E IH ; HI>? % % %

% % % NI F;

5;L;G?N?LM) HIH$MIOL=? ?F?G?HNM

-FC=E IH "5;L;G?N?LM"

5;L;G?N?LM) MCAH;F MIOL=?M

7IOL=? @!N" & ,

5;L;G?N?LM) MCAH;F MIOL=?M !'&)$&*#"

-IH@CAOL?>

7CGOF;N?) =L?;NCHA ; AL;JB

3?Q 0L;JB

7CGOF;N?) MJ?=C@SCHA QB;N NI JFIN

5FIN:B;N* & +a($a %#_%a #%

7CGOF;N?) MCGOF;NCIH L?MOFNM

+@N?L GCHIL =IMG?NC= =B;HA?M

5;N=B =F;GJ QCNB ?F?=NLI>? 6 ;H> -

Ve Re1 Vc Re2 Vm Iclamp Ce

soma(0.5)

3.9643 >?GI) >SH;GC= =F;GJ