Queues & : a Tale of Modeling and Experiment

Urbashi Mitra Ming Hsieh Department of Electrical Engineering Department of Computer Science Viterbi School of Engineering University of Southern California Thanks to ONR N00014-15-1-2550, NSF CPS-1446901, NSF CCF-1410009, [email protected] NSF CNS-1213128 , AFOSR FA9550-12-1-0215, NSF CCF-1117896 Bacteria and RF analogies?

5

for their discoveries concerning genec control of enzyme and virus synthesis Bacterial Communies

Analyze electron transport in bacterial cables Develop models for quorum sensing Why bacteria?

9

¨ Unicellular organisms

¨ One of the earliest forms of life (3-4 billion years ago)

¨ Aggregate biomass larger than all other plants/animals combined

¨ Relevant for bioremediaon, plant growth, human and animal digeson, global elemental cycles including the carbon-cycle, and water treatment

Escherichia coli cell Not as simple as they seem..

10

E. Coli genome

hp://ecocyc.org http://ecocyc.org What is communicaon?

The fundamental problem of communicaon is that of reproducing at one point either exactly or approximately a message selected at another point.

Claude Shannon, A Mathemacal Theory of Communicaon, 1948

TX RX What is capacity for bacteria?

13 (MFC) q Bacteria as “devices” that sense the environment, esmate, react q Idea: tools from informaon theory and communicaons to model & predict bacterial networks: § Interacons, dynamics, formaon § Do bacteria realize opmal control strategies? § E.g.: opmize MFC growth

) W-COMM INSPIRED ELECTRODE MODEL How to achieve the sweet spot?

14

¨ Microbes on a 2D electrode ¨ Populaon grows as a funcon of me q What happens at 15h? q How are bacteria interacng?

q What is electron transport?

From Ken Nealson group, USC, Earth Sciences e- transport chain (ETC)

15

q Fundamental (and well known) process in cellular respiraon

q e- flow creates proton gradient to pump synthesis of (ATP, cell’s energy currency) q Cells need a constant electron flow to produce energy (ATP) e- transport

16

q e- transport over cm-lengths Ø bacterial nanowires, biofilms, bacterial cables

mul-hop communicaon ``network’’ from e- donor (ED) to e- acceptor (EA)

Bacterial cable (1D), Source: [Reguera,Nature’12] e- transport

17

q e- transport over cm-lengths Ø bacterial nanowires, biofilms, bacterial cables

q Use communicaon strategies to opmize e- transfer

q Capacity as proxy for maximal e- transfer

q New dimension (& new models) informaon-energy coupling § Capacity boleneck, = vs wireless communicaons 6 § Instead, local interacons & Bacterial cable (1D), Source: [Reguera,Nature’12] energy coupling, cells may die Stochasc queuing model

19

q BEFORE: modeling via determinisc ODE/PDE • E.g., [Klapper&Dockery,’10], [Picioreanua et al.,’07], [Marcus et al.,’07] • Detailed master equaons & chemical reacons à huge complexity

q NOW: abstracon à stochasc queuing model inspired by W-COM • Chemical/electron concentraons à queues • Small-scale, local interacons as nuisances à stochasc model • Sll high complexity, but tools for complexity reducon from W-COM can be exploited Stochasc queuing model

20

q External high energy membrane: e- from previous cell

q e- parcipate in to produce ATP

e- from previous cell Stochasc queuing model

21

q Internal e- carrier pool: e- from external ED q e- parcipate in electron transport chain to produce ATP

Niconamide adenine dinucleode (NADH)

electro-chemical energy from environment (NADH pool) Stochasc queuing model

22

q ATP pool: ATP molecules produced via e- transport

q ATP consumed to sustain cell

ATP molecules generated via ET Stochasc queuing model

23

q External low energy membrane carries e- aer ETC. 2 paths: • Reducon to EA • e- transfer to next cell or electrode

e- then transferred to next cell Stochasc queuing model

24 q Molecular diffusion & e- transfer coexist q ATP level = energec state of the cell Ø Energy dimension: ATP low à cell may die Ø Informaon dimension: ATP level too high à clogging

BACTERIAL CABLE 23

ADP molecules in the cell is conserved, a higher ATP level corresponds to a lower ADP level. Therefore, as there is more ATP available in the cell, there are less ADP molecules available for ATP synthase to produce additional ATP molecules, which, in turn, results23 in reduction of the ATP production rate. Finally, the larger the ED concentration, the larger the ATP consumption ADP molecules in the cell is conserved, a higher ATP level corresponds to a lower ADP level. rate µ (s (t); s (t)). This is shown to be true experimentally, for instance in [25]. The reason Therefore, as there is more ATPAT PavailableI E in the cell, there are less ADP molecules available for behind this correlation is that cellular operations that consume ATP (e.g., ATP-ases) are directly ATP synthase to produce additional ATP molecules, which, in turn, results in reduction of the µ µ ATP production rate. Finally,regulated the larger by the the ED ED concentration. concentration, Note the that largerCH the, ATPCH and consumptionAT P further need to satisfy the constraints listed in Sec. II-A. We assume that all the cells are alive throughout the experiment, rate µAT P (sI (t); sE(t)). This is shown to be true experimentally, for instance in [25]. The reason behind this correlation isand that set cellular the death operations rate (s thatI (t); consumesE(t)) = ATP 0. (e.g., ATP-ases) are directly We define the parameter vector x =[, ⇢, ⇣, ], which is estimated via Maximum Likelihood regulated by the ED concentration. Note that CH, µCH and µAT P further need to satisfy the constraints listed in Sec.(ML) II-A. in We the assume next thatsection. all the Therefore, cells are the alive flow throughout matrix A the, defined experiment, in (15), is a linear function

of the entries of x. We write such a dependence as A(x, sE). Similarly, from (18), we write and set the death rate (sI (t); sE(t)) = 0. We define the parameterP( vectorx, sE)=x =[I +,⇢,A⇣(,x,],sE which). is estimated via Maximum Likelihood (ML) in the next section. Therefore, the flow matrix A, defined in (15), is a linear function B. Maximum Likelihood estimate of x of the entries of x. We write such a dependence as A(x, sE). Similarly, from (18), we write

P (x, s )=I + A(x, s For) a given time series (yk,tk),k=0, 1,...,N , and the piecewise constant profile of the E E . { } external state sE(t), in this section we design a ML estimator of x. Since the initial distribution ⇡0 B. Maximum Likelihoodis estimate unknown, of wex also estimate it jointly with x. Note that, since the death rate is zero, the entries of Isolated cell: T T For a given time series⇡0 (needy ,t to),k sum=0 to, one,1,...,Ni.e., 1, and⇡0 =1 the. piecewise Moreover, constant we further profile enforce of the the constraints ⇡0 Z = y0, { k k } t y external state sE(t), in thisi.e. section, theexperimental validaon expected we design values a ML of estimator the NADH of x. and Since ATP the poolsinitial atdistribution time 0 equal⇡0 the measurement 0. 25 T is unknown, we also estimateTherefore, it jointly we with havex the. Note linear that, equality since the constraint death rate⇡0 is[Z zero,, 1]=[ they entries0, 1], and of the inequality constraint Tq Setup [Ozalp&al.,Journal of Biological Chemistry,’10] T ⇡ need to sum to one, i.e.⇡0, 1 0⇡(component-wise).=1. Moreover, we We further denote enforce the constraint the constraints set as , so⇡ thatZ =⇡y0 , . Due to the Gaussian 0 0 P 0 02 P • Suspension of yeast Saccharomyces cerevisiae (isolated cells) i.e., the expected valuesobservation of the NADH model and (38), ATP the pools ML at estimate time t0 ofequal(x, ⇡0 the) is measurement given by y0. • ED added abruptly to starved cells T Therefore, we have the linear equality constraint ⇡0 [Z, 1]=[y0, 1], and the inequality constraint • NADH (electron-carrier pool) and ATP levels measured (xˆ, ⇡ˆ0)=arg min f(x, ⇡0), (40) x 0,⇡0 ⇡ 0 (component-wise). We denote the constraint set as , so that ⇡ .2 DueP to the Gaussian 0 P 0 2 P whereq we Use experimental data to fit parameters have defined the negative log-likelihood cost function observation model (38), the ML estimate of (x, ⇡0) is given by N k 2 1 T nj nj 1 (xˆ, ⇡ˆ0)=argf(x, ⇡ min0) , f(x, ⇡0y),k ⇡0 P(x, sE,j 1) (40) Z . (41) x 0,⇡0 2 2P j=1 Xk=0 Y F where we have defined the negative log-likelihood cost function For a fixed x, the optimization over ⇡0 is aState distribuon quadratic programming problem, which can be 2 solved1 efficientlyN using,k e.g., interior-point methods [27], [28]. On the other hand, for fixed T Measured ATP/NADH nj nj 1 f(x, ⇡0) , yk ⇡0 P(x, sE,j 1) Z . Fixed matrix: (41) 2 ⇡ , the optimizationk=0 overj=1 x is a non-convex optimization problem. Therefore, we resort to a 0 (average over populaon) F State indicator X Y For a fixed x, the optimization over ⇡0 is a quadratic programming problem, which can be November 18, 2013 DRAFT solved efficiently using, e.g., interior-point methods [27], [28]. On the other hand, for fixed

⇡0, the optimization over x is a non-convex optimization problem. Therefore, we resort to a

November 18, 2013 DRAFT Isolated cell: experimental validaon 26

3 Model Experimental [Ozalp&al.’10] 2.5 Glucose added (30mM)

EXPERIMENTAL DATA ) 2 [Ozalp et al.,’10] mM 1.5 ATP (mM) ATP ( 1

0.5 MODEL

0 0 500 1000 TimeTIME (s) (s) 30mM GLUCOSE ADDED q Good fing of experimental curve and model

¨ Stochasc queuing model simpler than detailed “physical” models, but captures salient features Informaon theorec aspects

M Mˆ

ENC DEC Mulhop Capacity

29

¨ Cascade of links • Persistently studied

¨ Via cut-set bounds can show that capacity is that of worst-case link

• Achieved by decode-and-forward

¨ What happens in biological systems? boleneck • Communicaon over channels with state! • There is coupling between the links Bacterial cable as communicaon system 31 M Mˆ CHANNEL STATE INFORMATION (CSI) ENC DEC

(t) [ , ] Cable state S(t) 2 min max Y (t) INPUT e- INTENSITY OUTPUT e- FLOW

CHANNEL q Challenge: signal is coupled with energec state of cells (channel state) q ATP level = energec state of the cell Ø Energy dimension: ATP low à cell may die Ø Informaon dimension: ATP high à clogging, e- transfer stopped Simplified cable model

32

q S(t)=Interconnecon of all SATURATION cells’ states  CLOGGING, e- FLOW = 0

q Simplified model: S(t)=# e- in the cable • Local interacons averaged out over ) spaal dimension • Global effects captured by clogging state A(s) § S(t) small à cells “starved” 1 A(s) § S(t) large à ATP saturaon, clogging § A(s) = rao of e- accepted 0 0 s Smax Communicaon over Poisson channels with states 34

M Mˆ CHANNEL STATE INFORMATION (CSI) ENC DEC

(t) [ , ] Cable state S(t) 2 min max Y (t) INPUT e- INTENSITY OUTPUT e- FLOW

X(t) (A(S(t))(t)) CHANNEL Y (t) (µ(S(t))) ⇠ P ⇠P Prior Work

35

¨ Uncontrolled case, no CSI: [Goldsmith&Varaiya’96] iid coding

¨ Uncontrolled case, paral CSIT, CSIR: [Yuksel&Takonda’07]

¨ I.i.d. state controlled by acons, CSIT: [Weissman’10], [Choudhuri&Mitra,’12]

¨ Controlled case, no CSI: [Pfister et al.’01] i.i.d. coding; with FB: [Takonda&Mier] POMDP

¨ Controlled case, full CSI & FB: [Chen&Berger’05] opmality of Markov coding, opmizaon via DP

¨ In our case, Markov state controlled by signal, full CSI case q Finite-state connuous-me Markov Poisson channel § Stac Poisson channels [Wyner’88] § We extend to finite-state Markov channels (dynamic) § Discrete-me: upper & lower bounds [Lapidoth&Moser’09] Discrete me model: approximaon 37

¨ Assumpon: “small” slot duraon [Wyner’88]

NO ELECTRONS IN/OUT 1 A(s)¯(s) µ(s) µ(s) A(s)¯(s) S-2 S-1 S S+1 S+2 1 OUT 1 IN

2 2 >1 OUT o( ) o( ) >1 IN

2 § Other events: probability o( ) à neglected § Only consider adjacent states Discrete me model: approximaon 38

CHANNEL STATE INFORMATION (CSI) ENC DEC

Cable state S 1 k 1 Xk Yk 0 0

CHANNEL

INPUT ELECTRON FLOW OUTPUT ELECTRON FLOW

¨ Assumpon: “small” slot duraon [Wyner’88] ¨ Channel with state (ENC controls intensity only)

¨ State “controlled” by input process; state-dependent output § Finite-state Markov [Chen&Berger’05] Capacity of finite-state Markov channel 39

¨ Staonary Markov coding opmal [Chen&Berger’05] ¨ Capacity as dynamic program § Acon/control: rate selecon § Reward: mutual informaon condioned on past state, control funcon

¨ THEOREM: capacity achieved by feedback independent inputs - [Chen&Berger’05] assumes feedback S+1

Asymptoc capacity of Poisson Markov channels 42

Smax

C⇤ = max max ⇡¯(s)If (s) ¯ f (¯) 2F s=0 X If (s)=E[f()] q Informaon-Energy coupling: MAX § Energy dimension: f() - Determined by ¯(s) min ¯(s) max § Informaon dimension: - PROP: Staonary distribuon depends only on ¯(s)=E[ s] | - Impact: Opmize over ENC funcons f with same ¯(s)

§ THEOREM: Binary signal opmal k min,max 2{ } - Impact: easier to operate with a binary signal

Myopic scheme

43

¨ Myopic scheme: maximizes in each slot If (s) § Can compute average rate in closed form § Independent of state s, no CSIT required (however, CSIR sll required)

¨ THEOREM: ¯⇤(s) ¯ (otherwise, fast recharges & low rate)  MP § Proof: properes of n-step value funcon ) , s ( I

min ¯MP max MP & OPT

46

q Myopic scheme: greedy 1 one step A(s) Amin 0 0 s Smax 0.8 q OPT 10% beer than MP 0.78 § MP: Greedy informaon OPT transfer maximizaon 0.76 achieves poor 0.74 performance 0.72 MP q Informaon transfer 0.7

maximizaon vs AVG ACHIEVABLE RATE 0.68 “wellness” maximizaon Amin 0.66 0 0.2 0.4 0.6 0.8 1 Queson

49

q Q: Can we apply these queueing theorec approaches to OTHER bacterial networks?

q A: YES! - Quorum sensing… Queuing models for Quorum Sensing Quorum Sensing 101

52

• Gene expression regulaon in response to fluctuaons in cell- populaon density • Bacteria produce autoinducers that increase in concentraon as a funcon of cell density • regulates many diseases hp://sites.tus.edu/quorumsensing/quorumsensing101/ Quorum Sensing

53

Vibrio Fischeria

Vibrio Fischeri luminescing Towards a model of QS

60 “Ingredients” of QS 1. Microbial community (e.g., Pseudomonas Aeruginosa) 2. Autoinducers only global parameter, others by cell, A(t) i - Emied by each cell in the environment

3. Receptors Ri(t) - They bind to autoinducers within each cell to form complexes

4. Complexes Ci(t) - 1 autoinducer bound to 1 receptor

5. Synthases Si(t) - “machines” that produce autoinducers 6. DNA binding sites - Complexes bind to them to produce synthases, receptors & virulence factors Gene Expression

61

¨ The promoter region between genes determines which proteins to make

genomic DNA

¨ Decision of whether or not to transcribe a gene depends on binding to operators and the promoter

From James Boedicker, USC Underlying processes

62

¨ Autoinducers bind to receptors to create complexes

§ Number of autoinducers/receptors decrease by 1

¨ Complexes can bind to one of three sites to create • Synthases (create autoinducers) • Receptors • Virulence factors (BAD) cause disease, don’t affect QS • Each have unique binding rates

¨ Complexes, synthases, receptors can degrade ¨ Cells do not die, but can duplicate Abstracon of QS

63

A(t) autoinducers CELL

Ri(t) receptors

Si(t) synthases Ci(t) complexes Virulence ✏C,3 factors ✏C,1 ✏C,2 Sites 1 2 3 Queuing model of QS

64

C1(t)✏1

C1(t)✏2 S2(t) R2(t)

R1(t) S1(t) C2(t)

C1(t) SYNTHASE RECEPTOR COMPLEXES QUEUE QUEUE QUEUE A(t)

S4(t) R4(t) R1(t)A(t) S1(t) C4(t) AUTOINDUCER QUEUE

S3(t) R3(t)

C3(t)

A(t)(N(t)) Compact state space representaon

65

q THEOREM:

is a sufficient stasc for QS acvaon

q Don’t need to track for each cell, ) track at the populaon level § Lower complexity!

R1(t)

TOTAL TOTAL TOTAL SYNTHASE RECEPTOR COMPLEXES AUTOINDUCER QUEUE QUEUE QUEUE QUEUE A(t)(N(t)) Simulaons

66 q Simulaon tools based on queuing model

§ Cell density is higher in 11 open system 10 - Larger concentraon of 10 autoinducers 10 - Faster acvaon me 9 10

Open system, time series Cell density [cells per mL] 8 10 Open system, QS activation time OPEN SYSTEM: cells grow in open space (no boundariesàleakage of autoinducers) Closed system, time series Closed system, QS activation time CLOSED SYSTEM: cells grow in finite box 7 10 (boundariesà no leakage) 0 2 4 6 8 10 TIME [hours] Simulaons

67 q Simulaon tools based on queuing model

100 Open system, time series 90 § Model allows realisc Open system, QS activation time performance 80 Closed system, time series Closed system, QS activation time evaluaon in short me 70 60 § Use of complexity reducon tools 50 from wireless comm 40 30

20 Autoinducers concentration [nM] OPEN SYSTEM: cells grow in open space (no 10 boundariesàleakage of autoinducers) 0 CLOSED SYSTEM: cells grow in finite box 0 2 4 6 8 10 (boundariesà no leakage) TIME [hours] Experimental results

68

¨ Closed system Model QS activation time experimental data ¨ 9 Escherichia coli cells 10 grown in LB media at 37C ¨ Green Fluorescence protein (GFP) monitored 8 (~ gene expression Cell density [cells per mL] 10 acvity)

0 2 4 6 8 10 TIME [hours] Experimental results

69 ¨ Closed system 200 ¨ Escherichia coli cells Open system, 180 time series Open system, grown in LB media at 37C 160 QS activation time Closed system, time series 140 Closed system, ¨ Green Fluorescence QS activation time 120 Experimental time series, protein (GFP) monitored GFP in arbitary units ×40 100 Experimental data, (~ gene expression QS activation time 80

acvity) 60

¨ Model & experiments: Synthases concentration [nM] 40 § Similar qualitavely 20 0 § 0 2 4 6 8 10 But GFP=0 before QS TIME [hours] acvaon (low basal expression rate) à further invesgaon needed It is much more complicated…

71

¨ The hierarchy quorum sensing network in Pseudomonas aeruginosa, Lee & Zhang, Protein&Cell 2015, 6(1):26–41

¨ Both smulatory and suppressive interacons

Schematic representation of the four QS signaling networks in P. aeruginosa and their respective regulons Bacterial Models

72 Large Scale Networks

¨ Can use CS on graphs

¨ Can learn cost funcons (e.g. throughput, SINR, BER, etc.)

¨ Can learn with two orders magnitude fewer samples Use your favorite CS method

Diffusion Wavelet based CS learning Standard Q learning

¨ Using Least Squares Residual Compressed sensing handles instability § Vaswani, IEEE Trans on SP OLD 8/2010 NEW

Levorato, M & Goldsmith, EURASIP Wireless Comm & Net, 8/12 Summary

76

q Stochasc queuing theorec model § Captures salient features, amenable to complexity reducon using W-COM tools (vs detailed physical models)

q IT aspects, coupling between § Coupling between wellness & informaon transfer § Impact: state esmaon to adapt signal and opmize this trade- off

¨ Quorum sensing modeling § Need to consider mulple signaling mechanisms § Experimental data analysis § Opmizaon to migate virulence factors Thanks to…

77 Nicolo PUBLICATONS Michelusi N. Michelusi, S. Pirbadian, M.Y. El-Naggar, U. Mitra, ”A Stochasc Model for Electron Transfer in Bacterial Cables”, IEEE JSAC– Series on Sahand Molecular, Biological, and Mul-Scale Communicaons 32 (12), , pp.2402-2416, Dec. Pirbadian 2014 N. Michelusi, U. Mitra, “Capacity of electron- based communicaon over bacterial cables: the full-CSI case“, IEEE Transacons on Molecular, Mohammed Biological, and Mul-Scale Communicaons, El-Naggar 1(1), pp. 62-75, March 2015. N. Michelusi, J. Boedicker, M. El-Naggar, U. Mitra, ``Queuing models for abstracng James interacons in Bacterial communies,’’ IEEE JSAC Issue on Emerging Technologies, to appear. Boedicker

hp://www.comsoc.org/tmbmc Editor-in-Chief Urbashi Mitra University of Southern California, USA Associate Editor-in-Chief Andrew W. Eckford York University, Canada

principles, design, and analysis of signaling and informaon systems beyond convenonal electromagnesm, small-scale and mul-scale applicaons. molecular, quantum, chemical and biological techniques

78 hps://mc.manuscriptcentral.com/tmbmc