Linear, Shift-Invariant Systems

• Linearity: scalar rule and additivity • Applied to impulse, sums of impulses • Applied to sine waves, sums of sine waves Summary: Linear Systems Theory

• Signals can be represented as sums of sine waves • Linear, shift-invariant systems operate “independently” on each sine wave, and merely scale and shift them. • A simplified model of neurons in the visual system, the linear receptive field, results in a neural image that is linear and shift-invariant. • Psychophysical models of the visual system might be built of such mechanisms. • It is therefore important to understand visual stimuli in terms of their spatial frequency content. • The same tools can be applied to other modalities (e.g., audition) and other signals (EEG, MRI, MEG, etc.).

From lecture 1 The Fourier transform is a useful change of basis for many signals

Original Remove most fourier components

50 50

100 100

150 150

200 200

250 250 50 100 150 200 250 50 100 150 200 250

Remove most pixels Fourier transform of above

50 50

100 100

150 150

200 200

250 250 50 100 150 200 250 50 100 150 200 250 The Fourier transform is a useful change of basis for many signals

x = imread('cameraman.tif'); F = fft2(x); % Fourier transorm n = 30; % Keep Fourier coefficients up to n F(n+1:end-n+1, n+1:end-n+1) = 0; % zero out the rest x2 = ifft2(F); % transform back to image domain

figure; colormap gray subplot(2,2,1), imagesc(x), title('Original') subplot(2,2,2), imagesc(x2), title('Remove most fourier components') subplot(2,2,4), imagesc(log(abs(F))), title('Fourier transform of above')

% now zero out most of the pixels for comparison x3 = x; x3(n+1:end-n+1, n+1:end-n+1) = 0;

subplot(2,2,3), imagesc(x3), title('Remove most pixels') 1. Homogeneity (scalar rule)

Neural activity fMRI response 2. Additivity Shift invariance Linear systems

A system (or transform) converts (or maps) an input signal into an output signal: y(t) = T[x(t)]

A linear system satisfies the following properties:

1) Homogeneity (scalar rule): T[a x(t)] = a y(t) 2) Additivity:

T[x1(t) + x2(t)] = y1(t) + y2(t)

Often, these two properties are written together and called superposition:

T[a x1(t) + b x2(t)] = a y1(t) + b y2(t) Shift invariance

For a system to be shift-invariant (or time-invariant) means that a time-shifted version of the input yields a time-shifted version of the output: y(t) = T[x(t)]

y(t - s) = T[x(t - s)]

The response y(t - s) is identical to the response y(t), except that it is shifted in time.

y(t) 1 5 1 1 1 1

0 1 2 3 4 5 ? ? 1 5 1 1 y(t-2) Neural Image - Reprise

A spatial receptive field may also be treated as a linear system, by assuming a dense collection of neurons with the same receptive field translated to different locations in the visual field. In this view, it is a linear, shift-invariant system.

See code on next slide Neural Image - Reprise

0.04

0.03

0.02

0.01

-0.01 20 20 10 10

0 0

% Make a Difference of Gaussian receptive field using fspecial DoG = fspecial('gaussian', 20,2) - fspecial('gaussian', 20,5); im = imread('cameraman.tif');

% Make a neural image by convolution neuralim = conv2(double(im), DoG);

% Show the image, the RF, and the neural image figure, subplot(1,3,1), imshow(im), subplot(1,3,2), surf(DoG) subplot(1,3,3), imagesc(neuralim); colormap gray, axis image off Linear, Shift-Invariant Systems

• Linearity: Scalar rule and additivity • Applied to impulse, sums of impulses • Applied to sine waves, sums of sine waves Convolution as sum of impulse responses

Input:

Impulse response:

Output:

+ Convolution as sum of impulse responses

% time vector 1 2 3 4 5 6 7 8 9 1011121314 Input S(t): dt = 0.01; T = 25; t = dt:dt:T-dt;

Impulse response I(t): % impulse response function h = exp(-(t));

% signal $1I(t t 1): x = zeros(size(t)); x(t<19) = 2; x(t<12) = 4; x(t<5) = 0; $ I(t t ): 2 2 % output of system by convolution r = conv(x,h/sum(h), 'full'); $3I(t t 3): r = r(1:length(t));

figure(1), clf, plot(t,x, 'k', 'LineWidth', 4); hold on $8I(t t 8): plot(t,r, 'LineWidth', 4);

% Suppose our sampling was just one time point per $9I(t t 9): second x2 = zeros(size(t)); x2(1:100:end) = x(1:100:end);

$14I(t t 14): r2 = conv(x2,h, 'full'); r2 =r2(1:length(t));

14 stem(t,x2) 3$i I(t ti): plot(t, r2, 'LineWidth', 4) i1 Convolution

Discrete-time signal: x[n] = [x1, x2, x3, ...]

A system or transform maps an input signal into an output signal: y[n] = T{x[n]}

A shift-invariant, linear system can always be expressed as a convolution: y[n] = 3 x[m] h[n-m] where h[n] is the impulse response. Convolution derivation

Homogeneity: T{a x[n]} = a T{x[n]}

Additivity: T{x1[n] + x2[n]} = T{x1[n]} + T{x2[n]}

Superposition: T{a x1[n] + b x2[n]} = a T{x1[n]} + b T{x2[n]}

Shift-invariance: y[n] = T{x[n]} => y[n-m] = T{x[n-m]} Convolution derivation (contd.)

Impulse sequence: d[n] = 1 for n = 0, d[n] = 0 otherwise

Any sequence can be expressed as a sum of impulses: x[n] = 3 x[m] d[n-m] where d[n-m] is impulse shifted to sample m x[m] is the height of that impulse

Example:

= + + Convolution derivation (cont) x[n]: input y[n] = T{x[n]}: output h[n] = T{d[n]}: impulse response

1) Represent input as sum of impulses: y[n] = T{x[n]} y[n] = T{ 3 x[m] d[n-m] } 2) Use superposition: y[n] =3 x[m] T{d[n-m]} 3) Use shift-invariance: y[n] = 3 x[m] h[n-m] Convolution as sum of impulse responses

Input S(t): 1 2 3 4 5 6 7 8 9 1011121314

Impulse response I(t):

$1I(t t 1):

$2I(t t 2):

$3I(t t 3):

$8I(t t 8):

$9I(t t 9):

$14I(t t 14):

14 3$i I(t ti): i1 Convolution as correlation with the “receptive field” (time-reversed impulse

response): Input S(t): 1 2 3 4 5 6 7 8 9 10 11 12 13 14

% time dt = 0.1; t = dt:dt:25; RF 1(t):

% signal (input) x = zeros(size(t)); RF (t)S(t): x(t<19) = 2; x(t<12) = 4; x(t<5) = 0; 1

% impulse response function h = exp(-(t)); RF 2(t):

% convert impulse response function to RF by time reversing it rf = flip(h)/sum(h); RF 2(t)S(t):

% output y = zeros(size(t)); RF 8(t):

% multiply time shifted RFs by (non-shifted) signal figure(1); clf; RF 8(t)S(t): for ii = 1:length(t) % shift the receptive field by one time step rf = rf([end 1:end-1]); RF 9(t):

% correlate (multiply) receptive field with signal

% note that the signal does not shift; only the rf shifts. RF 9(t)S(t): y(ii) = x*rf';

% plot plot(t,x ,'k', t, rf*10,'r',t, y, 'b', 'LineWidth', 4); F (t): legend('signal', 'rf', ‘output’); drawnow(); end Convolution as matrix multiplication

Linear system <=> matrix multiplication Shift-invariant linear system <=> Toeplitz matrix

⎛ ⎞ ! ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ 1 ⎛ ! ⎞ ! ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 5 ⎟ 1 2 3 0 0 0 0 2 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 2 ⎟ ⎜ 0 1 2 3 0 0 0 ⎟ ⎜ 0 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ −3 ⎟ = " 0 0 1 2 3 0 0 " 0 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 4 ⎟ ⎜ 0 0 0 1 2 3 0 ⎟ ⎜ −1 ⎟ ⎜ ⎟ ⎜ −6 ⎟ 0 0 0 0 1 2 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 2 ⎟ ⎝ ! ⎠ ⎜ ! ⎟ ⎜ −3 ⎟ ⎝⎜ ⎠⎟ ⎜ ⎟ ⎝ ! ⎠

Columns contain shifted copies of the impulse response. Rows contain time-reversed copies of impulse response. Convolution as matrix multiplication

Linear system <=> matrix multiplication Shift-invariant linear system <=> Toeplitz matrix

Columns contain shifted copies of the impulse response. Rows contain time-reversed copies of impulse response. Convolution as sequence of weighted sums

past present future 0 0 0 1 0 0 0 0 0 input (impulse)

1/8 1/4 1/2 weights

0 0 0 1/2 1/4 1/8 0 0 0 output (impulse response)

0 0 0 1 1 1 1 1 1 input (step)

1/8 1/4 1/2 weights

0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response)

Figure 3: Convolution as a series of weighted sums.

Using homogeneity,

Now let be the response of to the unshifted unit impulse, i.e., . Then by using shift-invariance, (4)

Notice what this last equation means. For any shift-invariant linear system , once we know its impulse response (that is, its response to a unit impulse), we can forget about entirely, and just add up scaled and shifted copies of to calculate the response of to any input whatsoever. Thus any shift-invariant linear system is completely characterized by its impulse response . The way of combining two signals speciﬁed by Eq. 4 is know as convolution. It is such a widespread and useful formula that it has its own shorthand notation, . For any two signals and , there will be another signal obtained by convolving with ,

Convolution as a series of weighted sums. While superposition and convolution may sound a little abstract, there is an equivalent statement that will make it concrete: a system is a shift- invariant, linear system if and only if the responses are a weighted sum of the inputs. Figure 3 shows an example: the output at each point in time is computed simply as a weighted sum of the inputs at recently past times. The choice of weighting function determines the behavior of the system. Not surprisingly, the weighting function is very closely related to the impulse response of the system. In particular, the impulse response and the weighting function are time-reversed copies of one another, as demonstrated in the top part of the ﬁgure.

7 Continuous-time derivation of convolution and Young (1983), and Oppenheim and Schafer (1989). and Young (1983), and Oppenheim and Schafer (1989). Continuous-Time and Discrete-Time Signals and Young (1983), and Oppenheim and Schafer (1989). Continuous-Time and Discrete-Time Signals In each of the above examples there is an input and an output, each of which is a time-varying signal.Continuous-TInWeeachwill oftreatthea signalaboimeve asexamplesaandtime-vDiscrtherearyingisfunction,ete-Tan inputimeand. anFSignalsoroutput,each timeeach, theof whichsignalishasa sometime-varying value signal., usuallyWe willcalledtreat“ aofsignal.” Sometimesas a time-varyingwe willfunction,alternatively.useFor eachtotimerefer,tothethesignalentirehas some signal v,aluethinking,ofusuallyas a freecalledvariable.“ of .” Sometimes we will alternatively use to refer to the entire InIn eachpractice,signalof the, thinkingabowillve ofusuallyexamplesas abefreethererepresentedvariable.is an inputas a finite-lengthand an output,sequenceeach ofofwhichnumbers,is a time-v,inarying signal. We will treat a signal as a time-varying function, . For each time , the signal has some which canIntakpractice,e integer valueswillbetweenusually 0beandrepresented, andaswherea finite-lengthis the lengthsequenceof theof numbers,sequence. ,in Thisvaluediscrete-time, usuallysequencecalledis“indeofxe.”d Sometimesby integers, weso wewilltakalternatie vtoelymeanuse “the nthto refernumberto thein entire signalwhich, thinkingcan takofe inteasgera freevaluesvariable.between 0 and , and where is the length of the sequence. sequenceThis,”discrete-timeusually calledsequence“ of ”isforindeshort.xed by integers, so we take to mean “the nth number in TheInsequenceindipractice,vidual,”numbersusuallywillcalledinusuallya sequence“ beof represented” for short.are calledas a finite-lengthsamples of thesequencesignal of .numbers,The word ,in “sample”which comescan takfrome intethegerfactvaluesthat thebetweensequence0 andis a discretely-sampled, and where versionis the oflengththe continuousof the sequence. The individual numbers in a sequence are called samples of the signal . The word signal.ThisImagine,discrete-timefor example,sequencethatisyouindearexedmeasuringby integers,membraneso we takpotentiale (orto meanjust about“the annthythingnumber in “sample” comes from the fact that the sequence is a discretely-sampled version of the continuous else,sequencefor that matter),” usuallyas itcalledvaries“oveofr time.” forYshort.ou will obtain a sequence of measurements sampled signal. Imagine, for example, that you are measuring membrane potential (or just about anything at evenly spaced time intervals. Although the membrane potential varies continuously over time, Theelse,indifor vidualthat matter)numbersas itinvariesa sequenceover time. Yoareu willcalledobtainsamplesa sequenceof theofsignalmeasurements. Thesampledword you will work just with the sequence of discrete-time measurements. “sample”at evenlycomesspacedfromtimetheintervfact thatals. theAlthoughsequencetheismembranea discretely-sampledpotential variesversioncontinuouslyof the continuousover time, signal.It isyouoftenImagine,willmathematicallyworkforjustexample,withconthevthatsequenceenientyoutoareofworkdiscrete-timemeasuringwith continuous-timemembranemeasurements.potentialsignals.(orButjustin aboutpractice,anything youelse,usuallyfor thatend upmatter)with asdiscrete-timeit varies ovsequenceser time. Ybecause:ou will obtain(1) discrete-timea sequencesamplesof measurementsare the onlysampled It is often mathematically convenient to work with continuous-time signals. But in practice, things that can be measured and recorded when doing a real experiment; and (2) finite-length, at evyouenlyusuallyspacedendtimeupintervwith als.discrete-timeAlthoughsequencesthe membranebecause:potential(1) discrete-timevaries continuouslysamples areovether time,only discrete-time sequences are the only things that can be stored and computed with computers. youthingswill workthatjustcanwithbe measuredthe sequenceand recordedof discrete-timewhen doingmeasurements.a real experiment; and (2) finite-length, In whatItdiscrete-timeis follooftenws,mathematicallywesequenceswill expressareconthemostvonlyenientofthingsthetomathematicswthatork canwithbecontinuous-timeinstoredthe continuous-timeand computedsignals.withdomain.Butcomputers.inButpractice, the examples will, by necessity, use discrete-time sequences. you usuallyIn whatendfolloup ws,withwediscrete-timewill expresssequencesmost of thebecause:mathematics(1) discrete-timein the continuous-timesamples aredomain.the onlyBut thingsthethatexamplescan bewill,measuredby necessityand ,recordeduse discrete-timewhen doingsequences.a real experiment; and (2) finite-length, discrete-time sequences are the only things that can be stored and computed with computers. Pulse and impulse signals. The unit impulse signal, written , is one at , and zero everywhereIn whatelse:follows, we will express most of the mathematics in the continuous-time domain. But Pulse and impulse signals. The unit impulse signal, written , is one at , and zero the examples will, by necessity, use discrete-time sequences. everywhere else: Pulses and impulses The impulse signal will play a very important role in what follows. Pulse and impulse signals. The unit impulse signal, written , is one at , and zero eOneverywhereTheveryimpulseusefulelse:signalway towillthinkplayofatheveryimpulseimportantsignalroleisinaswhata limitingfollows.case of the pulse signal, : One very useful wimpulseay to think of the impulse signal is as a limiting case of the pulse signal, : The impulse signal will play a very important role in what follows. The impulse signal is equal tounitthe pulsepulse signal when the pulse gets infinitely short: One very useful way to think of the impulse signal is as a limiting case of the pulse signal, The: impulse signal is equal to the pulse signal when the pulse gets infinitely short:

The impulse signal is equal to the pulse signal when the pulse gets inﬁnitely short: 2

2 = ... + + ++... = ... + + ++... = + + ++ Figure...1: Staircase approximation to a continuous-time signal. ... Figure 1: Staircase approximation to a continuous-time signal. Representing signalsFigurwithe 1:impulses.StaircaseStaircaseapproximationAny signal approximationtocana continuous-timebe expressed assignal.a tosum continuous-of scaled and shifted unitReprimpulses.esentingWsignalse beginwithwith impulses.the pulse orAn“staircase”y signal canapproximationbe expressed as toa suma continuousof scaled and signalshifted, aunits illustratedimpulses.inWFig.e be1.ginConceptuallywith the pulse, thisor “staircase”is tritimevial: for approximationsignaleach discrete sampleto a ofcontinuousthe Representing signals with impulses. Any signal can be expressed as a sum of scaled and originalsignalsignal, we, asmakillustratede a pulsein signal.Fig. 1. ThenConceptuallywe add,upthisallisthesetrivial:pulsefor signalseach discreteto maksamplee up theof the shifted unit impulses. We begin with the pulse or “staircase” approximation to a continuous approximateoriginalsignal.signal, Eachwe makofethesea pulsepulsesignal.signalsThencanweinaddturnupbeallrepresentedthese pulseassignalsa standardto makpulsee up the signal , as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the scaledapproximateby the appropriatesignal.valueEachandof shiftedthese pulseto thesignalsappropriatecan inplace.turn beIn mathematicalrepresented asnotation:a standard pulse original signal, we make a pulse signal. Then= we+ add up all these+ pulse signals++to make up the scaled by the appropriate value and shifted to...the appropriate place. In mathematical notation: ... approximate signal. Each of these pulse signals can in turn be represented as a standard pulse scaled by the appropriate value and shifted toFigurthee appropriate1: Staircase approximationplace. Into amathematicalcontinuous-time signal.notation:

% time t = dt:dt:10; As we let approach zero, the approximationRepresenting signalsbecomeswith impulses.betterAnandy signalbetter can, beandexpressedthe in asthea sumlimitof scaled and % signal (gaussian, centered at 5 with sd of 2) shifted unit impulses. We begin with the pulse or “staircase”x = exp(-(t-5).^2/2^2);approximation to a continuous equalsAs we. Therefore,let approach zero, the approximation becomes better and better, and the in the limit signal , as illustrated in Fig. 1. Conceptually, this %is discretelytrivial: samplefor signaleach at discreted = 1 second stepssample of the D = 1; equals . Therefore, original signal, we make a pulse signal. Then we add %up showall the thesesampling pulse signals to make up the As we let approach zero, the approximation becomes better andfigure(1),better clf;, andhold on the in the limit approximate signal. Each of these pulse signals can inforturn k = 1:10be represented as a standard pulse [~, whichTimePoint] = min(abs(t-k*D)); equals . Therefore, scaled by the appropriate value and shifted to the appropriate plot(t,x,place. 'k', Int, x(whichTimePoint)mathematical * (t-D*knotation: < D & t-D*k>0), 'b'); pause(.3) end Also, as , the summation approaches an integral, and the pulse approaches the unit impulse: % now do the same for finer sampling D = 0.2; Also, as , the summation approaches an integral, and the pulse approachesfigure(1), clf; hold onthe unit impulse: for k = 1:10/D [~, whichTimePoint] = min(abs(t-k*D)); plot(t,x, 'k', t, x(whichTimePoint) *(1) (t-D*k < D & t-D*k>0), 'b'); pause(.05) Also, as , the summationAs weapproacheslet approachanzero,intethegral,approximationand the pulsebecomesapproachesend better andthebetterunit, andimpulse:the in (1)the limit equals . Therefore, In other words, we can represent any signal as an infinite sum of shifted and scaled unit impulses. A digitalIncompactother words,disc, weforcanexample,representstoresanywholesignalcompleas an infinitex piecessumof musicof shiftedas lotsandofscaledsimpleunitnumbersimpulses.(1)A representingdigital compactvery shortdisc,impulses,for example,Also,andasthenstoresthe, thewholeCDsummationplayercompleapproachesaddsx piecesallantheinteofimpulsesgral,musicandasthebacklotspulseofapproachestogethersimple theonenumbersunit impulse: afterInanotherotherrepresentingwords,to recreateweverycanshorttherepresentcompleimpulses,xanmusicalyandsignalthenwasavtheaneform.infiniteCD playersumaddsof shiftedall theandimpulsesscaledbackunit impulses.together oneA digitalaftercompactanother todisc,recreatefor example,the complestoresx musicalwhole complewaveform.x pieces of music as lots of simple numbers (1) This no doubt seems like a lot of trouble to go to, just to get back the same signal that we representing very short impulses, and then the CD player adds all the impulses back together one originallyThisstartedno with,doubtbuseemst in fact,likIneweothera willlotwords,ofverywetroublecanshortlyrepresentto begoanableto,y signaljustto useastoanEq.getinfinite1backtsumo performoftheshiftedsameaandmarvsignalscaledelousunitthatimpulses.we A after another to recreate the compledigital compactx musicaldisc, forwaevxample,eform.stores whole complex pieces of music as lots of simple numbers trick. originally started with, but irepresentingn fact, wevwillery shortveryimpulses,shortlyandbethenablethetoCDuseplayerEq.adds1 toallperformthe impulsesa marvback togetherelous one trick.This no doubt seems likeaftera lotanotherof troubleto recreatetothegocompleto, xjustmusicalto wgetaveform.back the same signal that we originally started with, but in fact,Thiswenowilldoubtveryseemsshortlylike a lotbeofabletroubletotousego Eq.to, just1 ttoo performget back thea samemarvsignalelousthat we Lineartrick. Systems originally started with, but in fact, we will very shortly be able to use Eq. 1 to perform a marvelous Linear Systems trick. A system or transform maps an input signal into an output signal : Linear Systems LinearA systemSystemsor transform maps an input signal into an output signal :

A system or transform maps an input signal into an output signal : A system or transform maps an input signal into an output signal : where denotes the transform, a function from input signals to output signals. where denotes the transform, a function from input signals to output signals. Systems come in a wide varietywhereof types.denotes theOnetransform,importanta functionclassfromis knoinputwnsignalsas tolinearoutputsystemssignals. .To see whetherSystemsa systemcomeis linearin a wide, we vneedSystemsarietytooftestcometypes.whetherin a wideOnevitarietyimportantobeofystypes.certainclassOnerulesimportantis knothatwnclassallasislinearlinearknownsystemsassystemslinear systems.To .To obewherey. seeThewhethertwdenoteso basica systemtheteststransform,ofislinearitylinearsee,awhetherwfunctionaree needhomogeneitya systemtofromtestis linearwhetherinputand, wesignalsneedadditiit obeto testvitytoyswhetheroutput.certainitsignals.oberulesys certainthat allruleslinearthat allsystemslinear systems obey. The two basic tests of linearity are homogeneity and additivity. obeSystemsy. Thecometwo basicin a testswideofvarietylinearityof types.are homogeneityOne importantand additiclassvityis kno. wn as linear systems.To see whether a system is linear, we need to test4whether it obeys certain4 rules that all linear systems obey. The two basic tests of linearity are homogeneity4 and additivity.

4 Im pu ls es Im pu ls es IImmpuplsuels es Im pu ls e Res pon s e Im pu ls e Res pon s e Im pu ls e ImImpuplusles e Res pon s e

Ea ch Im pu ls e Crea tes a Sca lIemd apnud lSsheifstedE Iamchp uImls ep uRlessep Cornesaetes a Ea ch Im pu lsSe cCarleeda taens da Sh ifted Im pu ls e Res pon s e ScaIlmedp ua nlsde Sh ifted Im puImls ep uRlessep Roensspeon s e

For For exa m ple For exa m ple Eexaachm pImlepu ls e Crea tes a Sca led a n d Sh ifted Im pu ls e Res pon s e

Th e s u m of a ll th e im pu ls e res pon s es Th e s u m of a ll th e im pu ls e res pon s es is th e fin a l s ys tem res pon s e Th e s u m of a ll th e im pu ils et hres fpinoanls seyss tem res pon s e is th e fin a l s ys tem res pon s e For exa m ple

Figure 2Figur: CharacterizingTh ee s u2m: oCharacterizingf a ll th e aimlinearpu ls e resystemsapolinearn s es usingsystemitsusingimpulseits impulseresponse.response. Figure 2: Characterizingis th e afinlineara l s ys tesystemm res pon susinge its impulse response.

The wayThewewuseaytheweimpulseuse the impulseresponseresponsefunctionfunctionis illustratedis illustratedin Fig. 2.inWFig.e concei2. Weveconceiof theveinputof the input Thestimulus,way wstimulus,einusethisthecaseinimpulsethisa sinusoid,caseresponsea sinusoid,as functionif it wereas ifistheitillustratedweresumtheof asuminsetFig.ofof2.aimpulsessetWeofconceiimpulses(Eq.ve1).of(Eq.theWeinput1).knowWetheknow the stimulus,responsesin thisresponseswecasewoulda sinusoid,wegetwouldif eachasgetifimpulseifiteachwereimpulsewtheas presentedsumwofas presenteda setseparatelyof impulsesseparately(i.e., scaled(Eq.(i.e.,1).andscaledWshiftede knoandwcopiesshiftedthe ofcopies of responsesthe impulsewe wtheouldimpulseFigurresponse).geteif2:response).eachCharacterizingWeimpulsesimplyWewaddsimplyaaslineartogetherpresentedaddsystemtogetherallseparatelyusingof theallits(scaledofimpulse(i.e.,the and(scaledscaledresponse.shifted)andand shiftedshifted)impulsecopiesimpulseresponsesofresponsesto to the impulsepredictresponse).hopredictw thehosystemWwethesimplywillsystemrespondaddwilltogetherrespondto theallcompletetoofthethecomplete(scaledstimulus.andstimulus.shifted) impulse responses to predict how the system will respond to the complete stimulus. The waNoy wewuseeNowillthew repeatwimpulsee willallrepeatthisresponseinallmathematicalthisfunctionin mathematicalisnotation.illustratednotation.Ourin Fig.goalOur2.is toWgoalshoe conceiiswtothatshovethewofresponsethatthe theinputresponse(e.g., (e.g., stimulus,Nomembranew wine willthismembranerepeatcasepotentialaallsinusoid,potentialthisfluctuation)in mathematicalasfluctuation)if itofwerea shift-inofthenotation.avsumshift-inariantofOurlinearvaariantsetgoalofsystemlinearisimpulsesto sho(e.g.,systemw (Eq.thatpassi(e.g.,the1).veresponsepassiWneurale knovemembrane)neuralw(e.g.,the membrane) responsesmembranecan webepotentialwrittenwcanouldbegetfluctuation)aswrittenaifsumeachasofaimpulsescaledofsuma shift-inofandwscaledasshiftedpresentedvariantandcopiesshiftedlinearseparatelyofsystemcopiesthe system’(i.e.,of(e.g.,thescaledspassisystem’impulsevandesneuralimpulseresponseshiftedmembrane)copiesresponsefunction.of function. thecanimpulsebe writtenresponse).as a sumWofescaledsimplyandaddshiftedtogethercopiesall ofofthethe(scaledsystem’ands impulseshifted)responseimpulse responsesfunction. to predict how the system will respond to the completeConvolutionstimulus. The conThevolutionconvolutionintegral.integral.Begin byBeusinggin byEq.using1 toEq.replace1 to replacethe inputthesignalinput signalby its repre-by its repre- TheNosentationwconwevwillolutionsentationinrepeatRepresentingtermsintegral.allinoftermsthisimpulses:inofmathematicalBeimpulses: gintheby inputusingnotation. signalEq. 1 to Ourasreplace goala sumtheis toinput ofsho pulses:wsignalthat the responseby its repre-(e.g., sentationmembraneinpotentialterms offluctuation)impulses: of a shift-invariant linear system (e.g., passive neural membrane) can be written as a sum of scaled and shifted copies of the system’s impulse response function.

past present future The convolution 0integral. 0 0 1 Be 0gin 0 by 0 using 0 0Eq. 1inputto replace (impulse)the input signal by its representation in terms of impulses:1/8 1/4 1/2 weights Using additivity, Using additiUsingvityadditi0, 0 vity 0 , 1/2 1/4 1/8 0 0 0 output (impulse response) Using additivity,

Taking theTakinglimit,the0 0limit, 0 1 1 1 1 1 1 input (step) Taking the limit, Taking the 1/8limit, 1/4 1/2 weights Using additivity, 0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response) 6 6 Figure 3: Convolution as a series of weighted sums. Using homogeneity (scalar6 rule), Taking the limit, Using homogeneity,

Now let be theDefiningresponse ofh(t)to asthe theunshifted impulse6unit impulse, response,i.e., . Then by using shift-invariance, (4)

7 Linear, Shift-Invariant Systems

• Linearity: Scalar rule and additivity • Applied to impulse, sums of impulses • Applied to sine waves, sums of sine waves Im pu lses Shift-invariant linear Im pu lse Im pu lse Respon se systems and impulses

Each Impulse Creates a Scaled and Shifted Impulse Response

For example

The sum of all the impulse responses is th e fin al system respon se

Figure 2: Characterizing a linear system using its impulse response.

The way we use the impulse response function is illustrated in Fig. 2. We conceive of the input stimulus, in this case a sinusoid, as if it were the sum of a set of impulses (Eq. 1). We know the responses we would get if each impulse was presented separately (i.e., scaled and shifted copies of the impulse response). We simply add together all of the (scaled and shifted) impulse responses to predict how the system will respond to the complete stimulus. Now we will repeat all this in mathematical notation. Our goal is to show that the response (e.g., membrane potential ﬂuctuation) of a shift-invariant linear system (e.g., passive neural membrane) can be written as a sum of scaled and shifted copies of the system’s impulse response function.

The convolution integral. Begin by using Eq. 1 to replace the input signal by its representation in terms of impulses:

Using additivity,

Taking the limit,

Shift-Invariant Linear Systems and Sinusoids

Frequency Description of the system

Sinusoidal Scaled and Shifted Inputs sinusoidal outputs x

scaling x

frequ en cy

x x x shifting

frequ en cy

Figure 5: Characterizing a system using its frequency response. shift and the scale. This makes the job of measuring the response to sinusoids at many different frequencies quite practical. Often then, when scientists characterize the response of a system they will not tell you the impulse response. Rather, they will give you the frequency response, the values of the shift and scale for each of the possible input frequencies (Fig. 5). This frequency response representation of how the shift-invariant linear system behaves is equivalent to providing you with the impulse response function (in fact, the two are Fourier transforms of one another). We can use either to compute the response to any input. This is the main point of all this stuff: a simple, fast, economical way to measure the responsiveness of complex systems. If you know the coefﬁcients of response for sine waves at all possible frequencies, then you can determine how the system will respond to any possible input.

Linear filters. Shift-invariant linear systems are often referred to as linear filters because they typically attenuate (filter out) some frequency components while keeping others intact. For example, since a passive neural membrane is a shift invariant linear system, we know that injecting sinusoidally modulating current yields membrane potential fluctuations that are sinusoidal with the same frequency (sinusoid in, sinusoid out). The amplitude and phase of the output sinusoid depends on the choice of frequency relative to the properties of the membrane. The membrane essentially averages the input current over a period of time. For very low frequencies (slowly varying current), this averaging is irrelevant and the membrane potential fluctuations follow the injected current. For high frequencies, however, even a large amplitude sinusoidal current modulation will yield no membrane potential fluctuations. The membrane is called a low-pass