Linear Systems Theory Fourier Analysis

• Introduction – Receptive fields and Signals as sums of sine waves mechanisms • 1-d: time series • Fourier Analysis – Signals as sums of sine – fMRI signal from a voxel or ROI waves – mean firing rate of a neuron over time • Linear, shift-invariant systems – auditory stimuli – Definition • 2-d: static visual image, neural image – Applied to impulses, sums of impulses • 3-d: visual motion analysis – Applied to sine waves, sums of sine waves • Applications • 4-d: raw fMRI data

Linear Systems Analysis Spatial Vision

Systems with signals as input and output • Image representation or coding – At each stage, what information is kept and • 1-d: low- and high-pass filters in electronic what is lost? equipment, fMRI data analysis, or in sound • Image analysis production (articulators) or audition (the ear as a filter) • Nonlinear: pattern recognition • 2-d: optical blur, spatial receptive field • 3-d: spatio-temporal receptive field

Receptive Field Receptive Field A spatial receptive field is an image • In any modality: that region of the sensory apparatus that, when stimulated, can directly affect the firing rate of a given neuron • Spatial vision: spatial receptive field can be mapped in visual space or on the retina • Examples:

LGN V1 with its own Fourier transform. Neural Image Neural Image of a Sine Wave For a linear, shift-invariant system such as a linear model of A spatial receptive field may also be treated as a linear a receptive field, an input sine wave results in an identical system, by assuming a dense collection of neurons with the output sine wave, except for a possible lateral shift and same receptive field translated to different locations in the scaling of contrast. visual field:

Frequency Response Frequency Response This scaling of contrast by a linear receptive field in the This scaling of contrast by a linear receptive field in the neural image is a function of spatial frequency determined neural image is a function of spatial frequency determined by the shape of the receptive field. by the shape of the receptive field. Contrast Gain Contrast Gain

Spatial Frequency Spatial Frequency

Orientation Tuning Application Preview: SF Adaptation (Blakemore & Campbell, 1969) If a receptive field is not circularly symmetric, the scaling of contrast is also a function of orientation (for a given spatial frequency) determined by the shape of the receptive field. Contrast Gain

Orientation Application Preview: SF Adaptation Summary: Linear Systems Theory (Blakemore & Campbell, 1969) • Signals can be represented as sums of sine waves • Linear, shift-invariant systems operate “independently” on each sine wave, and merely scale and shift them. • A simplified model of neurons in the visual system, the linear receptive field, results in a neural image that is linear and shift-invariant. • Psychophysical models of the visual system might be built of such mechanisms. • It is therefore important to understand visual stimuli in terms of their spatial frequency content. • The same tools can be applied to other modalities (e.g., audition) and other signals (EEG, MRI, MEG, etc.).

Auditory example: Pure tones Frequency and amplitude

weak 100 Hz strong 100 Hz weak 1000 Hz strong 1000 Hz

Time

1/100 s Sound pressure level

Pure tones can be described by 3 numbers: Frequency = rate of air pressure modulation (related to pitch) Amplitude = sound pressure level (related to loudness) Phase = sin vs. cosine vs. another horizontal shift

Fourier components of a square wave Fourier components of a square wave

Ampl. Freq.

1 100 Square wave

-1.3 1/3 300 ‘f’ + ‘3f’/3

0 +.4

+0 1/5 500 ‘f’ + ‘3f’/3+’5f’/5

+0 -.2

1/7 700 ‘f’ + ‘3f’/3+’5f’/5+’7f’/7 =0 Prev Next = -1.1 Fourier Synthesis – Building Stimuli from Sine Waves Fourier spectrum representation of sound SPL Amplitude

0 10 20 30 0 100 200 300 Time (ms) Frequency (Hz)

Fourier spectra of some sounds Fundamental frequency and harmonics Pure tone

White noise Amplitude

Violin note

100 1000 10,000 Frequency (Hz)

Standing waves in Flute (open pipe) harmonics a vibrating string Flute (open pipe) harmonics

Other notes (shorten the pipe)

Reinforcing a harmonic (forcing a “node”)

Complex numbers and complex exponentials Discrete Fourier Transform z = a + bi = Aei! = A"cos! + i sin!$ where i [or j] = !1 # %

real imaginary amplitude phase N%1 imaginary %i(2& /N)kn part part X "!k $# = ( x "!n$#e 0 ' k ' N % 1 n=0 where: b a = Acos! b = Asin! A N%1 ! = x !n# cos((2& / N)kn) + i sin((2& / N)kn) 2 2 "1 ( " $( ) A = a + b ! = tan (b / a) a real n=0 Why bother? k is frequency in cycles/image (or cycles/signal) and is computed i!1 i!2 i(!1+!2 ) effectively only for frequencies zero (DC), 1, 2, ..., N/2. The A1e A2e = A1A2e ( )( ) vector you get back from MATLAB (fft or fft2, inverses are ifft and ifft2), however, continues redundantly (for real signals, that is): amplitudes phases multiply add 0, 1, ..., N/2-1, N/2=0, N/2-1, ..., 1. DFT of a Sine Wave

$ 2!kn ' 2!kn 2!kn cycles/scan Acos " # = Acos# cos + Asin# sin & N ) N N time points % ( The Fourier coefficient for this frequency of k cycles/signal is:

2!k / N a + bi = (Acos!) + (Asin!)i ( ! / 4 in this case) In other words, the amplitude is A and the phase is ! . frequency (cycles per scan) frequency (cycles per scan)

?

frequency frequency

Similarity Theorem

If f (x) ! F(") 1 # " & then f (ax) ! F a $% a '(

0 8 16 24 32 -16 -8 0 8 16 If you stretch the x-axis, you shrink the frequency axis

0 8 16 24 32 -16 -8 0 8 16 time Circular shift

f, 90 deg shift If x[n] X[k], then: 2f, 180 deg shift

Properties of the DFT x[(n-m)N] exp(-j2! km/N) X[k]

i.e., only phase (not magnitude) is affected

=

N-2 0 1 2 3 4 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 N-1 i.e., all cosine terms

To generate a signal or image from a transform: i.e., all sine terms

Other Properties of the Fourier Transform

Linearity f + g ! F + G

d cos2!"x d sin2!"x Derivative = #2!" sin2!"x, = 2!" cos2!"x x x df Hence: $ #i2!"F DTFT and DFS dx

iF Integral f dx " ! 2#$ Discrete-Time Fourier Transform (DTFT)

Two-dimensional Fourier transform

Edge effects: vs. Homogeneity (scalar rule) Linear, Shift-Invariant Systems Neural activity fMRI response • Linearity: scalar rule and additivity • Applied to impulse, sums of impulses • Applied to sine waves, sums of sine waves

Additivity Shift invariance

Linear systems Shift invariance

A system (or transform) converts (or maps) an input signal into an output signal: y(t) = T[x(t)] For a system to be shift-invariant (or time-invariant) means that a time-shifted version of the input yields a time-shifted version of the A linear system satisfies the following properties: output: y(t) = T[x(t)] 1) Homogeneity (scalar rule): T(a x(t)] = a y(t) y(t - s) = T[x(t - s)] 2) Additivity: The response y(t - s) is identical to the response y(t), except that it T(x1(t) + x2(t)] = y1(t) + y2(t) is shifted in time. Often, these two properties are written together and called superposition:

T(a x1(t) + b x2(t)] = a y1(t) + b y2(t)

Neural Image - Reprise Linear, Shift-Invariant Systems

A spatial receptive field may also be treated as a linear • Linearity: Scalar rule and additivity system, by assuming a dense collection of neurons with the same receptive field translated to different locations in the • Applied to impulse, sums of impulses visual field. In this view, it is a linear, shift-invariant system. • Applied to sine waves, sums of sine waves

Convolution as sum of impulse responses Convolution as sum of impulse responses

Input: Input S(t): 1 2 3 4 5 6 7 8 9 1011121314

Impulse response I(t):

$1I(t t 1): Impulse response: $2I(t t 2):

$3I(t t 3):

Output: $8I(t t 8):

$9I(t t 9):

$14I(t t 14):

14 3$i I(t ti): + i1

Convolution Convolution derivation

Homogeneity: Discrete-time signal: x[n] = [x1, x2, x3, ...] T{a x[n]} = a T{x[n]}

A system or transform maps an input signal into an output signal: Additivity: y[n] = T{x[n]} T{x [n] + x [n]} = T{x [n]} + T{x [n]} 1 2 1 2 A shift-invariant, linear system can always be expressed as a Superposition: convolution: T{a x [n] + b x [n]} = a T{x [n]} + b T{x [n]} 1 2 1 2 y[n] = 3 x[m] h[n-m] Shift-invariance: where h[n] is the impulse response. y[n] = T{x[n]} => y[n-m] = T{x[n-m]}

Convolution derivation (contd.) Convolution derivation (cont) x[n]: input Impulse sequence: y[n] = T{x[n]}: output d[n] = 1 for n = 0, d[n] = 0 otherwise h[n] = T{d[n]}: impulse response

Any sequence can be expressed as a sum of impulses: 1) Represent input as sum of impulses: x[n] = 3 x[m] d[n-m] y[n] = T{x[n]} y[n] = T{ x[m] d[n-m] } where 3 d[n-m] is impulse shifted to sample m 2) Use superposition: x[m] is the height of that impulse y[n] =3 x[m] T{d[n-m]} Example: 3) Use shift-invariance: = + + y[n] =3 x[m] h[n-m]

Convolution as sum of impulse responses Convolution as correlation with the “receptive field” (time-reversed impulse

Input S(t): 1 2 3 4 5 6 7 8 9 1011121314 response): 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Input S(t):

Impulse response I(t): RF 1(t):

$1I(t t 1): RF 1(t)S(t):

$2I(t t 2): RF 2(t):

$3I(t t 3): RF 2(t)S(t):

$8I(t t 8): RF 8(t):

RF (t)S(t): $9I(t t 9): 8

RF 9(t): $14I(t t 14):

RF (t)S(t): 14 9 3$i I(t ti): i1

F (t):

Convolution as matrix multiplication Convolution as sequence of weighted sums

Linear system <=> matrix multiplication Shift-invariant linear system <=> Toeplitz matrix

past present future 0 0 0 1 0 0 0 0 0 input (impulse)

1 1/8 1/4 1/2 weights 2 5 1 2 3 0 0 0 0 0 0 0 1/2 1/4 1/8 0 0 0 output (impulse response) 2 = 0 1 2 3 0 0 0 -3 0 0 1 2 3 0 -1 0 0 0 1 2 3 2 4 0 0 0 1 1 1 1 1 1 input (step)

1/8 1/4 1/2 weights

0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response)

Columns contain shifted copies of the impulse response. Figure 3: Convolution as a series of weighted sums. Rows contain time-reversed copies of impulse response.

Using homogeneity,

Now let be the response of to the unshifted unit impulse, i.e., . Then by using shift-invariance, (4)

Notice what this last equation means. For any shift-invariant linear system , once we know its impulse response (that is, its response to a unit impulse), we can forget about entirely, and just add up scaled and shifted copies of to calculate the response of to any input whatsoever. Thus any shift-invariant linear system is completely characterized by its impulse response . The way of combining two signals specified by Eq. 4 is know as convolution. It is such a widespread and useful formula that it has its own shorthand notation, . For any two signals and , there will be another signal obtained by convolving with ,

Convolution as a series of weighted sums. While superposition and convolution may sound a little abstract, there is an equivalent statement that will make it concrete: a system is a shift- invariant, linear system if and only if the responses are a weighted sum of the inputs. Figure 3 shows an example: the output at each point in time is computed simply as a weighted sum of the inputs at recently past times. The choice of weighting function determines the behavior of the system. Not surprisingly, the weighting function is very closely related to the impulse response of the system. In particular, the impulse response and the weighting function are time-reversed copies of one another, as demonstrated in the top part of the figure.

7 and Young (1983), and Oppenheim and Schafer (1989). and Young (1983), and Oppenheim and Schafer (1989).

Continuous-Tand Young (1983),imeand Oppenheimand Discrandete-TSchaferime(1989).Signals Continuous-Time and Discrete-Time Signals In each of the above examples there is an input and an output, each of which is a time-varying signal.Continuous-TWIneeachwill treatof thea signalaboimeve asexamplesaandtime-vDiscrtherearyingisfunction,ete-Tan inputimeand. anFSignalsoroutput,each timeeach, ofthewhichsignalishasa sometime-varying value signal., usuallyWe willcalledtreat“ aofsignal.” Sometimesas a time-varyingwe willfunction,alternatively.useFor eachtotimerefer, tothethesignalentirehas some signal v,aluethinking,ofusuallyas a freecalledvariable.“ of .” Sometimes we will alternatively use to refer to the entire In each of the above examples there is an input and an output, each of which is a time-varying In practice,signal , thinkingwill ofusuallyas abefreerepresentedvariable. as a finite-length sequence of numbers, , in signal. We will treat a signal as a time-varying function, . For each time , the signal has some which can take integer values between 0 and , and where is the length of the sequence. value In ,practice,usually calledwill“ usuallyof .” Sometimesbe representedwe willas aalternatifinite-lengthvely usesequencetoofrefernumbers,to the entire, in This discrete-time sequence is indexed by integers, so we take to mean “the nth number in signalwhich, thinkingcan takofe inteasgera freevaluesvariable.between 0 and , and where is the length of the sequence. sequenceThis,”discrete-timeusually calledsequence“ of ”isforindeshort.xed by integers, so we take to mean “the nth number in TheInsequenceindipractice,vidual,”numbersusuallywillcalledinusuallya sequence“ beof represented” for short.are calledas a finite-lengthsamples of thesequencesignal of numbers,. The word , in which can take integer values between 0 and , and where is the length of the sequence. “sample” comesThe indifromvidualthe fnumbersact that thein sequencea sequenceis a discretely-sampledare called samplesversionof theofsignalthe continuous. The word signal.ThisImagine,discrete-timefor example,sequencethatisyouindearexedmeasuringby integers,membraneso we takpotentiale (orto meanjust about“the annthythingnumber in “sample” comes from the fact that the sequence is a discretely-sampled version of the continuous else,sequencefor that matter),” usuallyas itcalledvaries“oveofr time.” forYshort.ou will obtain a sequence of measurements sampled signal. Imagine, for example, that you are measuring membrane potential (or just about anything at evenly spaced time intervals. Although the membrane potential varies continuously over time, Theelse,indifor vidualthat matter)numbersas itinvariesa sequenceover time. Yoareu willcalledobtainsamplesa sequenceof theofsignalmeasurements. Thesampledword you will work just with the sequence of discrete-time measurements. “sample”at evenlycomesspacedfromtimetheintervfact thatals. theAlthoughsequencetheismembranea discretely-sampledpotential variesversioncontinuouslyof the continuousover time, signal.It isyouoftenImagine,willmathematicallyworkforjustexample,withconthevthatsequenceenientyoutoareofworkdiscrete-timemeasuringwith continuous-timemembranemeasurements.potentialsignals.(orButjustinaboutpractice,anything youelse,usuallyfor endthat upmatter)with discrete-timeas it varies ovsequenceser time. Ybecause:ou will obtain(1) discrete-timea sequence samplesof measurementsare the onlysampled It is often mathematically convenient to work with continuous-time signals. But in practice, thingsat ethatvenlycanspacedbe measuredtime intervandals.recordedAlthoughwhenthedoingmembranea realpotentialexperiment;variesandcontinuously(2) finite-length,over time, you usually end up with discrete-time sequences because: (1) discrete-time samples are the only discrete-timeyou will wsequencesork just withare the sequenceonly thingsofthatdiscrete-timecan be storedmeasurements.and computed with computers. things that can be measured and recorded when doing a real experiment; and (2) finite-length, In whatItdiscrete-timeis follooftenws,mathematicallywesequenceswill expressareconthemostvonlyenientofthingsthetomathematicswthatork canwithbecontinuous-timeinstoredthe continuous-timeand computedsignals.withdomain.Butcomputers.inButpractice, the examples will, by necessity, use discrete-time sequences. you usuallyIn whatendfolloup ws,withwediscrete-timewill expresssequencesmost of thebecause:mathematics(1) discrete-timein the continuous-timesamples aredomain.the onlyBut thingsthethatexamplescan bewill,measuredby necessityand ,recordeduse discrete-timewhen doingsequences.a real experiment; and (2) finite-length, discrete-time sequences are the only things that can be stored and computed with computers. Pulse and impulse signals. The unit impulse signal, written , is one at , and zero everywhereIn whatelse:follows, we will express most of the mathematics in the continuous-time domain. But Pulse and impulse signals. The unit impulse signal, written , is one at , and zero the examples will, by necessity, use discrete-time sequences. everywhere else: Pulses and impulses Continuous-time derivation of convolutionThe impulse signal will play a very important role in what follows. Pulse and impulse signals. The unit impulse signal, written , is one at , and zero eOneverywhereTheveryimpulseusefulelse:signalway towillthinkplayofatheveryimpulseimportantsignalroleisinaswhata limitingfollows.case of the pulse signal, : One very useful way to think of the impulse signal is as a limiting case of the pulse signal, Im pu ls es : Im pu lIsmespu ls es Im pu ls e Im pu ls e Im pu ls e ReIsmpopnuslsee Res pon s e The impulse signal will play a very importantIm pu ls e roleIminpu lswhate Res pon sfolloe ws. The impulse signal is equal to the pulse signal when the pulse gets infinitely short: One very useful way to think of the impulse signal is as a limiting case of the pulse signal, The impulse signal is equal to the pulse signal when the pulse gets infinitely short: : ImEpauchl sImespu lEsae cChr eIma tpeus las e Crea tes a ESa cha l eImd paun dls SeS chCairlfeteedad tae Insm dap Suhlsifet eRde Ismpopnuslsee Res pon s e ScaIlmedp ua nlsde Sh ifted Im puImls ep uRlessep Roensspeon s e

For For The impulse signal is equal to the pulse signalFor exawhenm ple exathem ple pulse gets infinitely short: Eexaachm 2pImlepu ls e Crea tes a Sca led a n d Sh ifted Im pu ls e Res pon s e

Th e s u m of Tahlle tshuem im2 opf uallsl et hre sipmopnuslesse res pon s es Th e s u m of ailsl tthhee ifminpaul ilssy est htreems fpi nroeansls pseoysns tseem res pon s e is th e fin a l s ys tem res pon s e For exa m ple

2 Figure 2Figur: CharacterizingTh ee s u2m: oCharacterizingf a ll th e aimlinearpu ls e resystemsapolinearn s es usingsystemitsusingimpulseits impulseresponse.response. Figure 2: Characterizingis th e afinlineara l s ys tesystemm res pon susinge its impulse response. The wayThewewuseaytheweimpulseuse the impulseresponseresponsefunctionfunctionis illustratedis illustratedin Fig. 2.inWFig.e concei2. Weveconceiof theveinputof the input = + + + + ...... Thestimulus,way wstimulus,einusethisthecaseinimpulsethisa sinusoid,caseresponsea sinusoid,as functionif it wereas ifistheitillustratedweresumtheof asuminsetFig.ofof2.aimpulsessetWeofconceiimpulses(Eq.ve1).of(Eq.theWeinput1).knowWetheknow the stimulus,responsesin thisresponseswecasewoulda sinusoid,wegetwouldif eachasgetifimpulseifiteachwereimpulsewtheas presentedsumwofas presenteda setseparatelyof impulsesseparately(i.e., scaled(Eq.(i.e.,1).andscaledWshiftede knoandwcopiesshiftedthe ofcopies of = + + + + responses we would get if each impulse was presented separately (i.e., scaled and shifted copies of ...... the impulsethe impulseFigurresponse).e 2:response).CharacterizingWe simplyWeaddsimplya lineartogetheraddsystemtogetherall usingof theallits(scaledofimpulsethe and(scaledresponse.shifted)and shifted)impulseimpulseresponsesresponsesto to = + + + + the impulsepredictresponse).hopredictw thehosystemWwethesimplywillsystemrespondaddwilltogetherrespondto theallcompletetoofthethecomplete(scaledstimulus.andstimulus.shifted) impulse responses to Figure...1: Staircase approximation to a continuous-time signal. ... predict how the system will respond to the complete stimulus. The waNoy wewuseeNowillthew repeatwimpulsee willallrepeatthisresponseinallmathematicalthisfunctionin mathematicalisnotation.illustratednotation.Ourin Fig.goalOur2.is toWgoalshoe conceiiswtothatshovethewofthatresponsethe theinputresponse(e.g., (e.g., Figure 1: Staircase approximation to a continuous-time signal. stimulus,Nomembranew wine willthismembranerepeatcasepotentialaallsinusoid,potentialthisfluctuation)in mathematicalasfluctuation)if itofwerea shift-inofthenotation.avsumshift-inariantofOurlinearvaariantsetgoalofsystemlinearisimpulsesto sho(e.g.,systemw (Eq.thatpassi(e.g.,the1).veresponsepassiWneurale knovemembrane)neuralw(e.g.,the membrane) Representing signals with impulses. Any signal can be expressed as a sum of scaled and responsesmembranecan webepotentialwrittenwcanouldbegetfluctuation)aswrittenaifsumeachasofaimpulsescaledofsuma shift-inofandwscaledasshiftedpresentedvariantandcopiesshiftedlinearseparatelyofsystemcopiesthe system’(i.e.,of(e.g.,thescaledspassisystem’impulsevandesneuralimpulseshiftedresponsemembrane)copiesresponsefunction.of function. Figure 1: Staircase approximation to a continuous-time signal. shifted unit impulses. We begin with the pulse or “staircase” approximation to a continuous thecanimpulsebe writtenresponse).as a sumWofescaledsimplyandaddshiftedtogethercopiesall ofofthethe(scaledsystem’ands impulseshifted)responseimpulse responsesfunction. to RepresentingStaircasesignals with impulses.approximationAny signal can tobe econtinuous-xpressed as a sum of scaled and predict how the system will respond to the completeConvolutionstimulus. signalshifted, aunits illustratedimpulses.inWFig.e be1.ginConceptuallywith the pulse, thisor “staircase”is trivial: forapproximationeach discrete sampleto a continuousof the The conThevolutionconvolutionintegral.integral.Begin byBeusinggin byEq.using1 toEq.replace1 to replacethe inputthesignalinput signalby its repre-by its repre- Representing signals with impulses.timeAny signalsignalcan be expressed as a sum of scaled and TheNosentationwconwevwillolutionsentationinrepeatRepresentingtermsintegral.allinoftermsthisimpulses:inofmathematicalBeimpulses: gintheby inputusingnotation. Eq.signal1 toOur replaceasgoal a sumtheis toinput shoof wpulses:signalthat the responseby its repre-(e.g., originalsignalsignal, we, asmakillustratede a pulsein signal.Fig. 1. ThenConceptuallywe add,upthisallisthesetrivial:pulsefor signalseach discreteto maksamplee up theof the shifted unit impulses. We begin with the pulse or “staircase” approximation to a continuous sentationmembraneinpotentialterms offluctuation)impulses: of a shift-invariant linear system (e.g., passive neural membrane) approximateoriginalsignal.signal, Eachwe makofethesea pulsepulsesignal.signalsThencanweinaddturnupbeallrepresentedthese pulseassignalsa standardto makpulsee up the can be written as a sum of scaled and shifted copies of the system’s impulse response function. signal , as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the scaledapproximateby the appropriatesignal.valueEachandof theseshiftedpulseto thesignalsappropriatecan inplace.turn beIn representedmathematicalasnotation:a standard pulse original signal, we make a pulse signal. Then we add up all these pulse signals to make up the past present future scaled by the appropriate v=alue and+ shifted to the appropriate+ place.+ In mathematical+ notation: The convolution 0integral. 0 0 1 Be 0gin 0 by 0 using 0 0Eq. 1inputto replace (impulse)the input signal by its repre- approximate signal. Each of these... pulse signals can in turn be represented as a standard... pulse sentation in terms of impulses:1/8 1/4 1/2 weights scaled by the appropriate value and shifted to the appropriate place. In mathematical notation: Using additiUsingUsingvityadditi, additivityvity, , 0 0 0 1/2 1/4 1/8 0 0 0 output (impulse response) Figure 1: Staircase approximation to a continuous-time signal. Using additivity, As we let approach zero, the approximation becomes better and better, and the in the limit Representing signals with impulses. Any signal can be expressed as a sum of scaled and Taking theTakinglimit,the0 0limit, 0 1 1 1 1 1 1 input (step) equalsAs we. Therefore,let approachshifted unit impulses.zero, theWapproximatione begin with the pulse or “staircase”becomesapproximationbetter and betterto,aandcontinuousthe in the limit Taking the limit, Taking the limit, 1/8 1/4 1/2 weights equals . Therefore,signal , as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the As we let approachoriginal signal,zero,wethemakapproximatione a pulse signal. Then webecomesadd up allbetterthese pulseand signalsbetterto, andmaketheup thein the limit Using additivity, 0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response) approximate signal. Each of these pulse signals can in turn be represented as a standard pulse 6 6 equals . Therefore, Figure 3: Convolution as a series of weighted sums. scaled by the appropriate value and shifted to the appropriate place. In mathematical notation: 6 Also, as , the summation approaches an integral, and the pulse approaches the unit impulse: Using homogeneity (scalar rule), Taking the limit, Also, as , the summation approaches an integral, and the pulse approaches the unit impulse: Using homogeneity, (1) Also, as , the summation approaches an inte gral, and the pulse approaches the unit impulse:(1) As we let approach zero, the approximation becomes better and better, and the in the limit Now let be theDefiningresponse of h(t)to asthe theunshifted impulse6unit impulse, response,i.e., . Then by using In other words, we equalscan represent. Therefore,any signal as an infinite sum of shifted and scaled unit impulses. A shift-invariance, digitalIncompactother words,disc, wefor canexample,representstoresanywholesignalcompleas an infinitex piecessumof ofmusicshiftedas lotsandofscaledsimpleunitnumbersimpulses.(1)A (4) representingdigital compactvery shortdisc,impulses,for example,and thenstoresthewholeCD playercompleaddsx piecesall theofimpulsesmusic as backlots oftogethersimple numbersone Notice what this last equation means. For any shift-invariant linear system , once we know its afterInanotherotherrepresentingwords,to recreatewevAlso,erycanasshorttherepresentcompleimpulses,, the summationxanmusicalyandsignalapproachesthenwasavtheaneform.infiniteanCDinteplayergral,sumandaddstheof pulseshiftedall approachestheandimpulsesscaledthe unitbackunitimpulse:impulses.together oneA impulse response (that is, its response to a unit impulse), we can forget about entirely, and digital compact disc, for example, stores whole complex pieces of music as lots of simple numbers just add up scaled and shifted copies of to calculate the response of to any input whatsoever. after another to recreate the complex musical waveform. (1) Thus any shift-invariant linear system is completely characterized by its impulse response . representingThis no doubtveryseemsshortlikimpulses,e a lot ofandtroublethen theto goCDto,playerjust toaddsgetallbackthetheimpulsessame signalback togetherthat we one The way of combining two signals specified by Eq. 4 is know as convolution. It is such a originallyThisstartedno with,doubtIn otherbuseemswtords,in fact,welikcanewearepresentwilllot ofvanerytroubley signalshortlyastoanbegoinfiniteableto, sumjustto ofusetoshiftedEq.getand1backtoscaledperformtheunitsameimpulses.a marvsignalAelousthat we after another to recreate the complex musical waveform. widespread and useful formula that it has its own shorthand notation, . For any two signals and originally starteddigitalwith,compactbudisc,t in forfact,example,we willstoresverywholeshortlycomplebex piecesableoftomusicuse asEq.lots1oftosimpleperformnumbersa marvelous trick. , there will be another signal obtained by convolving with , trick.This no doubtrepresentingseems vlikeryeshorta lotimpulses,of troubleand thentothegoCDto,playerjustaddstoallgetthebackimpulsesthebacksametogethersignalone that we originally startedafterwith,anotherbuttoinrecreatefact, thewecomplewill vx erymusicalshortlywaveform.be able to use Eq. 1 to perform a marvelous trick. This no doubt seems like a lot of trouble to go to, just to get back the same signal that we Linear Systemsoriginally started with, but in fact, we will very shortly be able to use Eq. 1 to perform a marvelous Linear Systemstrick. Convolution as a series of weighted sums. While superposition and convolution may sound a little abstract,Shift-invariantthere is an equivalent statementlinearthat will make it concrete: a system is a shift- A system or transform mapsLinearan input signal, Shift-Invariantinto an output signal Systems: invariant, linear system if and only if the responses are a weighted sum of the inputs. Figure 3 Linear Systems shows an example:systemsthe output andat each impulsespoint in time is computed simply as a weighted sum of the A system or transformLinear Systemsmaps an input signal into an output signal : inputs at recently past times. The choice of weighting function determines the behavior of the system. Not surprisingly, the weighting function is very closely related to the impulse response of A system or transform•A systemLinearity:mapsor transforman inputmaps Scalaransignalinput signal ruleinto intoandananoutputoutput additivitysignalsignal : : the system. In particular, the impulse response and the weighting function are time-reversed copies where denotes the transform, a function from input signals to output signals. of one another, as demonstrated in the top part of the figure. where denotes• Appliedthe transform, toa impulse,function from inputsumssignals ofto impulsesoutput signals. Systems come in a wide variety of types. One important class is known as linear systems. To Systems comewhere indenotesa widethevtransform,ariety ofa functiontypes. fromOneinputimportantsignals toclassoutputissignals.known as linear systems. To see whether a system• Appliedis linear, we needto sineto test whetherwaves,it obe sumsys certain ofrules sinethat allwaveslinear systems wheresee whetherdenotesa systemtheSystemstransform,iscomelinearin a,awwidefunctione needvarietytofromoftesttypes.whetherinputOne importantsignalsit obetoclassysoutputcertainis knownsignals.rulesas linearthatsystemsall linear. To systems obey. The two basicseetestswhetherof alinearitysystem is linearare homogeneity, we need to test whetherand additiit obeysvitycertain. rules that all linear systems obeSystemsy. Thecometwoobebasicyin. Thea testswidetwo basicofvarietylinearitytests ofoflinearitytypes.are homogeneityareOnehomogeneityimportantandand additiadditiclassvityvityis. kno. wn as linear systems. To 7 see whether a system is linear, we need to test4whether it obeys certain rules that all linear systems obey. The two basic tests of linearity are homogeneity4 4 and additivity.

4 Example – Bass/Treble filters Convolution and multiplication

Signal Frequency content time sample Miles Davis Fourier transform “Half Nelson” Convolution: frequency sample

Filter x1[n] * x2[n] X1[k] X2[k]

Bass only: Low-pass or Bass Filter Multiplication: convolution

x1[n] x2[n] (1/N) X1[k] * X2[k] Treble only: High-pass or Treble Filter

Time Prev Next

The Big Payoff The Big Payoff input signal input transform

The signal: f ! F F The system T has impulse response i ! I ! cross-correlated with convolved with ! multiplied by The system's response is T(f ) = i " f ! IF ! In other words, the Fourier transform of the impulse ! F response is the modulation transfer function (MTF). receptive field impulse response! filter MTF = = The corresponding receptive field is the time- or space-reversed impulse response. !F output signal output transform 1-D Example: Gaussian Blur 1-D Example: Bandpass (DOG) Filter

F F input signal ! input signal !

F F impulse response ! impulse response !

F F output signal ! output signal !

1-D Example: Bandpass (Gabor) Filter

F ? input signal !

? F impulse response !

? F output signal ! Gabor function

1-d Example: fMRI Block alternation with Fourier transform of response with noise noise & drift and drift

1 1 0.8 HIRF 0.8 0.6 HIRF

activity 0.6 0.4 0.4

Relative neural 0.2

0 0.2 0 10 20 30 40 50 60 70 80 90 100

of neural activity 0 Relative amplitude 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1

+ noise + drift 0.4 0.5 + noise + drift 0.3 0

0.2 -0.5 fMRI response amplitude

(% change image intensity) 0.1 -1

0 10 20 30 40 50 60 70 80 90 100 fMRI response 0 Time (sec) 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Frequency (Hz) Bandpass  2-D Example: Gaussian Blur filtering   to remove  F input signal noise and fMRI response           ! drift Time (sec) 





amplitude  F • How do you make a fMRI response impulse response  low-pass filter?            How do you make a Frequency (Hz) ! •  high-pass filter?

 F

fMRI response output signal           Time (sec) !

2-D Example: Bandpass (DOG) Filter 2-D Example: Bandpass (DOG) Filter

F F input signal ! input signal !

F F impulse response ! impulse response !

F F output signal ! output signal !

2-D Example: Gabor Filter Applications: Line Spread Function (Campbell & Gubisch, 1966) F input signal !

F impulse response !

F output signal ! Applications: Line Spread Function Point-Spread Function (Campbell & Gubisch, 1966)

Applications: Multiple Mechanisms Applications: Multiple Mechanisms (Campbell & Robson, 1968) (Campbell & Robson, 1968)

Square wave

Near Contrast Sensitivity Function (CSF) Sine wave

Far

Ratio and peak- detector model

Applications: Multiple Mechanisms Applications: Summation Within and Between (Campbell & Robson, 1968) Channels (Graham & Nachmias, 1971)

f

CSF

3f

CSF shifted 3x left & down

Square vs sine Peaks discrimination Subtract

Peaks Add

Single-channel Multiple-channel Stimulus prediction prediction Applications: Summation Within and Between Applications: Summation Within and Between Channels (Graham & Nachmias, 1971) Channels (Graham & Nachmias, 1971)

Applications: SF Adaptation (Blakemore & Applications: SF Adaptation (Blakemore & Campbell, 1969) Campbell, 1969)

Applications: SF Adaptation (Blakemore & Applications: SF Adaptation (Blakemore & Sutton, 1969) Sutton, 1969) Applications: Pattern Masking Applications: Pattern Masking (Wilson et al., 1983) (Wilson et al., 1983) • Mask one pattern (Gabor, D6, ...) by another (e.g., a sine wave grating, tilted obliquely)

• Threshold is raised by the masker if channel being used is sensitive to both

• Many possible explanations of the rise in threshold with masker contrast:

- Weber’s Law, possibly as a result of multiplicative noise (noise whose SD is proportional to mean response): !I = k I - Nonlinearity followed by additive noise

Applications: Pattern Masking Applications: Pattern Masking (Wilson et al., 1983) (Wilson et al., 1983)

Applications: Pattern Masking Applications:Detection vs. Identification, (Wilson et al., 1983) Labeled Lines (Watson & Robson, 1981) Applications:Detection vs. Identification, Applications:Detection vs. Identification, Labeled Lines (Watson & Robson, 1981) Labeled Lines (Watson & Robson, 1981)

Applications:Detection vs. Identification, Applications:Detection vs. Identification, Most Discriminating Mechanism Most Discriminating Mechanism (Regan & Beverley, 1985) (Regan & Beverley, 1985)

Applications:Sampling and Reconstruction Applications:Sampling and Reconstruction

1.2 1

1 0.8 20 Band-limited 0.8 signal 0.6 15 0.6 10 0.4 0.4

0.2 0.2 5

0 0 10 20 30 40 50 60 0 0 −20 0 20 −20 0 20 Time

1 6 15 Supra-Nyquist Sufficient 0.8 sampling 5 sampling

10 4 0.6 3 5 0.4 2

0.2 1 0 0 10 20 30 40 50 60 0 0 Frequency 0 20 40 60 −20 0 20 Time Frequency Applications:Sampling and Reconstruction Applications:Sampling and Reconstruction

1 8 1 1 0.8 6 0.5 0.5 0.6 4 0 0 0.4

2 −0.5 −0.5 0.2

0 0 −1 −1 −20 0 20 −20 0 20

sampled function 1 2 Supra-Nyquist Insufficient 0.8 1 1 sampling 1.5 sampling 0.6 0.5 0.5 1 0.4 0 0 0.5 0.2 −0.5 −0.5 0 0 0 20 40 60 −20 0 20 −1 −1 Time Frequency 0 20 40 60 0 20 40 60

Applications:Sampling and Reconstruction Applications:Sampling and Reconstruction

1.2

20 Reconstruction of 15 1 sufficiently band-limited signal 10 0.8 5 Reconstruction of 0 0.6 −30 −20 −10 0 10 20 30 insufficiently band-limited signal 0.4 6

5 0.2 4

3 0 2

1 −0.2 0 −30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30 Frequency Time

Applications:Sampling and Reconstruction Applications:Sampling and Reconstruction

sinc(x) = sin(x)/x Box Filter

F !F

Time Frequency Sample spacing Applications:Sampling and Reconstruction Applications:Sampling and Reconstruction

F F

Applications:Sampling and Reconstruction Applications:Sampling and Reconstruction (Yellott, 1983) (Yellott, 1983)

Fovea Parafovea (approx. 6 deg ecc.)

Applications:Sampling and Reconstruction (Yellott, 1983)

Periphery (approx. 35 deg ecc.)