Spatial Pattern Vision and Linear Systems Theory Receptive Field

Spatial pattern vision and linear systems theory

• Receptive fields and neural images • Shift-invariant linear systems and convolution • Fourier transform and frequency response • Applications of linear systems to spatial vision – Contrast sensitivity – Spatial frequency and orientation channels – Spatial frequency and orientation adaptation – Masking

Receptive field

• In any modality: that region of the sensory apparatus that, when stimulated, can directly affect the firing rate of a given neuron • Spatial vision: spatial receptive field can be mapped in visual space or on the retina • Examples:

LGN V1

Receptive field

A spatial receptive field is an image: Orientation selective receptive field

No stimulus in receptive field: no response

Preferred stimulus: large response

Non-preferred stimulus: no response

Neural image: retinal ganglion cell responses

Input image Array of center-surround “Neural image” (cornea) receptive fields (retinal ganglion cells)

Neural image: simple cell responses

Input image Array of orientation- “Neural image” (cornea) selective receptive fields (V1 simple cells) Lots of neural images: V1 simple cells

Lots of neural images

Simple cells Complex cells

Neural image of a sine wave

For a linear, shift-invariant system such as a linear model of a receptive field, an input sine wave results in an identical output sine wave, except for a possible lateral shift and scaling. Frequency response

This scaling of contrast by a linear receptive field in the neural image is a function of spatial frequency determined by the shape of the receptive field. Contrast Gain

Spatial Frequency

Frequency response

This scaling of contrast by a linear receptive field in the neural image is a function of spatial frequency determined by the shape of the receptive field. Contrast Gain

Spatial Frequency

Orientation tuning

If a receptive field is not circularly symmetric, the scaling of contrast is also a function of orientation (for a given spatial frequency) determined by the shape of the receptive field. Contrast Gain

Orientation Linear systems analysis

Measure the impulse response Space/time method

Express as sum Calculate the Sum the of shifted and response to impulse scaled impulses each impulse responses Input stimulus Express as sum Calculate the Sum the of shifted and response to sinusoidal scaled sinusoids each sinusoid responses

Measure the Frequency method frequency response

Spatial pattern vision and linear systems theory

Functions and Systems

A function: a relation between a set of inputs and a set of permissible outputs with the property that each input is related to exactly one output.

Examples

• f(x) = x^2 • f(x) = cos(x) • f(x,y) = sqrt(x^2 + y^2)

wikipedia Functions and Systems

A system: a generalization of a function whereby inputs and outputs can be numbers or objects, discrete or continuous, any dimensionality

Examples

• Physiological optics: Input: scene spectral irradiance; Output: retinal image • Eye: Input: scene spectral irradiance; Output: retinal ganglion cell ﬁring rate • Human: Input: scene spectral irradiance; Output: knob adjustment (eg color match) • fMRI: Input: neuronal activity Output: T2*w image

Linear Systems Analysis

Systems with signals as input and output

• 1-d: low- and high-pass filters in electronic equipment, fMRI data analysis, or in sound production (articulators) or audition (the ear as a filter) y(t) = T{x(t)} • 2-d: optical blur, spatial receptive field g(x,y) = T{f(x,y)} • 3-d: spatio-temporal receptive field g(x,y,t) = T{f(x,y,t)}

Homogeneity (scaling)

Visual stimulus RGC firing rate Additivity

Visual stimulus RGC firing rate

Shift invariance

Visual stimulus RGC firing rate

Shift-invariant linear systems and impulses Convolution

Discrete-time signal: x[n] = [x1, x2, x3, ...]

A system or transform maps an input signal into an output signal: y[n] = T{x[n]}

A shift-invariant, linear system can always be expressed as a convolution: y[n] = x[m] h[n-m] m

where h[n] is the impulse response.

Convolution as sum of impulse responses

Input:

Impulse response:

Output:

Convolution as sequence of weighted sums

past present future 0 0 0 1 0 0 0 0 0 input (impulse)

1/8 1/4 1/2 weights

0 0 0 1/2 1/4 1/8 0 0 0 output (impulse response)

0 0 0 1 1 1 1 1 1 input (step)

1/8 1/4 1/2 weights

0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response)

Figure 3: Convolution as a series of weighted sums.

Using homogeneity,

Now let be the response of to the unshifted unit impulse, i.e., . Then by using shift-invariance, (4)

Notice what this last equation means. For any shift-invariant linear system , once we know its impulse response (that is, its response to a unit impulse), we can forget about entirely, and just add up scaled and shifted copies of to calculate the response of to any input whatsoever. Thus any shift-invariant linear system is completely characterized by its impulse response . The way of combining two signals speciﬁed by Eq. 4 is know as convolution. It is such a widespread and useful formula that it has its own shorthand notation, . For any two signals and , there will be another signal obtained by convolving with ,

Convolution as a series of weighted sums. While superposition and convolution may sound a little abstract, there is an equivalent statement that will make it concrete: a system is a shift- invariant, linear system if and only if the responses are a weighted sum of the inputs. Figure 3 shows an example: the output at each point in time is computed simply as a weighted sum of the inputs at recently past times. The choice of weighting function determines the behavior of the system. Not surprisingly, the weighting function is very closely related to the impulse response of the system. In particular, the impulse response and the weighting function are time-reversed copies of one another, as demonstrated in the top part of the ﬁgure.

7 Convolution as matrix multiplication Linear system <=> matrix multiplication Shift-invariant linear system <=> Toeplitz matrix

1 5 1 2 3 0 0 0 2 2 0 1 2 3 0 0 0 -3 = 0 0 1 2 3 0 0 4 0 0 0 1 2 3 -1 2

Columns contain shifted copies of the impulse response. Rows contain time-reversed copies of impulse response.

Convolution as matrix multiplication

0 3 1 2 3 0 0 0 0 2 0 1 2 3 0 0 1 1 = 0 0 1 2 3 0 0 0 0 0 0 1 2 3 0 0

Columns contain shifted copies of the impulse response. Rows contain time-reversed copies of impulse response.

Derivation: shift-invariant linear system => convolution

Homogeneity: T{a x[n]} = a T{x[n]}

Additivity:

T{x1[n] + x2[n]} = T{x1[n]} + T{x2[n]}

Superposition:

T{a x1[n] + b x2[n]} = a T{x1[n]} + b T{x2[n]}

Shift-invariance: y[n] = T{x[n]} => y[n-m] = T{x[n-m]} Convolution derivation (cont)

Impulse sequence: d[n] = 1 for n = 0, d[n] = 0 otherwise

Any sequence can be expressed as a sum of impulses: x[n] = x[m] d[n-m] m

where d[n-m] is impulse shifted to sample m x[m] is the height of that impulse

Example:

= + +

Convolution derivation (cont) x[n]: input y[n] = T{x[n]}: output h[n] = T{d[n]}: impulse response

1) Represent input as sum of impulses: y[n] = T{x[n]} y[n] = T{ x[m] d[n-m] } m

2) Use superposition: y[n] = x[m] T{d[n-m]} m

3) Use shift-invariance: y[n] = x[m] h[n-m] m

Matrix multiplication => scaling

NxN Toeplitz Nx1 vector matrix Nx1 vector

y1 x1 x2 y2 Impulse y3 x3 = response

yN xN

αy1 αx1

αy2 αx2 Impulse αy3 αx3 = response

αyN αxN Matrix multiplication => additivity

w1 y1 x1 z1 w2 y2 x2 z2 Impulse Impulse w3 y3 x3 and z3 = response = response

wN yN xN zN

x1 w1 y1 z1 x2 w2 y2 z2 Impulse y3 z3 x3 + w3 + = response

xN wN yN zN

Toeplitz matrix => shift invariance

0 3 1 2 3 0 0 0 0 2 0 1 2 3 0 0 1 1 = 0 0 1 2 3 0 0 0 0 0 0 1 2 3 0 0

Neural image and convolution

A spatial receptive field may also be treated as a linear system, by assuming a dense collection of neurons with the same receptive field translated to different locations in the visual field. In this view, it is a linear, shift-invariant system*§.

* This is the basis of CNNs (convolutional neural networks). § The nervous system doesn’t actually work like this. (It’s not linear and it’s not shift invariant!) Neural image and convolution

0.04

0.03

0.02

0.01

-0.01 20 20 10 10

0 0

% Make a Difference of Gaussian receptive field using fspecial DoG = fspecial('gaussian', 20,2) - fspecial('gaussian', 20,5); im = imread('cameraman.tif');

% Make a neural image by convolution neuralim = conv2(double(im), DoG);

% Show the image, the RF, and the neural image figure, subplot(1,3,1), imshow(im), subplot(1,3,2), surf(DoG) subplot(1,3,3), imagesc(neuralim); colormap gray, axis image off

Neural image and convolution

0.04

0.03

0.02

0.01

-0.01 20 20 10 10

0 0 % Shift the image and replot im2 = im(:, [n:end 1:n]); neuralim2 = conv2(double(im2), DoG);

% Make a Difference of Gaussian receptive field using fspecial 0.04 DoG = fspecial('gaussian', 20,2) - fspecial('gaussian', 20,5); im = imread('cameraman.tif'0.03);

0.02 % Make a neural image by convolution neuralim = conv2(double(im),0.01 DoG);

% Show the image, the RF, and0 the neural image figure, subplot(1,3,1), imshow(im),-0.01 subplot(1,3,2), surf(DoG) subplot(1,3,3), imagesc(neuralim);20 colormap gray, axis image off 20 10 10

0 0

Continuous-time derivation of convolution and Young (1983), and Oppenheim and Schafer (1989). and Young (1983), and Oppenheim and Schafer (1989). Continuous-Time and Discrete-Time Signals and Young (1983), and Oppenheim and Schafer (1989). Continuous-Time and Discrete-Time Signals In each of the above examples there is an input and an output, each of which is a time-varying signal. We will treat a signal as a time-varying function, . For each time , the signal has some In each of the above examples there is an input and an output, each of which is a time-varying valueContinuous-T, usually calledime“ of and.” SometimesDiscrete-Twe will imealternatiSignalsvely use to refer to the entire signal. We will treatLineara signal systemsas a time-varying requirementsfunction, . For each time , the signal has some signal , thinking of as a free variable. value , usually called “ of .” Sometimes we will alternatively use to refer to the entire InIn eachpractice,signalof the, thinkingabowillve ofusuallyexamplesA assystemabefreethererepresented (orvariable. istransform)an inputas a finite-lengthand convertsan output, (orsequence maps)each ofanof whichinputnumbers, signalis a time-v into,in arying whichsignal.canWtake wille intetreatgeravaluessignalan outputbetweenas a time-v signal:0 andarying function,, and where . Foris theeachlengthtimeof, the signalsequence.has some In practice, will usually be represented as a finite-length sequence of numbers, ,in Thisvaluediscrete-time, usuallysequencecalledis“indeofxe.”d ySometimesb(yt) inte= T[gers,x(t)] weso wewilltakalternatie vtoelymeanuse “the nthto refernumberto thein entire which can take inte ger v alues between 0 and , and where is the length of the sequence. sequencesignal ,,”thinkingusually calledof as“a offree”vforariable.short. This discrete-time sequence is indexed by integers, so we take to mean “the nth number in TheInsequenceindipractice,vidual,”numbersusuallywillAcalledin usuallylineara sequence“ systembeof represented” for satisfiesshort.are calledas thea finite-lengthsamples followingof properties.thesequencesignal of .numbers,The word ,in “sample”which comescan takfrome intethegerfact1)v Homogeneityaluesthat thebetweensequence (scalar0 andis a discretely-sampledrule):, and where versionis the oflengththe continuousof the sequence. The individual numbers in a sequence are called samples of the signal . The word signal.ThisImagine,discrete-timefor example,sequence that is youinde areT[xead measuringx(bty)] inte= a ygers,(tmembrane) so we takpotentiale (orto meanjust about“the annthythingnumber in “sample” comes from the fact that the sequence is a discretely-sampled version of the continuous else,sequencefor that matter),” usuallyas itcalledvaries2) Additivity:“oveofr time.” for Yshort.ou will obtain a sequence of measurements sampled signal. Imagine, for example, that you are measuring membrane potential (or just about anything at evenly spaced time interv als. Although T[xthe(t) +membrane x (t)] = y (potentialt) + y (t) varies continuously over time, else, for that matter) as it varies ove1r time.2 You will1 obtain2 a sequence of measurements sampled you willTheworkindijustvidualwithnumbersthe sequencein a ofsequencediscrete-timearemeasurements.called samples of the signal . The word “sample”at evenlycomesspacedfromtimetheintervfact thatals. theAlthoughsequencetheismembranea discretely-sampledpotential variesversioncontinuouslyof the continuousover time, signal.It isyouoftenImagine,willmathematicallyworkforjustexample,withOften,conthevthatsequence enienttheseyou totwoareofwork propertiesdiscrete-timemeasuringwith continuous-time membranearemeasurements. writtenpotential togethersignals.(orBut andjustin calledaboutpractice, anything youelse,usuallyfor thatend upmatter)with asdiscrete-timesuperposition:it varies ovsequenceser time. Ybecause:ou will obtain(1) discrete-timea sequence samplesof measurementsare the onlysampled It is often mathematically convenient to work with continuous-time signals. But in practice, things that can be measured and recorded when doing a real experiment; and (2) finite-length, at evenly spaced time interv als. AlthoughT[a x1(t)the + b membranex2(t)] = a y1(potentialt) + b y2(t)varies continuously over time, discrete-timeyou usuallysequencesend uparewiththe onlydiscrete-timethings thatsequencescan be storedbecause:and computed(1) discrete-timewith computers.samples are the only youthingswill wthatork justcanwithbe measuredthe sequenceand recordedof discrete-timewhen doingmeasurements.a real experiment; and (2) finite-length, In whatItdiscrete-timeis follooftenws,mathematicallywesequenceswill expressareconthemostvonlyenientofthingsthetomathematicswthatork canwithbecontinuous-timeinstoredthe continuous-timeand computedsignals.withdomain.Butcomputers.inButpractice, the examples will, by necessity, use discrete-time sequences. you usuallyIn whatendfolloup ws,withwediscrete-timewill expresssequencesmost of thebecause:mathematics(1) discrete-timein the continuous-timesamples aredomain.the onlyBut thingsthethatexamplescan bewill,measuredby necessityand ,recordeduse discrete-timewhen doingsequences.a real experiment; and (2) finite-length, discrete-timePulse and impulsesequencessignals.are theTheonlyunitthingsimpulsethatsignal,can bewrittenstored and ,computedis one at with computers., and zero everywhereIn whatelse:follows, we will express most of the mathematics in the continuous-time domain. But Pulse and impulse signals. The unit impulse signal, written , is one at , and zero the examplesverywherewill,else:by necessity, use discrete-time sequences. Pulses and impulses The impulse signal will play a very important role in what follows. Pulse and impulse signals. The unit impulse signal, written , is one at , and zero eOneverywhereTheveryimpulseusefulelse:signalway towillthinkplayofatheveryimpulseimportantsignalroleisinaswhata limitingfollows.case of the pulse signal, : One very useful way to think of the impulse signal is as a limiting case of the pulse signal, : The impulse signal will play a very important role in what follows. The impulse signal is equal to the pulse signal when the pulse gets infinitely short: One very useful way to think of the impulse signal is as a limiting case of the pulse signal, The: impulse signal is equal to the pulse signal when the pulse gets infinitely short:

The impulse signal is equal to the pulse signal when the pulse gets inﬁnitely short: 2

2 = ... + + ++... = ... + + ++... Figur= e...1: Staircase+ approximation+to a continuous-time signal.++... Figure 1: Staircase approximation to a continuous-time signal. Representing signalsFigurwithe 1:impulses.Staircase approximationAny signal tocana continuous-timebe expressed assignal.a sum of scaled and shifted unitReprimpulses.esentingWsignalse beginwithwith impulses.the pulse orAn“staircase”y signal canapproximationbe expressed as toa suma continuousof scaled and signalshifted, aunits illustratedStaircaseimpulses.inWFig.e be 1.ginapproximationConceptuallywith the pulse, thisor “staircase”is tri vial:to forapproximationeach discrete sampleto a continuousof the Representing signals with impulses. Any signal can be expressed as a sum of scaled and originalsignalsignal, we, asmakillustratede a pulsein signal.Fig. 1. ThenConceptuallywe add,upthisallisthesetrivial:pulsefor signalseach discreteto maksamplee up theof the shifted unit impulses.continuousWe begin withtimethe pulsesignalor “staircase” approximation to a continuous approximateoriginalsignal.signal, Eachwe makofethesea pulsepulsesignal.signalsThencanweinaddturnupbeallrepresentedthese pulseassignalsa standardto makpulsee up the signal , as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the scaledapproximateby the appropriatesignal.valueEachandof shiftedthese pulseto thesignalsappropriatecan inplace.turn beIn representedmathematicalasnotation:a standard pulse original signal, we make a pulse signal. Then we add up all these pulse signals to make up the scaled by the appropriate value and shifted to the appropriate place. In mathematical notation: approximate signal. Each of= these... +pulse signals can+ in turn be represented++as a standard... pulse scaled by the appropriate value and shifted to the appropriate place. In mathematical notation: Figure 1: Staircase approximation to a continuous-time signal. As we let approach zero, the approximation becomes better and better, and the in the limit Representing signals with impulses. Any signal can be expressed as a sum of scaled and equalsAs we. Therefore,let approachshifted unit impulses.zero, theWapproximatione begin with the pulse or “staircase”becomesapproximationbetter and betterto,aandcontinuousthe in the limit equals . Therefore,signal , as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the As we let approachoriginal signal,zero,wethemakapproximatione a pulse signal. Then webecomesadd up allbetterthese pulseand signalsbetterto, andmaketheup thein the limit equals . Therefore,approximate signal. Each of these pulse signals can in turn be represented as a standard pulse Also, as , thescaledsummationby the appropriateapproachesvalue andanshiftedintegral,to theandappropriatethe pulseplace.approachesIn mathematicalthenotation:unit impulse: Also, as , the summation approaches an integral, and the pulse approaches the unit impulse: (1) Also, as , the summation approaches an integral, and the pulse approaches the unit impulse: As we let approach zero, the approximation becomes better and better, and the in the limit (1) In other words, we equalscan represent. Therefore,any signal as an infinite sum of shifted and scaled unit impulses. A digitalIncompactother words,disc, wefor canexample,representstoresanywholesignalcompleas an infinitex piecessumof ofmusicshiftedas lotsandofscaledsimpleunitnumbersimpulses.(1)A representingdigital compactvery shortdisc,impulses,for example,and thenstoresthewholeCD playercompleaddsx piecesall theofimpulsesmusic as backlots oftogethersimple numbersone afterInanotherotherrepresentingwords,to recreatewevAlso,erycanasshorttherepresentcompleimpulses,, the summationxanmusicalyandsignalapproachesthenwasavtheaneform.infiniteanCDinteplayergral,sumandaddstheof pulseshiftedall approachestheandimpulsesscaledthe unitbackunitimpulse:impulses.together oneA digital compact disc, for example, stores whole complex pieces of music as lots of simple numbers after another to recreate the complex musical waveform. (1) representingThis no doubtveryseemsshortlikimpulses,e a lot ofandtroublethen theto goCDto,playerjust toaddsgetallbackthetheimpulsessame signalback togetherthat we one originallyThisstartednowith,doubtbuseemst in fact,likewea willlot ofverytroubleshortlyto begoableto, justto useto Eq.get 1backto performthe samea marvsignalelousthat we after another to Inrecreateother words,thewecomplecan representx musicalany signalwaasveform.an infinite sum of shifted and scaled unit impulses. A trick. originally starteddigitalwith,compactbudisc,t in fforact,example,we willstoresverywholeshortlycomplebex piecesableoftomusicuse asEq.lots1oftosimpleperformnumbersa marvelous trick.This no doubtrepresentingseems vlikeryeshorta lotimpulses,of troubleand thentothegoCDto,playerjustaddstoallgetthebackimpulsesthebacksametogethersignalone that we originally startedafterwith,anotherbuttoinrecreatefact, thewecomplewill vx erymusicalshortlywaveform.be able to use Eq. 1 to perform a marvelous trick. This no doubt seems like a lot of trouble to go to, just to get back the same signal that we Linear Systemsoriginally started with, but in fact, we will very shortly be able to use Eq. 1 to perform a marvelous Linear Systemstrick. A system or transform maps an input signal into an output signal : LinearA systemSystemsor transformLinear Systemsmaps an input signal into an output signal :

A system or transformA system mapsor transforman inputmaps ansignalinput signal intointoananoutputoutput signalsignal : : where denotes the transform, a function from input signals to output signals. where denotes the transform, a function from input signals to output signals. Systems come in a wide variety of types. One important class is known as linear systems.To where denotes the transform, a function from input signals to output signals. see whetherSystemsa systemcomeis linearin a wide, we vneedarietytooftesttypes.whetherOneitimportantobeys certainclassrulesis knothatwnallaslinearlinearsystemssystems.To where denotes the transform, a function from input signals to output signals. obey. seeThewhethertwo basica systemtestsSystemsofislinearitycomelinearin a, wwidearee needvhomogeneityarietytooftesttypes.whetherOneandimportantadditiit obevityclassys .certainis knownrulesas linearthatsystemsall linear.To systems obey. The twoseebasicwhethertestsa systemof linearityis linear, waree needhomogeneityto test whetherandit obeadditiys certainvityrules. that all linear systems Systems comeobeyin. Thea widetwo basicvarietytests ofoflinearitytypes.areOnehomogeneityimportantand additiclassvityis. known as linear systems.To see whether a system is linear, we need to test4whether it obeys certain rules that all linear systems obey. The two basic tests of linearity are homogeneity4 4 and additivity.

4 Shift invariance

Im pu ls es Im pu ls es Im pu ls es Im pu ls e Im pu ls e Res pon s e Im pu ls e ImImpupulsles e Res pon s e Im pu ls e Res pon s e For a system to be shift-invariant (or time-invariant) means that a time-shifted version of the input yields a time-shifted Im pu ls es version of the output:Ea ch Im pu ls e Crea tes a Im pEuaSlsceha l eImd paunlds eS hCirfeItmeadtEpe Iausmc lashpe uI mRlsepes upRloesnsep sCeornesaetes a y(tS)ca =led T[a n d xSh(itft)]ed SImcapluedls ea nRde sSphoinftseed Im pu ls e Res pon s e

y(t - s) = T[x(t - s)] For Ea cFho rIm peuxalsme pClerea tes Faor Sca led eaxnadm Sphliefted Im pu ls e xRaems ploen s e The response y(t - s) is identical to the response y(t), except

that it is shiftedTh ein s u mtime. of a ll th e im pu ls e res pon s es Th e s u m of a ilsl tthhee ifmTinhpaeul slssuyems tr eomsfp a orlenl stspheoesn ismepu ls e res pon s es is th e Ffionra l s ys tem riess tphoen fsine a l s ys tem res pon s e exa m ple

Th e s u m of a ll th e im pu ls e res pon s es Figure 2: Characterizingis th e fin a l s ysatelinearm res posystemn s e using its impulse response. Figure 2: CharacterizingFigure 2: Characterizinga linear systemausinglinearitssystemimpulseusingresponse.its impulse response.

The way we use the impulse response function is illustrated in Fig. 2. We conceive of the input The way we useThethewaimpulsey we useresponsethe impulsefunctionresponseis illustratedfunctioninis Fig.illustrated2. WeinconceiFig. 2.veWofetheconceiinputve of the input stimulus,stimulus,in this casein thisa sinusoid,case a sinusoid,as if it wereas if theit weresumtheof asumsetofofaimpulsesset of impulses(Eq. 1).(Eq.We 1).knoWwetheknow the stimulus,responsesin thiswecasewoulda sinusoid,get if eachas ifimpulseit werewtheas sumpresentedof a setseparatelyof impulses(i.e., (Eq.scaled1).andWeshiftedknowcopiesthe of responsesFigurewe2: Characterizingwould get if eacha linearimpulsesystemwausings presentedits impulseseparatelyresponse.(i.e., scaled and shifted copies of responsesthe impulsewe wouldresponse).get if eachWeimpulsesimplywaddas togetherpresentedallseparatelyof the (scaled(i.e.,andscaledshifted)and shiftedimpulsecopiesresponsesof to the impulse response).the impulseWeresponse).simply addWtogethere simplyalladdoftogetherthe (scaledall ofandtheshifted)(scaledimpulseand shifted)responsesimpulseto responses to predict hopredictw thehosystemw thewillsystemrespondwill respondto the completeto the completestimulus.stimulus. predictThehowawythewesystemuse thewillimpulserespondresponseto thefunctioncompleteis stimulus.illustrated in Fig. 2. We conceive of the input Now we will repeat all this in mathematical notation. Our goal is to show that the response (e.g., stimulus,Now wine willthis repeatcaseNowawallsinusoid,e willthis inrepeatmathematicalas ifallitthiswerein themathematicalnotation.sum ofOura setnotation.goalofisimpulsestoOurshowgoal(Eq.thatistheto1).shoresponseWwe knothatw(e.g.,thetheresponse (e.g., responsesmembranewe wmembraneouldpotentialget ifpotentialeachfluctuation)impulsefluctuation)ofwaasshift-inpresentedof avshift-inariantseparatelylinearvariantsystemlinear(i.e., scaled(e.g.,systempassiand(e.g.,shiftedve passineuralcopiesvemembrane)neuralof membrane) membranecan bepotentialwritten fluctuation)as a sum of scaledof a shift-inand shiftedvariantcopieslinearofsystemthe system’(e.g., passis impulseve neuralresponsemembrane)function. canthe impulsebe writtenresponse).canas abesumwrittenWofescaledsimplyas a sumandaddofshiftedtogetherscaledcopiesandall ofshiftedofthethe(scaledcopiessystem’andofs impulsetheshifted)system’responseimpulses impulseresponsesfunction.responseto function. predict howConvolutionthe system will respond to the complete stimulus. Now wThee willconrepeatvolutionall thisintegral.in mathematicalBegin bynotation.using Eq.Our1 tgoalo replaceis to shothewinputthat signalthe responseby(e.g.,its repre- The convolutionTheintegral.convolutionBeintegral.gin by usingBeEq.gin1bytousingreplaceEq.the1 toinputreplacesignalthe inputbysignalits repre- by its repre- membranesentationpotentialsentationin termsfluctuation)inoftermsimpulses:ofofimpulses:a shift-invariant linear system (e.g., passive neural membrane) sentation in termsRepresentingof impulses: the input signal as a sum of pulses: can be written as a sum of scaled and shifted copies of the system’s impulse response function.

past present future 0 0 0 1 0 0 0 0 0 input (impulse) The convolution integral. Begin by using Eq. 1 to replace the input signal by its representation in terms of impulses:1/8 1/4 1/2 weights Using0 0 additivity, 0 1/2 1/4 1/8 0 0 0 output (impulse response) Using additiUsingvityadditi, vity, Using additivity,

0 0 0 1 1 1 1 1 1 input (step)

Taking theTakingTakinglimit,the limit, the1/8 limit, 1/4 1/2 weights Taking the limit, Using additivity, 0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response)

Figure 3: Convolution as a series of weighted sums. 6 6 Using homogeneity (scalar6 rule), UsingTakinghomogeneitythe limit, ,

Now let be theDefiningresponse of h(tto) asthe unshiftedthe impulse6unit impulse, response,i.e., . Then by using shift-invariance, (4)

Convolution as a series of weighted sums. While superposition and convolution may sound a little abstract, there is an Measureequivalent statementthe that will make it concrete: a system is a shift- invariant, linear system ifimpulseand only responseif the responses are a weightedSpace/timesum of the methodinputs. Figure 3 shows an example: the output at each point in time is computed simply as a weighted sum of the inputs at recently past times. The choice of weighting function determines the behavior of the system. Not surprisingly, the weighting functionExpressis very asclosely sumrelated Calculateto the impulse theresponse Sumof the the system. In particular, the impulse responseofand shiftedthe weighting and functionresponseare time-re toversed copiesimpulse of one another, as demonstrated in the top partscaledof the ﬁgure.impulses each impulse responses Input stimulus Express as sum Calculate the Sum the of shifted and response to sinusoidal scaled sinusoids each sinusoid responses Measure the 7 Frequency method frequency response Spatial pattern vision and linear systems theory

Fourier transform and frequency response: summary

• Signals can be represented as sums of sine waves • Linear, shift-invariant systems operate “independently” on each sine wave, and merely scale and shift them. • A simplified model of neurons in the visual system, the linear receptive field, results in a neural image that is linear and shift- invariant. • Psychophysical models of the visual system might be built of such mechanisms. • It is therefore important to understand visual stimuli in terms of their spatial frequency content. • The same tools can be applied to other modalities (e.g., audition) and other signals (EEG, MRI, MEG, etc.).

Temporal frequency and Fourier decomposition Shift-invariant linear systems & sinusoids

Measure the scaling and shifting for each sinusoid

Shift-invariant linear systems & sinusoids

Frequency response

Linear systems analysis

Measure the impulse response Space/time method

time sample Fourier transform

Convolution: frequency sample

x1[n] * x2[n] X1[k] X2[k]

convolution Multiplication:

x1[n] x2[n] (1/N) X1[k] * X2[k]

Linear filter example

Signal Frequency content Miles Davis “Half Nelson”

Filter

Bass only: Low-pass or Bass Filter

Treble only: High-pass or Treble Filter

Prev Time Next

Linear filter example

Signal Frequency content Miles Davis “Half Nelson”

Filter

Bass only: Low-pass or Bass Filter

Treble only: High-pass or Treble Filter

Time Prev Next Linear filter example

Signal Frequency content Miles Davis “Half Nelson”

Filter

Bass only: Low-pass or Bass Filter

Treble only: High-pass or Treble Filter

Time Prev Next

Linear filter example

Signal Frequency content Miles Davis “Half Nelson”

Filter

Bass only: Low-pass or Bass Filter

Treble only: High-pass or Treble Filter

Prev Time Next

Input image (cornea) “Neural image” (V1) Linear Convolution with impulse response filter

Fourier transform

Multiplication with frequency response

Fourier spectrum Fourier spectrum of input image of neural image Fourier Analysis

Signals as sums of sine waves • 1d: time series – fMRI signal from a voxel or ROI – mean firing rate of a neuron over time – auditory stimuli • 2d: static visual image, neural image • 3d: visual motion analysis

Auditory example: Pure tones

Pure tones can be described by 3 numbers: Frequency = rate of air pressure modulation (related to pitch) Amplitude = sound pressure level (related to loudness) Phase = sin vs. cosine vs. another horizontal shift

Frequency and amplitude

weak 100 Hz strong 100 Hz weak 1000 Hz strong 1000 Hz

Time

1/100 s Soundlevel pressure Fourier spectrum representation of sound

Fourier spectra of some sounds

Pure tone

White noise Amplitude

Violin note

100 1000 10,000 Frequency (Hz)

Fundamental frequency and harmonics Lots of Fourier transforms

Name Time domain Freq domain

Fourier transform continuous, infinite continuous, infinite

Fourier series continuous, periodic discrete, infinite

DTFT discrete, infinite continuous, periodic

DFS discrete, periodic discrete, periodic

DFT discrete, finite discrete, finite

FFT algorithm

• Computes DFT of finite length input. • Efficient for inputs of length N = mn. • Produces 2 outputs, each of size/length equal to that of the input: real part (cosine coeffs), imaginary part (sine coeffs).

DFT of a cosine

DFT of a cosine cos(2πkn/N) ⎛ 2πkn⎞ cos ⎝⎜ N ⎠⎟ k = 4 cycles/scan N = 32 time points

2πk / N (π / 4 in this case)

frequency (cycles per scan) frequency (cycles per scan) frequencyFrequency (cycles (cycles) per scan) frequencyFrequency (cycles (cycles) per scan)

Real and imaginary parts

cosine sine

real part

imaginary part

magnitude

phase

DFT of impulse signal and constant signal

? ?

frequency frequency Real and imaginary parts

cosine sine

real part

imaginary part

magnitude

phase

DFT of impulse signal and constant signal

? ?

frequency frequency

Uncertainty principle

0 8 16 24 32 -16 -8 0 8 16

0 8 16 24 32 -16 -8 0 8 16 Time (sampleTime (TR) number) FrequencyFrequency (cyc/scan) (cycles) Multiplication and convolution

Discrete Fourier Transform (DFT)

Complex numbers and complex exponentials

amplitude phase imaginary

imaginary real part part real where:

Why bother with complex exponentials?

amplitudes phases multiply add Discrete Fourier transform matrix

Discrete Fourier transform matrix

Sine wave gratings and spatial frequency

1 cpd 4 cpd

Measured in cycles per degree (cpd or c/deg or c/º) of visual angle. Contrast

Contrast = 1

Contrast = 0.5

Two-dimensional Fourier spectra

Two-dimensional Fourier spectrum

Intensity in the Fourier spectrum at each location indicates amount of contrast (in the original image) for each spatial frequency and orientation. Input image (cornea) “Neural image” (V1) Linear Convolution with impulse response systems analysis

Fourier transform

Multiplication with frequency response

Fourier spectrum Fourier spectrum of input image of neural image

Spatial pattern vision and linear systems theory

Spatial contrast sensitivity Contrast sensitivity

Spatial frequency (c/deg) Spatial contrast sensitivity

Spatial frequency channels

Each channel is sensitive to a narrow range of frequencies. Overall contrast sensitivity depends on all of them together.

Theory of spatial pattern analysis by the visual system

Low sf filters encode High sf filters encode coarse-scale information fine-scale information (large objects, overall (small objects, detail) shape) Multi-resolution model

Psychophysical/perceptual evidence for spatial-frequency and orientation selective channels

Orientation and spatial frequency selective adaptation: appearance Emma by Chuck Close 28 CHAPTER 1. PATTERN SENSITIVITY Spatial frequency selective adaptation: appearance

(a) No adaptation (b) Low frequency (c) High frequency adaptation adaptation y y y t t t i i i v v v i i i t t t i i i s s s n n n e e e S S S

Spatial frequency Spatial frequency Spatial frequency

Test Test Test e e e s s s n n n o o o p p p s s s e e e R R R

Channel Channel Channel

Figure 1.13: A multiresolution model can explain certain aspects of pattern adaptation. (a) In normal viewing, the bar width is inferred from the relative responses of a collection of component-images, each responding best to a selected spatial frequency band. The spatial frequency selectivity of each component-image is shown above and the amplitude of the component-image encoding of the test stimulus is shown in the bar graph below. (b) Following adaptation to a low frequency stimulus (shown in inset), the sensitivity of the neurons comprising certain component-images is reduced. Con- sidering the responses of all the component-images, the response to the test is similar to the unadapted response to a high frequency target. (c) Following adaptation to a high frequency pattern (shown in inset), the neural representation is consistent with Spatialthe unadapted frequency response to a low frequencyselective target. adaptation: sensitivity

Spatial frequency selective adaptation: sensitivity Contrast sensitivity before & after adaptation

246 C. BLAKEMORE AND F. W. CAMPBELL to 23-8 c/deg., at one quarter octave steps, was determined by briefly inter- polating the low contrast test grating which the subject set to threshold. ContrastThe results aresensitivityshown with their beforeS.E. (n = 6). &It afteris clear that the sensitivity has been dramatically depressed in the region of 7 1 c/deg. but that adaptationthere is practically no effect beyond the frequencies 2-5 and 11 9 c/deg. A -P B c0

100 g1.0 F . W. c. 0 100~ ~~~n 0>

0 0

lo ~~~~~~~~~0.1 0 4-D~~~~~~~~~ 0

-0 I I I" 1" I I 1111"I 0-01 Il I ll l 1 I I I I ll 10 10 100 1-0 10 100 Spatial frequency (c/deg.) Text-fig. 6. The effect of adapting at 7-1 c/deg. A. The continuous curve from Text- fig. 5 is reproduced. The filled circles and vertical bars are the means and S.E. (n = 6) for re-determinations of contrast sensitivity at a number of spatial frequencies while F. W.C. was continuously adapting to a grating of 7-1 c/deg., 1-5 log. units above threshold. The exact procedure is described in the text. B. The depression in sensitivity due to adaptation at 7-1 c/deg. is plotted, with open circles, as relative threshold elevation against spatial frequency. The vertical difference between each point and the smooth curve in Text-fig. 6A is the ratio of sensitivity before and after adaptation. The relative threshold elevation is the anti- Contrastlogarithm of this difference minus 1, so that no change in threshold would give a value of zero on the ordinate. The continuous curve is the function [eif2e_(2f)2]2, sensitivityfitted by eye to the data points. The filled arrows show the adapting frequency of 7-1 c/deg. The open arrow marks the value on the ordinate for a threshold elevation beforeequivalent &to 212aftertimes an average s.E. for determining contrast sensitivity. adaptationIt is conceivable that adaptation at one specific frequency might have affected contrast threshold at all measurable spatial frequencies. If such were the case the over-all contrast sensitivity curve should have been uniformly lowered. This is not so, and we therefore conclude that the' adapting pattern is principally depressing the sensitivity of some 'channel', independently of others, and that this channel is adapted by a limited range of spatial frequency. In order to define its characteristics more specifically we have reduced Orientation selective adaptation

Orientation selective adaptation

++ + Applications: Optical Line Spread Function (Campbell & Gubisch, 1966)

Note the double passage

566 F. W. CAMPBELL AND R. W. GUBISCH 566 F. W. CAMPBELL AND R. W. GUBISCH 1.0 1.0 OPTICAL QUALITY OF THE HUMAN EYE 571 Applications:coming from Opticalscatter (Fry, 1965) orLinedefects of focus Spreadwhich increase with Function pupil diameter. A comparison of the linespread estimates of Flamant (1955), Westheimer & Campbell (1962), and Krauskopf (1962) with ours is (Campbellgiven in Fig. 11. & Gubisch, 1966) The linespread functions in Fig. 10 are about twice as narrow as the

30mm 38mm 0.1 0.1 1.0 G.L. 1.0 G.L.

2-4mm 4-9mm, I I 03 03

o i o i 2-0 mm 58m

15 mm A <,-J-J *--<_ C/ ...... ~~~~~~~~~~~~~~~~~~~~~~...... -----;'---- . 4 2 0 2 4 4 2 0 2 4 Angular distance (minutes of arc) Fig. 10. Optical linespread functions of the human eye. Each curve represents the normalized distribution of illuminance occurring on the fundus for a thin line source of light. Dots occur at 0-1 min increments. Narrower curve indicates the diffraction image of a line at the given pupil diameter.

Applications: Optical Line Spread Function00*1 00*1 4 8 1 2 1 6 20 (Campbell4 & Gubisch,8 1 2 1966)1 6 20 A-ngular distance (min) 568 F. W. CAMPBELLT AND1j _ t 1_B.TTYW.. GUBl-C1J1TY 7_ _ . T TTS . j s TS 75 7_ T distance A-ngular (min) 6. measured external to the of three 1 0r Fig. Linespreads eyes subjects. Angular Fig. 6. Linespreads measured external to the eyes of three subjects. Angular distances are taken from the centres of the synumetrical curves and only the right distances are taken from the centres of the synumetrical curves and only the right halves are shown. Pupil diameters in mm are: +, 2-0; *, 3 0; V, 3-8; A, 4-9; halves are shown.0.8 Pupil diameters in mm are: +, 2-0; *, 3 0; V, 3-8; A, 4-9; *, 5-8; *, 6-6. *, 5-8; *, 6-6.

0-6

1.0 G. L.

0-8

0-6

0 ~~~~~~20 20 40 ~~~~~~~~~~6060 0-4

0-2

0 20 40 60 Spatial frequency (c/deg.) Fig. 7. Modulation transfer functions for the eyes of three subjects. The curves display for various pupil sizes the transmission of contrast to an object imaged on the fundus; they are corrected for the double traverse of the eye made by the measured linespread function. Symbols are as in Fig. 6. Applications: Optical Line Spread Function (Westheimer, 1986)

Westheimer's Linespread Function. Analytic approximation of the human linespread function for an eye with a 3.0mm diameter pupil (Westheimer, 1986).

Applications: Optical Line Spread Function (Westheimer, 1986)

Modulation transfer function measurements of the optical quality of the lens made using visual interferometry (Williams et al., 1995; described in Chapter 3). The data are compared with the predictions from the linespread suggested by Westheimer (1986) and a curve fit through the data by Williams et al. (1995).

Applications: Optical Line Spread Function, Wavelength Dependency

OTF of Chromatic Aberration: Two views of the modulation transfer function of a model eye at various wavelengths. The model eye has the same chromatic aberration as the human eye (see Figure 2.23) and a 3.0mm pupil diameter. The eye is in focus at 580nm; the curve at 580nm is diffraction limited. The retinal image has no contrast beyond four cycles per degree at short wavelengths. (From Marimont and Wandell, 1993). Applications: Optical Line Spread Function, Wavelength Dependency

-60 410 -60 550 -60 690

-40 -40 -40

-20 -20 -20

0 0 0

20 20 20

40 40 40

60 60 60

-60 -40 -20 0 20 40 60 -60 -40 -20 0 20 40 60 -60 -40 -20 0 20 40 60 Position (um) Position (um) Position (um)