Andrew Stockman

NEUR3001: Advanced Visual

Introduction VISUAL NEUROSCIENCE

Andrew Stockman

Essentially what we’re trying to understand is:

How the works. What is it?

INPUT ? OUTPUT

Using any technique available to us.

Introduction to Advanced Visual Neuroscience 1 Andrew Stockman

PSYCHOPHYSICS PSYCHOPHYSICS

PHYSICAL STIMULUS PHYSICAL STIMULUS PERCEPTION

INPUT ? ? OUTPUT INPUT ? ? OUTPUT

Study the relationship between physical stimulus and perception… Can you think of some examples of psychophysical experiments From which we try to infer what is going on inside the black box. And what they might tell us about visual processing?

NEUROSCIENCE

• Bottom-Up Processing: information processing beginning at the “bottom” NEUROSCIENCE with raw sensory data sent “up” to the brain for higher-level analysis. • Top-Down Processing: information processing is modified by higher Study relationship between sensory input and the neural coding at different level processes from the “top” working “down”. stages of the visual system and from that infer the neural processing.

INPUT ? OUTPUT INPUT ? OUTPUT

NEURAL NEURAL SENSORY INPUT NEURAL CODING SENSORY INPUT NEURAL CODING PROCESSING PROCESSING

Introduction to Advanced Visual Neuroscience 2 Andrew Stockman

NEUROSCIENCE NEUROSCIENCE

• Information loss: information is lost as visual signals are encoded and • Information loss: information is lost as visual signals are encoded and processed by successive stages of the visual system. processed by successive stages of the visual system.

INFORMATION LOSS INFORMATION LOSS

INPUT ? OUTPUT INPUT ? OUTPUT

NEURAL NEURAL SENSORY INPUT NEURAL CODING SENSORY INPUT NEURAL CODING PROCESSING PROCESSING

Lost information cannot be retrieved, although some aspects can be Think of examples: What is lost? inferred by top-down processing (e.g., inference of 3D depth from a monocular image).

PSYCHOPHYSICS AND NEUROSCIENCE Let’s now step through this processing stream, beginning with… CAN BE COMPLEMENTARY

PHYSICAL STIMULUS PERCEPTION PHYSICAL STIMULUS PERCEPTION

INPUT ? OUTPUT INPUT ? OUTPUT

NEURAL NEURAL SENSORY INPUT NEURAL CODING SENSORY INPUT NEURAL CODING PROCESSING PROCESSING

Introduction to Advanced Visual Neuroscience 3 Andrew Stockman

Light 400 - 700 nm is most important for vision An image of an object is projected by the cornea and lens How do we see? onto the rear surface of the eye: the retina.

Inverted image

How do we see? Retina

The back of the retina is carpeted Part of the CNS by a layer of light-sensitive photoreceptors (rods and cones). About 60 identified cell types 4 Photoreceptors (+ ipRGCs) 10 bipolar cells 2 horizontal cells 30? amacrine cells 10? ganglion cells

Introduction to Advanced Visual Neuroscience 4 Andrew Stockman

Photoreceptors Initial encoding

rod

cone

Rods and cones respond differently inner segment | outer segment

Electron‐micrograph 800 

Other retinal cells

Introduction to Advanced Visual Neuroscience 5 Andrew Stockman

ON (greener) and Bipolar cells OFF (redder) varieties Horizontal cells Lateral interactions

From Rodieck (1998)

What sort of processing can be achieved by lateral interactions? Parallel processing? From Rodieck (1998)

Ganglion cells ON and OFF varieties Amacrine cells

From Rodieck (1998) Parallel processing? From Rodieck (1998)

Introduction to Advanced Visual Neuroscience 6 Andrew Stockman

Overview We can investigate what a cell encodes by “mapping” its receptive field

David Hubel Find the area in visual space to which the cell responds. From Rodieck (1998)

We can investigate what a cell encodes by Neural codes and signal processing “mapping” its receptive field

LIGHT

LIGHT

LIGHT

David Hubel And find out which types of stimuli optimally stimulate the cell. Webvision

Introduction to Advanced Visual Neuroscience 7 Andrew Stockman

Neural codes and signal processing (centre‐surround) Neural codes and signal processing (centre‐surround)

29 30

Red‐green colour opponency Parallel and serial processing in the retina

Colour coding

Introduction to Advanced Visual Neuroscience 8 Andrew Stockman

Retinotopic maps Retinotopic maps

David Hubel

Retinotopic maps Neural Coding Law of Specific Nerve Energies Stimulation by any cause has the same effect. Neurones code information by virtue of their connections not their biological structure. For example, electrical stimulation of your auditory nerve will cause you to hear sounds, or phospenes. Electrical stimulation of human cortex during neurosurgery causes Johannes Peter Müller hallucinatory (Penfield (1801‐1858) David Heeger experiments).

Introduction to Advanced Visual Neuroscience 9 Andrew Stockman

Neural Coding

• Single neurons represent sensory stimulus parameters with their rate From retina to or timing of action potential firing. • Will fire more spikes to some stimuli than others. brain… • Information might be contained in timing of spikes. • Relationship between stimulus parameters and neural responses is called the neural code

Visual pathways

L Andrew Stockman

Geniculo‐striate pathway

R Hannula, Simons & Cohen, 2005 From below

Introduction to Advanced Visual Neuroscience 10 Andrew Stockman

V1-V5

Cortical level

Neural codes and signal processing Neural codes and signal processing

Simple cells Complex cells

V1, layers 4 & 6 V1, layers 2, 3 & 5

Simple cells have narrow, elongated excitatory and inhibitory zones that have a Complex cells have large receptive fields without clear excitatory or inhibitory specific orientation. These cells are “line detectors”. Their receptive fields can be zones. They respond best to a moving edge of specific orientation and built from the convergent connections from lateral geniculate nucleus cells. direction of motion. They are powerful “motion detectors”. Their receptive fields could be built from the convergent connections of simple cells.

Introduction to Advanced Visual Neuroscience 11 Andrew Stockman

Neural codes and signal processing

V1 – primary visual cortex

www.cns.nyu.edu Feature extraction: Hypercomplex cells are like complex cells except for inhibitory flanks on the orientation ends of the receptive field, so that response increases with increasing bar length up to some limit, but is then inhibited. This property is called end‐stopping.

A tuning curve relates the response of a neuron to varied stimulus parameters. Visual areas

Levels of Processing: Feature extraction: Functional Hierarchy orientation

Introduction to Advanced Visual Neuroscience 12 Andrew Stockman

Streams of processing…

Dorsal stream – ‘Where’ pathway specialized for spatial location.

Parallel and serial processing Ventral stream – ‘What’ in the cortex pathway specialized for object identification and recognition.

Ventral stream – ‘What’ Ventral stream – ‘What’

• V1 V2  V4  IT • V1 V2  V4  IT • Neurons sensitive to features useful for object recognition

Face cell Border‐ownership neuron

Introduction to Advanced Visual Neuroscience 13 Andrew Stockman

Features (Pandemonium model, Selfridge 1959)

Processing strategies

The basic idea of the pandemonium architecture is that a pattern is first perceived in its parts before the “whole”: completely bottom-up.

Grandmother cell (Barlow, 1972) Localist representations

Knowledge is coded in a localist fashion: individual objects, words and simple concepts are coded distinctly with their own dedicated representation.

Grandmother! Easy to understand, but very inefficient. Separate cells for every colour/property of an object?

Localist coding schemes

Introduction to Advanced Visual Neuroscience 14 Andrew Stockman

Distributed/ Population coding Distributed representations

Perceptions are represented by the Knowledge is coded as a pattern of activation across rates and patterns of action potential many processing units, with each unit contributing to firing in populations of sensory neurons many different representations. As a consequence, there is no one unit devoted to coding a given word, object, or person.

Spatial frequency coding Each concept is represented by many neurons.

Each neuron participates in the representation of many concepts.

Low High

Top-down processing What next, though?

Beliefs, , and expectations can drive the pattern recognition process. For example, if you are expecting to come across a certain pattern, then you can focus your attention on looking for evidence consistent with that pattern.

So in the Pandemonium model, instead of all activity travelling up to the decision demon, information or activation can be sent the other way down to the feature demons. What is ? 60

Introduction to Advanced Visual Neuroscience 15 Andrew Stockman

fMRI Other methods

Measure localized brain PSYCHOPHYSICS activity by measuring localized increases in blood flow. PHYSICAL STIMULUS PERCEPTION Measure localized brain activity by measuring localized increases in blood flow INPUT ? OUTPUT Measures changes of oxygenated and deoxygenated blood to strong magnetic fields (BOLD signal). Study relationship between physical stimulus and perception..,

PSYCHOPHYSICS

PSYCHOPHYSICS PHYSICAL STIMULUS PERCEPTION

PHYSICAL STIMULUS PERCEPTION INPUT ? ? OUTPUT

INPUT What is happening in here ? OUTPUT ? In psychophysics, we have to In fact, we can learn a infer what is going on inside a remarkable amount about the “black box” just from studying internal processing from the input and output… psychophysical measurements. Study relationship between physical stimulus and perception.., Crucially, though, we can use knowledge from neuroscience to guide our experiments and models.

Introduction to Advanced Visual Neuroscience 16 Andrew Stockman

Which psychophysical tasks provide useful information?

Psychophysics DETECTION TASKS

Manipulate the properties of simple visual stimuli, and carefully measure the effect of that manipulation on the human subject’s response–in an attempt to understand the workings and organization of the human visual Is the circle flickering Is a circle present? Is the pattern (grating) or steady? system. visible?

In general, we use simple tasks. Can you think of other tasks?

Threshold detection measurements Threshold detection measurements

See it…

Don’t see it…

67

Introduction to Advanced Visual Neuroscience 17 Andrew Stockman

Experimental conditions

Colour normals

Variable wavelength As an example, human spectral sensitivity target, flickering at 1 Hz

Measuring the spectral sensitivity characterizes The subject’s task was to detect the 1-Hz wavelength‐dependence of the visual system flicker as a function of target wavelength. from which under some conditions we can infer

the spectral properties of the cone Very intense 580-nm, yellow photoreceptors. background that suppresses the M-cones, L-cones and rods.

-6

-8 S-cone data S -10

-12 Illusions -14

-16 The Normal data were quantal spectral sensitivity

10 obtained on an intense orange -18 Normals adapting background that was Log AS there to suppress the L- and CF BCMs HJ FB M-cone sensitivities. -20 LS KS TA PS

-22 400 450 500 550 600

Wavelength (nm)

Introduction to Advanced Visual Neuroscience 18 Andrew Stockman

Example: Grating adaptation Why study illusions? (population coding)

Seeing is not always believing.

Illusions can provide insights into how the visual system works

Source: Horace Barlow

Spatial frequency adaptation Visual processing: illusions explained?

Top down? Source: Barlow and Mollon, 1982

Introduction to Advanced Visual Neuroscience 19