Downloaded by guest on October 1, 2021 www.pnas.org/cgi/doi/10.1073/pnas.1812222116 a scale foveal Shelchkova the Natalya at exploration visual Task-driven foveated. is interest used of is region that the map once to priority microsaccades space finer-grained guide extrafoveal a to the also of but map pos- , priority system guide coarser visual a the only that across not suggest sesses maintained findings These idiosyncrasies when scales. used with spatial those scenes, resemble The larger scale scale. this foveal examining the at at used place takes strategies task, scanning process the a of goals exploration, the spatial by active driven resolu- localizing that high reveal at accurately position findings gaze for Our track tion. system continually and custom sight of a line the use of We expression meters. the many 1 judged approximately Observers covering pauses saccades. faces fixation large the between during in that occurs whether exploration an shown investigate visual we play Here, previously may task-driven vision. microsaccades, spatial have exploration fine particular in We role in visual important the scale. movements, for within smaller need large fixational this the over to the and studied extend tasks, complex, normally is everyday the This is in acuity. with exploration highest However, interest with visual scenes. of 2018) the 18, of objects July of review process region for inspect (received small 2019 to 31, the January foveola, approved saccades and NY, York, use New Physicians, Humans of College University Columbia Goldberg, E. Michael by Edited 02215 MA Boston, University, sarsl,we ugn ailepeso,hmn edto tend humans expression, facial features. judging diagnostic when most the result, at a acquisi- looking the As by fixations optimizes information When system two of visual 9). after the tion are period saturates (8, saccades this performance two During pattern as (10). first “T” relevant the most task, a the recognition using facial Gen- faces a faces. performing scan from information humans extracting erally, in specialized system visuomotor highly the is and abilities, human fundamental are tion use used we humans this way examine faces. To same scenes. human visual the large examine in explore to stimuli, saccades to used foveal are complex mechanism, patterns naturally recentering spatial rather simple microsaccades, fine a whether on being investigated than we fixation study, of this tuned locus In finely (7). preferred spatial are the fine microsaccades bring tasks, for laboratory crucial to In are 7). movements (6, eye vision These microsac- 5). with (4, and foveola drift, (<0.5 ocular saccades as small known cades, motion, jittery a intersaccadic with the during the foveola Can the to retina. of extend scale intervals? the exploration smaller this on visual much in else task-driven the stimulus anywhere top–down, input as of the complex concept receptors, as scenes retinal is natural covers of region of foveola scaling statistics The the fractal (2). and the task of because the process Nevertheless, of the this goals at that the look and only (1) to by scene high-acuity tend influenced the the humans is of with that regions inspected established informative be well most can is they It interest- that foveola. toward so gaze objects, of center ing the redirect saccades conditions, V maps priority eateto ersine nvriyo ohse eia etr ohse,N 42;and 14627; NY Rochester, Center, Medical Rochester of University , of Department prpitl nepeigfca xrsin n aedirec- gaze and expressions facial interpreting Appropriately move to continue but rest at never are the fixation During oe eaieylreprino h iulfil.I these In field. visual that the scenes of portion with large studied relatively a traditionally cover is exploration isual ≈1 ◦ fvsa nl,ls hn0.1% than less angle, visual of | clrdrift ocular a hiteTang Christie , | n pta vision spatial fine ◦ htke h tmlswti the within stimulus the keep that ) ◦ si iwdfo itneof distance a from viewed if as , b n atn Poletti Martina and , | ftevsa ed(3). field visual the of aeperception face | a,1 1073/pnas.1812222116/-/DCSupplemental. y at online information supporting contains article This been have figures in 1 (https://osf.io/tusgd/?view these data) repository produce Framework (containing to Science figures used Open scripts the the matlab all on the generate deposited and to manuscript necessary main data the The deposition: Data oeiaie ies . C BY-NC-ND) (CC 4.0 NoDerivatives License distributed under is article access open This Submission.y Direct PNAS a is article This interest.y of and conflict data; no per- declare analyzed M.P. authors M.P. The and and C.T., C.T., paper.y the N.S., N.S., wrote M.P. research; research; and the N.S. designed supervised M.P. M.P. and research; C.T., formed N.S., contributions: Author ce1e605d448e4014b50e6a7548ae37ba h edfrfrhrepoaini o meitl recognized. and immediately foveated, not is sim- is stimulus exploration system a investigated. further visual once for little the need fixation been that the maintain assumed has explo- to implicitly foveola needs of visual often ply the order why is of the it reasons scale First, in two the be at mainly may ration 1 are these features than There In different less away. arcminutes. the meters cover looking between many may is is tance face somebody person the whether the when circumstances, or whether even happy tell them, or can much at Humans angry scales ability studied. is spatial the typically somebody to and but those extends distances, fovea, than expressions different smaller the of facial humans range just recognize However, a to not periphery. from visual cover faces the view and stimuli These degrees the many also angle. spanning visual stimuli using of examined primarily been individual significant are there explo- subjects, (13–15). visual most variations task of in the seen patterns infer are these to ration possible while is Crucially, it (13). faces (12). scan performed at demands its the looking and task but observers positions of the nose, recogni- landing paths by the facial biased to examining for also by close Therefore, is gaze first important location the the most landing brings location, the exact normally eccentric is It an (12). face tion at the (10). presented upper to tasks is the recognition saccade with scanning face associated whereas the mostly 11), is When face (9, the region of mouth part the at look owo orsodnesol eadesd mi:martina Email: addressed. be should correspondence whom To edu. hrfr,rslsfo yeg fvsa rcse,motor processes, visual factors. of cognitive synergy and vision, behaviors, a spatial from Fine results scales. therefore, spatial dif- across task individual maintained resemble with the ferences scenes, closely larger on exploring strategies for based adopted scanning those stimulus visual the tiny These saccades, inspect demands. between the actively in localizing periods shifts the for gaze in that, methods show foveal high-precision we complex also gaze, examining using it when that By scale show gener- finer stimuli. we much is here a scales, exploration at spatial the occurs visual large of the While at portion studied bring foveola. high-acuity ally the which small field, the saccades, into visual by interest driven of objects is exploration Visual Significance iulepoaino ae,a iulepoaino cns has scenes, of exploration visual as faces, of exploration Visual y b eateto scooia n ri cecs Boston Sciences, and Psychological of Department ).y . y raieCmosAttribution-NonCommercial- Commons Creative www.pnas.org/lookup/suppl/doi:10. NSLts Articles Latest PNAS [email protected]. ◦ n h dis- the and , | f8 of 1 only=

PSYCHOLOGICAL AND COGNITIVE SCIENCES Second, whereas examining visuomotor scanning strategies over on the background. If exploration of complex foveal stimuli is a large visual scene is relatively straightforward, being able to top–down driven, we expect the pattern of eye movements to accurately localize gaze within a region as small as the foveola is a systematically change in the two tasks. The pattern of eye move- challenging task. ments on the stimulus was examined at high resolution while Here, we used high-resolution and a state-of- subjects performed the task. the-art system for gaze-contingent control that enables more accurate localization of the line of sight compared with standard Influence of the Task on the Examination of Foveal Stimuli. Our find- techniques (5). We examined the oculomotor behavior at fixation ings show that, despite the small size of the stimuli, and despite by precisely mapping gaze position onto the foveal stimulus. In the fact that they were already ideally placed within the foveola two experiments, we first examined whether visual exploration at to perform both tasks, subjects actively examined them using dif- the foveal scale is top–down guided based on the task demands, ferent scanning patterns in the two tasks. When asked to judge while the physical stimulus remains unchanged. Then, we investi- gaze direction, subjects’ gaze shifted toward the eyes region gated how visual exploration at the scale of the foveola compares (Fig. 2 A and B); on the other hand, when judging facial expres- to the exploration at a larger scale. sion, subjects spent more time on the mouth region (Fig. 2 C and D and Movie S1). Microsaccadic behavior was very consis- Results tent across subjects; most of the microsaccades landed on the To explore whether task-driven visual exploration extends to the eyes in the gaze direction task (0.70 ± 0.13 on the eyes vs. 0 fine scale of the foveola we conducted a simplified version of microsaccades landing on the mouth; P < 0.0001, two-tailed a “Yarbus experiment”; subjects performed two different tasks paired t test), but this pattern flipped when judging facial expres- with the same set of stimuli. In one task participants were asked sion, with most microsaccades landing on the nose and on the to judge whether a face was looking at them and in another mouth (0.1 ± 0.10 on the eyes vs. 0.5 ± 0.33 on the mouth; P = task whether the face was smiling at them (Fig. 1 A–D). Stimuli 0.02, two-tailed paired t test, Fig. 2 F and G). were presented foveally and covered approximately 1◦ of visual The oculomotor behavior in both tasks differed compared angle. The distance between the two task-relevant features, eyes with the normal physiological fixational instability when subjects and mouth, and the initial fixation location was the same (180, maintained fixation on a single point. When maintaining fixa- with prime indicating unit of arcminutes; Fig. 1C). We classified tion, the amplitude of microsaccades was lower (160 ± 20 in the gaze position based on where it was on the stimulus. Three main task vs. 130 ± 30 during sustained fixation; P = 0.007, paired two- regions were identified: eyes, nose, and mouth (Fig. 1E). If the tailed t test), and most microsaccades maintained the gaze close gaze was not in any of these regions, it was classified as being to the center of the display, the spatial location corresponding to

A 1o B C Drift Saccade

0.3°

Gaze direction 0.3°

Expression D Stimulus On E

Enlarged Display

variable 1.5 sec 2 sec

Time Response Time

Fig. 1. Methods (experiment 1). (A) An example of eye movements recorded by means of a high-precision eye tracker. Inset shows eye movements during a fixation period. (B) Stimuli were generated by changing gaze direction and shape of the mouth. The same face was presented in four different versions: gaze looking straight or looking away and smiling or neutral expression. In the gaze direction task, subjects judged gaze direction, and in the expression task they judged whether or not the face was smiling. (C) The distance between the eyes/mouth and the initial fixation location (blue cross) was the same. The face covered 1.46◦ of visual angle in height. (D) Experimental paradigm. After a brief period of fixation a face was presented for 1.5 s at the center of the display. Subjects could respond at any time during the stimulus presentation and after its offset. (E) Gaze position on the stimulus was mapped at high resolution based on which feature the gaze was on. The feature regions used for data analysis are shown here delimited by a pink bounding box.

2 of 8 | www.pnas.org/cgi/doi/10.1073/pnas.1812222116 Shelchkova et al. Downloaded by guest on October 1, 2021 Downloaded by guest on October 1, 2021 o60m rmsiuu ne 24ms/s (2.4 ms al. et 300 onset Shelchkova from stimulus interval course the from in time ms was task and microsaccades 600 direction rate gaze of to their the rate in also average higher but The based systematically. performed, different Microsac- varied task microsaccades of of the position Course on landing Time the and was only Rate the active cades. in in Changes engages Task-Driven but task. the stimulus of brief goals specific foveated main- the during by simply the guided even not exploration on does that, fixation system show visuomotor tain the further periods, findings fixation These tively). (0.52 task 0.14 the in nose the ( difference significant statistically tasks. a mark both Asterisks in onset. subject stimulus single from each ms of 300–600 proportions interval the the connect in lines tasks two the in paired mouth two-tailed the on and eyes the on (E (B SEM. tasks. are (D) regions Shaded time. (C response expression average the and (A) direction i.2. Fig. B Gaze position probability A

Microsaccades/s 0.25 0.75 0.5 ± 0 5 0 101500 1150 800 450 100 0 5 0 101500 1150 800 450 100 0 2 4 6 0 1 utemr,terslso xeiet1so ht not that, show 1 experiment of results the Furthermore, xeiet1rsls (A results. 1 Experiment .7 o oe ot,ee,adbcgon,respec- background, and eyes, mouth, nose, for 0.07, vrg aeo irscae ttebgnigada h n ftetilfrtetotss (F tasks. two the for trial the of end the at and beginning the at microsaccades of rate Average ) Gaze direction t igesbetpoaiiiso irscae adn ntemuhadns s ysi h w ak.The tasks. two the in eyes vs. nose and mouth the on landing microsaccades of probabilities Single-subject (G) significant). not n.s., test; Time [ms] Time [ms] ± and . s 0.23 vs. 0.2 ak (n tasks ) C 10’ ntegaze the in (Bottom) position landing microsaccade and (Top) distribution position gaze of probability Average ) 0.Dt aebe lee sn unn vrg iha10m idw ahdbaklnsmr the mark lines black Dashed window. 100-ms a with average running a using filtered been have Data 10). = ± and .,0.11 0.3, ± n h expression the and (B) direction gaze the in probability distribution gaze normalized 2D Average D) C D

. ssand ms/s 0.6 Gaze position probability

Microsaccades/s 0.25 0.75 0 5 0 1150 800 450 100 0.5 ± 0 5 0 101500 1150 800 450 100 0 2 4 6 0 1 .,and 0.1, Expression Eyes Nose Mouth Background n xeietw xmndhwtesaitmoa atr of pattern Scales. spatiotemporal the Spatial how Different examined we at experiment ond Strategies Scanning Visual paired two-tailed in ms t ms earlier (403 (327 ms task task 80 expression direction approximately The gaze task. peaked the the microsaccades by modulated of also rate was ms; course (600–900 time trial Microsaccade the respectively; of expression, rest and the during ms/s tasks 0.9 two the in same P ms/s 1.7 et n ihasml xto 31ms (391 fixation simple a with and test) .2,totie paired two-tailed 0.027, = Time [ms] Time [ms] ± 10’ ± ssfrgz ieto n xrsin respectively; expression, and direction gaze for ms/s 1 . ssad10ms/s 1.0 and ms/s 0.6 t .25 .5 .75 1 0 et Fg 3). (Fig. test) vrg rbblt fmcoacdslanding microsaccades of probability Average ) 1500 ± 0ms; 80 t G ,btwsvrulythe virtually was but 2E), (Fig. test) P Expression F E 0.2 0.4 0.6 0.8 .,totie paired two-tailed 0.3, = 0 1 Landing position probability microsaccades/s 00.20. 0.5 ± ± 3 0 4 2 1 0 1 P Landing positionprobability 7m)cmae ihthe with compared ms) 17 . ssfrgz direction gaze for ms/s 0.7

.1 w-aldpaired two-tailed 0.01, =

Eyes 300-600 300-600 NSLts Articles Latest PNAS * Gaze Direction } * } ± Gaze direction Expression Mouth

9ms; 49

N.S.

600-900 ms 600-900

* m } s } Eyes Mouth +Nose 0.6 P nasec- a In ∗ 0.005, = P .8 | t < test). f8 of 3 0.05, 1

PSYCHOLOGICAL AND COGNITIVE SCIENCES visual exploration at the foveal scale compares to that of visual 0.2, 0.25 ± 0.1, and 0.20 ± 0.1 probability of microsaccades exploration of larger stimuli. Subjects viewed human faces and landing on eyes, nose, and background, respectively; ANOVA judged whether or not the face’s expression was neutral. In the F(3,45) = 3.2; P = 0.03; Tukey’s HSD post hoc tests, mouth parafovea condition, each face covered an area of 11.5 deg2, as vs. eyes, P = 0.02; mouth vs. nose, P = 0.3; mouth vs. back- if it was viewed from a distance of 3 m. In the foveola condition, ground, P = 0.09] (Fig. 4 C, D, and F; SI Appendix, Fig. S2; instead, faces covered an area of 0.7 deg2, as if they were viewed and Movie S2). Overall, microsaccadic behavior in this task was from a distance of 13 m (Fig. 4A). less precise than the saccadic behavior, both within and across When the stimulus extended to the parafoveal region, almost subjects. This could be due to the fact that the stimuli used in all observers followed a very stereotyped scanning pattern (Fig. 4 experiment 2 were slightly smaller than those used in experi- B and E). Immediately before the stimulus onset subjects fixated ment 1; the distance between features ranged between 100 and 0 on a marker at the center of the display, so their initial gaze 15 . Critically, the decline in fine pattern vision reported across position upon stimulus presentation was on the upper part of the foveola is less steep than the decline from the fovea to the the nose region, approximately at the center of the face. After visual periphery. As a result, in the foveola condition there is a brief period of saccadic suppression following the presentation less of a drive to shift the gaze as precisely as in the parafovea of the stimulus, the rate of saccades sharply increased. During condition. A small microsaccade landing on the lower part of this time most of the saccades landed on the mouth (Fig. 4B, the nose region, or a microsaccade landing into the background 0 Right) [0.77 ± 0.3 vs. 0.15 ± 0.3, 0.05 ± 0.07, and 0.03 ± 0.03 region adjacent to a feature, would still land less than ≈5 away probability of landing on eyes, nose, and background, respec- from the target region and would still be precise enough for this tively; ANOVA F(3,45) = 30.3; P < 0.0001; Tukey’s honestly task. However, a microsaccade landing on the eye region or on significant differences (HSD) post hoc tests, mouth vs. eyes, P < its surrounding background likely shifts the preferred fixational 0.0001; mouth vs. nose, P < 0.0001; and mouth vs. background, locus too far from the mouth, the most informative feature for P < 0.0001] (Movie S2). The rate of saccades then gradually this task. Consistent with this idea, our data show that most of decreased back to baseline. This pattern of visual exploration the microsaccades landing on the background, or on the nose, is expected when the area of the stimulus covers many degrees. landed primarily in the lower part of these features closer to the A tendency to look over the mouth when judging facial expres- mouth region (0.65 ± 0.23 and 0.35 ± 0.23 probability of “nose” sion has been reported by a number of studies (9, 11, 15–17). microsaccades landing on the lower and upper part of the nose, Moreover, a bias toward the lower part of the face when judg- respectively, P = 0.03, paired two-tailed t test; 0.66 ± 0.22 and ing facial expression was also reported for the first saccade 0.34 ± 0.22 probability of “background” microsaccades landing bringing a face, presented in the visual periphery, to the center on the lower and upper part of the background, respectively, of gaze (12). P = 0.01, paired two-tailed t test) (Movie S2). In the foveola condition the exploratory behavior was driven Crucially, microsaccades that brought the center of gaze closer by microsaccades (average amplitude 150 ± 30; SI Appendix, to the task-relevant feature benefited performance in this task. Fig. S1). Similar to what happens in the parafovea condition The task was trivial, so to make sure that subjects remained for saccades, after an initial suppression period, the rate of engaged in the task and that performance did not saturate we microsaccades peaked at approximately 400 ms (371 ms ± lowered the contrast of the and included a number of 65 ms microsaccade rate peak time in the foveola condition vs. more ambiguous expressions. While the percentage of correct 403 ms ± 87 ms saccade rate peak time in the parafovea con- responses was well above chance for all subjects, there were dition; P = 0.23, two-tailed paired t test). During the period some variations in performance across individuals. The rate of in which microsaccade rate reached a peak (300–600 ms), most microsaccades landing on the mouth region was positively corre- microsaccades landed on the mouth region [0.40 ± 0.3 vs. 0.16 ± lated with the performance in the task across subjects (Pearson correlation coefficient r = 0.58, P = 0.02; SI Appendix, Fig. S3); that is, subjects characterized by a higher rate of microsaccades landing on this task-relevant region also showed higher perfor- mance in the task. This improvement was associated only with microsaccades landing on the mouth; performance was not cor- related with the global rate of microsaccades and with the rate of microsaccades landing on the eyes or background (r = −0.14, P = 0.60 for microsaccades landing on the eyes and r = 0.05, P = 0.85 for microsaccades landing on the background). To ensure that the pattern of recorded when subjects performed the task was, indeed, the result of an active exploration and not the mere outcome of the physiological insta- bility of the eye at fixation, similar to experiment 1, we examined fixational eye movements when subjects were required to keep their gaze on a marker at the center of the display. The rate of microsaccades was higher and the amplitude of microsac- cades lower during fixation compared with when the subjects performed the task (1.5 ms/s ± 0.8 m/s and 1.2 ms/s ± 0.6 ms/s fixation and task, respectively, P = 0.04, paired two-tailed t test; and 13.60 ± 3.60 and 150 ± 30 fixation and task, respectively, P = 0.04, paired two-tailed t test) (SI Appendix, Fig. S1). Moreover, microsaccade landing position and the overall spatial distribution of gaze position differed between fixation and the task. As illus- D SI Appendix Fig. 3. Temporal occurrence of microsaccades. Shown is average microsac- trated in Fig. 4 (red dashed line) and , Fig. S4, when cade rate over time in experiment 1 and during sustained fixation. Data subjects fixated on a central marker on a blank background, the have been filtered using a running average with a 100-ms window. Dashed probability of microsaccades landing on the spatial region corre- lines represent the average time when the rate of microsaccades reached a sponding to the mouth in the task was close to zero and it was peak. Error bars represent SEM. lower than the probability of landing anywhere else (0.06 ± 0.06

4 of 8 | www.pnas.org/cgi/doi/10.1073/pnas.1812222116 Shelchkova et al. Downloaded by guest on October 1, 2021 Downloaded by guest on October 1, 2021 l odto.Oua rf,teicsatjte fteee was eye, the of jitter incessant the fove- drift, the in Ocular changed condition. movements eye ola intersaccadic also but task, the by modulated was it and performed. task fixation, movements maintaining dif- eye simply task fixational of the when pattern during physiological behavior the motor from the fered that show findings these background, vs. mouth and eyes, vs. mouth tests, respectively, background, and deg 1 only spans it 4. Fig. hlhoae al. et Shelchkova ( 0.35 difference vs. significant when statistically shown a to also mark applies is Asterisks C) in mouth line). (key the the dashed code after to (red ms Average same corresponding stimulus 600 (E The (D) the region to parafovea SEM. time. of spatial ms the represent response 300 in absence bars the average probability from Error the on tests). the interval in landing hoc the mark marker post microsaccade in lines HSD a stimulus of Tukey’s black on the probability Dashed of fixation average SEM. regions maintained are the different subjects regions on comparison, landing Shaded For (foveola) window. onset. microsaccade 100-ms stimulus and a (parafovea) with saccade average of running probability a using filtered been (n condition neetnl,ntol eemcoacdsmdltdb the by modulated microsaccades were only not Interestingly, 2 prfvacniin 4.2 condition, (parafovea ae r omlyvee rmdfeetdsacs h aeo esnstanding person a of face the distances; different from viewed normally are Faces (A) results. 2 Experiment ± B .,0.32 0.1, C A 6.(C 16). =

◦ Gaze position probability Gaze position probability 0.25 0.75 0.25 0.75 hnteosre is observer the when 0.5 0.5 0 5 0 101500 1150 800 450 100 0 5 0 101500 1150 800 450 100 0 1 1 0 vrg itiuino aepsto (Left position gaze of distribution Average ) ± ~3 meters .,ad0.27 and 0.2, P < Eyes Nose Mouth Background n h oel (F foveola the and ) ◦ .01 ot s nose, vs. mouth 0.0001; Time [ms] P vrg itiuino aepsto (Left position gaze of distribution Average (B) height). Time [ms] = P 3maa.I xeiet2fcscvrdete nae f07deg 0.7 of area an either covered faces 2 experiment In away. m ≈13 .06.Smlrt xeiet1, experiment to Similar 0.0006). < ~13 meters .01 ue’ S othoc post HSD Tukey’s 0.0001; ± .,frmuh ys nose, eyes, mouth, for 0.1, conditions. ) P

< Saccades/s

Microsaccades/s 2.5 n irscaelnigpsto (Right position landing microsaccade and ) 0 1 2 0 5 0 101500 1150 800 450 100 0 5 0 5 0 101500 1150 800 450 100 0.0001; ipymitie xto nasnl on dfuincoeffi- S5 (diffusion point arcmin single 17 fixation a at on cient they subjects when fixation with when maintained compared stimuli coefficient simply foveal with diffusion task the smaller performed a by characterized ae ihrb h ako ytesailcaatrsiso the of characteristics spatial the by or modu- enhances task actively stimulus. the be further visual may by it These drift either as 18). intersaccadic lated (6, task stimulus that the this suggest of in findings content frequency beneficial spatial high be the may drift lar 4 .Rdcn h muto ipaeetitoue yocu- by introduced displacement of amount the Reducing ). Time [ms] Time [ms] o 2 1 n acd adn oiin( position landing saccade and ) ntefvoacondition, foveola the in o B and D

(E D. Landing position probability arcmin 0.5 0 1 2 and fvoacniin 1 condition, (foveola F E ntefvoacniin(n condition foveola the in ) Parafovea * F 2 vrg Dnraie aedistribution gaze normalized 2D Average ) ± wysas4 spans away m ≈3 * 5 Fixation arcmin * P Right 0.009; = * Foveola vrtm nteparafovea the in time over ) NSLts Articles Latest PNAS ◦ 2 egt ra rao 11.5 of area an or height) s 14 vs. 10’ 10’ Fig. Appendix, SI ◦ ntertn,but retina, the on arcmin 6.Dt have Data 16). = .25 0 .5 .75 1 ∗ P 2 | < ± f8 of 5 0.05, 4.3

PSYCHOLOGICAL AND COGNITIVE SCIENCES A Parafovea Foveola B 1 5 2 *

}

4 Mouth lookers Nose lookers 1.5 ** }

}

3 0.5 1 2

Probability

Saccades/s

Microsaccades/s 0.5 1

0 0 0 100 450 800 1150 1500 100 450 800 1150 1500 Mouth Nose Mouth Nose Time (ms) Time (ms) Parafovea Foveola

Fig. 5. Individual differences are maintained across scales. (A) Average rate of saccades (parafovea, Left) and microsaccades (foveola, Right) landing on the mouth during the course of the trial for nose lookers (n = 5) and mouth lookers (n = 16). (B) Probability of microsaccade and saccade landing over different regions of the stimulus for nose and mouth lookers. Probabilities are calculated in the interval from 300 ms to 600 ms from the stimulus onset. Asterisks mark a statistically significant difference (∗P < 0.05, two-tailed t test). Error bars represent SEM.

Individual Differences Are Maintained Across Scales. It has been swered primarily because of the challenges inherent in precisely previously reported that the pattern of eye movements when determining the portion of the scene covered by the foveola. Our viewing faces varies significantly across observers (13–15, 19, approach for accurate gaze localization has enabled us to cir- 20). Similarly, here we found that in the parafovea condition cumvent these limitations and investigate this issue. Here, we a small percentage of subjects (24% of the total, five subjects) show that task-driven visual exploration extends to the scale of maintained fixation around the center of the display (at the the foveola. This process follows scanning strategies qualitatively nose location) for the entire duration of the stimulus presen- similar, but smaller in scale, to those occurring during saccadic tation (0.29 ± 0.23 probability of saccades landing on the nose exploration. Microsaccades are modulated by the task demands and 0.30 ± 0.2 on the mouth for nose lookers vs. 0.05 ± 0.07 both in space and in time: They consistently target task-relevant and 0.77 ± 0.3 for the mouth lookers; nose vs. mouth lookers, locations within the fovea, and, for a given stimulus, their rate P = 0.001 and P = 0.005 for nose and mouth, respectively, two- and dynamics vary with the task at hand. These findings com- tailed t test) (Fig. 5 A and B). Although the nose lookers did plement our previous work on microsaccades. They show that not explore the face, their performance in the task was as good this oculomotor behavior is not the outcome of purely bottom– as that of the other subjects (88.7 ± 2.2 for nose lookers vs. up recentering mechanisms, but is the manifestation of active, 85.5 ± 6.4 for mouth lookers; P = 0.3, two-tailed t test). top–down-driven, visual scanning strategies. Because of their markedly different behavior, these subjects were The results reported here have important implications for the removed from the main analysis. Notably, however, our data study of priority maps. In our experimental paradigm, exposure show that these individual differences were maintained across to the stimulus was relatively long, and subjects were left free scales; the nose lookers showed a similar behavior in the foveola to perform multiple saccades. However, in experiment 1, sub- condition (0.40 ± 0.2 probability of microsaccades landing on the jects delivered their response about 200 ms before the offset nose and 0.21 ± 0.08 on the mouth for nose lookers vs. 0.25 ± of the stimulus and, in most trials, after only one microsaccade 0.13 and 0.40 ± 0.28 for the mouth lookers; P = 0.04, for mouth (1.5 ± 0.6 and 1.2 ± 0.7 microsaccades in the gaze direction vs. nose lookers microsaccades landing on the mouth, two-tailed and expression tasks, respectively). Thus, the first microsaccade t test) (Fig. 5B). Similarly, also in the foveola condition the per- following stimulus onset appeared to be critical for perform- formance in the task was the same for nose and mouth lookers ing the task. This first microsaccade, which generally occurred (78.4 ± 5 for nose lookers vs. 79.8 ± 7 for mouth lookers; P = within the initial 350 ms of exposure, was clearly driven by the 0.7, two-tailed t test). task. Later microsaccades were not as strongly directed toward Furthermore, even across the mouth lookers there were sig- a specific facial feature. These findings strongly suggest that the nificant variations in the proportion of microsaccades landing first microsaccade was driven by a priority representation of the on the eyes vs. those landing on the mouth. These differences, foveal input. Thus, contrary to the general idea that the function however, were also preserved across scales; the difference in the of priority maps is to represent the relevance of stimuli outside proportion of saccades/microsaccades landing on the eyes vs. on the fovea, our results show that these maps must also include the mouth was highly correlated across subjects in the parafovea foveal representations. and in the foveola condition (r = 0.77, P = 0.0005). These find- The existence of a foveal priority map is supported by the ings show that idiosyncrasies in the visual scanning patterns are notion that visual functions are not uniform across the foveola preserved across scales. (7). This map enables selection of the most relevant regions of the foveal landscape to guide visual exploration and is respon- Discussion sible for directing the first microsaccade following stimulus Previous studies have shown that microsaccades precisely posi- presentation. Priority maps of the extrafoveal space are known tion a preferred foveal locus of fixation in high-acuity tasks (7, to contribute to driving different effectors and behaviors, from 21). This observation raises the question of whether scanning and eye movements to reaching (22, 23). However, microsaccades exploration of visual objects and scenes, which have tradition- appear to be the only motor behavior that can be controlled ally been ascribed to large saccades, also apply to microsaccades at the fine scale of the foveola. This raises the question of at a finer spatial scale. This question has so far remained unan- whether this finer-grain priority map of the foveola is specifically

6 of 8 | www.pnas.org/cgi/doi/10.1073/pnas.1812222116 Shelchkova et al. Downloaded by guest on October 1, 2021 Downloaded by guest on October 1, 2021 hlhoae al. et Shelchkova xlie otepriiattemtra ntecnetfr.Tefr was form The form. consent the in and material reviewed the experimenter participant the the to partic- experiment explained all any from conducting obtained Before was Campus All ipants. consent River 2). Informed Charles (Fig. Board. 1 University Review experiment Boston 2 in Institutional the females) experiment by 6 in approved and were males part (4 experiments took observers y). females) 10 18–25 and 4 range 4) and (age (Fig. males experiments (17 the in observers Twenty-one participated study, the of purpose Observers. Methods and Materials locations foveal informative fixation. of most locus the preferred within the sampling falling with already by strate- patterns. stimulus foveola these the spatial the of explore component efficiently fine motor They visuo- inspect gies. main subtle the to are in strategies Microsaccades engage examining humans explorative fully for fixation, motor sufficient During stimu- not but stimulus. the necessary the Foveating is interest assumed. of commonly lus than complex more with interfere necessarily exploration. not foveal the may allo- that target peripherally suggesting saccade (31), next cating independently the and of anal- parallel selection that in the showed proceed and work stimuli Previous foveal fixation. of one ysis both of during when frame required time are conditions, the processing attention ecological peripheral of and more processing interplay in foveal funda- the movements of is eye associated understanding questions and better scanning these a visuomotor by for Addressing the leads mental exploration. fovea of foveal the suppression with from a far to attention Sec- itself details. allocating more in whether rich in is attention ond, stimulation of foveal allocation when condtions, peripheral to natural the continue by two microsaccades modulated prompts whether be observation First, This questions. gaze. of important center spa- visual minimal the the provide at studies conditions, stimulation these by viewing used paradigms natural cuing attention. with tial manipulating contrast when in shifts gaze However, small con- these of importance for the pat- trolling emphasized attention have microsaccade covert findings of These allocation how peripheral (28–30). the on by affected focused are terns mostly has microsaccades of in size the also and but 26) (27). (25, (25), pit density parafovea foveal cone the the in retina changes in with the only foveola, of the not shape reported the anatomy. been in retinal have differences and strategies individual different sensitivity strong these perceptual Indeed, that in possible is variations kept It reflect others While nose. mouth, the the reports. at on previous looked fixation primarily these observers the of move- of eye those most in resemble differences closely for individual centrally Yet ments period. presented long also relatively previ- (also were a smaller these they considerably of condition); just paradigms parafoveal not the in were from stimuli here Our ways used studies. of paradigm ous number experimental a The in regions. differs eye toward land or biases to strong nose saccade exhibit the individuals first 20). some the eyes, (14, of the peripherally tendency below of just general presented position a face is landing a there the Although toward we in saccade one reported the first difference been to the similar A already lookers (24). has mouth observed and artificial lookers central an nose when using between even change main- blocked not are is do they vision idiosyncrasies and occur (14), These time strategies over 20). scanning tained of (13–15, visual viewing individuals in During variations across scales. significant spatial face, across a maintained be are inves- also ration to can necessary or is work microsaccades question. Further this of systems. tigate other guidance by the accessed to restricted nsm u oksosta n clmtrbhvo is behavior oculomotor fine that shows work our sum, In on influences cognitive/attentional on research Previous explo- visual in differences individual that shows also work Our oa f3 merpchmnosres l av bu the about naive all observers, human emmetropic 31 of total A tmlspeetto n o eido fe h tmlswsturned these was In stimulus experiment. the the during after during interleaved s were time trials 4 any fixation of 1.5-s at period The controller off. a remote for a and on presentation button stimulus a pressing by responded of subject. order and the condition and each conditions for both randomized random- in was presentation was presented images were order images presentation same blocks’ face The The the ized. blocks. of in height run the were condition, conditions parafovea the 4.2 In not. measured or neutral and was task sion each for 2. randomized Experiment was presentation images conditions subject. of both in order in presented presentation presented the was blocks’ were images and The stimuli same blocks. of The in set randomized. run was same were order tasks The two display. The the conditions. of loca- both fixated center initially the face the 1.46 from at a distance measured tion or whether same face the ahead judge the at to of approximately straight were height asked eyes looking The were not. was they or face task smiling other was a the whether in asked whereas away, were they task one movements bar. bite 1. a microscopic Experiment on as even well compensate occur as to may electronics trial that the each in before by drifts position possible sight central for proce- the of calibration for manual line repeated The was 7). the dure (5, of magnitude of standard localization order than one 2D approximately position improving gaze cor- of procedures, cali- localization to single-step accurate dual-step joypad more then This a a were transformation. allows corrections used bration voltage-to-pixel These Subjects necessary. the screen. if into loca- location, the incorporated the calibration gaze automatic on while predicted the time grid the of the basis real rect the of in on points estimated displayed nine sight was of the line the of the refined each of phase, tion or this on In confirmed again calibration. fixated subjects automatic they the phase, by second given the mapping voltage-to-pixel In axes. vertical 3 1.32 and located a were of points points These experiments. nine motor sequentially the subjects positioned of phase, each first was the on In fixated subject phases. two The in pro- performed calibration minutes. was a Subsequently, cedure few apparatus. the in a comfortably and lasted optimally that operations Tasks. setup Experimental and task. Procedure 1. the experiment of in neutral. difficulty used the or were increase faces smiling to 186 and images of the total away looking A to manip- looking versions: added were and different was four neutral images noise in White or these presented smiling of was and mouth face straight the each images and of that database eyes so new ulated The a create the 1. to of experiment used subset was A for task. 2 the experiment of of difficulty of faces the contrast neutral increase the to were lowered 2 faces was experiment stimuli some the in the that Furthermore, and faces so others. obvious chosen than and too were ambiguous grayscale not faces more was The to mask. expressions converted oval between were an difference within images fit the facial to minimal of cropped had were All who females makeup. or or males white either their of views to frontal categories, were according main two prelabeled into (n faces were faces the neutral grouped used we 2 images experiment In The expression. 35). (34, databases on oculomotor stimulus estimated the of updates combination and desired gaze- the variables. time, flexible to real according in allows display them the which from processes signals tracker, processor, movement eye eye signal acquires the system digital This control. a display contingent on based for system recorded and kHz 1 at sampled were analysis. system positions subsequent 6 eye a horizontal Generation Technologies), and a of (Fourward tical noise of tracker internal means eye an by (DPI) with measured The spatial were patched. movements. Purkinje eye was and head Dual right eye prevented Hz left the headrest the of a 85 while and movements bar eye of bite right rate dental-imprint their A with refresh monocularly vertical task the a 2,048 at of resolution HM204DT) (Iyama itor Apparatus. and Stimuli voluntarily and study. the material in the part understood take fully to subject agreed the after only signed nbt xeiet tml eepeetdfr15sadsubjects and s 1.5 for presented were stimuli experiments both In online from taken faces of images using by generated were Stimuli custom-developed a (33), EyeRIS of means by rendered were Stimuli ◦ hra ntefvoacniini esrd1 measured it condition foveola the in whereas , ujcswr ntutdt efr w ifrn ak.In tasks. different two perform to instructed were Subjects ujcswr ntutdt ug hte aeexpres- face a whether judge to instructed were Subjects 2)adfcsepesn neoin(n emotion an expressing faces and 125) = × tml eedslydo atpopo R mon- CRT fast-phosphor a on displayed were Stimuli ,3 ies( ie 0.53 = pixel (1 pixels 1,536 ∼20 00 n pta eouino 1 of resolution spatial a and vr eso tre ihpreliminary with started session Every × rd si utmr noculo- in customary is as grid, 3 NSLts Articles Latest PNAS ◦ 0 .Osresperformed Observers ). pr ntehorizontal the on apart ◦ n ot and mouth and , 2) l faces All 125). = 0 5 2.Ver- 32). (5, ◦ h two The . | f8 of 7

PSYCHOLOGICAL AND COGNITIVE SCIENCES trials observers were instructed to fixate on a marker at the center of the Trials with blinks/loss of tracks (3.2%, 3.2%, and 4.9% of the total trials for display. parafoveal condition, foveal condition, and experiment 2, respectively) and trials with early responses (<700 ms, 6% of the total trials) were discarded. Data Analysis. Recorded eye movement traces were segmented into sep- To categorize gaze position during the task three regions were identified arate periods of drift and saccades. Classification of eye movements was on the stimulus: nose, eyes, and mouth. If the gaze was not in any of these performed automatically and then validated by trained laboratory person- regions, it was categorized as being on the background. Averages across nel with extensive experience in classifying eye movements. Periods of blinks observers in different conditions and tasks were examined by means of one- were automatically detected by the DPI eye tracker and removed from way within-subjects ANOVAs followed by Tukey post hoc tests. Comparisons data analysis. Only trials with optimal, uninterrupted tracking, in which the between two conditions and tasks across observers were tested using two- fourth Purkinje image was never eclipsed by the margin, were selected tailed paired t tests. for data analysis. Eye movements with minimal amplitude of 30 and peak On average, performance was evaluated over 153 trials per condition velocity higher than 3◦/s were selected as saccadic events. Saccades with an per observer. Figs. 2–5 show summary statistics across observers, and Fig. amplitude of less than 0.5◦ (300) were defined as microsaccades. Consecu- 2G shows average values for each individual observer. The data neces- tive events closer than 15 ms were merged together into a single saccade sary to generate all the figures (containing data) in the main manuscript to automatically exclude postsaccadic overshoots (36, 37). Saccade ampli- and the matlab scripts used to produce these figures have been deposited tude was defined as the vector connecting the point where the speed of on the Open Science Framework repository (https://osf.io/tusgd/?view only= the gaze shift grew greater than 3◦/s (saccade onset) and the point where ce1e605d448e4014b50e6a7548ae37ba). ◦ it became less than 3 /s (saccade offset). Periods that were not classified as ACKNOWLEDGMENTS. This work has been supported by National Science saccades or blinks were labeled as drifts. Foundation Grant NSF-BCS-1534932 (to M.P.).

1. Buswell GT (1935) How People Look at Pictures (Univ of Chicago Press, Chicago). 21. Ko HK, Poletti M, Rucci M (2010) Microsaccades precisely relocate gaze in a high visual 2. Yarbus AL (1967) Eye Movements and Vision (Plenum Press, New York). acuity task. Nat Neurosci 13:1549–1553. 3. Azzopardi P, Cowey A (1993) Preferential representation of the fovea in the primary 22. Bisley JW, Goldberg ME (2010) Attention, intention, and priority in the parietal lobe. . 361:719–721. Annu Rev Neurosci 9:1–21. 4. Rolfs M (2009) Microsaccades: Small steps on a long way. Vis Res 49:2415–2441. 23. Zelinsky GJ, Bisley J (2015) The what, where, and why of priority maps and their 5. Poletti M, Rucci M (2016) A compact field guide to the study of microsaccades: interactions with visual working memory. Ann N Y Acad Sci 1339:154–164. Challenges and functions. Vis Res 118:83–97. 24. Tsank Y, Eckstein MP (2017) Domain specificity of oculomotor learning after changes 6. Kuang X, Poletti M, Victor JD, Rucci M (2012) Temporal encoding of spatial in sensory processing. J Neurosci 37:11469–11484. information during active visual fixation. Curr Biol 20:510–514. 25. Curcio CA, Sloan KR, Kalina RE, Hendrickson AE (1990) Human photoreceptor 7. Poletti M, Listorti C, Rucci M (2013) Microscopic eye movements compensate for topography. J Comp Neurol 292:497–523. nonhomogeneous vision within the fovea. Curr Biol 23:1691–1695. 26. Li KY, Tiruveedhula P, Roorda A (2010) Intersubject variability of foveal cone 8. Williams CC, Henderson JM (2007) The face inversion effect is not a consequence of photoreceptor density in relation to eye length. Invest Ophthalmol Vis Sci 51: aberrant eye movements. Mem Cogn 35:1977–1985. 6858–6867. 9. Schurgin MW, et al. (2014) Eye movements during emotion recognition in faces. J Vis 27. Tick S, et al. (2011) Foveal shape and structure in a normal population. Invest 14:1–16. Ophthalmol Vis Sci 52:5105–5110. 10. Hsiao JH, Cottrell G (2008) Two fixations suffice in face recognition. Psychol Sci 28. Hafed ZM, Clark JJ (2002) Microsaccades as an overt measure of covert attention 19:998–1006. shifts. Vis Res 42:2533–2545. 11. Smith ML, Cottrell GW, Gosselin F, Schyns PG (2006) Transmitting and decoding facial 29. Engbert R, Kliegl R (2003) Microsaccades uncover the orientation of covert attention. expressions. Psychol Sci 16:184–189. Vis Res 43:1035–1045. 12. Peterson MF, Eckstein MP (2012) Looking just below the eyes is optimal across face 30. Yuval-Greenberg S, Merriam EP, Heeger DJ (2014) Spontaneous microsaccades reflect recognition tasks. Proc Natl Acad Sci USA 109:E3314–E3323. shifts in covert attention. J Neurosci 34:13693–13700. 13. Kanan C, Bseiso DNF, Ray NA, Hsiao JH, Cottrell GW (2015) Humans have idiosyncratic 31. Ludwig CJ, Davies JR, Eckstein MP (2014) Foveal analysis and peripheral selection and task-specific scanpaths for judging faces. Vis Res 108:67–76. during active visual sampling. Proc Natl Acad Sci USA 111:E291–E299. 14. Peterson MF, Eckstein MP (2013) Individual differences in eye movements during 32. Ko H-k, Snodderly DM, Poletti M (2016) Eye movements between saccades: Measuring face identification reflect observer-specific optimal points of fixation. Psychol Sci 33: ocular drift and tremor. Vis Res 122:93–104. 1216–1225. 33. Santini F, Redner G, Iovin R, Rucci M (2007) EyeRIS: A general-purpose sys- 15. Mehoudar E, Arizpe J, Baker CI, Yovel G (2014) Faces in the eye of the beholder: tem for eye movement contingent display control. Behav Res Methods 39:350– Unique and stable eye scanning patterns of individual observers. J Vis 14:1–10. 364. 16. Arizpe J, Kravitz DJ, Yovel G, Baker CI (2012) Start position strongly influences fixation 34. Minear M, Park DC (2004) A lifespan database of adult facial stimuli. Behav Res patterns during face processing: Difficulties with eye movements as a measure of Methods Instr Comput 36:630–633. information use. PLoS One 7:e31106. 35. Vieira TF, Bottino A, Laurentini A, Simone MD (2014) Detecting siblings in image pairs. 17. Blais C, Jack RE, Scheepers C, Fiset D, Caldara R (2008) Culture shapes how we look at Vis Comput 30:1333–1345. faces. PLoS One 3:e3022. 36. Deubel H, Bridgeman B (1995) Fourth Purkinje image signals reveal eye- 18. Rucci M, Iovin R, Poletti M, Santini F (2007) Miniature eye movements enhance fine deviations and retinal image distortions during saccades. Vis Res 35:529– spatial detail. Nature 447:852–855. 538. 19. Perlman SB, et al. (2009) Individual differences in personality predict how people look 37. Stevenson SB, Roorda A (2005) Correcting for miniature eye movements in high reso- at faces. PLoS One 4:1–6. lution scanning laser . Ophthalmic Technologies XV, Proceedings of 20. Peterson MF, Lin J, Zaun I, Kanwisher N (2016) Individual differences in face-looking SPIE, eds Manns F, Soderberg¨ PG, Ho A, Stuck BE, Belkin M (SPIE, Bellingham, WA), behavior generalize from the lab to the world. J Vis 16:1–18. Vol 5688, pp 145–151.

8 of 8 | www.pnas.org/cgi/doi/10.1073/pnas.1812222116 Shelchkova et al. Downloaded by guest on October 1, 2021