Task-driven visual exploration at the foveal scale Natalya Shelchkovaa, Christie Tangb, and Martina Polettia,1 aDepartment of Neuroscience, University of Rochester Medical Center, Rochester, NY 14627; and bDepartment of Psychological and Brain Sciences, Boston University, Boston, MA 02215 Edited by Michael E. Goldberg, Columbia University College of Physicians, New York, NY, and approved January 31, 2019 (received for review July 18, 2018) Humans use saccades to inspect objects of interest with the look at the mouth region (9, 11), whereas scanning the upper foveola, the small region of the retina with highest acuity. This part of the face is mostly associated with recognition tasks (10). process of visual exploration is normally studied over large When the face is presented at an eccentric location, the first scenes. However, in everyday tasks, the stimulus within the saccade to the face is the most important for facial recogni- foveola is complex, and the need for visual exploration may tion (12). It normally brings the gaze close to the nose, but its extend to this smaller scale. We have previously shown that exact landing location is also biased by the task demands (12). fixational eye movements, in particular microsaccades, play an Therefore, by examining saccade landing positions and the scan important role in fine spatial vision. Here, we investigate whether paths of observers looking at faces it is possible to infer the task task-driven visual exploration occurs during the fixation pauses performed (13). Crucially, while these patterns of visual explo- in between large saccades. Observers judged the expression of ration are seen in most subjects, there are significant individual faces covering approximately 1◦, as if viewed from a distance of variations (13–15). many meters. We use a custom system for accurately localizing Visual exploration of faces, as visual exploration of scenes, has the line of sight and continually track gaze position at high resolu- been primarily examined using stimuli spanning many degrees tion. Our findings reveal that active spatial exploration, a process of visual angle. These stimuli cover not just the fovea, but driven by the goals of the task, takes place at the foveal scale. The also the parafovea and the visual periphery. However, humans scanning strategies used at this scale resemble those used when view faces from a range of different distances, and the ability examining larger scenes, with idiosyncrasies maintained across to recognize facial expressions extends to spatial scales much spatial scales. These findings suggest that the visual system pos- smaller than those typically studied. Humans can tell whether sesses not only a coarser priority map of the extrafoveal space to somebody is angry or happy or whether somebody is looking guide saccades, but also a finer-grained priority map that is used at them, even when the person is many meters away. In these COGNITIVE SCIENCES PSYCHOLOGICAL AND to guide microsaccades once the region of interest is foveated. circumstances, the face may cover less than 1◦, and the dis- tance between the different features may be in the order of microsaccades j ocular drift j fine spatial vision j face perception j arcminutes. There are mainly two reasons why visual explo- priority maps ration at the scale of the foveola has been little investigated. First, it is often implicitly assumed that the visual system sim- isual exploration is traditionally studied with scenes that ply needs to maintain fixation once a stimulus is foveated, and Vcover a relatively large portion of the visual field. In these the need for further exploration is not immediately recognized. conditions, saccades redirect the center of gaze toward interest- ing objects, so that they can be inspected with the high-acuity Significance foveola. It is well established that humans tend to look at the most informative regions of the scene (1) and that this process Visual exploration is driven by saccades, which bring the is influenced by the goals of the task (2). The foveola covers objects of interest into the small high-acuity portion of the ◦ only ≈1 of visual angle, less than 0.1% of the visual field (3). visual field, the foveola. While visual exploration is gener- Nevertheless, because of the fractal statistics of natural scenes ally studied at large spatial scales, here we show that it also and the scaling of retinal receptors, the input stimulus in this occurs at a much finer scale when examining complex foveal region is as complex as anywhere else on the retina. Can the stimuli. By using high-precision methods for localizing the concept of top–down, task-driven visual exploration extend to gaze, we show that, in the periods in between saccades, tiny the much smaller scale of the foveola during the intersaccadic gaze shifts actively inspect the stimulus based on the task intervals? demands. These visual scanning strategies closely resemble During fixation the eyes are never at rest but continue to move those adopted for exploring larger scenes, with individual dif- with a jittery motion, known as ocular drift, and with microsac- ferences maintained across spatial scales. Fine spatial vision, ◦ cades, small saccades (<0.5 ) that keep the stimulus within the therefore, results from a synergy of visual processes, motor foveola (4, 5). These eye movements are crucial for fine spatial behaviors, and cognitive factors. vision (6, 7). In laboratory tasks, microsaccades are finely tuned to bring the preferred locus of fixation on fine spatial patterns Author contributions: N.S., C.T., and M.P. designed research; N.S., C.T., and M.P. per- (7). In this study, we investigated whether microsaccades, rather formed research; M.P. supervised the research; N.S., C.T., and M.P. analyzed data; and than being a simple recentering mechanism, are used to explore N.S. and M.P. wrote the paper.y naturally complex foveal stimuli, in the same way humans use The authors declare no conflict of interest.y saccades to examine large visual scenes. To examine this we used This article is a PNAS Direct Submission.y human faces. This open access article is distributed under Creative Commons Attribution-NonCommercial- Appropriately interpreting facial expressions and gaze direc- NoDerivatives License 4.0 (CC BY-NC-ND).y tion are fundamental human abilities, and the visuomotor system Data deposition: The data necessary to generate all the figures (containing data) in is highly specialized in extracting information from faces. Gen- the main manuscript and the matlab scripts used to produce these figures have been deposited on the Open Science Framework repository (https://osf.io/tusgd/?view only= erally, humans scan faces using a “T” pattern (8, 9). When ce1e605d448e4014b50e6a7548ae37ba).y performing a facial recognition task, the first two saccades are 1 To whom correspondence should be addressed. Email: martina [email protected]. the most relevant as performance saturates after two fixations edu.y (10). During this period the visual system optimizes the acquisi- This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10. tion of information by looking at the most diagnostic features. 1073/pnas.1812222116/-/DCSupplemental.y As a result, when judging facial expression, humans tend to www.pnas.org/cgi/doi/10.1073/pnas.1812222116 PNAS Latest Articles j 1 of 8 Downloaded by guest on October 1, 2021 Second, whereas examining visuomotor scanning strategies over on the background. If exploration of complex foveal stimuli is a large visual scene is relatively straightforward, being able to top–down driven, we expect the pattern of eye movements to accurately localize gaze within a region as small as the foveola is a systematically change in the two tasks. The pattern of eye move- challenging task. ments on the stimulus was examined at high resolution while Here, we used high-resolution eye tracking and a state-of- subjects performed the task. the-art system for gaze-contingent control that enables more accurate localization of the line of sight compared with standard Influence of the Task on the Examination of Foveal Stimuli. Our find- techniques (5). We examined the oculomotor behavior at fixation ings show that, despite the small size of the stimuli, and despite by precisely mapping gaze position onto the foveal stimulus. In the fact that they were already ideally placed within the foveola two experiments, we first examined whether visual exploration at to perform both tasks, subjects actively examined them using dif- the foveal scale is top–down guided based on the task demands, ferent scanning patterns in the two tasks. When asked to judge while the physical stimulus remains unchanged. Then, we investi- gaze direction, subjects’ gaze shifted toward the eyes region gated how visual exploration at the scale of the foveola compares (Fig. 2 A and B); on the other hand, when judging facial expres- to the exploration at a larger scale. sion, subjects spent more time on the mouth region (Fig. 2 C and D and Movie S1). Microsaccadic behavior was very consis- Results tent across subjects; most of the microsaccades landed on the To explore whether task-driven visual exploration extends to the eyes in the gaze direction task (0.70 ± 0.13 on the eyes vs. 0 fine scale of the foveola we conducted a simplified version of microsaccades landing on the mouth; P < 0.0001, two-tailed a “Yarbus experiment”; subjects performed two different tasks paired t test), but this pattern flipped when judging facial expres- with the same set of stimuli. In one task participants were asked sion, with most microsaccades landing on the nose and on the to judge whether a face was looking at them and in another mouth (0.1 ± 0.10 on the eyes vs. 0.5 ± 0.33 on the mouth; P = task whether the face was smiling at them (Fig.