Unraveling the distributed neural of facial identity through spatiotemporal analysis

Adrian Nestor1, David C. Plaut, and Marlene Behrmann

Department of Psychology, Carnegie Mellon University, and Center for the Neural Basis of Cognition, Pittsburgh, PA 15213

Edited by Charles G. Gross, Princeton University, Princeton, NJ, and approved May 12, 2011 (received for review February 13, 2011) Face individuation is one of the most impressive achievements of Furthermore, insofar as face individuation is mediated by a our visual system, and yet uncovering the neural mechanisms network, it is important to determine how information is distrib- subserving this feat appears to elude traditional approaches to uted across the system. Some interesting clues come from the fact functional brain data analysis. The present study investigates the that right fusiform areas are sensitive to both low-level properties neural code of facial identity with the aim of ascertain- of faces (16, 25) and high-level factors (26, 27), suggesting that ing its distributed nature and informational basis. To this end, these areas may mediate between image-based and conceptual we use a sequence of multivariate pattern analyses applied to representations. If true, such an organization should be reflected in functional magnetic resonance imaging (fMRI) data. First, we the pattern of information sharing among different regions. combine information-based brain mapping and dynamic discrimi- The current work investigates the nature and the extent of nation analysis to locate spatiotemporal that support face identity-specific neural patterns in the human ventral cortex. We classification at the individual level. This analysis reveals a network examined functional MRI (fMRI) data acquired during face of fusiform and anterior temporal areas that carry information individuation and assessed the discriminability of activation about facial identity and provides evidence that the fusiform face patterns evoked by different facial identities across variation in area responds with distinct patterns of activation to different face expression. To uncover the neural correlate of identity recogni- identities. Second, we assess the information structure of the tion, we performed dynamic multivariate mapping by combining fi network using recursive feature elimination. We nd that diagnos- information-based mapping (28) and dynamic discrimination tic information is distributed evenly among anterior regions of the analysis (29). The results revealed a network of fusiform and NEUROSCIENCE mapped network and that a right anterior region of the fusiform anterior temporal regions that respond with distinct spatiotem- gyrus plays a central role within the information network mediat- poral patterns to different identities. To elucidate the distribu- fi ing face individuation. These ndings serve to map out and tion of information, we examined the distribution of diagnostic characterize a cortical system responsible for individuation. More fi information across these regions using recursive feature elimi- generally, in the context of functionally de ned networks, they nation (RFE) (30) and related the information content of dif- provide an account of distributed processing grounded in ferent regions to each other. We found that information is evenly information-based architectures. distributed among anterior regions and that a right fusiform region plays a central role within this network. he neural basis of face perception is the focus of extensive re- Tsearch as it provides key insights both into the computational Results architecture of visual recognition (1, 2) and into the functional Participants performed an individuation task with faces (Fig. 1) organization of the brain (3). A central theme of this research and orthographic forms (OFs) (Fig. S1). Specifically, they recog- emphasizes the distribution of face processing across a network of nized stimuli at the individual level across image changes in- spatially segregated areas (4–10). However, there remains consid- troduced by expression (for faces) or font (for OFs). Response erable disagreement about how information is represented and accuracy was at ceiling (>95%) as expected given the familiar- processed within this network to support tasks such as individuation, ization with the stimuli before scanning and the slow rate of expression analysis, or high-level semantic processing. stimulus presentation. Thus, behavior exhibits the expected invari- One influential view proposes an architecture that maps dif- ance to image changes, and the current investigation focuses on ferent tasks to distinct, unique cortical regions (6) and, as such, the neural subserving this invariance. draws attention to the specificity of this mapping (11–20). As a case in point, face individuation (e.g., differentiating Steve Jobs Dynamic Multivariate Mapping. The analysis used a searchlight (SL) from Bill Gates across changes in expression) is commonly map- with a 5-voxel radius and a 3-TR (Time to Repeat) temporal ped onto the fusiform face area (FFA) (6, 21). Although recent envelope to constrain spatiotemporal patterns locally. These patterns were submitted to multivariate classification on the basis studies have questioned this role of the FFA (14, 15), overall they Methods SI Text agree with this task-based architecture as they single out other of facial identity ( and ). The outcome of the areas supporting individuation. analysis is a group information-based map (28) revealing the strength of discrimination (Fig. 2). Each voxel in this map rep- However, various distributed accounts have also been consid- fi ered. One such account ascribes facial identity processing to resents an entire region of neighboring voxels de ned by the SL mask. multiple, independent regions. Along these lines, the FFA’s sen- sitivity to individuation has been variedly extended to areas of the inferior occipital gyrus (5), the superior temporal sulcus (12), Author contributions: A.N., D.C.P., and M.B. designed research; A.N. performed research; and the temporal pole (22). An alternative scenario is that identity A.N., D.C.P., and M.B. analyzed data; and A.N., D.C.P., and M.B. wrote the paper. is encoded by a network of regions rather than by any of its separate The authors declare no conflict of interest. — components such a system was recently described for subordinate- This article is a PNAS Direct Submission. level face discrimination (23). Still another distributed account Data deposition: The fMRI dataset has been deposited with the XNAT Central database attributes individuation to an extensive ventral cortical area rather under the project name “STSL.” than to a network of smaller separate regions (24). Clearly, the 1To whom correspondence should be addressed. E-mail: [email protected]. degree of distribution of the information supporting face individ- This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10. uation remains to be determined. 1073/pnas.1102433108/-/DCSupplemental.

www.pnas.org/cgi/doi/10.1073/pnas.1102433108 PNAS Early Edition | 1of6 Downloaded by guest on September 29, 2021 Fig. 2. Group information-based map of face individuation. The map is computed using a searchlight (SL) approach and estimates the discrimina- < Fig. 1. Experimental face stimuli (4 identities × 4 expressions). Stimuli were bility of facial identities across expression (q 0.05). Each voxel in the map fi matched with respect to low-level properties (e.g., mean luminance), ex- represents the center of an SL-de ned region supporting identity discrimi- ternal features (hair), and high-level characteristics (e.g., sex). Face images nation. The four slices show the sensitivity peaks of the four clusters revealed courtesy of the Face-Place Face Database Project (http://www.face-place.org/) by this analysis. Copyright 2008, Michael J. Tarr. Funding provided by NSF Award 0339122. The results above suggest that identity coding relies on a dis- Our mapping revealed four areas sensitive to individuation: two tributed cortical system. Clarifying the specificity of this system located bilaterally in the anterior fusiform gyrus (aFG), one in the to face individuation is addressed by our region-of-interest right anterior medial temporal gyrus (aMTG), and one in the left (ROI) analyses. posterior fusiform gyrus (pFG). The largest of these areas cor- responded to the right (r)aFG whereas the smallest corresponded ROI Analyses. First, we examined whether the FFA supports reli- fi to its left homolog (Table 1). able face individuation as tested with pattern classi cation. Bi- fi To test the robustness of our mapping, the same cortical volume lateral FFAs were identi ed in each subject using a standard face (Fig. S2A) was examined with SL masks of different sizes. These localizer, and discrimination was computed across all features in — alternative explorations produced qualitatively similar results a region given the use of spatiotemporal patterns, our features X (Fig. S3 A–C). In contrast, a univariate version of the mapping (SI are voxel time-point pairings rather than voxels alone. The Text fi analysis revealed above-chance performance for rFFA (Fig. 3). ) failed to uncover any signi cant regions, even at a liberal fi threshold (q < 0.10), attesting to the strength of multivariate; To reduce over tting, we repeated the analysis above using subsets of diagnostic features identified by multivariate feature mapping. fi To further evaluate these results, we projected the four areas selection, speci cally RFE. The method works by systematically removing features, one at a time, on the basis of their impact on from the group map back into the native space of each subject and classification (SI Text). Following this procedure, we found above- expanded each voxel to the entire SL region centered on it. The chance performance bilaterally in the FFA (Fig. 3). In contrast, resulting SL clusters (Fig. S3D) mark entire regions able to sup- early visual cortex (EVC) did not exhibit significant sensitivity port above-chance classification. Examination of subject-specific either before or after feature selection. maps revealed that the bilateral aFG clusters were consistently − Second, we reversed our approach by using multivariate map- located anterior to the FFA [rFFA peak coordinates, 39, 46, ping to localize clusters and univariate analysis to assess face se- and −16; left (l)FFA, −36, −47, and −18]. However, we also found they consistently overlapped with the FFA (mean ± SD: 24 ± 14% of raFG volume and 35 ± 20% of laFG). Table 1. Areas sensitive to face individuation Finally, multivariate mapping was applied to other types of Coordinates (peak) discrimination: expression classification (across identities) and category-level classification (faces versus OFs). Whereas the Region xyzSL centers (voxels) Peak t value former analysis did not produce any significant results, the latter − − found reliable effects extensively throughout the cortical volume raFG 33 39 9 16 11.31 B raMTG 19 6 −26 8 9.90 analyzed (Fig. S2 ). In contrast, a univariate version of the latter − − − analysis revealed considerably less sensitivity than its multivariate lpFG 26 69 14 2 7.11 laFG −29 −39 −14 1 7.24 counterpart (Fig. S2C).

2of6 | www.pnas.org/cgi/doi/10.1073/pnas.1102433108 Nestor et al. Downloaded by guest on September 29, 2021 exclusively dedicated to individuation or even to face processing per se. One hypothesis, examined below, may explain this in- volvement in multiple types of perceptual discrimination simply by appeal to low-level image properties.

Impact of Low-Level Image Similarity on Individuation. To deter- mine the engagement of the network in low-level perceptual processing, image similarity was computed across images of different individuals using an L2 metric (Table S1). For each pair of face identities, the average distance was correlated with the corresponding discrimination score produced by every ROI (in- cluding the FFA). The only ROI susceptible to low-level image sensitivity was laFG (P < 0.05 uncorrected) (Fig. S4). Fig. 3. Sensitivity estimates in three ROIs. Facial identity discrimination was These results, along with the inability of the EVC to support computed using both the entire set of features in an ROI and a subset of di- individuation, suggest that low-level similarity is unlikely to be agnostic features identified by multivariate feature selection (i.e., RFE), the the main source of the individuation effects observed here. two types of classification are labeled as pre- and post-RFE. The average number of features involved in classification is superimposed on each bar. The Feature Ranking and Mapping. Having located a network of regions results indicate that the bilateral FFA, in contrast to an early visual area, con- sensitive to face identity, we set out to determine the spatial and tains sufficient information to discriminate identities above chance (P < 0.05). temporal distribution of the features diagnostic of face individ- uation. Specifically, we performed RFE analysis jointly across all SL clusters and recorded the ranking of the features within lectivity. Concretely, we examined the face selectivity of the SL each subject. To eliminate any spatial bias within the initial fea- clusters using the data from our functional localizers. No reliable ture set, we started with an equally large number of features for face selectivity was detected for any cluster. each cluster: the 1,000 top-ranked ones based on single-cluster Third, SL clusters along with the FFA were tested for their ability RFE computation. to discriminate expressions across changes in identity. Above- Fig. 4A shows the average ranking of the 400 most informative chance discrimination was found in rFFA and raMTG (P < 0.05). features across subjects and Fig. 4B summarizes their distribution NEUROSCIENCE Finally, we tested our clusters for OF individuation across across clusters and time points. We found a significant effect of variation in font. The analysis found sensitivity in two regions: cluster (two-way analysis of variance, P < 0.05) but no effect of rFFA and lpFG. time point and no interaction. Further comparisons revealed that These findings are important in several respects. They suggest lpFG contains fewer features than other clusters (P < 0.05), which conventionally defined face selectivity, although informative, may did not differ among each other. The time course of feature not be enough to localize areas involved in fine-level face rep- elimination revealed that lpFG features were consistently elimi- resentation. Also, they show that the identified network is not nated at a higher rate than features from other clusters (Fig. 4C). A

Fig. 4. Spatiotemporal distribution of information diagnostic for face individuation. (A) Group map of average feature ranking for the top 400 features— rows show different slices and columns different time points. Color codes the ranking of the features across space (the four regions identified by our SL analysis) and time (4–8 s poststimulus onset). The map shows a lower concentration of features in the lpFG relative to other regions but a comparable number of features across time. (B) Average feature distribution across subjects by cluster and time point (the bar graph quantifies the results illustrated in A). (C) Time course of feature elimination by ROI for 4,000 features (top 1,000 features for each ROI). This analysis confirms that lpFG features are eliminated at a higher rate, indicative of their reduced diagnosticity (shaded areas show ±1 SE across subjects).

Nestor et al. PNAS Early Edition | 3of6 Downloaded by guest on September 29, 2021 similar analysis across time points showed no substantial differ- the discovery of small fine-grained pattern differences underlying ences across time (Fig. S5). the perception of same-category exemplars. Feature mapping provides a bird’s eye view of information distribution across regions. In our case, it reveals a relatively even Multiple Cortical Areas Support Face Individuation. Multivariate division of diagnostic information among anterior regions. Fur- mapping located four clusters in the bilateral FG and the right ther pairwise comparisons of SL clusters were deployed to ex- aMTG encoding facial identity information. These results in- amine how areas share information with each other. dicate that individuation relies on a network of ventral regions that exhibit sensitivity to individuation independently of each Information-Based Pairwise Cluster Analysis. Whereas activation other. This account should be distinguished both from local ones patterns in different areas are not directly comparable with each (6, 21) and from other versions of distributed processing (23, 24). other (e.g., because they have different dimensionalities, there is With regard to the specific clusters identified, previous work no obvious mapping between them, etc.), the classification results uncovered face-selective areas in the vicinity of the FFA, both they produce serve as convenient proxies (Methods). Here, we posterior (9) and anterior (33) to it. Our clusters did not exhibit compared classification patterns for pairs of regions while con- face selectivity when assessed with a univariate test. However, this fl trolling for the pattern of correct labels. Thus, the analysis focuses lack of selectivity may re ect the variability of these areas (9, 33) on common biases in misclassification. and/or the limitations of univariate analysis (34). Alternatively, it First, we computed the partial correlation between classification is possible that face processing does not necessarily entail face patterns corresponding to different clusters. Group results are selectivity (35). More relevantly here, face individuation, rather than face selectivity, was previously mapped to an area in the displayed as a graph in Fig. 5. Similarity scores for all pairs of anterior vicinity of the FFA (14). Overall, our present results regions tested above chance (one-sample t test, P < 0.01). Simi- confirm the involvement of these areas in face processing and larity scores within the network were not homogeneous (one-way establish their role in individuation. analysis of variance, P < 0.05) mainly because raFG evinced higher P < On a related note, the proximity of these fusiform clusters to similarity estimates than the rest ( 0.01). In addition, we the FFA may raise questions as to whether they are independent computed similarity scores between the four clusters and the EVC. clusters or rather extensions of the FFA (7, 9). On the basis of fi Average similarity scores within the network were signi cantly differences in peak location and the lack of face selectivity, we t P < higher than scores with the EVC (paired-sample test, 0.01). treat them here as distinct from the FFA although further in- fi Second, to verify our ndings using an information-theoretic vestigation is needed to fully understand their relationship with measure, we computed the conditional mutual information be- this area. tween classification patterns produced by different clusters (Fig. Unlike the FG areas discussed above, an anterior region of the 5). Examination of information estimates revealed a relational right middle temporal cortex (36, 37) or temporal pole (22) was structure qualitatively similar to that obtained using correlation. consistently associated with identity coding. Due to its sensitivity We interpret the results above as evidence for the central role of to higher-level factors, such as familiarity (22), and its involve- the right FG in face individuation. More generally, they support ment in conceptual processing (38, 39), the anterior temporal the idea of redundant information encoding within the network. cortex is thought to encode biographical information (6). Whereas our ability to localize this region validates our mapping method- Discussion ology, the fact that our stimuli were not explicitly associated with The present study investigates the encoding of facial identity in any biographical information suggests that the computations hos- the human ventral cortex. Our investigation follows a multivariate ted by this area also involve a perceptual component. Consistent approach that exploits multivoxel information at multiple stages with this, another mapping attempt, based on perceptual dis- from mapping to feature selection and network analysis. We favor crimination (15), traced face individuation to a right anterior this approach because multivariate methods are more successful temporal area. Also, primate research revealed encoding of face– at category discrimination than univariate tests (31) and possibly space dimensions in the macaque anterior temporal cortex (40) as more sensitive to “subvoxel” information than adaptation tech- well as different sensitivity to perceptual and semantic processing niques (32). In addition, we extend our investigation to take ad- of facial identity (41). In light of these findings, we argue that this vantage of spatiotemporal information (29) and, thus, optimize area is part of the network for perceptual face individuation al- though the representations it hosts are likely to also comprise a conceptual component. In sum, our mapping results argue for a distributed account of face individuation that accommodates a multitude of experi- mental findings. Previous imaging research may have failed to identify this network due to limits in the sensitivity of the methods used in relation with the size of the effect. Inability to find sen- sitivity in more than one region can easily lead to a local in- terpretation. At the same time, combining information from multiple regions may counter the limitations of one method but overestimate the extent of distributed processing. Our use of dy- namic multivariate mapping builds upon these previous findings and is a direct attempt to increase the sensitivity of these mapping methods at the cost of computational complexity. Importantly, the regions uncovered by our analysis may rep- Fig. 5. Pairwise ROI relations. The pattern of identity (mis)classifications is resent only a subset of the full network of regions involved in separately compared for each pair of regions using correlation-based scores identity processing. We allow for this possibility given our limited (red) and mutual information (brown). Specifically, we relate classification results across regions while controlling for the pattern of true labels. These coverage (intended to boost imaging resolution in the ventral measures are used as a proxy for assessing similarity in the encoding of facial cortex) as well as the lower sensitivity associated with the imaging identity across regions. Of the four ROIs, the raFG produced the highest of the inferior temporal cortex. In particular, regions of the su- scores in its relationship with the other regions (connector width is pro- perior temporal sulcus and prefrontal cortex (4, 10) are plausible portional to z values). additions to the network uncovered here.

4of6 | www.pnas.org/cgi/doi/10.1073/pnas.1102433108 Nestor et al. Downloaded by guest on September 29, 2021 Individuation Effects Are Not Reducible to Low-Level Image Processing. assign information evenly across regions. At the other, its structure To distinguish identity representations from low-level image dif- may be shaped by the feed-forward flow of information and display ferences, we appealed to a common source of intraindividual im- a clear hierarchy of regions from the least to the most diagnostic. age variations: emotional expressions. Also, identity was not The latter alternative is consistent, for instance, with a posterior- predictable in our case by prominent external features, such as to-anterior accumulation of information culminating in the re- hair, or by image properties associated with higher-level charac- cruitment of the aMTG as the endpoint of identity processing. teristics (e.g., sex or age). Furthermore, we assessed the contri- Our results fall in between these two alternatives. Anterior bution of interindividual low-level similarity to discrimination regions appear to be at an advantage compared with the left pFG. performance. Of all regions examined, only laFG showed potential At the same time, there was no clear differentiation among an- reliance on low-level properties. Finally, an examination of an terior regions in terms of the amount of information represented, early visual area did not produce any evidence for identity suggesting that information is evenly distributed across them. encoding. Taken together, these results render unlikely an expla- nation of individuation effects based primarily on low-level The Right aFG May Be a Hub in the Facial Identity Network. As dif- image properties. ferent regions are not directly comparable as activation patterns, we used their classification results as a proxy for their comparison. The Face Individuation System Does Not Support General-Purpose Using this approach, we found that different regions do share in- Individuation. To address the domain specificity of the system formation with each other, consistent with redundancy in identity identified, we examined whether “abstract” OFs (i.e., independent encoding. Furthermore, we found that pairwise similarities are of font) can be individuated within the regions mapped for faces. more prominent in relation with the right aFG than among other Although highly dissimilar from faces, OFs share an important network regions, suggesting that the raFG plays a central role attribute with them, by requiring fine-grained perceptual dis- within the face individuation network. Thus, the raFG mirrors the crimination at the individual level. In addition, they appear to role played by the right FFA among face-selective regions as compete with face representations (42) and to rely on similar visual revealed by functional connectivity (4). One explanation of this processing mechanisms (43). Thus, they may represent a more role is that a right middle/anterior FG area serves as an interface suitable contrast category for faces than other familiar categories, between low- and high-level information. such as houses. Our attempt to classify OF identities revealed that Two lines of evidence support this hypothesis. Recent results two regions, the lpFG and the rFFA, exhibited sensitivity to this show, surprisingly, that the FFA exhibits sensitivity to low-level kind of OF information. face properties (16, 25). Additionally, the right FG is subject to NEUROSCIENCE Furthermore, to examine the task specificity of the network, notable top–down effects (26, 27). Maintaining a robust interface we evaluated the ability of its regions to support expression between low-level image properties and high-level factors is discrimination and found that the rFFA, along with the aMTG, likely a key requirement for fast, reliable face processing. Critical was able to perform this type of discrimination. for our argument, this requirement would lead to the formation Thus, it appears that the mapped network does not support of an FG activation/information hub. Future combinations of general object individuation, although it may share resources with information and activation-based connectivity analyses might be the processing of other tasks as well as of other visual categories. able to assess such hypotheses and provide full-fledged accounts of the flow of information in cortical networks. FFA Responds to Different Face Identities with Different Patterns. The FFA (44, 45) is one of the most intensely studied functional Summary. A broad body of research suggests that face perception areas of the ventral stream. However, surprisingly, its role in face relies on an extensive network of cortical areas. Our results show processing is far from clear. At one extreme, its involvement in that a single face-processing task, individuation, is supported by – face individuation (6, 18, 21) has been called into question (14 a network of cortical regions that share resources with the pro- 16); at the other, it has been extended beyond the face domain to cessing of other visual categories (OFs) as well as other face- visual expertise (17) and even to general object individuation (13). related (expression discrimination) tasks. Detailed investigation Previous studies of the FFA using multivariate analysis have not of this network revealed an information structure dominated by been successful in discovering identity information (15, 24) or anterior cortical regions, and the right FG in particular, con- even subordinate-level information (23) about faces. The study of firming its central role in face processing. Finally, we suggest that fi patient populations is also not de nitive. FG lesions associated a full understanding of the operation of this system requires a com- with acquired prosopagnosia (46) adversely impact face individ- bination of conventional connectivity analyses and information- fi uation, con rming the critical role of the FFA. However, indi- based explorations of network structure. viduals with congenital prosopagnosia appear to exhibit normal FFA activation profiles (47) in the presence of compromised Methods fi ber tracts connecting the FG to anterior areas (48). Thus, the An extended version of this section is available in SI Text. question is more pertinent than ever: Does the FFA encode identity information? Design. Eight subjects were scanned across multiple sessions using a slow Our results provide evidence that the FFA responds consistently event-related design (10-s trials). Subjects were presented with a single face across different images of the same individual, but distinctly to or OF stimulus for 400 ms and were asked to identify the stimulus at the different individuals. In addition, we show that the right FFA can individual level using a pair of response gloves. We imaged 27 oblique slices individuate OFs and decode emotional expressions, consistent covering the ventral cortex at 3T (2.5-mm isotropic voxels, 2-s TR). with its role in expression recognition (12). Thus, we confirm the FFA’s sensitivity to face identity using Dynamic Information-Based Brain Mapping. The SL was walked voxel-by-voxel fi pattern analysis. Moreover, we show that it extends beyond both across a subject-speci c cortical mask. The mask covered the ventral cortex a specific task, i.e., individuation, and a specific domain, i.e., faces. (Fig. S2) and was temporally centered on 6-s poststimulus onset. At each location within the mask, spatiotemporal patterns (29) were extracted for Further research is needed to determine how far its individuation each stimulus presentation. To boost their signal, these patterns were av- capabilities extend and how they relate with each other. eraged within runs on the basis of stimulus identity. Pattern classification was performed using linear support vector machines (SVM) with a trainable Informative Features Are Evenly Distributed Across Anterior Regions. c term followed by leave-one-run-out cross-validation. Classification was How uniformly is information distributed across multiple regions? separately applied to each pair of identities (six pairs based on four identi- At one extreme, the system may favor robustness as a strategy and ties). Discrimination performance for each pair was encoded using d′ and an

Nestor et al. PNAS Early Edition | 5of6 Downloaded by guest on September 29, 2021 average information map (28) was computed across all pairs. For the pur- of performance, we executed two types of cross-validation. We performed pose of group analysis, these maps were normalized into Talairach space and cross-validation, first, at each RFE iteration step to measure performance and, examined for above-chance sensitivity (d′ > 0), using voxelwise t tests across second, across iteration steps to find the best number of features. RFE analysis subjects [false discovery rate (FDR) corrected]. Expression discrimination was applied to all types of ROIs described above. followed a similar procedure. Analyses were carried out in Matlab with the Parallel Processing Toolbox Pairwise ROI Analysis. We computed the similarity of classification patterns running on a ROCKS+ multiserver environment. produced by each pair of ROIs, namely the patterns of classification labels obtained with test instances during cross-validation. However, patterns are ROI Localization. Three types of ROIs were localized as follows: (i) We iden- likely correlated across regions by virtue of the ability of SVM models to tified locations of the group information map displaying above-chance dis- approximate true labels. Therefore, we measured pattern similarity with criminability and projected their coordinates in the native space of each partial correlation while controlling for the pattern of true labels. Correla- subject. ROIs were constructed by placing spherical masks at each of these tions were computed for each pair of facial identities, transformed using — locations a set of overlapping masks gave rise to a single ROI. (ii) Bilateral Fisher’s z, and averaged within subjects. Critically, to eliminate common fi – FFAs were identi ed for each subject by standard face object contrasts. ROIs biases based on spatial proximity between regions (because of spatially were constructed by applying a spherical mask centered on the FG face- correlated noise) all activation patterns were z-scored before classification. selective peak. (iii) Anatomical masks were manually drawn around the To obtain estimates of the information shared between ROIs we also com- calcarine sulcus of each subject and a mask was placed at the center of these puted the conditional mutual information between ROI-specific classifica- areas. The results serve as a rough approximation of EVC. All masks used for tion patterns given the pattern of true labels. ROI localization had a 5-voxel radius. ACKNOWLEDGMENTS. This work was funded by National Science Foundation RFE-Based Analysis. SVM-based RFE (30) was used for feature selection and Grant SBE-0542013 to the Temporal Dynamics of Learning Center and by ranking—the order of feature elimination provides an estimate of feature National Science Foundation Grant BCS0923763. M.B. was partially supported diagnosticity for a given type of discrimination. To obtain unbiased estimates by a Weston Visiting Professorship at the Weizmann Institute of Science.

1. DiCarlo JJ, Cox DD (2007) Untangling invariant object recognition. Trends Cogn Sci 11: 24. Natu VS, et al. (2010) Dissociable neural patterns of facial identity across changes in 333–341. viewpoint. J Cogn Neurosci 22:1570–1582. 2. Jiang X, et al. (2006) Evaluation of a shape-based model of human face discrimination 25. Yue X, Cassidy BS, Devaney KJ, Holt DJ, Tootell RB (2011) Lower-level stimulus features using FMRI and behavioral techniques. Neuron 50:159–172. strongly influence responses in the fusiform face area. Cereb Cortex 21:35–47. 3. Kanwisher N (2010) Functional specificity in the human brain: A window into the 26. Bar M, et al. (2006) Top-down facilitation of visual recognition. Proc Natl Acad Sci USA functional architecture of the mind. Proc Natl Acad Sci USA 107:11163–11170. 103:449–454. 4. Fairhall SL, Ishai A (2007) Effective connectivity within the distributed cortical network 27. Cox D, Meyers E, Sinha P (2004) Contextually evoked object-specific responses in for face perception. Cereb Cortex 17:2400–2406. human visual cortex. Science 304:115–117. 5. Gauthier I, et al. (2000) The fusiform “face area” is part of a network that processes 28. Kriegeskorte N, Goebel R, Bandettini P (2006) Information-based functional brain faces at the individual level. J Cogn Neurosci 12:495–504. mapping. Proc Natl Acad Sci USA 103:3863–3868. 6. Haxby JV, Hoffman EA, Gobbini MI (2000) The distributed human neural system for 29. Mourão-Miranda J, Friston KJ, Brammer M (2007) Dynamic discrimination analysis: A face perception. Trends Cogn Sci 4:223–233. spatial-temporal SVM. Neuroimage 36:88–99. 7. Pinsk MA, et al. (2009) Neural representations of faces and body parts in macaque 30. Hanson SJ, Halchenko YO (2008) Brain reading using full brain support vector “ ” fi and human cortex: A comparative FMRI study. J Neurophysiol 101:2581–2600. machines for object recognition: There is no face identi cation area. Neural – 8. Rossion B, et al. (2003) A network of occipito-temporal face-sensitive areas besides Comput 20:486 503. the right middle fusiform gyrus is necessary for normal face processing. Brain 126: 31. Haxby JV, et al. (2001) Distributed and overlapping representations of faces and – 2381–2395. objects in ventral temporal cortex. Science 293:2425 2430. 9. Weiner KS, Grill-Spector K (2010) Sparsely-distributed organization of face and limb 32. Sapountzis P, Schluppeck D, Bowtell R, Peirce JW (2010) A comparison of fMRI fi activations in human ventral temporal cortex. Neuroimage 52:1559–1573. adaptation and multivariate pattern classi cation analysis in visual cortex. – 10. Tsao DY, Schweers N, Moeller S, Freiwald WA (2008) Patches of face-selective cortex Neuroimage 49:1632 1640. in the macaque frontal lobe. Nat Neurosci 11:877–879. 33. Rajimehr R, Young JC, Tootell RB (2009) An anterior temporal face patch in human – 11. Calder AJ, Young AW (2005) Understanding the recognition of facial identity and cortex, predicted by macaque maps. Proc Natl Acad Sci USA 106:1995 2000. 34. Kriegeskorte N, Bandettini P (2007) Analyzing for information, not activation, to facial expression. Nat Rev Neurosci 6:641–651. exploit high-resolution fMRI. Neuroimage 38:649–662. 12. Fox CJ, Moon SY, Iaria G, Barton JJ (2009) The correlates of subjective perception of 35. Hadj-Bouziane F, Bell AH, Knusten TA, Ungerleider LG, Tootell RBH (2008) Perception identity and expression in the face network: An fMRI adaptation study. Neuroimage of emotional expressions is independent of face selectivity in monkey inferior 44:569–580. temporal cortex. Proc Natl Acad Sci USA 105:5591–5596. 13. Haist F, Lee K, Stiles J (2010) Individuating faces and common objects produces equal 36. Leveroni CL, et al. (2000) Neural systems underlying the recognition of familiar and responses in putative face-processing areas in the ventral occipitotemporal cortex. newly learned faces. J Neurosci 20:878–886. Front Hum Neurosci 4:181. 37. Sugiura M, et al. (2001) Activation reduction in anterior temporal cortices during 14. Nestor A, Vettel JM, Tarr MJ (2008) Task-specific codes for face recognition: How they repeated recognition of faces of personal acquaintances. Neuroimage 13:877–890. shape the neural representation of features for detection and individuation. PLoS 38. Damasio H, Tranel D, Grabowski T, Adolphs R, Damasio A (2004) Neural systems ONE 3:e3978. behind word and concept retrieval. Cognition 92:179–229. 15. Kriegeskorte N, Formisano E, Sorger B, Goebel R (2007) Individual faces elicit distinct 39. Simmons WK, Reddish M, Bellgowan PS, Martin A (2010) The selectivity and response patterns in human anterior temporal cortex. Proc Natl Acad Sci USA 104: functional connectivity of the anterior temporal lobes. Cereb Cortex 20:813–825. – 20600 20605. 40. Leopold DA, Bondar IV, Giese MA (2006) Norm-based face encoding by single neurons 16. Xu X, Yue X, Lescroart MD, Biederman I, Kim JG (2009) Adaptation in the fusiform in the monkey inferotemporal cortex. Nature 442:572–575. – face area (FFA): Image or person? Vision Res 49:2800 2807. 41. Eifuku S, Nakata R, Sugimori M, Ono T, Tamura R (2010) Neural correlates of 17. Gauthier I, Skudlarski P, Gore JC, Anderson AW (2000) Expertise for cars and birds associative face memory in the anterior inferior temporal cortex of monkeys. – recruits brain areas involved in face recognition. Nat Neurosci 3:191 197. J Neurosci 30:15085–15096. 18. Freiwald WA, Tsao DY, Livingstone MS (2009) A face feature space in the macaque 42. Dehaene S, et al. (2010) How learning to read changes the cortical networks for vision – temporal lobe. Nat Neurosci 12:1187 1196. and . Science 330:1359–1364. 19. Andrews TJ, Ewbank MP (2004) Distinct representations for facial identity and 43. Hasson U, Levy I, Behrmann M, Hendler T, Malach R (2002) Eccentricity bias as an – changeable aspects of faces in the human temporal lobe. Neuroimage 23:905 913. organizing principle for human high-order object areas. Neuron 34:479–490. 20. Pourtois G, Schwartz S, Seghier ML, Lazeyras F, Vuilleumier P (2005) View-independent 44. Puce A, Allison T, Asgari M, Gore JC, McCarthy G (1996) Differential sensitivity of coding of face identity in frontal and temporal cortices is modulated by familiarity: human visual cortex to faces, letterstrings, and textures: A functional magnetic An event-related fMRI study. Neuroimage 24:1214–1224. resonance imaging study. J Neurosci 16:5205–5215. 21. Kanwisher N, Yovel G (2006) The fusiform face area: A cortical region specialized for 45. Kanwisher N, McDermott J, Chun MM (1997) The fusiform face area: A module in the perception of faces. Philos Trans R Soc Lond B Biol Sci 361:2109–2128. human extrastriate cortex specialized for face perception. J Neurosci 17:4302–4311. 22. Rotshtein P, Henson RN, Treves A, Driver J, Dolan RJ (2005) Morphing Marilyn into 46. Barton JJ, Press DZ, Keenan JP, O’Connor M (2002) Lesions of the fusiform face area Maggie dissociates physical and identity face representations in the brain. Nat Neurosci impair perception of facial configuration in prosopagnosia. Neurology 58:71–78. 8:107–113. 47. Avidan G, Behrmann M (2009) Functional MRI reveals compromised neural integrity 23. Op de Beeck HP, Brants M, Baeck A, Wagemans J (2010) Distributed subordinate of the face processing network in congenital prosopagnosia. Curr Biol 19:1146–1150. specificity for bodies, faces, and buildings in human ventral visual cortex. Neuroimage 48. Thomas C, et al. (2009) Reduced structural connectivity in ventral visual cortex in 49:3414–3425. congenital prosopagnosia. Nat Neurosci 12:29–31.

6of6 | www.pnas.org/cgi/doi/10.1073/pnas.1102433108 Nestor et al. Downloaded by guest on September 29, 2021