Memory and Perception-based Facial Image Reconstruction

by

Chi-Hsun Chang

A thesis submitted in conformity with the requirements for the degree of Master of Arts Department of Psychology University of Toronto

© Copyright by Chi-Hsun Chang 2016

Memory and Perception-based Facial Image Reconstruction

Chi-Hsun Chang

Master of Arts

Department of Psychology University of Toronto

2016 Abstract

Face perception and face memory have been the focus of extensive research to investigate the mechanisms of face processing. However, the nature of the representations underlying face perception and face memory remains unclear, and the relationship between them is not well- known given that they are usually studied separately. The current work examines these issues by adopting an image reconstruction approach, to perform behavioural perception and memory- based reconstructions. Significant features underlying the representation of face perception were first derived, and then used to reconstruct face images that were visually seen and recalled from memory. Reconstructions of perception and memory data were above chance for both unfamiliar faces, faces learned throughout the experiment, and celebrity faces retrieved from long-term memory. This not only provides new insights into the content of face memory and its relationship to face perception, but also opens a new path for practical applications such as computer-based ‘sketch artists’.

ii

Acknowledgments

I would first like to sincerely thank my supervisor Dr. Andy Lee and subsidiary advisor Dr. Adrian Nestor for their support and guidance over the past year. I would also like to thank Dr. Jonathan Cant as the third reader of this thesis and appreciate his valuable feedback on this thesis. Thank you to all members of the Lee and Nestor labs, particularly Dan Nemrodov, and Edward O’Neil for their useful ideas and comments on this work, as well as Ashutosh Patel and Deepika Elango for their help with data collection. I would like to thank Dr. Katherine Duncan for her kindness in providing me space in her lab for data collection. I am immensely grateful to people in my support network: Fang Liu, Tzu-Han Cheng, and Michelle Zhang, for their continuous encouragements. Lastly, I must express my profound gratitude to my parents and family for providing me unfailing support throughout my life. This accomplishment would not have been possible without them.

iii

Table of Contents

Acknowledgments ...... iii

Table of Contents ...... iv

List of Figures ...... vi

Chapter 1 Introduction ...... 1

Introduction ...... 1 1.1 Face Processing ...... 1 1.1.1 Face Perception ...... 1 1.1.2 Face Identification ...... 4 1.1.3 Interim Summary ...... 4 1.2 Representation underlying Face Space ...... 5 1.3 Face Memorability ...... 6 1.4 Face Image Reconstruction from memory ...... 6 1.5 Current Study ...... 7

Chapter 2 Materials and Methods ...... 9

Materials and Methods ...... 9 2.1 Experiment 1 ...... 9 2.1.1 Participants ...... 9 2.1.2 Stimuli ...... 9 2.1.3 Screening Questionnaires and Tests ...... 9 2.1.4 Task Design ...... 10 2.1.5 Experimental Procedures ...... 12 2.1.6 Analyses ...... 13 2.2 Experiment 2 ...... 15 2.2.1 Participants ...... 15 2.2.2 Stimuli ...... 15 2.2.3 Screening Questionnaires and Tests ...... 16 2.2.4 Two-alternative forced choice task ...... 16 2.2.5 Experimental Procedures ...... 16 2.2.6 Analyses ...... 17

iv

Chapter 3 Results ...... 18

Results ...... 18 3.1 Face Space and Classification Images ...... 18 3.2 Perception-based Reconstructions ...... 18 3.3 Memory-based Reconstructions ...... 19 3.4 Subjective Accuracy from Naïve Participants ...... 19 3.5 Regression Models ...... 20

Chapter 4 Discussions and Conclusions ...... 22

References ...... 31

v

List of Figures

Figure 1 ...... 26

Figure 2 ...... 27

Figure 3 ...... 29

Figure 4 ...... 30

vi 1

Chapter 1 Introduction Introduction

Face perception is an important ability in human beings as people are social animals. Infant shows the preference of faces at a very early stage in development (Johnson, Dziurawiec, Ellis, & Morton, 1991; Mondloch et al., 2015) and this ability becomes a highly developed skill across the whole life span. Specifically, humans are adept in face identification, a higher-level visual and memory process that involves the retrieval of the memory of faces and the identity information stored in memory (i.e. person semantic knowledge). These topics have been studied extensively, with a plethora of behavioral and neural evidence revealing the mechanisms underpinning face processing (Gobbini & Haxby, 2007; Haxby, Hoffman, & Gobbini, 2000).

1.1 Face Processing

1.1.1 Face Perception

Past studies have suggested that perceptual face information is represented in a multi- dimensional face space, in which individual faces deviate from the mean face along different dimensions (Johnston, Milne, Williams, & Hosie, 1997; Leopold, O’Toole, Vetter, & Blanz, 2001; Loffler, Yourganov, Wilkinson, & Wilson, 2005; Valentine, 1991). Early behavioral studies demonstrated that typical faces are represented closer to the mean face and distinctive faces are represented farther from the mean face in face space (Johnston et al., 1997; Valentine, 1991). Recent neural imaging research has added to this by revealing that faces are represented within a polar coordinate scheme, with faces along different axes being represented by distinct neuron populations (Loffler et al., 2005). Thus, these findings provide a conceptual framework for the representation of face perceptual information.

Regarding the specific localization of perceptual face information in the brain, early single-unit recording studies of non-human primate revealed that neurons in the temporal lobe respond selectively to face stimuli (Desimone, Albright, Gross, & Bruce, 1984; Perrett et al., 1985; Perrett, Hietanen, Oram, Benson, & Rolls, 1992). For example, Desimone et al. (1984) presented a series of stimuli including monkey faces, human faces and hands to monkeys and recorded the

2 firing rate of neurons in the inferior temporal (IT) cortex. They found that a group of neurons showed higher firing rate for monkey faces than for human faces or hands, and the activity of these neurons decreased when the eyes or nose of the monkey faces were removed. Furthermore, studies using functional magnetic resonance imaging (fMRI) have found several face-selective cortical patches in both the monkey’s anterior and posterior temporal lobe, in which the response of neurons were at least twice stronger for face stimuli than for other objects (Tsao, Freiwald, Knutsen, Mandeville, & Tootell, 2003; Tsao, Freiwald, Tootell, & Livingstone, 2006; Tsao, Moeller, & Freiwald, 2008). These results suggest that certain cortical areas in the primate brain are involved in the visual analysis of face information.

In human studies using fMRI, researchers have also found brain regions associated with the perceptual processing of faces, with these areas exhibiting greater activity during the viewing of faces compared to other stimulus categories (Gauthier et al., 2000; Grill-Spector, Knouf, & Kanwisher, 2004; Kanwisher, McDermott, & Chun, 1997; Puce, Allison, Asgari, Gore, & McCarthy, 1996). One area that has been suggested to be a human counterpart of the monkey face-sensitive middle face patch in posterior temporal lobe is the fusiform face area (FFA), which is located in the lateral middle fusiform gyrus (Tsao et al., 2003, 2006). This area has been found to be associated with the holistic processing of faces (Kanwisher et al., 1997; Liu, Harris, & Kanwisher, 2010; Mccarthy, Puce, Gore, & Truett, 1997; Schiltz, Dricot, Goebel, & Rossion, 2010; Zhang, Li, Song, & Liu, 2012) and the representation of higher-level features associated with the later stages of face processing such as facial expressions (Harry, Williams, Davis, & Kim, 2013; Hoffman & Haxby, 2000). Another region – occipital face area (OFA) – located near the inferior occipital gyrus, has also been proposed to play a role in the early stages of face processing (Pitcher, Walsh, & Duchaine, 2011; Pitcher, Walsh, Yovel, & Duchaine, 2007). Several studies have shown that this region represents the low-level features of the face such as eyes, nose and mouth (Liu, Harris, & Kanwisher, 2010; Pitcher et al., 2007). The superior temporal sulcus (STS) has also been proposed to be involved in face perception. Specifically, it represents the changeable features of the face, such as eye gaze (Hoffman & Haxby, 2000) or emotional expressions (Winston, 2004).

In addition to the above, a homologous region of the face-sensitive patches in the monkey anterior temporal lobe has also been discovered in the human brain in the anterior temporal cortex (Rajimehr, Young, & Tootell, 2009). Evidence from recent patient research has revealed

3 that an associative prosopagnosia patient with damage to the anterior temporal lobe (ATL), who is considered to be unable to recognize facial identity, is also significantly impaired at discriminating a target face from a perceptually similar distractor face (Busigny et al., 2014), suggesting that ATL also contributes to the holistic processing of faces. Also, a fMRI study using multivoxel pattern analysis (MVPA) revealed that the classification accuracy of novel faces was above chance level in the ATL voxels (Anzellotti, Fairhall, & Caramazza, 2013). Anzellotti and his colleague presented multiple 3D faces at different viewpoints generated by the computer to participants and asked them to determine whether the presented image was the target face identity or not. A linear SVM classifier was then used to classify the pattern activity in ATL ROI. Since the 3D faces were previously unknown to the participants, this evidence suggests that activity in this region reflects the visual information of faces that is not connected with semantic or social knowledge.

Meanwhile the perirhinal cortex (PRC), which is located in the anterior collateral sulcus, has been suggested to be another homologous region to the nonhuman primate anterior temporal face-sensitive patches (O’Neil, Cate, & Kohler, 2009; Rajimehr et al., 2009). In an fMRI study, O’Neil, Barkley, & Köhler (2013) found that activity in PRC was modulated by different representational task demands placed on face processing, including face individuation, which requires feature integration, and feature search which only requires separate feature processing. Furthermore, later research revealed that there was significant resting-state functional connectivity between PRC and other face-sensitive regions such as FFA (O’Neil, Hutchison, McLean, & Köhler, 2014). Similarly, a handful of studies have also demonstrated PRC activity during the perceptual judgments of faces (Barense, Henson, Lee, & Graham, 2010; Lee, Scahill, & Graham, 2008) and patients with PRC damage have been shown to be significantly impaired at complex visual discrimination tasks involving faces, for example, picking the odd face from an array of faces presented from different viewpoints (Lee, 2006; Lee et al., 2005). These findings are consistent with a wider suggested role of PRC in combining visual object features (Murray, Bussey, & Saksida, 2007; Saksida & Bussey, 2010) and support the importance of the conjunction of facial features in face perception.

4

1.1.2 Face Identification

Besides a role in face perception, the ATL has also been suggested to be involved in the processing of face identification (Collins & Olson, 2014; Gobbini & Haxby, 2007; Haxby et al., 2000). For example, patient studies have reported that people with damage to the ATL suffer from associative prosopagnosia or an impairment in face recognition (Barton, 2008; Barton & Cherkasova, 2003; Tippett, Miller, & Farah, 2000). Also, functional neuroimaging studies have found greater ATL activity for familiar compared to unfamiliar faces (Leveroni et al., 2000; Sugiura et al., 2001), and recent MVPA fMRI work has revealed that multivoxel patterns in ATL can be used to distinguish between different facial identities (Kriegeskorte, Formisano, Sorger, & Goebel, 2007; Nestor, Plaut, & Behrmann, 2011).

Beyond the process of face identification, the ATL is also believed to be important for person knowledge and the semantic information pertaining to a given identity. The ATL has been shown to respond more to faces for which participants have greater amount of semantic knowledge (e.g. faces of celebrities) than faces for which participants have less semantic knowledge (e.g. previously unfamiliar faces learned during experiment testing) (Leveroni et al., 2000; Nakamura et al., 2000; Sugiura et al., 2001). Converging with this, ATL activity has been reported to be modulated by the semantic richness or uniqueness of faces (Ross & Olson, 2012; Tsukiura et al., 2010) and patients with semantic dementia, a neurodegenerative disorder associated with a gradual cross-modal loss of semantic knowledge and atrophy to the ATL (Hodges, Patterson, Oxbury, & Funnell, 1992; Mummery et al., 2000; Snowden, Goulding, & Neary, 1989), usually exhibit an impairment in famous face/name recognition (Snowden, Thompson, & Neary, 2004).

1.1.3 Interim Summary

To summarize, a large number of studies have implicated the FFA, OFA and STS as the face- sensitive regions contributing to face perception, whereas the ATL contributes to both face perception and face identification. Notably, however, the precise nature of the representations underlying face perception and identification are not fully understood. For example, it is unclear what facial features or dimensions of the multi-dimensional face space are crucial to the representation of face information and moreover, what the differences and similarities are between the representations underlying face perception and face memory.

5

1.2 Representation underlying Face Space

Recent studies have used fMRI and MVPA to investigate the essence of face representations and further understand the facial features that are key to face processing (Goesaert & Op de Beeck, 2013; Liu et al., 2010). For example, Goesaert et al (2013) analyzed the SVM decoding sensitivity (a measure used in machine learning) to configuration and feature differences in several ROIs. The results revealed that some facial features (e.g. eyes, eyebrows and lips) and their configuration are represented in the face-sensitive areas such as FFA and OFA. An alternative approach to the use of machine learning to explore face representations is the use of image reconstruction, in which brain activity is used to reconstruct images that have been presented to participants. Early studies using this approach have achieved successful image reconstruction of visual information from the fMRI signal in the early visual areas (Brouwer & Heeger, 2009; Miyawaki et al., 2008; Naselaris, Prenger, Kay, Oliver, & Gallant, 2009; Nishimoto et al., 2011). More recently, three fMRI studies have successfully reconstructed face images from the neural activity in face-selective regions and demonstrated that some facial features are particularly important to this process (Cowen, Chun, & Kuhl, 2014; Lee & Kuhl, 2016; Nestor, Plaut, & Behrmann, 2016). For example, Nestor et al. collected both behavioral and fMRI data while presenting pairs of faces with two expressions and asking participants to judge if the presented faces were the same or different. Metric multidimensional scaling was then applied to behavioral and neural confusability matrices to create behavioural and neural face space. They found that features such as eyebrow thickness, skin tone, nose shape and mouth height were relevant to the facial identity in face space. Further, face images were reconstructed from behavioral and neural data and the reconstruction quality was assessed objectively and subjectively. Results showed that reconstruction accuracy of each identity was above chance and was significant in the bilateral fusiform gyrus ROI. These results provide a preliminary understandings of the nature of representations of faces.

It is important to note that the analyses and image reconstruction processes used in most prior studies were based on fMRI activity recorded while participants were physically viewing face stimuli. These results may, therefore, be unable to provide clear insight into how face information is represented in the brain since it is difficult to distinguish whether the observed representations are stimulus-driven or truly internally coded. Additionally, the representation

6 underlying face memory has not yet been elucidated. Thus, a crucial avenue of research will be the reconstruction of faces from memory.

1.3 Face Memorability

How well faces can be remembered is important for our ability to identify different faces and this has been a focus of extensive research. Past studies have shown that distinctive or atypical faces are remembered better than typical faces (Bartlett, Hurry, & Thorley, 1984; Shepherd, Gibling, & Ellis, Haydn, 1991; Winograd, 1981). In addition, familiarity and within-group face similarity have been suggested to be related with face memorability (Cheung & Gauthier, 2010; Yotsumoto, Kahana, Wilson, & Sekuler, 2007). Similarly, research has reported that faces of one’s own race (Byatt & Rhodes, 2004; Chiroro & Valentine, 1995; Meissner & Brigham, 2001) or faces of one’s own social group (Bernstein, Young, & Hugenberg, 2007) are remembered more accurately than other-race faces. Moreover, higher level facial traits such as emotional expression (D’Argembeau, Van der Linden, Etienne, & Comblain, 2003), attractiveness (Wiese, Altmann, & Schweinberger, 2014) and trustworthiness (Oosterhof & Todorov, 2008; Rule, Slepian, & Ambady, 2012) have been shown to influence face memory. Further, a recent study examined the simultaneous influence of an entire pool of facial factors on face memorability and found additional attributes, such as responsibility, kindness, happiness as significant predictors of successful memory retrieval (Bainbridge, Isola, & Oliva, 2013). However, even when all such attributes were controlled, a large amount of variance was still unexplained. Thus, while informative, previous work is largely constrained by the examination of specific factors warranted by specific hypotheses and theories in the field.

Our work aims to overcome the limitation above by addressing a related but more general question. That is, instead of asking what specific factors contribute to face memorability, we were interested in what comprises the detailed content of memory for faces. To this aim we deployed the image reconstruction methodology to reconstruct face images from memory. This allowed us to approximate and visualize the concrete pictorial content of face memory.

1.4 Face Image Reconstruction from memory

One recent study has reported that faces held in a retro-cue working memory task were successfully reconstructed from activity patterns of angular gyrus (Lee & Kuhl, 2016). The

7 authors performed principle component analysis (PCA) on a set of perceptual training faces to extract face components called eigenface scores such that each training face is defined by a weighted sum of the eigenface scores. Next, a regression model was estimated to map the eigenface scores to fMRI activity. Faces maintained in working memory were then successfully reconstructed from a linear combination of eigenface scores predicted by the neural pattern activity evoked by these faces. This finding suggests that the reconstruction approach can be applied not only to perceived faces but also to faces in memory. However, there are a number of caveats with this recent work. First, the face images that were used varied in terms of gender, expression, and ethnicity, and contained hair and other accessories. It is possible, therefore, that the reconstruction of faces from memory was influenced by these irrelevant features. Second, this work does not speak to the nature of the representation of perceptual face space and its memory counterpart face space was not investigated. Lastly, since the reconstruction of faces in memory were based on neural activity, it is still unknown whether the memory reconstructions can be achieved from behavioural data.

1.5 Current Study

The current study adopted the image reconstruction approach used in previous studies to examine whether face images held in memory can be successfully reconstructed based on individual behavioural data, and further explored what facial attributes are represented in face memory by investigating face space. To achieve this, I conducted two experiments. In the first experiment, a face space was first generated from a behavioural confusability matrix of perceptual face images. These face images contained only a specific subgroup of faces without any hair or accessory in order to avoid large variances in stimuli. The similarities between these visually presented faces and each of the memory-based reconstruction targets that were imagined by the participants were then collected. These similarity ratings provided information on how faces are represented in the absence of physical stimuli and allowed us to compute the location of each of the three learned faces in the multi-dimensional face space, and subsequently, to reconstruct them. The reconstructions were evaluated objectively with respect to the image pixel intensities. In the second experiment, the quality of the reconstructions was further assessed subjectively by participants in the first experiment and an independent group of subjects. It is important to note that compared to the previous studies conducting reconstructions at the group

8 level, the reconstruction analyses performed in my research were at an individual level, given that each individual has their own unique experience in forming memories of faces.

Thus, this study aims to provide both theoretical and practical contributions. On the one hand, successful reconstruction of faces in memory offers evidence of what facial features are crucial for the encoding and retrieval of faces. Additionally, if faces stored in memory can be reconstructed based on visual features represented in perceptual face space, then one may conclude that common representations underlie both face perception and memory. Finally, on a practical level, this work involves a behavioural experimental paradigm that could potentially lead to the automatic sketching of face images. The success of the reconstructions generated from individual participant’s behavioural data will allow us to understand what an individual is seeing or imagining and can be applied to different fields such as forensic science.

9

Chapter 2 Materials and Methods Materials and Methods 2.1 Experiment 1

This experiment aimed to reconstruct face images that were visually seen and were held in long- term memory. The reconstruction accuracy was assessed objectively via the pixelwise L2 metric for each individual’s reconstructions.

2.1.1 Participants

I aimed to assess the method on three participants and they were selected from 13 healthy adult volunteers based on their performance in a number of screening tests. On passing the screening criteria (Experimental Procedures), these three participants (NC, Female, 22 years; CB, Female, 21 years; SA, Male, 26 years) completed the whole experiment. All had normal or corrected-to- normal vision, with no history of neurological or psychiatric disorders.

2.1.2 Stimuli

Fifty-seven unfamiliar Caucasian male face images from a previous study (Nestor et al., 2016) and thirty face images of famous individuals collected from publicly available online sources were used in this study. Three additional unfamiliar faces from the same past study were chosen as targets for memory-based reconstruction. All facial images were front-view, with a neutral expression, and without hair or any facial accessories. These images were spatially normalized using the positions of the eyes and the nose, and normalized with the same mean and root mean square (RMS) contrast for each color channel in CIEL*a*b* color space.

2.1.3 Screening Questionnaires and Tests

Demographic questionnaire. Participant information such as age, ethnicity, and handedness was collected with the aid of this questionnaire.

Vividness of visual imagery questionnaire 2 (VVIQ2). This questionnaire (Marks, 1995) was used to assess participants’ proficiency with visual imagery (including face imagery). It contains

10

8 question sets, with each section consisting of 4 items that ask participants to imagine a given image or scene, and to rate the vividness of what they have imagined.

Familiarity rating of famous faces. This questionnaire measured each participant’s familiarity with 30 faces of famous individuals. They were asked to rate how familiar they were with each face on a 7-point scale (1: not familiar, 7: extremely familiar) and to fill in the name and occupation. The three famous individuals with the highest familiarity ratings were selected as targets in the memory-based similarity rating task for famous faces.

Cambridge face memory test (CFMT). The CFMT (Duchaine & Nakayama, 2006) is a standardized test of face memory. This test was used in this study in order to ensure that the participants possessed normal face recognition and memory abilities.

2.1.4 Task Design

Training task 1 (Familiarization task, Figure 1a). Three unfamiliar face images used as targets for the memory-based reconstruction were labeled as Face1, Face2 and Face3 separately. All three images were presented simultaneously with their corresponding numbers displayed below them. The order of the three images was randomized. Participants were instructed to memorize these three faces in as much as detail as possible. Participants were given as much time as needed to perform this task.

Training task 2 (Old/New recognition task, Figure 1b). This task was used to evaluate how well participants memorized the three faces in Training task 1 (Face1, Face2, and Face3) and to enhance their memory for these faces. Each trial started with a fixation cross at the center of the screen for 500 ms, followed by either a target face (out of the three faces learned in Training task 1) or by a face image out of the remaining 57 unfamiliar faces for 400 ms. Each face stimulus subtended a visual angle of 2.6° × 4° from 90 cm. Participants were asked to determine whether the presented face was old or new by pressing the right arrow key or left arrow key. Each trial was self-paced and participants were encouraged to make their responses as accurately and quickly as possible. Feedback was provided after each response (green checkmark for correct response and red cross for incorrect response). There were a total of 114 trials in this task: each of the three learned faces was presented 19 times and each of the 57 unfamiliar faces was presented once. The trial order was pseudorandomized under the constraint that the same face

11 image was not displayed successively and that old faces was not presented more than twice in a row.

Training task 3 (Visual noise task, Figure 1c). This task was also used to assess participants’ memory of the three learned faces and to enhance their memory of them. Each trial started with a fixation cross at the center of the screen for 500 ms, followed by one of the three faces that were learned in Training task 1 for 400 ms. Each face stimulus subtended a visual angle of 2.6° × 4° from 90 cm. Participants were instructed to determine the identity of the presented face (Face1, Face2 and Face3) by pressing the number keys 1 to 3. The presented face images were degraded by the addition of white noise that increased if the response to the previous trial was correct and decreased if it was incorrect. Adaptive noise levels (starting from 50% and varying between 0% and 100%) were determined with the use of the QUEST toolbox so as to maintain a target accuracy of 75% for behavioral performance. Feedback was provided after participants’ response (green checkmark for correct response and red cross for incorrect response). Participants were asked to be as accurate and quick as possible although they could take as much time as needed to make their responses. Each face learned in Training task 1 was presented 40 times and trial order was randomized.

Perception-based similarity rating task. Participants were presented with pairs of face images and were asked to rate the similarity between the two faces on a 7-point scale, for which 1 meant the two faces were very dissimilar in appearance and 7 meant that they were very similar. They viewed all possible pairs of 57 unfamiliar faces (1596 pairs) across 14 blocks – each face was presented 3 to 5 times in one block. Each trial started with a fixation cross at the center of the screen for 500 ms and was followed by a pair of face images presented on the left and right side of the center for 2000 ms. The stimuli subtended a visual angle of 2.6° × 4° and were displaced 2.4° from the center of the screen on either side. Participants made their judgment based on the 7-point scale by pressing a number key between 1 to 7 and were encouraged to use the whole range of the scale to respond. Trial order was randomized and the locations of the face images were counterbalanced. There were 12 practice trials at the beginning of the task to ensure participants fully understood the task.

Memory-based similarity rating task for learned faces. Participants were instructed to hold one of the three learned faces in their memory and rate its similarity to each of the 57 unfamiliar

12 faces. The similarity ratings of all possible pairs (171 trials) were collected from nine blocks (19 trials per block). Each block started with the presentation of the three learned faces (Face1, Face2 and Face3) and disappeared after participants held one of them in memory and pressed the space bar. In each trial, a fixation cross was displayed for 600 ms and followed by one of the 57 unfamiliar faces presented at the center for 400 ms, subtended a visual angle of 2.6° × 4°. Then participants rated the similarity between the presented face and the face they were required to hold in memory at the beginning of that block, by pressing a number key between 1 to 7. A gray- scale visual noise mask was presented during the inter-trial interval for 100 ms. They were encouraged to use the whole range of the scale to respond. At the end of each block, a scene image was presented in order to minimize the potential effect of visual interference from previous trials. The trial order within each block and the block order were randomized. There were 19 practice trials at the beginning of the task to ensure participants fully understand the task.

Memory-based similarity rating task for famous faces. At the beginning of each block, one of three famous names was presented (as selected via the familiarity ratings during the screening phase). Participants were instructed to imagine the appearance of that famous person in as much detail as possible, hold this mental image in memory and then rate its similarity to each of the 57 unfamiliar faces. All other procedures of this task were the same as the memory-based similarity rating task for learned faces.

2.1.5 Experimental Procedures

All participants first completed multiple screening questionnaires and tests, including the demographic questionnaire VVIQ2 (Marks, 1995), familiarity rating of famous faces, and the CFMT (Duchaine & Nakayama, 2006). Then they were trained to familiarize and remember the three target faces of memory-based reconstructions (learned faces) through the three training tasks. Participants who met all criteria below were qualified to complete the whole experiment: (1) VVIQ2 score (range 1-5) was higher than 3; (2) At least 3 faces were rated above 5 and named correctly in the familiarity rating of famous faces questionnaire; (3) CFMT accuracy was higher than 80%; (4) Accuracy in the training task 2 (old/new recognition task) was greater than 95% (5) Noise level at the end of the training task 3 (visual noise task) was higher than 75%.

13

Eligible participants proceeded to perform the perception-based similarity rating task for unfamiliar faces, memory-based similarity rating tasks for the learned and the famous faces (Figure 2a). Notably, participants did not see any image of the three targets of famous faces throughout the experiment except for in the familiarity rating of famous face questionnaire tested on the first day of the experiment. They did not view any of these celebrities via other means (e.g. media), as confirmed at the end of the experiment. In contrast, the three learned faces were presented multiple times in the experiment and at the beginning of each memory-based similarity rating block in order to refresh participants’ memory. The whole experiment lasted 4 hours and was completed across separate sessions in 4 days.

Data collection and analysis relied on Matlab with the aid of the Psychtoolbox 3.0.12 (Brainard, 1997; Pelli, 1997).

2.1.6 Analyses

All the analyses in Experiment 1 were performed separately for each participant.

Perceptual face space and classification image. First, the similarity ratings among the 57 unfamiliar faces obtained from the perception-based similarity rating task were organized into a confusability matrix, where each row and column represented a single face and the cells represented the similarities between two faces (Figure 2b). Second, a multidimensional face space was then computed by applying metric multidimensional scaling (MDS) to the confusability matrix (Figure 2c). Third, for each dimension of the face space, the coefficients were normalized to z scores. Theses z-transformed coefficients were then divided into positive and negative values and two average face templates were constructed by combining all faces weighted by the coefficients on the positive and negative side separately. The classification image (CI) of each dimension was created through calculating the differences between the corresponding positive and negative averaged face templates (raw CIs in Figure 2d). Lastly, the CIs were assessed pixel by pixel via permutation test in order to analyze which facial features accounted for the organization of facial identities in face space. For each dimension of face space, all coefficients of the 57 unfamiliar faces were randomly permuted 10,000 times and a CI was generated each time. The value of each pixel in the original CI was then compared with the pixel values in the distribution of the permutation using a two-tailed t-test with correction for multiple comparisons (FDR). The significant facial features were determined based on the

14 significant pixels (q < .10). This step was performed for all three color channels in the CIEL*a*b* color space (analyzed CIs in Figure 2d).

Perception-based face image reconstruction (unfamiliar faces) – Procedure. First, the to-be- reconstructed unfamiliar face image was left out, and a new multi-dimensional face space was created based on the confusability matrix that was computed according to the similarity scores among the remaining 56 faces. Second, the non-leave-one-out face space generated with all 57 unfamiliar faces was aligned to the leave-one-out face space using Procrustes analysis in order to obtain the location of the to-be-reconstructed face in the leave-one-out face space. Third, the CIs and significant features were obtained using the same procedures mentioned in paragraph 1.6.1. Lastly, the perception-based reconstruction was generated by summing the average face and the weighted significant features proportional to the coordinates in the face space (Figure 2e).

Memory-based face image reconstruction (learned and famous faces) – Procedure. First, the similarity scores between the to-be-reconstructed learned or famous face and every unfamiliar faces were organized into a confusability matrix and a new face space was then computed by applying MDS to the confusability matrix. Second, the location of the to-be-reconstructed face in the new face space was obtained by using Procrustes analysis to conform the new face space to the perceptual face space. Third, the same procedures mentioned in paragraph 1.6.1 were performed to compute the CIs and significant features. Lastly, the memory-based reconstruction was generated by summing the average face and the weighted significant features proportional to the coordinates in the face space (Figure 2e).

Perception and memory-based reconstructions – Objective evaluation. The objective accuracy of the reconstruction was measured by the similarity between the face images and their corresponding reconstructed images with a pixelwise L2 metric. For the reconstruction of the 57 unfamiliar faces, each face identity was compared with its reconstructed image and every other reconstructed face image. The reconstruction was counted as a correct one when the similarity of the reconstruction to its original face image was higher than that to other original faces. The reconstruction accuracies were averaged across all faces and were tested against chance level (50% accuracy) using a one-sample t-test (p < .05). The pairwise correlations between the participants’ objective accuracies were further tested (p < .05) in order to verify the consistency of the reconstructions across participants. For the reconstructed images of the three learned faces,

15 accuracies were assessed by comparing each reconstructed learned faces to the other two learned faces. Note that objective accuracy was not performed for famous faces since there were no physical corresponding face stimuli used in the experiment.

Regression models. A regression analysis was conducted to investigate whether the distance of a face image to the origin in the face space and the neighbourhood density of a face image in the face space are related to reconstruction accuracy. The distance to the origin was defined as the Euclidean distance between the unfamiliar face image and the origin in the face space. The neighbourhood density was defined as the sum of the inverse of the Euclidean distance. In addition to the face space mentioned above, I also examined the same two measurements in the Ideal Observer face space. The Ideal Observer face space was computed based on the confusability matrix that was computed according to the differences of the pixel values among the unfamiliar face images instead of the similarity ratings among them. Thus, there were four predictors included in the regression model.

2.2 Experiment 2

The purpose of Experiment 2 was to examine whether the reconstructions of face images from Experiment 1 could be identified by both the same participants who provided the data, and an independent group of naïve subjects.

2.2.1 Participants

In addition to the three participants who participated in Experiment 1(NC, CB and SA), an independent group of 67 participants also participated in this experiment. Twenty-seven of the them (15 females; age 18-32 years) who passed the screening criteria were included in the data analysis. The criteria included: (1) be familiar with the famous faces (rating above 4) in one of the three triplets in Experiment 1, and (2) CFMT score was greater than 70%. All participants possessed normal or corrected-to-normal vision, with no history of neurological or psychiatric disorders.

2.2.2 Stimuli

The same 57 unfamiliar faces, three learned faces, and thirty famous faces from Experiment 1 were used in this experiment. In addition, the reconstructed images of the 57 unfamiliar faces,

16 the learned faces and the famous faces from the three participants in Experiment 1 were also included.

2.2.3 Screening Questionnaires and Tests

The same familiarity rating of famous faces questionnaire and CFMT were used in Experiment 2.

2.2.4 Two-alternative forced choice task (face/face-name matching task, Figure 2f)

For the evaluation of the reconstructed unfamiliar face images, each trial consisted of three face images displayed for 2000 ms on the screen, with the reconstructed face image at the top and two original face images (target face plus foil randomly selected from the remaining 56 unfamiliar faces) at the bottom. The participants were asked to select one of the two bottom faces that was most similar to the top face. Each reconstruction was presented 8 times and there were a total of 4 blocks (of 114 trials each) in the task.

Regarding the evaluation of memory-based reconstructions for the learned faces, the task was similar to that for the unfamiliar faces, with the exception that faces were presented for 3000 ms and there were filler trials in which two face images at the bottom were both foils. As for the assessment of famous face reconstructions, instead of showing two original face images at the bottom, the names of two famous individuals were presented. Participants were instructed to select the name that matched the top face. Each memory-based reconstructed image was presented 36 times, and each of these two tasks for memory-based reconstructions consisted of 2 blocks of 54 trials.

2.2.5 Experimental Procedures

The three participants in Experiment 1 (NC, CB and SA) completed the 2AFC tests one day after Experiment 1 to evaluate their own reconstructions. In contrast, the naïve participants first completed the familiarity rating of famous faces and the CFMT, and then performed the 2AFC tests for the unfamiliar, learned and famous faces. Notably, the naïve participants only assessed the reconstructions obtained from one of the three participant in Experiment 1. There were 7, 12 and 8 naïve subjects assessing the reconstructions form NC, CB and SA, respectively.

17

Data collection and analysis relied on Matlab with the aid of the Psychtoolbox 3.0.12 (Brainard, 1997; Pelli, 1997).

2.2.6 Analyses

Perception and memory-based reconstructions – Subjective evaluation. The subjective accuracies of the reconstructed unfamiliar, learned and famous faces were assessed by the three participants in Experiment 1 and the naïve participants recruited in Experiment 2, using the 2AFC tests. The subjective accuracy of each reconstructed image was measured by the proportion of trials that the participants correctly matched the reconstruction to its corresponding original image or the famous individual. For the data of the three participants from Experiment 1, a one-sample t-test (p < .05) was applied on the unfamiliar faces at individual. The t-test was not to performed for the learned and famous faces due to the small sample size (the accuracy of learned or famous reconstructions was averaged across three faces only).

The naïve participants’ data were categorized into three different groups according to which Experiment 1 participant’s reconstructions they assessed. The mean accuracies were first analyzed by a two-way analysis of variance (ANOVA; 3 type of faces: unfamiliar, learned and famous x 3 sets of reconstructions from NC, CB and SA). The mean accuracies for each group and reconstruction type (i.e. unfamiliar, learned, and famous) were then tested against chance level using a one-sample t-test (p < .05). Similar to Experiment 1, the pairwise correlations between the accuracies of unfamiliar faces were further tested (p < .05).

18

Chapter 3 Results Results 3.1 Face Space and Classification Images

Multidimensional face spaces were generated for each individual participant (results from NC are shown in Figure 2c), with these approximating the organization of facial identities in face space. Each point in Fig. 1c represents an individual face and the distance between different points reflect the similarities between them. For illustration purpose, only the first two dimensions are presented and these dimensions explained about 20% of the variance of the distribution of face identities.

In order to examine the features representing the facial identities in the face space, the CIs were assessed pixel by pixel via permutation test (FDR correction across pixels, q < 0.1). The analyzed CIs (Figure 2d) revealed that some facial features such as eyebrows and nose shape are crucial in representing the face information in face space. In addition, these significant features were found not only in the luminance channel but also in the colour channels, suggesting that colour information also contributes to the organization of face space.

3.2 Perception-based Reconstructions

Reconstructions of the 57 perceived unfamiliar faces were generated by summing the average face and the weighted significant features proportional to the coordinates of the reconstruction target in the face space. Selected reconstructed facial images from a single subject (NC) were shown in Figure 3a. The reconstruction accuracies were analyzed objectively (with regards to the pixelwise L2 metric), and subjectively (the three participants in Experiment 1 judged their own reconstructions), and found that the comparisons against chance level at individual basis were significant (two-tailed t-test; p < .001; Figure 4a, and 4b for objective and subjective evaluation, respectively). This is consistent with previous studies (Nestor et al., 2016) that visually perceived faces can be successfully reconstructed based on the information of relative location in the multidimensional face space.

19

To further confirm the consistency of the reconstructions across participants, we correlated the objective accuracies of the 57 unfamiliar faces among the three participant and found significant pairwise correlations (Pearson correlation: rNC-CB = 0.66, rNC-SA = 0.65, rCB-SA = 0.6; all p < .001).

3.3 Memory-based Reconstructions

I reconstructed the face images from memory and assessed the accuracy of each reconstruction for each individual participant. The results showed that the accuracies of the objective and subjective comparisons of the reconstructed learned face images were all above chance for all three faces (see Figure 3b for exemplars of reconstructions and Figure 4a and 4b for reconstruction accuracies). Regarding the reconstructions of famous faces, we found that the subjective accuracy of each famous reconstruction measured by the face-name matching task was also above chance (see Figure 3c for reconstruction exemplars and Figure 4b for reconstruction accuracies). It is important to note that participants did not see any physical images of these famous faces or encounter these individuals via other means (e.g. media, as confirmed at the end of the study) during the experiment, except for the images that were shown in the screening familiarity questionnaire. Overall, these results indicate that we were able to successfully reconstruct not only visually perceived faces but also faces held in long-term memory, including those learned throughout the experiment as well as famous faces. Critically, these reconstructions were performed at the individual level, separately for each participant.

3.4 Subjective Accuracy from Naïve Participants

The reconstructions from the three participants in Experiment 1 were assessed by an independent group of subjects. This allowed me to (1) verify whether these reconstructions could be successfully recognized by subjects who were naïve to these images, and (2) to perform group level analyses for the reconstructed learned and famous faces. A two-way ANOVA (3 type of faces: unfamiliar, learned and famous x 3 sets of reconstructions from NC, CB and SA) showed 2 that the main effect of type of faces was significant (F(2,48) = 73.30, p < .001, η = .753), suggesting that there were differences between the accuracies of different type of faces. Post-hoc comparisons revealed that the accuracy of learned face was significantly greater than that of unfamiliar and famous faces (p < .001). This indicates that the learned faces were reconstructed better than other types of face, which may be possibly because the memory of the learned faces was refreshed throughout the experiment compared to the famous counterpart, and some of the

20 unfamiliar faces were poorly reconstructed. Additionally, the interaction effect between face 2 types and reconstruction sets was also significant (F(4,48) = 3.16, p < .05, η = .208). The accuracies of unfamiliar and learned faces differed among the three reconstruction sets 2 2 (unfamiliar face: F(2,24) = 5.79, p < .01, η = .328; learned face: F(2,24) = 4.53, p < .05, η = .276), whereas that of famous face did not. The main effect of the reconstruction set was not significant. Importantly, one-sample t-tests for the reconstructions of unfamiliar, learned and famous faces from each of the three participants found that all reconstructions were significantly above chance (all p < .05). Additionally, the pairwise correlation of the subjective accuracies of the unfamiliar faces between different reconstruction sets were significant (Pearson correlation: rNC-CB = 0.54, rNC-SA = 0.58, rCB-SA = 057; all p < .001). These significant independent subjective accuracies provide supportive evidence that the perception and memory-based reconstructions were successful.

3.5 Regression Models

Multiple regression analyses were performed on each of the three participant’s data from Experiment 1 to examine whether the distance of a face image to the origin in the face space and the neighbourhood density could predict reconstruction accuracy. It was found that the 2 regression models for all three participants were significant [NC: F(4,52) = 6.73, p < .05, R = 2 2 0.34; CB: F(4,52) = 16.43, p < .001, R = 0.56; SA: F(4,52) = 8.14, p < .001, R = 0.39]. This suggests that the four predictors, including the distance of the face image to the origin and the neighbourhood density of the face image in the psychological and Ideal Observer face spaces, explained a significant amount of variance in the reconstruction accuracy of the unfamiliar faces. In addition, investigating the contribution of each individual predictor showed significant partial correlations between the distance to the origin in the Ideal Observer face space and the reconstruction accuracy in all three participants [NC: partial correlation: -0.31, p < .05; CB: partial correlation: -0.43, p < .01; SA: partial correlation: -0.42, p < .01]. This further indicates that the significance of the regression models was driven mostly by the distance to the origin in the Ideal Observer face space.

21

22

Chapter 4 Discussions and Conclusions

The current work aimed to reconstruct face images from memory based on behavioural data. To my knowledge, it is the first study demonstrating that the reconstruction of faces from memory, in particular, long-term memory can be successful at an individual level. Several aspects of this study are worth discussing further, as follows.

This research first revealed that the visually perceived face images could be successfully reconstructed from the information in perceptual face space, the latter having been generated with the aid of a simple behavioural task that required participants to rate the similarities between face stimuli. Analyses of the face space and the CIs found that some visual features such as eyebrows were significantly represented across different colour channels in the perceptual face space. These findings are consistent with a previous study that reconstructed perceived face images from behavioural and neural data (Nestor et al., 2016). The same approach was used to obtain a combined perception-memory based face space by collecting the similarities between faces recalled from memory and multiple faces in the perceptual face space. These results showed that the reconstructions of faces retrieved from memory were also successful. The accuracies of the reconstructed face images were assessed by the same participants provided the data from which the reconstructions were derived, as well as by an independent group of naïve subjects. In both cases, accuracies were above chance, suggesting that a person can not only recognize his/her own reconstructed images but other naïve people can do so as well. Notably, the success of the memory-based reconstructions based on features represented in perceptual face space provides evidence of the facial features that are important for face memory encoding and retrieval, and suggests that face memory and face perception share common representations.

Notably, a recent study has also achieved successful reconstructions from memory (Lee & Kuhl, 2016). However, the current research is different from this work in several ways. First, the learned face images that were reconstructed in my study were recalled from long-term memory (these faces were learned over multiple days), whereas the memory reconstructions from Lee and Kuhl (2016) were generated from a retro-cue working memory paradigm. Thus, the present findings uniquely indicate that faces held in long-term memory can provide sufficient pictorial detail to support image reconstruction. Second, I only used Caucasian male faces without hair or

23 any accessories as stimuli while Lee and Kuhl (2016) used faces of different genders, ethnicities, and emotional expressions, including hair or accessories (i.e. low-level visual properties were not well controlled). The superior control of the appearance of the face images in my experiment allows one to claim that the reconstructions were truly based on facial features rather than low- scale irrelevant visual information. Third, in addition to the faces learned throughout the experiment, the current study also reconstructed famous faces recalled from memory, and found that the reconstruction accuracies were significantly above chance. It is important to note that participants did not see any image of the target famous face throughout the experiment (other than during screening), nor did they view any of them via other means (e.g. media) on the same day prior to the experimental tasks. Thus, the success of famous face reconstructions suggests that my results can address higher-level (e.g. semantic) representations of facial identity.

It is important to note that the reconstructions in the current work were all generated at the individual level, whereas previous reconstruction studies performed their analyses at the group level (Cowen et al., 2014; Nestor et al., 2016). Individual-based reconstruction is crucial given that each person’s memory of faces is shaped by their own subjective experience. This is specifically true of faces of celebrities since these individuals are usually associated with unique semantic knowledge (e.g. personality, trait). Hence, the above-chance reconstructions of faces in memory here provide more precise insight into the representations that underlie face memory.

More importantly, while the reconstruction approach was initially used for neural data (Thirion et al., 2006), it was performed on behavioural data in my study. It has been reported that the behaviour-based reconstruction is more robust than its neural-based counterpart (Nestor et al., 2016) and could be served to validate the latter. The current work not only replicates the behaviour-based reconstruction on perceived faces but also establishes its utilities in studying the representation of face memory. Additionally, it also points out the value of the reconstruction method in relating behavioural and neural processing of memory. Thus, while this study focused on behavioural estimates of facial representations, the approach here can be applied to investigate the neural representation of face memory and its relationship to face perception in future studies. For example, a face space could be derived from a neural confusability matrix, which is computed based on the fMRI activity while participants are performing the perception and memory-based similarity rating task. Face images will be reconstructed based on the significant features underlying the neural face space. I predict that the reconstructions of face

24 images held in memory could be successfully generated from the activity in the anterior temporal lobe (ATL), given that it is crucial for the processing of the semantic information related to a given identity (Leveroni et al., 2000; Nakamura et al., 2000; Ross & Olson, 2012; Sugiura et al., 2001;Tsukiura et al., 2010), as well as from other face-sensitive regions (e.g. FFA).

In addition to their theoretical relevance, the methods used here also possess significant practical value in that they can potentially provide the basis for an automatic ‘sketch artists’ that can be used in applied fields such as forensic science. For example, it may be more effective and accurate for a target individual face to be depicted and visualized using a behavioural similarity judgment paradigm rather than via current strategies such as verbal description.

Interestingly, the multiple regression analysis found that the distance of a face image to the origin in the Ideal Observer face space was significantly associated with the reconstruction accuracy. This suggests a possibility that the current reconstruction method can be improved by using a smaller set of stimulus set, including faces that are most informative in the face space, in order to develop a simple and optimized version of facial image reconstruction method. For instance, the time for collecting the similarity ratings among face images could be largely reduced with less stimuli, compared to the number of images used in the current study which took about 3 hours to complete all similarity rating tasks. This will make this reconstruction approach more efficient.

One caveat in this study is that the number of the naïve participants in Experiment 2 is not large (7, 12, and 8 subjects for reconstruction sets of NC, CB, and SA, respectively), which might undermine the reliability of the findings. However, it is important to note that the reconstruction accuracy of each type from each naïve participants was numerically above 50% (with only two exceptions that the subjective accuracies of the reconstructed famous face images in two subject were 49%). I believe that the statistical significance of the current results will remain the same with a greater sample size. Furthermore, my current work focused on individual-based reconstruction, and sought three participants with strict screening criteria. While the approach here is successful for each individual, more work will be needed in order to apply this method to the general population. For example, one question is whether the present reconstruction approach can be used successfully with people who are poor face recognizers or have poor ability in forming mental images.

25

In conclusion, this study successfully reconstructed face images that were visually perceived and maintained in memory using individual behavioural data. Theoretically, it sheds light on the contents of individual memory for faces and provides strong confirmation for the close relationship between face perception and face memory. Practically, it offers the foundation for concrete and translational applications such as computer-based ‘sketch artists’.

26

Figure 1. Three training tasks: a) training task 1 (familiarization task): three learned faces were shown simultaneously and participants were instructed to remember them in as much detail as possible; b) training task 2 (old/new recognition task): participants needed to determine whether the presented face image was an old or new face; c) training task 3 (visual noise task): participants had to determine the identity of the presented face, which was degraded by the addition of white noise.

27

Figure 2. Reconstruction procedure and result evaluation: a) participants rate the face similarity of two stimuli or of a single stimulus against a facial identity retrieved from memory; b) pairwise

28 face similarities are converted into a confusability matrix of n distinct facial identities, where face n is the target face for reconstruction purposes; c) face space is estimated from the similarity of n-1 different faces and the coordinates of the target face are approximated within that space (only 2 dimensions are displayed for convenience; PVE – percent variance explained); d) visual features corresponding to each dimension are derived through image classification from n-1 faces and analyzed, separately for each color channel in CIEL*a*b*, with a pixelwise permutation test (FDR-corrected across pixels; q<0.10); e) visual features are linearly combined to estimate the visual appearance of the target face (CIk – classification images corresponding to dimension k); f) face reconstructions are evaluated in a two-alternative forced choice task with face pairs, for novel faces, or with name pairs, for famous faces. As an illustration, c) and d) show intermediary reconstruction results for participant NC.

29

Figure 3. Examples of face reconstructions for participant NC. Results are shown separately for a) perceptual reconstructions, b) memory-based reconstructions of learned faces and c) memory- based reconstructions of famous faces that participants were familiar with from their individual experience. Reconstructions are paired with their corresponding stimuli for a) –b) or, for convenience, with examples of target famous faces from the public domain (pd) for c). Reconstruction accuracy was estimated objectively through pixelwise image similarity (top left) for a-b); additionally, accuracy was assessed experimentally by the same participant (top right) or by an independent group of naïve participants (bottom right) for all types of reconstruction.

30

Figure 4. Average reconstruction accuracy for three participants (NC, CB and SA). Accuracy was estimated via a) objective pixelwise similarity; b) subjective accuracy from the same participants, tested on their own reconstructions, and c) subjective accuracy from an independent group of participants (error bars show 1SE; *** - p<0.001; ** - p<0.01; * - p<0.05).

31

References

Anzellotti, S., Fairhall, S. L., & Caramazza, A. (2013). Decoding representations of face identity that are tolerant to rotation. Cerebral Cortex, 24(8), 1988–1995. http://doi.org/10.1093/cercor/bht046

Bainbridge, W. A., Isola, P., & Oliva, A. (2013). The intrinsic memorability of face photographs, 142(4), 1323–1334. http://doi.org/10.1037/a0033872

Barense, M. D., Henson, R. N. A., Lee, A. C. H., & Graham, K. S. (2010). Medial temporal lobe activity during complex discrimination of faces, objects, and scenes: Effects of viewpoint. Hippocampus, 401, NA-NA. http://doi.org/10.1002/hipo.20641

Bartlett, J. C., Hurry, S., & Thorley, W. (1984). Typicality and familiarity of faces. Memory & Cognition, 12(3), 219–228. http://doi.org/10.3758/BF03197669

Barton, J. J., & Cherkasova, M. (2003). Face imagery and its relation to perception and covert recognition in prosopagnosia. Neurology, 61(2), 220–225. http://doi.org/10.1212/01.WNL.0000071229.11658.F8

Barton, J. J. S. (2008). Structure and function in acquired prosopagnosia: lessons from a series of 10 patients with brain damage. Journal of Neuropsychology, 2, 197–225. http://doi.org/10.1348/174866407X214172

Bernstein, M. J., Young, S. G., & Hugenberg, K. (2007). The cross-category effect. Psychological Science, 18(8), 706–712.

Brainard, D. H. (1997) The Psychophysics Toolbox, Spatial Vision 10:433-436

Brouwer, G. J., & Heeger, D. J. (2009). Decoding and reconstructing color from responses in human visual cortex. Journal of Neuroscience, 29(44), 13992–14003. http://doi.org/10.1523/JNEUROSCI.3577-09.2009

Busigny, T., Van Belle, G., Jemel, B., Hosein, A., Joubert, S., & Rossion, B. (2014). Face- specific impairment in holistic perception following focal lesion of the right anterior temporal lobe. Neuropsychologia, 56(1), 312–333.

32

http://doi.org/10.1016/j.neuropsychologia.2014.01.018

Byatt, G., & Rhodes, G. (2004). Identification of own-race and other-race faces: implications for the representation of race in face space. Psychonomic Bulletin & Review, 11(4), 735–741. http://doi.org/10.3758/BF03196628

Cheung, O. S., & Gauthier, I. (2010). Selective interference on the holistic processing of faces in working memory. Journal of Experimental Psychology: Human Perception and Performance, 36(2), 448–461. http://doi.org/10.1037/a0016471

Chiroro, P., & Valentine, T. (1995). An investigation of the contact hypothesis of the own-race bias in face recognition. The Quarterly Journal of Experimental Psychology Section A, 48(4), 879–894. http://doi.org/10.1080/14640749508401421

Collins, J. a., & Olson, I. R. (2014). Beyond the FFA: The role of the ventral anterior temporal lobes in face processing. Neuropsychologia, 61(1), 65–79. http://doi.org/10.1016/j.neuropsychologia.2014.06.005

Cowen, A. S., Chun, M. M., & Kuhl, B. a. (2014). Neural portraits of perception: Reconstructing face images from evoked brain activity. NeuroImage, 94, 12–22. http://doi.org/10.1016/j.neuroimage.2014.03.018

D’Argembeau, A., Van der Linden, M., Etienne, A. M., & Comblain, C. (2003). Identity and expression memory for happy and angry faces in social anxiety. Acta Psychologica, 114(1), 1–15. http://doi.org/10.1016/S0001-6918(03)00047-7

Desimone, R., Albright, T. D., Gross, C. G., & Bruce, C. (1984). Stimulus-selective neurons in the macaque. Journal of Neuroscience, 4(8), 2051–2062.

Duchaine, B. C., & Nakayama, K. (2006). The Cambridge Face Memory Test: Results for neurologically intact individuals and an investigation of its validity using inverted face stimuli and prosopagnosic participants. Neuropsychologia, 44(4), 576–585. http://doi.org/10.1016/j.neuropsychologia.2005.07.001

Gainotti, G., & Marra, C. (2011). Differential contribution of right and left temporo-occipital and anterior temporal lesions to face recognition disorders. Frontiers in Human Neuroscience,

33

5(June), 55. http://doi.org/10.3389/fnhum.2011.00055

Gauthier, I., Tarr, M. J., Moylan, J., Skudlarski, P., Gore, J. C., & Anderson, a W. (2000). The fusiform “face area” is part of a network that processes faces at the individual level. Journal of Cognitive Neuroscience, 12(3), 495–504. http://doi.org/10.1162/089892900562165

Gobbini, M. I., & Haxby, J. V. (2007). Neural systems for recognition of familiar faces. Neuropsychologia, 45(1), 32–41. http://doi.org/10.1016/j.neuropsychologia.2006.04.015

Goesaert, E., & Op de Beeck, H. P. (2013). Representations of facial identity information in the ventral visual stream investigated with multivoxel pattern analyses. Journal of Neuroscience, 33(19), 8549–8558. http://doi.org/10.1523/JNEUROSCI.1829-12.2013

Grill-Spector, K., Knouf, N., & Kanwisher, N. (2004). The fusiform face area subserves face perception, not generic within-category identification. Nature Neuroscience, 7(5), 555–562. http://doi.org/10.1038/nn1224

Harry, B., Williams, M. A., Davis, C., & Kim, J. (2013). Emotional expressions evoke a differential response in the fusiform face area. Frontiers in Human Neuroscience, 7(October), 1–6. http://doi.org/10.3389/fnhum.2013.00692

Haxby, J. V. J., Hoffman, E. E. A., & Gobbini, M. I. M. M. I. (2000). The distributed human neural system for face perception. Trends in Cognitive Sciences, 4(6), 223–233. http://doi.org/10.1016/S1364-6613(00)01482-0

Hodges, J. R., Patterson, K., Oxbury, S., & Funnell, E. (1992). Semantic Dementia. Brain, 115, 1783–1806. http://doi.org/10.1093/brain/115.6.1783

Hoffman, E. a, & Haxby, J. V. (2000). Distinct representations of eye gaze and identity in the distributed human neural system for face perception. Nature Neuroscience, 3(1), 80–84. http://doi.org/10.1038/71152

Johnson, M. H., Dziurawiec, S., Ellis, H., & Morton, J. (1991). Newborns’ preferential tracking of face-like stimuli and its subsequent decline. Cognition, 40(1–2), 1–19. http://doi.org/10.1016/0010-0277(91)90045-6

34

Johnston, R. a., Milne, A. B., Williams, C., & Hosie, J. (1997). Do distinctive faces come from outer space? An investigation of the status of a multidimensional face-space. Visual Cognition, 4(1), 59–67. http://doi.org/10.1080/713756748

Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: a module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience : The Official Journal of the Society for Neuroscience, 17(11), 4302–4311. http://doi.org/10.1098/Rstb.2006.1934

Kriegeskorte, N., Formisano, E., Sorger, B., & Goebel, R. (2007). Individual faces elicit distinct response patterns in human anterior temporal cortex. Proceedings of the National Academy of Sciences.

Lee, A. C. H. (2006). Differentiating the roles of the hippocampus and perirhinal cortex in processes beyond long-term declarative memory: A double dissociation in dementia. Journal of Neuroscience, 26(19), 5198–5203. http://doi.org/10.1523/JNEUROSCI.3157- 05.2006

Lee, A. C. H., Buckley, M. J., Pegman, S. J., Spiers, H., Scahill, V. L., Gaffan, D., … Graham, K. S. (2005). Specialization in the medial temporal lobe for processing of objects and scenes. Hippocampus, 15(6), 782–797. http://doi.org/10.1002/hipo.20101

Lee, A. C. H., Scahill, V. L., & Graham, K. S. (2008). Activating the medial temporal lobe during oddity judgment for faces and scenes. Cerebral Cortex, 18(3), 683–696. http://doi.org/10.1093/cercor/bhm104

Lee, H., & Kuhl, B. A. (2016). Reconstructing perceived and retrieved faces from activity patterns in lateral parietal cortex. The Journal of Neuroscience, 36(22), 6069–6082. http://doi.org/10.1523/JNEUROSCI.4286-15.2016

Leopold, D. a, O’Toole, a J., Vetter, T., & Blanz, V. (2001). Prototype-referenced shape encoding revealed by high-level aftereffects. Nature Neuroscience, 4(1), 89–94. http://doi.org/10.1038/82947

Leveroni, C. L., Seidenberg, M., Mayer, A. R., Mead, L. a, Binder, J. R., & Rao, S. M. (2000).

35

Neural systems underlying the recognition of familiar and newly learned faces. The Journal of Neuroscience, 20(2), 878–886.

Liu, J., Harris, A., & Kanwisher, N. (2010). Perception of face parts and face configurations: an FMRI study. Journal of Cognitive Neuroscience, 22(1), 203–211. http://doi.org/10.1162/jocn.2009.21203

Loffler, G., Yourganov, G., Wilkinson, F., & Wilson, H. R. (2005). fMRI evidence for the neural representation of faces. Nature Neuroscience, 8(10), 1386–1391. http://doi.org/10.1038/nn1538

Marks, D. F. (1995). New directions for mental imagery research. Journal of Mental Imagery, 19(3–4), 153–167.

Mccarthy, G., Puce, A., Gore, J. C., & Truett, A. (1997). Face-specific processing in the human fusiforrn gyms. Journal of Cognitive Neuroscience, 9(5), 605–610. http://doi.org/10.1162/jocn.1997.9.5.605

Meissner, C. A., & Brigham, J. C. (2001). Own-race bias in memory for faces A meta-analytic review, 7(1), 3–35. http://doi.org/10.1037//1076-8971.7.1.3

Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M., Morito, Y., Tanabe, H. C., … Kamitani, Y. (2008). Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, 60(5), 915–929. http://doi.org/10.1016/j.neuron.2008.11.004

Mondloch, C. J., Lewis, T. L., Budreau, D. R., Maurer, D., James, L., Stephens, B. R., … Kleiner-gathercoal, K. A. (2015). Face perception during early infancy. Psychological Science, 10(5), 419–422.

Mummery, C. J., Patterson, K., Price, C. J., Ashburner, J., Frackowiak, R. S. J., & Hodges, J. R. (2000). A voxel-based morphometry study of semantic dementia: Relationship between temporal lobe atrophy and semantic memory. Annals of Neurology, 47(1), 36–45. http://doi.org/10.1002/1531-8249(200001)47:1<36::AID-ANA8>3.0.CO;2-L

Murray, E. a., Bussey, T. J., & Saksida, L. M. (2007). Visual Perception and Memory: A New

36

View of Medial Temporal Lobe Function in Primates and Rodents *. Annual Review of Neuroscience, 30(1), 99–122. http://doi.org/10.1146/annurev.neuro.29.051605.113046

Nakamura, K., Kawashima, R., Sato, N., Nakamura, A., Sugiura, M., Kato, T., … Zilles, K. (2000). Functional delineation of the human occipito-temporal areas related to face and scene processing. A PET study. Brain : A Journal of Neurology, 123 ( Pt 9, 1903–12. http://doi.org/10.1093/brain/123.9.1903

Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., & Gallant, J. L. (2009). Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6), 902–915. http://doi.org/10.1016/j.neuron.2009.09.006

Nestor, A., Plaut, D. C., & Behrmann, M. (2011). Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis. Proceedings of the National Academy of Sciences, 108(24), 9998–10003. http://doi.org/10.1073/pnas.1102433108

Nestor, A., Plaut, D. C., & Behrmann, M. (2016). Feature-based face representations and image reconstruction from behavioral and neural data. Proceedings of the National Academy of Sciences, 113(2), 201514551. http://doi.org/10.1073/pnas.1514551112

Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current, 21(19), 1641–1646. http://doi.org/10.1016/j.cub.2011.08.031

O’Neil, E. B., Barkley, V. a., & Köhler, S. (2013). Representational demands modulate involvement of perirhinal cortex in face processing. Hippocampus, 23(7), 592–605. http://doi.org/10.1002/hipo.22117

O’Neil, E. B., Cate, a. D., & Kohler, S. (2009). Perirhinal cortex contributes to accuracy in recognition memory and perceptual discriminations. Journal of Neuroscience, 29(26), 8329–8334. http://doi.org/10.1523/JNEUROSCI.0374-09.2009

O’Neil, E. B., Hutchison, R. M., McLean, D. A., & Köhler, S. (2014). Resting-state fMRI reveals functional connectivity between face-selective perirhinal cortex and the fusiform face area related to face inversion. NeuroImage, 92, 349–355.

37

http://doi.org/10.1016/j.neuroimage.2014.02.005

Oosterhof, N. N., & Todorov, A. (2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences of the United States of America, 105(32), 11087–92. http://doi.org/10.1073/pnas.0805664105

Pelli, D. G. (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies, Spatial Vision 10:437-442.

Perrett, D. I., Hietanen, J. K., Oram, M. W., Benson, P. J., & Rolls, E. T. (1992). Organization and functions of cells responsive to faces in the temporal cortex. Philosophical Transactions of the Royal Society B: Biological Sciences, 335(1273), 23–30.

Perrett, D. I., Smith, P. A. J., Potter, D. D., Mistlin, A. J., Head, A. S., Milner, A. D., & Jeeves, M. A. (1985). Visual cells in the temporal cortex sensitive to face view and gaze direction. Proceedings of the Royal Society of London B: Biological Sciences, 223(1232), 293–317.

Pitcher, D., Walsh, V., & Duchaine, B. (2011). The role of the occipital face area in the cortical face perception network. Experimental Brain Research, 209(4), 481–493. http://doi.org/10.1007/s00221-011-2579-1

Pitcher, D., Walsh, V., Yovel, G., & Duchaine, B. (2007). TMS evidence for the involvement of the right occipital face area in early face processing. Current Biology, 17(18), 1568–1573. http://doi.org/10.1016/j.cub.2007.07.063

Puce, A., Allison, T., Asgari, M., Gore, J. C., & McCarthy, G. (1996). Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study. The Journal of Neuroscience, 16(16), 5205–5215.

Rajimehr, R., Young, J. C., & Tootell, R. B. H. (2009). An anterior temporal face patch in human cortex , predicted by macaque maps. Proceedings of the National Academy of Sciences, 106(6).

Ross, L. a., & Olson, I. R. (2012). What’s unique about unique entities? An fMRI investigation of the semantics of famous faces and landmarks. Cerebral Cortex, 22(9), 2005–2015. http://doi.org/10.1093/cercor/bhr274

38

Rule, N. O., Slepian, M. L., & Ambady, N. (2012). A memory advantage for untrustworthy faces. Cognition, 125(2), 207–218. http://doi.org/10.1016/j.cognition.2012.06.017

Saksida, L. M., & Bussey, T. J. (2010). The representational–hierarchical view of amnesia: Translation from animal to human. Neuropsychologia, 48(8), 2370–2384. http://doi.org/10.1016/j.neuropsychologia.2010.02.026

Schiltz, C., Dricot, L., Goebel, R., & Rossion, B. (2010). Holistic perception of individual faces in the right middle fusiform gyrus as evidenced by the composite face illusion. Journal of Vision, 10(2), 25.1-16. http://doi.org/10.1167/10.2.25

Shepherd, J. W., Gibling, F., & Ellis, Haydn, D. (1991). The effects of distinctiveness, presentation time and delay on face recognition. European Journal of Cognitive Psychology, 3(1), 137–145. http://doi.org/10.1080/09541449108406223

Snowden, J. S., Goulding, P. J., & Neary, D. (1989). Semantic dementia: A form of circumscribed cerebral atrophy. Behavioural Neurology., 2(3), 167–182.

Snowden, J. S., Thompson, J. C., & Neary, D. (2004). Knowledge of famous faces and names in semantic dementia. Brain, 127(4), 860–872. http://doi.org/10.1093/brain/awh099

Sugiura, M., Kawashima, R., Nakamura, K., Sato, N., Nakamura, a, Kato, T., … Fukuda, H. (2001). Activation reduction in anterior temporal cortices during repeated recognition of faces of personal acquaintances. NeuroImage, 13(5), 877–90. http://doi.org/10.1006/nimg.2001.0747

Thirion, B., Duchesnay, E., Hubbard, E., Dubois, J., Poline, J. B., Lebihan, D., & Dehaene, S. (2006). Inverse retinotopy: Inferring the visual content of images from brain activation patterns. NeuroImage, 33(4), 1104–1116. http://doi.org/10.1016/j.neuroimage.2006.06.062

Tippett, L. J., Miller, L. A., & Farah, M. J. (2000). Prosopamnesia: A selective impairment in face learning. Cognitive Neuropsychology, 17(1–3), 241–255. http://doi.org/10.1080/026432900380599

Tsao, D. Y., Freiwald, W. A., Knutsen, T. A., Mandeville, J. B., & Tootell, R. B. H. (2003). Faces and objects in macaque cerebral cortex. Nature Neuroscience, 6(9), 989–995.

39

http://doi.org/10.1038/nn1111

Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H., & Livingstone, M. S. (2006). A cortical region consisting entirely of face-selective cells. Science, 311(5761), 670–674.

Tsao, D. Y., Moeller, S., & Freiwald, W. A. (2008). Comparing face patch systems in macaques and humans. Proceedings of the National Academy of Sciences of the United States of America, 105, 19514–19519. http://doi.org/10.1073/pnas.0809662105

Tsukiura, T., Mano, Y., Sekiguchi, A., Yomogida, Y., Hoshi, K., Kambara, T., … Kawashima, R. (2010). Dissociable roles of the anterior temporal regions in successful encoding of memory for person identity information. Journal of Cognitive Neuroscience, 22, 2226– 2237. http://doi.org/10.1162/jocn.2009.21349

Valentine, T. (1991). A unified account of the effects of distinctiveness, inversion, and race in face recognition. The Quarterly Journal of Experimental Psychology Section A, 43(2), 161– 204. http://doi.org/10.1080/14640749108400966

Wiese, H., Altmann, C. S., & Schweinberger, S. R. (2014). Effects of attractiveness on face memory separated from distinctiveness: Evidence from event-related brain potentials. Neuropsychologia, 56(1), 26–36. http://doi.org/10.1016/j.neuropsychologia.2013.12.023

Winograd, E. (1981). Elaboration and distinctiveness in memory for faces. Journal of Experimental Psychology. Human Learning and Memory, 7(3), 181–190. http://doi.org/10.1037/0278-7393.7.3.181

Winston, J. S. (2004). fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. Journal of Neurophysiology, 92(3), 1830–1839. http://doi.org/10.1152/jn.00155.2004

Yotsumoto, Y., Kahana, M. J., Wilson, H. R., & Sekuler, R. (2007). Recognition memory for realistic synthetic faces. Memory & Cognition, 35(6), 1233–1244. http://doi.org/10.3758/BF03193597

Zhang, J., Li, X., Song, Y., & Liu, J. (2012). The fusiform face area is engaged in holistic, not parts-based, representation of faces. PloS One, 7(7), e40390.

40 http://doi.org/10.1371/journal.pone.0040390