AN EVALUATION OF HYPERALIGNMENT ON REPRODUCIBILITY AND PREDICTION ACCURACY FOR FMRI DATA

by

Xuemin Zhu

A thesis submitted to Johns Hopkins University in conformity with the requirements for the degree of Master of Science in Engineering

Baltimore, Maryland May, 2021

© 2021 by Xuemin Zhu

All rights reserved Abstract

Functional magnetic resonance imaging (fMRI) is a technique which measures a person’s brain activity using changes in the flow in response to neural activity. Recently, resting state fMRI (rs-fMRI)has become a ubiquitous tool for measuring connectivity and examining the functional architecture of the . Here, we used a publicly available rs-fMRI dataset to investigate the performance of the hyperalignment algorithm, on several fMRI analyses. The research employs the use of the image intra-class correlation coefficient and functional connectome fingerprinting to evaluate the reproducibility of both the unaligned and hyperaligned data, and devel- oped a predictive model to investigate whether hyperalignment improves the prediction of certain behavioral measures. Overall, our results illustrate the utility of the hyperalignment algorithm for studying inter-individual variation in brain activity.

ii Thesis Committee

Primary Readers

Martin Lindquist (Primary Advisor) Professor Department of Biostatistics Johns Hopkins Bloomberg School of Public Health

Brian Caffo Professor Department of Biostatistics Johns Hopkins Bloomberg School of Public Health

James J. Pekar Professor Department of Radiology and Radiological Science Johns Hopkins School of

iii Acknowledgments

Throughout the writing of this thesis I have received a great deal of support and assistance.

First, I am eternally grateful and indebted to my thesis advisor, Prof. Mar- tin Lindquist, whose patience, support and understanding were invaluable in such a difficult time like this. I really appreciate everything that youhelped me both in academics and in life.

I would like to thank Farzad Vasheghani-Farahani for his wonderful col- laboration. Without his selfless help, this paper would not have been inits present form.

I would also like to thank my thesis committee members for their thought- ful and insightful comments, advice and help.

Finally, I would like to thank my dear family for their love and great faith in me for many years.I also want to thank my friends who have given me help and time to listen to my opinions and help me solve my problems during this difficult time.

iv Table of Contents

Abstract ii

Thesis Committee iii

Acknowledgments iv

Table of Contentsv

List of Tables vii

List of Figures viii

1 Introduction1

1.1 Background ...... 1 1.1.1 A brief history of neuroimaging ...... 1 1.1.2 Multivariate pattern analysis of fMRI data ...... 4

1.1.3 Hyperalignment ...... 5 1.2 Problem Statement ...... 6 1.3 Purpose of the Study ...... 7

1.3.1 Aim ...... 7 1.3.2 Objective ...... 7

v 1.3.3 Procedure ...... 7 1.3.4 Significance of the study ...... 8

1.3.5 Structure of the paper ...... 9

2 Literature Review 11

2.1 Neuroimaging ...... 11

2.2 Hyperalignment for fMRI Data ...... 13

3 Methodology 20

3.1 Human Connectome Project Data ...... 20

3.2 Hyperalignment ...... 21 3.3 Measures of Reliability: ICC, I2C2 and fingerprinting . . . . . 23 3.3.1 I2C2 ...... 23

3.3.2 Fingerprinting ...... 25 3.4 Predictive Modeling Approaches ...... 26

4 Results 29

4.1 Research Design ...... 29 4.2 Reliability ...... 30 4.3 Predictive Modeling ...... 33

5 Discussion and Conclusion 40

5.1 Discussion ...... 40 5.2 Limitations and future directions ...... 41

5.3 Conclusion ...... 42

vi List of Tables

4.1 Reliability Results for Unaligned Data of the Whole Brain . . . 30 4.2 I2C2 Results ...... 31

4.3 A table showing the fingerprinting accuracy of the aligned and unaligned data ...... 32 4.4 Evaluation Results ...... 37

4.5 AUC value for predicting gender ...... 38

vii List of Figures

4.1 Scatter plots of predicted values vs actual values for age. The results for the unaligned data are show to the left and the results

for the aligned data to the right...... 34 4.2 Scatter plots of predicted values vs actual values for BMI. The results for the unaligned data are show to the left and the results

for the aligned data to the right...... 35 4.3 Scatter plots of predicted values vs actual values for fluid intel- ligence. The results for the unaligned data are show to the left

and the results for the aligned data to the right...... 36 4.4 Scatter plots of predicted values vs actual values for depression score. The results for the unaligned data are show to the left

and the results for the aligned data to the right...... 37

viii Chapter 1

Introduction

1.1 Background

1.1.1 A brief history of neuroimaging

The first example of brain imaging dates back tothe 19th century in the form of the human circulation balance invented by the Italian physiologist Angelo Mosso. This precursor to modern neuroimaging devices was able to measure the redistribution of blood in the brain during emotional or intellectual activity. In this application, Mosso discovered important variables such as the signal to noise ratio, which are essential properties of modern methods of brain imaging.

According to Savoy, 2001, the beginning of modern neuroimaging dates back to the period between 1895-1973. During this time, Wilhelm Roentgen demonstrated the first-ever radiograph, which provided a gateway to modern forms of diagnosis. However, given the fact that most of the brain is made up of soft tissue, it was virtually impossible to view most of its parts using a standard radiograph. Therefore, in the early 20th century Walter Dandy, an American neurosurgeon, introduced ventriculography. This technique

1 involved taking images of the within the brain after the injection of filtered air into the lateral ventricles. Access to the ventricles was obtained by drilling a hole in the skull of the patients. Despite its success in being able to improve upon the performance of the radiograph, the technique presented a series of risks that jeopardized the health of the patients. These included potential infractions, threatening changes in the brain pressure, and possible hemorrhaging. Nevertheless, ventriculography proved successful in reducing the margin of error when performing neurosurgery.

With the continued discovery of brain-related problems, the need for brain imaging has never been greater. Therefore, the modern medical field has sought to build on these historical approaches by developing imaging tech- niques that reduce the risk of losing life, but at the same time increase accuracy and the amount of information gleaned. One such technique in modern neu- rosurgery and neuroimaging is cerebral . This technique was introduced by Egas Moniz, a neurologist based in Lisbon in 1927, and allows for the visualization of blood vessels in the brain. The technique ensured an enhanced sense of accuracy that its predecessors lacked. However, like ventriculography, in spite of the great strides that the technique took, the level of threat to the patients was equally dire. In this case, the technique could at times lead to cases of delirium in its patients.

For this reason, there was a clear need to develop techniques that both improved performance and risk management rate. One of the greatest im- provements came in the form of computerized techniques in the late 20th

2 century and early 21st centuries. The techniques developed in this era are typ- ically referred to as computerized tomography. The first computerized brain imaging technique is the computerized axial tomography which is commonly known as CAT scanning. This technique revolutionized brain imaging and neurosurgery as features in the brain that could not previously be viewed using older techniques become visible and available for both diagnosis and further research. Unlike previous techniques, the computer-based technique allowed the physicians to administer the imaging process in a painless, effec- tive, and nearly non-invasive way. This meant that the technique posed less risk compared to the earlier techniques. During the same period, radioactive neuroimaging also gained in popularity. Here technicians employed the used of photon emission tomography to conduct brain imaging. However, repeated use of this technique exposed the patients to potential radiation poisoning. As a result, a shift was made towards magnetic resonance imaging (MRI). This technique used the variation in the signals produced by protons in the body when the human head is placed next to a strong magnetic field to produce brain images. According to Hudd et al., 2019, MRI poses a risk to patients with metallic implants on their bodies such as hip implants and pacemakers. Furthermore, the technique has a claustrophobic characteristic that may be overwhelming to claustrophobic patients. However, the technique dramati- cally reduced the risks posed by the predecessors. In addition, it was found that the technique was also able to measure blood flow changes. As such, func- tional magnetic resonance imaging (fMRI) became the prevalent technique used to conduct functional brain imaging studies.

3 1.1.2 Multivariate pattern analysis of fMRI data

To date, the majority of fMRI studies have used the so-called approach. Here, the goal is to identify which brain areas encode a particular psychological condition or other outcome. The statistical procedures used to develop brain maps seek to test whether there is a non-zero effect of a particu- lar psychological manipulation or observed behavior on one or more brain voxels or regions. More recently, the field of neuroimaging has begun to move away from the traditional brain mapping approach towards the development of integrated, multivariate brain models of mental events. Multivariate Pat- tern Analysis (MVPA), brain decoding, and machine learning are terms used to describe overlapping subsets of these models and the algorithms used to develop them. These models make quantitative predictions about stimulus conditions, behaviors, or other outcomes.

The goals associated with the creation of a brain map and a brain model are different. While brain maps describe the local encoding of information, brain models attempt to specify the parts of a neural system and how their joint activity predict and behavior. Brain models can vary significantly in terms of their complexity. Some simply associate activity in a single brain region with a specific outcome. However, increasingly brain models are multi- variate and explain outcomes as patterns of brain activity and/or structure across large numbers of brain features, often distributed across anatomical regions and systems. They also vary widely in how well they explain the basis for their predictions.

4 1.1.3 Hyperalignment

While the use of predictive models on fMRI data has increased in recent years, a problem remains the large inter-individual differences in both brain anatomy and functional localization after anatomical alignment. Recent approaches towards predictive modeling have sought to circumvent this problem by mapping individual brains into a functional population-level reference space rather than an anatomically based brain space. One such approach is the

‘hyperalignment’ procedure (Haxby et al., 2011). Here brain activity patterns corresponding to stimuli and other cognitive events are represented as vectors in a neural representational space spanned by the voxels in a local neighbor- hood. Hyperalignment rotates each participant’s local voxel-wise activity patterns through multivariate voxel space using a Procrustes transformation to align the representational geometry across subjects. This dramatically re- duces inter-subject variability in functional anatomy, thereby increasing the accuracy and specificity of population-level models.

According to Yousefnezhad et al., 2020, hyperalignment has been widely favored in MVPA for determining the cognitive activity of the brain based on the multisubject fMRI datasets. When using hyperalignment, it is important to note that, the information shared in the brain is “encoded in idiosyncratic fine- scale functional topographies” (Haxby et al., 2020). Therefore, hyperalignment captures shared information between the brain by projecting patterns instead of aligning the topographies in a conical space for neural responses. The use of the hyperalignment algorithm is ideal for this project since the methods under hyperalignment tend to utilize unsupervised approaches. As

5 such, the technique can be used to maximize the correlation between the voxel and the same position in a time series when dealing with fMRI datasets.

Therefore, to get a better understand of the use of hyperalignment in fMRI data, our research focuses on comparing the information provided by the fMRI data before and after the hyperalignment algorithm has been added to the analysis.

1.2 Problem Statement

While many researchers have explored the use of fMRI for studying human brain function, a gap still exists when it comes to the assessing the benefits of using the hyperalignment algorithm on the data obtained from fMRI. This begs the question of whether using hyperalignment on the data increases the level of accuracy, maintains the same level of accuracy, or reduces it. In particular, it is of interest to determine how the application of hyperalignment influences the test/re-test reliability of fMRI data, as well as the accuracyof predicting certain behavioral variables. The answer to the first question is not clear as it is possible that by decreasing functional variation across subjects one might actually decrease test re-test reliability as large inter-subject variability compared to intra-subject variability lead to increased reliability. The answer to the second question appears more straightforward as one of the primary goals of hyperalignemnt is to align features across subjects, thus improving the performance of predictive models. However, we seek to replicate and confirm previous research findings.

6 1.3 Purpose of the Study

1.3.1 Aim

This project aims to investigate the benefits of using the hyperalignment al- gorithm on fMRI data. To achieve this goal, an empirical analysis of fMRI data is conducted both before and after performing the hyperalignment algo- rithm. This provides an empirical perspective on whether or not the use of the hyperalignment algorithm on the fMRI data increases the level of reliability and accuracy. We believe this will help fill in gaps in the research literature regarding the performance of the hyperalignmnet algorithm on fMRI data. Furthermore, it will provide a base for the comparative analysis between the findings of this research and that of other researchers. This will beachieved by performing an extensive analysis of previous research findings on the topic. With this perspective, our research will attempt to provide sufficient informa- tion for understanding how employment of the hyperalignment technique impacts fMRI data analysis.

1.3.2 Objective

To investigate the impact of hyperalignment on neuroimaging.

1.3.3 Procedure

In our work, the hyperalignment algorithm is applied to the original data to get aligned data. This is typically done in one of two ways, searchlight hyperalignment or alternatively ROI-based hyperalignment. In this study, we

7 only use searchlight hyperalignment. Once the data has been aligned, we will calculate the image intraclass correlation (I2C2) and fingerprinting accuracy, both measures of reliability, as well as perform predictions on various variables (age, handiness, height, weight, BMI, depression score, gender, and fluid intelligence). This is done both for the original unaligned and the hyperaligned data. For the I2C2 and fingerprint, we will evaluate the performance of these two data sets by comparing their respective values. For the prediction part, we compare the MSE (Mean Squared Error), RMSE (Root Mean Squared Error), and MAE (Mean Absolute Error) for the four continuous variables. As predicting gender is a binary classification problem, here we instead compare the receiver operating characteristic (ROC) curves and the corresponding area under the curve (AUC).

1.3.4 Significance of the study

The proposed research will help shed light on the benefits of using hyperalign- ment in the analysis of fMRI data. The results will help establish whether previous research findings are replicable in a new data set, as well as explore properties of the hyperalignment algorithm that have not previously been investigated. As such, neuroscientists can use the results to help improve the accuracy of their neuroimaging data analysis results.

8 1.3.5 Structure of the paper

The paper is divided into five sections, with each section working in cohesion with the other to arrive at an informed conclusion. The first section intro- duces the research with background information on the topic. The section also points out the essential part such as the aims and the objectives of the project. The second section is the literature review and offers detailed background information on the relationship between fMRI data and the hyperalignment algorithm. The focus of the section is to provide a historical overview from various sources related to hyperalignment algorithms, fMRI, and neuroimag- ing. This section will help develop a perspective on the relationship between fMRI and hyperalignment algorithms. The third section details the process of data collection and introduces the methods used in the data analysis. The fourth section presents the results of these analyses. The fifth and final section provides a summary of the project and introduce a few recommendations essential for future studies on the topic or related topics.

9 References

Savoy, Robert L (2001). “History and future directions of human brain map- ping and functional neuroimaging”. In: Acta psychologica 107.1-3, pp. 9– 42. Hudd, Fred, Anna Shiel, Matthew Harris, Paul Bowdler, Bryony McCann, Demitra Tsivos, Alfie Wearn, Michael Knight, Risto Kauppinen, Eliza- beth Coulthard, et al. (2019). “Novel blood biomarkers that correlate with cognitive performance and hippocampal volumetry: potential for early diagnosis of Alzheimer’s disease”. In: Journal of Alzheimer’s Disease 67.3, pp. 931–947. Haxby, James V, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge (2011). “A common, high-dimensional model of the representa- tional space in human ventral temporal cortex”. In: 72.2, pp. 404– 416. Yousefnezhad, Muhammad, Alessandro Selvitella, Liangxiu Han, and Dao- qiang Zhang (2020). “Supervised Hyperalignment for multi-subject fMRI data alignment”. In: IEEE Transactions on Cognitive and Developmental Sys- tems. Haxby, James V, J Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong (2020). “Hyperalignment: Modeling shared information encoded in id- iosyncratic cortical topographies”. In: Elife 9, e56601.

10 Chapter 2

Literature Review

2.1 Neuroimaging

Functional neuroimaging had its origin at the same time that modern brain imaging techniques were introduced in . The advancement of fMRI has been integral in allowing for the demonstration of the relative timing information used to determine the relationship between human perception and (Menon and Kim, 1999). The advancement of fMRI have al- lowed neurosurgeons to perform cognitive and basic neuroscience tasks on patients prior to brain surgery, allowing them to plan the procedure in a man- ner that does a minimum amount of damage . According to Raichle, 2010, the marriage of disciplines and techniques has bound the field of cognitive neuroscience, which has expanded to include fields like cell biology, and genetics of imaging signals. Raichle, 2010 adds that some of the neuroimaging technologies that have been at the forefront of this galvanization and growth in the field include positron emission tomography (PET) and fMRI.

11 Functional neuroimaging has grown exponentially in recent years. Ac- cording to Cabeza and St Jacques, 2007, the continued innovations in the techniques and technologies to obtain personal memories have availed the biomedical industry a chance to delve into the functional neuroanatomy of people’s past. The reason for this is that the use of these neuroimaging tech- nologies has shown that a recent autobiographic memory can trigger the activation of the hippocampus.

Neuroimaging has become a big part of the modern medical industry.

Klein, 2010 details that functional neuroimaging techniques like fMRI have revolutionized neuroscience by providing the essential tools needed to bring tighter cognitive psychology and the old neurosurgery model. According to

Talavage, Gonzalez-Castillo, and Scott, 2014, functional neuroimaging has been an essential part of the enhancement of auditory perception and language in the last 30 years. These functional neuroimaging techniques that made this breakthrough possible are associated with the localization of central response. In the research, the authors employed the use of PET and fMRI to ascertain the role that auditory neuroimaging has played especially when it comes to the perception of and communication of the acoustic world, especially in the brain regions. As such, neuroimaging has adopted an extensive clinical application with its effectiveness increasing exponentially. Siegle, Carter, and Thase, 2006 also show the important role that neuroimaging has played. In their research, Siegle, Carter, and Thase, 2006 present a scenario where neuroimaging can be used to prevent or predict recovery from unipolar depression.

On the same note, Phan et al., 2004 detail that neuroimaging studies have

12 allowed for the study of human emotions. This has been done through the use of technological advancement in the field such as fMRI and PET. From the research, it was clear that the use of PET and fMRI allowed the researcher to establish that some discrete parts of the brain were associated with emotions and emotional tasks while the others were involved in the general emotional perception and controlling of the emotions of people.

2.2 Hyperalignment for fMRI Data

Multivariate pattern classifications can map different cognitive states tothe brain task (Yousefnezhad and Zhang, 2017). However, a problem remains the large inter-individual differences in both brain anatomy and functional localization after anatomical alignment.

This leads to a mis-alignement of features in a machine learning algorithm which can have detrimental effects on model performance. Recent approaches towards predictive modeling have sought to circumvent this problem by mapping individual brains into a functional population-level reference space rather than an anatomically based brain space. An important step in the development was the introduction of the ‘hyperalignment’ procedure (Haxby et al., 2011). Here brain activity patterns corresponding to stimuli and other cognitive events are represented as vectors in a neural representational space spanned by the voxels in a local neighborhood. The procedure rotates each participant’s local voxel-wise activity patterns through multivariate voxel space using a Procrustes transformation to align the representational geometry across subjects. This dramatically reduce inter-subject variability in functional

13 anatomy, thereby increasing the accuracy and specificity of population-level models.

The relationship between hyperalignment and fMRI has been largely doc- umented in recent years. According to Yousefnezhad et al., 2020, hyper- alignment has been used widely in multivariate pattern analysis to establish the cognitive state of the human mind especially using fMRI data. In their research, the authors employ the use of supervised hyperalignment which ensures a better function alignment for MVPA. This technique ensures that the correlation among the stimuli belonging to the same category and minimizes the correlation between distinct categories of the stimuli. To achieve a superior fMRI outcome compared to other state-of-the-art hyperalignment algorithms,

Yousefnezhad and Zhang, 2017 introduce a new method in the form of deep hyperalignment. According to the author, deep hyperalignment is a scalable, deep extension and regularized hyperalignment method that is suited for the usage of fMRI datasets. Furthermore, the method uses a stochastic gradient descent for optimization and uses a parametric approach. The research shows that this technique offers a better performance metric when dealing with fMRI datasets compared to other methods used in hyperalignment. In another research, Yousefnezhad and Zhang, 2017 hint that hyperalignment is the most effective function alignment method especially since they can be mathemati- cally formulated by the Canonical Correlation Analysis (CCA) methods. In this research, the authors introduce a new hyperalignment method in the form of Local Discriminant Hyperalignment (LDHA). This is a supervised hyperalignment method that is associated with better functional alignment

14 for MVP analysis.

When dealing with multi-subject fMRI research, many researchers have ascertained that inter-subject alignment is an important pat of performing analysis. According to Chen et al., 2014, one of the methods that have proven to be at the forefront of achieving such an alignment is hyperalignment. The research introduces a new hyperalignment method in the form of the joint SVD-hyperalignment. In this case, Chen et al., 2014 present the joint SVD- hyperalignment as the most ideal method since it is more scalable and offers analytical and empirical results by using the fMRI datasets. From the results of the research, it is apparent that the method offers computational complexity and accuracy when determining the multi-subject alignment. On the same note, Xu et al., 2012stress the importance of inter-subject alignment when it comes to multi-subject fMRI research.

The researchers also employed the use of SVD-hyperalignment. The result of the research shows that the aligned functional fMRI datasets improved performance. In another research conducted by Al-Wasity et al., 2020, the researchers examine the Between Subject Classification (BSC) based on the hyperalignment of the motor cortex. From this examination, their outcomes were obtained. First, it was clear that the hyperalignment was effective in aligning neural responses in the motor cortex to enable BSC of motor imagery. Second, the effectiveness of the result was dependent on the order in which the participants of the research entered the hyperalignment algorithm. The final result indicated that the BSC of neural response patterns based onthe hyperalignment exceed the standard within the subject classification approach

15 and BSC based on anatomical alignment.

On the same note, a new method of alignment can be employed to im- prove the MVPA alignment. Lorbert and Ramadge, 2012 offer a regularized hyperalignment in the form of Kernel hyperalignment. A method like SDV- hyperalignment includes nonlinear measures of similarity. To add on, unlike other algorithms, this type of hyperalignment allows for the multiple subject alignment of large RIOs when FRMI data analysis is involved. At the same time, Lorbert and Ramadge, 2012 pointed out that the method enables the alignment of multiple datasets with a large number of base features. Although Rustamov and Guibas, 2013 agree that hyperalignment is essential in effective classification performance, the authors point out that the technique failsto include some potentially essential information synonymous with anatomy. To better improve the method, Rustamov and Guibas, 2013 take a different ap- proach to hyperalignment that allows for the incorporation of the anatomical information in a way that does not offset the original intent of the research. once the approach was alternated and other anatomic information factored in, the effectiveness of hyperalignment was observed in the classification performance.

Finally, searchlight hyperalignment (Guntupalli et al., 2016) has been devel- oped to allow for whole-brain coverage. This is an extension of the previous region of interest (ROI) hyperalignment algorithm proposed by Haxby et al.,

2011. While, standard hyperalignment aligns neural representational spaces of ROIs for each subject into a common model space, searchlight hyperalign- ment repeats the process in all cortical searchlights. Here, local searchlight

16 transformation matrices are combined into a single subject-specific matrix that maps data from that subject into the common model space.In the remainder of this thesis we explore the performance of searchlight hyperalignment.

17 References

Menon, Ravi S and Seong-Gi Kim (1999). “Spatial and temporal limits in cognitive neuroimaging with fMRI”. In: Trends in cognitive sciences 3.6, pp. 207–216. Raichle, Marcus E (2010). “the origins of functional brain imaging in humans.” In: Handbook of clinical neurology 95, pp. 257–268. Cabeza, Roberto and Peggy St Jacques (2007). “Functional neuroimaging of autobiographical memory”. In: Trends in cognitive sciences 11.5, pp. 219–227. Klein, Colin (2010). “Philosophical issues in neuroimaging”. In: Philosophy Compass 5.2, pp. 186–198. Talavage, Thomas M, Javier Gonzalez-Castillo, and Sophie K Scott (2014). “Auditory neuroimaging with fMRI and PET”. In: Hearing research 307, pp. 4–15. Siegle, Greg J, Cameron S Carter, and Michael E Thase (2006). “Use of FMRI to predict recovery from unipolar depression with cognitive behavior therapy”. In: American Journal of Psychiatry 163.4, pp. 735–738. Phan, K Luan, Tor D Wager, Stephan F Taylor, and Israel Liberzon (2004). “Functional neuroimaging studies of human emotions”. In: CNS spectrums 9, pp. 258–266. Yousefnezhad, Muhammad and Daoqiang Zhang (2017). “Deep hyperalign- ment”. In: arXiv preprint arXiv:1710.03923. Haxby, James V, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge (2011). “A common, high-dimensional model of the representa- tional space in human ventral temporal cortex”. In: Neuron 72.2, pp. 404– 416. Yousefnezhad, Muhammad, Alessandro Selvitella, Liangxiu Han, and Dao- qiang Zhang (2020). “Supervised Hyperalignment for multi-subject fMRI data alignment”. In: IEEE Transactions on Cognitive and Developmental Sys- tems.

18 Chen, Po-Hsuan, J Swaroop Guntupalli, James V Haxby, and Peter J Ramadge (2014). “Joint SVD-Hyperalignment for multi-subject FMRI data align- ment”. In: 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, pp. 1–6. Xu, Hao, Alexander Lorbert, Peter J Ramadge, J Swaroop Guntupalli, and James V Haxby (2012). “Regularized hyperalignment of multi-set fMRI data”. In: 2012 IEEE Statistical Signal Processing Workshop (SSP). IEEE, pp. 229–232. Al-Wasity, Salim, Stefan Vogt, Aleksandra Vuckovic, and Frank E Pollick (2020). “Hyperalignment of motor cortical areas based on motor imagery during action observation”. In: Scientific reports 10.1, pp. 1–12. Lorbert, Alexander and Peter J Ramadge (2012). “Kernel hyperalignment”. In: Advances in Neural Information Processing Systems 25, pp. 1790–1798. Rustamov, Raif M and Leonidas Guibas (2013). “Hyperalignment of multi- subject fMRI data by synchronized projections”. In: Machine Learning and Interpretation in Neuroimaging. Springer, pp. 115–121. Guntupalli, J Swaroop, Michael Hanke, Yaroslav O Halchenko, Andrew C Connolly, Peter J Ramadge, and James V Haxby (2016). “A model of rep- resentational spaces in human cortex”. In: Cerebral cortex 26.6, pp. 2919– 2934.

19 Chapter 3

Methodology

3.1 Human Connectome Project Data

We used the preprocessed and artifact-removed rs-fMRI data as provided by the 900 subject data release. Van Essen et al., 2013 provides a detailed explana- tion of the entire acquisition protocol. Data was acquired on a customized 3T Siemens connectome-Skyra 3T scanner. Participants completed two scanning sessions on separate days. A T1-weighted structural scan was acquired during each session (acquisition time = 7.6 min, TR/TE/TI = 2400/2.14/1000 ms, resolution = 0.7x0.7x0.7 mm3, SENSE factor = 2, flip angle = 8). A simultane- ous multi-slice pulse sequence with an acceleration factor of eight (U˘gurbil et al., 2013) was used to acquire two rs-fMRI runs during each session. Each consisted of 1200 volumes sampled every 0.72 seconds, at 2-mm isotropic spatial resolution (TE = 33.1 ms, flip angle = 52, 72 axial slices). Participants were instructed to keep their eyes open and fixated on a cross hair, while remaining as still as possible. Within sessions, phase encoding directions for the two runs were alternated between right-to-left (RL) and left-to-right (LR)

20 directions.

The preprocessing and the artifact-removing procedures performed are explained in detail elsewhere (Glasser et al., 2013; Smith et al., 2013; Griffanti et al., 2014; Salimi-Khorshidi et al., 2014), and briefly described below. Each run was minimally preprocessed (Glasser et al., 2013; Smith et al., 2013), and artifacts were removed using the Oxford Center for Functional MRI of the Brain’s (FMRIB) ICA-based X-noiseifier (ICA + FIX) procedure (Griffanti et al., 2014; Salimi-Khorshidi et al., 2014). At this point in the processing pipeline, rs-fMRI data from each run were represented as a time series of grayordinates, a combination of cortical surface vertices and subcortical standard-space voxels (Glasser et al., 2013). Each run was temporally demeaned and variance normalized (Beckmann and Smith, 2004).

In total we had 4 runs each consisting of 15 minutes of resting-state data. We denote the two scans from Day 1 as Rest1_LR and Rest1_RL based on the phase encoding direction used to acquire the data. Similarly, we will denote the two scans from Day 2 as Rest2_LR and Rest2_RL.

3.2 Hyperalignment

The research aims to determine the impact of hyperalignment algorithms on FMRI data analysis. From the literature review, it is apparent that more than one hyperalignment method exists that can be used to make the MVPA more effective when using fMRI datasets. Here we adapted searchlight hyperalign- ment algorithm.

Searchlight hyperalignment (Guntupalli et al., 2016) is an extension of

21 the previous region of interest (ROI) hyperalignment algorithm proposed by Haxby et al., 2011. Standard hyperalignment aligns neural representational spaces of ROIs for each subject into a common model space using a high- dimensional spatial rotation. In contrast, searchlight hyperalignment repeats the process in all cortical searchlights, thus providing whole-brain coverage.

Here, local searchlight transformation matrices are combined into a single subject-specific matrix that maps data from that subject into the common model space. The surface searchlights was defined in all subjects where each surface node was the center of a searchlight cortical disc of radius 5 mm in this thesis. Thus all these searchlights from all subjects at each cortical location were hyperaligned and aggregated into an N × N transformation matrix for each subject where N is the number of voxels in the gray matter mask. We used the transpose of each subject’s transformation matrix to get a reverse mapping from the common space to each subject’s cortical voxel space. Whole cortex hyperalignment transformation matrices were then applied to new data. In the remainder of this thesis we explore the performance of searchlight hyperalignment. Searchlight hyperalignment was applied to a training session of the rs-fMRI data (Rest1_LR), and subsequently the derived transformation was applied to the three other sessions.

After hyperalignment was performed both the unaligned and aligned data were parcellated using the HCP MMP 1.0 atlas, which is popularly referred to as the Glasser atlas. It consists of 180 regions per hemisphere. Once we have parcellated the data we compute connectivity matrices using pair-wise correlation between regions. We will use the image intra-class correlation

22 coefficient and functional connectome fingerprinting approaches to evaluate the difference in reproducibility of the computed connectivity matrices before and after the alignment. In addition, we will develop a predictive model to in- vestigate whether the hyperalignment algorithm contributes to improvement in the prediction of certain behavioral variables.

3.3 Measures of Reliability: ICC, I2C2 and finger- printing

3.3.1 I2C2

The image intra-class correlation (I2C2) coefficient is used as a global measure of reliability for functional connectivity where higher reproducibility yields a higher value of I2C2.

Let Xi(υ) be the true image and Wij(υ) be the proxy measurements of Xi(υ) at voxel υ. Further assume that all images can be represented as V × 1 vectors. Then the image measurement error can be expressed as follows:

Wij(v) = Xi(v) + Uij(v), (3.1)

Here the index i stands for subject and j stands for session. The observed {︁ }︁ proxy images can be expressed as W ij = Wij(v) : v = 1, . . . , V , while the latent true images can be expressed as Xi = {Xi(v) : v = 1, . . . , V}. The mea- {︁ }︁ surement error, Uij = Uij(v) : v = 1, . . . , V , is assumed to be independent across subjects.

23 Let KW, KX, and KU be the covariance of W ij, Xi, Uij, respectively. Thus,

(︁ )︁ KW = cov W ij, W ij

KX = cov (Xi, Xi) (︁ )︁ KU = cov Uij, Uij

The covariance operator of the observed data can be written KW = KX + KU, where KX represents the within-subject covariance, and KU is the covariance of the measurement error. The image intra-class correlation (I2C2) coefficient is defined as

trace (K ) trace (K ) − trace (K ) trace (K ) ρ = X = W U = 1 − U . (3.2) trace (KW ) trace (KW ) trace (KW )

A method of moments estimator is used to compute:

I J V 1 i 2 ( ) = {︁ ( ) − ( )}︁ traceˆ︁ KW I ∑ ∑ ∑ Wij v W.. v , ∑i=1 Ji − 1 i=1 j=1 v=1 and I J V 1 i 2 ( ) = {︁ ( ) − ( )}︁ traceˆ︁ KU I ∑ ∑ ∑ Wij v Wi. v . ∑i=1 (Ji − 1) i=1 j=1 v=1

Ji where W..(v) = Σi,j,vWij(v)/IJ and Wi.(v) = ∑j=1 Wij(v)/Ji. We used an R package to calculate the I2C2 http://www.biostat.jhsph. edu/~ccrainic/software.html. The input matrix of the I2C2 function is a N × p data matrix, where each row contains the observed correlation data from a particular session for a single subject computed as the upper triangle of the correlation matrix.

24 3.3.2 Fingerprinting

Since functional brain networks vary across individuals, sessions, and task states, researchers have developed a number of algorithms using functional MRI connectome to identify subjects from a large group. Finn et al., 2015 demonstrated that this individual variability is robust and reliable, and show that functional connectivity profiles can act as an identifying fingerprint.

In this model, Pearson correlation coefficients were calculated based on the connectivity matrices and normalized to z-scores using the Fisher trans- formation, resulting in a 360 * 360 symmetric connectivity matrix for each session for each subject. Then a database was created that consisted of a connectivity matrix of all individual subjects in a single condition, which is named as D = [Xi, i = 1, 2, ..., 50], where Xi is the 360 * 360 correlation matrix and the i denotes subject. Similarly, the target database, defined as Yi, consists of a connectivity matrix of all subjects from a different session. The similarity, defined as the Pearson correlation between two vectors of edge values taken from the target matrix and each of the database matrices, was computed and the predicted identity was that with maximal similarity score. If the target matrix is most similar to itself, thus the predicted identity matched the true identity, we assigned a score of 1, or 0, if it was not. We tested identification across all the target-database pairs, and the accuracy was measured as the percentage of subjects whose identity was correctly predicted out of the total number of subjects.

25 3.4 Predictive Modeling Approaches

Since establishing the relationship between individual differences in brain structure/function and individual differences in behavior is a major goal of modern neuroscience, here we build a connectome-based predictive model to investigate whether the hyperalignment algorithm contributes to predicting individual behavioral differences. In particular, we focus on five different measures, namely age, body mass index (BMI), depression score, gender, and fluid intelligence

For each subject, the 4 images are stitched into a single image with 4 channels. This would normalize the output value of the neural network, that is, the value becomes a 0-1 value. Once variables like gender and age have been normalized, we build the VGG-16 convolutional network and finally connect the fully connected network to output the five predicted values (age, BMI, depression score, gender, and fluid intelligence). To evaluate the prediction results, we calculated the MSE (Mean Squared Error), RMSE

(Root Mean SquaredError), and MAE (Mean Absolute Error) for the four continuous predicted values (all except gender). As predicting gender is a binary classification problem, we calculate the ROC curve and the AUC value instead.

26 References

Van Essen, David C, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, WU-Minn HCP Consortium, et al. (2013). “The WU-Minn human connectome project: an overview”. In: NeuroImage 80, pp. 62–79. U˘gurbil,Kamil, Junqian Xu, Edward J Auerbach, Steen Moeller, An T Vu, Julio M Duarte-Carvajalino, Christophe Lenglet, Xiaoping Wu, Sebastian Schmitter, Pierre Francois Van de Moortele, et al. (2013). “Pushing spatial and temporal resolution for functional and diffusion MRI in the Human Connectome Project”. In: Neuroimage 80, pp. 80–104. Glasser, Matthew F, Stamatios N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al. (2013). “The minimal prepro- cessing pipelines for the Human Connectome Project”. In: NeuroImage 80, pp. 105–124. Smith, Stephen M, Christian F Beckmann, Jesper Andersson, Edward J Auer- bach, Janine Bijsterbosch, Gwenaëlle Douaud, Eugene Duff, David A Fein- berg, Ludovica Griffanti, Michael P Harms, et al. (2013). “Resting-state fMRI in the human connectome project”. In: NeuroImage 80, pp. 144–168. Griffanti, Ludovica, Gholamreza Salimi-Khorshidi, Christian F Beckmann, Edward J Auerbach, Gwenaëlle Douaud, Claire E Sexton, Enik˝oZsoldos, Klaus P Ebmeier, Nicola Filippini, Clare E Mackay, et al. (2014). “ICA- based artefact removal and accelerated fMRI acquisition for improved resting state network imaging”. In: NeuroImage 95, pp. 232–247. Salimi-Khorshidi, Gholamreza, Gwenaëlle Douaud, Christian F Beckmann, Matthew F Glasser, Ludovica Griffanti, and Stephen M Smith (2014). “Au- tomatic denoising of functional MRI data: combining independent com- ponent analysis and hierarchical fusion of classifiers”. In: NeuroImage 90, pp. 449–468.

27 Beckmann, Christian F and Stephen M Smith (2004). “Probabilistic indepen- dent component analysis for functional magnetic resonance imaging”. In: Medical Imaging, IEEE Transactions on 23.2, pp. 137–152. Guntupalli, J Swaroop, Michael Hanke, Yaroslav O Halchenko, Andrew C Connolly, Peter J Ramadge, and James V Haxby (2016). “A model of rep- resentational spaces in human cortex”. In: Cerebral cortex 26.6, pp. 2919– 2934. Haxby, James V, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge (2011). “A common, high-dimensional model of the representa- tional space in human ventral temporal cortex”. In: Neuron 72.2, pp. 404– 416. Finn, Emily S, Xilin Shen, Dustin Scheinost, Monica D Rosenberg, Jessica Huang, Marvin M Chun, Xenophon Papademetris, and R Todd Constable (2015). “Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity”. In: Nature neuroscience 18.11, pp. 1664– 1671.

28 Chapter 4

Results

4.1 Research Design

In this section, the rs-fMRI data obtained from the Human Connectome Project was evaluated, both before and after application of the hyperalignment algo- rithm. Therefore, the data is processed using two different pipelines, denoted aligned and the unaligned data. The dataset used includes 900 subjects out of which 50 were randomly chosen to be included in our analysis.

For the unaligned pipeline we extracted data using two different parcella- tion schemes:

(A) HCP MMP 1.0 (Glasser atlas; 360 regions);

(B) Cole-Anticevic Brain Network Parcellation (Ji et al., 2019; 700 regions).

For the aligned pipeline we performed searchlight hyperalignment on surface data using group-averaged surface file (midthickness version) for training each hemisphere separately. We processed the data using Python 3.8.1 including libraries such as numpy, scipy, nibabel, nilearn, hcp_utils, and

29 mvpa2 and analyzed the model with "Surface Query Engine" in searchlights with a 5 mm radius. After alignment we parcellated the brain using the

HCP MMP 1.0 atlas. The hyperalignment was performed on data from the first scanning session (Rest1_LR), and the derived transformations were later applied to the remaining three sessions (Rest1_RL, Rest2_LR, and Rest2_RL).

For both processing pipelines, we calculated the image intra-class correla- tion coefficient and the functional connectome fingerprinting accuracy. We also build a convolutional neural network model to predict age, BMI, fluid intelligence and depression score. The values show the data before and after performing the hyperalignment algorithm. The final result of the analysis will be presented in the form of tables which can be used to visually show the degree of reliability between the two.

4.2 Reliability

Measure Unaligned Data I2C2 Rest1_LR vs. Rest2_LR 0.4468753 Rest1_RL vs. Rest2_RL 0.4274989 Rest1_LR vs. Rest1_RL 0.4436231 Rest2_LR vs. Rest2_RL 0.4362019 Fingerprinting Accuracy 0.98

Table 4.1: Reliability Results for Unaligned Data of the Whole Brain

We first evaluated reliability for the unaligned data at the whole brain level, using the image intra-class correlation coefficient (I2C2) and functional con- nectome fingerprinting accuracy as shown in Table 4.1. We use this as a

30 benchmark data set. However, we only performed searchlight hyperalign- ment on the surface data which includes 360 regions and we restrict our comparisons between aligned and unaligned data to these regions.

Table 4.1 shows the pairwise I2C2 scores between the different sessions. Here we show results comparing similar acquisition techniques across dif- ferent days and different acquisition techniques across different days. The results are similar across comparisons with a score around 0.44, indicating a moderate degree of reliability. The fingerprinting results show an impressive

98% accuracy rate.

Measure Unaligned Data Aligned Data I2C2 I2C2 Rest1_LR vs. Rest2_LR 0.2509014 0.2617642 Rest1_RL vs. Rest2_RL 0.3043583 0.3107248 Rest1_LR vs. Rest1_RL 0.2593981 0.2685964 Rest2_LR vs. Rest2_RL 0.1974292 0.2069266

Table 4.2: I2C2 Results

Table 4.2 shows the I2C2 scores for the fMRI data before and after hyper- alignment of the surface 360 regions. By restricting the data to surface regions instead of including subcortical regions, we can notice a significant decrease in the reliability of the unaligned data. However, importantly the reliability of the aligned and unaligned data is similar. Thus, any concern of reduced reliability due to the removal of between-subject variation by transforming the data into a standard space did not occur. In fact, reliability improved slightly after hyperalignment. It is apparent that hyperalignment slightly increases the efficiency of the analysis. These findings are in line with the assessment ofXu

31 et al., 2012 who point out that the most successful form of functional alignment is hyperalignment. This sentiment is shared by Yousefnezhad and Zhang,

2017 who hint that hyperalignment is the most effective function alignment method. Future work will attempt to confirm that the results carry-over when performing hyperalignment on the whole brain. Interestingly, the combination of sessions that had the highest I2C2 score after alignment was Rest1_RL and Rest2_RL even though they were both test datasets in the analysis (i.e. the data was trained on Rest1_LR and the transformation was applied to the other sessions). This may indicate that the transformations effectively carries over from the training to the test datasets.

Unaligned data Aligned data Accuracy 0.56 0.54

Table 4.3: A table showing the fingerprinting accuracy of the aligned and unaligned data

Table 4.3 shows the results for functional connectivity fingerprinting. Again, there are significnat costs involved with not including the subcor- tical regions, as there is a decrease in accuracy for the unaligned data. The accuracy of the unaligned one is slightly higher than that of the aligned data. However, both are above the 50% mark making which illustrates that there is important information in the connectivity between surface regions that is unique to individual subjects. This outcome slightly contradicts the argument of authors like Yousefnezhad and Zhang, 2017 who insist that alignment offers grounds for improvement in the accuracy and effectiveness of neuroimaging.

32 However, we note that this may be an error due to the small size of the data set.

4.3 Predictive Modeling

Despite the findings of the fingerprinting, the predictive methods show that using aligned data works better than unaligned data when analyzing fMRI data. Figure 4.1, 4.2, 4.3 and 4.4 show the prediction results of the using aligned and unaligned data. To evaluate the performance of the prediction results, we calculate the Mean Absolute Error (MAE) of the regression problem, where a smaller MAE value indicates that the prediction result is closer to the true result (Table 4.4). Table 4.5 employs the use of the AUC value of a ROC curve to determine the relationship between the using aligned and unaligned data of the gender of patients showing that aligned data had a higher performance threshold than the unaligned data.

For both the MAE values and the AUC values, they both indicate aligned data had a higher performance threshold than the unaligned data. Therefore, this means that when a hyperalignment algorithm is introduced in the align- ment of multi-subject fMRI research, the analysis improved. Although Lorbert and Ramadge, 2012 focus on Kernel hyperalignment, they share the same sentiments as the finding of the research that methods of alignment canbe employed to improve the MVPA alignment. Hyperalignment is an essential aspect of neuroimaging since it helps reduce the limitations faced when using fMRI. This is because the alignment of the features of the fMRI data across brains is essential in ensuring that the high variability in individual responses

33 is accounted for.

Figure 4.1: Scatter plots of predicted values vs actual values for age. The results for the unaligned data are show to the left and the results for the aligned data to the right.

34 Figure 4.2: Scatter plots of predicted values vs actual values for BMI. The results for the unaligned data are show to the left and the results for the aligned data to the right.

35 Figure 4.3: Scatter plots of predicted values vs actual values for fluid intelligence. The results for the unaligned data are show to the left and the results for the aligned data to the right.

36 Measure Unaligned Data Aligned Data MAE MAE Age 3.64 2.98 BMI 3.99 3.85 Fluid Intelligence 6.84 4.46 Depression Score 2.7 2.0

Table 4.4: Evaluation Results

Figure 4.4: Scatter plots of predicted values vs actual values for depression score. The results for the unaligned data are show to the left and the results for the aligned data to the right.

37 AUC Value Unaligned Data 0.79 Aligned Data 0.81

Table 4.5: AUC value for predicting gender

38 References

Ji, Jie Lisa, Marjolein Spronk, Kaustubh Kulkarni, Grega Repovš, Alan An- ticevic, and Michael W Cole (2019). “Mapping the human brain’s cortical- subcortical functional network organization”. In: Neuroimage 185, pp. 35– 57. Xu, Hao, Alexander Lorbert, Peter J Ramadge, J Swaroop Guntupalli, and James V Haxby (2012). “Regularized hyperalignment of multi-set fMRI data”. In: 2012 IEEE Statistical Signal Processing Workshop (SSP). IEEE, pp. 229–232. Yousefnezhad, Muhammad and Daoqiang Zhang (2017). “Local discriminant hyperalignment for multi-subject fmri data alignment”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. 1. Lorbert, Alexander and Peter J Ramadge (2012). “Kernel hyperalignment”. In: Advances in Neural Information Processing Systems 25, pp. 1790–1798.

39 Chapter 5

Discussion and Conclusion

5.1 Discussion

Using a publicly available dataset, the present work assessed the use of the hyperalignment algorithm in fRMI data analysis to reliably characterize inter- individual variation in functional brain networks. To achieve this goal, first, a literature analysis was conducted. This employed the use of retrospective data on the topic and related fields. The inclusion of a literature review was necessary, especially when it comes to the comparative analysis of the outcomes of the research. Second, the research conducted data analysis on the impact of hyperalignment on fMRI data analysis. From the perspective of reliability, across the 4 scan sessions, we were able to demonstrate that the use of the hyperalignment algorithm does not reduce the reproducibility of the data, but instead somewhat enhances it. This was not a given as we hypothesized that by decreasing functional variation across subjects one might actually decrease test re-test reliability, as large inter-subject variability compared to intra-subject variability lead to increased reliability.

40 Finally, the outcome of our predictive model shows that the introduction of hyperalignment alignment to the multi-subject fMRI research improves the predictive performance. This is consistent with findings from other research studies. As such, our findings increase our confidence in applying the hyper- alignment algorithm to fMRI studies as to map inter-individual differences in brain function.

5.2 Limitations and future directions

Although our results are promising, there are several limitations to the findings of the present work which merit further consideration. First, due to the time- consuming nature of hyperalignment processing, as of the completion of this work, we have only completed the processing of 50 subjects. It may be that such a small data set does not accurately and completely reflect the characteristics of the hyperalignment algorithm on the entire HCP data set.

Therefore, future work will focus on increasing the size of the data set. This will necessitate developing more optimized computational algorithms for computing hyperalignment.

Second, the parcellation used in our research consisted of 360 nodes de- fined on the surface of the brain. However, it is clear that when addingthe subcortical areas and analyzing a more fine-grained scheme with 718 nodes, performance on both I2C2 and fingerprinting accuracy improved substantially.

This highlights the importance of the subcortical regions in analysis of this type. It clearly indicates that whole-brain hyperalignment should be done in a future work.

41 5.3 Conclusion

In sum, the presented work demonstrates that the inclusion of the hyper- alignment algorithm to fMRI preprocessing provides desirable effects and improves the overall prediction performance of a series of behavioral vari- ables. Moreover, the apparent improvement in results using an atlas with better spatial coverage suggests future directions of work.

42