IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014 2039 Comparative Evaluation of Registration Algorithms in Different Brain Databases With Varying Difficulty: Results and Insights Yangming Ou*, Hamed Akbari, Michel Bilello, Xiao Da, and Christos Davatzikos

Abstract—Evaluating various algorithms for the inter-subject resent the same anatomical structures. Image registration, espe- registration of brain magnetic resonance images (MRI) is a neces- cially deformable image registration, is a fundamental problem sary topic receiving growing attention. Existing studies evaluated in medical image computing. It is usually an indispensable com- image registration algorithms in specific tasks or using specific databases (e.g., only for skull-stripped images, only for single-site ponent in many analytic studies, including studies aiming to images, etc.). Consequently, the choice of registration algorithms understand population trends of imaging phenotypes, to mea- seems task- and usage/parameter-dependent. Nevertheless, re- sure longitudinal changes, to fuse multi-modality information, cent large-scale, often multi-institutional imaging-related studies to guide computerized interventions, to capture structure-func- create the need and raise the question whether some registration tion correlations, and many others (two recent comprehensive algorithms can 1) generally apply to various tasks/databases posing various challenges; 2) perform consistently well, and while surveys can be found in [1], [2]; various other surveys can be doing so, 3) require minimal or ideally no parameter tuning. In found in [3]–[9]). seeking answers to this question, we evaluated 12 general-purpose The past two decades have witnessed the development of registration algorithms, for their generality, accuracy and robust- many deformable registration algorithms. A comprehensive ness. We fixed their parameters at values suggested by algorithm evaluation of different registration methods has thus become developers as reported in the literature. We tested them in 7 databases/tasks, which present one or more of 4 commonly-en- a research topic of interest. It is the basis for users to choose countered challenges: 1) inter-subject anatomical variability in the most suitable methods for the problems at hand, and for skull-stripped images; 2) intensity homogeneity, noise and large algorithm developers to be better informed theoretically. A structural differences in raw images; 3) imaging protocol and comprehensive evaluation is a fairly complicated problem, field-of-view (FOV) differences in multi-site data; and 4) missing though. It oftentimes requires public databases, expert-labeling correspondences in pathology-bearing images. Totally 7,562 regis- trations were performed. Registration accuracies were measured of ground truth regions/landmarks, a comprehensive evaluation by (multi-)expert-annotated landmarks or regions of interest protocol, careful tunings of parameters, considerable computa- (ROIs). To ensure reproducibility, we used public software tools, tional resources, and a proper choice of data to reflect certain public databases (whenever possible), and we fully disclose the registration challenges or to meet certain (pre-)clinical needs. parameter settings. We show evaluation results, and discuss the performances in light of algorithms’ similarity metrics, transfor- mation models and optimization strategies. We also discuss future A. Literature on Evaluation of Algorithms in Brain MRI directions for the algorithm development and evaluations. Registration Index Terms—Brain magnetic resonance imaging (MRI), de- West et al. [10] evaluated 3 registration methods for formable image registration, evaluation, registration accuracy. multi-modal registration of the same subject undergoing neu- rosurgery, where the accuracy was measured on the fiducial markers. Hellier et al. [11] evaluated six methods (ANIMAL, I. INTRODUCTION Demons, SICLE, Mutual Information, Piecewise Affine, and MAGE registration is a process of transforming different the authors’ own method) in a database containing brain MRI I images into the same spatial coordinate system, so that after from 18 healthy subjects; they measured registration accuracy registration, the same spatial locations in different images rep- on expert-defined cortical regions. Yanovsky et al. [12] eval- uated three methods (fluid and two variations of the authors’ Manuscript received March 05, 2014; revised May 29, 2014; accepted May own methods, namely symmetric and asymmetric unbiased 31, 2014. Date of publication June 13, 2014; date of current version September methods), in mapping brains during longitudinal scans for the 29, 2014. Asterisk indicates corresponding author. detection of atrophy. They used 20 subjects from the ADNI *Y. Ou is with the Center for Biomedical Image Computing and Analytics (CBICA), Department of Radiology, University of Pennsylvania, Philadelphia, database, and all images were preprocessed to exclude skull PA, 19104 USA, and also with the Athinoula A. Martinos Center for Biomed- and dura. Yassa et al. [13] evaluated three methods (DARTEL, ical Imaging, Massachusetts General Hospital, Harvard Medical School, LDDMM, and Diffeomorphic Demons) in inter-subject regis- Charlestown, MA 02129 USA. H. Akbari, M. Billelo, X. Da, and C. Davatzikos are with the Center for tration of images especially at the medial temporal lobe, which Biomedical Image Computing and Analytics (CBICA), Department of Radi- is a crucial region for the study of memory. Christensen et ology, University of Pennsylvania, Philadelphia, PA 19104 USA. al. introduced the NIREP project [14], [15], which in the first Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. phase contains 16 publicly-available MR images, each having Digital Object Identifier 10.1109/TMI.2014.2330355 32 expert-defined regions of interests (ROIs). They also defined

0278-0062 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. 2040 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014 a comprehensive set of evaluation criteria (including inten- an attribute-based algorithm and 11 other intensity-based algo- sity-based variations, region-based overlaps, and transitivity rithms. Our work built upon and significantly expands previous errors). Based on these criteria they evaluated six methods evaluation studies of the similar nature in many aspects. (rigid, affine, AIR, Demons, SICLE, SLE) in [16]. Klein et al. 1) Our work evaluated registration methods under var- in [17] evaluated 14 publicly-available registration tools for ious tasks and databases presenting a wide variety of inter-subject registration of brain MR images. Four databases, challenges, rather than in a specifictaskcontainingspe- each containing images of multiple subjects, were used. While cific challenges. We identified four typical challenges in it evaluated perhaps the largest number of registration tools inter-subject registration of brain MRI (as will be de- so far, [17] only focused on skull-stripped, high quality, and scribed in Section II). We chose seven databases, each single-site images from healthy subjects. To evaluate registra- containing images of multiple subjects, to represent some tion methods under more challenges, Klein et al. in another or all of those challenges. This helped reveal whether study [18] included both skull-stripped and raw images. Here, a registration method could be generally applicable and raw images refer to those acquired directly from scanner, prior robust with regard to various challenges. to any image processing steps. 2) To reflect the robustness of registration algorithms es- pecially in multiple large-scale translational studies B. Need for a More Comprehensive Evaluation Study involving various tasks/databases, we fixed the parameters While all the aforementioned studies provided insightful and for each registration method throughout this paper (i.e., informative evaluations, each study only tested registration task-independent parameter settings). Particularly, in all methods in a specific task. For instance, for healthy subjects tasks/databases in this paper, we used the parameters as only (e.g., [11], [13], [16], [17]), for multi-modal fusion for the ones reported in [17] whenever applicable. Using such pathological subjects only (e.g., [10]), for skull-stripped images parameter settings was because that the parameters had only (e.g., [12], [13], [17]), for raw images only (e.g., [10], [11], been “optimized” by authors/developers of each specific [19]), for skull-stripped and raw images only (e.g., [18]), for algorithm for the registration of skull-stripped, prepro- single-site data only (e.g., [10], [11], [13], [16], [17]), and for cessed and normal-appearing brain MR images, which multi-site data only (e.g., [12]). In addition, different evaluation is a typical task in the inter-subject registration [17]. We studies included different registration methods for evaluation. realize that this set of parameters might not be optimal Moreover, they used different parameter settings for a same for other databases/tasks (e.g., those containing raw im- registration method. As a result, the choice of registration ages, pathology-bearing images, or multi-site images). algorithms seemed to be task- and database-dependent, and However, having a fixed set of parameters is perhaps was sensitive to parameter settings. how those registration algorithms would be used or first Nevertheless, many of today’s large-scale, pre-clinical tried in daily imaging-related translational studies, where and imaging-related studies present a wide variety of heavy parameter tunings are not only tedious, but also challenges—they may contain normal-appearing and/or less practical or reproducible. From another perspective, pathology-bearing images; they may contain skull-stripped it would be preferable if some registration algorithms and/or raw images; and they may contain images acquired could apply widely and could perform consistently well in from single- and/or multi-institutions. Facing all these pos- various tasks/databases giving a fixed set of parameters. sible challenges, there is an increasing need for registration To maintain reproducibility of our study, we used public algorithms that are publicly-available, that can widely apply to databases wherever possible (six out of seven databases used in various tasks/databases, that can perform relatively accurately this paper are public); we included registration algorithms/tools and robustly, that can be easily used by people with varying that are publicly available; and we will fully disclose the exact expertise in image registration, and that are without much need parameter settings in Appendix B. for parameter tunings. The rest of the paper is organized as follows. In Section II, we While automatically and effectively tuning parameters for identify four typical challenges in the inter-subject registration specific database/tasks is an important and active area of re- of brain MR images. In Section III, we present the protocol to search (e.g., [20]–[24]), having registration algorithms that can evaluate the accuracies of registration algorithms. In Section IV, be widely applicable to many tasks/databases is a very desir- we show the evaluation results. Finally, we discuss and conclude able property. This need stems not only from the size of studies, this paper in Section V. but also from the rapidly increasing number of studies under- taken in, for instance, translational neuroscience. Due to the lack II. TYPICAL CHALLENGES IN INTER-SUBJECT BRAIN of ground truth, tuning parameters is a difficult task even for MRI REGISTRATION technical experts. Moreover, when images are acquired and pro- Brain images from different subjects may present one or more cessed in multiple collaborative institutions, having registration of the following challenges to registration. algorithms that perform robustly and consistently well with a Challenge 1: Inter-Subject Anatomical Variability. Sub- fixed set of parameters becomes almost entirely necessary. jects may vary structurally (Fig. 1 shows some examples). Inter- subject variability is a common challenge in many registration C. Overview and Contributions of Our Evaluation Study tasks investigating neuro-development, neuro-degeneration, or Towards meeting this need, this paper evaluates 12 publicly- neuro-oncology. It is the main challenge against which regis- available and general-purpose registration algorithms, including tration methods were evaluated in the literature [13], [15]–[17]. OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2041

Fig. 1. Four randomly-chosen subjects in the NIREP database (the top two rows) and four randomly-chosen subjects in the LONI-LPBA40 database (the bottom two rows). For each subject, both the intensity image and the expert-an- notated ROI image are shown. Different colors represent different ROIs in each database. These two databases were used to evaluate how registration methods perform facing challenges arising from the inter-subject variability (Challenge 1).

Challenge 2: Intensity Inhomogeneity, Noise and Structural Difference in Raw Images. In addition to inter-subject anatom- ical variability, images may suffer from intensity inhomogeneity (due to bias field), background noise, and low contrast. With skulls, ears, neck structures present in the raw images, subjects may also present larger deformations in those nonbrain struc- tures compared to cortical structures (see Fig. 2 for example). Registration of raw images is necessary when 1) skull stripping is erroneous, so one has to work with the with-skull raw images; or 2) when registration itself is part of the skull-stripping step (e.g., in multi-atlas-based skull stripping approaches [19], [25]). Challenge 3: Protocol and FOV Differences in Multi-site Databases. Many of today’s large-scale translational imaging- Fig. 2. Images and annotations of two randomly chosen subjects from each of related studies involve brain MR images acquired from multiple the three databases we used to represent Challenge 2 (intensity inhomogeneity, institutions. Since MR scanners, imaging protocols, and FOVs noise and structural differences in raw brain images). (a) From the BrainWeb may vary from institution to institution, the acquired images database. (b) From the IBSR database. (c) From the OASIS database. may vary greatly. Especially when the FOV is different, which is not uncommon in multi-site databases, some images may con- tain structures that do not show up in other images (see Fig. 3 method that is relatively more robust to imaging protocol and for an example). One can rely on experts to interactively crop FOV differences is therefore of interest. images, so that images from various institutions cover roughly Challenge 4: Pathology-induced Missing Correspon- the same FOV. However, the manual cropping is labor-inten- dences in Pathology-Bearing Images. Spatially normalizing a sive, subject to intra-/inter-rater variability, and may become in- number of pathology-bearing images into a normal-appearing tractable for today’s large-scale studies. Seeking a registration template space offers opportunities to understand the common 2042 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

Fig. 4. Database to evaluate how registration methods perform facing the chal- lenge arising from the pathology-induced missing correspondences (i.e., Chal- lenge 4). Red arrows point out the regions that contain the cavity (after the re- section of the original tumors) and the recurrent tumors. Their correspondences are difficult to find in the normal-appearing template image (second row).

is ideal if a general-purpose registration algorithm, segmenta- tion-free in itself, can perform well in pathological-to-normal subject registration. That is, without prior knowledge of the presence or the location of the pathologies, nor any partition of the pathological versus normal regions, it is ideal if a registration algorithm can find correspondences in places where correspon- dences can be found, while relaxing the deformation (or reducing the pathology-induced bias) in regions where correspondences can hardly be established.

III. EVALU AT ION PROTOCOL In the following, Section III-A presents the databases we chose to represent these four challenges aforementioned in Section II. Then Section III-B briefly introduces the registration methods/tools we included in this evaluation. Section III-C Fig. 3. Three-plane view of the intensity images and annotation images from elaborates parameter settings for all methods, with an emphasis three randomly-chosen subjects in the ADNI database. White color in the an- on how to maintain the fairness, transparency, and repro- notation images denotes the brain masks, and red denotes hippocampus masks. Blue contours in panel (a) point to the region that exists in one image, but does ducibility in our evaluation. Section III-D describes the criteria not exist in other images, due to the FOV differences in multiple imaging in- to measure registration accuracies. stitutions. The ADNI database was used to represent Challenge 3 (on top of Challenges 1, 2). (a) A normal control (NC) subject. (b) A mild-cognitive-im- A. Databases pairment (MCI) subject. (c) An Alzheimer’s Disease (AD) subject. Seven databases were used. Of them, six are publicly avail- able. Specifically, we used two public databases to represent spatial patterns of diseases. Pathologies present in the patients’ challenge 1 (Section III-A1); three public databases to repre- images, but not in the normal-appearing template. This poses sent challenge 2 (Section III-A2); one public database to repre- the so-called missing correspondence problem (see Fig. 4 for sent challenge 3 (Section III-A3); and one in-house database to example). An ideal registration approach should accurately represent challenge 4 (Section III-A4). They are summarized in align the normal regions (which do have correspondences across Table I and introduced in the following. images), and relax the deformation in the pathology-affected 1) Databases Representing Challenge 1: Two publicly- regions, where no correspondences can be found [26], [27]. Lit- available and single-site databases, NIREP and LONI-LPBA40, erature has suggested to either mask out the pathological regions were used to represent Challenge 1 (inter-subject variability). from the registration process (i.e., the cost-function-masking Both databases contain multiple normal subjects. Images in approach [27]), or, to simulate a pathological region in the the two databases have been skull-stripped by neuroradiolo- normal-appearing template (i.e., the pathology-seeding ap- gists. Both databases contain T1-weighted (T1w) MR images proach [28]–[34]). However, both approaches require a careful (sequence parameters in Table I). Neuroradiologists annotated segmentation of the pathological regions, which in itself is not those images into a number of ROIs (32 ROIs in the NIREP an easy or affordable task, especially in large-scale studies. It database and 56 ROIs in the LONI-LPBA40 database). The OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2043

TABLE I The detailed lists of those ROIs can be found in Appendix C. DATABASES USED IN OUR STUDY.ABBREVIATIONS:UCLA—UNIVERSITY OF The detailed information about how ROIs were annotated can CALIFORNIA AT LOS ANGELES; MGH—MASSACHUSETTS GENERAL HOSPITAL; be found in [14] and [35]. Fig. 1 shows the intensity images BIRN—BIOMEDICAL INFORMATICS RESEARCH NETWORK; SPGR—SPOILED GRADIENT ECHO PULSE SEQUENCE; MP-RAGE—MAGNETIZATION-PREPARED and the corresponding expert-annotated ROI images from RAPID ACQUISITION WITH GRADIENT ECHO;TR—REPETITION TIME; four subjects in the NIREP database and four subjects in the TE—ECHO TIME;FA—FLIP ANGLE, N/A—NOT AVA I L A B L E LONI-LPBA40 database. These databases were also used to evaluate the performance of registration methods in other similar studies (e.g., [15], [17]). Registration was carried out from every subject to every other subject in the same database. This removed any bias in the selec- tion of source and target images in the registration. This led to 240 ,or210 , registrations in the NIREP, or LONI-LPBA40, database, for each registration method. Be- fore registration, we removed the bias field inhomogeneity by the N3 algorithm (using the default parameters) [36], and re- duced the intensity difference between the two images by a his- togram matching step. 2) Databases Representing Challenge 2: Three public databases were used, containing raw brain MR images from multiple subjects. They were: BrainWeb, IBSR and OASIS databases. Specifically, the BrainWeb database [37] contains raw brain images of 20 healthy subjects. In each subject, every image voxel has been annotated as one of the 11 brain or nonbrain tissue types or structures: cerebrospinal fluid (CSF), gray matter (GM), white matter (WM), fat, muscle, muscle/skin, skull, vessels, around fat, dura matter, and bone marrow. Fig. 2(a) presents two randomly-chosen subjects from the BrainWeb database, including their intensity images and the corresponding annotation images. We have randomly picked 11 BrainWeb subjects, leading to 110 pair-wise registrations for each registration algorithm. The IBSR database [38] consists of raw T1-weighted MRI scans of 20 healthy subjects from the Center for Morphometric Analysis at the Massachusetts General Hospital. In this database, the brain masks have been manually delineated by trained investigators for each subject. We randomly picked up 10 IBSR subjects, leadingto90 inter-subject registrations for each registration algorithm. Fig. 2(b) shows the raw intensity images and the corresponding brain masks of 2 randomly-chosen IBSR subjects. The OASIS database [39] contains cross-sectional T1-weighted MRI Data in young, middle aged, nondemented, and demented older adults, to facilitate basic and clinical dis- coveries in neuroscience. The brain masks were first generated by an automated method based on a registration to an atlas, and then proofread and corrected by human experts before the release. We randomly selected 10 OASIS subjects, leading to 90 inter-subject registrations for each registration algorithm. Fig. 2(c) shows the raw intensity images and the corresponding brain masks of 2 randomly-chosen OASIS subjects. Similar to those databases used for Challenge 1, the expert annotations were not used in the registration process; rather, they only served as references to evaluate registration accuracy, as we will explain later in Section III-D2. 3) Database Representing Challenge 3: One example annotated ROIs are located in the frontal, parietal, temporal multi-site database is the ADNI database. ADNI, or Alzheimer’s and occipital lobes, cingulate gyrus, insula, cerebellum, and Disease Neuroimaging Initiative, is a large-scale longitudinal brainstem. Note that, those ROIs were not used in the registra- study for better understanding and diagnosing Alzheimer’s tion process; instead, they only served as references to evaluate Disease. It contains images acquired at 57 collaborative in- the accuracy of registration (explained later in Section III-D1). stitutions or companies, all of which are publicly available. 2044 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

TABLE II REGISTRATION ALGORITHMS TO BE EVALUATED FOR THE INTER-SUBJECT REGISTRATION OF BRAIN IMAGES.THIS TABLE IS ONLY A BRIEF SUMMARY OF THEM. MORE DETAIL CAN BE FOUND IN APPENDIX A. ABBREVIATIONS:DIFF.—DIFFEOMORPHISM;MI—MUTUAL INFORMATION;SSD—SUM OF SQUARED DIFFERENCE; MSD—MEAN SQUARED DIFFERENCE;CC—CORRELATION COEFFICIENT;NCC—NORMALIZED CC

Different imaging sites used different MRI devices, imaging (Health Insurance Portability and Accountability Act), the protocols, and FOVs. Registration among ADNI subjects is public release of this database is still an ongoing effort. usually needed in the data preprocessing, or for the spatial normalization of subjects. This multi-site database has most B. Registration Algorithms Included of the challenges a multi-site database typically presents. Moreover, the ADNI protocol has now become a standard for Twelve general-purpose, publicly-available image registra- the studies of aging and neurodegenerative disorders such as tion methods were included in our study. They are summa- AD. Therefore, the performance on the ADNI database is of rized in Table II. We note that they are only a fraction of great importance for registration algorithms applied to data the large number of registration algorithms developed in the from older individuals and individuals with neurodegenerative community. The pool can always be expanded to include diseases. We randomly selected the baseline images of 10 other general-purpose algorithms (e.g., LDDMM [40], elastix ADNI subjects, including three normal controls (NC), four [41], NiftyReg [42], plastimatch [43], etc.) and brain-specific mild-cognitive-impairment (MCI), and three AD subjects. For methods (which often needs or incorporates tissue segmen- many subjects, the brain mask and hippocampus are available tation and/or preprocessing such as skull-stripping or surface at the data release website. They were used in our experiments construction, e.g., DARTEL [44], HAMMER [45], FreeSurfer as the references to evaluate the registration accuracy. Fig. 3 [46], Spherical Demons [47], etc.). In general, we chose the displays the raw intensity images and the brain/hippocampus 12 methods listed in Table II, because they represent a wide masks for three randomly-chosen ADNI subjects (1 CN, 1 variety of choices for similarity measures, deformation models MCI, and 1 AD subjects). Please note the presence of the and optimization strategies, which are the most important com- inter-subject variability in the ventricle size, sulci, gyri, etc.; ponents for registration algorithms (see Table II). Out of those the image inhomogeneity, noise and large deformation; and 12 registration methods, nine methods were included in a re- especially the FOV differences due to the image acquisition cent brain registration evaluation study [17]: flirt1 [48], fnirt2 in multiple imaging sites [e.g., the neck can be seen in (a) [49], AIR3 [50], [51], ART4 [52], ANTs5 [53], CC-FFD6 [54], (highlighted by the blue contours), but is barely seen in (b)]. SSD-FFD [54], MI-FFD [54], and Diffeomorphic Demons7 As in the previously-mentioned database, we performed all [55]. In addition, we included DRAMMS8 [56], and two regis- pair-wise registrations to avoid subject/template bias. This led tration methods that were not included in study [17]. They are: to 90 registrations for each registration method. (the nondiffeomorphic, or additive, version of) Demons [57] 4) Database Representing Challenge 4 (In Combination (with an ITK-based public software available), and DROP9 With Challenge 1): An in-house database containing eight [58] (a novel discrete optimization strategy that dramatically patients with recurrent brain tumors was used. T1-weighted increases registration speed while maintaining high accuracy). images were collected with the image size 192 256 192 For the completeness of this paper, more detail of these image and the voxel size 0.977 0.977 1.0 mm .Imagescontain registration algorithms can be found in Appendix A. And how both the cavity, caused mainly by the blood pool after the resection of the original tumor, and the recurrent tumors. We 1flirt: http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/flirt registered those pathology-bearing images into a common 2fnirt: http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/fnirt T1-weighted MR image (i.e., the template), which was col- 3AIR: http://bishopw.loni.ucla.edu/air5/ lected from a healthy subject (image size 256 256 181, 4ART: http://www.nitrc.org/projects/art/ voxel size 1.0 1.0 1.0 mm ). In this database, we have 5ANTs: http://www.picsl.upenn.edu/ANTS collected landmarks and ROIs annotated by two independent 6CC/MI/SSD-FFD: http://www.doc.ic.ac.uk/dr/~software/ experts (HA and MB). Those landmarks and ROIs served as 7(Diff.) Demons: http://www.insight-journal.org/browse/publication/154 references for measuring the registration accuracy (the criteria 8DRAMMS: http://www.cbica.upenn.edu/sbia/software/dramms to be presented in Section III-D4). Due to the HIPPA regulation 9DROP: http://www.mrf-registration.net/ OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2045 their parameters were set will be presented in the next subsec- “testing” in this paper is less of concern, since 1) the pa- tion (for the general rules) and Appendix B (for the detailed rameters of all other registration methods included in this parameter values). paper were also optimized in the LONI-LPBA40 database Note that, while including a large number of registration and other similar skull-stripped databases (IBSR, CUMU, algorithms/tools is preferable, including all available registra- MGH databases) as reported in [17]; and moreover, 2) in tion algorithms/tools seems less practical. We have included a [17], the registration methods seemed to be trained and number of the best performing methods (ANTs, ART, Demons, tested in the same exact databases (IBSR, CUMU, MGH, MI-FFD, etc.) as previously reported in [17] and several recent LONI-LPBA40), while in our study, we only trained/op- ones (DROP, DRAMMS) representative of new advancements timized the registration methods in one skull-stripped in optimization strategies and/or similarity designs. Our focus, database representing Challenge 1, and we tested all regis- though, was not only on the number of algorithms/tools being tration methods in six other unseen databases representing included in this study, but more importantly on comprehen- Challenges 1–4, respectively. sively evaluating registration methods in various tasks other • Rule 2: For each method, we fixed its parameters in all than one or two specific tasks as in many previous evaluation registration tasks. Put differently, we used the same param- studies. Doing so could provide an valuable insight to the gen- eters for a registration method, no matter it was used for erality, accuracy and robustness of registration algorithms/tools skull-stripped brain images, raw brain images, multi-site and an inspiration for future algorithm development. Some data, or tumor-recurrent brain images. It should be ad- algorithms/tools were not included. One reason was that they mitted that the optimized parameters for skull-stripped were not included in [17], and hence their best parameters were brain MR images are not necessarily optimal for raw not reported on the same training databases that other methods images or pathology-bearing images. However, using the used to optimize their parameters. We wanted to avoid the same set of parameters has two advantages: 1) most users potential bias introduced by us selecting parameters of various or algorithm-developers will start from the parameters algorithms, therefore we chose to use the optimal parameters that have already been optimized in normal-appearing, sets whenever available in [17]. Moreover, some algorithms skull-stripped images (e.g., [18], [59]–[63]); 2) it helps already had their closely-related methods included in our study. reveal the generality of registration methods and their ro- bustness levels facing various registration challenges. The For example, LDDMM is in line with ANTs but does not have second point is especially important, because a registration the symmetric design; elastix is an implementation of many method that can successfully applytoawidevarietyof transformation/similarity criteria, for which we have already registration tasks without the need for the task-specific had 12 methods in this paper to represent the variety; niftyReg parameter tuning should be desirable for the routine use in and plastimatch are based on the GPU implementation of the many large-scale pre-clinical research studies. MI-FFD algorithm, which has already been included in this Based on these two rules, we set the parameters which are study, and the GPU-implementation should be expected to disclosed in Appendix B of this paper. improve the speed but not necessarily the registration accuracy. On the other hand, the evaluation framework in this paper is D. Criteria to Measure Registration Accuracy general to include, in the future, many other popular registration Having described the databases and registration methods in algorithms/tools. the previous sub-sections, this sub-section introduces the cri- teria to evaluate registration accuracy. C. Parameter Configurations for Registration Algorithms 1) Criteria in Databases Representing Challenge 1: We measured the accuracies of inter-subject registrations in the We had the following two rules to set the parameters for each NIREP and LONI-LPBA40 databases by the Jaccard Overlap method. [64] between the deformed ROI annotations and the ROI • Rule 1: We used the optimized parameters as reported annotations in the target image space. A greater overlap often in [17] whenever applicable. Those parameters were indicates a more accurate spatial alignment. This was also the optimized by the methodology/software developers them- accuracy criterion used in many other evaluation studies such selves on skull-stripped brain image databases that are as [14], [17], [63]. Rohlfing in [65] demonstrated that, as long similar to the ones we used in this paper (specifically, they as the ROIs are localized (e.g., those (sub-)cortical structures), used four databases—IBSR, LONI-LPBA40, CUMC, which is the case in the two databases we used, the regional MGH—for training, which are similar to the databases overlap of ROIs is a faithful indicator of registration accuracy we used in this paper, which also contain skull-stripped in various locations in the image space. Mathematically, given T1-weighted images from 1.5T scanners). We can treat two regions and in a 3-D space, and given the volume of those databases in [17] as the “training” database for aregionasdefined by , the Jaccard overlap [64] the databases we used in this paper. For registration between the two regions is defined as algorithms that were not included in [17]—DRAMMS, (Additive) Demons and DROP—we took the same logic: (1) to optimize their parameters in, and only in, the task of registering skull-stripped images (the LONI-LPBA40 Some other studies used the Dice overlap [66], defined as database specifically). The fact that this LONI-LPBA40 . It should show database was also used as one of the seven databases for the same trend and should be directly linked with the Jaccard 2046 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

Fig. 5. Measuring registration accuracies in different zones. Panel (a) is the sketch of dividing the whole images into various zones. The solid contour filled with yellow texture denotes the abnormal zone (Zone 1), which contains the post-resection cavity and the recurrent tumor. Zones 2 and 3 are normal-appearing regions immediately close to, and far away from, Zone 1. Zone 4 is the whole brain boundary. The definition of the zones can be found in the main context in Section III-D4. Panel (b) shows landmark/ROI definitions for an example pair of images. Blue contours are expert-defined ROIs in Zone 1. Red crosses are expert-defined landmarks in Zone 2. Yellow crosses are expert-defined landmarks in Zone 3. Green contours are the automatically-computed brain boundaries (through Canny edge detection of the brain masks), to measure the registration accuracy in Zone 4. Please note that the landmark/ROI definitions from a second expert (which are notshownhere)maydiffer.Thisfigure is best viewed in color. overlap by . Therefore, reporting either one registration accuracy in various locations. Specifically, we de- overlap metric should be sufficient for our purpose. fined4zonesintheentire image space, as can be seen in Fig. 5. 2) Criteria in Databases Representing Challenge 2: In the Those zones were defined by the distances to the abnormal re- BrainWeb database, the annotations of 11 relatively localized gions. Therefore, they helped reflect how the existence of cavi- brain and nonbrain structures [see Fig. 2(a)] were available. ties and recurrent tumors influenced the registration accuracy in Therefore, we measured registration accuracy by the Jaccard various regions of the image. overlap between the warped ROI annotations and the target ROI • Zone 1: Abnormal region. Experts HA and MB together annotations. In the IBSR and OASIS databases, only the brain contoured the abnormal region that contains 1) the post-re- masks from raw brain images [see Fig. 2(b) and (c)] were avail- section cavity and 2) the recurred tumor. Within this con- able. Since the brain mask is not a localized structure, the Jac- tour was what we defined as Zone 1, the abnormal re- card overlap alone, according to [65], might not be sufficient gion [see Fig. 5(a)]. The experts referred to FLuid-Attenu- to represent the registration accuracy. Therefore, we used the ated Inversion-Recovery (FLAIR) MR image for this con- 95-percentile Hausdorff Distance (HD) between the warped and touring, because of its high sensitivity and specificity in the target brain masks as an additional accuracy surrogate. The delineating brain tumors [69]–[71]. Then they indepen- HD between two point sets and is defined as dently found the corresponding regions in the normal tem- plate image. We measured the accuracy of registration in Zone1bytwo metrics: 1) the average Dice overlap, and 2) the 95-th percentile Hausdorff Distance, between the algorithm-warped abnormal region and two rater-warped (2) abnormal regions in the template space. A higher regional overlap and a smaller distance reflect a better alignment of where is the Euclidean distance between the spatial lo- two images in Zone 1. cations of two points. The HD is symmetric to two input im- • Zone 2: Regions immediately neighboring the ab- ages, with a smaller value indicating a better alignment of brain normal region. A 30 mm-wide band immediately outside boundaries. We used the 95th percentile other than the max- the abnormal region was defined, by morphologically di- imum HD, to avoid the influence of outliers, as suggested in lating the abnormal region mask agreed by the two experts [67] and [68]. in the patient’s image space [see Fig. 5(a)]. Anatomical 3) Criteria in Databases Representing Challenge 3: Since landmarks were identified in this band, which served as the annotations of both the brain mask and the left and right the references to reflect the registration accuracy in the im- hippocampi were available in the ADNI database, we used the mediate neighborhood of abnormalities. One expert (HA) Jaccard overlap between the warped and target ROIs to indicate labeled 10 anatomical landmarks in Zone 2 within the the registration accuracy in this multi-site database—a higher patient image. The two experts then independently labeled overlap usually means a better spatial alignment of two images. the corresponding anatomical landmarks in the template 4) Criteria in Databases Representing Challenge 4: The space. The average Euclidean distance between the algo- landmark and ROI annotations from two independent experts rithm-calculated corresponding landmark locations and in the in-house brain tumor database enabled us to measure the the rater-labeled corresponding landmark locations in the OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2047

IV. RESULTS AND ANALYSIS In this section, we use four subsections to present the eval- uation results in the databases representing the four aforemen- tioned challenges.

A. Results in Databases Representing Challenge 1 Fig. 7 shows the average Jaccard overlap over all ROIs in the NIREP and LONI-LPBA40 databases. A detailed table of Jaccard overlap per ROI can be found in Appendix C for all algorithms evaluated in this paper. Several observations can be made from this set of results. Fig. 6. Depiction of inter-expert and algorithm versus expert landmark errors. 1) DRAMMS and ANTs obtained the highest Jaccard overlaps in both databases. Between the two methods, DRAMMS had a slightly higher accuracy in the NIREP template space was used to measure the registration accu- database ( for ANTs and racy in Zone 2. Smaller landmark errors point to higher for DRAMMS, ); whereas registration accuracy. The concept is depicted in Fig. 6. ANTs had a slightly higher accuracy in the LONI-LPBA40 Given a set of expert-annotated landmarks in database ( for ANTs and the patient image, their corresponding landmark loca- for DRAMMS, ). In both tions (independently by expert HA) and cases, the differences were tiny ( difference be- (independently by expert MB) in the tem- tween the average Jaccard overlaps in the 0–1 scale). plate image, and the algorithm-calculated corresponding Following ANTs/DRAMMS were DROP, Demons and landmark locations also in the template ART registration methods. Among these methods, ANTs image, we defined the inter-expert landmark errors (the and ART were included in the evaluation study [17] and length of the solid line in Fig. 6) as were found to be the two most accurate methods. Our findings here showed a similar trend. In addition, the three (3) methods—DRAMMS, DROP and (the nondiffeomorphic, and the algorithm-to-expert landmark errors (the average or additive, version of) Demons, which were not included length of the dashed lines in Fig. 6) as in [17], showed highly competitive performances. 2) Methods such as SSD-FFD, fnirt, DROP use intensity differences (SSD) as the similarity metric. On average, they had reasonable Jaccard overlaps. However, they (4) also showed larger variations of regional overlaps, and where is the Euclidean distance between two voxel therefore they were less stable in our experiments than locations. those methods using CC, MI, or attribute-based similarity • Zone 3: Regions far away from the abnormal region. measures. Zone 3 was defined as all the normal regions outside Zone 2 3) The high degree of freedom allowed by a deformation [see Fig. 5(a)]. We used landmarks to evaluate the registra- mechanism (such as the FFD model as used in DRAMMS tion accuracy in Zone 3. This could show how the recurrent and the diffeomorphism LDDMM model as used in ANTs) tumor and the cavities have influenced registration in far- is perhaps another factor contributing to the higher reg- away normal-appearing regions. One expert (HA) labeled istration accuracy, compared to deformation mechanisms 40 anatomical landmarks in Zone 3. Then two experts inde- with relatively few degrees of freedom (such as fifth-order pendently labeled corresponding landmarks in the template polynomials in AIR). space. Registration accuracy in this zone was measured by B. Results in Databases Representing Challenge 2 the average Euclidean distance between the algorithm-cal- culated corresponding landmarks and the rater-labeled cor- For the three databases containing raw images, we had two responding landmarks in the template space. Smaller land- scenarios—one focusing on the localized structures and tissue mark errors point to higher registration accuracy in Zone 3. types throughout the image space (the BrainWeb database); • Zone 4: Brain boundaries. The existence of cavities and and the second scenario focusing on the brain masks (the IBSR recurrent tumors inside the patient image may even influ- and OASIS databases), which is crucial for (multi-)atlas-based ence the alignment of the brain boundaries between the pa- skull-stripping. tient and the normal-appearing template images. To cap- Scenario 1. Registration Accuracy in Multiple Localized ture this influence, we measured the dice overlap and the ROIs/Structures in the Raw Images: Fig. 8 shows the average 95th percentile Hausdorff Distance between the warped Jaccard overlaps of the 11 ROIs in the BrainWeb database. and the template brain masks. A higher dice overlap and Several observations can be made. smaller distance indicate a higher level of robustness of a 1) DRAMMS, DROP and Demons obtained similarly high registration algorithm with regard to the abnormality-in- accuracies, followed closely by the ANTs and ART duced negative impact. methods. These registration methods were also among 2048 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

Fig. 7. Box-and-Whisker plots of registration accuracies in the NIREP and LONI-LPBA40 databases, as indicated by the Jaccard overlaps averaged across 32 (in NIREP) or 56 (in LONI-LPBA40) ROIs. This figure shows how registration methods perform facing Challenge 1 (inter-subject variability).

Fig. 8. Box-and-Whisker plots of registration accuracy in the BrainWeb database, as indicated by the Jaccard overlaps averaged across 11 available ROIs. This is Scenario 1 in the testing of registration methods facing Challenge 2 (intensity inhomogeneity, noise and structural differences in raw images).

the most accurate ones in registering skull-stripped brain 2) On the other hand, the average Jaccard overlap in var- images as shown in the previous sub-section. ious ROIs by the best-performing algorithm in this raw, OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2049

Fig. 9. Registration accuracy in raw brain images, in the IBSR and OASIS databases, as indicated by the Jaccard overlap (the first row) and 95th percentile Hausdorff Distance (the second row), between the warped and the target brain masks. This is Scenario 2 in the testing of registration methods facing Challenge 2 (intensity inhomogeneity, noise and structural differences in raw images). “prctile” in the title of the second subfigure means “percentile”.

with-skull database (i.e., DRAMMS) was only around 0.32 difficulty for registration in a database. In the OASIS data- (Fig. 8). Compared to the average Jaccard of 0.52–0.57 in base, which exhibits a lower level of inter-subject FOV registering skull-stripped brain images (Fig. 7), this clearly differences and intensity inhomogeneity [as can be seen underlined the increased level of difficultyinregistering in Fig. 2(c)], ANTs scored the highest accuracy, followed raw, with-skull brain images. closely by Demons and DRAMMS. In the IBSR database, Scenario 2. Registration Accuracy in Brain Masks of the Raw however, which exhibits a higher level of inhomogeneity, Images: In this registration scenario, we focused on the accu- background noise and larger deformations, DRAMMS racy of registration in warping the brain masks, which is the scored the highest accuracy, followed, with relatively basis for the atlas-based skull-stripping framework (e.g., [19], bigger distances, by Demons and ANTs. [25]). Fig. 9 shows the accuracies of three registration methods 3) Overall, a Jaccard overlap of 0.75–0.93 could be ex- (ANTs, Demons and DRAMMS), which were forerunners in the pected for the brain masks when we registered raw, results in Scenario 1. Several observations can be made. with-skull images within a same database. This should 1) The Jaccard regional overlap and the 95th percentile provide a promising starting point for (multi-)atlas-based Hausdorff Distance showed the same trend for aligning skull-stripping (e.g., [19], [25]). On the other hand, one the brain masks—a higher Jaccard overlap corresponded needs to be aware that the registration accuracy might de- to a smaller distance; crease when images are from multi-site databases, usually 2) The ranking and the difference of methods seemed to be with larger imaging and FOV differences (to be shown in highly dependent on the database, especially the level of the next subsection). 2050 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

Fig. 10. Jaccard overlaps in the ADNI database, for a) the brain mask (the left three columns); b) the left hippocampus (the middle three columns); and c)the right hippocampus (the right three columns). This figure shows how registration methods perform in a typical multi-site database, where additional challenges arise from the imaging and FOV differences in different imaging institutions (Challenge 3).

C. Results in the Database Representing Challenge 3 ventricle sizes, atrophy patterns, and hippocampus sizes (see Fig. 11 for some examples of very difficult cases, Fig. 10 shows the overlap results averaged over all 90 pair- which will be described in item 4 below). wise inter-subject registrations in the ADNI database. Com- 3) Considering the quantitative results in all three regions pared to the registration within the single-site database, the reg- (brain mask, left, and right hippocampi) in those with-skull istration of raw images acquired from multiple imaging sites raw images acquired from multiple institutions, DRAMMS encountered an increased level of difficulty. Specifically, three showed the greatest promise. observations can be made. 4) Besides the quantitative results in the limited number of 1) DRAMMS and ANTs performed better than Demons in structures such as the brain masks and the hippocampi, the aligning the brain masks. They showed higher levels of visual inspection of the registration results in the whole robustness with regard to the FOV differences, the back- images could actually reveal much greater differences ground noise and the presence of skull or other nonbrain among registration methods. Fig. 11 shows some regis- structures. tration results from Demons, ANTs and DRAMMS in 2) The presence of the skull, the background noise, and two pairs of subjects from the multi-site ADNI database. especially the FOV difference, in the ADNI multi-site As pointed out by blue arrows in the figure, DRAMMS database, had a clearly visible impact on the accuracy showed a clear advantage to align the largely different of aligning deep brain structures. When registering anatomies such as the ventricles. To capture this large skull-stripped images from single-site databases such difference, many algorithms may have to increase their as in the LONI-LPBA40 database, ANTs, Demons and search ranges. However, this usually requires consider- DRAMMS could align hippocampi at Jaccard overlaps able efforts for task-specific, or even individual-specific, around 0.6, and they differed by less than 0.05 Jaccard parameter adjustments. Adjusting search ranges is a non- overlap on average (see Table V in Appendix C). How- trivial research topic. It often requires specifictheoretic ever, when the raw images from the multi-site ADNI designs [73], or requires the introduction of anatomical database were used, even the best-performing algorithms landmarks [74]–[80]. The main difficulty is to effec- (DRAMMS and Demons as shown in Fig. 10) could tively balance between capturing the large differences only align hippocampi at Jaccard overlaps around 0.5 on and capturing the local subtle displacements. Therefore, average, and algorithms had greater differences in terms algorithms that can capture and balance between both, and of the Jaccard overlaps they obtained. Another factor that do not require additional parameter adjustments, become caused the decrease in the accuracy and the increase in the favorable. Actually, in our recent studies that spatially nor- differences among methods could be that those subjects malized all ADNI subjects into a common template in the ADNI database have highly variable degrees of (from a normal control subject), a small portion of the neuro-degeneration (three normal controls, four MCI, and Alzheimer’s Disease (AD) subjects have unusually large three AD subjects), and hence they have largely different ventricles than many other AD subjects. We considered a OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2051

Fig. 11. Demons, ANTs and DRAMMS registration results of two pairs of images having large anatomical variations especially in ventricles, mainly due to their different levels of neuro-degeneration. All subjects are from the multi-site ADNI database. Blue arrows point to some typical locations where registration results from three methods differ greatly.

registration “failure” if there were more than 5 mm errors Compared to the 80%–85% success rate by ANTs and in the ventricle boundaries (visually pronounced errors). Demons when used with the fixed sets of parameters, Accordingly, the success rate was defined by DRAMMS, also using a fixed set of parameters as in other cases, achieved a success rate of 96%, which was a clear # improvement of registration accuracy in this large-scale # multi-institutional database. 2052 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

Fig. 12. Landmark errors or the 95th percentile Hausdorff Distance in various zones in the pathology-to-normal subject registrations. In addition to the errors, we have shown the average Jaccard overlap in Zone 1 in this figure. This figure shows how registration methods perform in the presence of pathology-induced missing correspondences (Challenge 4).

D. Results in Database Representing Challenge 4 respectively), and also comparable to the inter-expert errors. Fig. 12 shows the quantitative registration errors in the 3) Further away from the abnormal regions, Zone 3 had pathology-to-normal subject registration. As a reference, this larger landmark errors than Zone 2. The average errors figure also includes inter-expert errors between the two inde- pendent experts in Zones 1–3. A desirable registration should were 5.3 mm between experts, 5.8 mm for DRAMMS, and 1) accurately register the normal-appearing regions, where 5.9 mm for ANTs. Diffeomorphic Demons and especially correspondences can be established (i.e., small landmark er- fnirt started to have larger landmark errors (6.6 and 9.6 rors in Zone 2–3); 2) accurately align the brain boundaries mm on average). This difference may, in part, be attributed (i.e., small 95%-percentile HD distances in Zone 4); and 3) to the fact that DRAMMS used texture features and ANTs map the pathological regions to the right location but relax used correlation coefficient as similarity metric, which the deformation within the pathological regions, where cor- were perhaps more robust and reliable than the intensity respondences could hardly be established (i.e., high regional difference which Demons and fnirt used as their similarity overlaps in Zone 1). In Fig. 12, several observations can be metrics. made. 4) Another interesting finding was in Zone 4 (brain 1) Two independent experts agreed with each other only at a boundary). Because of the sharp contrast between fore- 0.48 Jaccard overlap on average in the cavity and tumor ground and background in a skull-stripped image, the recurrence regions (Zone 1). This reflected the difficulty, brain boundaries ought to be among the easiest parts or the ambiguity, for the human experts to deal with the to register. In the presence of pathologies, however, missing correspondences. Among the four methods eval- this was surprisingly not always the case. Fnirt, for uated, ANTs agreed with experts at the highest level (av- example, had 11.1 mm as the 95th percentile Hausdorff erage Jaccard 0.40), followed closely by DRAMMS (av- Distance at brain boundaries, which meant registration erage 0.37 Jaccard overlap). The 95th percentile Hausdorff failures in several cases. ANTs had, on average, 3.3 Distances showed the same trend. Overall, experts showed mm as the 95th percentile Hausdorff Distance in the better agreement between each other than between algo- brain boundary, which was even bigger than the average rithms and experts. errors ANTs produced in the abnormal regions (2.1 2) In Zone 2 (the immediate neighborhood of the abnormal mm). By carefully examining the output images, we regions), the average landmark error was 4.1 mm between found that this average boundary error by ANTs was experts. Landmarks errors for DRAMMS, ANTs and Dif- mainly caused by misalignments in the boundaries close feomorphic Demons were similar (at 3.9, 4.1, and 4.4 mm, to the pathology sites in several cases. This showed OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2053

Fig. 13. Registration of a brain image with tumor recurrence to a normal brain template by DRAMMS, for a series of slices in the coronal view. This figures shows how the mutual-saliency mechanism (a spatial-varying utilization of voxels) helped DRAMMS in the pathological-to-normal subject registration scenario. Without segmentation, initialization, or prior knowledge, the automatically-calculated mutual-saliency map (d), defined in the target image space, effectively assigned low weights to those regions that correspond to those outlier regions (pointed out by arrows) in the source image (a). This way, the negative impact of outlier regions could be largely reduced; registration was mainly driven by regions that could establish good correspondences. Red arrows point to the post-surgery cavity regions. Blue arrows point to the recurrent tumors.

that the pathology regions may impact a wide area in needed to segment and specifically deal with the pathology-af- ANTs registration. On the other hand, DRAMMS and fected regions. However, the fact that DRAMMS, as a gen- Diffeomorphic Demons had the smallest boundary errors eral-purpose registration algorithm, performed stably and ro- (1.8 and 2.2 mm, respectively). Both errors were smaller bustly in all Zones 1–4, highlighted the effect of using at- than those in the abnormal regions, indicating good tributes to measure voxel similarities and to quantify voxel- alignments of the brain boundaries. This showed that wise matching reliabilities. To better illustrate this, Fig. 13 the negative effect of pathological regions was more shows a set of representative DRAMMS registration results localized in DRAMMS and Diffeomorphic Demons (registration from a tumor-recurring patient’s brain image to registration algorithms, which should be desirable. the normal-appearing brain template image). First, DRAMMS It should be emphasized that general-purpose registration extracted high-dimensional Gabor texture attributes to repre- algorithms are usually not designed for registering pathology- sent each voxel. The attributes should be more informative bearing images. Task-specific registration algorithms are often than intensities in the search for correspondences. Moreover, 2054 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014 at each voxel, DRAMMS automatically calculated a so-called B. Understanding the Differences Among Registration “mutual-saliency” weight, also based on the attributes. The Algorithms mutual-saliency quantified the chance of each and every voxel to establish a reliable correspondence between the two im- Registration algorithms differ in similarity metrics, transfor- ages. As Fig. 13(d) shows, the mutual-saliency map effec- mation models, and the optimization strategies. Table II sum- tively identified outlier regions (dark blue), where correspon- marized the registration methods included in this paper, and dences could be hardly established. The identified outlier more details can be found in Appendix A. Such differences are regions coincided with the recurrent tumor regions [as red likely the major factors for their different performances in this arrows pointed out in panel (a)]. Note that, this was obtained paper. without any segmentation, manual masking, or any prior In terms of similarity metrics, 11 out of 12 methods in- knowledge of the presence or the location of the tumor recur- cluded in this paper measure the image similarity based on the rence. Being segmentation-free is a feature that differentiates gray scale intensities or intensity distributions. DRAMMS, on DRAMMS from those task-specific cost-function-masking ap- the other hand, measures the image similarity by a rich set of proaches or pathology-seeding approaches. As a result of this multi-scale and multi-orientation Gabor attributes. Intensities attribute-based similarity measurement and mutual-saliency alone may not necessarily carry anatomical or geometric weighting, the registration by DRAMMS was mainly driven information of voxels. That is, voxels having similar or by the regions where correspondences could be well estab- even identical intensities may belong to different anatom- lished. This led to visually plausible results as shown in ical structures. Consequently, a common challenge in inten- Fig. 13(c). sity-based similarity metrics is how to effectively deal with matching ambiguities. Methods such as ANTs measure the similarity of two voxels by the correlation coefficient of in- V. D ISCUSSION AND CONCLUSION tensities in local patches centered at those two voxels. The In this last section, we first summarize the work and findings local patches carry, to some extent, the local texture or geo- in Section V-A. Then, Section V-B discusses the theoretical metric information. Therefore, in our experiments they were differences among the registration algorithms included in relatively more robust to noise, partial volume effects and this paper, which may, at least partly, explain the different magnetic field inhomogeneities, compared to measuring the performances among registration methods in our experiments. voxel-wise similarity using intensities alone. Attribute-based Section V-C discusses the limitations of the whole evaluation methods such as DRAMMS extend this to the explicit char- work and the future directions. Finally, Section V-D concludes acterization of voxels by the high-dimensional, often more this paper. informative, texture or geometric attributes. This could reduce matching ambiguities, but at the cost of an increased compu- A. Summary of Work and Findings tational burden. This observation has been documented in the This study evaluated several registration algorithms and their literature by several research groups (e.g., [45], [81]–[83]). publicly-available software tools. Our evaluation had the fea- The generality and accuracy of DRAMMS in our exper- tures summarized below. iments, especially its performances in raw, multi-site and First, compared to existing studies that evaluated registra- pathology-bearing images, provided new evidence for using tion algorithms in specific tasks and/or databases, our study attributes to measure image similarities. On the other hand, utilized multiple databases to represent a wide range of chal- there is also ongoing research on extending intensity-based lenges for the inter-subject registration. As Table I showed, the similarity metrics (CC or MI) into more robust measures to databases we included in this evaluation work covered a va- reduce matching ambiguity for mono- and multi-modality reg- riety of imaging scanners (GE, Siemens, Philips), field strengths istration [22], [84], [85]. (1.5T, 3T), age groups, imaging FOVs, and imaging protocols Another related issue is how image voxels are used when (varying pulse sequence parameters). The purpose was to exten- calculating the similarity between two images. General- sively evaluate the generality, robustness and accuracy of regis- purpose registration methods, such as most of the ones in- tration algorithms. cluded in our study, often use all voxels equally to define the Second, our study found out that, in general, registration al- image similarity. On the other hand, DRAMMS introduced gorithms differed greatly in terms of their performances, when the notion of “mutual-saliency.” The central idea was to use facing different databases or challenges. For skull-stripped im- all voxels, but at different levels of confidence as measured by ages included in our study, ANTs and DRAMMS led to the the mutual-saliency metric. Specifically, those voxels having highest overlaps of expert-annotated (sub-)cortical structures, higher confidence to establish reliable correspondences were followed by ART, Demons, DROP, and FFD. Whereas for more associated with higher mutual-saliencies (e.g., Fig. 13), and challenging tasks in databases containing raw, multi-site and they were accordingly used with higher weights in calculating pathology-bearing images, the attribute-based DRAMMS algo- the image similarity. They were the main driving force for the rithm obtained relatively more stable and higher accuracies, fol- registration. An immediate advantage was in the registration lowed closely by the intensity-based and symmetric ANTs reg- of pathology-bearing images such as shown in Fig. 13. istration algorithm. Without prior knowledge for tumor presence, or any prior OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2055 tumor segmentation, DRAMMS examined voxels one by a 16 GB memory. It should take another controlled study to one and attached with each one of them a “mutual-saliency” further investigate the impact of the optimization strategies number that reflected its ability to find correspondences. This on registration accuracies. One thing to note is that many way, the mutual-saliency map in Fig. 13(d) automatically registration methods are able to be parallelized into GPU ac- and effectively found out a temporal lobe region that had celerations [93]–[95]. difficulty to establish correspondences, and the location of this region agreed with that of the abnormal regions. By this, C. Limitations of Our Evaluation Study and Future Work the deformation within the abnormal region was relaxed; the We also note some drawbacks of this study, and hence our other normal-appearing regions were matched well, which future work. drove the registration of the whole image. The idea of First, like many other studies [11], [12], [16], [83], [96]–[98], spatially-varying treatment of voxels has also been adopted this study was also conducted by the authors of one of the algo- in other registration approaches (e.g., [86]–[88]), showing rithms to be evaluated. Questions may naturally arise for the great promise in many challenging registration problems reproducibility and fairness in such comparative evaluations. involving, for example, topology-changing tumor changes, We tried to address these questions when designing and con- pathology-induced outliers, and cardiac/lung motion-induced ducting this study: 1) for reproducibility, we fully disclosed subtle changes. d In terms of the transformation models, the ones with more the parameters, used public databases whenever possible, an degrees of freedom typically led to higher registration ac- constrained our evaluation within publicly-available registra- curacies in our experiments. For instance, the geometric tion algorithms; 2) for fairness, we used the parameters sug- cubic B-spline-based FFD transformation model as used in gested by the authors of the registration methods as reported CC/MI/SSD-FFD, DRAMMS and DROP, and the velocity by [17], and all of those parameters were “optimal” inthereg- fields used in Demons and ANTs, could perhaps explain istration of skull-stripped and preprocessed brainMRimages. their relatively higher registration accuracies than other less Not all self-conducted evaluation studies in the literature had flexible transformation models (e.g., the fifth polynomial as these features, but we felt that it was really important to comply used in AIR). The symmetric feature as introduced in ANTs with these high standards in our study. Our future plan includes seemed to at least partly contribute to its accuracy and ro- the participation in third-party-organized challenges, such as in bustness. Specifically, in the pathological-to-normal subject [17], [99], and [100]. registration, where two images differ greatly, ANTs had high The second point worth discussing in our study is that accuracies in abnormal regions and in the immediately neigh- we fixed the parameters for each registration method. Two boring normal regions. This was perhaps due to the symmetric questions may arise: whether we should fixparameters and setting, which constrained both images to deform towards the at which values we should fix the registration parameters. “hidden middle template” between the two images. This way, For the first question, we fixed registration parameters be- adifficult inter-subject registration problem was decoupled cause the aim was to test whether some methods can be into two relatively simpler subproblems. Such a symmetric used in many large-scale, often multi-institutional, transla- setting is also advocated in many other approaches such as in tional studies. This was a high standard and quite an ambi- the linear registration [89] and deformable registration [90], tious aim that did not appear in previous evaluation studies. [91]. Furthermore, the diffeomorphic setting in ANTs and Facing many registration tasks and many databases, normal Diffeomorphic Demons also contributed to the accuracy and or pathological, single- or multi-site, in daily translational re- robustness, since the regularization of the transformation in search, it is usually less practical to tune parameters for each the diffeomorphisms seemed to account for the real-world specific task/database. Therefore, we fixed parameter values anatomical deformations. in our study. We note that this does not necessarily reflect When it comes to the optimization strategies, methods in- a registration method’s stability or sensitivity with regard to cluded in this paper used optimizers either in the discrete the parameter changes. Stability is another preferable feature space (DRAMMS, DROP) or in the continuous space (all of registration methods. To better reveal stability or sensi- others). The discrete optimization helped to reduce the com- tivity, a future study is needed to examine how registration putational time to 3–5 min in DROP [58], [92], compared to algorithms perform over a wide range of parameter values. 10–20 min in MI/CC/SSD-FFD, which have the same simi- That is, to thoroughly investigate how registration accuracy larity metric and transformation model. Their accuracies were comparable in our experiments. Another interesting compar- changes when the values of registration parameters change. ison in our experiments was the 30–50 min computational time The difficulty lies in the determination of effective, often of DRAMMS, which used a discrete optimizer on high-di- multivariate, parameter ranges, and the objective comparison mensional attribute-based similarities, versus about 1–1.5 h of parameters (or parameter ranges) among different registra- for ANTs, which used a continuous optimizer on patch-based tion algorithms. For the second question—at which values we correlation coefficient similarities. Both computational times should fixtheparameters, the parameter values we used in were for a pair-wise registration of some typical brain images this paper were those optimal for one typical registration task. (e.g., image size 256 256 200), and on a Linux opera- They were the values suggested by authors of the algorithms tion system with an Intel Xeon x5667 3.06-GHz CPU and themselves, and hence most likely to be adopted by ordinary 2056 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014 users, or to be tried in the first pass by algorithm developers. text of registration smoothness, and evaluated both properties. We note that they are not necessarily optimal for the other Debates exist, though, for two reasons. First, whether the three tasks, nor necessarily optimal for the four tasks alto- emphasis should be on the accuracy or on the smoothness gether. In the future, a more complete study may be needed is usually more dependent on the specific application. For to “learn” the (range of) parameter values that are best overall instance, atlas-based segmentation frameworks may need a in the four tasks included in this paper. Highly sophisticated more aggressive registration to obtain higher structural over- learning framework needs to be designed, such as [21] and its laps; whereas on the other hand, population studies using extensions. It will require a larger data size for training and voxel-wise statistics may need a smoother registration to better testing, and the familiarization of the implementation details balance between the difference among individuals and the in each registration algorithm. commonality within a population. Second, while the criteria In our experiments, we used fewer images and fewer for registration accuracy can be indicated on landmark errors databases to represent Challenges 3–4 than Challenges 1–2. or regional overlaps, the criteria for registration smoothness Specifically, only 10 ADNI subjects (three AD, four MCI, is relatively loosely defined. Some studies used Jacobian de- three NC), or 90 pair-wise registrations, were used to test terminants [99], [101]—negative Jacobian determinants indi- registration methods against challenges arising in multi-site cating self-folding (i.e., nondiffeomorphism) should be penal- databases; and only eight patients with recurrent tumors were ized, as human organs deform smoothly. In the existence of used to test registration methods against pathology-induced cross-individual anatomical differences, whether the deforma- missing correspondences. Because of the relatively small tion should be strictly diffeomorphism remains a topic of de- data size, a decisive conclusion on how registration methods bate. Plus, the computation of Jacobian determinants requires perform facing Challenges 3–4 may need to be deferred to a numerical approximation of the continuum from the discrete future larger-scale studies. What we want to emphasize is that, image space, and usually varies by different software pack- although small in data size, those experiments were among ages. Consequently, other studies (e.g., [14]–[16], [89], [102]) the very first ones appearing in the literature to evaluate gen- used additional metrics such as the transitivity and the inverse eral-purpose registration methods in multi-site data and in consistency to measure the smoothness and diffeomorphism of pathological data. Therefore, those experiments could serve as deformation fields. Nevertheless, the wide adoption of those a proof of concept that some registration algorithms may bear metrics needs more studies, and the balance between accuracy the potential to work reasonably well in those difficult cases. A and smoothness seems a task-specific choice. This is another future study with a larger data size is needed in this direction. reason that our future studies need to thoroughly examine a Acquiring multi-site or pathological data, especially acquiring range of parameter values to look at how registration accuracy expert annotations of landmarks and/or ROIs on those data, is and smoothness change as parameter values change, so that in itself a nontrivial problem. users may make a more informed choice about the algorithms, The same as in studies [10], [11], [14]–[18], [98], expert-de- their implementation tools and a proper set of parameters for fined annotations of ROIs or landmarks served as references problems at hand. for the evaluation of registration accuracy in our study. One Besides registration accuracy or smoothness, a perhaps more thing to note is that expert annotations may be subject to important task for our future study is the utility of registra- intra-/inter-expert variability, and may have a certain level of tion methods in various clinical applications. For example, in uncertainty or even errors. Therefore, a perfect and completely multi-atlas-based segmentation propagation, the most accurate error-free registration algorithm may still present some minor registration for single-atlas registration may not necessarily errors in the current criteria to assess registration accuracy, achieve the highest overall segmentation accuracy. Choosing because of the uncertainty in the expert annotation of land- a proper registration methods or a proper set of registration marks or ROIs. Quantifying such uncertainty is another topic parameters in this case is subject to other interleaved factors for future studies. To reduce the influence by the variability such as the label fusion. Another example is in the detection or uncertainty of expert annotations, we either used two inde- of atrophy or growth patterns, which is usually an important pendent experts in some databases, or used multiple databases component in the study of neuro-degenerations [103]–[108] for a specific task, where different databases were annotated or neuro-development [109]–[112]. There, the most accurate by different experts to reduce the chances of systematic uncer- or the most smooth registration may not be necessarily the tainties or errors. optimal choice to decipher the subtle patterns [113]. Sim- In our study, we only evaluated the accuracy of registra- ilar situations also occur for the brain extraction (e.g., [19]), tion, which was what most previous studies did [12], [17], the quantification of longitudinal disease change (e.g., [83], [96]–[98]. Some recent studies (e.g., [16], [99]) started to [114]), and in cognitive neuroscience (e.g., [115]). Therefore, look at more comprehensive criteria including the registra- much interest is in putting registration into the big picture of tion smoothness. The argument was that, many registration end (pre-)clinical goals and considering its interactions with algorithms might achieve a fairly high structural overlaps at other factors. very aggressive underlying deformations. Therefore, studies To further improve registration accuracy, one interesting like [16], [99] started to put registration accuracy in the con- topic is to utilize multiple registration methods in a meta- OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2057 analysis framework, where one method may underperform Our future work will include the participations in third- or fail in a task/database/individual, but others may make party-organized grand challenges, the further investigation of up the loss. Several newly published articles supported this registration performances with regard to changes of varying pa- consensus process, or the so-called meta-registration process, rameter values, and a more extensive evaluation by including such as [72], [116]–[118]. Doshi et al. recently combined more data, more registration methods, more comprehensive ANTs and DRAMMS in a multi-atlas labeling framework, evaluation criteria, and especially more clinical-oriented ap- resulting in a top-performing method in a MICCAI seg- plications. Another very interesting direction is to develop a mentation challenge [72]. Muenzing et al. recently combined meta-registration paradigm, where registration methods com- ANTs, NiftyReg and DROP and showed a significant reduc- plement rather than compete with each other. This has the tion of registration errors in pulmonary images [118]. These potential to bring accuracy, robustness, and generality to a new studies further underline the potentially synergistic value of and higher level. various registration methods. Another interesting avenue for improving registration accuracy, especially when registration APPENDIX A algorithms face several challenges from large anatomical vari- SUMMARY OF THE 12 PUBLICLY-AVA IL AB LE METHODS abilities or imaging device differences (e.g., 2-D ultrasound to INCLUDED IN THIS EVAL UAT IO N 3-D MRI), is to better initialize general-purpose registration algorithms with anatomic or geometric landmarks. Several re- Appendix A supplements Section III-B in providing more de- cent studies are in this direction, on how to accurately and tail for the 12 publicly-available registration methods included automatically extracting useful landmarks [77], [79], [119], in this evaluation study. and how to effectively combine landmarks and voxel-wise • flirt. An affine transformation assumes that all voxels in informationinanelegantandpractical framework [76], [92], the image move together with 12 degrees of freedom (dof) [120]–[123]. in a 3-D space. This includes three dof for scaling, three dof for translation, three dof for rotation, and three dof for D. Conclusion shearing. The publicly-available FMRIB’s Linear Image Registration Tool (flirt) [48] from FSL package is perhaps In conclusion, this paper conducted a comparative eval- the most cited work for affine registration, as evidenced uation of 12 publicly-available and general-purpose image by more than 1500 citations in the Google Scholar search registration methods, in the context of inter-subject registra- engine since its publication in 2001 (as of March 2013). The tion of brain MR images. In contrast to existing works that main contribution of flirt is in the optimization strategy. evaluated registration methods in a specific task or a specific A global, multi-start and multi-resolution optimization database, the emphasis of this work was on evaluating the strategy specifically for affine image registration problems generality, accuracy and robustness of registration methods in was proposed. The fundamental idea is to combine a fast various inter-subject brain MR image registration tasks with local optimization (Powell’s conjugate gradient descent various challenges involving multiple databases containing method [124]) with an initial search phase. The flirt tool skull-stripped images, noisy and raw images, multi-site data, supports a variety of intensity-based similarity metrics, and pathology-bearing images. Our experiments showed that including Least Square Intensity Difference (LS), (Normal- DRAMMS and ANTs gained relatively higher accuracy in ized) Correlation Coefficient ((N)CC) and (Normalized) skull-stripped images, followed by ART, Demons, DROP and Mutual Information ((N)MI). The flirt tool is publicly avail- FFD. DRAMMS showed a relatively higher accuracy when able at http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FLIRT. raw, multi-site or pathology-bearing data were involved. The • AIR. The Automatic Image Registration (AIR) registra- main reasons might be 1) DRAMMS measures the simi- tion tool was developed by researchers (Woods et al.)at larity between voxels by a rich set of texture attributes the University of California, Los Angeles (UCLA) in the while other methods by intensities; and 2) DRAMMS has 1990s [50], [51]. AIR makes contributions in the deforma- the mutual-saliency mechanism to automatically quantify the tion models. It models the deformation by second-, third-, matching reliability of voxels, and to use those highly reliable fourth- and fifth-order nonlinear polynomials (30, 60, 105, matching to drive the registration, while other methods use and 168 deformation parameters, respectively). To find the voxels equally. The fact that ANTs, ART, Demons, DROP, and optimal values for the deformation parameters, AIR min- SSD/MI-FFD performed relatively accurately among all inten- imizes a cost function between the deformed image and sity-based algorithms highlight the importance of 1) choosing the target image. Three different cost functions are sup- image similarity metrics, such as correlation coefficient and ported in AIR. One is the ratio image uniformity (RIU). mutual information, that are relatively more robust with re- The ratio of the intensity in the deformed image to the in- gard to the noise, intensity inhomogeneity, and partial volume tensity in the target image is computed at each voxel lo- effects, and 2) choosing proper transformation models that are cation. The deformed and the target images are assumed of sufficient degrees of freedom but well regularized by the to be aligned when the ratio is homogeneous (i.e., high diffeomorphism or even the symmetric design with regard to mean value and low standard deviation) in the entire target the two input images. image space. The second cost function is Sum of Square 2058 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

Difference (SSD) of image intensities. The SSD cost func- at each voxel by searching a most similar voxel in the tion assumes that the same anatomical structure in dif- other image, according to whichever similarity metric ferent images share the same intensity. Therefore, two im- users choose (by default correlation coefficient). Then the ages are aligned when the differences in their intensities velocity fieldissmoothedbyGaussianfilters before incor- are minimized. The third cost function is a variant of the porated into the total deformation. The ANTs registration second cost function, with some relaxation. Instead of as- tool is publicly available at http://stnava.github.io/ANTs/. suming that the same anatomical structure shares the same • Demons and Diffeomorphic Demons. The Demons intensity in different images, the third cost function as- algorithm [57], [128] considers deformable image regis- sumes the same anatomical structure shares the same in- tration as a nonparametric diffusion process. It introduces tensity with a global scaling factor. Therefore, it is a glob- “demons” to push voxels to their correspondences ac- ally-weighted SSD cost function. To minimize the cost cording to the local intensity characterizations. Using function, a numerical optimizer based on either Newton’s intensity difference as the similarity metric, a force is method [125] or Levenberg–Marquardt method [126] is computed from the optical flow equations to push voxels used. The AIR registration tool is publicly available at by some velocity that is iteratively addedtothetotaldis- http://bishopw.loni.ucla.edu/air5/. placement (initially zero). The total displacement is then • ART. The Automatic Registration Toolbox (ART) was smoothed with a Gaussian filter serving as regularizations developed by researchers (Ardekani et al.)attheNathan of the deformation. In its original version [57], the velocity Kline Institute for Psychiatric Research and in the New field is simply added to the current deformation in each York University in 2005 [52]. It makes contributions in iteration, which may cause self-foldings in the deformation defining a new similarity metric. In the ART framework, field and is hence not diffeomorphic. To solve this problem, each voxel is characterized by a high-dimensional feature the Diffeomorphic Demons was developed in [128], by vector, which is constructed by stacking the intensity composing the deformation with the exponential of the values of all the voxels in the neighborhood. Compared velocity field. The Diffeomorphic Demons version has with most voxel-wise methods which establish correspon- been shown to produce much smoother deformation (mea- dences by the intensity at each voxel, ART establishes sured by Jacobians and Harmonic Energy of the obtained correspondences by the high-dimensional feature vector deformation) at an accuracy comparable to that of the orig- at each voxel. The similarity of two voxels is defined on inal Demons [128]. In both approaches, the deformation their feature vectors as the inner-product of two vectors is iteratively updated by a velocity field, which computes and an idempotent and symmetric matrix that removes the voxel-wise displacements according to a specific similarity mean of the vector it pre-multiplies. ART is implemented metric; and the increment and the total deformation are in multi-resolution fashion. In the middle and low image smoothed by Gaussian filters to maintain smoothness. resolutions, ART searches correspondences for all voxels In this paper, both Demons and Diffeomorphic Demons in the target image. In the highest image resolution, ART are included in the evaluation. The Demons software (in- only searches correspondences at those voxels whose cluding Diffeomorphic Demons) is publicly available at gradient norms are in a certain upper percentile of the http://www.insight-journal.org/browse/publication/154. gradient magnitude histogram. The ART registration tool • DRAMMS. Deformable Registration via Attribute is publicly available at http://www.nitrc.org/projects/art/. Matching and Mutual-Saliency Weighting [56] is a gen- • ANTs. The Advanced Normalization Tools (ANTs) was eral-purpose deformable registration algorithm. It makes developed by researchers (Avants et al.)attheUniver- contributions in defining a new similarity metric with two sity of Pennsylvania in the 2000s [53]. It is based on features. One feature is that it finds voxel correspondences the LDDMM algorithm [40], but improves LDDMM’s based on high-dimensional Gabor texture attributes. The computational efficiency and introduces symmetry into other feature is that it does not force each and every voxel to the LDDMM framework. In particular, ANTs decom- find its correspondences as most other general-purpose reg- poses the diffeomorphic deformation into two symmetric istration methods do. Rather, it weights voxels differently components. The idea is that, instead of deforming one based on automatically detecting the ability of this voxel image into the space of the other image, which is usually to establish correspondences between images. This way, not symmetric to the input images, ANTs simultaneously the registration is mainly driven by voxels/regions that can deforms two input images, each towards the “midpoint” establish reliable correspondences; at the same time, it can image. Therefore, the formulation becomes symmetric effectively reduce the negative impact of outlier regions if to the input images. ANTs supports three intensity-based they exist. DRAMMS uses the cubic B-spline transforma- similarity metrics: SSD, which LDDMM uses; CC, which tion model, as will be described later in the FFD algorithm. is used by default in ANTs; and MI. The gradient descent The DRAMMS software package is publicly available at optimization strategy [127] is used to numerically find http://www.cbica.upenn.edu/sbia/software/dramms/. the optimal deformation in the above formulation. ANTs • DROP. DROP is the implementation of a deformable iteratively updates the transformation by a velocity field, image registration pipeline developed by researchers which, subject to the symmetry constraints, is computed (Glocker et al.) at the Ecole Centrale de Paris, France, OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2059

the Technische Universität München, Germany, and the cian of the deformation (also known as the bending en- University of Grete, Greece [58]. The main contribution ergy of the deformation). The main contribution is in the of DROP is in the numerical optimization. Discrete opti- optimization process [129]. The optimization is based on mization is, for the first time, introduced into the field of multi-resolution Levenberg-Marquardt strategy [126]. The medical image registration, brining in significant speedup registration is initialized and run to the convergence in the compared with other continuous optimizers such as gra- down-sampled images, generating a deformation field with dient descent (8 min by discrete optimization versus 3 h 50 low resolutions and a high regularization weight. The im- min by gradient descent for the same set of brain images, ages and the deformation field from the first step are then as reported in [58]). DROP uses FFD as its deformation up-sampled, with the regularization modified. And it is model (described later in this section), and supports 12 again run to the convergence. This is repeated until the re- intensity-based similarity metrics: Sum of Absolute Dif- quired high-resolution and the required level of regular- ferences (SAD), which is used by default, Sum of Absolute ization is achieved. The fnirt registration tool is publicly Difference plus Sum of Gradient Inner Products (SADG), available at http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/fnirt. Sum of Squared Differences (SSD), Normalized Correla- tion Coefficient (NCC), Normalized Mutual Information APPENDIX B (NMI), Correlation Ratio (CR), Sum of Gradient Inner PARAMETER SETTINGS FOR REGISTRATION METHODS Product (SGAD), and others. The DROP registration tool is publicly available at http://www.mrf-registration.net/. In Section III-C, we presented two rules to set the parameters • FFD and its variants (CC-/MI-/SSD-FFD). The free for each registration method, in order to maintain fairness of the form deformation (FFD) model is a geometric transfor- evaluation. Appendix B below provides the details of parameter mation model that was introduced to image registration settings for transparency and reproducibility in the evaluation. by [54] in 1999. Since then it has been widely used and • flirt. cited for more than 2400 times in Google Scholar search engine (as of March 2013). In the FFD model, a regular - grid of so-called “control points” is superimposed on - top of the dense image lattice. A FFD model basically - states that the movement of an image voxel is a smooth, - cubic B-spline-based interpolation of the displacement - of the control points surrounding this voxel. Therefore, - the task of finding movements at each voxel was trans- - lated into finding the displacements at regularly-spaced - control points. In contrast to voxels-based methods, FFD - • AIR. has three nice properties: 1) control points are regularly spaced in the image, providing guidance throughout the image domain; 2) deformation is smooth by the cubic B-spline-based interpolation; 3) landmarks moves by finding its own correspondences, whereas control point - moves by finding the most possible corresponding patch - for the image patch it controls, or represents. In the original - work [54], FFD was combined with normalized mutual - information (NMI) similarity metric. It can be combined • ART. with several other similarity metrics, including correlation coefficient (CC) and sum of squared difference (SSD). In - this paper, we used the IRTK software package, which is - the original implementation of the FFD-based registration. - NMI-FFD, CC-FFD and SSD-FFD were all included - in the evaluation. The software is publicly available at - http://www.doc.ic.ac.uk/~dr/software/. - • fnirt. The FMRIB’s Nonlinear Image Registration Tool • ANTs. (fnirt) was developed by researchers (Anderson, Smith, and Jenkinson) at the University of Oxford, Oxford, U.K., - in 2007 [49], [129]. The fnirt method uses Sum of Squared - Differences (SSD) as the similarity metric, therefore it is - only suitable for mono-modality image registration tasks. - It implements the free form deformation (FFD) model. The - deformation is regularized by the magnitude of the Lapla- - 2060 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

• (Additive) Demons.

------• Diffeomorphic Demons.

- - . - Note that, in [17]; we set - to increase smoothness of the deformation. - • MI-FFD. • DRAMMS. The usage is the same as in CC-FFD above, with the only difference being one item in : – . – • SSD-FFD. – TheusageisthesameasinCC-FFDabove, with the – only difference being two items in : • DROP. . • fnirt.

where specifies all parameters. They are:

. • CC-FFD.

- - where specifies all parameters. They are:

APPENDIX C JACCARD OVERLAP FOR ALL ROISINTHENIREP AND LONI DATABASES Table III shows Jaccard overlaps, averaged over all pair-wise registrations, in all 32 ROIs in the NIREP database. Methods that obtained highest average Jaccard overlap in each ROI are noted in bold texts. Similarly, Table V shows overlaps, aver- OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2061

TABLE III TABLE IV JACCARD OVERLAP FOR EACH OF THE 32 ROISINTHENIREP DATABASE, JACCARD OVERLAP FOR EACH OF THE 11 ROISINTHEBRAINWEB DATABASE, AVERAGED AMONG ALL 240 REGISTRATIONS .FOR EACH ROI, AVERAGED AMONG ALL 110 REGISTRATIONS .FOR EACH ROI, BOLD TEXTS AND LIGHT-GRAY TEXTS INDICATE THAT THE CORRESPONDING BOLD TEXTS AND LIGHT-GRAY TEXTS INDICATE THAT THE CORRESPONDING REGISTRATION METHODS OBTAIN THE HIGHEST AND THE SECOND HIGHEST REGISTRATION METHODS OBTAIN THE HIGHEST AND THE SECOND HIGHEST OVERLAP COMPARED TO OTHER REGISTRATION METHODS OVERLAP COMPARED TO OTHER REGISTRATION METHODS

aged over all pair-wise registrations, in all 56 ROIs in the LONI- LPBA40 database. They provide more information than Fig. 7 APPENDIX D especially on what level of accuracy we can expect for each JACCARD OVERLAP FOR ALL ROISINTHE of the ROI structure in inter-subject registration. For instance, BRAINWEB DATABASE hippocampus may have over 0.6 Jaccard overlap in inter-sub- Table IV shows the average Jaccard overlap in all 11 ROIs in ject registration, while superior occipital gyrus may only have the BrainWeb database. Methods that obtained highest average around 0.45 Jaccard overlap. Jaccard overlap in each ROI are noted in bold texts. 2062 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

TABLE V JACCARD OVERLAP FOR EACH OF THE 56 ROISINTHELONI-LPBA40 DATABASE,AVERAGED AMONG ALL 210 REGISTRATIONS .FOR EACH ROI, BOLD TEXTS AND LIGHT-GRAY TEXTS INDICATE THAT THE CORRESPONDING REGISTRATION METHODS OBTAIN THE HIGHEST AND THE SECOND HIGHEST OVERLAP COMPARED TO OTHER REGISTRATION METHODS OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2063

REFERENCES [25] K. K. Leung, J. Barnes, M. Modat, G. R. Ridgway, J. W. Bartlett, N. C. Fox, and S. Ourselin, “Brain maps: An automated, accurate and robust [1]A.Sotiras,C.Davatzikos,andN. Paragios, “Deformable medical brain extraction technique using a template library,” NeuroImage, vol. image registration: A survey,” IEEE Trans. Med. Imag., vol. 32, no. 55, no. 3, pp. 1091–1108, 2011. 7, pp. 1153–1190, Jul. 2013. [26] S. Parisot, W. Wells, III, S. Chemouny, H. Duffau, and N. Paragios, [2] A. Sotiras, Y. Ou, N. Paragios, and C. Davatzikos, “Graph-based de- “Concurrent tumor segmentation and registration with uncer- formable image registration,” in Handbook of Biomedical Imaging; tainty-based sparse non-uniform graphs,” Med. Image Anal.,vol.18, Methodologies and Clinical Research, N. Paragios, J. Duncan, and N. no. 4, pp. 647–659, 2014. Ayache, Eds. New York: Springer, 2015. [27] M. Brett, A. P. Leff, C. Rorden, and J. Ashburner, “Spatial normaliza- [3] J. B. A. Maintz and M. A. Viergever, “A survey of medical image reg- tion of brain images with focal lesions using cost function masking,” istration,” Med. Image Anal., vol. 2, no. 1, pp. 1–36, 1998. NeuroImage, vol. 14, no. 2, pp. 486–500, 2001. [4] H. Lester and S. Arridge, “A survey of hierarchical non-linear med- [28] B. M. Dawant, S. Hartmann, and S. Gadamsetty, “Brain atlas ical image registration,” Pattern Recognit., vol. 32, no. 1, pp. 129–149, deformation in the presence of large space-occupying tumors,” 1999. in Medical Image Computing and Computer-Assisted Interven- [5] D. L. Hill, P. G. Batchelor, M. Holden, and D. J. Hawkes, “Medical tion—MICCAI’99. New York: Springer, 1999, pp. 589–596. image registration.,” Phys. Med. Biol., vol. 46, no. 3, pp. 1–45, 2001. [29] M. Cuadra, C. Pollo, A. Bardera, O. Cuisenaire, J. Villemure, and J. [6] B. Zitova and J. Flusser, “Image registration methods: A survey,” Thiran, “Atlas-based segmentation of pathological MR brain images Image Vis. Comput., vol. 21, no. 11, pp. 977–1000, 2003. usingamodeloflesiongrowth,”IEEE Trans. Med. Imag., vol. 23, no. [7]J.P.W.Pluim,J.B.A.Maintz,andM.A.Viergever,“Mutual-infor- 10, pp. 1301–1314, Oct. 2004. mation-based registration of medical images: A survey,” IEEE Trans. [30] A. Mohamed et al., “Deformable registration of brain tumor images via Med. Imag., vol. 22, no. 8, pp. 986–1004, Aug. 2003. a statistical model of tumor-induced deformation,” Med. Image Anal., [8] W. R. Crum, T. Hartkens, and D. L. Hill, “Non-rigid image registration: vol. 10, no. 5, pp. 752–763, 2006. Theory and practice,” Br.J.Radiol., vol. 77, no. 2, pp. 140–153, 2004. [31] E. Zacharaki, D. Shen, S. Lee, and C. Davatzikos, “ORBIT: A multires- [9] M. Holden, “A review of geometric transformations for nonrigid body olution framework for deformable registration of brain tumor images,” registration,” IEEE Trans. Med. Imag., vol. 27, no. 1, pp. 111–128, Jan. IEEE Trans. Med. Imag., vol. 27, no. 8, pp. 1003–1017, Aug. 2008. 2008. [32] A. Gooya, G. Biros, and C. Davatzikos, “Deformable registration of [10] J. West et al., “Comparison and evaluation of retrospective inter- glioma images using em algorithm and diffusion reaction modeling,” modality brain image registration techniques,” J. Comput. Assist. IEEE Trans. Med. Imag., vol. 30, no. 2, pp. 375–390, Feb. 2011. Tomogr., vol. 21, no. 4, pp. 554–568, 1997. [33]B.Wang,M.Prastawa,A.Saha,S.P.Awate,A.Irimia,M.C.Cham- [11] P. Hellier, C. Barillot, I. Corouge, B. Gibaud, G. Le Goualher, L. bers,P.M.Vespa,J.D.VanHorn, V. Pascucci, and G. Gerig, “Mod- Collins, A. Evans, G. Malandain, and N. Ayache, “Retrospective eling 4D changes in pathological anatomy using domain adaptation: evaluation of inter-subject brain registration,” in Medical Image Com- Analysis of TBI imaging using a tumor database,” in Multimodal Brain puting and Computer-Assisted Intervention—MICCAI 2001.New Image Analysis.NewYork: Springer, 2013, pp. 31–39. York: Springer, 2001, pp. 258–265. [34] D. Kwon, M. Niethammer, H. Akbari, M. Bilello, C. Davatzikos, and [12] I. Yanovsky, A. D. Leow, S. Lee, S. J. Osher, and P. M. Thompson, K. Pohl, “PORTR: Pre-operative and post-recurrence brain tumor reg- “Comparing registration methods for mapping brain change using istration,” IEEE Trans. Med. Imag., vol. 33, no. 3, pp. 651–667, Mar. tensor-based morphometry,” Med. Image Anal.,vol.13,no.5,p.679, 2014. 2009. [35] D. Shattuck, M. Mirza, V. Adisetiyo, C. Hojatkashani, G. Salamon, [13] M. A. Yassa and C. E. Stark, “A quantitative evaluation of cross-par- K. Narr, R. Poldrack,R.Bilder,andT.AW,“Constructionofa3-D ticipant registration techniques for MRI studies of the medial temporal probabilistic atlas of human cortical structures.,” NeuroImage,vol.39, lobe,” NeuroImage, vol. 44, no. 2, pp. 319–327, 2009. pp. 1064–1080, 2008. [14] G. E. Christensen, X. Geng, J. G. Kuhl, J. Bruss, T. J. Grabowski, I. [36] J.G.Sled,A.P.Zijdenbos, and A. C. Evans, “A nonparametric method A. Pirwani, M. W. Vannier, J. S. Allen, and H. Damasio, “Introduc- for automatic correction of intensity nonuniformity in MRI data,” IEEE tion to the non-rigid image registration evaluation project (NIREP),” Trans. Med. Imag., vol. 17, no. 1, pp. 87–97, Jan. 1998. in Biomedical Image Registration. New York: Springer, 2006, pp. [37] B. Aubert-Broche, M. Griffin, G. Pike, A. Evans, and D. Collins, 128–135. “Twenty new digital brain phantoms for creation of validation image [15]J.H.Song,G.E.Christensen,J.A.Hawley,Y.Wei,andJ.G.Kuhl, data bases,” IEEE Trans. Med. Imag., vol. 25, no. 11, pp. 1410–1416, “Evaluating image registration using NIREP,” in Biomedical Image Nov. 2006. Registration. New York: Springer, 2010, pp. 140–150. [38] O. Tsang, A. Gholipour, N. Kehtarnavaz, K. Gopinath, R. Briggs, and I. [16] Y. Wei, “Non-rigid image registration evaluation using common eval- Panahi, “Comparison of tissue segmentation algorithms in neuroimage uation databases,” M.S. thesis, Univ. Iowa, Iowa City, 2009. analysis software tools,” in Proc. 30th Annu. Int. Conf. IEEE EMBS, [17] A. Klein et al., “Evaluation of 14 nonlinear deformation algorithms 2008, pp. 3924–3928. applied to human brain MRI registration,” NeuroImage, vol. 46, no. 3, [39]D.Marcus,T.Wang,J.Parker,J.Csernansky,J.Morris,andR. pp. 786–802, 2009. Buckner, “Open access series of imaging studies (OASIS): Cross-sec- [18]A.Klein,S.S.Ghosh,B.Avants,B.T.Yeo,B.Fischl,B.Ardekani,J. tional MRI data in young, middle aged, nondemented, and demented C. Gee, J. J. Mann, and R. V. Parsey, “Evaluation of volume-based and older adults,” J. Cognitive Neurosci., vol. 19, no. 9, pp. 1498–1507, surface-based brain image registration methods,” NeuroImage, vol. 51, 2007. no. 1, pp. 214–220, 2010. [40] M. F. Beg, M. I. Miller, A. Trouvé, and L. Younes, “Computing large [19] J. Doshi, G. Erus, Y. Ou, B. Gaonkar, and C. Davatzikos, “Multi-atlas deformation metric mappings via geodesic flows of diffeomorphisms,” skull stripping,” Acad. Radiol., vol. 20, no. 12, pp. 1566–1576, 2013. Int. J. Comput. Vis., vol. 61, no. 2, pp. 139–157, 2005. [20] G. Wu, F. Qi, and D. Shen, “Learning-based deformable registration [41] S. Klein, M. Staring, K. Murphy, M. A. Viergever, and J. P. W. Pluim, of MR brain images,” IEEE Trans. Med. Imag., vol. 25, no. 9, pp. “elastix: A toolbox for intensity-based medical image registration,” 1145–1157, 2006. IEEETrans.Med.Imag., vol. 29, no. 1, pp. 196–205, Jan. 2010. [21] B. T. Yeo, M. R. Sabuncu, T. Vercauteren, D. J. Holt, K. Amunts, K. [42] M. Modat, G. Ridgway, Z. Taylor, M. Lehmann, J. Barnes, D. Hawkes, Zilles, P. Golland, and B. Fischl, “Learning task-optimal registration N. Fox, and S. Ourselin, “Fast free-form deformation using graphics cost functions for localizing cytoarchitecture and function in the cere- processing units.,” Comput. Methods Programs Biomed., vol. 98, no. bral cortex,” IEEE Trans. Med. Imag., vol. 29, no. 7, pp. 1424–1441, 3, pp. 278–284, 2010. Jul. 2010. [43] G. Sharp, R. Li, J. Wolfgang, G. Chen, M. Peroni, M. Spadea, S. Mori, [22] F.Michel,M.Bronstein,A.Bronstein,andN.Paragios,“Boostedmetric J. Zhang, J. Shackleford, and N. Kandasamy, “Plastimatch—An open learning for 3-D multi-modal deformable registration,” in Proc. IEEE source software suite for radiotherapy image processing,” presented at Int. Symp. Biomed. Imag.: From Nano to Macro, 2011, pp. 1209–1214. the 16th Int. Conf. Use Comput. Radiother., Amsterdam, The Nether- [23] M. Kim, G. Wu, P.-T. Yap, and D. Shen, “A general fast registration lands, 2010. framework by learning deformation-appearance correlation,” IEEE [44] J. Ashburner et al., “A fast diffeomorphic image registration algo- Trans. Image Process., vol. 21, no. 4, pp. 1823–1833, Apr. 2012. rithm,” NeuroImage, vol. 38, no. 1, pp. 95–113, 2007. [24] A. Valsecchi, J. Dubois-Lacoste, T. Stutzle, S. Damas, J. Santamaria, [45] D. Shen and C. Davatzikos, “HAMMER: Hierarchical attribute and L. Marrakchi-Kacem, “Evolutionary medical image registration matching mechanism for elastic registration,” IEEE Trans. Med. using automatic parameter tuning,” in 2013 IEEE Congr. Evolut. Imag., vol. 21, no. 11, pp. 1421–1439, Nov. 2002. Comput., 2013, pp. 1326–1333, IEEE. [46] B. Fischl, “Freesurfer,” NeuroImage, 2012. 2064 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 33, NO. 10, OCTOBER 2014

[47] B. Yeo, M. R. Sabuncu, T. Vercauteren, N. Ayache, B. Fischl, and P. [70] J. Liu, J. K. Udupa, D. Odhner, D. Hackney, and G. Moonis, “A system Golland, “Spherical demons: Fast diffeomorphic landmark-free sur- for brain tumor volume estimation via MR imaging and fuzzy connect- face registration,” IEEE Trans. Med. Imag., vol. 29, no. 3, pp. 650–668, edness,” Comput. Med. Imag. Graph.,vol.29,no.1,p.21,2005. Mar. 2010. [71] R. Verma, E. I. Zacharaki, Y. Ou, H. Cai, S. Chawla, S.-K. Lee, [48] M. Jenkinson and S. Smith, “A global optimisation method for robust E. R. Melhem, R. Wolf, and C. Davatzikos, “Multi-parametric tissue affine registration of brain images.,” Med. Image Anal.,vol.5,no.2, characterization of brain neoplasms and their recurrence using pattern pp. 143–156, 2001. classification of MR images,” Acad. Radiol.,vol.15,no.8,p.966, [49] J. Andersson, S. Smith, and M. Jenkinson, “FNIRT—FMRIB non- 2008. linear image registration tool,” in 14th Annu. Meet. Organizat. Human [72] J. Doshi, G. Erus, Y. Ou, and C. Davatzikos, “Ensemble-based Brain Map., Melbourne, Australia, 2008. medical image labeling via sampling morphological appearance [50] R. Woods et al., “Rapid automated algorithm for aligning and manifolds,” in Medical Image Computing and Computer-Assisted reslicing PET images,” J. Comput. Assist. Tomogr., vol. 16, no. 4, Intervention—MICCAI’13 Challenge Workshop on Segmentation: p. 620, 1992. Algorithms, Theory and Applications. New York: Springer, 2013. [51] R. Woods, S. Grafton, C. Holmes, S. Cherry, and M. JC, “Automated [73] L. Tang and G. Hamarneh, “Random walks with efficient search and image registration: I. General methods and intrasubject, intramodality contextually adapted image similarity for deformable registration,” validation,” J. Comput. Assist. Tomogr., vol. 22, pp. 139–152, 1998. in Medical Image Computing and Computer-Assisted Intervention [52] B. Ardekani et al., “Quantitative comparison of algorithms for inter- (MICCAI). New York: Springer, 2013, vol. 8150, LNCS, pp. 43–50. subject registration of 3-D volumetric brain MRI scans,” J. Neurosci. [74] Z. Xue, D. Shen, and C. Davatzikos, “Determining correspondence Methods, vol. 142, no. 1, pp. 67–76, 2005. in 3-D MR brain images using attribute vectors as morphological [53] B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric signatures of voxels,” IEEE Trans. Med. Imag., vol. 23, no. 10, pp. diffeomorphic image registration with cross-correlation: Evaluating 1276–1291, Oct. 2004. automated labeling of elderly and neurodegenerative brain,” Med. [75] Y. Zhan, Y. Ou, M. Feldman, J. Tomaszeweski, C. Davatzikos, and D. Image Anal., vol. 12, no. 1, pp. 26–41, 2008. Shen, “Registering histologic and MR images of prostate for image- [54] D. Rueckert, L. Sonoda, C. Hayes, D. Hill,M.Leach,andD.Hawkes, based cancer detection,” Acad. Radiol., vol. 14, no. 11, pp. 1367–1381, “Nonrigid registration using free-form deformations: Application 2007. to breast MR images,” IEEE Trans. Med. Imag.,vol.18,no.8,pp. [76] A. Sotiras, Y. Ou, B. Glocker, C. Davatzikos, and N. Paragios, “Simul- 712–721, Aug. 1999. taneous geometric-iconic registration,” in Medical Image Computing [55] T. Vercauteren, X. Pennec, A. Perchant, and N. Ayache, “Diffeo- and Computer-Assisted Intervention—MICCAI 2010.NewYork: morphic demons: Efficient non-parametric image registration,” Springer, 2010, pp. 676–683. NeuroImage,vol.45,no.1,pp. S61–S72, 2009. [77] Y. Ou, A. Besbes, M. Bilello, M. Mansour, C. Davatzikos, and N. Para- [56] Y. Ou, A. Sotiras, N. Paragios, and C. Davatzikos, “DRAMMS: gios, “Detecting mutually-salient landmark pairs with MRF regulariza- Deformable registration via attribute matching and mutual-saliency tion,” in Proc.IEEEInt.Symp.Biomed.Imag.,FromNanotoMacro, weighting,” Med. Image Anal., vol. 15, no. 4, pp. 622–639, 2011. 2010, pp. 400–403. [57] J. Thirion, “Image matching as a diffusion process: An analogy with [78] J. Yang, J. P. Williams, Y. Sun, R. S. Blum, and C. Xu, “A robust hybrid Maxwell’s demons,” Med. Image Anal., vol. 2, no. 3, pp. 243–260, method for nonrigid image registration,” Pattern Recognit., vol. 44, no. 1998. 4, pp. 764–776, 2011. [58] B. Glocker et al., “Dense image registration through MRFs and ef- [79] H. Lu, L.-P. Nolte, and M. Reyes, “Interest points localization for brain ficient linear programming,” Med. Image Anal., vol. 12, no. 6, pp. image using landmark-annotated atlas,” Int. J. Imag. Syst. Technol., 731–741, 2008. vol. 22, no. 2, pp. 145–152, 2012. [59] R. A. Heckemann, S. Keihaninejad, P. Aljabar, D. Rueckert, J. V. [80] Y. H. Le, U. Kurkure, and I. A. Kakadiaris, “PDM-ENLOR: Learning Hajnal, and A. Hammers, “Improving intersubject image registration ensemble of local PDM-based regressions,” in Proc. IEEE Conf. using tissue-class information benefits robustness and accuracy of Comput. Pattern Recognit., 2013, pp. 1878–1885. multi-atlas based anatomical segmentation,” NeuroImage, vol. 51, no. [81]M.P.Heinrich,M.Jenkinson,M.Bhushan,T.Matin,F.V.Gleeson, 1, pp. 221–227, 2010. S. M. Brady, and J. A. Schnabel, “Mind: Modality independent neigh- [60] H. Jia, G. Wu, Q. Wang, and D. Shen, “ABSORB: Atlas building by bourhood descriptor for multi-modal deformable registration,” Med. self-organized registration and bundling,” NeuroImage, vol. 51, no. 3, Image Anal., vol. 16, no. 7, pp. 1423–1435, 2012. pp. 1057–1070, 2010. [82] M. Toews and W. M. Wells, III, “Efficient and robust model-to-image [61] M. Toews, W. Wells, III, D. L. Collins, and T. Arbel, “Feature-based alignment using 3-D scale-invariant features,” Med. Image Anal., vol. morphometry: Discovering group-related anatomical patterns,” Neu- 17, no. 3, pp. 271–282, 2013. roImage, vol. 49, no. 3, pp. 2318–2327, 2010. [83]Y.Ou,S.P.Weinstein,E.F.Conant,S.Englander,X.Da,B. [62] J. Hamm, D. H. Ye, R. Verma, and C. Davatzikos, “GRAM: A frame- Gaonkar,M.-K.Hsieh,M.Rosen,A.DeMichele,C.Davatzikos,and work for geodesic registration on anatomical manifolds,” Med. Image D. Kontos, “Deformable registration for quantifying tumor changes Anal., vol. 14, no. 5, pp. 633–642, 2010. during neoadjuvant chemotherapy,” Magn. Reson. Med., 2014, to be [63] D. H. Ye, J. Hamm, D. Kwon, C. Davatzikos, and K. M. Pohl, “Re- published. gional manifold learning for deformable registration of brain MR im- [84] H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image ages,” in Medical Image Computing and Computer-Assisted Interven- registration by maximization of quantitative-qualitative measure of tion—MICCAI 2012. New York: Springer, 2012, pp. 131–138. mutual information,” Pattern Recognit., vol. 41, no. 1, pp. 285–298, [64] P. Jaccard, Etude comparative de la distribution florale dans une 2008. portion des Alpes et du Jura. Lausanne, Switzerland: Impr. Corbaz, [85] H. Rivaz, Z. Karimaghaloo, and D. L. Collins, “Self-similarity 1901. weighted mutual information: A new nonrigid image registration [65] T. Rohlfing, “Image similarity and tissue overlaps as surrogates for metric,” Med. Image Anal., vol. 18, no. 2, pp. 343–358, 2014. image registration accuracy: Widely used but unreliable,” IEEE Trans. [86] B. Qin, Z. Gu, X. Sun, and Y. Lv, “Registration of images with outliers Med. Imag., vol. 31, no. 2, pp. 153–163, Feb 2012. using joint saliency map,” IEEE Signal Process. Lett., vol. 17, no. 1, [66] L. Dice, “Measures of the amount of ecologic association between pp. 91–94, Jan. 2010. species,” Ecology, vol. 26, no. 3, pp. 297–302, 1945. [87] X. Zhuang, S. Arridge, D. J. Hawkes, and S. Ourselin, “A nonrigid [67] O. Colliot, O. Camara, and I. Bloch, “Integration of fuzzy spatial rela- registration framework using spatially encoded mutual information and tions in deformable models—Application to brain MRI segmentation,” free-form deformations,” IEEE Trans. Med. Imag., vol. 30, no. 10, pp. Pattern Recognit., vol. 39, no. 8, pp. 1401–1414, 2006. 1819–1828, Oct. 2011. [68] H. Elhawary, S. Oguro, K. Tuncali, P. R. Morrison, S. Tatli, P. B. Shyn, [88] D. Pace, S. Aylward, and M. Niethammer, “A locally adaptive regular- S. G. Silverman, and N. Hata, “Multimodality non-rigid image regis- ization based on anisotropic diffusion for deformable image registra- tration for planning, targeting and monitoring during CT-guided per- tion of sliding organs,” IEEE Trans. Med. Imag., vol. 32, no. 11, pp. cutaneous liver tumor cryoablation,” Acad. Radiol., vol. 17, no. 11, pp. 2114–2126, 2013. 1334–1344, 2010. [89] M. Reuter, H. D. Rosas, and B. Fischl, “Highly accurate inverse con- [69] M. Essig, H. Hawighorst, S. O. Schoenberg, R. Engenhart-Cabillic, sistent registration: A robust approach,” NeuroImage, vol. 53, no. 4, M. Fuss, J. Debus, I. Zuna, M. V. Knopp, and G. van Kaick, “Fast pp. 1181–1196, 2010. fluid-attenuated inversion-recovery (FLAIR) MRI in the assessment [90] H. D. Tagare, D. Groisser, and O. Skrinjar, “Symmetric non-rigid regis- of intraaxial brain tumors,” J. Magn. Reson. Imag., vol. 8, no. 4, pp. tration: A geometric theory and some numerical techniques,” J. Math. 789–798, 2005. Imag. Vis., vol. 34, no. 1, pp. 61–88, 2009. OU et al.: COMPARATIVE EVALUATION OF REGISTRATION ALGORITHMS IN DIFFERENT BRAIN DATABASES WITH VARYING DIFFICULTY 2065

[91] I. Aganj, M. Reuter, M. R. Sabuncu, and B. Fischl, “Symmetric [110]G.Erus,H.Battapady,T.D.Satterthwaite,H.Hakonarson, non-rigid image registration via an adaptive quasi-volume-preserving R. E. Gur, C. Davatzikos, and R. C. Gur, “Imaging patterns of constraint,” in Proc. IEEE 10th Int. Symp. Biomed. Imag., 2013, pp. brain development and their relationship to cognition,” Cerebral 230–233. Cortex, p. bht425, 2014. [92] B. Glocker, A. Sotiras, N. Komodakis, and N. Paragios, “Deformable [111]M.Ingalhalikar,D.Parker,Y.Ghanbari,A.Smith,K.Hua,S.Mori, medical image registration: Setting the state of the art with discrete T.Abel,C.Davatzikos,andR.Verma, “Connectome and maturation methods*,” Annu. Rev. Biomed. Eng., vol. 13, pp. 219–244, 2011. profiles of the developing mouse brain using diffusion tensor imaging,” [93] X. Han, L. S. Hibbard, and V. Willcut, “GPU-accelerated, gradient- Cerebral Cortex, p. bhu068, 2014. free mi deformable registration for atlas-based MR brain imageseg- [112] Y. Ou, N. Reynolds, R. Gollub, R. Pienaar, Y. Wang, T. Wang, D. mentation,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Sack,K.Andriole,S.Pieper,C.Herrick, S. N. Murphy, P. E. Grant, Recognit. Workshop, 2009, pp. 141–148. and L. Zöllei, “Developmental brain ADC atlas creation from clinical [94] R. Shams, P. Sadeghi, R. Kennedy, and R. Hartley, “A survey of images,” in Org. Human Brain Mapp., Hamburg, Germany, 2014. medical image registration on multicore and the GPU,” IEEE Signal [113] S. Baloch and C. Davatzikos, “Morphological appearance manifolds Process. Mag., vol. 27, no. 2, pp. 50–60, Mar. 2010. in computational anatomy: Groupwise registration and morphological [95] X. Gu, H. Pan, Y. Liang, R. Castillo, D. Yang, D. Choi, E. Castillo, A. analysis,” NeuroImage, vol. 45, no. 1, pp. S73–S85, 2009. Majumdar, T. Guerrero, and S. B. Jiang, “Implementation and evalua- [114] B. Baumann, B. Teo, K. Pohl, Y. Ou, J. Doshi, M. Alonso-Basanta, tion of various demons deformable image registration algorithms on a J. Christodouleas, C. Davatzikos, G. Kao, and J. Dorsey, “Multipara- GPU,” Phys. Med. Biol., vol. 55, no. 1, p. 207, 2010. metric processing of serial MRI during radiation therapy of brain tu- [96] J. A. Schnabel, C. Tanner, A. D. Castellano-Smith, A. Degenhard, M. mors: “Finishing with flair?”,” Int. J. Radiat. Oncol. Biol. Phys., vol. O. Leach, D. R. Hose, D. L. Hill, and D. J. Hawkes, “Validation of 81, p. S794, 2011. nonrigid image registration using finite-element methods: Application [115] S. S. Ghosh, S. Kakunoori, J. Augustinack, A. Nieto-Castanon, I. to breast MR images,” IEEE Trans. Med. Imag.,vol.22,no.2,pp. Kovelman, N. Gaab, J. A. Christodoulou, C. Triantafyllou, J. D. 238–247, Feb. 2003. Gabrieli, and B. Fischl, “Evaluating the validity of volume-based and [97] X. Li et al., “Validation of an algorithm for the nonrigid registration of surface-based brain image registration for developmental cognitive longitudinal breast MR images using realistic phantoms,” Med. Phys., neurosciencestudiesinchildren4to11yearsofage,”NeuroImage, vol. 37, p. 2541, 2010. vol. 53, no. 1, pp. 85–93, 2010. [98]B.B.Avants,N.J.Tustison,G.Song, P. A. Cook, A. Klein, and J. [116] S. Seshamani, P. Rajan, R. Kumar, H. Girgis, T. Dassopoulos, C. Gee, “A reproducible evaluation of ANTs similarity metric perfor- G. Mullin, and G. Hager, “A meta registration framework for mance in brain image registration,” NeuroImage, vol. 54, no. 3, pp. lesion matching,” in Medical Image Computing and Computer- 2033–2044, 2011. Assisted Intervention—MICCAI 2009. New York: Springer, 2009, [99] K. Murphy et al., “Evaluation of registration methods on thoracic CT: pp. 582–589. The EMPIRE10 challenge,” IEEE Trans. Med. Imag., vol. 30, no. 11, [117] S. E. Muenzing, B. van Ginneken, and J. P. Pluim, “On combining pp. 1901–1920, Nov. 2011. algorithms for deformable image registration,” in Biomedical Image [100] R. Castillo, E. Castillo, R. Guerra, V. E. Johnson, T. McPhail, A. K. Registration. New York: Springer, 2012, pp. 256–265. Garg, and T. Guerrero, “A framework for evaluation of deformable [118] S. E. Muenzing, B. van Ginneken, M. A. Viergever, and J. P. Pluim, image registration spatial accuracy using large landmark point sets,” “Dirboost—An algorithm for boosting deformable image registration: Phys. Med. Biol., vol. 54, no. 7, p. 1849, 2009. Application to lung CT intra-subject registration,” Med. Image Anal., [101] Y. Ou, D. H. Ye, K. M. Pohl, and C. Davatzikos, “Validation of vol. 18, no. 3, pp. 449–459, 2014. DRAMMS among 12 popular methods in cross-subject cardiac [119] Y. Guo, G. Wu, J. Jiang, and D. Shen, “Robust anatomical correspon- MRI registration,” in Biomedical Image Registration.NewYork: dence detection by hierarchical sparse graph matching,” IEEE Trans. Springer, 2012, pp. 209–219. Med. Imag., vol. 32, no. 2, pp. 268–277, Feb. 2013. [102] X. Geng, D. Kumar, and G. E. Christensen, “Transitive inverse-con- [120] U. Kurkure, Y.H.Le,N.Paragios,J.P.Carson,T.Ju,andI.A.Kakadi- sistent manifold registration,” in Information Processing in Medical aris, “Landmark/image-based deformable registration of gene expres- Imaging. New York: Springer, 2005, pp. 468–479. sion data,” in Proc. IEEE Conf, Comput. Vis. Pattern Recognit.,2011, [103] M. V. Zanetti,M.S.Schaufelberger,J.Doshi,Y.Ou,L.K.Ferreira, pp. 1089–1096. P. R. Menezes, M. Scazufca, C. Davatzikos, and G. F. Busatto, [121] G. Sharp et al., “SU-E-J-64: Landmark and ROI-enhancement-assisted “Neuroanatomical pattern classification in a population-based sample inter-patient deformable registration of 3-D bladder CT images,” Med. of first-episode schizophrenia,” Progr. Neuro-Psychopharmacol. Biol. Phys., vol. 40, no. 6, pp. 164–164, 2013. Psychiatry, vol. 43, no. 116, p. 125, 2012. [122] A. Gupta, H. K. Verma, and S. Gupta, “A hybrid framework for regis- [104] R.Bansal,L.H.Staib,A.F.Laine,X.Hao,D.Xu,J.Liu,M.Weissman, tration of carotid ultrasound images combining iconic and geometric and B. S. Peterson, “Anatomical brain images alone can accurately di- features,” Med. Biol. Eng. Comput., vol. 51, no. 9, pp. 1043–1050, agnose chronic neuropsychiatric illnesses,” PloS One,vol.7,no.12,p. 2013. e50698, 2012. [123] S. Wörz and K. Rohr, “Spline-based hybrid image registration [105] X. Da, J. Toledo, D. Wolk, Y. Ou, A. Shacklett, P. Parmpi, L. Shaw, J. using landmark and intensity information based on matrix-valued Trojanowski, and C. Davatzikos, “Prediction of conversion from MCI non-radial basis functions,” Int. J. Comput. Vis., vol. 106, no. 1, pp. to AD: Integration and relative values of brain atrophy patterns, clinical 76–92, 2014. scores, CSF biomarkers, and APOE genotype,” presented at the RSNA [124] M. Powell, “An efficient method for finding the minimum of a function Annu. Meet., Chicago, IL, 2013. of several variables without calculating derivatives,” Comput. J., vol. [106] X.Da,J.B.Toledo,J.Zee,D.A.Wolk,S.X.Xie,Y.Ou,A.Shacklett, 7, no. 2, pp. 155–162, 1964. P. Parmpi, L. Shaw, J. Q. Trojanowski, and C. Davatzikos, “Integration [125] M. Avriel, Nonlinear Programming: Analysis and Methods.Mi- and relative value of biomarkers for prediction of MCI to AD progres- neola, NY: Dover, 2003. sion: Spatial patterns of brain atrophy, cognitive scores, APOE geno- [126] J. More, “The Levenberg-Marquardt algorithm: Implementation and type and CSF biomarkers,” NeuroImage: Clin., vol. 4, pp. 164–173, theory,” Numer. Anal., pp. 105–116, 1978. 2014. [127] A. Cauchy, “Méthode générale pour la résolution des systemes [107] N. Koutsouleris et al., “Accelerated brain aging in schizophrenia d’équations simultanées,” Comp. Rend. Sci. Paris, vol. 25, no. 1847, and beyond: A neuroanatomical marker of psychiatric disorders,” pp. 536–538, 1847. Schizophrenia Bull., p. sbt142, 2013. [128] T. Vercauteren et al., “Diffeomorphic demons: Efficient non-para- [108] M. H. Serpa et al., “Neuroanatomical classification in a population- metric image registration.,” NeuroImage, vol. 45, no. 1, pp. S61–S72, based sample of psychotic major depression and bipolar I disorder with 2008. 1 year of diagnostic stability,” BioMed Res. Int., vol. 2014, 2014. [129] J. Andersson, M. Jenkinson, and S. Smith, Non-linear registration, Aka [109] T. D. Satterthwaite et al., “Neuroimaging of the philadelphia neurode- spatial normalisation FMRIB FMRIB Analysis Group, Univ. Oxford, velopmental cohort,” NeuroImage, vol. 86, pp. 544–553, 2013. Tech. Rep. TR07JA2, 2007.