NeuroImage 17, 515–531 (2002) doi:10.1006/nimg.2002.1188 Appearance-Based Segmentation of Medial Temporal Lobe Structures S. Duchesne,1 J. C. Pruessner, and D. L. Collins McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, 3801 University Street, Montreal, Canada H3A 2B4 Received October 8, 2001 high contrast and spatial resolution allow morpholog- A new paradigm for the characterization of struc- ical studies of MTL structures by extracting their po- ture appearance is proposed, based on a combination sitions, shapes, volumes, and other properties from the of gray-level MRI intensity data and a shape descrip- information contained in the images. tor derived from a priori principal components anal- The work presented in this article is concerned with ysis of 3D deformation vector fields. Generated with- the segmentation of 3D brain structures, such as those out external intervention, it extends into 3D more found in the MTL, from MR images. Manual segmen- classic, 2D manual landmark-based shape models. Ap- tation, as shown in Fig. 1, is considered highly accu- plication of this novel concept led to a method for the rate, but subject to intra/inter-observer variability, es- segmentation of medial temporal lobe structures from brain magnetic resonance images. The strategy em- pecially in the absence of a thorough segmentation ployed for segmentation aims at synthesizing, using protocol. It is also obvious for researchers that this is a the appearance model, a deformation field that maps a time-consuming endeavor. Available automatic seg- new volume onto a reference target. Any information mentation techniques include ANIMAL (Automatic defined on the reference can then be propagated back Nonlinear Image Matching and Anatomical Labeling), on the new volume instance, thereby achieving seg- a registration and segmentation tool based on image mentation. The proposed method was tested on a data intensity features, and developed at the Montre´al Neu- set of 80 normal subjects and compared against man- rological Institute (MNI) by Collins et al. (1995). While ual segmentation as well as automated segmentation the segmentation accuracy of ANIMAL is comparable results from ANIMAL, a nonlinear registration and to that of expert manual segmentation, there is no segmentation technique. Experimental results demon- statistical information embedded in the process, mak- strated the robustness and flexibility of the new ing each application of the segmentation task a com- method. Segmentation accuracy, measured by overlap pletely new one from the system’s point of view. Fur- statistics, is marginally lower (< 2%) than ANIMAL, ther, one must use an iterative multiscale approach while processing time is six times faster. Finally, the leading to a segmentation time of around 2 h for the applicability of this concept toward shape deforma- MTL region, which is not practical for many tasks and tion analysis is presented. © 2002 Elsevier Science (USA) studies. Since the inception of ANIMAL, simple models have been developed elsewhere that could be inte- grated in some fashion to enhance the capabilities of an 1. INTRODUCTION intensity-based technique without “memory” or prior knowledge of the structure of interest. For example, Brain structures like the hippocampus (HC) and the shape descriptors such as the point distribution models amygdala (AG), located in the medial temporal lobe (PDMs) of Cootes et al. (1993) are now being used (MTL), have received a lot of attention due to their extensively in a number of applications, including seg- importance in neurological diseases and disorders. For mentation in medical images (Kelemen et al., 1999). example, clinical consensus is now clear on the impli- There exist a few hybrid systems that combine inten- cation of the HC and the AG in temporal lobe epilepsy, sity and shape data into a description of the appear- and study of these structures (and others in the MTL) ance of an object, one of which has been successfully cannot but help in understanding other forms of epi- applied to medical imaging segmentation tasks (Cootes leptic activity. Current approaches in the study of MTL et al., 1998). structures are heavily reliant on in vivo imaging tech- The original motivation of the work described in this niques such as magnetic resonance imaging (MRI). Its article was to develop a new segmentation method, embedding the paradigm of statistically relevant a pri- 1 To whom correspondence should be addressed. Fax: (514) 398- ori information, that could achieve or surpass the ac- 2975. E-mail: {duchesne, jens, louis}@bic.mni.mcgill.ca. curacy of ANIMAL while possibly reducing computa- 515 1053-8119/02 $35.00 © 2002 Elsevier Science (USA) All rights reserved. 516 DUCHESNE, PRUESSNER, AND COLLINS FIG. 1. Example of T1 MR image with manually labeled HC (blue, green, red) and AG (yellow). Image courtesy N. Bernasconi, BIC, MNI. tional costs (Fig. 11). The reader is first invited to read ing on these models, an appearance-based method in- an overview of applicable segmentation strategies corporating analysis of dense deformation fields is in- forming the necessary background for this work. Draw- troduced and rigorously developed. The main FIG. 2. Processing pipeline. All MRI data are processed through the pipeline shown. After preprocessing to correct for intensity nonuniformity, the data are linearly registered into stereotaxic space and resampled onto a 1-mm isotropic grid. Nonlinear transformation to stereotaxic space is used to produce 3D dense deformation fields for PCA analysis and construction of the WDM. APPEARANCE-BASED SEGMENTATION OF MTL STRUCTURES 517 contribution of this work is the extension of 2D appear- These drawbacks motivated Pruessner et al. (2001) ance-based techniques into three dimensions by in- to conduct a manual segmentation study which has cluding nonlinear registration vector fields into a 3D been retained in this work as the gold standard for HC Warp Distribution Model. The image matching and and AG segmentation. Following a thorough literature segmentation strategy to be employed by the new tech- review, Pruessner et al. developed a protocol for con- nique is presented in the same section. The perfor- tour tracing using high-resolution, uniform, and stan- mance of the appearance-based method is character- dardized T1-weighted MR images with 1-mm isotropic ized through a series of experiments. The accuracy of voxels. He used a 3D analysis software called DIS- the segmentation method is evaluated against manual PLAY, developed at the Montreal Neurological Insti- segmentation results from an expert neuroanatomist, tute, which allows simultaneous display of coronal, and also compared with results from ANIMAL using transverse, and sagittal images for visualization and the same input data. A discussion of experimental re- segmentation. The results indicate that with 3D visu- sults and properties of the technique is followed by alization, the irregularly shaped HC and AG can be some general observations on this current work and reliably and precisely segmented. The reported intra- ideas for future research. class interrater overlap coefficients are in the range ␬ ϭ 0.83–0.94, which indicates very good accord be- 2. BACKGROUND tween raters (see Section 3 for a mathematical descrip- tion of the overlap coefficient ␬). Regarding the relative Segmentation of the HC/AG on MR images has been volumes of the targeted structures, comparison of left approached in a number of ways. While a complete and right HC and AG volumes in normal controls re- review of segmentation techniques is beyond the scope vealed a significantly larger right HC volume and no of this article, this section endeavors to describe those differences in the AG volume. that have been applied to the MTL. 2.2. Automated Segmentation Techniques 2.1. Manual Segmentation To discuss some of the many segmentation tech- Manual segmentation techniques involve contour de- niques that have been developed for purposes related lineation of MTL structures by one or more trained to brain imaging, we introduce a simple classification neuroanatomists. While expert human intervention re- scheme based on two design features: mains the most accurate segmentation technique, se- rious drawbacks undermine its usefulness in a number ● Segmentation paradigm: The prevalent paradigm of situations. is one of forward segmentation, where a predefined The main difficulty resides in the subjective inter- template is made to match the new volume to achieve pretation of anatomic variations (Hogan et al., 2000). segmentation. In most cases experts are required to In the case of HC and AG, for example, differences in initialize the segmentation process by choosing land- border definitions among research groups—and for marks. The matching process can be done in the way of that matter, among investigators of the same group— an initial contour which will be propagated through have hindered the comparison of results (Pruessner et some form of elastic matching onto the image until a al., 2000). Commonality in the definition of segmenta- proximity criterion is maximized. An example of this tion protocols is needed if results are to be compared. technique is the high-dimensional brain mapping tech- Second, research groups use different software pack- nique using fluid transformations, proposed by Chris- ages to trace the targeted structure. Most employ 2D tensen et al. (1997). Other techniques, such as the visualization tools for brain images, without the possi- active shape model (ASM) of Cootes and Taylor et al. bility of adjusting resolution or image contrast. A com- (2000), relies on the placement of landmarks on the mon 2D error is interslice misregistration, which leads image to derive a model that is then globally deformed to nonsmooth 3D surfaces (Hogan et al., 2000). On the to match some intensity features of the new image. other hand, scalable 3D imaging is available in some Backward segmentation, on the other end, can be centers, allowing for precise display and enlargement thought as a reversion of the original paradigm.