Software Approach to Merging Molecular with Anatomic Information Piotr J. Slomka, PhD Departments of Imaging and Medicine, Cedars-Sinai Medical Center and UCLA School of Medicine, Los Angeles, California; and Department of Diagnostic Radiology and Nuclear Medicine, University of Western Ontario, London, Ontario, Canada ized and are used clinically in several centers. In this article, Software image registration is a powerful and versatile tool that issues related to the practical implementation of software allows the fusion of molecular and anatomic information. Image for merging anatomic and functional information are dis- registration can be applied to compare anatomic information cussed, including: (a) description of computer algorithms with function, localize organs and lesions, and plan radiation for automatic retrospective image registration; (b) valida- therapy, biopsy, or surgery. Automatic volume-based image tion of accuracy for such algorithms; (c) visualization tech- registration techniques have been devised for both linear and nonlinear image alignment. Challenges remain in the validation niques for display of fused images; (d) clinical applications; of the accuracy of software registration. Image registration has and (e) comparison with hardware PET/CT technology. been applied clinically in neurology and oncology and may be particularly practical in radiotherapy applications. Potential new applications in cardiology could allow the combination of CT IMAGE REGISTRATION ALGORITHMS angiography with perfusion and viability images obtained by Several approaches have been proposed for the retrospec- PET, SPECT, or MRI. Software methods allow versatility in the tive automatic registration of multimodal images. Broadly, choice of modalities and facilitate retrospective and selective these registration algorithms could be grouped as feature application. Fully automatic registration algorithms are needed for routine clinical applications. Connectivity, compatibility, and based (using extracted image features) and volume based cooperation between various clinical departments are essential (using statistical voxel dependencies). The algorithms also for the successful application of software-based image fusion in could be classified as linear, when computed alignment a hospital setting. transformation between 3D image volumes is limited to Key Words: image registration; image fusion; PET/CT; nonlin- translation, rotation, and possibly scaling, or nonlinear, ear registration; mutual information which allow more complex transformations. Nonlinear J Nucl Med 2004; 45:36S–45S techniques can be feature based or volume based. Feature-Based Algorithms Feature-based registration algorithms seek to align cor- responding anatomic landmarks, organ surfaces, or other usion of images containing molecular and anatomic F features. Such techniques consist of 2 steps: (a) extraction of information could aid clinicians in a variety of clinical relevant features (points, contours, surfaces) from the im- applications, including comparison of anatomic information ages; and (b) spatial alignment of these features. Two rep- with function, localization and boundary definition of or- resentative examples of the feature-based approach are the gans and lesions, planning of radiation therapy and biopsy, “head and hat” method (1) and the “iterative closest point” and integration of PET or other functional modalities with method (2). Accurate image segmentation is required be- image-guided surgery. cause the registration relies on only the extracted features. Merging of multimodality images requires accurate im- Therefore, errors in image segmentation will inevitably lead age alignment, which is typically referred to as image reg- to errors in image registration. Although the registration of istration. Such image registration can be accomplished by accurately segmented surfaces is computationally straight- software algorithms that retrospectively align 3-dimensional forward, the identification of such surfaces may be difficult (3D) data acquired by stand-alone modalities to common and prone to errors. This step may require significant user spatial coordinates. Practical systems using software regis- interaction, even in the case of brain registration. Automatic tration algorithms for image fusion have been commercial- extraction of features needs to be customized for each imaging modality and for each organ of interest. Some Received Sep. 18, 2003; revision accepted Nov. 7, 2003. registration techniques rely on external fiducial markers that For correspondence or reprints contact: Piotr J. Slomka, PhD, Department can be identified and matched (automatically or manually) of Imaging, Cedars-Sinai Medical Center, A047 8700 Beverly Blvd., Los Angeles, CA 90048. on the acquired images (3,4). The main limitations of such E-mail: [email protected] marker-based approaches are the increased complexity of 36S THE JOURNAL OF NUCLEAR MEDICINE • Vol. 45 • No. 1 (Suppl) • January 2004 the imaging procedures and the lack of information about reliable than the registration of emission or transmission internal organ displacements. data alone (11). It is also possible to adjust the image resolution “on the fly” during the iterative search for the Volume-Based Algorithms best alignment. Initially, the stand-alone images can be More recently, volume-based image registration tech- grossly misaligned, and small matrix sizes are sufficient to niques have been introduced to maximize measures of sim- search for approximate alignment. The matrix size can be ilarity (cost function) between images. The proposed mea- progressively increased as the images become closely sures of the alignment quality include the standard deviation aligned, thus allowing fine adjustments. Such multiresolu- of the histogram (5), joint entropy (6), and mutual informa- tion techniques can decrease the time of the computations tion (MI) (7,8). In particular, methods implementing MI and may avoid entrapment in “local minima” (15). A measures (Fig. 1) have been proven versatile and successful method of search for the optimal value of the cost function in clinical applications (9). Volume-based techniques usu- may affect the calculation time or entrapments in local ally do not depend on image segmentation but exploit the minima. Furthermore, the number of registration parameters statistical voxel dependencies of the raw image pairs to find will determine the time required to find a solution. For the appropriate alignment. These techniques were initially example, the search for a transformation that includes ad- designed for the registration of MRI, CT, and PET images ditional scaling parameters will take longer than rigid-body of the brain (5,7,8) but recently have been extended to other registration. Current practical implementations allow fully organs (10–12). Volume-based techniques have been automated image registration of thoracic CT and PET in Ͻ1 shown to achieve better accuracy than surface-based meth- min (10,11) with accuracy at Ͻ1 cm. ods (13,14). Several possible modifications to volume-based algo- Nonlinear Registration rithms can improve their reliability and speed. For example, Image registration of thoracic and abdominal scans may PET transmission maps acquired with emission data can be require nonlinear transformation to compensate for changes used in the calculation of the cost function, because these in body configuration, breathing patterns, or movements of maps do not reflect the physiologic variations of the radio- internal organs. Nonlinear image alignment (image warp- isotope uptake. The emission images, however, contain ing) uses advanced interpolation schemes, such as the thin- important information that can correlate with anatomic fea- plate spline method (16) or piecewise-linear methods (12) tures on CT or MRI. Therefore, the registration of the adapted to 3 dimensions. A major difficulty with nonlinear combined emission and transmission data with CT is more warping is the determination of the correct nonlinear trans- FIGURE 1. Concept of image registration based on mutual information (MI) is explained using example of PET and CT. Separate PET and CT image intensity histograms are derived from PET and CT, which contain frequencies (f) of occurrence for specific voxel values in 3D volumes (p ϭ PET, c ϭ CT). Additional 2D image histogram is created from combination of PET and CT data, in which frequencies of occurrence for particular PET/CT voxel intensity pairs (p, c), both at same location, are calculated. Subsequently, PET and CT image entropies are calculated from PET and CT histograms, and 2D PET/CT histogram is used to calculate joint entropy. Joint entropy is smallest and, consequently, MI is largest when images are closely aligned and 2D histogram is least dispersed. Search is performed, which continuously modifies 3D shifts (X,Y,Z) and rotations (XY, XZ, YZ), each time transforming PET data. Although it is possible to perform image registration using joint entropy only, inclusion of separate PET and CT entropies is needed when portions of PET volume could move outside of overlapping field of view. SOFTWARE REGISTRATION AND FUSION • Slomka 37S form from the functional/molecular images. The transmis- the algorithms (18). Moreover, no available phantoms sim- sion images are acquired during the same tidal breathing as ulate complex patient motion. The accuracy of the volume- the emission images and, therefore, may provide an approx- based PET-to-CT image registration for the brain has been