Brain MRI Landmark Identification and Detection

Ali Asaei

In Partial Fulfilment of the Requirements for the Degree of MASTER OF SCIENCE

In The Department of Electrical and Computer Engineering

Thesis advisors: Dr. Babak Ardekani, The Nathan S. Kline Institute for Psychiatric Research Dr. Faramarz Vaziri, The Department of Electrical and Computer Engineering

State University of New York New Paltz, New York 12561

December 2015

Brain MRI Landmark Identification and Detection

Ali Asaei

State University of New York at New Paltz

We, the thesis committee for the above candidate for the Master of Science Degree, hereby recommend acceptance of this thesis

Babak Ardekani, PhD Faramarz Vaziri, PhD

Department of Electrical and Computer Engineering State University of New York at New Paltz

Approved by:

Babak Ardekani, PhD (Advisor)

Faramarz Vaziri, PhD (Advisor)

Approved on:

Babak Izadi, PhD (Chair) Abstract

Knowledge of the location of anatomical landmarks on the brain is important in neuroimaging. Applications include landmark-based image registration, segmentation of brain structures, electrode placement in deep brain stimulation, and prospective subject positioning in longitudinal imaging. Landmarks are speciﬁc structures with distinguishable morphological characteristics. In this study, we only consider point landmarks on magnetic resonance imaging (MRI) brain scans. The most basic method for locating anatomical landmarks on MRI is manual placement by a trained operator. However, manual landmark detection is a strenuous and tedious task, especially if large databases are involved and/or multiple landmarks need to be located. Therefore, automatic landmark detection on MRI has become an active area of research. Model-based methods are popular for detecting brain landmarks. Generally, model-based landmark detection includes a training set of MRI scans on which the location of certain landmarks are known, usually by manual placement. The location of landmarks on the training set is then used to derive and store models for individual landmarks. Then, when the same landmarks are to be located on a test MRI volume, the models are recalled and their information is used to automatically detect the landmarks.

In this thesis, we propose a new unsupervised landmark identiﬁcation method for the training phase of this process to replace manual landmark identiﬁcation on the training set of MRI volumes. This method employs an iterative algorithm for detecting a set of landmarks on the training set that are leave-one-out consistent. In addition, we suggest a detection method to locate the corresponding points on a given test volume. In this study, the method was implemented and applied to a dataset of sixty 3D MRI volumes. The training was performed on 30 volumes. The remaining 30 volumes were used as a test set on which the detection algorithm located the corresponding landmarks.

iii In the landmark identiﬁcation approach, a set of candidate seeds are necessary as the initial guesses of landmark positions. The position and number of the seeds are optional. In this study, we used 154 candidate seeds spread uniformly across the entire brain volume. All the identiﬁed and detected landmarks were inspected manually using a graphical user interface. To further evaluate the performance of the introduced method, we registered a set of 152 brain images to a reference space employing this method. Brain overlap of the registered volumes improved as a result of landmark based registration.

As a further application, we used landmark detection for rigid-body registration of longitudinal MRI volumes. These are MRI volumes scanned from the same individual over time. We show that landmark detection is a fast method that can be used to obtain a good initial rigid-body registration which can then be followed by ﬁne-tuning of the registration parameters.

Keywords: landmark identiﬁcation, landmark detection.

iv Acknowledgements

I would like to express my gratitude to my supervisors Dr. Babak Ardekani of the Nathan Kline Institute for Psychiatric Research (NKI) and Dr. Faramarz Vaziri of the State University of New York at New Paltz, for introducing me to the topic, engagement in the learning process, and their supports in the research work and academic studies of my thesis. I would like to thank Dr. Baback Izadi, the chair of the Electrical and Computer Engineering department for his support and for providing the ﬁnancial assis- tance. I also thank all those who have helped me in any respect during the completion of this work, especially my parents who have encouraged and supported me through life.

v Contents

List of Figures viii

List of Tablesx

1 Introduction1

1.1 Motivations...... 1 1.2 Challenges...... 2 1.3 Thesis overview...... 2

2 Background3

2.1 3D digital images...... 3 2.1.1 Spatial resolution...... 5 2.2 Image I/O...... 7 2.3 Image interpolation...... 8 2.3.1 Nearest neighbor interpolation...... 9 2.3.2 Trilinear interpolation...... 9 2.4 Coordinate systems...... 10 2.5 Image orientation...... 12 2.6 Spatial transformation...... 14 2.6.1 Linear transformation matrices...... 14 2.7 MRI Scans...... 17

vi Contents

2.8 Brain landmarks...... 18 2.9 Current brain landmark detection methods...... 20 2.9.1 Model-based landmark detection...... 22 2.9.1.1 Landmark template...... 22 2.9.1.2 Searching space...... 23 2.9.1.3 Similarity measure...... 24

3 Landmark identiﬁcation and detection methods 26

3.1 Method overview...... 26 3.2 Image normalization...... 28 3.3 Supervised landmark detection...... 29 3.4 Unsupervised landmark identiﬁcation...... 32 3.4.1 Initial seeds...... 34

4 Registration method 35

5 Results 39

5.1 Image data...... 39 5.2 Test setup...... 39 5.3 Evaluation...... 41 5.3.1 Overlap Index...... 43 5.3.2 Application...... 45

6 Conclusion 47

Bibliography 47

A Least-squares aﬃne transformation estimationI

B Registration FiguresIII

vii List of Figures

2.1 A rectangular parallelepiped representing the 3D imaging FOV.....3

2.2 (a) Discrete sampling scheme. (b) A digital image with the size of nx =

5, ny = 7, nz = 3 voxels...... 4 2.3 Images of the same object with diﬀerent resolutions [1]...... 6 2.4 (a) Schematic of an MR image slice [2]. (b) Parallel imaging slices to cover the brain entire volume [3]...... 6 2.5 Mapping between voxel indices [i, j, k] and memory locations v.....8 2.6 Schematic of a 3D sampling grid [4]...... 9 2.7 Three medical imaging coordinate systems and corresponding axes [5].. 11 2.8 (a) Anatomical reference planes [6]. (b) Anatomical axes and cross sections of the brain [7]...... 12 2.9 A slice from a PIL image...... 13 2.10 Brain mid-sagittal scheme and some anatomical landmarks [8]...... 19 2.11 Spherical voxel templates...... 23

3.1 Framework of the proposed method...... 26 3.2 Flow chart of the identiﬁcation algorithm...... 27 3.3 Flow chart of the identiﬁcation algorithm...... 28

viii List of Figures

3.4 The MSP of an MR image in PIL space. (a) AC (green) and PC (red) landmarks are detected using the algorithm in [9]. (b) Reoriented image according to the detected AC and PC...... 29

3.5 Flowchart for computing aﬃne transformation TLM and TA...... 30 3.6 Fuzzy brain template...... 34

4.1 Flowchart of the proposed registration method...... 35 4.2 The 8 selected landmarks on the MSP...... 36

4.3 Flowchart for computing aﬃne transformation TLM and TA...... 37

4.4 The sample plot of similarity function f versus dx, dy, dz, θx, θy, and θz. 38

5.1 An example of an identiﬁed landmark in 8 sample images. The red crosses (ﬁrst row) and the blue crosses (second row) display the seeds and converged points in the coronal plane, respectively...... 40 5.2 First row displays the average volume in PIL space. Second row displays the average volume in the original space...... 43 5.3 (a) AC (green) and PC (red) on MSP. (b) 8 landmarks on MSP..... 45 5.4 MSP of 6 sample images and the 8 detected landmarks...... 46

B.1 Same cross sections of original longitudinal images from one subject...III B.2 Same cross sections of registered longitudinal images from one subject.IV

ix List of Tables

5.1 Number of the identiﬁed (total or symmetric) and detected landmarks on 30 training images and 30 test images for 3 template radii...... 41

x 1

Introduction

1.1 Motivations

MRI scanners provide high resolution 3D digital image data for clinical assessment or research studies. Manual identiﬁcation of anatomical landmarks on brain MR images is a common practice in neuroimaging. Although, manual labeling is still the gold standard in landmark detection methods, manual procedures are labor-intensive and subjective. Therefore, a computer-aided method to automatically detect landmarks would resolve the diﬃculties of manual labeling. Automatic model-based landmark detection algorithms look for a pattern in a given test image. Hence, a learning algorithm is required to extract the pattern from a training set. Landmark patterns are created from the information provided by the training set such as image intensity distribution and have common features among the training volumes.

An objective of this thesis is to provide an automated, reliable, and practical method for identifying a set of landmarks in a set of training volumes of human brain MRI scans. A detection method is also proposed to locate landmark patterns in a given test image. This method is able to identify a set of landmarks in any region of the brain and could be employed for various applications such as deﬁning a reference space, image registration, and structural change detection. The landmark identiﬁcation

1 1. Introduction method discussed in this thesis was originally developed and optimized for brain MR images, but is applicable for other medical imaging ﬁelds and modalities.

1.2 Challenges

In this study, the first challenge stems from differences in input volumes. Images have different orientation, spacing, and origin. The proposed method assumes images are spatially normalized, therefore, input volumes must be registered to a reference space before applying the identification or detection methods. Another major challenge springs from the definition of landmarks. A proper landmark has common characteristics among the subjects and is distinguishable in each image. For example few anatomical landmarks on the brain are salient and can be located accurately. The vast majority of the landmarks have localization uncertainties, hence, precise labelling is often very difficult even for a human expert. In this thesis, approaches to solve the above mentioned problems are presented and possible improvements are discussed.

1.3 Thesis overview

In this thesis an automatic landmark identiﬁcation and detection method is presented. The motivation and challenges for this study are noted in Chapter1. Chapter2 provides background knowledge on medical imaging. Chapter3 describes the identiﬁcation and detection methods in detail. Chapter5 presents the results and evaluates the performance of the proposed method. Conclusions and possible future works are mentioned in Chapter6.

2 2

Background

2.1 3D digital images

Consider a function f(x, y, z) of three continuous variables (x, y, z) deﬁned on the rectangular parallelepiped domain shown in Figure 2.1, that is, the set of all (x, y, z) points such that:

−FOVx/2 < x < FOVx/2

−FOVy/2 < y < FOVy/2 (2.1)

−FOVz/2 < z < FOVz/2

where FOVx, FOVy, and FOVz represent the imaging ﬁeld-of-view (FOV) in x, y, and z directions, respectively.

FOVz

FOV x y y

FOVx

Figure 2.1: A rectangular parallelepiped representing the 3D imaging FOV.

3 2. Background

A 3D digital image can be regarded as a set of discrete samples from f(x, y, z) on a regular 3D grid of points. In particular, we assume that nx, ny, and nz samples are acquired along the x, y, and z directions, respectively, at corresponding equally- spaced intervals of ∆x, ∆y, and ∆z millimeters. The numbers of samples and sampling intervals are selected to uniformly cover the entire 3D FOV in Figure 2.1, that is,

FOVx = nx∆x, FOVy = ny∆y, and FOVz = nz∆z. The 3D digital image thus obtained can be considered as a function f[i, j, k] of discrete variables [i, j, k], where i = 0, 1, . . . , nx − 1, j = 0, 1, . . . , ny − 1, and k = 0, 1, . . . , nz − 1, are integer indices of samples along the x, y, and z directions, respectively. It is said that the digital image

3 has a matrix size of nx × ny × nz voxels and a voxel size of ∆x × ∆y × ∆z mm . The sampling scheme described above is shown in Figure 2.2a for a 2D case with nx = 14 and ny = 7. Figure 2.2b shows a 3D case with nx = 5, ny = 7, nz = 3 voxels.

∆x

i ∆y j FOV O x y y

FOVx

(a) (b)

Figure 2.2: (a) Discrete sampling scheme. (b) A digital image with the size of

nx = 5, ny = 7, nz = 3 voxels.

In the sampling scheme described above and schematically shown in Figure 2.2a, the precise relationship between indices [i, j, k] and real world points (x, y, z) in the

4 2. Background imaging FOV (Figure 2.1) is:

x = i∆x − ∆x(nx − 1)/2

y = j∆y − ∆y(ny − 1)/2 (2.2)

z = k∆z − ∆z(nz − 1)/2 or in matrix notation:

        x ∆x 0 0 i ∆x(nx − 1)/2                          y  =  0 ∆y 0   j  −  ∆y(ny − 1)/2  (2.3)                 z 0 0 ∆z k ∆z(nz − 1)/2 Thus nx − 1 ny − 1 nz − 1 f[i, j, k] = f(i∆x − ∆x , j∆y − ∆y , k∆z − ∆z ) (2.4) 2 2 2 Note that parenthesis, as in f(.), are used to enclose continuous variables and square brackets, as in f[.], are used to enclose discrete variables.

2.1.1 Spatial resolution

Spatial resolution refers to size of the elements constructing an image. These elements are referred to as pixels in 2D and voxels in 3D images. An image with smaller voxel size has higher spatial resolution. Assume, there are two images acquired from the same subject and similar FOV. The one with higher resolution has more and smaller voxels and provides more precise details of the scanned subject. Hence, higher resolution images are desired. Beside the technical limitations, the only drawback for obtaining high resolution images is cost. Figure 2.3 displays images of the same object with diﬀerent resolutions. The left image has low resolution and lacks in details. However, as one progresses to right, the spatial resolution becomes higher and results in more accurate images.

5 2. Background

Figure 2.3: Images of the same object with diﬀerent resolutions [1].

In MRI, the scanned volume is sliced into parallel planes as shown in Figure 2.4b. A slice is selected by applying a gradient to the static magnetic ﬁeld and the RF signals are collected for that individual plane. Each slice taken through the brain, creates a 2D image and contains the collected information. Pixels in each of the 2D images are encoded by applying the phase and frequency encoding pulses [10]. All the 2D images are stacked together to create a 3D image. Figure 2.4a represents a slice of an MR image. The slice thickness is the interval between the slices. Thicker slices tend to produce volume averaging but have more signal to noise ratio. Although, the penalty is losing the accuracy in small structures [2].

(a) (b)

Figure 2.4: (a) Schematic of an MR image slice [2]. (b) Parallel imaging slices to cover the brain entire volume [3].

6 2. Background

Assume the scanned volume slices are perpendicular to the z axis. Then, nz represents number of the slices and slice thickness is denoted by ∆z. Each slice contains nx × ny voxels and the covered area is FOVx × FOVy. In this case, the voxel size is

∆x × ∆y × ∆z and the entire scanned volume is FOVx × FOVy × FOVz.

2.2 Image I/O

Storage of an image of matrix size nx ×ny ×nz requires nv = nxnynz computer memory locations, where nv is the number of image voxels. Let the speciﬁc memory locations be indexed by v ranging from 0 to nv −1. The convention in this thesis is to write voxel values f[i, j, k] into memory locations v = i + jnx + knp, where np = nxny. Conversely, values in memory locations v are read as voxel values f[i, j, k], where:

i = v mod nx

j = b(v mod np)/nxc (2.5)

k = bv/npc brc denotes the ﬂoor function which gives the largest integer not greater than r, and “mod” denotes the modulo operator.

The mapping scheme between voxel indices [i, j, k] and memory locations v is shown schematically in Figure 2.5. In this example nx = 4, ny = 3, and nz = 2. Note for example that voxel value at [i, j, k] = [3, 1, 1] is f[3, 1, 1] = 29 and stored in memory location v = i + jnx + knp = 3 + 4 + 12 = 19. Conversely, we note that the voxel location [i, j, k] of say memory index v = 22 can be found from (2.5) to be: i = v mod nx = 22 mod 4 = 2, j = b(v mod np)/nxc = b(22 mod 12)/4c = 2, and k = bv/npc = b22/12c = 1. Thus the value at memory location 22 which happens to be 94 is stored at voxel [2, 2, 1], that is, f[2, 2, 1] = 94.

7 2. Background i i i i i i i i 0 = 1 = 2 = 3 = 0 = 1 = 2 = 3 =

j =0 18 75 54 38 j =0 4459 14 26 j =1 91 11 43 65 j =1 98 65 54 29 j =2 23 88 71 34 j =2 45 76 94 31 k =0 k =1 1875 54 38 91... 29 45 7694 31 =0 =1 =2 =3 =4 = 19 = 20 = 21 = 22 = 23 v v v v v v v v v v

Figure 2.5: Mapping between voxel indices [i, j, k] and memory locations v.

2.3 Image interpolation

As described previously, a 3D digital image stores samples of f(x, y, z), a function of continuous spatial variables (x, y, z), at nx × ny × nz grid points. Thus, values of f(x, y, z) are only known at these discrete locations. Interpolation methods are used to estimate the value of f at arbitrary locations (x, y, z) within the 3D FOV that generally do not coincide with a grid location. For example, value of point P in Figure 2.6 is unknown since it is not located on a sampling point of the 3D grid. Two interpolation methods are described below. In this study, the trilinear interpolation was utilized to estimate the value of any point in MR volumes.

8 2. Background

Figure 2.6: Schematic of a 3D sampling grid [4].

2.3.1 Nearest neighbor interpolation

The simplest method of interpolation is the nearest neighbor interpolation which simply estimates the value of f(x, y, z) (denoted by fˆ(x, y, z)) to be the value of the nearest sampled point (voxel). The value of the nearest voxel is found by ﬁrst inverting (2.3) to obtain:       ˜i 1/∆x 0 0 x + ∆x(nx − 1)/2                    ˜j  =  0 1/∆y 0   y + ∆y(ny − 1)/2  (2.6)             k˜ 0 0 1/∆z z + ∆z(nz − 1)/2

Then fˆ(x, y, z) is taken to be f[i, j, k] where [i, j, k] are integer indices that are nearest to (˜i, ˜j, k˜). For example, if (˜i, ˜j, k˜) = (5.4, 3.9, 12.6), then [i, j, k] = [5, 4, 13] and fˆ(x, y, z) = f[5, 4, 13].

2.3.2 Trilinear interpolation

Trilinear interpolation is used extensively in this study because it is a good compromise between accuracy and speed. In this method, f(x, y, z) for an arbitrary point (x, y, z) is

9 2. Background approximated by a weighted average of the eight grid points that surround the (˜i, ˜j, k˜) obtained from (2.6). Speciﬁcally, let i = b˜ic, j = b˜jc, k = bk˜c, wi = ˜i − i, wj = ˜j − j, and wk = k˜ − k, then f(x, y, z) is approximated by:

fˆ(x, y, z) = (1 − wi)(1 − wj)(1 − wk)f[i, j, k]

+ (1 − wi)wj(1 − wk)f[i, j + 1, k]

+ (1 − wi)(1 − wj)wkf[i, j, k + 1]

+ (1 − wi)wjwkf[i, j + 1, k + 1]

+ wi(1 − wj)(1 − wk)f[i + 1, j, k]

+ wiwj(1 − wk)f[i + 1, j + 1, k]

+ wi(1 − wj)wkf[i + 1, j, k + 1]

+ wiwjwkf[i + 1, j + 1, k + 1] (2.7)

Note that if (˜i, ˜j, k˜) corresponded to a grid point [i, j, k] exactly, then wi = wj = wk = 0 and fˆ(x, y, z) = f[i, j, k].

2.4 Coordinate systems

Figure 2.7 shows three coordinate systems that are involved in medical imaging. The world coordinate system (Figure 2.7a), is typically a Cartesian coordinate system and the axes are denoted as x, y, and z. The parameters, ∆ (voxel size) and FOV (ﬁeld of view) are deﬁned in this system. Position of each sample point in this system is relative to the scanner.

The anatomical coordinate system (Figure 2.7b), is the common coordinate system to describe the standard anatomical position of a human. It consists 6 directions as described in Section 2.5 and 3 planes. Figure 2.8a shows the three basic anatomical planes on a human body model. The anatomical planes and directions are displayed

10 2. Background on the brain schematic in Figure 2.8b. The image coordinate system (Figure 2.7c), determines the position of a voxel in a 3D image as discussed in Section 2.2.

(a) World space (b) Anatomical space (c) Image space

Figure 2.7: Three medical imaging coordinate systems and corresponding axes [5].

To reference an exact location or part of the human body, there are standard terms and anatomical references which form an anatomical space. This unique space is vital in medical imaging and prevents any possible confusions. Three basic anatomical planes are [11]:

• Axial plane: parallel to the ground and separates the top and bottom.

• Sagittal plane: perpendicular to the ground and separates the left and right.

• Coronal plane: perpendicular to the ground and separates the front and back.

11 2. Background

(a) (b)

Figure 2.8: (a) Anatomical reference planes [6]. (b) Anatomical axes and cross sections of the brain [7].

2.5 Image orientation

In brain imaging the x, y, and z axes usually point in one of six directions with respect to the subject’s anatomy: subject’s Left, Right, Anterior (towards the front of the brain also known as the rostral direction), Posterior (towards the back of the brain also known as the caudal direction), Superior (towards the top of the brain also known as the dorsal direction), or Inferior (towards feet also known as the ventral direction).

To specify the image orientation, we used a three-letter code using letters from the set {L,R,A,P,S,I}.For example, a ‘PIL’ image code means that the x axis points Posteriorly, the y axis points Inferiorly, and the z axis points towards subject’s Left. As another example, an image is said to be ‘RAS’ if the x axis points towards the subject Right, the y axis points Anteriorly, and the z axis points Superiorly. There are a total for 48 possible image orientation codes. An example slice from a PIL image is displayed in Figure 2.9. A volume in PIL space

12 2. Background has the following properties:

1. The z = 0 plane is the brain’s MSP.

2. The origin (ﬁeld-of-view center) is the mid-point between the AC and PC on the MSP.

3. The x axis is on the MSP, parallel to the AC-PC line and points posteriorly.

4. The y axis is on the MSP, perpendicular to the AC-PC line and points inferiorly.

5. The z axis is perpendicular to the MSP and points from subject’s right to left.

o x

PC AC y

Figure 2.9: A slice from a PIL image.

Images are acquired in diﬀerent orientations. Reorienting an images into a new space such as PIL space is performed by a spatial transformation. Spatial transformation maps an image from the subject space to the target space that are discussed in Section 2.6.

13 2. Background

2.6 Spatial transformation

A spatial transformation deﬁnes a relationship between points on a subject space and the corresponding points on a target space. Since MR images are represented as 3D matrices, we only consider 3D spaces. Although, the equations can be generalized for higher dimensions. A general R3 → R3 mapping function can be expressed as:

(x0, y0, z0) = X(x, y, z),Y (x, y, z),Z(x, y, z) (2.8) where (x, y, z) and (x0, y0, z0) represent the coordinates of the source and target points, respectively. X,Y, and Z are the mapping functions which construct the spatial transformation. If the points of the source volume are mapped to the points of the target volume, the transformation is referred as the forward mapping. Similarly, if the mapping is a bijection function, the inverse mapping will map the points from the target volume to the reference volume [12].

2.6.1 Linear transformation matrices

A linear transformation is a particular type of spatial transformation when there is a geometric relationship between the reference and the mapped points. This mapping is always a bijection function, in other word, it’s invers exits. Some of the basic linear transformations in image processing are translation, rotation, scaling, and shearing.

linear transformations are divided into 4 main categories: rigid, projective, and affine. In rigid transformation distance between corresponding points in subject and target space remains the same. A rigid transformation can be represented as a combination of a translation and a rotation. If any line is mapped onto a line, it is a projective transformation. If parallel lines stay parallel, it is an affine transformation. Any category can be a subset of the other categories as ( rigid ∈ projective ∈ affine

14 2. Background

). For instance, the projective transformation is a special kind of affine transformations. Combination of two or more transformations, is a transformation. The combined transformation will be categorized as the biggest category of individual transformations [13]. For example, combination of a rigid, affine, and elastic transformations is an elastic transformation. Likewise, combination of two affine transformations, is an affine too.

An aﬃne transformation {f : ~p → p~0} is the combination of a linear transformation (multiplying by matrix A) and a translation (adding by matrix d~) as stated in Equation 2.9. Where, ~p and p~0 are the reference and mapped vectors.

p~0 = f(~p) = A~p + d.~ (2.9)

In this study, vectors ~p and p~0 are 3×1 matrices that indicate the location of corresponding voxels in the subject and target images, respectively. An aﬃne transformation (f) can be represented by a single matrix multiplication as following:

    0 x x           ~0 ~     p A3x3 d3x1 ~p   ~0  0       ~p = y , p = y  ,   =     (2.10)     1 0 0 0 1 1    0 z z | {z } | {z } |{z} P 0 T P

T is a 4×4 aﬃne transformation matrix comprised of 12 parameters that maps point P to P 0. Where P and P 0 are 3D vectors represented in the 4D vectors.

      x0 x A A A d      11 12 13 1        0     y  y A21 A22 A23 d2   0     P =   ,P =   ,T =   (2.11)  0     z  z A31 A32 A33 d3             1 1 0 0 0 1

In this section some geometric aﬃne transformations in R3 are presented using a 4×4 matrix. Equation 2.12 displays a translation transformation, with oﬀset values of

15 2. Background

d1, d2, and d1 on x, y, and z directions, respectively.       0 x 1 0 0 d1 x              0     y  0 1 0 d2 y         =     (2.12)  0     z  0 0 1 d3 z             1 0 0 0 1 1

Scaling of the image by the scale factor of A11,A22, and A33 on x, y, and z axes, respectively, is be presented as Equation 2.13. Any negative scale factor means reﬂecting with respect to the corresponding axis.       0 x A11 0 0 0 x              0     y   0 A22 0 0 y         =     (2.13)  0     z   0 0 A33 0 z             1 0 0 0 1 1

Shearing transformation applies changes in each dimension based on a linear combination of other dimension. The general shearing in a R3 space transforms a cube into a general parallelepiped.       0 x 1 A12 A13 d1 x              0     y  A21 1 A23 d2 y         =     (2.14)  0     z  A31 A32 1 d3 z             1 0 0 0 1 1

Let the rotation transformation matrix around vector ~v, by the rotation angle of

α, be denoted as Rv(α). This rotation can be represented as a combination of rotations around the x, y, and z axes, by the angle of θx, θy, and θz, respectively. Such as:

Rv(α) = Rx(θx)Ry(θy)Rz(θz) (2.15)

The rotation transformations around the coordinate system axes can be calculated

16 2. Background as following:       1 0 0 0 cos(θy) 0 sin(θy) 0 cos(θz) −sin(θz) 0 0                   0 cos(θx) −sin(θx) 0  0 1 0 0 sin(θz) cos(θz) 0 0                   0 sin(θx) cos(θx) 0 −sin(θy) 0 cos(θy) 0  0 0 1 0             0 0 0 1 0 0 0 1 0 0 0 1 | {z } | {z } | {z } Rx(θx) Ry(θy) Rz(θz) (2.16)

2.7 MRI Scans

MRI is a medical imaging modality that uses magnetic ﬁelds and radio frequency (RF) waves to produce cross-sectional images from internal organs of the body. An MRI scanner outputs a multidimensional image that represents spatial samplings of a physical quantity. These images can be 2D or 3D acquired at any orientation. Unlike positron emission tomography (PET) and some other imaging techniques, in MRI no injection of radioactive isotopes is necessary. MRI operates in RF range. Therefore, ionizing radiation is not involved. An MR image contains very rich information contents of the scanned subject which depend on parameters such as T1 and T2 relaxation times, and the nuclear spin density ρ. Therefore, the information content of MRI is diﬀerent from other medical modalities such as PET or CT (computed tomography) [14].

An MR scanner is constructed from four main parts: the main magnet, the shim coils, the gradient system, and the RF system [14]. The main magnet provides a strong static magnetic field (B0). When the patient is placed in an MR scanner, the magnetic moments of protons in the body aligns with the direction of the B0 field. The main magnet field is usually not homogeneous enough, hence, a set of shim coils are included to provide the desire homogeneity. The gradient system includes three gradient coils which produce magnetic field gradients in 3 orthogonal orientations. Gradient coils

17 2. Background encode the position of a sampling point across the scanned volume. The RF system includes a transmitter and receiver. The transmitter coil generates a rotating magnetic

field (B1). This magnetic field alters the magnetization alignment of protons relative to the B0. When the magnetic field from an RF pulse is ended, protons reorient to their equilibrium orientation and emit RF signals. These signals are collected with the RF receiver and utilized to create an MR image. The image quality also depends on operator-selected parameters such as repetition time (TR), echo time (TE), and flip angle (α). TR refers to the time interval between the RF pulses. The interval between sending the signal and receiving the echo signal is denoted as TE [15].

2.8 Brain landmarks

In medical imaging, landmark is a structure that helps to navigate and perceive a spe- cific position in the images. In neuroimaging, a landmark refers to a brain structure which is detectable and has similar characteristics in different subjects. A landmark can be a volume, surface, curve, or a point. The following examples are classified as brain landmarks but they have different structure types [7]. Corpus callosum is bundle of neural fibers that connects the right and left hemispheres. Hippocampus is a small region of brain that is primarily associated with memory and spatial navigation. MSP is a plane that divides the right and left sides of the brain into two cerebral hemispheres. AC and PC are point landmarks on the MSP as shown in Figure 2.9.

In this study we only consider point landmarks. Point landmarks represent points in the image and can be divided into two groups [16]. The ﬁrst group is Anatomical point landmarks. These landmarks have a distinctive structural characteristics and low localization uncertainty in 3 dimensions. A neuroscientist or morphologist is able to locate the anatomical landmarks visually on a 3D MR image. Figure 2.10 illustrates

18 2. Background some of the anatomical landmarks and structures on the MSP. The second group is Geometrical point landmarks. These landmarks have speciﬁc geometric properties that are not necessarily distinctive for an operator in all three spatial dimensions of an MR volume. These landmarks can be localized by computer-aided methods to detect the similar patterns.

Figure 2.10: Brain mid-sagittal scheme and some anatomical landmarks [8].

19 2. Background

2.9 Current brain landmark detection methods

Many of the algorithms in brain imaging are based on landmarks, such as landmark- based registration methods [17]. Hence, an accurate method for locating the landmarks is crucial. Commonly, anatomical landmarks are labeled manually on an image display software tool such as like ITK-SNAP [18]. Manual procedures are labor-intensive and they suﬀer from low intra/inter-rater reliability and results are not reproducible. There- fore, semi-automated or fully-automated procedures for detecting brain landmarks have attracted attention in this ﬁeld.

MSP is a hypothetical plane that divides the brain into two symmetric hemispheres. Detecting the MSP is one of the preliminary processes in most landmark detection algorithms. Symmetry-based [19, 20, 21] and feature-based [22] methods are currently the most typical approaches for MSP detection. In a symmetry-based method, MSP is deﬁned by the optimum point of symmetry measure function in a parameter space. The symmetry criteria is measured from both sides of the hypothetical plane. Parameter space is a set of parameters that describes the MSP. Three parameters are required to deﬁne a plane in 3D space. Usually, MSP detection methods do not make any assumptions about image orientation, therefore, algorithms are applicable to any given image [7].

AC and PC are very important anatomical landmarks. Some methods and atlases are based on position of the AC/PC, such at the Talairach coordinate system. Ardekani and Bachman [9] proposed a fully automatic model-based algorithm for AC/PC detection. The approach consists of training and detection parts. The detection approach utilizes a template matching algorithm. Templates are obtained from model images with 3 known landmarks including AC/PC. In medical imaging, 3D template matching

20 2. Background techniques are frequent for landmark detection methods. Although, template matching algorithms can be highly computational. Wörz and Rohr [23] proposed an algorithm by ﬁtting a 3D parametric intensity model. Depending on size of the landmark template, the searching space is adjusted. This method improves the accuracy and prevents lead- ing to local extrema of the similarity measure functions and decreas the execution time drastically.

Some landmark detection methods are based on diﬀerential operators and use multiple order partial derivative of the image, therefore, they are relatively complex and time consuming. On the other hand, having plenty of information, provides a precise detection result. Rohr [24] proposed four 3D diﬀerential operators. The operators are generalized from existing 2D operators and only require low order partial derivatives to detect points with high intensity variations.

Atlas based registration is one of the most common techniques for segmentation or landmark detection. A brain atlas is an image built from one or more models. Landmarks and structures of the brain are located and labeled in the training models manually. In this method, ﬁrst, a given image is registered to an atlas space. Since, the position of the landmarks are known in the atlas space, landmarks can be located in the original space by using the registration transformation from the atlas space to the original space. The major drawbacks of this method are long execution time and dependency on the atlas models [7]. Therefore, researchers [25] proposed multi-atlas registration methods to decrease the biased inﬂuences of the training models.

21 2. Background

2.9.1 Model-based landmark detection

Model-based landmark detection methods look for the best matching point in a search space that ﬁts the model for the corresponding landmark. Details of this method are presented in this section.

2.9.1.1 Landmark template

In model-based detection methods, a landmark model includes a feature vector that deﬁnes the characteristics of that point. The feature vector can be considered as a vector-valued function extracted from the neighboring voxels at each point. To determine the neighbor voxels, we used a template. This template is similar for all the points in the image, although the feature vector values are diﬀerent. In our implementation, a spherical template was employed to decrease the directional biases compare to the cubical templates. The spherical template for point p = [ic, jc, kc], includes all voxels for q 2 2 2 2 2 2 which ∆x (i − ic) + ∆y (j − jc) + ∆z (k − kc) ≤ r, where [i, j, k] is the position of a neighbor voxel and r is the radius of the sphere.

Figure 2.11 shows two spherical templates with radii of 2 and 7. Depending on the image, size and shape of the templates play an important role. Each template should include enough information based on the application. Also, size of the template has a direct correlation with execution time. This can be an important factor in high computational methods or online algorithms. For instance, the template with radius of 2 includes 33 voxels while the template with radius of 7 includes 1419 voxels. We tested the algorithm with diﬀerent template sizes. Results are discussed in chapter5.

22 2. Background

(a) r=2 (b) r=7

Figure 2.11: Spherical voxel templates.

2.9.1.2 Searching space

When the landmark template and feature vector are deﬁned, the detection method will look for the best matching point in a searching space. Searching space is a parameter space denoted as S ∈ Rl, where l is number of the searching parameters. For instance, the searching parameters can be the position of the landmark in a 3D image. In this case, l = 3 and the searching parameters are translations along the axes. If the orientation is considered in addition to the position, then l = 6 and rotations around the axes will be evaluated as well. The searching region can be expanded by adding scale factor or other parameters.

In the proposed method, ﬁrst images are spatially normalized and reoriented to the reference space using the method in Section 3.2. Therefore, all the landmarks will be in a similar but not exact orientation and position. In our implementation, translation along the axes are the searching parameters and we do not consider rotation.

Accordingly, the searching space S ∈ R3 is a sphere around the searching center, with radius of 10. The radius constraint limits the searching space to 4169 voxels around the expected center. This limitation avoids ﬁnding local extrema in other parts of the

23 2. Background image and speeds up the searching time. Applying this constraint is possible due to the preliminary normalization. The feature vector for all the points in the searching space are calculated. The point with the most similar feature vector to the model feature vector is detected as the landmark.

2.9.1.3 Similarity measure

A similarity measure is needed to measure the similarity between two feature vectors. There is no single deﬁnition for the similarity measure. Also, a similarity measure can be interpreted as the inverse of a distance measure. Therefore, to ﬁnd the best matching feature vector there are two approaches, maximizing the similarity function or minimizing the distance function. Cha [26] reviewed various distance/similarity measures that are applicable for comparing two probability density functions. One of the simplest distance measures is the Euclidean distance as:

s X 2 f(x, y, z) − g(x, y, z) (2.17) x,y,z As mentioned, various similarity measures exist but in this study we used the cross correlation (CC) as the similarity measure of two feature vectors. CC is a common method in image processing which is simple and fast. Any other similarity measure can be substituted for CC without changing the rest of the algorithm. MR images can have diﬀerent intensity ranges due to diﬀerent scanner setups. Therefore, before measuring the CC, vectors should be normalized. This is by subtracting the mean and dividing by the standard deviation. Let f(x, y, z) and g(x, y, z) be two vector-valued functions.

If for vector function f, f represents mean and σf represents standard deviation, then, the CC of f and g is given by the equation

1 X (f(x, y, z) − f)(g(x, y, z) − g) CC(f, g) = (2.18) N x,y,z σf σg where N is the dimension of each vectors and

24 2. Background

v u u 1 X 2 1 X σf = t (f(x, y, z) − f) f = f(x, y, z). (2.19) N x,y,z N x,y,z

25 3

Landmark identiﬁcation and detection methods

3.1 Method overview

In this thesis we introduce a landmark identiﬁcation and detection method. The frame work of the proposed method includes two main phases, the training and the application.

Figure 3.1: Framework of the proposed method.

The training stage is an oﬀ-line process to identify landmarks on a set of training volumes. Currently, an operator locate the landmarks on the training set manually. In this thesis, we propose a method substitute manual training phase with an automatic computer-aided method. Figure 3.2 shows a schematic diagram of the training phase. All the images in the training set are ﬁrst normalized and reoriented to a standard PIL

26 3. Landmark identification and detection methods space as explained in Section 2.5. The output of this phase is the model of the identified landmarks including the feature vectors and the searching centers. The landmark models are saved and later used in the application stage. The unsupervised landmark identification method is explained step by step in Section 3.4.

Figure 3.2: Flow chart of the identiﬁcation algorithm.

The application phase is a supervised landmark detection method that detect the identiﬁed landmarks on a new test image. Figure 3.3 shows a schematic diagram of the application phase. Similar to the identiﬁcation method, the test image should be reoriented to the PIL space at the beginning. The output of the detection phase is a set of locations representing the position of each landmark. This set of landmarks can be employed for any application such as registration. The supervised landmark detection method is presented in Section 3.3.

27 3. Landmark identiﬁcation and detection methods

Figure 3.3: Flow chart of the identiﬁcation algorithm.

In this study, we checked all the identified and detected landmarks using a graphical user interface program. This part can be done for inspection purposes but it is not required for utilizing the identification and detection methods. In this chapter, first, the image normalization method is described.

3.2 Image normalization

The algorithms presented in this thesis relies on a fast and fully automatic method for transforming an arbitrarily oriented volume (V ) to a standard posterior-inferior-left (PIL) space. The PIL transformation can be determined by first finding the (MSP) and then locating the AC and PC on it. Multiple algorithms have been published to perform these tasks automatically and efficiently on T1-weighted structural MRI scans. In our implementation, we employed the algorithm proposed by Ardekani et al in [19]

28 3. Landmark identiﬁcation and detection methods and [9]. The MSP and AC/PC locations are used to determine a rigid-body transformation

TPIL that would transform V to PIL space. The detected AC and PC are shown in Figure 3.4a. Also, Figure 3.4b displays the MSP of an MRI volume in PIL space.

(a) (b)

Figure 3.4: The MSP of an MR image in PIL space. (a) AC (green) and PC (red) landmarks are detected using the algorithm in [9]. (b) Reoriented image according to the detected AC and PC.

3.3 Supervised landmark detection

This section describes the supervised landmark detection block in the ﬂowchart in Fig- ure 3.5. Assume that we have a set of M training volumes V = {Vm}(m = 1, 2, ..., M) which are spatially normalized in PIL space as explained in section 2.5 and that the location of a given landmark n is known on each volume.

29 3. Landmark identiﬁcation and detection methods

Let these points be represented by set Qn = {qmn} where qmn denotes the location of landmark n on the mth training volume. Section 3.4 presents an automatic method for deﬁning qmn on {V m}. But for now, assume that Qn = {qmn} are given.

Figure 3.5: Flowchart for computing aﬃne transformation TLM and TA.

This section describes a method for estimating the location of landmark n on a test volume Vt in PIL space given the knowledge of (V,Qn). The landmark detection algorithm can be thought of as a function pn = D(Vt,V,Qn), where pn denotes the detected location of landmark n on the test volume Vt. Based on Qn, the expected location of landmark n in PIL space can be estimated to be:

M 1 X qn = qmn (3.1) M m=1

Let Ω(qn) denote a neighborhood of point qn. The landmark detection algorithm assumes that pn ∈ Ω(qn). Furthermore, the landmark detection algorithm relies on a feature vector that can be obtained for any point p on a volume V . The feature vector can be considered a vector-valued function f(V, p) ∈ Rl that deﬁnes an l-dimensional

30 3. Landmark identiﬁcation and detection methods vector extracted from volume V at point p. Given these deﬁnitions, the landmark detection algorithm can be written as:

" M # 1 X pn = arg max f(Vt, p) ∼ f(Vm, qmn) (3.2) p∈Ω(qn) M m=1

where f(Vt, p) ∼ f(Vm, qmn) denotes the degree of similarity of the feature vector at point p on the test volume Vt and the feature vector at point qmn on the training volume Vm. The summation calculates the average similarity between f(Vt, p) and f(Vm, qmn) for m = 1, 2, ..., M. The algorithm detects the landmark pn on Vt as the point in the neighborhood Ω(qn) with maximum average similarity with points Qn on V .

In our current implementation, as feature vector f(V, p), we simply take the gray levels of V in a neighborhood ω(p) of point p and normalize them so that the elements of vector f(V, p) have zero mean and unit variance. The neighborhood ω(p) is explained in subsubsection 2.9.1.1 as the landmark template. It is possible to include other information in the feature vector function f(V, p), for example, intensity gradient and/or Laplacian values measured at diﬀerent resolution levels. However, this line of research has not been pursued in the current work.

As the similarity operation “∼” in Equation 3.2 we simply use the CC of the feature vectors. The advantage of this choice is that, because the CC is a linear operation, Equation 3.2 simpliﬁes to:

h i pn = arg max f(Vt, p) ∗ f¯n (3.3) p∈Ω(qn) where

M 1 X f¯n = f(Vm, qmn). (3.4) M m=1

31 3. Landmark identiﬁcation and detection methods

The computational advantage of Equation 3.3 is that it depends on the average feature vector (3.4) obtained from the training set. The average feature vector f¯n can be computed oﬀ-line once along with the average landmark location qn. These quantities can then be recalled during landmark detection (shown as auxiliary inputs to the supervised landmark detection block on Figure 3.5) and used in Equation 3.3 in a computationally eﬃcient manner to detect the landmark location pn on Vt.

3.4 Unsupervised landmark identiﬁcation

The supervised landmark detection algorithm described in section 3.3 relies on the knowledge of the locations of a landmark n on a set of M training volumes V = {V m}.

This set of training landmarks are denoted by Qn = {q1n, q2n, ··· , qMn}. This appendix describes an algorithm for deﬁning Qn automatically on V in an unsupervised way.

Central to this is the the concept of leave-one-out consistency of a set of landmarks with respect to a supervised landmark detection algorithm. Suppose that we have a supervised landmark detection algorithm, denoted by D(Vt,V,Qn), that can estimate the location of the landmark n on a test volume Vt in PIL space using the knowledge of training data (V,Qn). Such an algorithm was described in section 3.3. We deﬁne the set of landmarks Qn to be leave-one-out consistent (LOOC) with respect to algorithm

D if and only if qmn = D(Vm,V \{Vm},Qn\{qmn}) for all m, where the notation X\Y indicates the relative complement of set Y in set X. To be LOOC means that if for a particular m the landmark qmn and its corresponding training volume Vm are removed from Qn and V , respectively, then the algorithm would be able to correctly estimate its location based on the knowledge of the remaining M − 1 landmarks Qn\{qmn} on volumes V \{V m}.

32 3. Landmark identiﬁcation and detection methods

In this section we present an algorithm for automatically ﬁnding N sets of LOOC landmarks on a set of M training volumes V = {Vm} in PIL space. Before starting the algorithm, we deﬁne a set of K candidate seed locations S = {sk}(k = 1, 2, ··· ,K) on {v : RPIL(v) ≥ T hreshold}, where RPIL is the fuzzy brain template described in Section 3.4.1 in details.

(i) The algorithm for ﬁnding LOOC landmark sets is iterative. Let qmn denote the th (i) (i) n landmark location on training volume Vm at iteration i and Qn = {qmn}(m = 1, 2, ··· ,M). The algorithm proceeds as follows:

1. Initialize n = 0 and k = 0.

2. Increment k = k + 1.

(0) 3. Initialize qmn = sk for all m.

(i+1) (i) 4. Perform the following iteration until convergence (i.e. qmn = qmn for all m) or until a maximum number of iterations is reached without convergence:

(i+1) (i) (i) qmn = D(Vm,V \{Vm},Qn \{qmn}) (3.5)

(i) 5. If convergence occurred in step 3, then increment n = n + 1 and save Qn = {qmb} as a LOOC set of landmarks.

6. If k ≤ K go to step 1.

7. All seed points have been used (k = K), set N = n and end, where N is the total number of LOOC landmark sets identiﬁed.

The above procedure results in N sets of LOOC landmarks Qn = {qmn}, each set with M landmarks, one on each of the Vm training volumes. These landmarks will be in the vicinity of RPIL from which the seeds S were sampled. Once Qn = {qmn}

33 3. Landmark identiﬁcation and detection methods are found for m = 1, 2, ··· ,M and n = 1, 2, ··· ,N, the average landmark locations

{qn}(n = 1, 2, ··· ,N) are computed using Equation 3.1. In addition, the average feature vectors {f¯n} are computed using Equation 3.4. The training data {qn} and

{f¯n} are saved in auxiliary ﬁles on the computer system. Flowchart of the landmark identiﬁcation method is display in Figure 3.2.

3.4.1 Initial seeds

The unsupervised identiﬁcation method is an iterative algorithm. For each landmark we select a candidate seed and the location of the landmark updates in each iteration. As mentioned in previous section, A set of K candidate seed S = {sk}(k = 1, 2, ··· ,K) is selected based on a fuzzy brain template. The brain template is created from averaging brain masks for 152 volumes in the PIL space. This fuzzy brain template RPIL is a probabilistic function from the training masks. Each voxel in this brain template has a value between 0 to 100 that represents the probability of being part of the brain for that voxel. Figure 3.6 displays the fuzzy brain template in PIL space. To form S we sampled every 20 voxels in each dimension on {v : RPIL(v) ≥ T hreshold}. v is the volume that includes the probabilities higher than the value of T hreshold. In our implementation we set the T hreshold, zero. It means we include any voxel that might be part of the brain.

Figure 3.6: Fuzzy brain template.

34 4

Registration method

In this study, the proposed landmark identification and detection methods were employed to develop a rigid-body registration algorithm for registering two MRI volumes. Figure 4.1 displays the framework of the registration method. Two MRI volumes are passed to the system and outputs are the registered images. First, input volumes are reoriented to PIL space. Then, volumes are again registered to another space based on a set of landmarks. At the end, both images are registered to a mid-space by fine-tuning the registration parameters. The transformation to PIL space is performed as noted in 3.2. The landmark-based registration and fine registration modules are explained in this section.

Figure 4.1: Flowchart of the proposed registration method.

35 4. Registration method

To obtain a better normalization than PIL space for the rigid-body registration, we identiﬁed a set of landmarks on the MSP. All the landmarks were manually inspected and the best 8 landmarks were selected as shown in Figure 4.2. The pattern and expected location of the 8 landmarks deﬁned a new reference space.

Figure 4.2: The 8 selected landmarks on the MSP.

When the 8 landmarks on a given test image are detected, the best fitting plane that fits to these point was calculated. The plane fitting was implemented based on principle component analysis (PCA) algorithm. The normal vector of the plane is the Eigen vector corresponding to the smallest Eigen value of the covariance matrix calculated from the landmarks location. The detected plane is registered to the plane z = 0. Then, we found the image of the landmarks location on that plane. These points are located on one plane, therefore, a 2D rigid-body transformation will fit them to the reference landmarks on the MSP. These rigid body transformations are denoted as TLM which transforms an image from PIL space to LM space as the reference.

36 4. Registration method

Normalizing the images in the LM space provides a good initial point for ﬁne rigid- body registration, especially for longitudinal MRI volumes of the same subject. MRI scans contain the other parts of the head, neck, and the face. In brain registration, only the brain tissues are interested. Therefore, we deﬁned a brain template based on the fuzzy brain template in Section 3.4.1 to approximately cover the entire brain volume. This template is in LM space and contains a set of spherical templates with radius of 5. Figure 4.3 displays the overlay of this template on a brain MR volume.

Figure 4.3: Flowchart for computing aﬃne transformation TLM and TA.

The implemented registration algorithm is based on a similarity measure. Algo- rithm finds a point that the similarity function is maximum in the searching space. We defined a similarity measure based on the mentioned spherical brain template. The similarity measure is calculated on the registered volumes. Let brain template consists of H spherical templates and the weight factor for the hth sphere is denoted as wh. The similarity measure we utilized in this implementation can be represented as H P th f = wh (τ1h ∼ τ2h) where τ1h and τ2h denote the corresponding vectors to the h h=1 template (τh) in the first and second registered volumes, respectively. We used the CC as the similarity operation "∼". If transformation matrix T1, transforms the volume

V1 and similarly T2 transforms the volume V2 to the registered space, then τ1h = T1τh

37 4. Registration method

and τ2h = T2τh. In this registration method, both volumes are registered to a com-

−1 mon space. The registration is symmetric and unbiased, if and only if T1 = T2 or −1 T1 = T2, and the common space is noted as the mid-space. If the registration is symmetric, swapping the V1 and V2 as the inputs does not aﬀect the registration results. Also, ﬁnding the T1 transformation matrix provides the T2 matrix, and vice versa.

In our implementation, the T1 and T1 are rigid-body transformations. As explained in Section 2.6.1, a rigid-body transformation can be deﬁned by 6 parameters,

3 translation and 3 rotations. Therefore, the similarity function f(dx, dy, dz, θx, θy, θz) depends on 6 parameters. By finding these 6 parameters that maximize the similarity function f, the T1 and T1 matrices can be calculated. When images are registered using the 8 landmarks the similarity function is close to the maximum point. As shown in Figure 4.4, the similarity function f is convex around the maximum. Therefore, convex optimization methods are applicable to find the maximum point. We used the parabolic interpolation and golden section search optimization methods to find the maximum.

Figure 4.4: The sample plot of similarity function f versus dx, dy, dz, θx, θy, and θz.

38 5

Results

5.1 Image data

The database for this study comprehend T1-weighted magnetization prepared rapid gradient echo (MP-RAGE) images from ADNI archive. Images are 1.5 Tesla and acquired from diﬀerent scanners. Therefore, the volume matrix sizes might be variable. The PIL transformation discussed in section 3.2 was applied on the volumes and transformed them to PIL space with matrix size of 255 × 255 × 189. A set of 30 volumes were used as the training set. The detection algorithm was applied on 30 other images. Results were manually inspected using a graphical user interface (GUI). For further investigate the performance of the proposed method, a set of 152 volumes and brain masks were used for statistical analysis. In this chapter, results and the evaluation method are explained.

5.2 Test setup

The algorithm was implemented in MATLAB on a Windows 8 OS with intel i7 pro- cessor. A nifti toolbox [27] was used to load and save the MR images. Processing and analyzing the data was done in MATLAB workplace. Also a GUI program was developed to inspect the identiﬁed and detected landmarks.

39 5. Results

A set of 30 T1 wighted MR images from the ADNI database were selected as the training set. They were all transformed to PIL space using the ART(acpcdetect) program [28] on a linux machine. Unsupervised landmark identification method was applied on the training images in PIL space. According to Section 3.4.1, a set of 154 candidate seeds were chosen to cover the entire brain. Seeds were symmetric with respect to the MSP. From the identified landmarks, the symmetric points and the points on the MSP were selected. Then for each landmark, the model containing the average feature vector {f¯n} and searching centers {qn}(n = 1, 2, ··· ,N) was calculated and saved to be used in the detection phase. N is the total number of LOOC landmark sets identified. The training process is a longer process compare to the detection phase. Although, this part is off-line and need to be done once. Another set of 30 T1 wighted MR images from the ADNI database were used as the testing set. Identified landmarks were detected on the testing volumes. Figure 5.1 displays the initial seeds and the converged points for an identified landmarks, in 8 sample images.

Figure 5.1: An example of an identiﬁed landmark in 8 sample images. The red crosses (ﬁrst row) and the blue crosses (second row) display the seeds and converged points in the coronal plane, respectively.

In this algorithm there is no guarantee for the convergence of the candidate seeds in all the images. If they converge there is a landmarks with the similar and distinguishable

40 5. Results feature vector in all the images. Also, the program works ﬁne for most of the seeds but still there are some errors. Therefore, identiﬁed landmarks on the training volumes and detected landmark on the testing volumes were inspected using the GUI to make sure they are converged to the similar point in all the images. GUI program displays the neighbor voxels of the landmarks in axial, coronal, and sagittal planes.

5.3 Evaluation

The entire process including the identification and detection phases was done for 3 different template sizes. Templates were spheres with the radii of 5, 7, and 10 as described in 2.9.1.1. Table 5.1 demonstrates number of the seeds and identified landmarks. The "identified symmetric" row refers to the landmarks which are converged and symmetric with respect to MSP. The execution time depends on the number of training images, number of seeds, and size of the template and searching area. The best results in our experiment was obtained from the landmark patterns with radius of 7 and searching region with radius of 10.

Table 5.1: Number of the identiﬁed (total or symmetric) and detected landmarks on 30 training images and 30 test images for 3 template radii.

Number of the points Type of the points r=5 r=7 r=10

Seeds 154 154 154 Identiﬁed 83 87 59 Identiﬁed & symmetric 56 67 47 Detected 38 63 46

41 5. Results

From the training phase of the previous section, we have the average landmark locations and the the average feature vectors. Let these landmarks define a reference space named AFF space. If the corresponding set of landmark is detected in a test volume, then we are able to find an affine transformation matrix TA to map the landmarks from the original space to AFF space. The estimation method we used to find the affine transformation TA from two sets of landmarks, is explained in Appendix

A. Let TLM denote the aﬃne transformation that transform a volume from PIL space to AFF space, then TA = TLM TPIL will transform a volume from original to AFF space.

Referencing section 3.4.1, a fuzzy brain template RPIL was calculated from averaging 152 brain masks in the PIL space. The reference landmarks were detected on these 152 volumes and transformation matrix TA was calculated. Volumes were transformed to the PIL and AFF space and averages. Figure 5.2 shows the average of the volumes in the original and PIL space. The average volume in PIL space is less fuzzy because volumes are registered to a reference space. This visual improvement from PIL space to AFF space is not signiﬁcant, hence, we employed a method to analyze the results statistically.

42 5. Results

Figure 5.2: First row displays the average volume in PIL space. Second row displays the average volume in the original space.

5.3.1 Overlap Index

By applying the TA on the brain masks in the original space, we have a set of masks in the AFF space. Averaging these masks in AFF space produces a new fuzzy brain template RAF F . We utilized an overlap index to compare RPIL with RAF F . More overlap between brain masks means they are normalized better.

The Dice similarity coeﬃcient (K ∈ [0, 1]) is a normalized measure of overlap between two sets X and Y given by Dice [29]:

2|A ∩ B| K = (5.1) |A| + |B|

43 5. Results

The Dice K has been used extensively in evaluating medical image segmentation and registration algorithms. In this context, K is usually used to quantify the degree of agreement between two labels, for example, when a given structure on a volume is labeled by two diﬀerent methods. Let In(v) ∈ {0, 1} (n = 1, 2) represent a pair of binary-valued label volumes deﬁned over a set of common voxels v. The Dice can be also written as:

P 2 v I1(v)I2(v) K = P P (5.2) v I1(v) + v I2(v)

In this part, we present a generalization of Dice K to the case where one wishes to quantify the degree of overlap or agreement between a set of N label volumes

In(v) (n = 1, 2, . . . ,N). This situation could arise, for example, when N observers label a given structure on a common MRI volume, or when a given structure on a common MRI volume, or when a given structure is labeled on N different registered volumes by the same observer. To this end, let I(v) ∈ [0, 1] represent the average label given by P I(v) = (1/N) n In(v). We define a generalized similarity coefficient α to be:

P v I(v) log NI(v) α = P (5.3) v I(v) log N

where x log x is explicitly deﬁned to be zero when x = 0. It can be seen that

0 ≤ α ≤ 1. When there is perfect overlap between ln(v), then l(v) ∈ {0, 1}, P P v I(v) log NI(v) = v I(v) log N, and α = 1. On the other hand, when there is P no overlap between ln(v), then l(v) ∈ 0, 1/N, v I(v) log NI(v) = 0, and α = 0. Fur- thermore, it can be veriﬁed that for the special case of two labels (e.g., N = 2), the α in (5.3) reduces to the K in (5.2).

44 5. Results

The overlap index was measured for the brain probabilistic volume in the PIL space and in the AFF space for r = 7 and r = 10. The overlap index in PIL space was 0.96918 which already has a huge improvement compare to the original space. In the AFF space it was 0.98067 for r = 10 and 0.98096 for r = 7, respectively. These results show that brains masks have a little more overlap in AFF space than PIL space. Higher overlap in AFF space demonstrated a better normalization using the proposed method in this thesis.

5.3.2 Application

Identifying a set of landmarks in any desired region of interest and ability to detect them in a new volume is useful and these set of landmarks can be employed in many applications. For instance, image registration, segmentation, and change detection. We developed an image registration program employing the landmark identiﬁcation and detection method as described in Chapter4. Figure 5.3.2 shows an MRI brain volume and the landmarks detected on the MSP.

(a) (b)

Figure 5.3: (a) AC (green) and PC (red) on MSP. (b) 8 landmarks on MSP.

45 5. Results

Figure 5.4 displays these landmarks detected on 6 sample volumes. Using these method gives us a better and more robust normalization than PIL since the alignment is based on 8 detected landmarks instead of only AC and PC. We tested this program on more than 1000 images and the landmarks were detected correctly in every case.

Figure 5.4: MSP of 6 sample images and the 8 detected landmarks.

After normalizing the images in PIL and LM space, the ﬁne registration algorithm register the volumes. The registration method uses an iterative algorithm and in each iteration measures the similarity function and updates the registration parameters. This procedure continues till the similarity function reaches the maximum value. We comparing the similarity measure value for the volumes in PIL and LM space in- dividually and results show the normalized images are more similar using registering based on the 8 landmarks than only AC and PC. This gives a better initial point for the registration method and speed up the registration time. Figure B.1 and Figure B.2 display the cross sections of an MRI brain volume for a subject before and after registration, repsectively.

46 6

Conclusion

This study introduced a new method for landmark identification and detection in brain MR images. The proposed method identifies a set of landmarks from a training set and detects them in a given volume. The identification part only depends on the training set. Therefore, the suggested methods can be extended for other applications or in different fields. We implemented the identification and detection method for the brain MRI volumes and tested them. The landmarks were manually inspected and a registration method was developed as a further application. The methods were evaluated and results demonstrate an improvement compare to some of the existing methods. The registration method works based on 8 landmarks and the experiments on different datasets show the method is robust.

The results proof the ability and potential of the proposed method to be employed in practical and clinical applications. However, there are some aspects that still can be improved. For instance, we selected the candidate seeds based on spatial intervals and ROI masks. They can be selected automatically based on morphological characteristics. The template size and shape can be dynamic for each point or the feature vector includes more information than intensity value-vectors. The searching space can include more parameters and adaptive to the template size. Also, using a robust and intelligent optimization method to ﬁnd the best matching points will improve the timing.

47 Bibliography

[1] 3d pixel / voxel, 2010. [Online: http://www.bilderzucht.de/blog/3d-pixel-voxel/].

[2] Whats an mri, 2015. [Online: http://multiple-sclerosis-research. blogspot.com/2015/01/education-whats-mri.html].

[3] Suren Maharjan. Medical imaging technology, 2012. [ Online: http://suren634634.blogspot.com/2012/05/ t2-axial-sequence-of-brain-is-planned.html].

[4] Interpolation. [Online: http://mmc2.geofisica.unam.mx/luiggi/projects/ FLOWVIZ/Interpolation.html].

[5] Coordinate systems. [Online: http://www.slicer.org/slicerWiki/index.php/Coordinate_systems].

[6] Yassine Marbet. Human anatomy planes, 2012. [Online: https: //en.wikipedia.org/wiki/File:Human_anatomy_planes.svg#filehistory].

[7] Ganesh Jayachandran, Ahmet Ekin, and Ir Gerard De Haan. Landmark detection in mr brain images using surf. 2012.

48 Bibliography

[8] Overview of the central nervous system, 2003. [Online: http://what-when-how.com/neuroscience/overview-of-the-central\ -nervous-system-gross-anatomy-of-the-brain-part-1/].

[9] Babak A Ardekani and Alvin H Bachman. Model-based automatic detection of the anterior and posterior commissures on mri scans. Neuroimage, 46(3):677–682, 2009.

[10] Dr Jeremy Jones and Dr J Yeung. Spatial resolution mri. [Online: http://radiopaedia.org/articles/spatial-resolution-mri-1].

[11] Wikipedia. Anatomical plane — wikipedia, the free encyclopedia, 2015. [Online; https://en.wikipedia.org/w/index.php?title=Anatomical_plane&oldid= 644060248"].

[12] George Wolberg. Geometric transformation techniques for digital images: a survey. 1988.

[13] Twan Maintz. Digital and medical image processing. Universiteit Utrecht, 2005.

[14] Zhi-Pei Liang and Paul C Lauterbur. Principles of magnetic resonance imaging. SPIE Optical Engineering Press, 2000.

[15] Barry R Friedman, JP Jones, G Chavez-Munoz, AP Salmon, and CRB Merritt. Principles of mri. 1989.

[16] Wei Lu. A method for automated landmark constellation detection using evolutionary principal components and statistical shape models. 2010.

[17] Sönke Frantz, Karl Rohr, and H Siegfried Stiehl. Development and validation of a multi-step approach to improved detection of 3d point landmarks in tomographic images. Image and Vision Computing, 23(11):956–971, 2005.

49 Bibliography

[18] Itk-snap. [Online:http://www.itksnap.org/pmwiki/pmwiki.php].

[19] Babak Ardekani, Jeﬀ Kershaw, Michael Braun, I Kanuo, et al. Automatic detection of the mid-sagittal plane in 3-d brain images. Medical Imaging, IEEE Transactions on, 16(6):947–952, 1997.

[20] Yanxi Liu, Robert T Collins, and William E Rothfus. Robust midsagittal plane extraction from normal and pathological 3-d neuroradiology images. Medical Imaging, IEEE Transactions on, 20(3):175–192, 2001.

[21] Leonid Teverovskiy and Yanxi Li. Truly 3d midsagittal plane extraction for robust neuroimage registration. In Biomedical Imaging: Nano to Macro, 2006. 3rd IEEE International Symposium on, pages 860–863. IEEE, 2006.

[22] Ahmet Ekin. Feature-based brain mid-sagittal plane detection by ransac. In Signal Processing Conference, 2006 14th European, pages 1–4. IEEE, 2006.

[23] Stefan Wörz and Karl Rohr. Localization of anatomical point landmarks in 3d medical images by ﬁtting 3d parametric intensity models. Medical Image Analysis, 10(1):41–58, 2006.

[24] Karl Rohr. On 3d diﬀerential operators for detecting point landmarks. Image and Vision Computing, 15(3):219–233, 1997.

[25] Jyrki Lötjönen, Juha Koikkalainen, Lennart Thurfjell, and Daniel Rueckert. Atlas-based registration parameters in segmenting sub-cortical regions from brain mri-images. In Biomedical Imaging: From Nano to Macro, 2009. ISBI’09. IEEE International Symposium on, pages 21–24. IEEE, 2009.

[26] Sung-Hyuk Cha. Comprehensive survey on distance/similarity measures between probability density functions. 2007.

50 Bibliography

[27] MATLAB 2012a and Tools for NIfTI and ANALYZE image, The MathWorks, Inc., Natick, Massachusetts, United States.

[28] Art-acpcdetect. [Online:https://www.nitrc.org/projects/art/].

[29] Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302, 1945.

[30] Moo K. Chung. Aﬃne registration, 2012. [Online: http://pages.stat.wisc. edu/~mchung/teaching/768/reading/lecture06-affine.pdf].

51 A

Least-squares aﬃne transformation estimation

Suppose that a set of landmarks are extracted from a destination image and the aﬃne transformation T is desired to map the set of detected landmarks on the set of reference landmarks. The most common approach is using the least square for the mapping error. Moo [30] demonstrated a matrix-based solution for ﬁnding the optimum values of the matrix A and vectro d. The transformation equation for a

3 landmark ~p = (p1, p2, p3) ∈ R to the reference landmark ~q = (q1, q2, q3) is:

~q = T (~p) = A~p + d~ (A.1)

Where A represent an linear mapping and d is a translation.

      ~q A d~ ~p         =     (A.2) 1 0 ... 0 1 1 | {z } T

For n detected points ~pi = (pi1, pi2, pi3) and given reference points ~qi = (qi1, qi2, qi3) we have qi = Api + d, where i = 1, 2, . . . , n. Let p¯ and q¯ be the average of pi and qi points, respectively. Considering pˆi= pi − p¯ and qˆi= qi − q¯ we have:

qˆi = Apˆi (A.3)

I A. Least-squares aﬃne transformation estimation

The least square estimation of A is given by

2 Aˆ = arg min kqî − Apîk (A.4) A∈Ω where Ω is the parameter space of affine elements.

Q = pˆ1pˆ2 ... pˆn ,P = qˆ1qˆ2 ... qˆn (A.5)

      Q A 0 P         =     (A.6) 1 0 ... 0 1 1 From the above equations the optimum matrix that satisﬁes the least square equation is given as: A = QP 0(PP 0)−1 (A.7)

Consequently the translation vector would be

d =q ¯ − Ap¯ (A.8)

II B

Registration Figures

Figure B.1: Same cross sections of original longitudinal images from one subject.

III B. Registration Figures

Figure B.2: Same cross sections of registered longitudinal images from one subject.