ΠΑΝΕΠΙΣΤΗΜΙΟ ΠΑΤΡΩΝ

ΔΙΑΤΜΗΜΑΤΙΚΟ ΠΡΟΓΡΑΜΜΑ ΜΕΤΑΠΤΥΧΙΑΚΩΝ ΣΠΟΥΔΩΝ ΣΤΗΝ ΙΑΤΡΙΚΗ ΦΥΣΙΚΗ

ΑΞΙΟΛΟΓΗΣΗ ΑΛΓΟΡΙΘΜΩΝ ΑΝΤΙΣΤΟΙΧΙΣΗΣ ΕΙΚΟΝΑΣ ΣΤΗΝ ΥΠΟΛΟΓΙΣΤΙΚΗ ΤΟΜΟΓΡΑΦΙΑ ΠΟΛΛΑΠΛΩΝ ΑΝΙΧΝΕΥΤΩΝ ΘΩΡΑΚΑ

ΔΗΜΗΤΡΙΟΥ ΔΗΜΗΤΡΙΟΣ ΜΕΤΑΠΤΥΧΙΑΚΗ ΔΙΠΛΩΜΑΤΙΚΗ ΕΡΓΑΣΙΑ 2015 UNIVERSITY OF PATRAS

INTERDEPARTMENTAL PROGRAM OF POSTGRADUATE STUDIES IN MEDICAL PHYSICS

EVALUATION OF IMAGE REGISTRATION ALGORITHMS IN THORAX MDCT

DIMITRIOU DIMITRIOS MSc THESIS 2015 THREE MEMBER EXAMINING COMMITTEE

Professor, Lena Costaridou (Supervisor) Associate Professor, Christina Kalogeropoulou Associate Professor, Ioannis Pratikakis

ΤΡΙΜΕΛΗΣ ΕΞΕΤΑΣΤΙΚΗ ΕΠΙΤΡΟΠΗ

Καθηγήτρια, Ελένη Κωσταρίδου (Επιβλέπουσα) Αναπληρώτρια Καθηγήτρια, Χριστίνα Καλογεροπούλου Αναπληρωτής Καθηγητής, Ιωάννης Πρατικάκης Ευχαριστίες

Τις βαθύτερες ευχαριστίες μου στην επιβλέπουσα της παρούσας μεταπτυχιακής εργασίας κ. Κωσταρίδου Ελένη, Καθηγήτρια Ιατρικής Φυσικής του Πανεπιστήμιου Πατρών για την σημαντική ευκαιρία την οποία μου έδωσε να εργαστώ σε ένα τόσο ενδιαφέρον και σύγχρονο θέμα ιατρικής απεικόνισης. Η επιστημονική της καθοδήγηση και η ηθική της υποστήριξη συντέλεσε τα μέγιστα στην ολοκλήρωση της όλης ερευνητικής προσπάθειας.

Ευχαριστώ θερμά τον κ. Πρατικάκη Ιωάννη, Αναπληρωτή Καθηγητή Ηλεκτρολόγων Μηχανικών & Μηχανικών Υπολογιστών του Δημοκρίτειου Πανεπιστημίου Θράκης, καθώς και την κ. Καλογεροπούλου Χριστίνα, Αναπληρώτρια Καθηγήτρια του τμήματος Ιατρικής Πανεπιστημίου Πατρών για τη συμμετοχή τους στη τριμελή επιτροπή.

Ιδιαίτερα ευχαριστώ τον κ. Βλαχόπουλο Γεώργιο, υποψήφιο Διδάκτορα Ιατρικής Φυσικής για την πολύτιμη συνεισφορά του στο πειραματικό μέρος και στη γενικότερη επιστημονική καθοδήγηση και συνεισφορά του, καθώς επίσης και ηθική υποστήριξη για την ολοκλήρωση αυτής της ερευνητικής προσπάθειας. Abstract

The follow-up of disease progression is one of the most challenging tasks in CT lung analysis. The main image analysis method behind the development of automated tools for image based quantification and estimation of disease progression is image registration. Alterations in lung field radiological appearance, caused by disease progression in the lung, as well as the elastic nature of lung tissue, challenge registration of disease affected lung fields. In this thesis, lesions associated to Interstitial Lung Disease (ILD) are considered. These lesions are characterized by diffuse lung appearance alterations. Specifically, the selection of the evaluation of registration algorithms suitable for ILD quantification is critical for a registration scheme.

A registration scheme consists of the following basic components; the transformation (or deformation model), the cost function (or matching criteria or similarity metric) and the optimizer. In this study the effect of two registration optimizers, the Quasi-Newton (QS) and the Simultaneous Perturbation (SP), is assessed.

A pilot clinical data set was analyzed consisting of 5 pairs of MDCT scans corresponding to 5 patients diagnosed with ILD secondary to connective tissue diseases, at two different instances, abstaining in time approximately two years. All patients were scanned with a 16-row multidetector CT (MDCT) scanner (GE Lightspeed 16, General Electric Medical Systems, Milwaukee, WI) at 120 kVp, rotation time of 0.5 s, automatic modulation of mA, collimation thickness of 16 × 0.625 mm, and slice thickness of 1.25 mm, using a protocol obtaining volumetric 3D data at full inspiration, in supine position. Each scan volume comprised of approximately 200–250 slices/patient. The mean volume CT dose index and the mean dose-length product, provided by CT application panel, were 11.2 mGy and 266.6 mGy cm, respectively. Assuming 0.017 mSv/mGy cm for a standard chest CT examination, the effective radiation dose for the volumetric chest CT protocol used was 4.5 mSv, complying with European Working Group for Guidelines on Quality Criteria in CT.

It is proposed in literature that a spatial multiresolution procedure from coarse to fine image resolution can be used in the registration in order to improve speed, accuracy and robustness. The basic idea of multiresolution is that registration is first performed at a coarse scale where the images have much fewer pixels, which is fast and can help eliminate local optima. In that case, we talk about a “pyramid”. In this study we use a 4-level resolution pyramid, performing rigid transformation for the first three levels (for computational time considerations) and non-rigid transformation for the 4th level.

The registration schemes were performed in the frame of elastix, which is an open source software based on the Insight Segmentation and Registration Toolkit (ITK). A total of 64 different registration schemes were generated by considering all possible combinations among: 2 different types of pyramids (Shrinking, which applies no smoothing, but only down-sampling by a factor of 2 in all three dimensions, and Gaussian Smoothing, which applies smoothing and down-sampling by a factor of 2 in all three dimensions), 4 different cost functions (Sum of Square Differences – SSD, Normalized Correlation Coefficient – NCC, Mutual Information – MI and Normalized Mutual Information – NMI), 4 different types of transforms for the first three levels (Euler, Similarity, Affine and 3rd order B-Splines) in order to obtain a coarse initial alignment, while in all cases for the 4th level, corresponding to the highest image resolution, the 3rd order B-Spline transform was selected for refinement purposes. Finally, 2 different types of optimizers were considered (Quasi-Newton (QS) and Simultaneous Perturbation (SP)) in order to compare their performances and evaluate their effect to registration accuracy.

The registration schemes are evaluated using two distance metrics, the distance between corresponding points: (a) in ILD affected regions and (b) in normal lung parenchyma (NLP). Prior to evaluation, the registered lung volumes were prescreened for exclusion of schemes introducing folding regions. Irregular deformations were assessed in terms of the determinant of the Jacobian matrix of each voxel of registered volumes (deformation field). Registration schemes whose determinants of Jacobian matrices include negative values are excluded for subsequent analysis, since such areas correspond to singularities of deformation fields (foldings).

In order to estimate registration distance error, 2 sets of landmark points were generated, corresponding to 2 different types of tissue in the lung fields: NLP and ILD affected tissue. In order to identify normal parenchyma lung tissue, an ILD region segmentation algorithm was applied to create binary masks corresponding to ILD affected and normal parenchyma regions.

By considering the optimal schemes, 16 of 32 registration schemes were initially selected. These schemes obtained submillimeter registration accuracies in terms of average distance errors; 0.33 ± 0.01 mm for NLP and 0.33 ± 0.01 mm for ILD affected regions in the case of Quasi-Newton optimizer, 0.49 ± 0.09 mm for NPL and 0.48 ± 0.05 mm for ILD affected regions in the case of Simultaneous Perturbation.

Best performance was achieved by the registration scheme using: the Gaussian smoothing pyramid, the Affine Transform for the first 3 resolution levels and the 3rd order B-Spline for the 4th resolution level, the Mutual Information Cost Function and the Quasi-Newton Optimizer. Non-parametric Wilcoxon signed-rank test indicates that there was statistical significant better performance in the case of Quasi-Newton compared to performances achieved using the Simultaneous Perturbation optimizer.

Future efforts should include different tuning of the optimizer parameters, as well as regularization terms which prevent folding areas, in order to achieve higher registration accuracies. In addition, future steps should include constraints on full nonrigid schemes, that will not allow irregular deformations.

Taking into account that registration is a fundamental subsystem of an integrated quantification and follow-up system, future efforts should also focus on integrate the quantification and registration system in order to create an integrated monitoring system of the ILD. Περίληψη

Η παρακολούθηση της εξέλιξης της νόσου είναι μία από τις πιο απαιτητικές εργασίες στην ανάλυση της αξονικής τομογραφίας πνεύμονα. Η κύρια μέθοδος ανάλυσης εικόνας, πίσω από την ανάπτυξη αυτοματοποιημένων εργαλείων με υποβοήθηση εικόνας για την ποσοτικοποίηση και την εκτίμηση της εξέλιξης της νόσου, είναι η αντιστοίχιση εικόνας (image registration). Οι μεταβολές στο πνευμονικό πεδίο που προκαλούνται από διάχυτη εξέλιξη της νόσου, καθώς και η ελαστική φύση του πνευμονικού ιστού, δυσχεραίνουν την αντιστοίχιση (registration). Σε αυτή την εργασία μελετώνται περιοχές του πνεύμονα που έχουν προσβληθεί από διάμεση νόσο (ΔΝΠ-ILD). Αυτές οι περιοχές χαρακτηρίζονται από διάχυτες μεταβολές στην όψη του πνεύμονα. Η αξιολόγηση των αλγορίθμων αντιστοίχισης στην ποσοτικοποίηση της ΔΝΠ είναι κρίσιμη για την αξιόπιστη επιλογή ενός σχήματος αντιστοίχισης (registration scheme).

Ένα σχήμα αντιστοίχισης εικόνας αποτελείται από τα παρακάτω: τον μετασχηματισμό, την συνάρτηση κόστους (cost function) και την μέθοδο βελτιστοποίησης (optimizer). Σε αυτή την εργασία εξετάζεται η επίδραση δύο ειδών μεθόδων βελτιστοποίησης αντιστοίχισης εικόνας, ο Quasi-Newton (QS) και ο Simultaneous Perturbation (SP).

Χρησιμοποιήθηκαν πιλοτικά κλινικά δεδομένα που αποτελούνται από 5 ζεύγη λήψεων από αξονικό τομογράφο πολλαπλών τομών (MDCT) και αντιστοιχούν σε 5 ασθενείς (περίπου 5x250 τομές) οι οποίοι έχουν διαγνωσθεί με ΔΝΠ, σε δύο διαφορετικούς χρόνους, που απέχουν περίπου δύο χρόνια μεταξύ τους. Οι λήψεις έχουν πραγματοποιηθεί από έναν αξονικό τομογράφο πολλαπλών ανιχνευτών (MDCT) 16 τομών (GE Lightspeed 16, General Electric Medical Systems, Milwaukee, WI) στα 120 kVp, με χρόνο περιστροφής 0.5 s, αυτόματης διαμόρφωσης, πάχος ευθυγράμμισης 16 × 0.625 mm και πάχος τομής 1.25 mm, χρησιμοποιώντας πρωτόκολλο 3Δ δεδομένων σε πλήρη εισπνοή, σε ύπτια θέση. Κάθε όγκος λήψης αποτελείται από περίπου 200-250 τομές ανά ασθενή. Η ενεργός δόση σύμφωνα με το πρωτόκολλο που χρησιμοποιήθηκε ήταν 4.5 mSv, ακολουθώντας τις κατευθυντήριες γραμμές του European Working Group για τα κριτήρια ποιότητας. Στη βιβλιογραφία προτείνεται ότι μία χωρική διαδικασία πολλαπλής ανάλυσης (multiresolution), από μία αδρή έως την καλύτερη ανάλυση εικόνας, μπορεί να χρησιμοποιηθεί έτσι, ώστε να βελτιστοποιηθεί η ταχύτητα και η ακρίβεια. Η βασική ιδέα της πολλαπλής ανάλυσης είναι ότι η αντιστοίχιση γίνεται πρώτα σε μια αδρή κλίμακα όπου οι εικόνες έχουν λιγότερα ογκοστοιχεία (voxel), το οποίο είναι γρήγορο και μπορεί να βοηθήσει να απαλειφθούν τα τοπικά ακρότατα. Σε αυτή την περίπτωση μιλάμε για μια «πυραμίδα». Στην παρούσα εργασία χρησιμοποιούμε μία πυραμίδα τεσσάρων (4) επιπέδων, πραγματοποιώντας έναν άκαμπτο (rigid) μετασχηματισμό για τα πρώτα τρία επίπεδα (για λόγους υπολογιστικού κόστους) και έναν ελαστικό (non-rigid) μετασχηματισμό για το 4ο επίπεδο.

Τα σχήματα αντιστοίχισης εκτελέστηκαν στο elastix, το οποίο είναι μια ανοιχτή πηγή λογισμικού βασισμένη στο Insight Segmentation and Registration Toolkit (ITK). Παράχθηκαν συνολικά 64 διαφορετικά σχήματα, συνυπολογίζοντας όλους τους πιθανούς συνδυασμούς μεταξύ: 2 ειδών πυραμίδων (Shrinking and Recursive Gaussian Smoothing), 4 διαφορετικών συναρτήσεων κόστους (cost functions) (Sum of Square Differences – SSD, Normalized Correlation Coefficient – NCC, Mutual Information – MI and Normalized Mutual Information – NMI), 4 ειδών μετασχηματισμού για τα πρώτα τρία επίπεδα (Euler, Similarity, Affine and 3rd order B-Splines) ώστε να σχηματιστεί μια χονδροειδής αρχική ευθυγράμμιση, ενώ σε όλες τις περιπτώσεις για το τέταρτο επίπεδο, που αντιστοιχεί στην μέγιστη ανάλυση της εικόνας, επιλέχτηκε ο 3ης τάξης B-Spline μετασχηματισμός για λόγους βελτίωσης. Τελικά, 2 τύποι μεθόδων βελτιστοποίησης (optimizers) επιλέχτηκαν (Quasi-Newton (QΝ) and Simultaneous Perturbation (SP)) έτσι, ώστε να συγκρίνουμε τις επιδόσεις τους και να αξιολογήσουμε την επίδρασή τους στην ακρίβεια της αντιστοίχισης εικόνας (registration accuracy).

Τα σχήματα αξιολογήθηκαν με δύο υπολογισμούς απόστασης, της απόστασης μεταξύ αντίστοιχων σημείων: α) στις περιοχές που έχουν προσβληθεί από ΔΝΠ και β) στο φυσιολογικό παρέγχυμα στον πνεύμονα. Πριν από την αξιολόγηση, έχουν εξαιρεθεί σχήματα τα οποία εισαγάγουν περιοχές που εμφανίζουν πτυχώσεις (folding regions). Αυτές οι παράτυπες παραμορφώσεις εκτιμήθηκαν με την ορίζουσα του Ιακωβιανού πίνακα (Jacobian matrix) για κάθε voxel (ογκοστοιχείο) των αντιστοιχισμένων εικόνων. Τα σχήματα των οποίων οι ορίζουσες των Ιακωβιανών πινάκων περιέχουν αρνητικές τιμές, εξαιρούνται από περαιτέρω ανάλυση.

Για να εκτιμηθεί το σφάλμα στις αποστάσεις της αντιστοίχισης, παράχθηκαν δύο σειρές διακριτικών σημείων (landmark points), που αντιστοιχούν στα δύο είδη ιστών στα πνευμονικά πεδία: περιοχές φυσιολογικού παρεγχύματος και περιοχές που έχουν προσβληθεί από ΔΝΠ. Για να αναγνωριστούν οι περιοχές φυσιολογικού παρεγχύματος, χρησιμοποιείται ένας αλγόριθμος τμηματοποίησης των περιοχών με ΔΝΠ ώστε να δημιουργηθούν δυαδικές μάσκες που αντιστοιχούν στις περιοχές με ΔΝΠ και φυσιολογικού παρεγχύματος.

Λαμβάνοντας υπόψιν τα βέλτιστα σχήματα, 16 από τα 32 επιλέχθηκαν. Αυτά τα σχήματα απέδωσαν ακρίβεια αντιστοίχισης σε όρους μέσου σφάλματος απόστασης της αντιστοίχισης: 0.33 ± 0.01 mm για περιοχές φυσιολογικού παρεγχύματος και 0.33 ± 0.01 mm για περιοχές που έχουν προσβληθεί από ΔΝΠ στην περίπτωση του Quasi-Newton και 0.49 ± 0.09 mm για περιοχές φυσιολογικού παρεγχύματος και 0.48 ± 0.05 mm για περιοχές που έχουν προσβληθεί από ΔΝΠ στην περίπτωση του Simultaneous Perturbation.

Η καλύτερη επίδοση επιτεύχθηκε από το σχήμα που χρησιμοποίησε: Recursive Gaussian smoothing πυραμίδα, Affine μετασχηματισμό για τα 3 πρώτα επίπεδα ανάλυσης και 3ης τάξης B-Spline μετασχηματισμό για το 4ο επίπεδο ανάλυσης, Mutual Information cost function και Quasi-Newton optimizer. Σε όλα τα σχήματα ο QN optimizer είχε σημαντικά υψηλότερη επίδοση συγκριτικά με τον SP.

Μελλοντικές προσπάθειες θα πρέπει να γίνουν σε πραγματικά κλινικά δεδομένα και τα σχήματα να περιέχουν διαφορετική ρύθμιση των παραμέτρων των μεθόδων βελτιστοποίησης, καθώς και την προσθήκη παραγόντων κανονικοποίησης που αποτρέπουν τις πτυχώσεις, ώστε να επιτευχθούν υψηλότερες επιδόσεις στην αντιστοίχιση εικόνας.

Λαμβάνοντας υπ’όψιν ότι η αντιστοίχιση είναι ένα θεμελιώδες υποσύστημα ενός ολοκληρωμένου συστήματος ποσοτικοποίησης και παρακολούθησης, μελλοντικές προσπάθειες θα πρέπει να εστιάσουν στην ενοποίηση ενός συστήματος ποσοτικοποίησης και αντιστοίχισης έτσι, ώστε να δημιουργηθεί ένα πλήρες σύστημα παρακολούθησης της ΔΝΠ. Contents

1. Introduction and Outline ...... 1

1.1 Computed Tomography ...... 1 1.2 Multidetector Computed Tomography ...... 3 1.3 Lung Anatomy ...... 5 1.4 CT Appearance of ILD Patterns ...... 8

1.4.1 Nodular Opacities ...... 10 1.4.2 Linear and Reticular Opacities ...... 10 1.4.3 Ground Glass Opacities ...... 10 1.4.4 Decreased Lung Attenuation ...... 11 1.4.5 Consolidation ...... 12

1.5 Thesis and Outline...... 13

2. Image Registration ...... 15

2.1 Introduction ...... 15 2.2 Algorithm Classification ...... 16

2.2.1 Intensity-based vs Feature-based ...... 16 2.2.2 Transformation models ...... 16 2.2.3 Spatial vs Frequency Domain methods ...... 17 2.2.4 Intramodal vs Intermodal Registration ...... 17 2.2.5 Automatic vs Interactive methods ...... 18

2.3 Image Registration Framework ...... 18

2.3.1 Deformation Models – Transformation Parameterization . . . . .19 2.3.2 Matching Criteria – Cost Function Design ...... 26 2.3.3 Optimization Methods ...... 32 2.3.4 Multiresolution Scheme ...... 34 2.3.5 Interpolators ...... 35

2.4 Evaluation Methods ...... 36

2.4.1 Landmark Matching Accuracy ...... 36 2.4.2 Vessel Matching Accuracy ...... 37 2.4.3 Fissure Alignment Distance ...... 37 2.4.4 Lung Segmentation Overlap ...... 38 2.4.5 Evaluation Using Transformation Properties ...... 39 2.5 Clinical Significance and “State of the Art” of Medical Image Registration...... 40 2.6 Summary ...... 42

3. Materials and Methods ...... 43

3.1 Method Overview ...... 43 3.2 Transformation ...... 50

3.2.1 Rigid Transformation ...... 50 3.2.2 Similarity Transformation ...... 51 3.2.3 Affine Transformation ...... 51 3.2.4 B-Spline Transformation ...... 52

3.3 Interpolator ...... 53 3.4 Cost Function ...... 53

3.4.1 Sum of Squared Difference (SSD) ...... 53 3.4.2 Normalized Cross Correlation (NCC) ...... 54 3.4.3 Mutual Information (MI) ...... 54 3.4.4 Normalized Mutual Information (NMI) ...... 55

3.5 Optimizer ...... 56

3.5.1 Quasi-Newton LBFGS ...... 57 3.5.2 Simultaneous Perturbation...... 59

3.6 Data complexity – Pyramids ...... 60 3.7 Evaluation ...... 61 3.8 Summary ...... 64

4. Results ...... 65

4.1 Results ...... 65 4.2 Results and Discussion ...... 71

References ...... 73

List of Figures

1.1 A 2D stack corresponding to a certain CT scan length (z axis). The volumetric character of the technique enables representation at 3 orthogonal planes (Figure 1.1b axial , d sagittal and f coronal) also lending its self to 3D representation ...... 2

1.2 Multislice Computed Tomography: a) arrows indicate tube and table movement, b)single detector array and c) multidetector arrays ...... 4

1.3 Basic lung anatomy ...... 6

1.4 Volume-rendered coronal image depicting vessel tree ...... 6

1.5 Airway anatomy ...... 7

1.6 Diagram illustrating the three main compartments of the pulmonary interstitium at the level of the secondary pulmonary lobule. The axial compartment surrounds the lobular artery and bronchiole, whereas the peripheral compartment is located at the level of the interlobular septum. These compartments are connected to each other by the intralobular interstitium (Adapted from Beigelman-Aurbry et al.) . . . . . 8

1.7 CT appearance of ILD. (a) Normal lung parenchyma. (b) Reticular opacities, (c) Ground glass opacities, (d) Honeycomb, (e) Emphysema and (f) Consolidation, with corresponding regions of interest patterns ...... 12

2.1 A diagrammatic example of registration of two images ...... 15

2.2 Basic components of the registration framework ...... 19

2.3 Classification of deformation models...... 21

2.4 Classification of geometric transformations derived from physical models. . . . .23

2.5 Classification of geometric transformations derived from interpolation theory...... 24

2.6 Classification of knowledge-based geometric transformations ...... 25

2.7 Classification of task-specific constraints ...... 26

2.8 Classification of cost functions ...... 28

2.9 Classification of geometric methods ...... 29

2.10 Classification of iconic methods ...... 31

List of Figures

2.11 Classification of hybrid methods ...... 31

2.12 Classification of optimization methods ...... 32

2.13 Classification of continuous optimization methods ...... 33

2.14 Classification of discrete optimization methods ...... 34

2.15 Multiresolution pyramid ...... 35

2.16 Illustration of the landmark error calculation ...... 36

2.17 Vessel matching distance (mm) on target vessel tree...... 37

2.18 Lobe division ...... 38

3.1 100 well-distributed landmark points were automatically generated in the baseline 3D lung segment volume and matched semi-automatically in the follow-up 3D lung segment, using method proposed by Murphy et al ...... 44

3.2 An example of the landmark points identified in a fixed scan ...... 45

3.3 Procedure for warping a scan artificially ...... 45

3.4 Image registration procedure ...... 46

3.5 Procedure for selecting the optimal registration scheme among i = 32 schemes generated for each optimizer tested ...... 47

3.6 Components combinations of the registration schemes ...... 48

3.7 Iterative optimization. Example for registration with a translation transformation model ...... 56

3.8 Multiresolution pyramid – Images for each resolution level ...... 61

3.9 Example of a rejected registration scheme. The registered lung fused with the deformation field, which includes negative values ...... 63

3.10 (a) Lung fields affected by interstitial disease. Gray color indicates the normal lung parenchyma, while green color indicates the interstitial lung disease affected regions. (b) Landmark points in normal lung parenchyma. (c) Landmark points in interstitial lung disease affected regions ...... 63

List of Figures

3.11 An example of a landmark point for an image pair. The top row shows the landmark identified in the fixed image in the sagittal, coronal and axial directions. The second row shows the point selected in the registered image...... 64

4.1 Segmented lung fields of the 5 patients baseline scans ...... 65

4.2 Registration scheme performance example: (a) baseline image, (b) unregistered warped image, (c) difference image of (a) and (b), (d) optimal registered image (QN), (e) difference image of (a) and (d), (f) suboptimal registered image (SP), and (g) difference image of (a) and (f)...... 66

List of Tables

3.1 Parameters used in the registration schemes applied ...... 49

4.1a Results using Quasi-Newton LBFGS optimizer for normal parenchyma areas..68

4.1b Results using Quasi-Newton LBFGS optimizer for ILD regions ...... 68

4.1c Results using Simultaneous Perturbation optimizer for normal parenchyma areas ...... 69

4.1d Results using Simultaneous Perturbation optimizer for ILD regions ...... 69

4.1e Average distance error...... 70

4.2 Illustration of the results ...... 70

Chapter 1 – Introduction and Outline

1. Introduction and Outline

Computed tomography (CT) imaging is a powerful modality for the quantification of Interstitial Lung Disease (ILD) disease extent and monitoring of change in a follow-up framework. ILD response quantification involves comparison of quantified disease extend between scans obtained at different time instances. Alterations in lung field radiological appearance, caused by diffused disease progression in the lung, as well as the elastic nature of lung tissue, challenge registration of ILD affected lung fields. Evaluation of registration algorithms in ILD quantification is critical for a reliable selection of a registration scheme. Due to lack of a gold standard, the selection of the optimal registration scheme remains an open issue.

In this study the effect of two types of registration optimizers, Quasi Newton (QS) and Simultaneous Perturbation (SP), are assessed. This introduction aims to familiarize the reader with the basic concepts of CT and ILD dealt by in this thesis and provide the thesis outline.

1.1 Computed Tomography

X-ray Computed Tomography CT brought body slice imaging into widespread use, representing a major breakthrough in medical imaging. Today, CT is an essential part of radiological diagnostics and is considered as a mature and clinically accepted technology. The first clinical CT images were produced at the Atkinson Morley Hospital in London in 1972. The very first patient examination performed with CT offered convincing proof of the effectiveness of the method by detecting a cystic frontal lobe tumor. CT made also possible to view the human body in a 3D manner as a build-up of a finite number of discrete slices, composed of volume elements, with each single transverse slice (section) representing the composition of tissue resulting from the attenuation of the transversing x-ray beam. A CT slice can be considered as a 2D matrix composed of discrete volume elements, assuming a certain slice thickness.

-1-

Chapter 1 – Introduction and Outline

The value of each volume element is displayed in one picture element of the 2D digital image matrix representing the slice. The term “voxel” corresponds to the volume element while, the term “pixel” stands for picture element (Figure 1.1) [1]. Figure 1.1a displays a 2D stack corresponding to a certain CT scan length (z-axis). The volumetric character of the technique enables representation at 3 orthogonal planes (Figure 1.1 b axial , d sagittal and f coronal) also lending its self to 3D representation.

Figure 1.1: A 2D stack corresponding to a certain CT scan length (z axis). The volumetric character of the technique enables representation at 3 orthogonal planes (Figure 1.1b axial , d sagittal and f coronal) also lending its self to 3D representation (Adapted from P. Korfiatis - PhD Thesis, 2010, Al. Kazantzi - Phd Thesis 2013)

-2-

Chapter 1 – Introduction and Outline

CT measures the spatial distribution of the linear attenuation coefficient 휇(푥, 푦), which describes the fraction of a beam of x-rays that is absorbed or scattered per unit length of the absorber. This value basically accounts for the number of atoms in a cubic cm3 of volume of material and the probability of a photon being scattered or absorbed from the nucleus or from an electron of one of these atoms. However, the physical quantity 휇 is not very descriptive and is strongly dependent on the spectral energy used. So, to render feasible direct comparisons between scanners with different beam voltage and beam filtration the attenuation coefficients are converted to the so-called CT value relative to the attenuation of the water. CT values are measured in Hounsfield units (HU). The CT values of water and air are independent of the energy of the x-rays and therefore constitute the fixed points for the CT value scale. While most of the body areas exhibit positive CT values, lung tissue exhibits negative CT values due to its lower density resulting in lower attenuation coefficient (휇푙푢푛푔 < 휇푤푎푡푒푟). The Hounsfield scale has no upper limit. For medical CT scanners a range from -1024 to 3071 is typically provided. Consequently, 4096 different values are available assuming a 12 bits depth.

1.2 Multidetector Computed Tomography

Various types of CT scanners have involved based on combination of different motion types (tube, table and detector array), beam geometries and detector types, resulting up to now to 4 CT scanners generations. Multi-Detector Computed Tomography (MDCT), also known as multislice, multidetector CT, or multisection CT, is the latest breakthrough in CT technology. It has transformed CT from a transaxial cross-sectional technique into a truly three- dimensional imaging modality. While dual-slice spiral scanning has been available since 1992, the first 4-slice units were introduced in 1998. Systems with 6-, 8-,10-, and 16-detector arrays have become available over the past few years, and scanners with 32-, 40-, 64-, and 128-detector rows are now available. MDCT has been more rapidly accepted in the radiological

-3-

Chapter 1 – Introduction and Outline

community than single-slice spiral CT, with exponential growth in the use of these scanners in clinical practice worldwide.

Figure 1.2: Multislice Computed Tomography: a) arrows indicate tube and table movement, b) single detector array and c) multidetector arrays

A significant development for 3rd and 4th generation scanners was the development of the so-called helical or spiral scanners. This approach incorporated a moving table during the rotation of the x-ray tube. The net effect is that the x-ray path describes a spiral (helical) pattern around the scan volume of a patient (Figure 1.2a). The major advantage of spiral scanning is the volume of coverage for a given rotation of x-ray exposure. In single slice spiral scanning the information to reconstruct a given slice originates from the slice above and below from the slice of interest exploiting interpolation techniques to establish an effective slice at a given position. The main problem with spiral CT scanning is the inverse relationship between scan length and spatial resolution along the patient axis (z-axis resolution). Thus, a volumetric acquisition with high spatial resolution in all spatial directions can only cover small areas during one breath hold. The solution to this problem is to acquire multiple simultaneous slices and use a higher speed of rotation. As a result, a four-detector row scanner with 0.5 sec rotation, for example, has a performance that is up to eight times greater than a conventional 1 sec single-slice scanner. This allows high spatial resolution to be achieved over a long scan range. Furthermore, with multidetector scanners the information of a slice of interest is better restored as compared with

-4-

Chapter 1 – Introduction and Outline

the interpolations used to extract a slice in single slice spiral scans (Figure 1.2b,c). The main breakthrough of MDCT has been in the area of thinner sections, which makes it possible to acquire a near isotropic data volume. The advantages of MDCT come with the problem of increased data load. More than 1,000 images can be produced, particularly with near-isotropic volume acquisition techniques. While such a volumetric acquisition is used less frequently with most 4-slice scanners, it is a standard technique for 16-slice scanners. The number of images in a CT scan depends on the scan length and the reconstruction interval. The increased data load means that more time is needed to analyze the data than with single-spiral CT, while new ways of viewing, processing and archiving images are necessary.

1.3 Lung Anatomy

Since this thesis is focused on image processing of Interstitial Lung Disease (ILD) in CT lung imaging, a small overview of basic lung anatomy is provided. Each lung is encased in a thin membranous sac called the pleura, and each is connected with the trachea (windpipe) by its main bronchus (large air passageway) and with the heart by the pulmonary arteries. The lungs are soft, light, spongy, elastic organs that normally, after birth, always contain some air (Figure 1.3). In the inner side of each lung, about two-thirds of the distance from its base to its apex, is the hilum, the point at which the bronchi, pulmonary arteries (Figure 1.4) and veins, lymphatic vessels, and nerves enter the lung. The main bronchus subdivides many times after entering the lung; the resulting system of tubules resembles an inverted tree. The diameters of the bronchi diminish eventually to less than 1 mm. The branches are 3 mm and less in diameter and are known as bronchioles, which lead to minute air sacs called alveoli (see pulmonary alveolus), where the actual gas molecules of oxygen and carbon dioxide are exchanged between the respiratory spaces and the blood capillaries (Figure 1.5).

-5-

Chapter 1 – Introduction and Outline

Figure 1.3: Basic lung anatomy

Figure 1.4: Volume-rendered coronal image depicting vessel tree

-6-

Chapter 1 – Introduction and Outline

Figure 1.5: Airway anatomy

Each lung is divided into lobes separated from one another by a tissue fissure. The right lung has three major lobes; the left lung, which is slightly smaller because of the asymmetrical placement of the heart, has two lobes. Internally, each lobe further subdivides into hundreds of lobules. Each lobule contains a bronchiole and affiliated branches, a thin wall, and clusters of alveoli (Figure 1.6).

-7-

Chapter 1 – Introduction and Outline

Figure 1.6: Diagram illustrating the three main compartments of the pulmonary interstitium at the level of the secondary pulmonary lobule. The axial compartment surrounds the lobular artery and bronchiole, whereas the peripheral compartment is located at the level of the interlobular septum. These compartments are connected to each other by the intralobular interstitium. (Adapted from Beigelman-Aurbry et al.)

In addition to respiratory activities, the lungs perform other bodily functions. Through them, water, alcohol, and pharmacologic agents can be absorbed and excreted.

1.4 CT Appearance of ILD Patterns

ILD is a non specific term used to refer to any acute or chronic disease that results in a widespread or generalized lung involvement. They can reflect the presence of interstitial or airspace abnormalities. ILD represents a heterogeneous group of diseases with a wide range of associated histological abnormalities, treatment and prognosis. They comprise over 200 entities and include a wide spectrum of diseases, many uncommon and many of unknown etiology. Acute causes of ILD include pulmonary edema, diffuse alveolar damage, diffuse pulmonary hemorrhage,

-8-

Chapter 1 – Introduction and Outline

hypersensitivity pneumonitis, acute interstitial pneumonia and acute infectious bronchiolitis. Chronic causes of ILD include diffuse infiltrative lung disease, emphysema, chronic obstructive pulmonary disease and obliterative bronchiolitis. Diffuse infiltrative lung disease are characterized by diffuse infiltration of the lung parenchyma (see figure 1.5, intersitium and /or alveolar spaces) by inflammatory cells, fibrous tissue or other tissues or substances. The abnormalities present may be reversible or irreversible. Common causes of diffuse infiltrative lung disease include Idiopathic Pulmonary Fibrosis (IPF), sarcoidosis, collagen - vascular diseases (e.g. rheumatoid lung disease, progressive systemic sclerosis), pneumoconiosis (e.g. asbestosis, silicosis), cryptogenic organizing pneumonia and hypersensitivity pneumonitis [2]. Clinically however, the diseases have similar presentations with progressive dyspnoea, restriction on pulmonary function tests and widespread shadowing on the chest radiograph. A chest radiograph is the initial imaging investigation of suspected ILD, but its diagnostic value is limited due to its low sensitivity and specificity. High Resolution CT (HRCT) is the method of choice to image and manage diffuse lung disease, due to high spatial resolution (up to 0.5 mm), providing anatomical detail similar to that seen by gross pathological examination. HRCT protocols are acquired with thin collimation (usually 1 mm thick sections), slices obtained at 10 or 20 mm intervals, field of view adjusted to the size of the lungs and a high spatial frequency reconstruction algorithm. HRCT plays a crucial role in the investigation of ILD and is routinely used in the management of specific problems such as: detection and assessment of the pattern and distribution of the disease, assessment of activity and potential reversibility of diffuse lung disease, prediction of the response to therapy and likelihood of survival, selection of the type and location of the biopsy when required, follow-up of disease and evaluation of effectiveness of medical therapy.

On the other hand, thin slice MDCT protocols are acquired with thin collimation (0.5 - 1.0 mm thick sections), continuous rotation of the X-ray tube and continuous table feed, and a high spatial frequency reconstruction algorithm. In this way nearly isotropic volumetric high-resolution data are generated, allowing contiguous 3D visualization of lung parenchyma, with the capacity to create high-quality two

-9-

Chapter 1 – Introduction and Outline

dimensional (2D) and 3D reformatted images. These images have proven to be very useful in the evaluation of lung parenchyma and in the assessment of disease characteristics [2]. MDCT facilitates detection of various patterns of ILD. With this modality, it is easier to recognize the predominant pattern of distribution, which is important for developing the differential diagnosis [2.3]. Abnormalities are also more easily identified in relation to the underlying vascular, bronchial, and lobular anatomy. In the following subsections, a taxonomy of major ILD types, as well as examples of associated CT image patterns are presented.

1.4.1 Nodular Opacities

Nodular patterns include airspace nodules and interstitial nodules, both referring to multiple round opacities varying in diameter from 1 mm to 1 cm, with smooth or irregular margins in presence or absence of cavitation. According to their distribution, nodules can be classified as: perilymphatic nodules, centrilobular nodules and diffuse nodules [4].

1.4.2 Linear and Reticular Opacities

Linear and reticular opacities v, producing innumerable, interlacing line shadows suggesting a mesh are due to thickening of lung interstitium resulting from fibrosis, edema and infiltration of cells or other material [4,3]. The reticular pattern can be classified as large (thickening of interlobular septa), intermediate (concomitant with the honeycomb pattern) and fine (thickening of intra-lobular septa).

1.4.3 Ground Glass Opacities

Ground Glass Opacitiess is defined as “hazy” increased attenuation of lung, with preservation of bronchial and vascular margins [4]. It is a common but nonspecific

-10-

Chapter 1 – Introduction and Outline

finding indicating parenchymal abnormalities escaping the spatial resolution of HRCT. It can be due to partial filling of air spaces, interstitial thickening, partial collapse of alveoli, normal expiration and increased capillary blood volume. In the respective clinical and radiological context, it may lead to a specific diagnosis or more commonly represent areas of active inflammation, but ground glass can also be caused by microscopic fibrosis.

1.4.4 Decreased Lung Attenuation

Decreased lung attenuation is a characteristic finding in lung cysts, honeycombing and emphysema. A cyst is defined as a thin-walled well-defined and circumscribed, air- or fluid-containing lesion, with an epithelial or fibrous wall [4]. The presence of a definable wall differentiates cysts from centrilobular emphysema. A cystic pattern can be indicative of histiocytosis X, lymphangioleiomyomatosis and honeycombing. Honeycombing consists of cystic air space having thick, clearly definable walls lined with bronchiolar epithelium, predominantly in the basal and subpleural areas[5]. It is a radiological feature of alveolar destruction and of the loss of the acinar architecture indicating the end stage of the lung. Honeycombing, a typical feature of pulmonary idiopathic fibrosis, allows a confident diagnosis of the disease.

-11-

Chapter 1 – Introduction and Outline

1.4.5 Consolidation

A consolidation is characterized by a replacement of alveolar air by fluid, cells, tissue, or other substances that obscure underlying vascular structures. It is frequently associated with air bronchograms. Consolidation is commonly seen in cryptogenic organizing pneumonia, chronic eosinophilic and lipoid pneumonia[4]. Figure 1.7 illustrates normal lung (a) and patterns (b-f) in CT image data. Black arrows in (b), (c), (d), (e) and (f) indicate reticular, ground glass, honeycombing, emphysema and consolidation patterns, respectively.

Figure 1.7: CT appearance of ILD. (a) Normal lung parenchyma. (b) Reticular opacities, (c) Ground glass opacities, (d) Honeycomb, (e) Emphysema and (f) Consolidation, with corresponding regions of interest patterns (Adapted from P. Korfiatis - PhD Thesis, 2010).

-12-

Chapter 1 – Introduction and Outline

1.5 Thesis and Outline

This thesis deals with the effect of two types of registration optimizers, Quasi- Newton (QN) and Simultaneous Perturbation (SP), in registration accuracy of multiresolution registration schemes. Specifically, 64 multiresolution registration schemes comprised of 3 levels of rigid transformations and B-Spline transformation for the 4th level, as well as fully non-rigid schemes, have been generated and tested, corresponding to 4 different transform combinations (Euler, Similarity, Affine and B- Spline), 4 different cost functions (SSD, NCC, MI and NMI), 2 types of pyramids (Shrinking and Gaussian Smoothing) and one of the optimizers mentioned above, each time.

Artificially warped scan pairs simulate lung deformations as a result of respiration, without taking into account differences in lung density (intensity appearance), caused by the amount of air in the lungs.

The accuracy of the registration schemes tested was evaluated with the determinant of the Jacobian matrix of each voxel of the registered volumes, in order to exclude registration schemes whose determinants of Jacobian matrices include negative values. Then, the registration distance error was estimated using 2 sets of landmark points, corresponding to 2 different types of tissue in the lung fields: NLP and ILD affected tissue. In order to identify normal parenchyma lung tissue, an ILD region segmentation algorithm was applied to create binary masks corresponding to ILD affected and normal parenchyma regions.

The manuscript is organized as follows: In chapter 2 a review of medical image registration methodology is presented, including: algorithm classification, the basic components of the registration framework; deformation models, matching criteria (cost function) and optimization methods (optimizers), as well as the evaluation metrics. Chapter 3 presents the materials and methods of image registration used in this study. Chapter 4 presents and discusses the results of this study.

This study has been presented as a poster presentation titled “The importance of registration optimizer to ILD quantification and follow-up: Quasi-Newton and

-13-

Chapter 1 – Introduction and Outline

Simultaneous Perturbation” in BIOMEP 2015 – Conference on Bio-Medical Instrumentation and related Engineering and Physical Sciences, 18-20 June 2015, T.E.I. of Athens, Greece, as well as an oral presentation titled “The effect of mass preservation in the accuracy of image registration algorithms in thorax multidetector computed tomography”, in the 22nd Inter-University Conference of Radiology, 12-14 November 2015, Athens, Greece.

-14-

Chapter 2 – Image Registration

2. Image Registration

2.1 Introduction

Image registration is used to find the spatial correspondence between two images and plays an important role in pulmonary image analysis. In a sequence of pulmonary scans, image registration provides the spatial locations of corresponding voxels [6]. The computed correspondences describe the motion of the lung between a pair of images at the voxel level. Image registration of lung volumes has clinical significance in both radiotherapy, in order to calculate localized dose distributions, and monitoring of disease progression. Registration of lung volumes across time or across modalities has been utilized to establish lung atlases, estimate regional ventilation and local lung tissue expansion, assess lobar slippage during respiration, and measure pulmonary function change following radiation therapy.

A typical example of registration is intra-modal registration, where two medical images from the same patient and the same modality are taken at different time instances. It is very likely that the patient assumes a different position or volume during each acquisition. Additionally, in the case of lung, the images can have been taken in different phases of the respiration cycle. A registration procedure considers an image pair and uses a spatial transformation to find the corresponding pixels from one image into the other.

One of the images is referred to as the reference, source, target, fixed or baseline and the other is respectively referred to as the moving or template image. Image registration involves spatially registering the moving image to align it with the reference image.

Figure 2.1: A diagrammatic example of registration of two images

-15-

Chapter 2 – Image Registration

2.2 Algorithm Classification

2.2.1 Intensity-based vs Feature-based

Image registration algorithms can be classified into intensity-based and feature- based. Intensity-based methods compare intensity patterns in images via correlation metrics. Intensity-based methods register entire images.

Feature-based methods find correspondence between image features such as points, lines, and contours. Feature-based methods establish a correspondence between features in images. Knowing the correspondence between features, a geometrical transformation is then determined to map the baseline image to the moving image, thereby establishing point-by-point correspondence between the moving and baseline images.

2.2.2 Transformation Models

Image registration algorithms can also be classified according to the transformation models they use to relate the moving image volume to the reference image volume. The first broad category of transformation models includes linear (or rigid) transformations, which include rotation, scaling, translation, and other affine transforms. Linear transformations are global in nature, thus, they cannot model local geometric differences between images.

The second category of transformations allow 'elastic' or 'nonrigid' transformations. These transformations are capable of locally warping the moving image to align it with the reference image. Nonrigid transformations include radial basis functions (thin-plate or surface splines, multiquadratics, and compactly-supported transformations), physical continuum models (viscous fluids), and large deformation models (diffeomorphisms).

In the case of deformable registration, a common approach is to use mixed schemes with rigid and nonrigid transformation components in order to achieve low computational cost and high accuracy.

-16-

Chapter 2 – Image Registration

2.2.3 Spatial vs Frequency Domain methods

Spatial domain methods operate in the image domain, matching intensity patterns or features in images. Some of the feature matching algorithms are outgrowths of traditional techniques for performing manual image registration, in which an operator chooses corresponding control points in images. When the number of control points exceeds the minimum required to define the appropriate transformation model, iterative algorithms can be used to robustly estimate the parameters of a particular transformation type (e.g. affine) for registration of the images.

Frequency domain methods find the transformation parameters for registration of the images while working in the frequency domain. Such methods work for simple transformations, such as translation, rotation, and scaling.

2.2.4 Intramodal vs Intermodal Registration

In intramodal registration, we are registering images that stem from the same modality; an example is CT-to-CT registration of volumes acquired at different times, as in our case. Such a procedure is extremely helpful when doing time series evaluation, for instance when tracking the effect of chemo- or radiotherapy on tumor growth. Here, a number of CT images is taken at different time instances. The one quantity of interest is tumor size, which can be identified easily provided a sufficient uptake of contrast agent. But with varying orientation of the patient, segmentation of tumor and, above all, assessment of direction of tumor growth or shrinkage becomes extremely cumbersome. By registration of the volumes, this task is greatly simplified since all slices can be viewed with same orientation.

When fusing image data from different modalities, we are dealing with intermodal registration. Examples include registration of brain CT/MRI images or whole body PET/CT images for tumor localization, registration of contrast- enhanced CT images against non-contrast-enhanced CT images for segmentation of

-17-

Chapter 2 – Image Registration

specific parts of the anatomy, and registration of ultrasound and CT images for prostate localization in radiotherapy.

2.2.5 Automatic vs Interactive methods

Registration methods may be classified based on the level of automation they provide. Manual, interactive, semi-automatic, and automatic methods have been developed. Manual methods provide tools to align the images manually. Interactive methods reduce user bias by performing certain key operations automatically while still relying on the user to guide the registration. Semi-automatic methods perform more of the registration steps automatically but depend on the user to verify the correctness of a registration. Automatic methods do not allow any user interaction and perform all registration steps automatically.

2.3 Image Registration Framework

As previously mentioned, the goal of registration is to find the spatial mapping that aligns the moving image with the fixed image [6].

The input data to the registration framework are two image volumes: one is defined as the moving or template image, and the other is defined as the reference, source, target, fixed or baseline image [7].

The transform is used to map points between the two images spatially.

The cost function represents the similarity measure of how well the fixed image is aligned by the deformed moving image.

This measure forms the quantitative criterion which is optimized by the optimizer over the search space defined by the transformation parameters.

The interpolator is used to obtain the moving image intensities at non-grid positions using information from neighboring grid positions (Fig. 2.2).

-18-

Chapter 2 – Image Registration

Moving Image

optimization

transform optimizer

Registered Image

interpolator cost function

Fixed Image

Figure 2.2: Basic components of the registration framework

2.3.1 Deformation Models - Transformation Parameterization

The choice of deformation model is of great importance for the registration process as it entails an important compromise between computational efficiency and richness of description. It also reflects the class of transformations that are desirable or acceptable, and therefore limits the solution to a large extent. The parameters that registration estimates through the optimization strategy correspond to the degrees of freedom of the deformation model [8]. Their number varies greatly, from six in the case of global rigid transformations, to millions when non-parametric dense transformations are considered. Increasing the dimensionality of the parameters, results in enriching the descriptive power of the model. This model enrichment may be accompanied by an increase in the model’s complexity which, in turns, results in a more challenging and computationally demanding inference.

The transformation is assumed to map homologous locations from the moving anatomy to the fixed anatomy (backward mapping). While from a theoretical point

-19-

Chapter 2 – Image Registration

of view, the mapping from the fixed physiology to the moving physiology is possible (forward mapping), from an implementation point of view, this mapping is less advantageous.

In both cases, the moving image is warped to the target domain through interpolation resulting to a deformed image. When the forward mapping is estimated, every voxel of the moving image is pushed forward to its estimated position in the deformed image.

On the other hand, when the backward mapping is estimated, the pixel value of a voxel in the deformed image is pulled from the moving image.

The difference between the two schemes is in the difficulty of the interpolation problem that has to be solved. In the first case, a scattered data interpolation problem needs to be solved because the voxel locations of the moving image are usually mapped to nonvoxel locations, and the intensity values of the voxels of the deformed image have to be calculated. In the second case, when voxel locations of the deformed image are mapped to nonvoxel locations in the moving image, their intensities can be easily calculated by interpolating the intensity values of the neighboring voxels.

To represent the locally varying geometric distortions, the transformation can be represented by various forms of basis function, such as Fourier transform, thin-plate splines, and B-splines. B-splines are well suited for shape modeling and are efficient to capture the local nonrigid motion between two images, as in our case.

Geometric transformations can be classified into three main categories (Figure 2.3): 1) those that are inspired by physical models, 2) those inspired by interpolation and approximation theory, 3) knowledge-based deformation models that opt to introduce specific prior information regarding the sought deformation, and 4) models that satisfy a task-specific constraint [8].

-20-

Chapter 2 – Image Registration

Figure 2.3: Classification of deformation models.

Of great importance for biomedical applications are the constraints that may be applied to the transformation such that it exhibits special properties. Such properties include, but are not limited to, inverse consistency, symmetry, topology preservation and diffeomorphism. The value of these properties was made apparent to the research community and were gradually introduced as extra constraints.

Despite common intuition, the majority of the existing registration algorithms are asymmetric. As a consequence, when interchanging the order of input images, the registration algorithm does not estimate the inverse transformation. As a consequence, the statistical analysis that follows registration is biased on the choice of the target domain.

-21-

Chapter 2 – Image Registration

Inverse Consistency: Inverse consistent methods aim to tackle this shortcoming by simultaneously estimating both the forward and the backward transformation. The data matching term quantifies how well the images are aligned when one image is deformed by the forward transformation, and the other image by the backward transformation. Additionally, inverse consistent algorithms constrain the forward and backward transformations to be inverse mappings of one another. This is achieved by introducing terms that penalize the difference between the forward and backward transformations from the respective inverse mappings. Inverse consistent methods can preserve topology but are only asymptotically symmetric. Inverse- consistency can be violated if another term of the objective function is weighted more importantly.

Symmetry: Symmetric algorithms also aim to cope with asymmetry. These methods do not explicitly penalize asymmetry, but instead employ one of the following two strategies. In the first case, they employ objective functions that are by construction symmetric to estimate the transformation from one image to another. In the second case, two transformation functions are estimated by optimizing a standard objective function. Each transformation function map an image to a common domain. The final mapping from one image to another is calculated by inverting one transformation function and composing it with the other.

Topology Preservation: The transformation that is estimated by registration algorithms is not always one-to-one and crossings may appear in the deformation field. Topology preserving/homeomorphic algorithms produce a mapping that is continuous, onto, and locally one-to-one and has a continuous inverse. The Jacobian determinant contains information regarding the injectivity of the mapping and is greater than zero for topology preserving mappings. The differentiability of the transformation needs to be ensured in order to calculate the Jacobian determinant.

Diffeomorphism: Diffeomoprhic transformations also preserve topology. A transformation function is a diffeomorphism, if it is invertible and both the function and its inverse are differentiable. A diffeomorphism maps a differentiable manifold to another.

-22-

Chapter 2 – Image Registration

In the following four subsections, the most important methods of the four classes are presented with emphasis on the approaches that endow the model under consideration with the above desirable properties.

A. Geometric Transformations Derived From Physical Models

Currently employed physical models can be further separated in five categories (see Fig. 2.4): 1) elastic body models, 2) viscous fluid flow models, 3) diffusion models, 4) curvature registration, and 5) flows of diffeomorphisms [8].

Elastic body models

Viscous fluid flow models

Geometric transformations Diffusion models derived from physical models

Curvature registration

Flows of diffeomorphisms

Figure 2.4: Classification of geometric transformations derived from physical models.

B. Geometric Transformations Derived From Interpolation Theory

Rather than being motivated by a physical model, the models of this class are derived from either interpolation theory or approximation theory. In interpolation theory, displacements, considered known in a restricted set of locations in the image, are interpolated for the rest of the image domain. In approximation theory, we assume that there is an error in the estimation of displacements. Thus, the transformation smoothly approximates the known displacements rather than taking

-23-

Chapter 2 – Image Registration

the exact same values. These models are rich enough to describe the transformations that are present in image registration problems, while having low degrees of freedom and thus facilitating the inference of the parameters. Among the most important families of interpolation strategies, one may cite (see Fig. 2.5): 1) radial basis functions, 2) elastic body splines, 3) free-form deformations, 4) basis functions from signal processing, and 5) piecewise affine models.

Radial basis functions

Elastic body splines

Geometric transformations Free-form deformations derived from interpolation theory

Basis functions from signal processing

Locally affine models

Figure 2.5: Classification of geometric transformations derived from interpolation theory.

C. Knowledge-Based Geometric Transformations

In medical image analysis, there are registration scenarios that involve a specific well-defined task. More specifically, registration is either performed between any image and a specific target image or involves image acquisitions of specific anatomical organs. In these cases, it is possible to introduce knowledge about the deformations one tries to recover.

Introducing knowledge regarding the deformation may be achieved in two ways. In the case that the target domain is fixed in registration because it exhibits desired properties (e.g., it is manually annotated), one can learn a high dimensional

-24-

Chapter 2 – Image Registration

statistical model of deformations by performing pairwise registrations between the target image and the data that one has at their disposition. Subsequently, when a new image is to be registered to the target image, the learned model can be used to penalize configurations that diverge from it. The second method consists of exploiting our knowledge about the deformability of the tissues and constructing biomechanical/biophysical deformation models that mimic their properties (see Fig. 2.6).

The main motivation behind creating more informed priors is to render the registration method more robust and stable. A registration method is characterized as robust, when its performance does not drastically degrade for small deviations of the input images from the nominal assumptions. In other words, the presence of a small fraction of artifacts or outliers results in small changes in the result. A registration method is characterized as stable, when small changes in the input data result in small changes in the result.

Statistically-constrained geometric transformations

Knowledge-based geometric transformations Geometric transformations inspired by biomechanical/biophysical models

Figure 2.6: Classification of knowledge-based geometric transformations.

D. Task-Specific Constraints

Unregularized optimization of similarity measures for high-dimensional deformable transformation models is, in general, an ill-posed problem. In order to cope with the

-25-

Chapter 2 – Image Registration

difficulty associated with the ill-posedness of the problem, regularization is necessary.

There are two possible ways to regularize the problem: implicitly and explicitly. Implicit regularization may be achieved by parameterizing the deformation field with smooth functions. Explicit regularization may be achieved through the use of either hard constraints or soft constraints. Hard constraints are the constraints that the solution must satisfy in order for the registration to be successful. Soft constraints are introduced as additional terms in the energy function that penalize nonregular configurations. Soft constraints encode our preference regarding specific con figurations, but deviations from the preferred configurations are allowed if driven by the other term(s) of the energy function. Physics-based deformation models are typical examples of explicit regularization. Moreover, explicit regularization may be used to achieve specific goals that are tailored to the problem at hand. Such goals include (see Fig. 2.7): 1) topology preservation, 2) volume preservation, and 3) rigidity constraints.

Topology preservation

Task-specific constraints Volume preservation

Rigidity constraints

Figure 2.7: Classification of task-specific constraints

2.3.2 Matching Criteria - Cost Function Design

We can distinguish three groups of registration methods according to how they exploit the available information to drive the matching process (Figure 2.8) [8].

On one hand, geometric methods opt for the establishment of correspondences between landmarks. The landmarks are assumed to be placed in salient image

-26-

Chapter 2 – Image Registration

locations which are considered to correspond to meaningful anatomical locations. The underlying assumption is that saliency in the image level is equivalent to anatomical regions of interest. Geometric registration is robust with respect to the initial conditions and the existence of large deformations. The solution of the registration problem is obtained in a relatively straightforward way once landmarks have been extracted. However, locating reliable landmarks is an open problem and an active topic of research. Most importantly, the sparse set of directly obtained correspondences gives rise to the need for extrapolation. Interpolation results in a decrease in accuracy as the distance from the landmarks increases. Geometric methods constitute a reliable approach for specific applications. They are of interest when intensity information is undermined due to the presence of pathologies while geometric structures remain stable. Geometric registration has also important applications in image-guided interventions.

On the other hand, iconic methods, often referred to as either voxel-based or intensity-based methods, quantify the alignment of the images by evaluating an intensity-based criterion over the whole image domain. When compared to the geometric methods, this approach has the potential to better quantify and represent the accuracy of the estimated dense deformation field. Nonetheless, it comes at the cost of increased computational expense. Where geometric methods use a small subset of image voxels to evaluate the matching criterion, iconic methods may use the mall. Moreover, due to the fact that salient points are not explicitly taken into account by the matching criterion, the important information they contain is not fully exploited to drive the registration. In addition, initial conditions greatly influence the quality of the obtained result due to the nonconvexity of the problem.

Hybrid methods combine both types of information in an effort to get the best of both worlds.

-27-

Chapter 2 – Image Registration

Figure 2.8: Classification of cost functions

A. Geometric Methods

Geometric methods aim to register two images by minimizing a criterion that takes into account landmark information. The known variables consist of two sets of landmarks which can be created using a key-point detector strategy. The first set of landmarks contains points belonging to the baseline domain Ωbaseline, while the second contains points that belong to the moving one Ωmoving. The set of unknown variables comprises: 1) the correspondence, and 2) the transformation .

-28-

Chapter 2 – Image Registration

The first step in geometric registration is to detect points of interest. Images that contain sufficient details facilitate point detection. Medical images are not as rich in details as natural images. That is why, point detection has mainly drawn the interest of the computer vision community. Landmark extraction has been studied more in the case of 2D images and less in the case of 3D images.

The detection and the matching of points of interest are inherently coupled with the way the landmarks are described. The richness of the description is important in order to detect salient points and better disambiguate between close potential candidates during matching. Moreover, as the imaged objects undergo deformations, the appearance of the points of interest will vary between images. Therefore, descriptors should be invariant to such changes in order to allow robust detection and matching under deformations.

Three classes of methods can be separated based on which unknown variable is estimated (see Fig.2.9): 1) methods that infer only the correspondence, 2) methods that infer only the spatial transformation, and 3) methods that infer both variables.

Matching by descriptor Methods that infer only distance the correspondenses Matching through geometric constraints

Known correspondences Methods that infer only Geometric methods the spatial transformation Unknown correspondences

Methods that infer both the correspondences and the transformation

Figure 2.9: Classification of geometric methods

-29-

Chapter 2 – Image Registration

B. Iconic Methods

In iconic methods, the matching term integrates the evaluation of a dissimilarity criterion that takes into account the intensity information of the image elements. Devising an appropriate criterion is an important and difficult task. The criterion should be able to account for the different physical principles behind the acquisition of the two images and thus for the intensity relation between them. Moreover, the properties of the similarity function (e.g. its convexity) may influence the difficulty of the inference and thus the quality of the obtained result.

An ideal dissimilarity criterion would take low values when points belonging to the same tissue class are examined and high values when points from different tissue classes are compared. Moreover, an ideal criterion should be convex, allowing for accurate inference. There is an important balance that should be struck between the convexity and ability to distinguish between points belonging to different tissues. On the one hand, convexifying the objective function will facilitate the solution of the problem. On the other hand, it may lead to a less realistic problem because the problem is nonconvex in its nature.

At this point, two cases should be distinguished regarding the iconic methods (see also Fig. 2.10): 1) the mono-modal case (intramodal), involving images from one modality, and 2) the multi-modal one (intermodal), involving images from multiple modalities.

-30-

Chapter 2 – Image Registration

Intensity-based methods

Mono-modal registration Attribute-based methods Iconic methods Information-theoretic approaches Multi-modal registration Reduction to mono-modal registration

Simulating one Mapping both modality from another modalities to a common one

Figure 2.10: Classification of iconic methods

C. Hybrid Methods

Iconic and geometric registration methods each bear certain advantages while suffering from shortcomings. Hybrid methods try to capitalize on the advantages of each by using complementary information in an effort to get the best of both worlds. Among hybrid methods, the subclasses shown in Fig. 2.11 may be distinguished based on the way the geometric information is exploited.

Additional information used independently

Hybrid methods Additional information used as constraint

Coupled approaches

Figure 2.11: Classification of hybrid methods

-31-

Chapter 2 – Image Registration

2.3.3 Optimization Methods

The aim of optimization is to infer the optimal transformation that best aligns two images according to an objective function comprising a matching term and a regularization term. As a consequence, the choice of the optimization methods impacts the quality of the obtained result.

Optimization methods may be separated into two categories based on the nature of the variables that they try to infer: 1) continuous, and 2) discrete (Figure 2.12) [8].

The first class of methods solves optimization problems where the variables take real values and the objective function is differentiable.

On the contrary, methods in the second class solve problems where the variables take values from a discrete set. Both classes of methods are constrained with respect to the nature of the objective function as well as the structure to be optimized.

Figure 2.12: Classification of optimization methods

-32-

Chapter 2 – Image Registration

A. Continuous Optimization

Continuous optimization methods are constrained to problems where the variables take real values and the objective function is differentiable. Image registration is a problem where the application of continuous optimization methods has been studied. Continuous optimization methods estimate the optimal parameters following an update rule of the following form: 휽푡+1 = 휽푡 + 푎푡퐠푡(휽푡) , where 휽 denotes the vector of parameters of the transformation, t indexes the number of iteration, 푎푡 is the step size or gain factor, and 퐠 defines the search direction. The search direction is calculated by taking into account both the matching and the regularization term.

There are various ways to define the previous parameters. The search direction can be specified by exploiting only first-order information or, for example, by also taking into consideration second-order information. It is the choice of these parameters that distinguishes different methods. Commonly used methods include (see also Fig. 2.13): 1) gradient descent (GD), 2) conjugate gradient (CG), 3) Powell’s conjugate directions, 4) Quasi-Newton (QN), 5) Levenberg-Marquardt (LM) , and 6) stochastic gradient descent.

Gradient descent methods

Conjugate gradient methods

Powell’s conjugate directions methods

Continuous methods Quasi-Newton methods

Gauss-Newton method

Levenberg-Marquardt algorithm

Stochastic gradient descent methods

Figure 2.13: Classification of continuous optimization methods

-33-

Chapter 2 – Image Registration

B. Discrete Optimization

Discrete optimization methods are constrained to problems where the variables take discrete values. Discrete optimization methods can be classified according to the techniques they employ into three categories (see also Fig. 2.14): 1) graph-based methods, 2) message passing methods, and 3) linear-programming (LP) approaches.

Graph-based methods

Discrete methods Belief propagation methods

Linear-programming approaches

Figure 2.14: Classification of discrete optimization methods

2.3.4 Multiresolution Scheme

Multiresolution strategy helps improve the computational efficiency and avoid some local minima. A spatial multiresolution procedure from coarse to fine resolution is used in the registration in order to improve speed, accuracy and robustness. The basic idea of multiresolution is that registration is first performed at a coarse scale where the images have much fewer pixels, which is fast and can help eliminate local minima. The resulting spatial mapping from the coarse scale is then used to initialize registration at the next finer scale. This process is repeated until registration is performed at the finest scale. In that case, we talk about a “pyramid” (Fig. 2.15).

In this study we use a 4-level resolution pyramid, performing rigid transformation for the first three levels (for computational time considerations) and non-rigid transformation for the 4th level.

-34-

Chapter 2 – Image Registration

Figure 2.15: Multiresolution pyramid

2.3.5 Interpolators

During the optimization, some pixel values are evaluated at non-voxel positions, for which intensity interpolation is needed. Several methods for interpolation exist, varying in quality and speed [29].

Nearest neighbour: This is the most simple technique, low in quality, requiring little resources. The intensity of the voxel nearest in distance is returned.

Linear: The returned value is a weighted average of the surrounding voxels, with the distance to each voxel taken as weight.

N-th order B-spline: The higher the order, the better the quality, but also requiring more computation time. In fact, nearest neighbour (N = 0) and linear interpolation (N = 1) also fall in this category.

-35-

Chapter 2 – Image Registration

During registration a first-order B-spline interpolation, i.e. linear interpolation, often gives satisfactory results. It is a good trade-off between quality and speed. To generate the final result, i.e. the deformed result of the registration, a higher-order interpolation is usually required.

2.4 Evaluation methods

2.4.1 Landmark Matching Accuracy

Landmarks are point features of an object. Anatomical landmarks have biological meaning. In the lung CT image, the vascular tree and airway tree can be extracted by intensity contrast, and their bifurcations can then be identified. Points at bifurcations of vessel and airways provide good candidates for landmarks [7].

The distance between corresponding landmarks is called the landmark error. Landmark error is calculated by the Euclidean distance from its estimated to real position (Fig. 2.16). The landmarks are well dispersed throughout the lungs and lie in the regions of good grayscale contrast. Large error reduction mostly happens with landmarks located in the regions near diaphragm.

Figure 2.16: Illustration of the landmark error calculation (Adapted from K. Cao PhD Thesis 2012)

-36-

Chapter 2 – Image Registration

2.4.2 Vessel Matching Accuracy

Vessels in the lung form tree structures and they keep their tubular shape and tree structures during the respiratory process. The vascular tree provide us rich spatial and shape information in parenchyma regions. Therefore, evaluating the alignment on vessel trees is an important perspective to validate matching accuracy at the lung feature level [7].

The registration accuracy on the vessel tree are evaluated by vessel matching distance which is calculated as the distance between a point on the target vessel tree and its closet point on warped template vessel tree.

Figure 2.17: Vessel matching distance (mm) on target vessel tree. Arrows denote regions of large discrepancies between the deformed source and target vessel trees. (Adapted from K. Cao PhD Thesis 2012)

2.4.3 Fissure Alignment Distance

The human lungs are divided into five independent compartments which are called lobes. Lobar fissures are the division between adjacent lung lobes. The left lung is divided into the left upper (LUL) and left lower (LLL) lobes, separated by the oblique fissure. The right lung is partitioned into the right upper lobe (RUL), middle lobe

-37-

Chapter 2 – Image Registration

(RML), and the lower lobe (RLL), separated by the oblique and horizontal fissures (Fig. 2.17). Since fissures represent important physical boundaries within the lungs, their alignment result is included as an evaluation category [7].

Figure 2.18: Lobe division

To evaluate the fissure alignment, the fissure positioning error (FPE) is used and determined by comparing the distance between the transformed fissure and target fissure. The FPE is defined as the minimum distance between a point on the deformed fissure and the closest point on the corresponding target fissure.

2.4.4 Lung Segmentation Overlap

A more global feature-based registration validation approach compared to methods mentioned above is to calculate the relative overlap (RO) between the segmentations of images. The alignment of objects or regions of interests (ROIs) is one perspective to indicate how well two images are matched [7].

The RO metric has several limitations for evaluating registration algorithms: it gives larger (better) values for large object compared to smaller objects; there may be biases and errors due to differences in anatomy and error in the hand segmentations; a high relative overlap does not ensure the shape of the two segmentations are close enough since this metric does not reflect the local shape difference. The RO metric is limited to measure how well two binary segmentations align, however, it cannot give a measure of the inside microscopic local structure

-38-

Chapter 2 – Image Registration

matching. Therefore, volume overlap may be considered a better global performance measure than local performance measure.

2.4.5 Evaluation Using Transformation Properties

Good matching accuracy on the feature locations does not guarantee that the parenchymal tissues are correctly aligned. In order to reveal the lung tissue deformation pattern, the Jacobian determinant of the transformation field derived by image registration is used to estimate the local tissue deformation.

The Jacobian determinant (often simply called the Jacobian) [31], [32], [33] is a measurement to estimate the pointwise expansion and contraction during the deformation. The Jacobian of the transformation 퐽(퐡(퐱)) is defined as

휕ℎ1(퐱) 휕ℎ1(퐱) 휕ℎ1(퐱) | 휕푥1 휕푥2 휕푥3 | 휕ℎ (퐱) 휕ℎ (퐱) 휕ℎ (퐱) 퐽(퐡(퐱)) = 2 2 2 (2.1) 휕푥1 휕푥2 휕푥3 | | 휕ℎ3(퐱) 휕ℎ3(퐱) 휕ℎ3(퐱)

휕푥1 휕푥2 휕푥3

Using a Lagrangian reference frame, a Jacobian value of one corresponds to zero expansion or contraction. Local tissue expansion corresponds to a Jacobian greater than one and local tissue contraction corresponds to a Jacobian less than one.

퐽 > 1, local expansion

퐽 > 0, preserve orientation { 퐽 = 1, no deformation 0 < 퐽 < 1, local contraction (2.2)

퐽 = 0, non − injective { 퐽 < 0, reverse orientation

-39-

Chapter 2 – Image Registration

2.5 Clinical Significance and “State of the Art” of Medical Image Registration

The term interstitial lung disease (ILD) refers to more than 200 chronic lung disorders that are classified to the same group because of their similar clinical, radiological, physiologic, and pathologic features. Accurate chest CT quantification of ILD extent is crucial for patient management, since no robust biomarkers for monitoring disease progression exist. Currently, ILD extent estimation is performed by several semi- quantitative scoring systems, estimating extent on 2D axial slices, utilizing high- resolution CT (HRCT) protocols. Such scoring systems are deprived of information regarding localization of disease, demonstrating moderate inter- and intra-observer agreement. Thus, the development of accurate and reproducible image analysis methods, adapted to volumetric datasets, for the automated estimation of ILD extent and progression is emerging.

In the frame of quantitative image-based follow-up ILD monitoring and response to therapy in CT, image registration methods have the important role to ensure that any measured volume change between follow-up scans is caused by ILD patterns change and not by patient’s breathing or positioning during CT scanning. Up to now, only one study has reported on ILD follow-up monitoring in the context of 2D axial slices of baseline and follow-up scans, subjected to volume registration.

Besides ILD monitoring, image registration has received considerable attention in lung CT image analysis applications, such as atlas registration-based segmentation, lung nodule monitoring, emphysema monitoring and estimation of local ventilation and perfusion, and motion estimation for radiotherapy planning, exploiting various inspiration/expiration protocols.

Multiresolution nonrigid registration, capable for capturing local lung tissue deformations, accounts for a commonly used approach for lung CT registration utilized in all resolution levels, or in combination with rigid transforms applied in low

-40-

Chapter 2 – Image Registration

resolution levels for computational time considerations. Fully nonrigid registration schemes had also been exploited adopting constraints to avoid unrealistic deformations introduced by folding areas, or local rigidity penalty term to penalize the deformation of rigid objects such as bones. Recently, boosting of deformable registration by targeting erroneously registered lung regions has been introduced.

Besides the transformation model, the choice of cost function and optimizer accounts for critical selections in a registration scheme. Commonly used cost functions are sum of squared differences (SSD) and its variants, mutual information (MI) and its variants, normalized mutual information (NMI), and normalized correlation coefficient (NCC), while commonly used optimizers considered include variants of gradient descent optimizers and quasi-Newton. Finally, while most applications adopt the Gaussian pyramid (GP), simple downsampling and the Laplacian pyramid have also been adopted.

Quantitative state-of-the-art approaches for evaluating registration accuracy consider distance measures as the metric of choice. Such metrics include distance of corresponding landmark points, as well as alignment of anatomical structures such as lung boundary, fissures, and vessel tree center lines. The lung overlap error accounts for a coarse registration performance evaluation metric, which however has been commonly used. Finally, the determinant of Jacobian matrix of the deformation field is used to detect singularities of deformation fields.

Quantitative evaluation of image registration using clinical data is susceptible to both algorithm and intrinsic image data variability. Thus, the artificial data generation obtained by applying a known deformation to a fixed image has been suggested, ensuring that the variability is only due to the registration algorithm alone.

-41-

Chapter 2 – Image Registration

2.6 Summary

In this chapter, we introduced the algorithm classification and the components of image registration. We also highlighted the importance of a multiresolution scheme for avoiding local minima and for computational time reduction. Furthermore, we discussed some approaches to evaluate registration schemes, specifically in the case of lung images. Finally, the clinical significance and the “state of the art” of medical image registration.

The next chapter presents the registration schemes that we generated which, according to this chapter, can be classified as: intensity based, rigid for the first 3 levels of the pyramid and non-rigid for the final level, spatial domain, intramodal (CT images), automatic, using geometric transformations and continuous optimization methods.

-42-

Chapter 3 – Materials and methods

3. Materials and Methods

3.1 Method Overview

A pilot clinical data set was analyzed consisting of 5 pairs of MDCT scans corresponding to 5 patients diagnosed with ILD secondary to connective tissue diseases, at two different instances, abstaining in time approximately two years. All patients were scanned with a 16-row multidetector CT (MDCT) scanner (GE Lightspeed 16, General Electric Medical Systems, Milwaukee, WI) at 120 kVp, rotation time of 0.5 s, automatic modulation of mA, collimation thickness of 16 × 0.625 mm, and slice thickness of 1.25 mm, using a protocol obtaining volumetric 3D data at full inspiration, in supine position. Each scan volume comprised of approximately 200–250 slices/patient. The mean volume CT dose index and the mean dose-length product, provided by CT application panel, were 11.2 mGy and 266.6 mGy cm, respectively. Assuming 0.017 mSv/mGy cm for a standard chest CT examination, the effective radiation dose for the volumetric chest CT protocol used was 4.5 mSv, complying with European Working Group for Guidelines on Quality Criteria in CT.

Each lung field dataset resulted from the application of a 3D lung segmentation algorithm on original thorax CT scan data [34]. Five (5) artificially warped 3D lung field generated by the clinical dataset by the following procedure: 100 well- distributed landmark points were automatically generated in the baseline 3D lung segment volume and matched semi-automatically in the follow-up 3D lung segment, using method proposed by Murphy et al. [35] and also used by Vlachopoulos et al. [24] (see Fig. 3.1, 3.2).

The method can simply described as following: A well-distributed set of 푛 landmarks is detected fully automatically in one scan of a pair to be registered. Using a custom- designed interface, observers define corresponding anatomic locations in the second scan for a specified subset of 푠 of these landmarks. The remaining 푛 − 푠 landmarks

-43-

Chapter 3 – Materials and methods

are matched fully automatically by a thin-plate-spline based system using the 푠 manual landmark correspondences to model the relationship between the scans.

A thin-plate kernel spline [36],[26] model was created using these 100 pairs of matching points defining a thin-plate transformation model, which was then applied to each 3D baseline image segment. Using this model, the baseline 3D lung segment was warped to create the artificial 3D lung segment, which simulates lung deformations in follow-up data (Fig. 3.3). The artificial data preserve size and slice thickness of the original dataset. Thus, an artificially warped lung segment pair consists of the original baseline lung segment and the artificially warped version of it. Artificially warped scan pairs simulate lung deformations as a result of respiration, without taking into account differences in lung density (intensity appearance), caused by the amount of air in the lungs.

An inherent error involved with the warped data generation is related with matching accuracy error of the semi-automated method, which according to Murphy et al. [35] is expected to be less than 1 mm in case of 15 points used as training set, as in our case. This approach was adopted as a first step in performance evaluation, in order to avoid intrinsic errors introduced by disease progression or regression, while providing repeatable ground truth data of realistic appearance.

Figure 3.1: 100 well-distributed landmark points were automatically generated in the baseline 3D lung segment volume and matched semi-automatically in the follow-up 3D lung segment, using method proposed by Murphy et al, 2011 IEEE

-44-

Chapter 3 – Materials and methods

Figure 3.2: An example of the landmark points identified in a fixed scan. Landmarks have been projected onto a single slice (maximum intensity projection image is shown here) and markers are increased in size for visualization (image adapted by Murphy et aI,2011 IEEE).

Warped Baseline segmentation Baseline voxels Lung Field Scan Lung Field (Fixed) (Moving)

100 landmark points thin-plate-spline 100 corresponding model landmark points

Follow-up segmentation Follow-up Scan Lung Field

Figure 3.3: Procedure for warping a scan artificially: The baseline and the follow-up scan images are segmented to create a baseline and a follow-up lung field image. A thin-plate- spline model is created using 100 pairs of landmark points between the baseline and the follow-up lung field. Using this thin-plate-spline model and linear interpolation, the baseline image is warped to create an image with the same image size and spacing as the follow-up scan.

-45-

Chapter 3 – Materials and methods

Warped Lung Field (Moving) deformation optimization

transform optimizer

Registered Lung Field

interpolator cost function

Baseline Lung Field (Fixed)

Figure 3.4: Image registration procedure

The registration schemes were performed in the frame of elastix, which is an open source software based on the Insight Segmentation and Registration Toolkit (ITK). The software consists of a collection of algorithms that are commonly used to solve (medical) image registration problems. The modular design of elastix allows the user to quickly configure, test, and compare different registration methods for a specific application. A command-line interface enables automated processing of large numbers of data sets, by means of scripting.

A total of 64 different registration schemes were generated by considering all possible combinations among: 2 different types of pyramids (Shrinking, which applies no smoothing, but only down-sampling by a factor of 2 in all three dimensions, and Gaussian Smoothing, which applies smoothing and down-sampling by a factor of 2 in all three dimensions), 4 different cost functions (Sum of Square Differences – SSD, Normalized Correlation Coefficient – NCC, Mutual Information – MI and Normalized Mutual Information – NMI), 4 different types of transforms for

-46-

Chapter 3 – Materials and methods

the first three levels (Euler, Similarity, Affine and 3rd order B-Splines) in order to obtain a coarse initial alignment, while in all cases for the 4th level, corresponding to the highest image resolution, the 3rd order B-Spline transform was selected for refinement purposes. Finally, 2 different types of optimizers were considered (Quasi- Newton (QS) and Simultaneous Perturbation (SP)) in order to compare their performances and evaluate their effect to registration accuracy (Figure 3.4, 3.5, 3.6 Table 3.1).

The registration schemes are evaluated using two distance metrics, the distance between corresponding points: (a) in ILD affected regions and (b) in normal parenchyma. Prior to evaluation, the registered lung volumes were prescreened for exclusion of schemes introducing folding regions. Irregular deformations were assessed in terms of the determinant of the Jacobian matrix of each voxel of registered volumes (deformation field). Registration schemes whose determinants of Jacobian matrices include negative values are excluded for subsequent analysis, since such areas correspond to singularities deformation fields (foldings).

Baseline Lung Field

Registration Registered Evaluation Scheme 푖 Lung Field 푖 Warped Lung Field

Optimal Scheme

Figure 3.5: Procedure for selecting the optimal registration scheme among 풊 = ퟑퟐ schemes generated for each optimizer tested

-47-

Chapter 3 – Materials and methods

Transform Pyramid Cost Function Optimizer (4 levels)

Euler/B-Spline SSD

Recursive Quasi Gaussian Newton Similarity/B-Spline NCC

Affine/B-Spline MI

Shrinking Simultaneous Perturbation B-Spline/B-Spline NMI

Figure 3.6: Components combinations of the registration schemes

-48-

Chapter 3 – Materials and methods

Parameter Cost Transform Pyramid Optimizer set Function

par01 Euler par02 Similarity Recursive par03 Affine Gaussian par04 B-Spline SSD par05 Euler par06 Similarity Shrinking par07 Affine par08 B-Spline par09 Euler par10 Similarity Recursive par11 Affine Gaussian par12 B-Spline NCC par13 Euler par14 Similarity Shrinking par15 Affine par16 B-Spline QN / SP par17 Euler par18 Similarity Recursive par19 Affine Gaussian par20 B-Spline MI par21 Euler par22 Similarity Shrinking par23 Affine par24 B-Spline par25 Euler par26 Similarity Recursive par27 Affine Gaussian par28 B-Spline NMI par29 Euler par30 Similarity Shrinking par31 Affine par32 B-Spline

Table 3.1: Parameters used in the registration schemes generated

-49-

Chapter 3 – Materials and methods

3.2 Transformation

A frequent confusion about the transformation is its direction. In elastix the transformation is defined as a coordinate mapping from the fixed image domain to

d d the moving image domain: 푇: 훺퐹 ⊂ R → 훺푀 ⊂ R [29]. The confusion usually stems from the phrase: “the moving image is deformed to fit the fixed image”. Although one can speak about image registration like this, such a phrase is not meant to reflect mathematical underlyings: one deforms the moving image, but the transformation is still defined from fixed to moving image. The reason for this becomes clear when trying to compute the deformed moving image (the registration result) 퐼푀(푻휇(푥)) (this process is frequently called resampling). If the transformation would be defined from moving to fixed image, not all voxels in the fixed image domain would be mapped to (e.g. in case of a scaling), and holes would occur in the deformed moving image. With the transformation defined as it is, resampling is quite simple: loop over all voxels 푥 in the fixed image domain 훺퐹, compute its mapped position 푦 = 푻휇(푥), interpolate the moving image at 푦, and fill in this value at 푥 in the output image.

The transformation model used for 푻휇 determines what type of deformations between the fixed and moving image you can handle.

3.2.1 Rigid (Euler) Transformation

A rigid transformation is defined as:

푻휇(푥) = 푹(푥 − 푐) + 푡 + 푐 (3.1) with the matrix 푹 a rotation matrix (i.e. orthonormal and proper), 푐 the centre of rotation, and 푡 the translation vector. The image is treated as a rigid body, which can translate and rotate, but cannot be scaled/stretched. The rotation matrix is parameterised by the Euler angles (one in 2D, three in 3D). The parameter vector 휇 consists of the Euler angles (in rad) and the translation vector. In 2D, this gives a

-50-

Chapter 3 – Materials and methods

푇 vector of length 3: 휇 = (휃푧, 푡푥, 푡푦) , where 휃푧 denotes the rotation around the axis 푇 normal to the image. In 3D, this gives a vector of length 6: = (휃푥, 휃푦, 휃푧, 푡푥, 푡푦, 푡푧) . The centre of rotation is not part of 휇; it is a fixed setting, usually the centre of the image.

3.2.2 Similarity Transformation

Α similarity transformation is defined as:

푻휇(푥) = 푠푹(푥 − 푐) + 푡 + 푐 (3.2) with 푠 a scalar and 푹 a rotation matrix. This means than the image is treated as an object, which can translate, rotate, and scale isotropically. The rotation matrix is parameterised by an angle in 2D, and by a so-called “versor” in 3D (Euler angles could have been used as well). The parameter vector 휇 consists of the angle/versor, the translation vector, and the isotropic scaling factor. In 2D, this gives a vector of 푇 length 4: 휇 = (푠, 휃푧, 푡푥, 푡푦) . In 3D, this gives a vector of length 7: 푇 휇 = (푞1, 푞2, 푞3, 푡푥, 푡푦, 푡푧, 푠) , where 푞1, 푞2 and 푞3 are the elements of the versor.

3.2.3 Affine Transformation

An affine transformation is defined as:

푻휇(푥) = 푨(푥 − 푐) + 푡 + 푐 (3.3) where the matrix 푨 has no restrictions. This means that the image can be translated, rotated, scaled, and sheared. The parameter vector 휇 is formed by the matrix elements 푎푖푗 and the translation vector. In 2D, this gives a vector of length 6: = 푇 (푎11, 푎12, 푎21, 푎22, 푡푥, 푡푦) . In 3D, this gives a vector of length 12.

-51-

Chapter 3 – Materials and methods

3.2.4 B-Spline Transformation

For the category of non-rigid transformations, B-splines [23] are often used as a parameterisation:

푥 − 푥푘 푻 (푥) = 푥 + ∑ 푝 훽3 ( ) (3.4) 휇 푘 휎 푥푘∈푁푥

3 with 푥푘 the control points, 훽 (푥) the cubic multidimensional B-Spline polynomial, 푝푘 the B-spline coefficient vectors (loosely speaking, the control point displacements), 휎 the B-spline control point spacing, and 푁푥 the set of all control points within the compact support of the B-Spline at 푥.

The control points 푥푘 are defined on a regular grid, overlayed on the fixed image. In this context we talk about ‘the control point grid that is put on the fixed image’, and about ‘control points that are moved around’. Note that 푻휇(푥푘) ≠ 푥푘 + 푝푘 , a common misunderstanding. Calling 푝푘 the control point displacements is, therefore, actually somewhat misleading. The control point grid is defined by the amount of space between the control points

휎 = (휎1, … , 휎푑)(with 푑 the image dimension), which can be different for each direction. B-splines have local support (|푁푥| is small), which means that the transformation of a point can be computed from only a couple of surrounding control points. This is beneficial both for modelling local transformations, and for fast computation. The parameters 휇 are formed by the B-spline coefficients 푝푘. The number of control points 푃 = (푃1, … , 푃푑) determines the number of parameters 푀, by 푀 = (푃1 × … × 푃푑) × 푑. 푃푖 in turn is determined by the image size 푠 and the B- spline grid spacing, i.e. 푃푖 ≈ si/휎푖 (where we use ≈ since some additional control points are placed just outside the image). For 3D images, 푀 ≈ 10000 parameters is not an unusual case, and 푀 can easily grow to 105 − 106. The parameter vector (for 푇 2D images) is composed as follows: 휇 = (푝1푥, 푝2푥, … , 푝푃1, 푝1푦, 푝2푦, … , 푝푃2) .

-52-

Chapter 3 – Materials and methods

3.3 Interpolator

During the optimization, some pixel values are evaluated at non-voxel positions, for which intensity interpolation is needed. Several methods for interpolation exist, varying in quality and speed [29]. In this work, 푁-th order B-Spline interpolator is used; 1st order B-Spline interpolator for each resolution level and 3rd order B-Spline interpolator for applying the final deformation.

3.4 Cost Function

Several choices for the (dis)similarity measure can be found in the literature. The choices for this work are described below.

3.4.1 Sum of Squared Difference (SSD)

Minimizing the intensity difference at corresponding points between two images is an intuitive method to register grayscale images. A simple and common cost function is the sum of squared difference (SSD) defined by

2 퐶푆푆퐷 = ∑[훪2(퐱) − 퐼1(퐡(퐱))] (3.5) 퐱∈훀 where I1 and I2 are the baseline and moving image intensity functions, respectively [6],[7]. Ω denotes the union of lung regions in target image and deformed template image. The underlying assumption of SSD is that the image intensity at corresponding points between two images should be similar. This is true when registering images of the same modality. However, considering the change in CT intensity as air inspired and expired during the respiratory cycle, the grayscale range is different within the lung region in two CT images acquired at different inflation levels. To balance this gray scale range difference, intensity normalization is needed.

-53-

Chapter 3 – Materials and methods

For example, a histogram matching procedure can be used before SSD registration to modify the histogram of template image so that it is similar to that of target image.

3.4.2 Normalized Cross Correlation (NCC)

Normalized cross correlation measures the pixelwise cross-correlation between image intensities normalized by the square root of the autocorrelation of each image. Mathematically, the negative normalized cross correlation measure is given by [6],[11]

∑퐱∈훀 퐼2(퐱) · 퐼1(퐡(퐱)) 퐶푁퐶퐶 = − (3.6) 2 2 √∑퐱∈훀 퐼2(퐱) ∙ ∑퐱∈훀 퐼1(퐡(퐱)) where the negative sign was added so that the optimal transformation 퐡 is found by minimization. When the factor of the intensity patterns from two images is a constant, the measure equals −1. Misalignments between the images will result in decrease of the normalized cross correlation, and thus, increase of the similarity cost

CNCC .

3.4.3 Mutual Information (MI)

Mutual information (MI) similarity cost function can accommodate intensity difference between two images and is therefore well-suited to accommodate the CT intensity change during inflation and deflation of the lung. Mutual information expresses the amount of information that one image contains about the other one. The negative mutual information cost of two images is defined as [6],[7],[9],[10]

푝(푖, 푗) 퐶MI = − ∑ ∑ 푝(푖, 푗) log (3.7) 푝퐼 °ℎ(푖)푝퐼 (푗) 푖 푗 1 2 where p(i,j) is the joint intensity distribution of transformed template image I1 ◦h and target image I2; pI1◦h(i) and pI2(j) are their marginal distributions, respectively.

The histogram bins of I1 ◦h and I2 are indexed by i and j. Misregistration results in a

-54-

Chapter 3 – Materials and methods

decrease in the mutual information, and thus, increases the similarity cost CMI. Note that the MI metric does not assume a linear relationship between the intensities of the two images.

3.4.4 Normalized Mutual Information (NMI)

For a discrete variable A, where pA(a) is the probability that A has value a, the Shannon entropy H is defined as [13]

퐻(퐴) = − ∑ 푝퐴(푎)푙표𝑔푝퐴(푎) (3.8) 푎

For two discrete random variables A and B the Shannon entropy of their joint distribution is defined as

퐻(퐴, 퐵) = − ∑ 푝퐴퐵(푎, 푏)푙표𝑔푝퐴퐵(푎, 푏) (3.9) 푎,푏

To counter the effect of increasing MI with decreasing registration quality we use normalized mutual information, NMI. NMI is a well-established registration quality measure which can be defined in terms of image entropies [12],[13]

퐻(퐼1) + 퐻(퐼2) 퐶MI 퐶 = = (3.10) NMI 퐻(퐼 , 퐼 ) 1 2 √퐻(퐼1)퐻(퐼2)

-55-

Chapter 3 – Materials and methods

3.5 Optimizer

Image registration can be formulated as an optimization problem:

흁̂ = arg min(퐶(흁, 퐼퐹, 퐼푀)) (3.11) 휇 where the cost function 퐶 represents the negated similarity measure that is optimized. 퐼퐹 and 퐼푀(휇) are the fixed and moving images, respectively, 흁 contains the external transform parameters that are applied to 퐼푀(휇) and 흁̂ represents the transform that align the images.

To solve the optimization problem, i.e. to obtain the optimal transformation parameter vector 흁̂, commonly an iterative optimization strategy is employed:

흁푘+1 = 흁푘 + 푎푘풅푘, 푘 = 0,1,2, … , (3.12)

with 풅푘 the “search direction” at iteration 푘, 푎푘 a scalar gain factor controlling the step size along the search direction [29]. The optimization process is illustrated in Figure 3.7.

Figure 3.7: Iterative optimization. Example for registration with a translation transformation model. The arrows indicate the steps 푎푘풅푘 taken in the direction of the optimum, which is the minimum of the cost function.

-56-

Chapter 3 – Materials and methods

3.5.1 Quasi-Newton LBFGS (QN L-BFGS)

QN methods [14], [15] are inspired by the well-known Newton–Raphson algorithm, which is given by

−1 흁푘+1 = 흁푘 − [퐻(흁푘)] 품(흁푘) (3.13)

where 퐻(흁푘) is the Hessian matrix of the cost function, evaluated at 흁푘 and

품(흁푘) = 휕퐶⁄휕휇 is the gradient of the cost function 퐶. The computation of the Hessian matrix and its inverse is computationally expensive, especially in high- dimensional optimization problems such as nonrigid registration. QN methods tackle this problem by using an approximation to the inverse of the Hessian: 퐿푘 ≈ −1 [퐻(흁푘)] . The approximation is updated in every iteration 푘 . Second-order derivatives of the cost function are not needed for this update; only the already computed first-order derivatives are used. Direct approximation of the inverse of the Hessian avoids the need for a matrix inversion. QN methods are typically implemented in combination with an inexact line search routine, determining a gain factor 푎푘 that ensures sufficient progress towards the solution. This results in the following QN algorithm:

흁푘+1 = 흁푘 − 푎푘퐿푘품(흁푘) (3.14)

Given certain conditions, many QN methods can be shown to be superlinearly convergent [14]

||흁푘+1 − 흁̂|| lim → 0 (3.15) 푘→∞ ||흁푘 − 흁̂||

Many ways to construct the series {퐿푘} are proposed in the literature [14], [15], most notably Symmetric-Rank-1 (SR1), Davidon-Fletcher-Powell (DFP), and Broyden-

-57-

Chapter 3 – Materials and methods

Fletcher-Goldfarb-Shanno (BFGS). Numerical experiments indicate that BFGS is very efficient in many applications [15]. It uses the following update rule for 퐿푘:

풔푦푇 풚풔푇 풔풔푇 퐿 = (퐼 − ) 퐿 (퐼 − ) + (3.16) 푘+1 풔푇풚 푘 풔푇풚 풔푇풚

where 퐼 is the identity matrix, 풔 = 흁푘+1 − 흁푘, and 풚 = 품푘+1 − 품푘. In our study, we use the limited memory BFGS (LBFGS) [16], which eliminates the need for storing the matrix 퐿푘 in memory. Following the implementation described in [16], we use the inexact line search routine described by Moré and Thuente [17]. It determines such that the so-called strong Wolfe conditions are satisfied

푇 퐶(흁푘+1) ≤ 퐶(흁푘) + 푐1푎푘풅푘 품(흁푘) (3.17)

푇 푇 |풅푘 품(흁푘+1) ≤ 푐2풅푘 품(흁푘) (3.18)

with user-defined scalars 푐1 and 푐2 satisfying 0 < 푐1 < 푐2 < 1 . Recall that 풅푘 represents the search direction of the optimization algorithm, which equals

−퐿푘품(흁푘) in the case of QN methods. The first Wolfe condition demands a sufficient decrease of the cost function value. The second Wolfe condition enforces reasonable progress towards a stationary point of the cost function, where the derivative vanishes. For optimization problems where the computational cost of evaluating the gradient 품푘 is high compared to the cost of computing 퐿푘, the values −4 푐1 = 10 and 푐2 = 0.9 are suggested in [18]. To realize superlinear convergence it is important to always try a gain factor 푎푘 = 1 first [15]. If this step size does not satisfy the strong Wolfe conditions, the iterative Moré–Thuente line search procedure is started to find a suitable gain. If no gain factor satisfying the strong Wolfe conditions can be found, the optimization is assumed to have converged.

-58-

Chapter 3 – Materials and methods

3.5.2 Simultaneous Perturbation (SP)

The simultaneous perturbation method (SP) is a stochastic gradient descent method [19] described by the following equation:

흁푘+1 = 흁푘 − 푎푘품̂푘 (3.19)

푎푙푝ℎ푎 where 푎푘 = 푎/(푘 + 퐴) . The constants 훼, 훢 and 푎푙푝ℎ푎 are user-defined and problem specific with 푎 > 0 , 퐴 ≥ 1 and 0 ≤ 푎푙푝ℎ푎 ≤ 1 . Convergence to the solution 흁̂ can only be guaranteed [20] if the bias of the approximation error goes to zero

E(품̂푘) → 품(흁푘), as 푘 → ∞ (3.20) where E(∙) denotes expectation. A stochastic gradient descent method is often applied when computation of the exact derivative is very costly. Using an approximation of the exact derivative could decrease the computation time per iteration, but may have negative effects on the speed of convergence.

The simultaneous perturbation method, first described by Spall [21], also bases its derivative estimate on approximate evaluations of the cost function. The SP method uses only two evaluations, independent of 푁

̃+ + 퐶푘 = 퐶(흁푘 + 푐푘횫푘) + 휀푘 (3.21) and

̃− − 퐶푘 = 퐶(흁푘 − 푐푘횫푘) + 휀푘 (3.22)

In these expressions, 횫푘 denotes the “random perturbation vector” of which each element is randomly assigned ±1 in each iteration, with equal probability. The approximation errors are represented by the 휀 terms. The 푖th element of the derivative vector 품̂푘 is then computed by

̃+ ̃− 퐶푘 − 퐶푘 [품̃푘]푖 = (3.23) 2푐푘[횫푘]푖

-59-

Chapter 3 – Materials and methods

The scalar 푐푘 is defined as

푐 푐 = (3.24) 푘 (푘 + 1)훾 where 푐 > 0 and 0 ≤ 훾 ≤ 1 are used-defined constants.

The simultaneous perturbation method has been used for rigid registration [22], but its performance has not been compared to other optimization methods so far.

3.6 Data complexity - Pyramids

It is common to start the registration process using images that have lower complexity, e.g., images that are smoothed and possibly downsampled. This increases the chance of successful registration. A series of images with increasing amount of smoothing is called a scale space. If the images are not only smoothed, but also downsampled, the data is not only less complex, but the amount of data is actually reduced. In that case, we talk about a “pyramid”. However, confusingly, the word pyramid is used by us also to refer to a scale space. Several scale spaces or pyramids are found in the literature, amongst others Gaussian and Laplacian pyramids, morphological scale space, and spline and wavelet pyramids [29]. The Gaussian pyramid is the most common one. In elastix we have:

Gaussian pyramid: Applies smoothing and down-sampling by a factor of 2 in all three dimensions.

Gaussian scale space: Applies smoothing and no down-sampling by a factor of 2 in all three dimensions.

Shrinking pyramid: Applies no smoothing, but only down-sampling by a factor of 2 in all three dimensions.

-60-

Chapter 3 – Materials and methods

In this study we used Gaussian and Shrinking 4-level resolution pyramids, performing rigid transformation for the first three levels (for computational time considerations) and non-rigid transformation for the 4th level with a maximum of 500 iterations in every resolution level. We also used a pyramid with only non-rigid transformation.

Figure 3.8: Multiresolution pyramid – Images for each resolution level

3.7 Evaluation

Good matching accuracy on the feature locations does not guarantee that the parenchymal tissues are correctly aligned. In order to reveal the lung tissue deformation pattern, the Jacobian determinant of the transformation field derived by image registration is used to estimate the local tissue deformation.

Given the Jacobian matrix of a voxel 휐 = (푥1, 푥2, 푥3), the determinant is:

-61-

Chapter 3 – Materials and methods

휕푇 (휐) 휕푇 (휐) 휕푇 (휐) 푥1 푥1 푥1 휕푥1 휕푥2 휕푥3 휕푇푥 (휐) 휕푇푥 (휐) 휕푇푥 (휐) det (퐽(푇(휐))) = det 2 2 2 (3.25) 휕푥 휕푥 휕푥 1 2 3 휕푇 (휐) 휕푇 (휐) 휕푇 (휐) 푥3 푥3 푥3 ([ 휕푥1 휕푥2 휕푥3 ])

⁄ where 휕푇푥푖(휐) 휕푥푗 is the partial derivative of the transform of voxel 휐 in the direction 푥푖 with respect to 푥푗 variable.

The Jacobian determinant (often simply called the Jacobian) [31], [32], [33] is a measurement to estimate the pointwise expansion and contraction during the deformation. The Jacobian determinant 퐽 at a given point gives important information about the behavior of the transformation near that point. Using a Lagrangian reference frame, a Jacobian value of one corresponds to zero expansion or contraction. Local tissue expansion corresponds to a Jacobian greater than one and local tissue contraction corresponds to a Jacobian less than one.

퐽 > 1, local expansion

퐽 > 0, preserve orientation { 퐽 = 1, no deformation 0 < 퐽 < 1, local contraction (3.26)

퐽 = 0, non − injective { 퐽 < 0, reverse orientation

Registration schemes whose determinants of Jacobian matrices include negative values are excluded for subsequent analysis, since such areas correspond to singularities deformation fields (foldings) [30] (see also Fig. 3.9).

-62-

Chapter 3 – Materials and methods

Figure 3.9: Example of a rejected registration scheme. The registered lung fused with the deformation field, which includes negative values.

In order to estimate registration distance error, 2 sets of landmark points were generated, corresponding to 2 different types of tissue in the lung fields: NLP and ILD affected tissue. In order to identify normal parenchyma lung tissue, an ILD region segmentation algorithm [14] was applied to create binary masks corresponding to ILD affected and normal parenchyma regions. An example of landmark points generated in NLP regions and ILD affected regions is presented in Fig. 3.10.

Figure 3.10: (a) Lung fields affected by interstitial disease. Gray color indicates the normal lung parenchyma, while green color indicates the interstitial lung disease affected regions. (b) Landmark points in normal lung parenchyma. (c) Landmark points in interstitial lung disease affected regions

-63-

Chapter 3 – Materials and methods

Figure 3.11: An example of a landmark point for an image pair. The top row shows the landmark identified in the fixed image in the sagittal, coronal and axial directions. The second row shows the point selected in the registered image.

The Euclidean distance error between the original landmark coordinates

(푥_ILD1푖 , 푥_ILD2푖 , 푥_ILD3푖), (푥_NLP1푖 , 푥_NLP2푖 , 푥_NLP3푖) and the coordinates

(푥_ILD1푖′ , 푥_ILD2푖′ , 푥_ILD3푖′), (푥_NLP1푖′ , 푥_NLP2푖′ , 푥_NLP3푖′) of the registered

points was calculated by the following equations:

푀 1 푑̅̅̅_̅ILD̅̅̅̅ = ∙ ∑ √(푥_ILD − 푥_ILD ′)2 + (푥_ILD − 푥_ILD ′)2 + (푥_ILD − 푥_ILD ′)2 푀 1푖 1푖 2푖 2푖 3푖 3푖 푖=1

(3.27)

1 푑̅̅̅_̅NPL̅̅̅̅̅ = ∙ 푀

∑푀 2 2 2 푖=1 √(푥_NPL1푖 − 푥_NPL1푖′) + (푥_NPL2푖 − 푥_NPL2푖′) + (푥_NPL3푖 − 푥_NPL3푖′)

(3.28)

3.8 Summary

In this chapter the materials and methods of this study were presented extensively. The next chapter presents and discusses the results of the registration schemes generated using these methods.

-64-

Chapter 4 – Results and Discussion

4. Results and Discussion 4.1 Results

Figure 4.1: Segmented lung fields of the 5 patients baseline scans

-65-

Chapter 4 – Results and Discussion

(a)

(b) (c)

(d) (e)

(f) (g)

Figure 4.2: Registration scheme performance example: (a) baseline image, (b) unregistered warped image, (c) difference image of (a) and (b), (d) optimal registered image (QN), (e) difference image of (a) and (d), (f) suboptimal registered image (SP), and (g) difference image of (a) and (f).

-66-

Chapter 4 – Results and Discussion

As mentioned in chapter 3, registration schemes whose determinants of Jacobian matrices include negative values are excluded for subsequent analysis, since such areas correspond to singularities deformation fields (foldings). 16 out of 32 registration schemes included negative value of the determinant of the Jacobian Matrices and excluded from subsequent analysis.

Tables 4.1 (a,b,c,d,e) depicts the results obtained by the above evaluation method from the 16 remained schemes of each optimizer for normal parenchyma and ILD regions. The distance error is represented in mm.

-67-

Chapter 4 – Results and Discussion

Registration Patient 1 Patient 2 Patient 3 Patient 4 Patient 5 Average scheme code par 17 0.3536 0.3486 0.3303 0.3256 0.3211 0.3358 par 18 0.3427 0.3354 0.3310 0.3226 0.3169 0.3297 par 19 0.3461 0.3410 0.3304 0.3176 0.2991 0.3268 par 20 0.3421 0.3319 0.3274 0.3186 0.3124 0.3264 par 21 0.3692 0.3516 0.3431 0.3316 0.3176 0.3426 par 22 0.3535 0.3395 0.3305 0.3286 0.3258 0.3355 par 23 0.3705 0.3517 0.3338 0.3246 0.3144 0.3390 par 24 0.3518 0.3380 0.3349 0.3226 0.3226 0.3339 par 25 0.3385 0.3351 0.3260 0.3236 0.3194 0.3285 par 26 0.3715 0.3529 0.3345 0.3216 0.3175 0.3396 par 27 0.3620 0.3449 0.3335 0.3176 0.3102 0.3336 par 28 0.3443 0.3262 0.3167 0.3166 0.3070 0.3221 par 29 0.3609 0.3457 0.3359 0.3226 0.3201 0.3370 par 30 0.3468 0.3358 0.3314 0.3226 0.3093 0.3291 par 31 0.3596 0.3425 0.3363 0.3176 0.3111 0.3334 par 32 0.3397 0.3231 0.3210 0.3146 0.3117 0.3220

Table 4.1a: Distance error (in mm) using Quasi-Newton LBFGS optimizer for normal parenchyma areas

Registration Patient 1 Patient 2 Patient 3 Patient 4 Patient 5 Average scheme code par 17 0.3513 0.3450 0.3240 0.3205 0.3159 0.3313 par 18 0.3346 0.3310 0.3277 0.3159 0.3139 0.3246 par 19 0.3413 0.3385 0.3298 0.3154 0.2971 0.3244 par 20 0.3340 0.3311 0.3231 0.3156 0.3075 0.3222 par 21 0.3610 0.3464 0.3386 0.3233 0.3170 0.3372 par 22 0.3480 0.3404 0.3264 0.3199 0.3220 0.3313 par 23 0.3713 0.3514 0.3305 0.3238 0.3101 0.3374 par 24 0.3477 0.3296 0.3329 0.3201 0.3160 0.3292 par 25 0.3329 0.3350 0.3200 0.3218 0.3120 0.3243 par 26 0.3660 0.3482 0.3333 0.3195 0.3161 0.3366 par 27 0.3586 0.3435 0.3308 0.3135 0.3052 0.3303 par 28 0.3425 0.3229 0.3140 0.3155 0.2987 0.3187 par 29 0.3568 0.3399 0.3342 0.3158 0.3112 0.3315 par 30 0.3451 0.3347 0.3258 0.3178 0.3018 0.3250 par 31 0.3532 0.3410 0.3305 0.3141 0.3105 0.3298 par 32 0.3385 0.3184 0.3177 0.3086 0.3063 0.3179

Table 4.1b: Distance error (in mm) using Quasi-Newton LBFGS optimizer for ILD regions

-68-

Chapter 4 – Results and Discussion

Registration Patient 1 Patient 2 Patient 3 Patient 4 Patient 5 Average scheme code par 17 0.4756 0.5576 0.5213 0.4706 0.5131 0.5076 par 18 0.5757 0.6034 0.4690 0.3986 0.5899 0.5273 par 19 0.3801 0.3420 0.5204 0.4166 0.3401 0.3998 par 20 0.6401 0.3709 0.4954 0.5366 0.4594 0.5004 par 21 0.5572 0.4926 0.4771 0.4386 0.5156 0.4962 par 22 0.3965 0.4485 0.3785 0.4706 0.6198 0.4627 par 23 0.5745 0.5937 0.4718 0.5586 0.5634 0.5524 par 24 0.4888 0.4470 0.4579 0.5616 0.3956 0.4701 par 25 0.5145 0.6261 0.5120 0.4226 0.3714 0.4893 par 26 0.6645 0.6489 0.6345 0.5196 0.4435 0.5822 par 27 0.3630 0.4439 0.4645 0.6176 0.4962 0.4770 par 28 0.6163 0.5642 0.3927 0.5276 0.5000 0.5201 par 29 0.5119 0.4977 0.5109 0.4146 0.3731 0.4616 par 30 0.4838 0.4818 0.4354 0.5576 0.3503 0.4617 par 31 0.4926 0.5765 0.6163 0.4846 0.5531 0.5446 par 32 0.5337 0.3741 0.4930 0.3216 0.5187 0.4482

Table 4.1c: Distance error (in mm) using Simultaneous Perturbation optimizer for normal parenchyma areas

Registration Patient 1 Patient 2 Patient 3 Patient 4 Patient 5 Average scheme code par 17 0.4073 0.5600 0.5280 0.6105 0.3219 0.4855 par 18 0.4186 0.4180 0.4957 0.4269 0.4079 0.4334 par 19 0.4813 0.4875 0.3848 0.5584 0.4601 0.4744 par 20 0.3670 0.5441 0.4121 0.3426 0.4705 0.4272 par 21 0.6230 0.6404 0.6106 0.3333 0.5940 0.5602 par 22 0.6450 0.4004 0.5724 0.5039 0.3890 0.5021 par 23 0.5113 0.5864 0.5045 0.4738 0.5621 0.5276 par 24 0.4017 0.5206 0.5989 0.5741 0.3680 0.4926 par 25 0.3399 0.5990 0.3260 0.4068 0.4870 0.4317 par 26 0.5600 0.5202 0.6023 0.3925 0.5351 0.5220 par 27 0.3906 0.6055 0.5048 0.3955 0.3832 0.4559 par 28 0.5135 0.4319 0.3190 0.4155 0.4897 0.4339 par 29 0.5868 0.3659 0.3802 0.4468 0.5972 0.4753 par 30 0.6191 0.3727 0.5468 0.4668 0.3318 0.4674 par 31 0.3882 0.6230 0.5055 0.4751 0.5145 0.5012 par 32 0.3425 0.3674 0.4457 0.4476 0.6033 0.4413

Table 4.1d: Distance error (in mm) using Simultaneous Perturbation optimizer for ILD regions

-69-

Chapter 4 – Results and Discussion

Quasi-Newton Simultaneous Perturbation Registration Average Average scheme code Average ILD Average ILD Normal Normal par 17 0.34 0.33 0.51 0.49 par 18 0.33 0.32 0.53 0.43 par 19 0.33 0.32 0.40 0.47 par 20 0.33 0.32 0.50 0.43 par 21 0.34 0.34 0.50 0.56 par 22 0.34 0.33 0.46 0.50 par 23 0.34 0.34 0.55 0.53 par 24 0.33 0.33 0.47 0.49 par 25 0.33 0.32 0.49 0.43 par 26 0.34 0.34 0.58 0.52 par 27 0.33 0.33 0.48 0.46 par 28 0.32 0.32 0.52 0.43 par 29 0.34 0.33 0.46 0.48 par 30 0.33 0.33 0.46 0.47 par 31 0.33 0.33 0.54 0.50 par 32 0.32 0.32 0.45 0.44

Table 4.1e: Average distance error (in mm)

Table 4.2: Illustration of the results

-70-

Chapter 4 – Results and Discussion

4.2 Results and Discussion

In this study, a set of multiresolution combinations of rigid and nonrigid registration schemes are evaluated in the framework of ILD CT follow-up analysis. Although nonrigid registration schemes seem the natural choice for lung analysis, due to the elastic nature of lung tissue, combination schemes that comprised of rigid and nonrigid transforms are recently proposed, considering the trade-off between sufficient accuracy and time efficiency in the clinical environment [25]. The registration schemes evaluated in the present study were initially selected on the basis of artificially warped ground truth follow-up data, generated from original baseline images of 5 clinical volumetric CT scans, in order to avoid intrinsic clinical follow-up data variability. Subsequently, the performance of selected registration schemes was validated utilizing the clinical follow-up counterparts of the original baseline images.

These schemes obtained submillimeter registration accuracies in terms of average distance errors; 0.33 ± 0.01 mm for NLP and 0.33 ± 0.01 mm for ILD affected regions in the case of Quasi-Newton optimizer, 0.49 ± 0.09 mm for NPL and 0.48 ± 0.05 mm for ILD affected regions in the case of Simultaneous Perturbation optimizer.

Best performance was achieved by the registration scheme using: Gaussian smoothing pyramid, Affine Transform for the first 3 resolution levels and the 3rd order B-Spline for the 4th resolution level, the Mutual Information Cost Function and the Quasi-Newton Optimizer. Non-parametric Wilcoxon signed-rank test indicates that there was statistical significant better performance in the case of Quasi-Newton compared to performances achieved using the Simultaneous Perturbation optimizer.

Although the approach of artificially warped data has been recently adopted by several studies reporting on evaluation of registration algorithm performance [26], [27], [28], studying registration accuracy with artificially generated follow-up data may not simulate faithfully the actual lung field deformations, which in case of ILD affected lung field data analyzed by the present study, is further complicated by disease progression/regression.

-71-

Chapter 4 – Results and Discussion

Future efforts should include different tuning of the optimizer parameters, as well as regularization terms which prevent folding areas, in order to achieve registration accuracies compared to 0.2 mm [15],[24]. In addition, future steps should include constraints on full nonrigid schemes, that will not allow irregular deformations, even though this study showed that fully nonrigid schemes do not show statistically significantly better performance compared to combined rigid/nonrigid schemes, although having significant higher computational cost. Furthermore, an important step would be the implementation of the Quasi-Newton based registration schemes on clinical follow-up data. The fact that the schemes using Quasi-Newton optimizer in this study didn’t achieve comparable registration accuracies to the study of Gradient Descent optimizer of Vlachopoulos et. al [24] could be due to the selection of the optimizer parameters and the lack of regularization terms.

Taking into account that registration is a fundamental subsystem of an integrated quantification and follow-up system, future efforts should also focus on integrate the quantification and registration system in order to create an integrated monitoring system of the ILD.

-72-

References

[1] P. Korfiatis, “Image analysis methods for diagnosis of diffuse lung disease in multi-detector computed tomography”, PhD Thesis, University of Patras, 2010.

[2] M. Zompatori, N. Sverzellati, V. Poletti, C. Bnà, F. Ormitti, E. Spaggiari, and E. Maffei, “High-resolution ct in diagnosis of diffuse infiltrative lung disease,” Semin Ultrasound CT MR, vol. 26, pp. 332–47, Oct 2005.

[3] C. Beigelman-Aubry, C. Hill, A. Guibal, J. Savatovsky, and P. A. Grenier, “Multi-detector row ct and postprocessing techniques in the assessment of diffuse lung disease,” Radiographics, vol. 25, no. 6, pp. 1639–52, 2005.

[4] C. D. Collins, A. U. Wells, D. M. Hansell, R. A. Morgan, J. E. MacSweeney, R. M. du Bois, and M. B. Rubens, “Observer variation in pattern type and extent of disease in fibrosing alveolitis on thin section computed tomography and chest radiography,” Clin Radiol, vol. 49, pp. 236–40, Apr 1994.

[5] C. S. White, R. Pugatch, T. Koonce, S.W. Rust, and E. Dharaiya, “Lung nodule cad software as a second reader: a multicenter study,” Acad Radiol, vol. 15, pp. 326–33, Mar 2008.

[6] Kunlin Cao, Kai Ding, Joseph M. Reinhardt, and Gary E. Christensen, “Improving Intensity-Based Lung CT Registration Accuracy Utilizing Vascular Information”, Research Article, International Journal of Biomedical Imaging, Volume 2012.

[7] Kunlin Cao, “Mechanical analysis of lung CT images using nonrigid registration”, PhD diss., University of Iowa, 2012.

[8] A. Sotiras, Ch. Davatzikos and N. Paragios, “Deformable Medical Image Registration: A Survey”, IEEE transactions on medical imaging, vol. 32, no. 7, 2013.

[9] D. Mattes, D. R. Haynor, H. Vesselle, T. K. Lewellen, and W. Eubank, “PET-CT image registration in the chest using freeform deformations,” IEEE Transactions on Medical Imaging, vol. 22, no. 1, pp. 120–128, 2003.

[10] P. Thévenaz and M. Unser, “Spline pyramids for inter-modal image registration using mutual information,” in Wavelet Applications in Signal and Image Processing V, vol. 3169 of Proceedings of SPIE, pp. 236–247, July 1997.

[11] T. S. Yoo, Insight Into Images: Principles and Practice for Segmentation, Registration, and Image Analysis, AK Peters, 2004.

[12] Yokoi T., Soma T., Shinohara H., Matsuda H., “Accuracy and reproducibility of co-registration techniques based on mutual information and normalized mutual information for MRI and SPECT brain images”, Annals of Nuclear Medicine Vol. 18, No. 8,659-667, 2004.

-73-

[13] Z.F. Knops, J.B.A. Maintz, M.A. Viergever, J.P.W. Pluim, “Normalized mutual information based registration using k-means clustering and shading correction”, Medical Image Analysis 10 (2006) 432–439.

[14] J. E. Dennis, Jr. and J. J. Moré, “Quasi-Newton methods, motivation and theory,” SIAM Rev., vol. 19, no. 1, pp. 46–89, 1977.

[15] J. Nocedal and S. J. Wright, Numerical Optimization. New York: Springer-Verlag, 1999.

[16] J. Nocedal, “Updating quasi-Newton matrices with limited storage,” Math. Comput., vol. 35, no. 151, pp. 773–782, 1980.

[17] J. J. Moré and D. J. Thuente, “Line search algorithms with guaranteed sufficient decrease,” ACM Trans. Math. Software, vol. 20, no. 3, pp. 286–307, 1994.

[18] F. A. Potra and Y. Shi, “Efficient line search algorithm for unconstrained optimization,” J. Optim. Theor. Appl., vol. 85, no. 3, pp. 677–704, 1995.

[19] H. J. Kushner and G. G. Yin, Stochastic Approximation and Recursive Algorithms and Applications, 2nd ed. New York: Springer-Verlag, 2003.

[20] H. J. Kushner and D. S. Clark, Stochastic Approximation Methods for Constrained and Unconstrained Systems. New York: Springer-Verlag, 1978.

[21] J. C. Spall, “Multivariate stochastic approximation using a simultaneous perturbation gradient approximation,” IEEE Trans. Autom. Control, vol. 37, no. 3, pp. 332–341, 1992.

[22] A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1495–1511, Dec. 2003.

[23] Rueckert et al., “Nonrigid Registration Using Free-Form Deformations: Application to Breast MR Images”, IEEE Trans. on Medical Imaging, vol. 18, no. 8, August 1999

[24] Vlachopoulos et al.: “Registration schemes in ILD CT follow-up analysis”, Medical Physics, Vol. 42, No. 8, August 2015

[25] S. Gu, D. Wilson, J. Tan, and J. Pu, “Pulmonary nodule registration: Rigid or nonrigid?”, Med. Phys. 38(7), 4406–4414 (2011)

[26] K. Murphy et. al, “Evaluation of registration methods on thoracic CT: The EMPIRE10 challenge”, IEEE Trans. Med. Imaging 30(11), 1901–1920 (2011)

-74-

[27] J. Salvi, C. Matabosch, D. Fofi, and J. Forest, “A review of recent range image registration methods with accuracy evaluation”, Image Vision Comput. 25(5), 578–596 (2007)

[28] H. Zhong, J. Kim, and I. J. Chetty, “Analysis of deformable image registration accuracy using computational modeling,” Med. Phys. 37(3), 970–979 (2010)

[29] Klein S, Staring M, Murphy K, Viergever MA, Pluim JP., “elastix: a toolbox for intensity- based medical image registration”, IEEE Trans Med Imaging. 2010

[30] G. E. Christensen and H. J. Johnson, “Consistent image registration”, IEEE Trans. Med. Imaging 20(7), 568–582 (2001)

[31] L. Fan, Chang Wen Chen, J. M. Reinhardt, and E. A. Hoffman, “Evaluation and application of 3D lung warping and registration model using HRCT images,” in Medical Imaging 2001: Physiology and Function from Multidimensional Images, Proceedings of SPIE, pp. 17–22, February 2001.

[32] G. E. Christensen, R. D. Rabbitt, M. I. Miller et al., “Topological properties of smooth anatomic maps,” in Information Processing in Medical Imaging, Y. Bizais, C. Braillot, and R. D. Paola, Eds., vol. 3, pp. 101–112, 1995.

[33] R. C. Buck, Advanced Calculus, McGraw-Hill Book Company, St. Louis, Miss, USA, 3rd edition, 1978.

[34] P. Korfiatis, C. Kalogeropoulou, A. Karahaliou, A. Kazantzi, S. Skiadopoulos, and L.Costaridou, “Texture classification-based segmentation of lung affected by interstitial pneumonia in high-resolution CT”, Med. Phys. 35(12), 5290–5302 (2008).

[35] K. Murphy, B. van Ginneken, S. Klein, M. Staring, B. J. de Hoop, M. A. Viergever, and J. P. W. Pluim, “Semi-automatic construction of reference standards for evaluation of image registration”, Med. Image Anal. 15(1), 71–84 (2011).

[36] M. M. Coselmon, J. M. Balter, D. L. McShan, and M. L. Kessler, “Mutual information based CT registration of the lung at exhale and inhale breathing states using thin-plate splines”, Med. Phys. 31(11), 2942–2948 (2004).

-75-