Computed Tomography from Imagery Generated by Fluoroscopy Along an Arbitrary Path

COMPUTED TOMOGRAPHY FROM IMAGERY GENERATED BY FLUOROSCOPY ALONG AN ARBITRARY PATH

Christopher L. Baker

A thesis submitted to the Faculty and Board of Trustees of the Colorado School of Mines in partial fulfillment of the requirements for the degree of Master of Science (Engineering Systems)

Golden Colorado Date ______

Signed: ______Christopher L. Baker

Approved:______Dr. Christian Debrunner Thesis Advisor

Approved: ______Dr. Mohamed Mahfouz Thesis Advisor Golden Colorado Date ______

______Dr. David Munoz Professor and Head Department of Engineering

ABSTRACT

An accurate geometrical three-dimensional (3D) model of human bone is required in many medical procedures including computer assisted surgery. Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are commonly used to obtain these models by reconstructing a 3D dataset from image projections. However, these methods are expensive and time consuming, and the instruments are not available in most operating rooms. An alternative to CT and MRI that is available in most operating rooms is fluoroscopy. If the viewing geometry is known, fluoroscopic imagery collected over a range of viewpoints can be processed using cone-beam tomographic reconstruction methods to produce a 3D volumetric image similar to that obtained from CT. Although there are fluoroscopy machines available that will process the projections in this manner, they must be carefully instrumented and constrained to gather images along a specific pre-defined path.

We propose an alternative based on tomographic processing of fluoroscopic imagery collected along an arbitrary path, which will allow use of most of the fluoroscopy machines that are currently available. We use metallic markers (beads) attached to the object being scanned and 3D model matching techniques to recover the actual path of the source/detector from the image data. Given this path, we can use cone-beam tomographic reconstruction methods to reconstruct the 3D volume. With our method, careful control and instrumentation of the trajectory is not necessary, and the required motion can be obtained by moving the fluoroscopic sensor or the patient. This capability would provide 3D volumetric imaging capabilities to most facilities with fluoroscopes.

We have performed various experiments to develop and demonstrate our approach. We first generated a set of synthetic phantom data to test the reconstruction algorithm along multiple paths. Next we used a fluoroscope to collect imagery of a physical phantom. We generated two sets of physical phantom data, the first of which we used to verify our pose estimation method. The next set was used to reconstruct the physical phantom from imagery

iv collected along an arbitrary path. Finally we imaged a cadaver along an arbitrary path and reconstructed the 3D volume.

v TABLE OF CONTENTS

Abstract...... iv

Table of Contents...... vi

List of Figures...... x

List of Tables...... xvi

Acknowledgments ...... xviii

Dedication...... xx

Chapter 1...... xx

Chapter 2...... 7

2.1 Pose Estimation ...... 7

2.2 Cone-Beam Tomographic Reconstruction...... 8

2.2.1 Feldkamp’s method ...... 9

2.2.2 Grangeat’s method...... 11

2.2.3 Defrise and Clack method...... 12

2.2.4 LINCON method...... 13

2.3 Reconstruction Algorithm Speed Increase ...... 15

Chapter 3...... 17

3.1 Data Collection ...... 19

3.2 Sensor Calibration...... 19

vi 3.2.1 Geometric distortion...... 20

3.2.2 Radiometric distortion...... 21

3.3 Pose Estimation...... 23

3.3.1 Image feature matching to find the pose ...... 23

3.3.1.1 Finding the beads ...... 23

3.3.1.2 Marker tracking correspondence...... 24

3.3.1.3 Pose computation with least squares iterative technique...... 26

3.4 Image Correction...... 27

3.4.1 Finding the optimal origin ...... 27

3.4.2 Re-orientation of images ...... 31

3.5 Tomographic Reconstruction ...... 34

3.5.1 Reconstruction algorithm geometry...... 34

3.5.2 Reconstruction algorithm...... 36

3.5.2.1 Grangeat’s development ...... 36

3.5.2.2 First derivative of the radon transform from the x-ray projections ...... 40

3.5.2.3 Resample to spherical coordinates...... 41

3.5.2.3.1 Backward mapping scheme with the source searching method ...... 42

3.5.2.3.2 Forward mapping scheme with the volume minimization method...... 46

3.5.2.4 Radon inversion by consecutive backprojections...... 49

3.6 Summary ...... 52

Chapter 4...... 53

vii 4.1 Experimental Setup...... 53

4.1.1 Synthetic images of four cylinders...... 54

4.1.2 Synthetic images of ellipsoids ...... 55

4.1.3 Physical phantom description...... 56

4.1.4 Cadaver image...... 57

4.1.5 Calibration accuracy experiments ...... 59

4.1.5.1 Geometric calibration...... 59

4.1.5.2 Radiometric calibration ...... 60

4.1.6 Pose estimation accuracy experiments...... 62

4.1.6.1 Ground truth from a rotational stage...... 63

4.1.6.2 Ground truth from the Optotrak...... 65

4.2 Tomographic Reconstruction Results ...... 70

4.2.1 Synthetic phantom volume reconstruction...... 71

4.2.1.1 Cylinder data...... 71

4.2.1.2 Ellipsoid data...... 72

4.2.2 Physical phantom reconstruction...... 75

4.2.3 Cadaver reconstruction...... 78

4.2.4 Synthetic data error analysis...... 78

4.2.4.1 Density analysis...... 79

4.2.4.2 Geometric accuracy analysis...... 82

4.3 Summary of Results...... 83

viii Chapter 5...... 85

Chapter 6...... 87

6.1 Epipolar Geometry Included to Remove the Beads ...... 87

6.2 Reconstructions from Missing Data...... 87

6.2.1 Metal artifact reduction...... 88

6.2.2 Incomplete datasets...... 88

6.3 Reconstruction of Other Data Sets...... 89

6.4 Other Methods of Reconstruction...... 89

6.5 Reconstruction on Other Fluoroscopy Machines...... 89

6.6 Move All Code to C/C++ and Package...... 90

6.7 Algorithm Speed-up (Hardware and Software) ...... 90

6.8 Noise Analysis...... 90

6.9 Resolution Improvements ...... 91

6.9.1 Resolution of the fluoroscope...... 91

6.9.2 Virtual detector plane...... 91

6.9.3 Radon resolution...... 92

6.9.4 Output array resolution ...... 92

References...... 93

Appendix A ...... 97

ix LIST OF FIGURES

Figure 1 On the left is a 3D fluoroscope sold by Siemens, called the SIREMOBIL Iso-C 3D. The right shows the General Electric version. [1]...... 2

Figure 2 Display of an elbow fracture with fissure extending into the joint in three MPR planes (left) and Hand in SSD (Surface-Shaded Display) from three different views (right) [1]...... 2

Figure 3 A Pictorial Overview of the Entire Reconstruction Process...... 18

Figure 4 Fluoroscopic image of geometric calibration grid, before distortion removal (left) and after (right). Note the “pincushion” effect visible in the left image...... 21

Figure 5 Calibration spots can be seen on the left and right. These spots should maintain constant intensity (if there is no auto gain adjustment or change in x-ray power) and position throughout the data collection since they are attached to the fluoroscopy machine. If we scale and offset the raw image data to make these calibration spot intensities constant from frame to frame we can remove the effect of auto-gain and small changes in the x-ray power...... 22

Figure 6 Beads found in fluoroscope image displayed as connected components (left) and projected pose for all beads in the bead model (right)...... 24

Figure 7 Interpretation search tree. Each level of the tree is searched to find a possible set of correspondences. When enough hypothesized correspondences have been found, the pose is calculated and tested. If it is found to be an incorrect correspondence, the tree is searched further...... 25

Figure 8 The five coordinate systems (CAS , CIS , WAS , WIS , VPS ) which govern the re-orientation of the images...... 29

Figure 9 Example of 3 different source positions and associated parameters used in finding the optimal WIS origin...... 30

Figure 10 Transformation from VPS to WIS ...... 32

Figure 11 The scaling of the real detector plane to the virtual detector plane. Notice that the virtual detector plane passes through the VPS origin, which is coincident with the WIS origin. Also notice that the uˆ -axis is in the ijˆˆ-plane...... 33

x Figure 12 Typical radon plane shown in VPS , where nˆ is the unit normal to this plane

and Pi is the camera center...... 34

Figure 13 Illustration demonstrating the definition of ψ h and ψ v as well as the relationship nˆ has with the two coordinate system, WIS (iˆˆ,,j kˆ) and VPS (uˆˆ,,v wˆ)...... 35

Figure 14 Radon plane parameterization ...... 37

Figure 15 Grangeat’s cone-beam geometry. The description of the cone-beam geometry

is shown in (a). The plane formed by the line CpQ and the line SC p is shown in (b)...... 38

Figure 16 Illustration of a typical discrete path and the closest intersections...... 44

Figure 17 Geometry relating A and ρ ...... 47

Figure 18 Description of the first backprojection process. All the values that would contribute to the point B are added in and found based the fact that a line through B generated their value in the radon space...... 50

Figure 19 The horizontal backprojection is basically the same as the vertical except now we perform for all the horizontal planes...... 51

Figure 20 Illustration of the partial volume effect. If the plane shown is at the top edge of the image, then the shaded region of the cylinder will not be imaged from this source position. However, if the cylinder is imaged from the right side, the shaded region will be imaged. The partial volume effect arises from the fact that parts of the volume are imaged in some views and not in others, while other parts of the volume (the center for example) are imaged in all views...... 55

Figure 21 External (left) and Internal (right) picture of the physical phantom. Notice the beads placed along the grid on the outside in the left figure and along the inside on the right...... 57

Figure 22 Path of the sensor relative to the cadaver shown as a 3D plot...... 58

Figure 23 Fluoroscopic image of a calibration grid before distortion removal (a) and after (b). Note the rotation and “pincushion” effects visible in (a)...... 59

xi Figure 24 Fit relating pixel intensity value to absorption (t) ...... 60

Figure 25 Auto gain correction. The dashed lines are before the correction, and the dotted lines are after. The two lower plots are the control points, and the upper plot is another constant density within the image...... 62

Figure 26 Illustration of the two components of the RADIUS error measure. The first is the distance of the CAS origin from the plane of the best fitting circle, and the second is the deviation from the circle in a direction perpendicular to the rotation axis...... 64

Figure 27 Top view of Pose estimation verification setup. The Optotrak and fluoroscope reference frames do not move during the data collection. The probe (OPS ) is attached to the outside of the PVC pipe whose origin (WAS ) is defined somewhere (arbitrarily) in the center of all the beads...... 65

Figure 28 The two paths used to verify the pose finding algorithm. Path 1 (bottom) takes

us from the OPS0 through OBS to OPSi where the comparison is made. The second path (top) takes us from the same starting reference coordinate system,

OPS0 , through WAS0 to CAS back through WASi and into OPSi where the comparison is made...... 66

Figure 29 The Euclidean distance between the probe origin found from the pose finding algorithm to the origin found using the Optotrak method...... 69

Figure 30 The error in the rotation angle between two consecutive image frames for 67 frames in the sequence. This error is calculated as the rotational error from the angle-axis representation of the residuals from the two pose calculation paths...... 69

Figure 31 Sample pose projected for specific correspondences. Left: Original image, Right: Bead model projected onto the image for the estimated pose...... 70

Figure 32 Cross sectional views of synthetic phantom for multiple paths, The top row is the original data, the second row is reconstructed from the half circle path, the third row from the full circle path, the fourth from the circle plus arc path, and the fifth from the partial volume path. Column (a) is slice 35, (b) is 70, (c) is 105, and (d) is 140...... 73

Figure 33 Since the VolumePro output looks the same for all of the given paths, we only show the full path reconstruction (a) alongside the partial volume (b). These

xii reconstructions are rendered voxel data in Amira™ using the VolumePro hardware with an appropriate alpha scale...... 74

Figure 34 Synthetic phantom data reconstruction of ellipsoids along 3 different paths with the original data to compare. (a) shows the original data. (b) is reconstructed with only 180 degrees. (c) is reconstructed with 360 degrees. Finally, (d) is reconstructed with full path of a circle plus arc. The slice shown here is slice 70 out of 210...... 74

Figure 35 Images taken of the reconstructed synthetic phantom from VolumePro. There are two different views of the same reconstructed volume with the original volume displayed next to it. The original volumes are shown on the left while the reconstructed volumes are shown on the right...... 75

Figure 36 Data collection path of the physical phantom generated by the arbitrary motion...... 76

Figure 37 Three slices from the physical phantom arbitrary path reconstruction. (a) is slice number 53, (b) is slice number 105, and (c) is slice number 158 our of 211 slices...... 77

Figure 38 Reconstruction of the physical phantom displayed here as a volume rendering from Amira™ with an appropriate alpha scale...... 78

Figure 39 The reconstructed volume from projections generated along an arbitrary path using our proposed algorithm...... 79

Figure 40 Plot of the linear transform from the original density values to the reconstructed values. The error bars represent the standard deviation found in the segmented reconstruction volume for a particular ellipsoid...... 80

Figure 41 Slice 48 out of the reconstructed physical phantom from an arbitrary path is shown in (a). The acrylic rods segmented are shown in (b). The PCV wall segmented is shown in (c). The air was taken to be everything else...... 81

Figure 42 Pictorial description of radon transform ...... 97

Figure 43 Image with non-zero pixels as specified in Table 6...... 99

Figure 44 Radon transform of Figure 43 with the radon lines labeled...... 100

Figure 45 MATLAB®’s phantom image...... 101

Figure 46 Reconstruction of phantom using only 4 different projection angles...... 102

xiii Figure 47 Reconstruction of the phantom using 10 degree angle steps...... 102

Figure 48 Phantom reconstructed using 1.5 degree spacing ...... 103

Figure 49 Description of the fan beam resampling process. Notice that not only does Source 2 have a perpendicular ray to its own sensor plane, but Source one’s data will also contain a perpendicular ray to Source 2’s sensor plane. The data from Source 1 can be resampled to Source 2 in order to perform parallel backprojection...... 104

Figure 50 The integration plane is defined by the distance from the origin (ρ ) , and the orientation of a unit vector, nˆ , parameterized by θ and ϕ ...... 107

Figure 51 Redundant data in a circular path around an object created by two source centers lying on the same plane...... 113

Figure 52 For the data sufficiency criteria, every plane that passes through the object must intersect the source path. This is an example where that is not the case...... 114

Figure 53 To cover the missing radon plane from Figure 52 a new arc has been introduced whose path will intersect the missing plane. With this configuration (the circle plus arc data acquisition) all planes that pass through the object will intersect a source path, therefore the data sufficiency criteria is fulfilled [11]...... 114

xiv

LIST OF TABLES

Table 1 Al Thicknesses Used...... 22

Table 2 List of ellipsoid parameters ...... 56

Table 3 Pose verification Results...... 63

Table 4 Mean and standard deviations of the segmented reconstructed physical phantom...... 81

Table 5 The differences found in the reconstructed ellipsoids vs. the original data. All data given in this table is in pixels...... 82

Table 6 Image points...... 99

xvi

ACKNOWLEDGMENTS

First I would like to thank my advisor Dr. Christian Debrunner for his support, confidence and guidance over the past two years. The experience and help he has provided me in this and other projects is greatly appreciated. Without his help I would have been lost.

I would also like to thank my advisor Dr. Mohamed Mahfouz for his help and insight as well as providing many of the datasets, equipment, and funding from Rocky Mountain Musculoskeletal Research Lab (RMMRL) mentioned in this thesis.

Thank you also to the members of my thesis committee: Dr. William Hoff, and Dr. Roel Snieder. Their constructive criticism and advice has helped to shape both the course of my research and this manuscript.

I would like to thank Dr. Hoff and Jamon Bowen for the work done on the pose estimation sections.

Finally, I would like to thank my wife for her continued support and dedication in my education adventures. She has been behind me 100% in everything I have achieved and I could not have accomplished this without her. She truly has been my inspiration from the beginning.

xviii

DEDICATION

I would like to dedicate this thesis to my two wonderful little girls, Erin and Katie. May I instill in them the same desire to better themselves by loving to learn and learning to think as my father did for me.

xx 1

Chapter 1

INTRODUCTION AND THESIS OVERVIEW

Many medical procedures require three dimensional (3D) imaging of the human body. For example, when planning or executing surgical procedures, 3D imaging yields tremendous advantages over conventional 2D static radiographs. This is because when using 3D volumetric data, it is simpler to visualize complex geometries and understand the relationship between foreground and background objects. Intra-operative 3D imaging is extremely valuable during orthopedic surgery for the precise reconstruction of complex fractures, joint surfaces, and the exact positioning of screws and implants [1]. 3D image data can be obtained from CT and MRI, but these procedures are expensive and time consuming, and not available in most operating rooms. Fluoroscopy machines are much less expensive than CT or MRI instruments, and are readily available in many operating rooms. However, a fluoroscopy image is only a single plane projection, similar to a 2D x-ray. To obtain 3D data, the fluoroscope can be rotated in order to get multiple views of the patient’s anatomy, and the projections can be processed using cone-beam tomographic reconstruction methods [2-12] to produce a 3D volumetric image, similar to that obtained from CT.

Recently, Siemens has created a 3D fluoroscope using cone-beam tomography, called the SIREMOBIL Iso-C 3D (Figure 1). This instrument has a fluoroscope on a mobile C-arm that rotates 190-degree along a circular arc around the patient. The reconstructed 3D data are visualized as Multi-planar Reconstruction (MPR) or as a shaded surface display (SSD), see Figure 2. The system is especially suited for imaging of high-contrast objects such as bones and joints. General Electric has produced a similar 3D fluoroscopic imaging and navigation system called FluoroCAT, an add-on for their OEC® 9800 Plus fluoroscope.

Figure 1 On the left is a 3D fluoroscope sold by Siemens, called the SIREMOBIL Iso-C 3D. The right shows the General Electric version. [1]

Figure 2 Display of an elbow fracture with fissure extending into the joint in three MPR planes (left) and Hand in SSD (Surface-Shaded Display) from three different views (right) [1].

Although the method used in these systems gives good results, it has the disadvantage of requiring a specialized fluoroscope that is designed to precisely and automatically rotate the source and image intensifier combination around an axis. The precise mounting, motors, and sensors increase the cost of the unit, and there are great numbers of existing fluoroscopes in use that do not have this extra hardware and could therefore not easily be upgraded to this capability.

These commercial products illustrate that there is a need for such capabilities in the operating room. Our approach makes such a capability more broadly available by allowing most existing fluoroscopes to be upgraded. Our approach does not rely on instrumentation of the fluoroscope to determine its position, but instead recovers an accurate position directly from the imagery. Our method also does not require that the fluoroscope move along a predefined path, but can reconstruct 3D data from arbitrary (yet complete) sets of views (see Section A-9). This allows much greater flexibility in imaging configurations, even allowing reconstruction from images collected as the patient moves in the field of view of a fixed fluoroscope.

The work presented in this thesis has the potential to make a significant impact on the health care industry by providing a low cost 3D imaging capability to most medical facilities that have a fluoroscope (including many operating rooms and most clinics). In addition to the standard applications of 3D imaging in general medical diagnosis, this capability has applications in biomechanics research and will improve the cost effectiveness of Computer Aided Orthopedic Surgery (CAOS) and other forms of surgical navigation. Variations of this approach could also be applied to 3D imaging using highly portable systems such as might be used in field hospitals in disaster relief or combat situations.

To perform this cone-beam tomographic reconstruction, the relative position and orientation (pose) of the x-ray source, the sensor array, and the object being imagined must be known in order to reconstruct the 3D data. In order to perform accurate cone-beam CT you must know the path of the source/detector combination during the data collection, therefore we have developed a method that can detect the arbitrary motion for the reconstruction algorithm. The uniqueness of this work is in the estimation of this motion from the image data rather than from precisely instrumented actuators (as in the commercial products shown in Figure 1). Once the arbitrary motion is determined, the reconstruction algorithm of Grangeat [3], modified to handle an arbitrary path,, can reconstruct the 3D volume. The specific original contribution of this thesis to the medical field is the ability to use an existing fluoroscope to perform CT using motion estimated from the gathered imagery which includes the following:

• Correcting the images based on the found pose to align with the assumptions of the reconstruction algorithm (see Section 3.4) and modifying the cone-beam reconstruction algorithm suggested by Grangeat [3] to allow for arbitrary path reconstructions by extending the circle plus arc acquisition geometry suggested by Wang and Ning [11]

• Developing an appropriate search method for the resampling step as well as an appropriate scoring to determine the best plane matches.

• Developing a normalization factor similar to [8] which is used to account for data that may be missing due to a particular path that was taken during the data collection.

While the originality of the following items may not be new to computer vision and image processing, their application to tomography in the medical field is original. The following list also contains items completed for this thesis work.

• Calibration procedures that enable the raw data from a fluoroscopy machine to be related to the actual densities of the object being imaged (this allows the data to be used in tomographic reconstruction).

• Gathering data and calculating the pose using fiducial markers and the pose finding algorithm developed by Dr. Hoff and Jamon Bowen.

• Analyzing the pose finding algorithm for error by two different and independent methods as well as analyzing the reconstruction of the phantoms, both synthetic and physical, for error.

• Developing and gathering multiple datasets and performing reconstructions of those data sets including synthetically generated phantoms, physical phantoms, and a cadaver.

The outline of the remainder of this thesis is as follows. Chapter 1 is primarily focused on the previous work done in the field, mainly in the various cone-beam reconstruction algorithms. This is followed by a detailed description of our approach in Chapter 1. First we cover the data collection and calibration procedures, then we move to the description of the pose estimation and image correction algorithms, and finally we describe the reconstruction algorithm. Chapter 1 discusses the experiments performed to demonstrate and verify the processes explained in Chapter 1. This is followed by Chapter 1 and Chapter 5, which discuss the conclusions and future work respectively. Also included are two appendices, the first of which includes supporting information and proofs for other sections of this thesis. The second appendix contains the full commented code detailing the procedure and implementation of the algorithms discussed in this thesis.

Chapter 2

PREVIOUS WORK

In our work we propose a method of obtaining the pose information from the image data. We also develop a cone-beam tomographic reconstruction algorithm to process images collected along an arbitrary path to recover the 3D volumetric data. To solve these problems, we looked at the following previous works. First we discuss the related work in the pose estimation problem and follow with a discussion of the current methods in tomographic cone- beam reconstruction.

2.1 Pose Estimation

Tomographic reconstruction from projections requires an accurate knowledge of the position and orientation (the pose) of the sensor relative to the object being imaged [13]. A standard technique of determining the pose of a three dimensional object from a two dimensional image is to incorporate a set of fiducial markers of known 3D structure into the object. Once the images of these fiducial markers are correctly associated with their counterparts in a 3D model of the object, the pose of the model with respect to the imaging sensor can be determined using an “absolute orientation” algorithm [14]. Examples of 2D-to- 3D registration in x-ray images using the fiducial technique include [15-17]. The algorithm of [14] is described in more detail in Section 3.3.

Other approaches for determining the pose of the x-ray source during data collection include using external camera systems to track markers on the rotating c-arm of a fluoroscope. When the fluoroscope is mounted on a rotating c-arm, it is conceivable that the internal parameters may change slightly from one image to the next due to mechanical distortions of

the c-arm. Therefore a method of determining not only the extrinsic parameters of the camera system (the pose), but also the intrinsic parameters must be used. In [13] such a system is analyzed. They demonstrate a method for the use of external tracking sensors for recovering the projection geometry of a moving x-ray c-arm system and compare it to the use of a CCD camera attached to the x-ray source. They then use the found projective geometry to perform 3D tomographic reconstruction and compare the results of the two methods. They have found that while the external tracking system provides a better estimation of the motion of the x-ray, the mounted CCD camera provides better reconstruction results. This is because the main part of the translation error for the integrated CCD camera is always towards the optical axis of both the x-ray system and the CCD system, and therefore has only a small influence on the quality of the 3D reconstruction. For the external sensor, the main part of the translation error is always perpendicular to the optical axis of the x-ray system, and therefore has a large influence on the quality of the 3D reconstruction. Our method uses the better of their two methods because we are estimating the motion using projective geometry aligned with our x- ray source as their work describes. Our work uses the actual x-ray system to determine the pose and avoids the need for an external camera to determine the projective geometry of our system. While we are right now not solving for the full projective geometry, this could be added to our system. Since we are using a c-arm fluoroscope that does not move during the data collection (for our system the patient moves in the field of view of the fluoroscope) our projective geometry will remain constant throughout the data collection and therefore will not need to be solved for. In order for our method to be applied to other fluoroscope systems with rotating c-arms, we may need to include the full projective calculations in our approach.

2.2 Cone-Beam Tomographic Reconstruction

The Siemens SIREMOBIL Iso-C 3D mentioned in Chapter 1 is an example of a commercial cone-beam tomography unit using fluoroscopy. In addition, many approaches to cone-beam tomography have been developed or enhanced in earlier work [2-10, 18-21],

9 including many that process data collected along a specific path such as a circle [8, 22], a circle plus an arc [11], a helix [6, 23], two orthogonal circles (also [6]), and many others. We have selected the method of Grangeat, [3] as the most suitable for our problem since it is a well established and fairly efficient method of inverting the 3D radon transform for exact reconstruction in cone-beam tomography. This algorithm is developed for the special case of data collected along a circular path, so we modified it to allow reconstruction from imagery collected along an arbitrary path. We also incorporate into [3] the adaptations proposed in [11] for processing the specific path of a circle plus an arc. They have treated the circle path and the arc path as specifically different paths and derived the transformation matrices for each path independently. This allows them to process the circular path and then the arc path to satisfy the data sufficiency criteria (A-9). We combine these into one transformation to handle arbitrary path trajectories.

The following sections describe the most relevant cone-beam reconstruction methods and their benefits and drawbacks for our application.

2.2.1 Feldkamp’s method

Feldkamp, Davis, and Kress formulated a method for reconstructing cone-beam projections from a single circular orbit, which is commonly referred to as the Feldkamp method [4]. Feldkamp’s method is the most common method used today for cone-beam reconstruction [18] due to the fact that it is similar to 2D CT, so this method is easy to implement in existing hardware for people with experience in 2D CT. The single circular orbit however does not actually gather complete 3D radon data and therefore this method is an approximation when used on a cone-beam machine. However, even if complete 3D radon data is gathered, it is still an approximate reconstruction in the sense that the reconstruction result will deviate somewhat from the actual density (regardless of the angular step size) for any plane other than the mid-plane. The mid-plane is defined to be the plane that coincides with

plane of the circular path S(λ), where S(λ) is the source position indexed by λ . These deviations are small for small deviations from the mid-plane and are often acceptable but will increase as one moves further from the mid-plane. The Feldkamp method for 3D is just a generalization of 2D filtered backprojection applied to fan-beam projections [5]. Although this algorithm is approximate, Feldkamp, Davis, and Kress observed the following three attractive properties [18]:

• It is exact in the mid-plane. This is obvious since it is identical to the fan-beam reconstruction in this plane. The reconstruction becomes less accurate as one moves away from this plane.

• It is exact for objects homogeneous in the zˆ -direction, i.e. orthogonal to the circle formed by the source path S(λ).

• The integral value ∫ fx(,,yz)dz, where f (x, y, z) is the density function being reconstructed, is preserved. This is due to the fact that all three-dimensional radon data at z = 0 are measured with the single circular scan.

Turbel [18] demonstrates nicely how to use resampling with Feldkamp’s algorithm to produce parallel projection data for each row, or slice, separately. Once parallel projection data is obtained, simple parallel backprojection methods can be implemented. However, Feldkamp’s method does not lend itself to true 3D resampled parallel data. Instead Feldkamp’s method just resamples data from multiple projection views to produce parallel projection data when viewed along the axis of rotation. The resampling of the cone angle perpendicular to the rotation is considered to be an acceptable source of error in the reconstructed data since the cone angle is usually very small.

2.2.2 Grangeat’s method

The main contribution that Grangeat made in the development of CT was to establish an exact formula relating the cone-beam x-ray transform to the first derivative of the 3D radon transform. He uses 2D radon transforms of the pre-weighted input images as samples of the 3D radon transform, which he then interpolates to the standard spherical coordinate system of the 3D radon transform. The desired density function fx( ,y,z) is then recovered by inverting the 3D radon transform. The inversion of the 3D radon transform from Section A-7 is

1 ∂ f ()xR=− ' f()ρnˆˆdn 4πρ2 ∫S 2 ∂ where nˆ is a vector normal to the 3D radon plane, ρ is the distance to that plane from the origin, S is the set of all source positions, and R ' f is the derivative of the 3D radon transform of our object space. In order to perform this inversion, the cone-beam data is interpolated from the coordinate system of the detection plane to spherical coordinates so that parallel backprojection can be used to perform the reconstruction.

In order for Grangeat’s method to be computed exactly, there is a data sufficiency condition that must be adhered to (see Section A-9). So, although the continuous analytical formulation produces an exact reconstruction, in practice we will have approximate solutions due to the interpolation of the discrete samples of our images. We may also have to interpolate data if our arbitrary collection path does not satisfy the data sufficiency condition fully. Our method does not require that we stay on an exact circle, so we can tilt the object out of the circular plane to obtain samples from the areas that will not be covered by a single circular scan. By moving the object slowly in the field of view at 30 frames per second collection rate we can minimize the interpolation errors due to discrete sampling along the sensor path.

We decided to use Grangeat’s method because it is an exact method, and because of its straightforward implementation. In order to modify Grangeat’s single circular trajectory algorithm to handle 3 degrees of freedom, we reference [11] which modifies Grangeat’s

method for a circle-plus-arc data acquisition system. Resampling equations are provided for both the circle and the arc around the object of interest as two separate paths. We combine these into one rotation matrix to handle a more general trajectory.

2.2.3 Defrise and Clack method

The Defrise/Clack method (DC) is an exact reconstruction method if there is complete data to support it (see Section A-9). DC is based on Grangeat’s result, but the main difference is they backproject the 3D detector data directly without the resampling. [5]

The DC method can be summarized by the following steps

• Pre-weight the x-ray transform to give the weighted x-ray transform

XW ()u,v,inum , where (u,v) are the image coordinates, and inum is the image index.

• Compute the 2D radon transform the images.

• Differentiate perpendicular to the line-integration of the radon transform.

1 • Post-weight by , where β is the cone angle, which yields the derivative of cos2 β the 3D radon transform of the object, Rf',(ρnˆ inum) , where R' is the first derivative of the radon transform, and f here is parameterized by the normal to the radon plane and the image index inum .

• Weight by the M -function to average redundant data. M is a count of the number of times a particular 3D radon value is filled from the source data.

• Differentiate along the perpendiculars to the radon plane intersections with the image planes.

• Perform 2D backprojection in the detector planes. We now have X F f ()p, q,inum .

• Finally, perform 3D weighted backprojection for the resulting f ()x, y, z .

The DC method inherits all the advantages of any 3D exact reconstruction method. One nice feature of this method is the ability to process one projection totally while the next projection is measured which saves memory and shortens the remaining reconstruction time once the scanning has terminated. Since our method uses the method proposed by Grangeat, we could extend our method to use the DC method, however the advantages of this are limited and it would make more sense to implement our method with the LINCON method since its advantages are much greater.

2.2.4 LINCON method

LINCON is an efficient method to produce projections and backprojections in Fourier space. Therefore, to a large extent the LINCON can be applied to many reconstruction algorithms including the DC method and Grangeat’s method. The key to this method is computing the 2D radon transform in the Fourier domain. Taking the Fourier and chirp z - transforms puts the data into the linogram samples of the Fourier domain.

The general chirp z -transform described in Oppenheim and Schafer [19] can be used to compute samples of the z -transform if they lie on a spiral contour equally spaced in angle over some portion of the spiral in the complex z -plane. The Fourier transform of a function can be found along the unit circle in the z -plane. The unit circle is a special case of a spiral and thus, equidistant Fourier transform points can be calculated using a special case of the chirp z -transform, which gives us our desired Fourier transform. It should be noted that the Fourier slice theorem provides us with the Fourier transform of a projection of our x-ray data along the diameters of the radially interpolated Fourier space. However, the interpolation stage

to get to the radial representation in Fourier space must be made with care. Traditionally Fourier domain interpolation is considered to be very risky. The LINCON method, using this special case of the chirp- z transform, avoids this interpolation. Instead, the chirp- z transform computes the Fourier transform at the desired sample points. The detector data is processed along two separate paths. After pre-weighting, computation along the first path provides the vertical Fourier transform and the horizontal chirp- z transform. The computation along the second path provides the horizontal Fourier transform and then the vertical chirp- z transform. For each of these paths, we then compute the radial inverse Fourier transform of the derivative in the Fourier domain, and then combine the results and post weight to obtain the radon data for the inversion formula.

Note here, as in Grangeat’s method, the 3D radon data is obtained as 2D radon transforms of the cone-beam projections and must be interpolated to parallel projection data. Also, depending on the path and sampling density used, there may be missing or redundant data (in the shadow zone, see Section A-9) which will need to be handled somehow.

The radon inversion for LINCON is implemented using the 3D direct Fourier method with linogram sampling. The inversion is done along three paths, one for each coordinate axis direction, ( xˆˆ,,yzˆ). The radial Fourier transform is computed for each direction and filtered with an inverse derivative filter since we have the derivative of the radon space. Because of the linogram sampling, the radial Fourier transforms have samples at unit sampling in distance along each of the three axes, on the faces of concentric cubes. Therefore, only 2D interpolation on each cube face is required. Next, the inverse chirp- z transform is applied to all three paths along their respective directions. Summing the inverse FFT from each direction will produce the reconstructed object. For more details on the LINCON method see [5].

An obvious improvement to our approach would be to compute the various steps of our algorithm using this LINCON method. The 2D radon and 2D backprojection steps in Grangeat’s method are time consuming and involve interpolations which cause smoothing of the result. In LINCON, the 2D linogram method (a relative to the direct Fourier method) is

15 used to perform these steps with less computation than the Grangeat method. However, an even faster alternative is to use the 3D linogram method for the reconstruction phase. Either way the computational complexity is reduced from ON( 4 ) in Grangeat’s method to ON( 3 )log N in LINCON. Furthermore, the interpolations during line-integration and 2D backprojection are eliminated, which improves the image quality considerably.

2.3 Reconstruction Algorithm Speed Increase

There has also been much work done in methods to speed reconstruction algorithms in hardware [21], as well as software, [2, 9]. We leave such methods to future enhancements of this algorithm and present here the proof-of-concept experiments illustrating that our approach works. Some such methods are discussed in Section 6.7.

Chapter 3

APPROACH

The processing required for 3D tomographic reconstruction from cone-beam projections collected along an unknown, arbitrary path consists of two steps: recovery of the unknown pose (position and orientation) and cone-beam tomographic reconstruction. In addition, most fluoroscopes are not designed for the application we propose here, and therefore produce image data that is poorly suited to our application without calibration. The raw data has two types of distortion that are of concern, geometric and radiometric, both of which can be corrected. After these corrections, the image intensity at each image location will be proportional to the density integral along a straight ray projecting from the x-ray source to the image location on the detection plane. Such projection data is referred to as the x-ray transform of the 3D density function.

Reconstruction of a volumetric data set from x-ray cone-beam projections can be accomplished by computing the 3D radon transform from the x-ray projections, taking the derivative along the radial direction, and then using the 3D radon inversion formula to reconstruct the 3D density function [3, 5, 18]. The 3D radon transform can be determined by computing 3D radon values from 2D image radon values and resampling them from the coordinates of the detector plane to the spherical coordinates used in the 3D radon transform of the object. This results in parallel projection data, which can be processed into the 3D density function using parallel back projection with derivative filtering. The implementation of this for a specified path is well known for a variety of specific path types [8, 11, 18]. However, the approach we use here will allow reconstruction from projections gathered along arbitrary paths. The key difference lies in the resampling equations and the method for finding the

18 appropriate source location for given radon data. The process has been broken down into five steps and shown in somewhat more detail in Figure 3.

Figure 3 A Pictorial Overview of the Entire Reconstruction Process.

• Data collection from the fluoroscopy unit (described in Section 3.1)

• Calibration (geometric and radiometric, see Section 3.2).

• Pose estimation (described in Section 3.3)

• Image correction for source orientation (described in Section 3.4)

• Tomographic reconstruction (described in Section 3.5)

3.1 Data Collection

The data used for experiments in this thesis was collected on the fluoroscopy unit at Rocky Mountain Musculoskeletal Research Laboratory (RMMRL)†. A fluoroscopy machine is the same as an x-ray machine except that instead of capturing images on film while the patient remains motionless, the images are sensed on a phosphorous screen and captured on a CCD video camera. The camera’s output is sent to a VCR or to a frame grabber card on a computer. The data for the experiments described in this paper were captured directly to a computer at 30 frames per second and stored as individual images on hard drive in 8-bit grayscale windows bitmap (BMP) format.

We test the algorithms on multiple data collections in this thesis: computed projections of a synthetic volumetric phantom data set, fluoroscopic images of a physical phantom of known geometry, and fluoroscopic images of a cadaver leg. The physical phantom experiment data was collected along two paths: a random path (by rotating the physical phantom by hand in the field of view of the fluoroscope), and a rotation about a fixed axis. A jig was build for the latter path to allow precise measurement of the path to verify the pose estimation algorithms. Another jig was built to rotate the cadaver leg by hanging the leg from a turn table and rotating it by hand. For each of these setups, the data collected needed to be calibrated as the following section describes. The details of the multiple data collections and experimental setups can be found in Section 4.1.

3.2 Sensor Calibration

Before we can process the images using our proposed method, we must adjust the images through two calibration steps. There are two distortions present in the raw data from the fluoroscope. First, the images are geometrically distorted, as is common in many camera

† The fluoroscope unit was built by PNF Dynamics custom for RMMRL in June 1998. Control model type and number are CF3 and 1003 respectively. The tube was manufactured by N. Am. Imaging. The tube type and housing SN are E 7242 GX and 7K185 respectively.

systems. Using a calibration target we can estimate this distortion and remove it from subsequent images. Second, there is a radiometric distortion: the recorded intensity values are not proportional to the integrated density along the x-ray path as assumed in the tomographic reconstruction. Both these problems are discussed in the following subsections. Note that the computation of the calibration parameters only needs to be performed once for each fluoroscope (with periodic maintenance updates to correct for long-term temporal drift in the parameters).

3.2.1 Geometric distortion‡

Our reconstruction algorithm assumes a cone-beam projection image formation model, which treats the fluoroscope sensor as consisting of an x-ray point source and a planar phosphor screen upon which the image is formed. The first step in our calibration procedure is to estimate any 2D image geometric distortion. By taking an x-ray of a known rectangular grid of metal beads, we can estimate a 2D spatial transform for each small square sub-image that is bounded by four beads. Using standard techniques in geometric distortion removal (e.g., [24]), a local bilinear model is used to model the spatial mapping, as well as the gray level interpolation. Once the 2D distortion has been removed, the effective source-to-image plane distance (focal length) can be computed by a two-plane calibration grid with a known displacement between the planes (e.g. [25]). Figure 4 illustrates the before and after of a typical geometric distortion removal. We compute the bilinear transform for each set of four grid points that transforms the image positions of the beads in the left image to regularly spaced grid locations. It is clear that the calibration procedure removes the pin-cushion distortion so that the grid points lie along straight lines.

‡ The geometric distortion removal has been implemented from work done by Dr. Mohamed Mahfouz in earlier work.

Figure 4 Fluoroscopic image of geometric calibration grid, before distortion removal (left) and after (right). Note the “pincushion” effect visible in the left image.

3.2.2 Radiometric distortion

The second correction is for the radiometric distortion. The reconstruction algorithm assumes that the value recorded at a particular pixel is the integral of the 3D density function along the ray that projects to that pixel. The actual recorded value must be converted to such a density integral value. In addition, the sensitivity of the sensor is not uniform across the field of view, so the radiometric calibration must also correct for this effect. We determine the radiometric calibration parameters using stacks of aluminum sheet metal with varying total thickness in the field of view, and determine for every pixel the function relating the pixel value to the total thickness. The thicknesses used are shown in Table 1. Since the absorption is proportional to the aluminum thickness, this gives us a transformation from intensity to a quantity proportional to absorption. We assume that the image intensity I is related to the aluminum thickness t by the equation I = ΩeΓt + Ψ , so our problem reduces to finding values of Ω , Γ , and Ψ at every pixel location. We compute these parameters for a particular pixel using the non-linear optimization method of Marquadt [26] to find a fit to the image intensities recorded for the pixel at the various thicknesses of aluminum.

Table 1 Al Thicknesses Used # of Al Thickness # of Al Thickness Sheets (in) Sheets (in) 10 0.075 70 0.53 20 0.15 80 0.61 30 0.225 90 0.675 40 0.3 100 0.75 50 0.37 120 0.9 60 0.45

Some fluoroscopes have automatic gain control on the video output that introduces an unknown scaling and offset in the intensity values for each image frame. We dealt with this unknown by inserting two calibration spots of two different known thicknesses of aluminum into the field of view (near the edges as shown in Figure 5). These spots would remain a constant intensity if there were no variation due to unknown gain and offset, so we correct for this effect by linearly transforming the intensities of each image such that the average intensity in each calibration spot is a specified constant value.

Calibration Spots

3.3 Pose Estimation§

The next step in our method is to recover the unknown pose for all the images in the video sequence. We do this using a set of metal marker beads that are rigidly attached to the subject so that their positions are known and remain fixed relative to the subject. The markers can then be automatically detected in the images using morphological operations. The placement of the markers on the subject is chosen to cover as large an image area as possible in all the frames. Enough markers were used to allow accurate pose estimation even when the feature detection algorithm missed some of them, or when some of them were outside the field of view. Figure 5 shows the beads in the un-calibrated image as dark dots located around the knee.

3.3.1 Image feature matching to find the pose

The first step in determining the pose of the model with respect to our camera system (the fluoroscope) is to detect the markers in the image. After that we need to find the correct correspondence between the marker locations in the model and these image locations. Given these correspondences we can determine the pose.

3.3.1.1 Finding the beads

By thresholding at various levels and removing (with morphological operations) any objects that contain lines longer than the bead diameter, we can isolate these beads as shown in the left part of Figure 6. We find the centroids of the beads by finding the connected components and then determining the centroid of each component. We expect three types of errors from this detector. If two beads overlap, they will either be missed, or they will be

§ The pose finding algorithms were developed and implemented in part by William Hoff and Jamon Bowen.

interpreted as one bead with a centroid between the actual centroids depending on the degree of overlap. The detector may miss some of the beads, or finally, the detector may find false positives. The right image of Figure 6 shows an example of the detected markers found in a sample image.

Figure 6 Beads found in fluoroscope image displayed as connected components (left) and projected pose for all beads in the bead model (right)

3.3.1.2 Marker tracking correspondence

We compute the pose of the sensor by finding correct associations (or correspondences) between the detected marker locations in the image and the 3D marker locations. This is accomplished using an interpretation tree search method [16] as shown in Figure 7. In this method, each level of the tree represents the correspondences between an image point and all possible model points, and each path from the root to a leaf node in the complete tree represents one possible correspondence. When a search descending the tree reaches a node at which at least four correspondences have been hypothesized, the correspondences are used to calculate a pose solution. In general, we only need three points to solve for a pose, but there

25 may be more than one solution since three points are always co-planner. In fact there is more than one solution for many arrangements points that are symmetric. To help resolve this ambiguity, we require that at least four points be used and ensure that the beads are placed randomly on the object being scanned to avoid the unwanted symmetries.

Root

Image Point 1 = 1 2 3 4 Model Point #

Image Point 2 = 2 3 4 1 3 4 1 2 4 1 2 3 Model Point #

Image Point 3

= 3 4 2 4 2 3 3 4 1 4 1 3 2 4 1 4 1 2 2 3 1 3 1 2 Model Point # Figure 7 Interpretation search tree. Each level of the tree is searched to find a possible set of correspondences. When enough hypothesized correspondences have been found, the pose is calculated and tested. If it is found to be an incorrect correspondence, the tree is searched further.

Once we have four correspondences we can calculate a pose. This pose solution is used to project the object points back onto the image for comparison to the detected markers as a measure of how well this solution fits the complete data set. If the pose solution fits the data well then the correspondence and computed pose are correct. If the pose solution does not fit the data, then the correspondence is wrong, and the tree is traversed further to check additional correspondences. Since for a large number of points this tree may be very large, we do not search the entire tree. In fact, we search the tree further only if the current correspondence produces an error larger than some threshold. If it is below, we assume we

have found a correct correspondence. Since we are processing a continuous video file, we have the added benefit of using the correspondence found in the previous image as a starting guess at the correspondence for the current image. If the object in the field of view has not moved far (for small inter-frame motion) then our correspondence will be near the beginning of the search. Because of this, we also put a limit on how far the tree can be searched before the algorithm gives up and moves to the next image. This can occur for an image that has many false positives or where two beads overlap slightly and the detection algorithm finds only one centroid that does not match well with either bead.

3.3.1.3 Pose computation with least squares iterative technique

Once the correct correspondences have been found, we compute the pose using a non- linear least squares method as described in [27]. Specifically, given an image point P , a 3D model point Q , and the six-vector containing the correct pose parameters β , let the transformation to project Q onto the image be given by the function f (β,Q) such that P = f (β ,Q). The vector function f also represents a perspective projection (whose parameters are known from calibration). Linearizing around a current pose solution β we get

⎛⎞∂f ∆P =⎜⎟∆β . (1) ⎝⎠∂β

Given at least four point correspondences, we can solve for the correction term ∆β . This process is iterated until the solution converges. The final solution is the pose β that minimizes the squared Euclidean distance between the observed image points and the projected model points on the image plane.

3.4 Image Correction

Because our data is collected along an arbitrary path, there are six degrees of freedom in pose that have to be considered in the algorithm for 3D tomographic reconstruction. However the reconstruction algorithm makes several assumptions. It assumes 1) the world origin is near the center of the imaged volume, 2) the image plane is orthogonal to the line from the x-ray source to the world origin, 3) the horizontal axis of the image plane is contained within the world’s horizontal plane, and 4) the image plane passes through the world origin. Assumptions 2) and 3) effectively remove the rotational elements of pose from the reconstruction algorithm, so that it requires only the position of the sensor (three degrees of freedom) to represent sensor pose. These rotational parameters can also be removed from our images by reprojecting them to the image plane assumed by the reconstruction algorithm. Note that if the camera location is not changed, an image can be reprojected to any desired image plane using a projective transformation of the image. The following subsections will detail this process, which involves two steps. First we determine the world origin as the point in space that minimizes the average distance to the camera principal ray (the ray perpendicular to the image plane) in all the views. Next we determine the projective transformation of the image that produces an image meeting assumptions 2) and 3) above.

3.4.1 Finding the optimal origin

Since the sensor orientations are arbitrary, we have no guarantee that they will all be pointing to a common point in space as our algorithm assumes. Therefore, we need to reproject our images such that they all point at the same location in space and define this point as our reconstruction world origin. We do assume that the anatomy being imaged is kept in the field of view throughout the sequence. The question then becomes how to find the optimal location in 3D space for this world origin.

Let us define WAS , WIS , CAS , CIS , and VPS as the actual world coordinate system, the ideal world coordinate system, the actual sensor coordinate system, the ideal sensor coordinate system, and the virtual detector plane coordinate system respectively. WAS is chosen arbitrarily when the model of the marker locations is constructed, and is typically defined somewhere in the middle of all the markers. We define the WIS coordinate system as a translation of the WAS such that the WIS origin is the point that minimizes the average distance to all our sensor principal rays. CAS is the coordinate system attached to the sensor,

with its origin at the sensor focal point, its z -axis ( zˆCAS ) along the sensor principal ray, and its

x -axis ( xˆCAS ) pointing to the right in the image. The CIS is the ideal sensor system whose origin is at the same location as CAS , but whose principal ray ( zˆCIS ) points to the origin of ˆˆ WIS , and whose x -axis ( xˆCIS ) is parallel to the WIS ijWIS WIS -plane. Finally, VPS is a

coordinate system whose origin is aligned with WIS and whose w -axis, wˆVPS is the negative ˆˆ of zˆCIS . It follows then that uˆVPS will always be in the ijWIS WIS -plane (see Figure 8). Note that we can think of the image pixels as a set of rays (each passing through an image pixel) defined in CAS . If we transform these rays into CIS , we have a representation of the image in the desired coordinate system (meeting assumptions 2) and 3) above). Thus the desired projective transformation of the image is given by the rotational transformation from CAS to CIS . If we then scale the resulting image (or, equivalently, change the focal length) we can shift the image plane to the virtual detector plane so that the image plane lies within VPS to meet assumption 4).

zˆWAS xˆ kˆ CIS WIS ˆ xˆ ˆ yWAS xˆ CAS yˆCASzCAS WAS zˆ ˆj CIS WIS vˆ wˆ VPS VPS ˆ iWIS yˆCIS P0 Arbitrary Path uˆVPS Figure 8 The five coordinate systems (CAS , CIS , WAS , WIS , VPS ) which govern the re-orientation of the images.

The pose estimation software that is discussed in Section 3.3 provides us with the pose of

the CASi (where the subscript i refers to a particular camera location) relative to the WAS .

CASi CASi This information is in the form of the rotation matrix RWAS and the translation vector PA which will transform the WAS coordinates of a point, PWAS , to CAS coordinates as in

PR=CASiP+PCASi. Consider the unit vector ζˆ the principal ray (along zˆ ) of the CASi WAS WAS A CASi

th CASi th i sensor expressed in WAS (the third row of RWAS for camera i ), and Pi, the i sensor

T CASiCASi center in WAS (PRiW=−()ASPA for camera i ). See Figure 9 for an illustration of these vectors and points for cameras i = 1, 2, and 3.

P1 PCP,1 P CP,2 ˆ r1 ζ1 ˆ r ζ 2 2 P r P P2 0 3 CP,3

ζˆ Arbitrary 3 Path

P3 Figure 9 Example of 3 different source positions and associated parameters used in finding the optimal WIS origin.

We first define the point P0 as the point that minimizes the average perpendicular distance to the principal rays of all the sensors. We will use this point as the origin of the WIS . We also ˆ define PCP,i as the closest point to P0 along ζ i . We can then formulate the point PCP,i in the WAS as (see Figure 9 for reference).

ˆˆˆ PPCP,0i =+i (−Pi •ζζi +P•i )ζi (2) ˆˆTTˆˆ =+PPii0ζζi−Piζζii

ˆ where again, ζ i is a unit vector in the direction of zˆCAS,i expressed in WAS coordinates.

Now, ri is just PCP,i − P0 , or

ˆˆTˆˆT rIii=−()ζζiPi−(I−ζζii) P0 . (3)

This equation is linear in P0 , so we can minimize the sum of the squared magnitude of ri over all cameras i and solve for P0 using linear least-squares methods. This will give us the 3D translation from the WAS origin to the WIS origin given in WAS coordinates.

3.4.2 Re-orientation of images

Given the computed WIS origin P0 , we now must find the 3D rotation for each image from CAS to CIS that aligns the sensor principal axis with the WIS origin. We determine this rotation as a sequence of transformations from CAS to WAS to WIS to CIS to VPS as

CIS CIS VPS WIS WAS RRCAS = VPS RWIS RWAS RCAS (4)

CIS VPS WIS WAS where RVPS , RWIS , RWAS and RCAS are the rotations from VPS to CIS , WIS to VPS , WAS

WAS WIS to WIS , and CAS to WAS respectively. RCAS is given from the pose estimation. RWAS is just

VPS the identity matrix since WIS and WAS differ only by a translation. RWIS can be derived from the geometry that relates our camera center to the WIS coordinate system. Consider VPS (meeting all of the assumptions 2), 3), and 4)) with coordinate axes uˆ,vˆ,wˆ as illustrated in Figure 8 and Figure 10. The camera center always lies somewhere along the wˆ -axis, and the

CIS uvˆˆ-plane is the image plane. We can construct the rotation RCAS more simply by defining two intermediate rotations as follows. First we write the transform from VPS (uˆˆ,,v wˆ) (where

XXX (•) refers to coordinate system XXX , whose axis labels are • ) to (uzˆ,,ˆ wˆ') (as

indicated in Figure 10), then transform from (uzˆ,,ˆ wˆ') to WIS (iˆˆ,,j kˆ) . From Figure 10 we have

⎡u ⎤ ⎡1 0 0 ⎤⎡ u ⎤ ⎢v ⎥ = ⎢0 cosψ − sinψ ⎥⎢ k ⎥ ⎢ ⎥ ⎢ v v ⎥⎢ ⎥ ⎣⎢w⎦⎥ ⎣⎢0 sinψ v cosψ v ⎦⎥⎣⎢w'⎦⎥ (5) ⎡ u ⎤ ⎡cosψ h sinψ h 0⎤⎡i ⎤ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ k ⎥ = ⎢ 0 0 1⎥⎢ j⎥ ⎣⎢w'⎦⎥ ⎣⎢sinψ h − cosψ h 0⎦⎥⎣⎢k⎦⎥

VPS or, for the overall transform RWIS , we have

⎡⎤ui⎡10 0⎤⎡cosψψhhsin 0⎤⎡⎤ ⎢⎥vj=−⎢0cosψψsin ⎥⎢ 0 0 1⎥⎢⎥ ⎢⎥ ⎢ vv⎥⎢ ⎥⎢⎥ ⎣⎦⎢⎥wk⎣⎢0 sinψψvvcos ⎦⎥⎣⎢sinψh−cosψh0⎦⎥⎣⎢⎦⎥ (6) ⎡⎤cosψψhhsin 0 ⎡i ⎤ =−⎢⎥sinψψsin sinψcosψcosψ⎢j⎥. ⎢⎥vh v h v⎢⎥ ⎣⎦⎢⎥cosψψvhsin −cosψvcosψhsinψv⎣⎢k⎦⎥

CIS Finally, RVPS is really just a negation of the second and third coordinates:

⎡⎤xCIS ⎡10 0⎤⎡u ⎤ ⎢⎥y=−⎢010⎥⎢v⎥. (7) ⎢⎥CIS ⎢ ⎥⎢⎥ ⎣⎦⎢⎥zwCIS ⎣⎢00−1⎦⎥⎣⎢⎦⎥

kˆ vˆ

wˆ

ψ wˆ ' v

ψ h ˆj

ˆ i uˆ Figure 10 Transformation from VPS to WIS

Since the change from CAS to CIS does not involve a translation of the coordinate

CIS origin (or camera center), the image from the CAS can be transformed into CIS using RCAS as a projective transformation as described in [28]. This has the effect of removing three of the

six degrees of freedom in the sensor pose, namely those associated with orientation. The ideal

sensor pose is thus captured entirely in the location Pi of the sensor focal point in WIS coordinates. In the reconstruction algorithm the point, Pi , is represented in spherical coordinates (ψ h,i ,ψ v,i , Di ) which are derived from Pi in the WIS . Finally, the last step is to scale the images such that the virtual detector plane passes through the VPS origin (assumption 4)) as illustrated in Figure 11. If the focal length of our fluoroscope is f , we scale the image by Di / f to transform to the desired virtual detector plane. Combining equations (6) and (7) with this scaling, we see that our overall projective transformation is

⎡⎤xxCIS ⎡cosψψh sin h 0 ⎤ ⎡CAS ⎤ ⎢⎥yR=−⎢sinψψsin sinψcosψ−cosψ⎥⎡⎤WAS ⎢y⎥, ⎢⎥CIS ⎢ v h v h v ⎥⎣⎦CAS ⎢CAS ⎥ ⎣⎦⎢⎥Dfiv⎣⎢−−cosψψsin hcosψvcosψhsinψv⎦⎥ ⎣⎢ ⎦⎥

where the actual image coordinates are computed by dividing the first two components of this result by the third.

vkˆ, ˆ iˆ wˆ ˆj uˆ zˆCIS

ˆ yCIS xˆ CIS Figure 11 The scaling of the real detector plane to the virtual detector plane. Notice that the virtual detector plane passes through the VPS origin, which is coincident with the WIS origin. Also notice that the uˆ -axis is in the ijˆˆ- plane.

3.5 Tomographic Reconstruction

The final step, cone-beam tomographic reconstruction, modifies the result of Grangeat [3] to reconstruct from data collected on an arbitrary path by extending the circle plus arc geometry suggested by [11]. In order to describe this process we will first need to describe the geometry that is associated with the reconstruction algorithm. We then move into a detailed description of the algorithm with our adaptations.

3.5.1 Reconstruction algorithm geometry

After applying the image correction process described in Section 3.4, all the images are in the form assumed by the reconstruction algorithm. In the following description of the tomographic reconstruction geometry, the reader should refer to Figure 12 and Figure 13.

nˆ t ρ A α D Pi w u

Figure 12 Typical radon plane shown in VPS , where nˆ is the unit normal to

this plane and Pi is the camera center.

As discussed before, WIS is fixed with respect to the object being reconstructed. We also have VPS , which was described in 3.4.1, and whose origin is coincident with the WIS origin. The uvˆˆ-plane is the transformed (rotated and scaled as described in Section 3.4.2) virtual detector plane, shown in Figure 11 and Figure 12. The rotation between VPS and WIS is

defined in terms of the angles ψ h and ψ v , the azimuth and elevation angles respectively (see Figure 13).

kˆ vˆ

nˆ ψ wˆ v ˆj ϕ ˆ i ψ h uˆ

Figure 13 Illustration demonstrating the definition of ψ h and ψ v as well as the relationship nˆ has with the two coordinate system, WIS (iˆˆ,,j kˆ) and VPS ()uˆˆ,,v wˆ.

We now define the radon plane, which is specified by its unit normal nˆ and its distance ρ from the origin (Figure 12). We can also parameterize nˆ by the inclination angle θ it makes with the kˆ -axis and the angle ϕ its projection in the ijˆˆ-plane makes with the iˆ-axis (see

Figure 13). Note that this plane always passes through the sensor focal point Pi located at

distance D from the origin on the wˆ -axis. The radon plane is shown as the shaded triangular plane in Figure 12.

The intersection of this plane with the image plane creates a line t as shown in Figure 12. The perpendicular distance in the image plane from the origin to the line t is A , and t is parameterized by A and the angle α that A makes with the uˆ -axis. This defines the geometry needed to describe the reconstruction formulation.

3.5.2 Reconstruction algorithm

In order to provide an adequate explanation of our reconstruction process, we must first gain an understanding of the radon transform and the radon inversion formula. We must also understand the x-ray transform, which models our cone-beam projections. Finally, we must understand how to relate the x-ray transform to the radon transform, as used in Grangeat’s method. These ideas are presented in the next section. We can then continue by describing the steps of our algorithm in the subsequent four subsections.

3.5.2.1 Grangeat’s development

Most of the derivation in this section has been adapted from [5]. We have defined a 3D radon plane that passes through the object and the current source location by its unit normal vector nˆ and the plane’s distance ρ from the origin. Then the x-ray absorption density function values defined on such a plane can be written as

f (ρnrˆ,,γ ) (8)

where r , the distance from the source, and γ, the angle from the ray through ρnˆ , define a point on this plane (see Figure 14). The radon transform of this object function is given as the

integral of f (•) over the 3D radon plane specified by ρnˆ : f (ρnrˆ,,γ ) . Note that the integration is taken over the half of the 3D radon plane in front of the sensor under the assumption that f (•) is zero in the other half-plane.

π /2 ∞ Rfn()ρˆ= fn(ρˆ,,rγ)rdrdγ (9) ∫∫−π /2 0

Assuming that our images are formed by the integration of densities as a ray passes through an object, we see that the images collected are the x-ray transform Xf (ρnˆ,γ ) of the x-ray absorption density we seek to recover.

∞ Xfn()ργˆ,= fn(ρˆ,r,γ)dr (10) ∫0

Integration of the x-ray transform over γ gives (11), which unfortunately differs from (9) by a factor of r inside the integral.

ππ/2 /2 ∞ Xfn()ρ ˆ,γγd= f(ρnˆ,r,γ)drdγ (11) ∫∫−−ππ/2 /2∫0

vˆ

ρnˆ r γ

source wˆ uˆ

Figure 14 Radon plane parameterization

In the ideal situation the 3D radon transform of our object function would just be the 2D radon transform of the x-ray projection data, allowing us to simply invert the 3D radon transform to reconstruct our object function. However, this is not the case as we have demonstrated. Grangeat developed a relationship between the x-ray transform and the derivative of the 3D radon transform with respect to ρ. We define our cone-beam reconstruction space more clearly with the following figures. Figure 15a shows the cone-beam geometry for two different planes. The first plane is formed by the lines CpQ and SC p . The second plane is formed in the same way, but offset slightly by dβ giving us dρ and dA . Figure 15b shows the two planes viewed from the bottom and indicates the values of r , γ , and rcosγ .

O t D uˆ α Q S β ρnˆ A r dβ C Q t p γ dρ S C dA r cosγ p

(a) (b)

Figure 15 Grangeat’s cone-beam geometry. The description of the cone-beam geometry

is shown in (a). The plane formed by the line CpQ and the line SCp is shown in (b).

The radial derivative of the 3D radon transform, (9), can be expressed as

∂∂π /2 ∞ Rfn()ρ ˆ= fn(ργˆ,,r)rdrdγ. (12) ∂∂ρρ∫∫−π /2 0

dρ(r) From Figure 15, using the small angle approximation for sin(dβ ), we have that dβ = . rcosγ From this we can make the substitution by the chain rule,

d d dβ d 1 = = dρ dβ dρ dβ r cosγ

which gives us

∂∂π /2 ∞ 1 Rfn()ρρˆˆ= fn(),,rγdrdγ ∂∂ρβ∫∫−π /2 0 cosγ (13) ∂ π /2 1 = Xf ()ργnˆ,.dγ ∂βγ∫−π /2 cos

This is Grangeat’s main result. It states that one can compute the derivative of the radon transform from the x-ray projections. From Figure 15 we can derive the following relationships.

A = D tan β D dA = dβ cos2 β

By making this substitution in (13) we have

∂∂D π /2 1 Rfn()ρ ˆˆ= Xf()ργn, dγ. ∂∂ρβcos2 A ∫−π /2 cosγ

We can also note the relationships

tS= Cp tanγ SC SQ . dt ==p dγγd cos2 γγcos

By making these substitutions to express (13) in our detector plane coordinate system we have our final result that will guide the first step of our reconstruction process as

∂∂1 ∞ D R ',f ()A αρ,inum ==Rf ()nˆ Xf t ()A,α,inum dt .(14) 2 ∫−∞ () ∂∂ρ cos β A Du22++v2

Note here we have used the fact that SQ = D22++u v2.

This computation can be broken down into three steps. First we pre-weight our (calibrated D and corrected) image data by . Next, we take the radon transform of all our D2 + u2 + v 2 individual images as denoted in (14) by the integration. Finally, we take the derivative with 1 respect to the 2D radial coordinate A from the radon transform and post weight by . cos2 β Our method of performing these operations is described in detail in the following sections.

3.5.2.2 First derivative of the radon transform from the x-ray projections

As described in the previous section, the first step is to perform the pre-weighting similar to that used in most cone-beam algorithms (e.g., [3, 5, 8])

D Xfw ()u,v,inum = Xf (u,v,inum). (15) D2 + u 2 + v2

The next step is to compute the radon transform of each image. This gives us samples of the 3D radon data for our volume. The radon transform is computed using the MATLAB® “radon” function, a sample application of which is described in Section A-1.2.

Then we differentiate in the 2D radon space along A, by filtering the data with the well- known Sobel filter. Finally we apply the post weighting to all our images as given in (14), which gives us R' f ()A,α,inum .

3.5.2.3 Resample to spherical coordinates

Now that we have the 1st derivative of our radon data, we would like to perform the 3D inverse radon transform to build our reconstructed volume. (16) is the inversion formula that needs to be applied, however as (14) is written, our 1st derivative of the radon data is parameterized by A, the radial component of the 2D radon transform, α , the angular component of our 2D radon transform, and the index of the images inum . In order to perform the radon inverse shown in (16), we must have our data parameterized by (ρ,θ,ϕ), which suggests a resampling step to those coordinates. For a proof of (16), the radon inversion formula, please see Section A-7.

1 ππ/2 ∂ f ()xy,,z =− R'f()ρ,θϕ, sinθdθdϕ (16) 4πρ2 ∫∫0/−π 2∂

In general when resampling from one set of coordinates (the source space) to another (the destination space) there are two approaches that can be used, forward and backward mapping schemes. For a forward mapping scheme you calculate all the bins that your source space will map into and simply place the data into those bins. However, there may be many cases, depending on step sizes and the relating geometries, where you will have multiple data mapping to the same destination bin. There also may be the case where destination bins will not be filled from your source space due to missing data. One way to avoid such problems is to use a backward mapping scheme. In this method each bin of the destination array is filled by finding the exact location in the source space that would map to the destination bin and interpolating in the source space for the value to be placed in the destination bin. There are various methods of interpolation that can be used, some common methods would include nearest neighbor, bilinear interpolation or spline curve fitting. For our particular problem we started with the backward mapping scheme with bilinear interpolation and found it to be rather slow because of the searching that was needed to find the best estimate of our mapped value. We therefore moved to a forward mapping scheme and found that by choosing for each destination bin the value from the source bin that maps the closest to the destination bin, our

destination space was mostly filled, speed was increased by nearly a factor of ten, and the final output was not significantly degraded. In fact, visually there was almost no difference detected.

Independent of which method (forward mapping scheme or backward mapping scheme) is chosen; one still needs to determine some criterion for choosing the best matching radon plane in the resampling step. One method, which we implemented for the backward mapping, is to use the fact that in order for a particular image to have information about a particular radon plane, the x-ray source will have to be located on that plane. Therefore we can derive the equation of the desired radon plane in our source space and select the source location that is closest to our desired plane. We refer to this method as the source search method. Another method, which we implemented in the forward mapping scheme, is based on the volume between the two planes within the reconstruction region. We calculate this volume and compare it to the volume calculated from the plane that is currently stored in the destination array and take the minimum found for all the planes that project to a particular destination array location. We refer to this method as the volume minimizing method. The source search method may suffer from the possibility that the plane that is closest to our x-ray source at the source may deviate significantly from the desired plane in the actual reconstructed volume area. The volume minimization method takes care of this problem by matching the radon planes in the actual reconstruction area.

Both mapping schemes are discussed in the next subsections along with the corresponding plane matching method that we used in each case. Although the reader should note that the plane matching method is independent of the use of a mapping direction (forward or backward), we present them here together since that is the way they have been implemented.

3.5.2.3.1 Backward mapping scheme with the source searching method

Because we allow for an arbitrary path, no pre-computation of a resampling equation can be formulated. Therefore we must implement a search method to find the best possible radon

value to use in our resampling step. We resample the data into (ρ,θ,ϕ) coordinates which will allow us to perform parallel backprojection as in [3]. In the backward mapping scheme, this involves stepping through all values of (ρ,θ,ϕ) in the output array, determining which sensor locations intersect the corresponding radon plane, and selecting the appropriate radon value in the image from these sensor locations.

Since we are using a backward mapping implementation where we transform from the output array values (ρ,θ,ϕ) to the sensor location and radon values (A,α ), we need a representation of the source position in VPS (uˆˆ,,v wˆ) in the WIS (iˆˆ,,j kˆ) coordinate system.

As described in Section 3.5.1, we represent the source position with the two angles, ψ h and

ψ v , and a distance D . If we use the fact that the center of our image is at the WIS origin, and the fact the sensor is at w = D in VPS (uˆˆ,,v wˆ) coordinate system, we have the following equations for sensor location in WIS (iˆˆ,,j kˆ) from (6),

xD= sinψψhvcos

yD=− cosψhcosψv (17)

zD= sinψ v .

Now, if we substitute these values into the following equation for the radon plane,

x()sinθ cosϕ + y(sinθ sinϕ)+ z(cosθ )− ρ = 0 , (18)

we can determine if the sensor location is on the given radon plane (a necessary condition for the radon transform of the sensor image to contain radon values for this radon plane). The

(ψ h ,ψ v , D) combination for which this equation holds denotes an intersection of the path with the radon plane, and values close to zero will indicate a source point very close to intersecting the plane. For a general path, we may have multiple intersections with a particular radon plane, or none at all (Figure 16). Because of our discrete system it is unlikely that a given radon plane will exactly intersect one of the source locations, so we simply choose the closest

source locations to the intersection. We search through all our camera center locations to find a source position that is closest on either side of the radon plane of interest and then interpolate between the radon planes selected at those two sensor locations to find the radon plane value we will use.

(a)

(b)

(c)

Figure 16 Illustration of a typical discrete path and the closest intersections.

There are four cases that can result from our search if we do not have an exactly intersecting source location, three of which can be seen in Figure 16. The first case is that our path passes through the radon plane. In this case, we will have two consecutive points on the path on either side of the radon plane that will be close to the desired plane. This case is shown in Figure 16a. The second case is also a case where the source trajectory crosses the desired radon plane, but in this case the two closest points are not consecutive. This is shown in Figure 16b. The third case occurs when we have valid camera centers only on one side of the desired radon plane as shown in Figure 16c. In this case we do not have any points on the path on either side of the plane of interest, but we do have a point that is very close (we define “very close” as within a threshold of 0.5 pixels) so we extract our radon plane from the 2D radon transform of the image collected that source location. The fourth case is when there are

no points that come close and there is no intersection with the plane of interest. In this case we do not fill that radon value and consider it to be in the shadow zone, i.e., we do not have data for it. Furthermore, we count how many destination positions we have data for (something we refer to as the shadow zone) and create another array to keep track of the shadow zone to be used as a normalization in the backprojection steps. This will give us a measure of how to normalize a particular bin based on how many contributions it had from various projections. This counting array, N(ρ,θ,ϕ), is processed through the backprojection steps exactly along side Rf',(ρ θϕ,) and is used at the end of the process as a normalization factor.

If the search described above produces one or two sensor locations, then the 2D radon transform of the pre-weighted image collected at these locations should include radon values for a plane close to the desired 3D radon plane, (ρ,θ,ϕ). For each of these sensor locations we need to calculate the A and α for these 2D radon values. From Figure 13 we know that

α = atan2(nnˆˆ'v, 'u), where nˆ '= nˆ *sign(ρ) and atan2 is the four quadrant arctangent. To

obtain nˆ 'uvw , we can write nˆ 'ijk directly and transform nˆ 'ijk to the VPS (uˆˆ,,v wˆ) coordinate

VPS system using RWIS (from Section 3.4.2). We can write α as

α = atan2(nˆ'v , nˆ'u ) (19)

where

nˆ 'vh=−sign (ρ )(cosψθcos sinθsinψvsin (ψh−ϕ))

nˆ 'suh=−ign()ρθ()sincos(ψϕ).

In general, α may be out of range since we only compute the radon transform for angles 0 to 180 degrees and (19) will compute values in the range −180o to +180o . Since the radon transform is redundant for angles offset by 180o, we can add or subtract 180o from α and negate the value of A to bring α back into range.

Now that we have α , we can compute A . We first find use ρ and D to find A. From Figure 12, we have

ρ D A = ± . D2 − ρ 2

Here the sign of A is ambiguous and depends on whether α is found to be in range or not. The sign of A does not necessarily match ρ since the sign of A depends on whether the radon plane is in the +vˆ hemisphere, and the sign of ρ depends on whether the normal to

the radon plane is in the +kˆ hemisphere (see Figure 12 and Figure 13). For a general rotation these hemispheres may not coincide. We use this computation to find the magnitude of A and then depending on the angle α , we change the sign of A and adjust α accordingly.

Given these equations for finding the appropriate α and A values for a given 3D radon plane and sensor location, we compute these values for the one or two sensor locations obtained as described above. Since the computed α and A values may not be in the set of 2D radon values computed for the pre-weighted image, we interpolate the available 2D radon values. If two sensor locations were returned by the search, we then perform a weighted sum of the interpolated 2D radon values with weights depending on the source location’s distance to the 3D radon plane of interest. The resulting value is then stored in the output array at location (ρ,θ,ϕ), completing the backward mapping resampling step.

3.5.2.3.2 Forward mapping scheme with the volume minimization method

As an alternative to help increase the speed of our algorithm we also decided to implement a forward mapping scheme. Now instead of stepping through our destination space and searching for the best source location, we step through our source data and forward project it to our destination space. Depending on our path taken during the data collection, we will have

a set of ()A,,α inum available to us. From these coordinates we must determine the appropriate (ρ,θϕ,) bin to place our radon data into. From Figure 17 and using similar triangles we have

AD ρ = . A2+ D2

Also note from Figure 15 we can compute the (uv,,w) -components of our radon plane normal as

ρ 2 ρnˆ = cosα u A ρ 2 ρnˆ = sinα v A Aρ ρnˆw = . A22+ D

D S O

ρnˆ A AD22+

Figure 17 Geometry relating A and ρ .

Now that we have our plane in VPS (u,,v w) we can use the transform given in (6) to transform this vector into WIS (i,,j k ) and parameterize the normal to the plane with

(ρ,,θϕ) so that we can put the radon plane value in the correct destination space bin.

With multiple images and a completely arbitrary path, we may have data from multiple sources that will project into the same destination bin. Therefore we need a way to handle this extra information. [8] uses a counting array where they determine how many contributions they will have to a single resampled point. They can then divide by this count to find an average for their resampled data. We address the issue of multiple data for a single resampled point by using the best estimate of that point based on a volume measure. Our criterion for measuring the closeness of two radon planes is based on the volume captured between the two planes within a specified region. We define two planes which can be calculated from their unit normals given by (ρ,,θϕ) as the plane bin in our destination array or P1, and the plane which we have data for from our source space, or P2 ,

Px1:[ ,y,z]•=nˆ11ρ

Px2:[],y,z•=nˆ22ρ .

We can write the perpendicular distance between an arbitrary point in P1 and P2 as

[xy,,z]nˆ2− ρ2, (20) where [x,,yz] is a vector to an arbitrary point on P1. Solving P1 for z **, substituting into (20), computing the product suggest by (20), squaring so that we can look at the squared distances, and integrating over a pre-defined square space of 2d by 2d gives us our match score correlating to how well the two planes are aligned.

** In general we need to solve for whichever variable, (x,,yz) is most aligned with the normal nˆ . However, right now we simply just solve for z . Note also that this will take care of any division by zero in (21) since the coefficient in the divisor will always be the greatest in magnitude of the three (i.e. not equal to zero).

2 dd⎧⎫⎡⎤ ⎪⎪DA11−−xB1y ˆ MatchScore =+xnˆˆ22xyyn +nˆ2z −D2dxdy (21) ∫∫−−dd⎨⎬⎢⎥ ⎩⎭⎪⎪⎣⎦C1

Each time a new plane is found that will project into a particular point in the destination array, we calculate this volume and compare it to the volume calculated from the plane that is currently stored in the destination array (the distances are initialized to infinity). If the volume between the new radon plane and the plane of the destination array cell is smaller than the previously stored volume, we use the new radon plane.

3.5.2.4 Radon inversion by consecutive backprojections

Now that we have our data parameterized correctly after the resampling step we can perform the radon inversion given in (16) and repeated here for convenience.

1 ππ/2 ∂ f ()xy,,z =− R'f()ρ,θϕ, sinθdθdϕ 4πρ2 ∫∫0/−π 2∂

An efficient method to compute the 3D backprojection is to split the computation of this double integral into two steps. First an integration over all θ for the vertical planes, and next an integration over all ϕ for the horizontal planes as described in [3]. Note, since our data is now in our desired (ρ,θ,ϕ) space, both backprojection steps are independent of the sensor path during data collection and only dependent on the overall image sizes.

On each vertical plane indexed by ϕ , we get the following relation for the first backprojection step,

1 π /2 ∂ B()xzB ,,ϕ =− R'f()ρθ,,ϕ sinθdθ. 4πρ2 ∫−π /2 ∂

First we differentiate with respect to ρ , then multiply by sinθ and integrate. We step through

the output array B(xB , z,ϕ) for each plane ϕ , and sum up along all the angles θ the parallel

projections that would contribute to our current point B( xB , z) (refer to Figure 18). Since the

specific values of ρ for a particular angle θ contributing to a particular point B(xB , z) are independent of the arbitrary path taken during the data collection, these values can be pre- computed for each point B(xB , z) to generate a lookup table of all locations that need to be summed into a particular point B()xB , z.

zˆ ρ 1 ρ2 B γ θ1 ρ3 ε1

ρ4

xˆ

The final step is to backproject onto horizontal planes. This is basically the same procedure as above except now we are summing up along each plane ϕ to fill in the horizontal planes as in [3].

π f ()xy,,z = B(x,z,ϕ)dϕ (22) ∫0 B

Here again we step through the output array f (xy,,z) and compute all the perpendicular contributors to the current location. We again can perform the computation of the indices for each f (xy,,z) point off-line to help increase the speed of the algorithm. Once all the needed values have been determined for a particular point, we can sum them together and put them in the final f (xy,,z) to have a final reconstructed volume (see Figure 19). As the last step, we divide by the normalization factor ( N ) , which takes into account the missing data from the resampling step and normalizes those points that may have more data contributions than other, to give the resulting reconstructed volume f (x, y, z).

B3 B x3 ϕ 3 B Bx2 2 ϕ 2 yˆ '

Bx1 ϕ1 φ

ˆ x' Figure 19 The horizontal backprojection is basically the same as the vertical except now we perform for all the horizontal planes.

3.6 Summary

We have shown how we can fully reconstruct a 3D volume from the projected images using the 3D radon inversion formula of Grangeat along with a resampling step and backprojection. First we compute the derivative of the 2D radon transform of the pre- weighted images, which is related to the radial derivative of the 3D radon data. However, because of the parameterization of our 3D radon space, we do not have a simple implementation of the backprojection steps unless we resample to spherical coordinates with parallel projections. This allows us to use simple parallel backprojection algorithms by splitting the double integral of the radon inversion into two separate steps. We speed the algorithm up considerably by implementing each integral separately as a 2D backprojection. We can further speed the algorithm up by performing some pre-computations in the resampled space. This results in a fairly efficient method of recovering the 3D volumetric data from Fluoroscopy projections along an arbitrary source path.

Chapter 4

EXPERIMENTS AND VERIFICATION

In order to test our algorithm’s performance, we ran two categories of tests: those testing the experimental setup and those testing the quality of the tomographic reconstructions. In these experiments, we used three types of data: synthetic phantom, physical phantom, and a cadaver. The synthetic phantom was created from a synthetic 3D data set that we projected onto images using a geometry modeling that of a fluoroscope. We used the synthetic phantom primarily to test the reconstruction algorithm itself and measure the effect of various types of reconstruction algorithm errors. The physical phantom data is fluoroscopy imagery of a PCV and acrylic phantom of known structure. We used the physical phantom data to evaluate the error in the pose estimation, to analyze the consistency in our reconstructed densities, and to evaluate the algorithm’s ability to reconstruct volumes from projections collected along arbitrary paths. The cadaver data is fluoroscopy data of a cadaver knee, and was used to generate 3D reconstructions of an object comparable to what would be seen in a clinical application.

4.1 Experimental Setup

In this section we describe the setup used for the various experiments. We first describe how the two sets of synthetic phantom images were produced, and then describe the physical phantom and the physical phantom imagery in detail. Finally we describe the imagery collected of the cadaver.

4.1.1 Synthetic images of four cylinders

We used two different sets of synthetic phantom volumetric data to test our reconstruction. The first was a set of four cylinders of constant density and radius. We generated cone-beam projections of this volume along a circular path with uniform sensor spacing in ψh (every degree), and along a 121 degree arc with one degree spacing along the ψ v direction. We generated reconstructions from three different subsets of the full path. The first subset included projections along a 180 degree arc in ψ h which we refer to as the half circle path. The second included a full 360 degree rotation in ψh which we refer to as the full circle

path. The third added the perpendicular arc along the ψv direction as well (to satisfy the data sufficiency criteria – see Section A-9) which we refer to as the circle plus arc path. In addition to using subsets of the sensor path, we also used a subset of the images on the full path to study the partial volume effect. This effect is illustrated in Figure 20, where we can see that if we are imaging an object that does not fit completely into the field of view of the fluoroscope, there will in general be portions of the object that are not imaged from all sensor positions. To simulate this effect we trimmed the top and bottom of the synthetic phantom projections such that they did not include the projection of the complete cylinder. We refer to this path as the partial volume path.

Source

4.1.2 Synthetic images of ellipsoids

The second set of synthetic phantom data that we used was a set of arbitrarily placed ellipsoids of different sizes and shapes. Table 2 shows the ellipsoid parameters used to create the synthetic phantom volume. The ellipsoids are parameterized by the (x,,yz) positions of their centroids and the length of their axis along each dimension. The last column in the table indicates the intensity value that was assigned to that row’s ellipsoid. We generated synthetic projections of this volume along the same path used for the synthetic images of the four cylinders and reconstructed from the same three subsets of that path, the half circle path, the full circle path, and the circle plus arc path. We did not, however, analyze the partial volume effects with this dataset.

Table 2 List of ellipsoid parameters X-Cen Y-Cen Z-Cen X-Axis Y-Axis Z-Axis Intensity 0 0 0 48 54 48 63 0 0 0 12 18 6 42 12 -12 -18 24 30 6 105 12 -12 26.4 18 18 18 210 36 -12 0 9.6 18 18 252 36 -12 -36 6 12 18 94.5 -18 -12 24 12 18 12 126 -12 24 0 12 12 12 210 -2424241818 6 84 -3636122412 6252

4.1.3 Physical phantom description

The physical phantom we chose to use for testing was constructed out of 6” PVC pipe. It was capped on both ends and four acrylic rods with a diameter of approximately 1.3cm were placed inside parallel to the axis of the pipe and attached at only one end. We measured the maximum motion of the unattached ends of the rods due to gravity (as the physical phantom is rotated into different positions) to be approximately 3mm. We embedded metal beads in the outside of the pipe at known locations for pose estimation purposes. Figure 21 shows external and internal views of the physical phantom.

Figure 21 External (left) and Internal (right) picture of the physical phantom. Notice the beads placed along the grid on the outside in the left figure and along the inside on the right.

We generated two separate paths of the physical phantom. One path was generated specifically to measure the accuracy of our pose estimation method. We generated this data by attaching the physical phantom to a rotational stage so that very precise rotations could be generated at one-degree rotational steps.

The second path we generated with the physical phantom was completely arbitrary. The physical phantom was placed on the floor as in the left of Figure 21. It was then rotated by hand in the field of view of the fluoroscope with no care in orientation or angular spacing.

4.1.4 Cadaver image

In addition to the fluoroscopy images of the phantoms, we also collected images of a cadaver leg. These images were taken as the cadaver leg suspended from a jig mounted on a turntable. No care was taken to keep the angular spacing between images uniform, and some images were omitted upon transfer to the computer performing the reconstruction. The result

of this data collection can be considered a truly arbitrary path of continuous video data. The path is shown in Figure 22.

Figure 22 Path of the sensor relative to the cadaver shown as a 3D plot.

Our pose finding algorithm depends on knowing the 3D structure of the beads for matching purposes. For the physical phantom it was straightforward to measure the positions of the beads since they were placed on a uniform grid on the cylinder, and since the cylinder is of known diameter. The beads on the cadaver leg were placed randomly, so we performed a CT scan of the knee to determine their 3D structure. After computing the pose for each image, we used the estimated poses in an optimization of the bead locations over all the images to get a better estimate of the bead model. We performed this optimization for both the physical phantom and the cadaver datasets. We then re-ran the pose finding algorithm to obtain our final estimate of the pose for each image. It should be noted that in application to live subjects we will design a plastic frame or brace that will contain the beads in a known, rigid configuration and the CT scan to determine their structure will only be performed one time

for this frame. We could then optimize the bead locations of the rigid shell as we have done for the cadaver and physical phantom bead locations. A third alternative is to apply standard structure-from-motion algorithms from the computer vision literature to recover the pose and the structure simultaneously (these would only recover structure up to a scale factor).

4.1.5 Calibration accuracy experiments

As mentioned earlier (Section 3.2), we must perform two types of calibration. Here we describe the results of these calibration processes to demonstrate that they achieve the desired results.

4.1.5.1 Geometric calibration

As described in Section 3.2.1, our approach for removing the geometric distortion is to take a picture of a known rectangular grid of beads and estimate a 2D spatial transform for each small square sub-image that is bounded by four beads. Using standard techniques in geometric distortion removal (e.g., [25]), we can transform images to a regularly spaced grid. Figure 23 shows an image of the calibration pattern, before and after distortion removal.

(a) (b) Figure 23 Fluoroscopic image of a calibration grid before distortion removal (a) and after (b). Note the rotation and “pincushion” effects visible in (a).

4.1.5.2 Radiometric calibration

As described in Section 3.2.2, the reconstruction algorithm assumes that the value recorded at a particular pixel is the integral of the 3D density function along the ray that projects to that pixel. The actual recorded value is proportional to the energy left in the x-ray beam after attenuation caused by the 3D density function and so must be converted to a density integral value. In addition, the sensor’s sensitivity is not uniform across the field of view, but we can also correct for this effect in the same step. As discussed in Section 3.2.2, we use Marquadt’s method to find a fit to the function I = ΩeΓt + Ψ at every pixel location [26]. Figure 24 shows the fit found for a typical pixel location. From this plot we can see that the proposed model accurately describes the recorded pixel value as a function of the thickness of the aluminum calibration targets.

Figure 24 Fit relating pixel intensity value to absorption (t) .

Also discussed in Section 3.2.2 is the fact that some fluoroscopes also have automatic gain control on the video output that introduces an unknown scaling in the intensity values. We dealt with this unknown by inserting calibration spots of two different thicknesses of aluminum into the field of view (near the edges of the image).

We studied the variation in the intensity of the calibration spots as a third stack of aluminum sheets was moved from a position close to the x-ray source to a position near the image plane. This had the affect of covering more of the field of view when the aluminum was closer to the x-ray source and less of the field of view when it was further away. This caused the the automatic gain feature of the fluoroscope to vary the brightness of the imaged aluminum. We know that for all the images taken with this setup we have constant thicknesses of aluminum, so without automatic gain control we would expect to see no variation in recorded intensities. We corrected for the automatic gain by linearly transforming (scale and offset) each image depending to keep the average intensity of each calibration spot constant across all frames. Figure 25 shows the calibration spot intensity before and after the calibration. The dashed lines are before and the dotted lines are the corrected intensities. The thicker aluminum calibration spot is shown in the middle plots, and the thinner one is shown in the lower plots. The top plot is another thickness of aluminum that we can use to measure how well this method corrects for the auto gain problem. A large variation could be seen for all thicknesses of aluminum before calibration, but as seen in Figure 25, the variability was reduced by the linear transformation.

230

220

210

200 y t i s

n 190 te In 180

170

160

150 1 2 3 4 5 6 7 8 Image Number Figure 25 Auto gain correction. The dashed lines are before the correction, and the dotted lines are after. The two lower plots are the control points, and the upper plot is another constant density within the image.

4.1.6 Pose estimation accuracy experiments

Since one of the novel aspects of our approach is using the image data to determine the path of our sensor for reconstruction, and since the accuracy of the pose finding algorithm to determine the path greatly influences the quality of the reconstruction, we need to quantify the error in the pose calculation. We used two different methods to analyze our pose accuracy. The first compares estimated poses to ground truth measurements recorded with a precise rotational stage, and the second compares estimated poses to ground truth measurements collected with an optical tracker.

4.1.6.1 Ground truth from a rotational stage

The data used in this experiment were images of the physical phantom rotated to precisely measured angles about a constant vertical axis. The physical phantom was rotated at one- degree increments about this axis using a Newport 481-A rotational stage. The pose estimation software was then applied to the images and four parameters based on the estimated poses were compared with the ground truth parameter values. The results are shown in Table 3 and explained in the following paragraphs.

Table 3 Pose verification Results

Parameter Mean Max Min Units RADIUS 2.97 13 0.024 Pix RMS 1.68 2.98 0.644 Pix Orien 1.079 1.659 0.618 deg E ψ 0.106 1.23 -0.85 deg

RADIUS is a measure of how close the pose estimates are to a circle. For a 360o rotation about a fixed axis the CAS origin should make a perfect circle. The center of that circle is the center of rotation which will not in general coincide with the WAS origin. Because the physical phantom was attached to a jig and a rotational stage, we know that the CAS origin, for all the different images, will lie in a single plane. We find the plane and project the CAS origin into that plane and take the average to find the center of rotation. The radius of our circle in the plane is this average which we refer to as RAD . There are two components of possible error: the distance of the CAS origin from that plane, and the radial deviation of the CAS origin from the circle. Figure 26 will help clarify these two quantities. We report the square root of the sum of the squares of these values in Table 3. We found RAD to be 862.3 voxels.

RAD

CAS

RMS in Table 3 reflects the range of the root mean square deviation between the bead locations found in a particular image and projections of the bead model onto the image using the estimated pose parameters. This is the metric minimized by the pose algorithm during its optimization process. The errors reported in Table 3 are the mean, maximum, and minimum errors over the set of images in the sequence. If RMS error is extremely high for a particular image we can allow the algorithm to disregard that image and continue on to the next.

The third reported error is a measure of the error in the rotation component of the pose (Orien). We first compute the angle axis representation of images that are approximately 90 degrees apart. The average of all these axes will give us a vector orthogonal to our circle’s plane, or parallel to the rotation axis. The angle reported in Table 3 is the average deviation of each image pair’s rotation axis from the average rotation axis in degrees.

The final reported error, Eψ , is based on the deviation from the measured 1-degree step size around the physical phantom. Here we look at the vector RAD and compute the angle between RAD and the x -axis. We know the step size should be 1-degree, so we can look at the difference between these angles and compare them to a step size of 1-degree. The mean

absolute deviation, max, and min of the residuals from this comparison are reported in Table 3.

4.1.6.2 Ground truth from the Optotrak

The second method that we used to measure the accuracy of our pose estimation software used an Optotrak optical tracker†† to record ground truth position during the image data collection. The setup and the various coordinate systems we used are illustrated in Figure 27. We refer to the Optotrak probe coordinate system as OPS and the Optotrak base coordinate system as OBS .

Rigidly WAS Attached HOPS CAS WASi HCAS OPS Fluoroscope

OPSi HOBS

OBS

†† The Optotrak used is from Northern Digital model #3020

In order to verify the pose accuracy we need to compare the same relative pose motion in the same coordinate reference frame using two different transform paths. The paths are illustrated in Figure 28. The first path (shown on the bottom of Figure 28) takes us from the OPS coordinate system’s initial location through the Optotrak base coordinate system (OBS ) to the ith frame of OPS where the comparison can be made. The second path (shown across the top of Figure 28) takes us from the same starting reference coordinate system of OPS through the initial frame of WAS to the fluoroscope (CAS ) , which does not change during the data collection, back through the ith frame of WAS , and into the ith frame of OPS where again the comparison can be made.

Path 2

CAS

H CAS H WASi WAS0 CAS

WAS0 WASi

H WAS0 H OPSi OPS0 WASi

OPS0 OPSi OBS H OPSi OPS0 HOBS

OBS

Path 1

Figure 28 The two paths used to verify the pose finding algorithm. Path 1

(bottom) takes us from the OPS0 through OBS to OPSi where the comparison is made. The second path (top) takes us from the same starting

reference coordinate system, OPS0 , through WAS0 to CAS back through

WASi and into OPSi where the comparison is made.

The transforms between OBS and OPS can be calculated from the Optotrak. The Optotrak probes have six infrared emitters on them whose 3D positions (relative to the fixed Optotrak base) the Optotrak can sense in sequence. Based on the position of those six emitters we can calculate the pose of the probe with respect to the Optotrak base. The transforms between CAS and WAS are given from the pose estimation algorithm discussed in Section 3.3. This is the transform we wish to verify.

The transform from OPS to WAS can be found with physical measurements. Since OPS and WAS are rigidly attached as Figure 27 suggests, we are able to determine this transform by measuring the physical distances from the OPS origin to the individual beads. However, the best way to make that measurement is to use another Optotrak probe to measure each particular bead location. In order to get accurate measurements of the bead positions we need to remove the beads from the physical phantom and place the tip of the probe in the hole that would result from the bead removal. However we desired to conduct this experiment non-destructively, so approximate measurements were performed and an approximate transform was found. We then performed an optimization to this transform across all images to get a better estimate of the rigid transform. We define the transform

OPS between WAS and OPS as HWAS . Note also that since this is a rigid transform we know that

HHOPS ==OPSi HOPS0 and these notations will be used interchangeably to make consecutive WAS WASi WAS0 transform cancellations more clear. We define the transformation from the CAS to WAS at

th WASi the time of the i image as HCAS , and the transform between OBS and OPS at the time of

th OPSi WASi OPSi the i image as HOBS . Note for both HCAS and HOBS , where i = 0 refers to the initial image.

We measure our pose error by finding the change in pose of OPS relative to its initial

position, or H OPSi , in two ways, one using the Optotrak and the other using our pose OPS0 estimation algorithm, or the two paths shown in Figure 28. Then we compare the results to

measure our error. We call the result of the first method of computing H OPSi (using the OPS0

Optotrak) 1H OPSi , and readily see that it can be computed as OPS0

1HHOPSii= OPS HOPS0 −1. (23) OPS0 OBS OBS

We call the result of the second method of computing H OPSi (using fluoroscopy imagery and OPS0 our pose estimation algorithm) 2H OPSi . Given the transform H WAS (which is fixed over time OPS0 OPS since the probe is rigidly attached to the physical phantom) we see that we can compute

2H OPSi from the transforms H CAS and H WASi computed by our pose estimation algorithm as OPS0 WAS0 CAS

2HHOPSii= OPS HWASiHCAS HWAS0 . (24) OPS0WASi CAS WAS0OPS0

By comparing these two results we can analyze the error in our pose finding method. We can

compare these two paths by computing 11HOPSi− 2HOPSi. If the two transforms were equal this OPS0OPS0 would produce an identity matrix, so we convert the relative rotation from this transform to angle-axis form and use the magnitude of the angle as our orientation error measure, and the magnitude of the translation as our position error measure. The position errors are shown in Figure 29, while the orientation errors are shown in Figure 30 for 67 frames. The mean translation error is 3.37mm, with a standard deviation of 1.53. The mean error in the angular deviation is 0.023 degrees, with a standard deviation of 0.0104.

Total distance error between each method 7 ) m

(m 6 od h t e

m 5 h c a e

by 4 und o f

n i 3 g Ori n e

e 2 w t be e c

n 1 a t s i D 0 0 10 20 30 40 50 60 70 Frame # Figure 29 The Euclidean distance between the probe origin found from the pose finding algorithm to the origin found using the Optotrak method.

Theta error between each method 0.07

0.06

0.05

0.04 s e egre

D 0.03

0.02

0.01

0 0 10 20 30 40 50 60 70 Frame # Figure 30 The error in the rotation angle between two consecutive image frames for 67 frames in the sequence. This error is calculated as the rotational error from the angle-axis representation of the residuals from the two pose calculation paths.

Figure 31 shows a typical pose estimate found by our algorithm. The figure shows an image of the physical phantom, with and without circles showing the model points projected into the image at the estimated pose.

Figure 31 Sample pose projected for specific correspondences. Left: Original image, Right: Bead model projected onto the image for the estimated pose.

4.2 Tomographic Reconstruction Results

The previous section has shown that the calibration and pose estimation algorithms produce accurate results. The next step in the analysis of our method is to evaluate the results of the complete reconstruction process. This includes both subjective evaluation of the appearance of the resulting volumetric data, and quantitative evaluation of the geometric accuracy reconstructions and the accuracy of the recovered densities.

First we consider the subjective evaluation of reconstructions of two sets of synthetic phantom data, then the physical phantom data, and finally the cadaver data. Then we analyze the synthetic data for geometric accuracy. Finally we verify that the density is consistent across various parts of the reconstruction that are of the same material.

4.2.1 Synthetic phantom volume reconstruction

In order to verify the reconstruction algorithm independently of the pose finding software we reconstructed synthetic data generated from modeling the x-ray process by developing a cone-beam projection operator and designing a path for the x-ray source to traverse during the data generation as described in Section 4.1. We first reconstruct the cylinder synthetic phantom data described in Section 4.1.1 and then the ellipsoidal synthetic phantom data described in Section 4.1.2 and analyze the results for each.

4.2.1.1 Cylinder data

Figure 32 shows a cross sectional slice of the reconstructed volume near the center. The top row shows the original data which is made up of 4 cylinders as described in Section 4.1.1. The second column shows the reconstruction for the half circle path, the third column is the reconstruction for full circle path, fourth is the circle plus arc path, and finally, the fifth is the reconstruction from the partial volume path. Column (a) refers to slice 35, (b) is slice 70, (c) is slice 105, and (d) is slice 175 out of 211 slices. Notice that in the half circle path the contrast is not as strong as the other paths. The full circle path shows some artifacts in between the cylinders, but the contrast is much stronger and sharper. The circle plus arc path looks very similar to the full circle path with the addition of some horizontal streaks. This is probably due to a bias in the reconstruction resulting from the arc being imaged along that direction. The best contrasting result can be seen in the circle plus arc reconstruction. For the partial volume

72 path reconstruction we can see how the cone-beam projections clip the top and bottom of the volume as a sphere. The center of the images for (a) and (d) show a portion of the volume inside the sphere, which is inside the white ring, and the portion outside the white ring is outside the sphere. The portion outside of the sphere is outside the region that we support for reconstruction and can therefore be ignored. The artifacts are due to regions that are imaged in some of the views, but not in all. The rendered reconstructions are shown in Figure 33 with (a) showing the reconstruction of the full path, and (b) showing the partial volume reconstruction. For the rendered reconstructions, the different paths produce the same results visually, so we only show the full path reconstruction here.

4.2.1.2 Ellipsoid data

The ellipsoid model described in Section 4.1 was reconstructed from projections on the same paths as the cylinder data except that no partial volume data was processed. Slices out of the reconstructed volume can be seen in Figure 34, where (a) shows the original volume that is being reconstructed, (b) shows reconstruction along the half circle path, (c) is reconstructed along the full circle path, and (d) is reconstructed along the circle plus arc path. Looking very closely, we can see that the half circle path reconstruction has some artifacts, but the object and intensities are very clear and crisp and segmenting this would not be difficult. In (c), we see some artifacts, but they are reduced from (b) some. Finally, for the full path reconstruction in (d), we see very few artifacts and very clear crisp boundaries between the various intensities in the volume.

Figure 35 shows the rendered reconstructed volume results of the circle plus arc path. The left column of the figure shows the original volume, while the right side is the reconstructed result. There is a slight difference in the intensity values from the original data, which will be discussed in Section 4.2.4.1. The boundaries of the various ellipsoids can be seen clearly.

(a) (b) (c) (d) Figure 32 Cross sectional views of synthetic phantom for multiple paths, The top row is the original data, the second row is reconstructed from the half circle path, the third row from the full circle path, the fourth from the circle plus arc path, and the fifth from the partial volume path. Column (a) is slice 35, (b) is 70, (c) is 105, and (d) is 140.

(a) (b) Figure 33 Since the VolumePro output looks the same for all of the given paths, we only show the full path reconstruction (a) alongside the partial volume (b). These reconstructions are rendered voxel data in Amira™ using the VolumePro hardware with an appropriate alpha scale.

(a) (b) (c) (d) Figure 34 Synthetic phantom data reconstruction of ellipsoids along 3 different paths with the original data to compare. (a) shows the original data. (b) is reconstructed with only 180 degrees. (c) is reconstructed with 360 degrees. Finally, (d) is reconstructed with full path of a circle plus arc. The slice shown here is slice 70 out of 210.

4.2.2 Physical phantom reconstruction

We generated tomographic reconstructions from sensor data collected on an arbitrary path using the physical phantom described in Section 4.1. We generated arbitrary paths by moving the physical phantom by hand in front of the fluoroscopy unit. The path recovered by our pose estimation algorithm is shown in Figure 36.

-20

-40

-60

-80

-100 1000 1000 500 500 0 0 -500 -500 -1000 -1000

Figure 36 Data collection path of the physical phantom generated by the arbitrary motion.

Given the pose of the sensor relative to the physical phantom, the images were corrected, and the volume was reconstructed using our reconstruction method. Figure 37 shows three of the axial slices out of the reconstructed volume, where (a) shows slice 53, (b) is slice 105, and (c) is slice 158 out of 211 slices.

Some of the artifacts that can be seen are a result of the metal beads that were used for registration. During the data collection, the attenuation of a particular beam is the result of all the densities passed through by that beam. However if a beam is completely attenuated the intensity collected at the imaging plane will be zero and may not accurately reflect the integral of the densities it passed through. This happens when you have a metal object in the field of view (the metal beads). The correction of this effect is commonly referred to as metal artifact reduction (MAR) and has been given much attention in the literature [29-31].

Another source of error may be from the fact that the rods are not attached at one end of the PVC tube. We measured the maximum deflection of these rods due to gravity to be approximately 3mm, however during the data collection process it is possible that some other accelerative forces may have moved them more than 3mm. Although the data appears to be somewhat noisy, the various densities of the physical phantom can be identified and segmented with ease.

(a) (b) (c) Figure 37 Three slices from the physical phantom arbitrary path reconstruction. (a) is slice number 53, (b) is slice number 105, and (c) is slice number 158 our of 211 slices.

Figure 38 shows two views of the rendered reconstructed volume. The four rods are clearly visible in the right figure, and the solid PVC wall can also be seen. Some artifacts and partial volume effects are noticeable as would be expected since the object being scanned is not completely in the field of view as Section 4.1.1 discusses. Again, some of the noise seen is a result of using the metal beads.

Figure 38 Reconstruction of the physical phantom displayed here as a volume rendering from Amira™ with an appropriate alpha scale.

4.2.3 Cadaver reconstruction

Figure 39 shows the reconstructed cadaver knee using our proposed algorithm. Notice that there is some noise in the reconstruction. Some of the effects from the partial volume can be seen at the top and bottom of the reconstruction. We also notice some metal artifacts from the beads. Obviously there is still some work that needs to be done in post processing and algorithm verification to improve these initial results as described in Chapter 1.

4.2.4 Synthetic data error analysis

We used the synthetic phantom reconstruction results to analyze the accuracy of the recovered densities relative to the original volumetric data, and to measure the geometric accuracy of our reconstruction algorithm.

Figure 39 The reconstructed volume from projections generated along an arbitrary path using our proposed algorithm.

4.2.4.1 Density analysis

In order to test the consistency of our final voxel density information we can compare the final reconstructed voxel values with the density values of the original data. We fit a linear function to transform our reconstructed density information to the original density information and then look at the deviations from this fit to quantify our error. We compute the linear transformation on the ellipsoid data by segmenting the reconstructed volume, finding a mean intensity for each ellipsoid, and then finding the linear transform that fits these mean intensities to those used to construct the synthetic phantom ellipsoid volume. Figure 40 shows the linear transform with the data used to derive the linear function. The error bars represent the standard deviation of the densities in the segmented region corresponding to each ellipsoid. The mean signed deviations is 3.64 (from a range of densities from zero to 255), and the standard deviation is 2.62.

Linear Transform from Original Density Values to Reconstructed Density Values 300

250 lues a V 200

y = 0.852x + 28.673

Density 150 d e uct 100 econstr

R 50

0 0 50 100 150 200 250 300 Original Density Values Figure 40 Plot of the linear transform from the original density values to the reconstructed values. The error bars represent the standard deviation found in the segmented reconstruction volume for a particular ellipsoid.

We also would like to verify the reconstruction densities of real data as gathered from the fluoroscopy unit. We have no way of determining what the actual density is, so we instead check that for a particular material we get a consistent density throughout the reconstruction of the object, and that objects of materials (presumably with different densities) are reconstructed with different densities (e.g., to aid in segmentation). Because the materials in our physical phantom appear to be of different densities, we can segment the physical phantom data into three different regions and look at the variability of the densities for these three regions. We arbitrarily choose slice 48 for this analysis as shown in Figure 41a. The three regions we look at are the wall of the physical phantom, constructed out of PVC, the 4 acrylic rods, and the air that makes up the rest of the area in the image. Figure 41b and c show the hand segmentations used. A circle was fit to the apparent position of the PVC wall, and circles were fit to the acrylic rods for segmentation. We then took the remaining area and considered that to be the air.

(a) (b) (c) Figure 41 Slice 48 out of the reconstructed physical phantom from an arbitrary path is shown in (a). The acrylic rods segmented are shown in (b). The PCV wall segmented is shown in (c). The air was taken to be everything else.

Table 4 Mean and standard deviations of the segmented reconstructed physical phantom. Mean STD PVC 143.31 17.69 Acrylic 122.96 11.23 Air 93.17 13.62

Table 4 shows the results of this analysis. Each of the three materials are given along with the average intensity found in the image and the standard deviation of that intensity. Since we know that each of the materials is of constant density, we would expect the reconstructed densities to be of constant density as well. The errors shown in Table 4 are rather high for this assumption showing around 7% error for the PVC, 4.4% for the Acrylic, and 5.3% for the air. Some probable causes for this are the accuracy of the segmentation, the various artifacts from the beads, and the relatively small regions being tested (they suffer from the blurring effects and low pass filtering effects of the algorithm). One thing worth noting is the high contrast between the various materials demonstrated by the mean intensity for each material.

Table 5 The differences found in the reconstructed ellipsoids vs. the original data. All data given in this table is in pixels. X-Cen Y-Cen Z-Cen X-Axis Y-Axis Z-Axis 10 000 0 1 20 000-20 3-1 0 0-1-1 0 40 0-0.4-2-21 5 0 0 0 -1.6 -2 1 6 0 1 0 -2 -2 0 7 0 0 0 -1 -1 1 8 0 0 0 -2 -2 1 91-10-2-20 10 0.5 0 0 -4.5 -3 1 Mean 0.25 0.20 0.04 1.61 1.70 0.60 STD 0.50 0.47 0.13 1.29 0.82 0.52

4.2.4.2 Geometric accuracy analysis

In order to determine whether the reconstruction algorithm introduced any geometric errors (e.g., position offsets and scaling), we compared the position and size of each reconstructed ellipsoid of the ellipsoid data set to those of the corresponding ellipsoid in the original volume. We first found the three axes of each reconstructed ellipsoid by measuring the extremities of the segmented ellipsoid in the x , y , and z -directions. We then found the center of the ellipsoid by averaging the axis endpoints. Table 5 summarizes the results. The errors in the axis lengths are larger than the error in the position and are probably due to the blurring effect of the algorithm (accurate segmentation is very difficult with blurred edges). We also notice the increase in the axis lengths along the z -direction and the decrease in the x and y -direction, which is possibly caused in part due to less than perfect segmentation. The actual position of the ellipsoids was very reliable. Section 6.9 discusses some methods that could be applied to help sharpen the results.

4.3 Summary of Results

In order to demonstrate our method and approach, we have performed various analysis of our approach, and have as well reconstructed multiple datasets, including two sets of synthetic phantom data, physical phantom data generated along an arbitrary path, and cadaver data generated along an arbitrary path. We analyzed the various steps of our algorithm starting with an analysis of the radiometric calibration procedures. We looked at the radiometric calibration procedures and demonstrated the relationship between the densities passed by the x-ray and the recorded image intensities. We then looked at the removal of the auto gain problems by performing a linear transformation of the pixel intensities based on the calibration spots in the images. Next we verified our approach to finding an accurate pose of our model with respect to the x-ray system through two different methods. The first method used the pose finding algorithm only and made some assumptions about the path that the physical phantom would take during a particular data collection. In the second method we used the Optotrak optical tracker as our ground truth and measured deviations between our estimated pose and the pose from the Optotrak. To verify the reconstruction process we have looked at geometric distortions both visually and analytically using synthetically generated projections. We have also verified the densities of the reconstructed volumes using synthetic data. We have verified full arbitrary path reconstructions of a physical phantom, and a cadaver leg. While the cadaver leg has been verified only visually, we have made an attempt to segment the physical phantom and verify the consistencies of the reconstructed volume.

Chapter 5

CONCLUSIONS

This thesis presents preliminary results of a method of tomographically reconstructing 3D volumetric data from fluoroscopic imagery using pose estimates computed from the fluoroscopy image data. Our method differs from earlier applications of reconstruction from cone-beam projections in that it does not require a precisely instrumented data collection unit to provide the sensor trajectory information, nor does it require any external pose measurements that would have to be synchronized with the data collection. Instead, we extract the necessary pose information from the markers detected in the cone-beam images themselves.

The results from this work will provide an inexpensive and readily available 3D imaging capability that can benefit many medical applications, but we expect it to particularly increase the use of computer aided orthopedic surgery (CAOS). Advances in visualization and localization will allow minimally invasive procedures to be used, thus reducing the trauma to the patient. It is expected that long term health care costs will be reduced because of the high efficiency of computer-aided procedures, and from the shorter hospitalization and recovery times that will be required [32]. CAOS has been shown to improve outcomes and reduce patient trauma, but these benefits depend on increasing the accuracy and reducing the invasiveness of the system. The methods that use patient-specific models and intra-operative 3D registration provide the highest accuracy. The surgeon can plan the optimal strategy using patient-specific computer models of anatomy (typically, bones) and then, during surgery, a navigation system can guide the surgeon in completing the planned strategy. Our system provides an inexpensive alternative to CT and MRI for the creation of patient-specific 3D orthopedic models in any facility with a fluoroscope. This approach will also provide the ability

86 to perform reconstructions in the operating room, emergency room, or field hospitals where otherwise obtaining 3D data may be impractical or impossible.

We have shown that our approach using fluoroscopy data gathered along an arbitrary path can be used to gather 3D models of the data required in many medical procedures. We have demonstrated this approach by first detailing the calibration procedures necessary for the gathered data to be appropriate for the reconstruction process. We then demonstrated how the pose estimation from the fluoroscopic imagery can be used to gather information about the path of the x-ray during the data collection, which is used for the rebinning process of the reconstruction algorithm. We then applied Grangeat’s method of exact cone-beam reconstruction demonstrating accurate reconstructions from synthetic phantom data, real imagery of a physical phantom with controlled rotation, real imagery of the physical phantom with a random path, and real imagery of a cadaver. We tested the reconstruction algorithm by first processing the synthetic data. We tested the pose estimation using the physical phantom and verified the accuracy of our approach through two separate methods. Finally, we tested the combined overall process by reconstructing the physical phantom as well as a cadaver knee.

We have demonstrated that our process produces accurate 3D models. We have judged the success of our reconstructions by analyzing the densities found in the reconstructed volume for both the synthetic phantom data as well as the physical phantom data. These judgments have been based on verifying the relative positions of objects in the reconstructed volume for the synthetic phantom data, analyzing reconstructions from data generated along various paths as well as completely arbitrary paths, and visually analyzing the reconstructed cadaver knee. From these experiments and analysis we have concluded that our approach is a valid method to generate CT-like data from a fluoroscopy machine. However, the method is still in an infantile stage and there is much work and research that still needs to be done before it is a deliverable product, some of which is outlined in the following chapter.

Chapter 6

FUTURE WORK

It is the goal of this thesis to demonstrate an initial method for using an existing fluoroscope as a CT machine. There are other advances needed to improve the process and streamline it for commercial application. Some of these advancements are detailed below.

6.1 Epipolar Geometry Included to Remove the Beads

One could in principal determine the epipolar geometry of a scene by matching radon data around proposed epipolar points. Given the epipolar geometry one can recover the relative pose of the sensors in the two views (up to a global unknown scale factor). More testing and development would be required to use this method on a real and noisy dataset, while considering things such as partial volume effects, efficient search methods, and sensitivity issues.

6.2 Reconstructions from Missing Data

There are various types of missing data that can affect the quality of the reconstructions. Since high density objects such as metal cause the x-rays to be attenuated beyond recovery, the metal in the scanning region may cause incomplete data. There is also the possibility that the images collected do not form a complete projection data set (see Section A-9 for a definition of completeness).

6.2.1 Metal artifact reduction

Metal Artifact Reduction (MAR) is a method for removing the deleterious effects of metal objects in the reconstructed images. MAR may be a concern to us because the use of metal beads for our registration process may produce metal artifacts. Even if the epipolar plane matching approach is successfull and we don’t need the beads to determine pose, we may still need to consider MAR for patients with metal implants. There are many methods for reducing metal artifacts in CT [30, 31, 33] and the inclusion of these methods in our approach will produce higher quality results.

6.2.2 Incomplete datasets

Analysis of the effects of missing data has not been performed in this thesis and is left to the future work section. We would like to see the effects of incomplete data of many forms. The arbitrary image collection path may not sufficiently cover the 3D radon domain, which can cause significant degradation in the reconstructions. We also would like to study the effect of partial scans and analyze the degradation of the final reconstruction in relationship to the amount of data collected so that we may reconstruct the best 3D volume with the smallest amount of data to process as possible. The partial volume effects that we analyzed in Section 4.2.1.1 are an example of missing data from a partial scan because the data in the partial volume region is scanned from some of the views, but not all. We would also like to see the effects of slower frame rates (larger angular spacing) in capturing the video data from the fluoroscopy machine. Decreasing the sampling rate of the video by a factor of two may have negligible effect on the quality of the results while decreasing the data processing requirements by a factor of two. Certain types of missing data, such as the lower sampling of the video data, may be corrected for by interpolation with minimal effects on the reconstruction quality.

6.3 Reconstruction of Other Data Sets

Testing on other datasets is necessary to further characterize our approach. For example, another cadaver experiment would provide valuable feedback of how our algorithm will perform on live subjects. We would also like to process other datasets of small models and phantoms to provide further methods and datasets to analyze our algorithm.

6.4 Other Methods of Reconstruction

There are conceivably other approaches to the reconstruction process. In our approach we are summing along planes to model the x-ray transform and relate it to the radon transform. However, the x-ray transform could also be modeled as a summation along rays and inverted directly. Furthermore, the reconstruction directly from the x-ray data is another area of interesting research.

6.5 Reconstruction on Other Fluoroscopy Machines

There are a multitude of fluoroscopy machines of various types in existence. They vary in focal length, the cone (or view) angle, and the mobility/construction of the unit. We would like to test this method on other machines and verify our calibration processes and reconstruction methods on other machines to verify our results. For example, we do not consider changes in the source-to-sensor geometry of the fluoroscope during our pose estimation procedure. This is because our fluoroscopy machine is completely rigid and stationary during the data collection while the patient rotates in front of the collection unit. However there are some fluoroscopy machines that will rotate and move around the patient with a gliding C-arm. In this case, small changes in the source-to-sensor geometry during the rotation may have a large effect on our accuracy in the pose estimation. This is a known problem and there are ways of

90 dealing with this during the data collection such as [13] but we have not addressed them here since they do not apply to the fluoroscope that we are currently using.

6.6 Move All Code to C/C++ and Package

The code is currently in both MATLAB® and C. As a final commercial product this application must be developed into a software suite for collection and processing of the data real time both for inter-operative and pre-operative imaging studies. Ideally, the software would be modified to provide feedback to the technician as the data is being gathered so that he would know when a complete data set has been collected (see Section A-9).

6.7 Algorithm Speed-up (Hardware and Software)

Much research has been done on speeding up the computationally expensive reconstruction process. This includes Fourier techniques [2, 5, 12] for the reconstruction, algebraic techniques [31], and hardware implementation [21], all of which could be used to speed up our algorithm. So far the only speed-ups that we have implemented have been more efficient ways of searching our space, better coding in MATLAB®, and some pre-computing to cut down on the runtime computations.

6.8 Noise Analysis

Although the processing is spatially variant, it is a linear process and would lend itself to a fairly straightforward analysis of the effects of image noise. Some of the other errors mentioned in this section as well as in Chapter 1 might also be analyzed analytically. There is literature on the sources of error and propagation of error in computed tomography; however their application to our problem needs to be explored in more detail.

6.9 Resolution Improvements

Resolution limits in our data source and limitations of our algorithms reduce the resolution of the reconstructed 3D data and may also cause artifacts in the reconstruction. There are various areas of resolution in our process and any/all of them could be researched further to determine the effects each has on the final reconstructed volume.

6.9.1 Resolution of the fluoroscope

If the data were collected on a higher resolution fluoroscopy machine the reconstruction artifacts may be reduced and resolution in the final volume would probably be improved. However one of the main goals of this project is to convert existing fluoroscopy machines, so we will in general not be able to control the resolution at which we are collecting the data. A future area of work and research however would be analysis of the algorithm to determine the effects of sensor resolution on such errors and to determine a method of correction or compensation.

6.9.2 Virtual detector plane

There are also a number of other resolution issues in the actual processing of our data. Since we transform and scale our image size down to the virtual detector plane size (as in Figure 11), we lose resolution proportional to this scale factor. One method to avoid this would be to include the initial transformation and scaling of the images in the interpolation of the 3D radon resampling process which would have the added benefit of replacing two interpolations with one. This should reduce the blurring of the result..

6.9.3 Radon resolution

Another issue affecting the resolution of the output is the resolution at which we store the 3D radon data internally to the algorithm. Currently we take the radon transform of each image in our dataset at one-degree steps, and the radial component at one-pixel step sizes. We can reduce both the angular component as well as the radial component independently to analyze each of their effects on the final output.

6.9.4 Output array resolution

We could also increase the output array resolution to produce higher final resolution datasets. During the resampling step we use angle step sizes in θ and ϕ of one-degree, and again we use the radial component ρ at one-pixel step sizes, all three of which could be reduced.

We did not implement any of these resolution improvement methods due partially to the data size required for higher resolution. For large datasets and large volume reconstruction sizes, the memory required on a typical computer increases very quickly past the physical memory, and therefore the system slows down to unacceptable speeds once paging begins to occur. We can avoid this problem by either processing partial parts of the volume at a time, or develop better data handling systems to more efficiently store the data.

REFERENCES

1. Siemens, Siremobil Iso-C 3D. 2002.

2. Axelsson, C. and P.E. Danielsson, Three-dimensional reconstruction from cone-beam data in O(N^3log N) time. Physics in Medicine and Biology, 1994. 39(3): p. 477.

3. Grangeat, P., Mathematical Framework of Cone Beam 3D Reconstruction via the First Derivative of the Radon Transform, in Mathematical Methods in Tomography. 1992: Springer Verlag. p. 66.

4. Feldkamp, L.A., L.C. Davis, and J.W. Kress, Practical cone-beam algorithm. Journal of the Optical Society of America A: Optics and Image Science, 1984. 1(6): p. 612-619.

5. Jacobson, C., Fourier Methods in 3D-Reconstruction from Cone-Beam Data. Linkoping Studies in Science and Technologry Dissertations, 1996. 427.

6. Kudo, H. and T. Saito, Derivation and implementation of a cone-beam reconstruction algorithm for nonplanar orbits. IEEE Transactions on Medical Imaging, 1994. 13(1): p. 196-211.

7. Mueller, K., R. Yagel, and J.J. Wheller, Fast implementations of algebraic methods for three- dimensional reconstruction from cone-beam data. IEEE Trans. on Medical Imaging, 1999. 18(6): p. 538-548.

8. Noo, F., M. Defrise, and R. Clack, Direct reconstruction of cone-beam data acquired with a vertex path containing a circle. IEEE Trans. on Image Processing, 1998. 7(6): p. 854-867.

9. Schaller, S., T. Flohr, and P. Steffen, Efficient Fourier method for 3-D radon inversion in exact cone-beam CT reconstruction. IEEE Transactions on Medical Imaging, 1998. 17(2): p. 244- 250.

10. Tuy, H.K., INVERSION FORMULA FOR CONE-BEAM RECONSTRUCTION. SIAM Journal on Applied Mathematics, 1983. 43(3): p. 546-552.

11. Wang, X. and R. Ning, Cone-beam reconstruction algorithm for circle-plus-arc data-acquisition geometry. IEEE Transactions on Medical Imaging, 1999. 18(9): p. 815-824.

12. Xiao, S., Y. Bresler, and D.C. Munson, O(N^2logN) Native Fan-Beam Tomographic Reconstruction. IEEE, 2002.

13. Mitschke, M. and N. Navab, Recovering the X-ray projection geometry for three-dimensional tomographic reconstruction with additional sensors: Attached camera versus external navigation system. Medical Image Analysis, 2003. 7: p. 65-78.

14. Horn, B.K.P., Closed-form solution of absolute orientation using unit quaternions. Journal Optical Soc. of America, 1987. 4(4): p. 629-642.

15. Kall, B.K., Comprehensive multimodality surgical planning and interactive neurosurgery, in Computers in Stereotactic Neruosurgery, P.J. Kelly and B.K. Kall, Editors. 1992, Blackwell Scientific: Boston. p. 209-229.

16. Grimson, W.E.L., Object recognition by Computer. 1990, Cambridge, Massachusetts: MIT Press.

17. Penney, G., et al., A comparison of similarity measures for use in 2D-3D medical image registration. IEEE Trans. on Medical Imaging, 1998. 1496: p. 586-595.

18. Turbell, H. and D. Per-Erik, Helical cone-beam tomography. International Journal of Imaging Systems and Technology, 2000. 11(1): p. 91-100.

19. Oppenheim, A.V. and R.W. Schafer, Digital Signal Processing. 1975: Prentice-Hall.

20. Mueller, K., R. Yagel, and J.J. Wheller, Anti-aliased three-dimensional cone-beam reconstruction of low-constrast objects with algebraic methods. IEEE Trans. on Medical Imaging, 1999. 18(6): p. 519-537.

21. Mueller, K. and R. Yagel, On the use of graphics hardware to accelerate algebraic reconstruction methods. Proceedings of SPIE - The International Society for Optical Engineering, 1999. 3659(II): p. 615-625.

22. Hsieh, J., Reconstruction Algorithm for Single Circular Orbit Cone Beam Scans. IEEE, 2002: p. 836-838.

23. Kudo, H., et al., Performance of Quasi-Exact Cone-Beam Filtered Backprojection Algorithm for Axially Truncated Helical Data. IEEE Transactions on Nuclear Science, 1999. 46(3): p. 608-617.

24. Banks, S.A. and W.A. Haodge, Accurate measurement of three-dimensional knee replacement kinematics using single-plane fluoroscopy. IEEE Transactions on Biomedical Engineering, 1996. 43(6): p. 638-649.

25. Pratt, W.K., Digital Image Processing. 2nd ed. 1991, New York: Wiley & Sons.

26. Bevington, P.R. and K.D. Robinson, Data Reduction and Error Analysis for the Physical Sciences. 1992, Boston, Massachusetts: McGraw-Hill.

27. Haralick, R. and L. Shapiro, Computer and Robot Vision. 1993: Addison-Wesley Inc.

28. Hartley, R. and A. Zisserman, Multiple View Geometry in Computer Vision. 2001: Cambridge University Press.

29. Wang, G., et al., Iterative Deblurring for CT Metal Artifact Reduction. IEEE Transactions on Medical Imaging, 1996. 15(5): p. 657-664.

30. Zhao, S., et al., X-Ray CT Metal Artifact Reduction Using Wavelets: An Application for Imaging Total Hip Prostheses. IEEE Transactions on Medical Imaging, 2000. 19(12).

31. Benac, J., Alternating Minimization Algorithms for Metal Artifact Reduction in Transmission Tomography., in Electrical Engineering. 2002, Washington University: St. Louis, Missouri.

32. Jolesz, F., Kikinis, and F. Shtern, The Vision of Image-Guided Computer Surgery. The High Tech Operating Room in Computer Integrated Surgery - Technology and Clinical Applications, ed. e.a. R. Taylor. 1996, Cambridge, Massachusetts: MIT Press. 717-721.

33. Murphy, R.J., et al., Incorporating known information into image reconstruction algorithms for transmission tomography. Proceedings of SPIE - The International Society for Optical Engineering, 2002. 46841: p. 29-37.

34. Jain, A.K., Fundamentals of Digital Image Processing. 1986: Prentice Hall.

Appendix A

A-1 2D Radon Transform Summary

The 2D radon transform of a function fx( ,y), denoted as Rf ( A,α ), is defined as a set of line integrals along a line at a distance A from the origin whose normal is inclined at an angle α from the x -axis (see Figure 42) [34]. The mathematical equation is

∞ R()Af,αα=−(Acos ssinα, Asinα+scosα)ds ∫−∞ where (A, s) is a rotated coordinate system inclined at an angle α from the x -axis. We can also write this in xy -coordinates with a delta function as

∞∞ Rf ()A,,αδ=f (x y)(x cosα+y sinα−A)dxdy . (25) ∫∫−∞ −∞

yˆ

A α xˆ

Figure 42 Pictorial description of radon transform

A-1.1 Sample implementation

In this section we give an example of a (relatively inefficient) method to compute the 2D radon transform that illustrates the radon transform The pseudo-code for the operation is given below.

for all desired A's for all desired α 's for all x 's across the image calculate the corresponding y's I _,out ()A α +=I ( x,y); } } }

This code demonstrates a way to implement the 2D radon transform of a discrete image. It iterates through all the output points and adds into that location all the values in the image along the line specified by the given (A,α). In general, the image locations along this line will not fall directly into a particular bin in the image. Therefore some type of interpolation must be used to choose the value to be used in the summation. A fairly efficient implementation of the 2D radon transform with interpolation is MATLAB®’s RADON command, which is described in Section A-1.2.

A-1.2 Example of the radon transform using MATLAB®

Here we demonstrate the 2D radon transform of a simple image in which there are five non-zero points located at the positions given in the following table.

Table 6 Image points Point u v pix val 155 1 21020 4 32530 2 48403 52540 1

The image is of size 50x50, and (u,v) are the image coordinates with the origin at the upper left hand corner of the image, and the pixel value is the intensity assigned to that particular point. The intensities are normalized in the following images for better visualization. The image is shown in Figure 43.

1 4 2

3 5

Figure 43 Image with non-zero pixels as specified in Table 6.

100

The 2D radon transform of this image is computed in MATLAB® using the command radon(I,[0:179]); to produce the image shown in Figure 44. Note that the second argument to the radon function give the angles at α = (0,1,2,3,...,178,179) at which the transform is computed. Figure 44 includes labels for the curves that correspond to the labels of the points in Figure 43 that generate the curves.

1 5 2 3

Figure 44 Radon transform of Figure 43 with the radon lines labeled.

Each input image point will be transformed to a sine wave in the radon domain. The intensity values of each sine wave are proportional to the input intensities of the corresponding input image points. The frequency of each sine wave is the same; it depends on the number of angles and the angle spacing the transform is computed at. The fraction of the period of the sine wave that is shown is dependent on the range that the angles cover. Finally, the amplitude and phase of the sine wave is dependent on the original position of the points in the image. Wave 1 has the largest amplitude because the corresponding point 1 is the furthest from the image center. Wave 2 has the highest intensity because point 2 has the highest intensity.

101

A-2 Example of Reconstruction from Projections in MATLAB®.

The goal of this section is to give a basic overview of the process for computing the 2D radon transform and inverse radon transform. The image we use is the Shepp-Logan Phantom from MATLAB®, which is shown in Figure 45.

Slice = phantom('modified shepp-logan'); figure,imshow(Slice)

Figure 45 MATLAB®’s phantom image.

The iradon function in MATLAB® can be used to reconstruct the image again. An example of such a reconstruction is shown in Figure 46. This reconstruction is the inverse radon transform of the 2D radon transform of Figure 45 computed at only four angles.

102

Figure 46 Reconstruction of phantom using only 4 different projection angles.

One can see that this produces a very poor reconstruction of the image because the angular spacing is extremely coarse. If we perform the same computation with 18 projection angles (every 10-degrees) we obtain the result shown in Figure 47.

Figure 47 Reconstruction of the phantom using 10 degree angle steps

103

This produces a better reconstruction of the original image shown in Figure 45, however it is still quite distorted, so we next consider the reconstruction obtained from a radon transform computed at 120 angles (1.5 degree increments).

Figure 48 Phantom reconstructed using 1.5 degree spacing

This yields a much better result and illustrates the possible effects coarse angular spacing can have on reconstructions.

A-3 Parallel and Fan Beam Reconstruction Overview

Most projection data is actually not acquired with a parallel projection scheme, but with a fan-beam geometry. The reconstruction cannot be performed on this data directly by the inverse radon transform, so a resampling step is necessary in order to resample the fan beam data to parallel beams. Since every ray in a fan beam data corresponds to a ray in a parallel set of rays, this conversion can be easily completed. Figure 49 demonstrates this idea pictorially. The fan beam data generated by Source 1 contains a ray that is parallel to the principle ray in

104 the Source 2 image. More generally stated, each source will have a ray that belongs to a parallel set of rays, and if the parallel set of rays are grouped together (or resampled) then parallel backprojection can be performed. This is the 2D example of what we must do in 3D to get from the cone beam projections to the parallel 3D projections.

Source 1 vˆ

D A s Source 2 α β uˆ

The equations that relate the fan beam projection to parallel projection and control the resampling process are (see Figure 1)

105

sD A = AD22+ , (26) −1 ⎛⎞A αβ=+cos ⎜⎟ ⎝⎠s

where s is the distance from the origin in sensor plane one which provides parallel data for sensor plane two, A is the perpendicular distance corresponding to the radon transform, D is the distance between the origin and the x-ray source, and α and β are the angles that each sensor plane makes with the uˆ -axis. α also corresponds to the angular component of the radon transform. The implementation of the reconstruction is straightforward using MATLAB®’s iradon function so the heart of this problem becomes the resampling step. MATLAB® provides a convenient resampling function, interp2, which can be used to interpolate the parallel-beam projections from the fan-beam projection. Then we can then use the iradon function to reconstruct the original image.

A-4 2D Fourier Slice Theorem

The 2D Fourier Slice Theorem states that the Fourier transform of a parallel projection of an object fx(,y) obtained at angle α equals a line in a two-dimensional Fourier transform of f (x, y) taken at the same angle [34]. In order to prove this, we start with the projection along lines parallel to the y -axis. The projection, p(x,0) is related to f (x, y) by

∞ px,0 = fx,y dy. ()∫−∞ ()

Taking the Fourier transform with respect to x on both sides gives

∞∞∞ −ju2πx −j2πux Po ()u==p(x,0)edx f(x,y)edxdy. (27) 0 ∫∫−∞ −∞ ∫−∞

106

Now, we consider the 2D Fourier transform of the original function f (x, y) evaluated at v = 0 ,

∞∞ F ()u,0 = ⎡⎤f (x,y)e−+ju2π ()xvydxdy ⎢∫∫−∞ −∞ ⎥ ⎣⎦v=0 (28) ∞∞ = f ()x,.y e− ju2π xdxdy ∫∫−∞ −∞

Comparing the right hand sides of (27) and (28), we conclude that the two equations are identical. This states that the Fourier transform of the 0o projection of an object is the same as the v = 0 line in the two-dimensional Fourier transform of the same object. Since the coordinate system is chosen arbitrarily, this conclusion is valid in any rotated coordinate system [34], or to state mathematically,

Puθ ( ) = Fr (u,θ ) , (29) where the subscript r denotes the radial form of F .

A-5 3D Radon Data from Projection Data

For a 2D image, radon data is the set of integrals along a complete set of lines in an image. As described in A-1, it is parameterized by the normal A to the projection line and the angle α , that the normal to the line makes with the x -axis (Figure 42). Mathematically we can state this as in Equation (25) repeated here for convenience.

∞∞ Rf ()A,,αδ=+f (x y)(x cosαy sinα−A)dxdy ∫∫−∞ −∞

There also exists a 3D radon transform, which instead of integration along lines in an image, involves integration along planes in a volume. The 3D radon transform can be written as

∞ ∞ ∞ Rf ρnˆ = fx,y,z δ x sinθ cosϕ + y cosθ + zsinθ sinϕ − ρ dxdydz, (30) ()∫−∞ ∫−∞ ∫−∞ ()()

107

where fx(,y,z) is the data being projected, and the delta function selects an integration plane parameterized by its distance from the origin ρ , and the orientation of a unit vector normal to the integration plane nˆ . The orientation of nˆ is parameterized by two angles, θ and ϕ , where θ is declination angle measured from the kˆ -axis and ϕ is measured from the

projection that nˆ makes into the ijˆˆ-plane with the iˆ -axis, see Figure 50.

k ρ

θ nˆ

Figure 50 The integration plane is defined by the distance from the origin (ρ ), and the orientation of a unit vector, nˆ , parameterized by θ and ϕ .

Thus, the 3D radon transform is the integral of f (•) on all 3D planes that pass through the object. Since our x-ray source can be modeled as a projection operator integrating along lines, when we take the 2D radon transform of these images, we are actually integrating over planes in 3D, giving us the 3D radon plane data‡‡.

‡‡ Note that this is slightly oversimplified. See Section 3.5.2.1 for details.

108

A-6 3D Projection Slice Theorem

The 3D projection slice theorem states that the one-dimensional Fourier transform along

∞ nˆ of the 3D radon transform, Rf(ρ,nˆ), (defined as RFn()ε,ˆ= Rf(ρρ,nˆ)e− j2περ d) is ∫−∞

the central slice Fr (ε,nˆ) in the direction nˆ of the 3D Fourier transform of f ()x, y, z .

The first step to show this is to write the equation for the 3D radon transform,

∞∞∞ Rf ()ρ,θ,ϕ=f (x, y, z)δ(x cosϕθsin +y sinϕθsin +z cosθ−ρ)dxdydz . ∫∫∫−∞ −∞ −∞

Now we define a rotated coordinate system such that one of the axes of our new coordinate system (say ρˆ ' ) aligns with the perpendicular to our radon plane (or nˆ ). Also note that since this transformation is a rotation between two Cartesian coordinate systems, the Jacobian is unity and no scale factor is needed to integrate the transformed system.

⎡⎤ρϕ'c⎡ sθsϕsθcθ⎤⎡x⎤ ⎢⎥⎢ ⎥⎢⎥ ⎢⎥my=−⎢ scϕϕ0⎥⎢⎥, ⎣⎦⎢⎥qz⎣⎢−ccϕθ sϕθc sθ⎦⎥⎣⎢⎦⎥

where c and s are the cosine and sine functions respectively. Notice also, that we can invert this transform and get expressions for x , y, and z by simply taking the transpose. This

allows us to define fnˆ ()ρ ', mq, as f (x, y, z) rotated into this new coordinate system which is defined by nˆ . Now the radon transform becomes

∞∞∞ Rff()ρθ,,ϕ =ˆ (ρ',m,q)δ(ρ'−ρ)dρ'dmdq. (31) ∫∫∫−∞ −∞ −∞ n

By the sifting property we have,

∞∞ Rf ()ρθ,,ϕ = f ˆ (ρ,m,q)dmdq . ∫∫−∞ −∞ n

Now if we take the 1D Fourier transform of Rf along ρ , we will have

109

∞ RF ()εθ,,ϕ = Rf (ρ,,θϕ)e− j2πρε d ρ ∫−∞

∞∞∞ − j2πρε = fnˆ ()ρρ,,m q e d dmdq. ∫∫∫−∞ −∞ −∞

However, this is just the 3D Fourier transform of fnˆ evaluated along the direction of nˆ . This becomes clear when we transform back to the un-rotated coordinate system.

∞∞∞ RF ()εθ,,ϕ = f (x,y,z)e−+jx2πε ()cosϕ sinθ ysinϕ sinθ +zcosθ dxdydz ∫∫∫−∞ −∞ −∞ = F (εϕcos sinθ,εsinϕcosθ,εcosθ) (32)

= Fr ()εθ,,ϕ.

Now we can prove the radon inversion formula which is given in the following section.

A-7 Proof of 3D Radon Inversion Formula

We would like to give a simple proof of the 3D radon inversion formula,

1 2ππ∂ f ()xy,,z =− R'f()ρ,θϕ, sinθdθdϕ. (33) 4πρ2 ∫∫00∂

We can prove the radon inversion formula as follows. First we need to write the 3D inverse Fourier transform of f ()x, y, z ,

∞ ∞ ∞ fx,y,z = F ε ,ε ,ε e j 2π ()xε1 +yε 2 +zε 3 dε dε dε . ()∫−∞ ∫−∞ ∫−∞ ()1 2 3 1 2 3

Expressing this in polar form in the frequency domain we have

ε1 = εcosϕ sinθ

ε2 = εsinϕ sinθ ,

ε3 = εcosθ or,

110

ππ/2 ∞ 2 jx2πε()cosϕθsin ++ysinϕθsin zsinθ f ()xy,,z = ε sinθεFp (),θ,ϕe dεdθdϕ ∫∫0/−−π 2∫∞() ππ/2 ∞ = εθ22sin RF ()ε,θ,ϕe j περ dεdθdϕ ∫∫0/−−π 2∫∞() ππ/2 = Id()ρθ,,ϕ sinθ θdϕ ∫∫0/−π 2

∞ where I ()ρ,,θϕ = ε2RF (ε,,θϕ)e j2περ dε is the inverse Fourier transform of ∫−∞ RF (ε,,θ ϕ ) weighted by ε2, and we have made use of (32). Note that by the Fourier derivative theorem, I(ρ,θ,ϕ) is just the second derivative of Rf (ρ,,θ ϕ ) with respect to ρ,

1 ∂2 IR()ρ,,θϕ = f()ρ,,θϕ ()j2π 2 ∂ρ 2

1 ∂2 =− Rf ()ρ,,θϕ. 4πρ22∂

So now we have

2 1 ππ/2 ∂ f ()xy,,z =− Rf()ρ,θϕ, sinθdθdϕ 4πρ22∫∫0/−π 2∂

1 ππ/2 ∂ =− R ',fd()ρ θϕ,sinθ θdϕ. 4πρ2 ∫∫0/−π 2∂

This states that we can reproduce the original function fx( ,y,z) from the first derivative of the radon transform, R' f ()ρ,θ,ϕ .

A-8 Backprojection

In our algorithm we use two 2D backprojection steps in the radon inversion process. This backprojection step was originally used in a 2D tomographic reconstruction algorithm called filtered backprojection. In this section we describe 2D filtered backprojection, which recovers

111

the original image data fx(,y) from its 2D radon transform Rf . First we write the definition of the inverse Fourier transform,

∞ ∞ fx,y = Fu,v e j 2π ()ux+vy dudv . (34) ()∫−∞ ∫−∞ ()

If we exchange the rectangular coordinate system in the frequency domain, (u,v), for a polar coordinate system, (w,θ ), by making the substitutions

u = wcosθ v = wsinθ dudv = wdwdθ

we can write the inverse Fourier transform as

2π ∞ fx(),,y= F(wθθ)ejw2cπθ()xos+ ysinθwdwd ∫∫00p π ∞ = Fw(),θθejw2cπθ()xos+ ysinθwdwd ∫∫00 p π ∞ ++Fw(),θπejw2cπθ()xos()++πysin()θ+πwdwdθ ∫∫00 p π ∞ = Fw(),θθejw2cπθ()xos+ ysinθwdwd ∫∫00 p π 0 +−Fw(),θ +πθejw2cπθ(−−)()xos()−ysin()θ()−wdwd ∫∫0 −∞ p

and using the fact that Fwp()−+,θ π=Fp(w,θ) we can write

π ⎡ ∞ jw2cπθ()xos+ ysinθ⎤ f ()xy,,= Fp (wθ )we dwdθ ∫∫0 ⎣⎢ −∞ ⎦⎥

π ∞ ⎡⎤jw2π t = Fwp (),θθwe dwd ∫∫0 ⎣⎦⎢⎥−∞

where tx=cosθ +ysinθ . Using the Fourier slice theorem (Section A-4), we can substitute

the Fourier transform of the projection at angle θ , Pθ (w) , defined in (29) in Section A-4, for

the two-dimensional Fourier transform Fwp ( ,θ ) , we get

112

π ∞ f ()xy, = ⎡ P(w)wejw2π tdw⎤ dθ . (35) ∫∫0 ⎣⎢ −∞ θ ⎦⎥

This integral may be expressed in two steps as

π f ()xy,c=Q(xosθ +ysinθ)dθ (36) ∫0 θ

∞ Qt()= P(w)wejw2π tdw (37) θθ∫−∞

Thus fx(,y) can be reconstructed with two steps: the filtering step in (37), and the backprojection step in (36). For a particular image point ( x, y) , the backprojection step integrates Qθ (t) for all lines (θ,t) that pass through the point ( x, y) . This process is shown pictorially and described in Section 3.5.2.4 for our specific application.

A-9 Data Sufficiency Criteria

The 3D radon transform of an object consists of the integrals over all possible planes through that object. If some planes are missing then the full object cannot be reconstructed exactly. For a general path around an object, an x-ray cone-beam scanner will provide image data which when weighted and processed with the 2D radon transform, provides the 3D radon values integrated on many of the planes in 3D. In fact, many of the radon planes may be redundant since the x-ray source may lie on the same plane in multiple source locations, see Figure 51.

113

Plane covered by both Source 1 and Source 2

Figure 51 Redundant data in a circular path around an object created by two source centers lying on the same plane.

More likely, however, is the possibility that some planes cannot be computed from the x- ray projection data. This observation leads us to the data sufficiency criteria: in order to reconstruct an object from its cone-beam projections exactly, the path that the x-ray projection center traverses must intersect all possible planes through the object. A single circular orbit does not satisfy this condition as shown in Figure 52. However if an arc orthogonal to the circle is added to the path as in Figure 53, then the criteria is satisfied.

114

Figure 52 For the data sufficiency criteria, every plane that passes through the object must intersect the source path. This is an example where that is not the case.

There are many orbits that will satisfy this condition [11, 18], and many that will not. Since we do not define the orbit that will be used during the data collection (i.e. it is an arbitrary path) we may suffer from non-exact reconstructions as a result. Our algorithm checks for this

115 condition and keeps track of the locations in 3D radon space that are not filled (commonly referred to as the shadow zone) and those that are collected from multiple sensor positions. A normalization step is applied to the entire dataset at the end of the process to help account for the missing data that may result from arbitrary sensor paths.