BULLETIN (New Series) OF THE AMERICAN MATHEMATICAL SOCIETY Volume 00, Number 0, Pages 000–000 S 0273-0979(XX)0000-0

MATHEMATICAL METHODS IN MEDICAL IMAGE PROCESSING

SIGURD ANGENENT, ERIC PICHON, AND ALLEN TANNENBAUM

Abstract. In this paper, we describe some central mathematical problems in medical . The subject has been undergoing rapid changes driven by better hardware and software. Much of the software is based on novel methods utilizing geometric partial differential equations in conjunction with standard signal/image processing techniques as well as fa- cilitating man/machine interactions. As part of this enterprise, researchers have been trying to base biomedical engineering principles on rigorous mathe- matical foundations for the development of software methods to be integrated into complete therapy delivery systems. These systems support the more ef- fective delivery of many image-guided procedures such as radiation therapy, biopsy, and minimally invasive surgery. We will show how may impact some of the main problems in this area including image enhancement, registration, and segmentation.

Contents 1. Introduction 2 2. Outline 3 3. 3 3.1. Generalities 3 3.2. Imaging Modalities 4 4. Mathematics and Imaging 6 4.1. Artificial Vision 7 4.2. Algorithms and PDEs 7 5. Imaging Problems 9 5.1. Image Smoothing 9 5.2. 15 5.3. 19 6. Conclusion 26 References 27

Key words and phrases. medical imaging, artificial vision, smoothing, registration, segmentation. This research was supported by grants from the NSF, NIH (NAC P41 RR-13218 through Brigham and Women’s Hospital), and the Technion, Israel Insitute of Technology. This work was done under the auspices of the National Alliance for Medical Image Computing (NAMIC), funded by the National Institutes of Health through the NIH Roadmap for Medical Research, Grant U54 EB005149. The authors would like to thank Steven Haker, Ron Kikinis, Guillermo Sapiro, Anthony Yezzi, and Lei Zhu for many helpful conversations on medical imaging and to Bob McElroy for proof- reading the final document.

c 0000 (copyright holder) 1 2 SIGURD ANGENENT, ERIC PICHON, AND ALLEN TANNENBAUM

1. Introduction Medical imaging has been undergoing a revolution in the past decade with the advent of faster, more accurate, and less invasive devices. This has driven the need for corresponding software development which in turn has provided a major impetus for new algorithms in signal and image processing. Many of these algorithms are based on partial differential equations and curvature driven flows which will be the main topics of this survey paper. Mathematical models are the foundation of biomedical computing. Basing those models on data extracted from images continues to be a fundamental technique for achieving scientific progress in experimental, clinical, biomedical, and behavioral research. Today, medical images are acquired by a range of techniques across all biological scales, which go far beyond the visible light and microscope images of the early 20th century. Modern medical images may be considered to be geometrically arranged arrays of data samples which quantify such diverse physi- cal phenomena as the time variation of hemoglobin deoxygenation during neuronal metabolism, or the diffusion of water molecules through and within tissue. The broadening scope of imaging as a way to organize our observations of the biophys- ical world has led to a dramatic increase in our ability to apply new processing techniques and to combine multiple channels of data into sophisticated and com- plex mathematical models of physiological function and dysfunction. A key research area is the formulation of biomedical engineering principles based on rigorous mathematical foundations in order to develop general-purpose software methods that can be integrated into complete therapy delivery systems. Such systems support the more effective delivery of many image-guided procedures such as biopsy, minimally invasive surgery, and radiation therapy. In order to understand the extensive role of imaging in the therapeutic process, and to appreciate the current usage of images before, during, and after treatment, we focus our analysis on four main components of image-guided therapy (IGT) and image-guided surgery (IGS): localization, targeting, monitoring, and control. Specifically, in medical imaging we have four key problems:

(1) Segmentation - automated methods that create patient-specific models of relevant anatomy from images; (2) Registration - automated methods that align multiple data sets with each other; (3) - the technological environment in which image-guided pro- cedures can be displayed; (4) Simulation - softwares that can be used to rehearse and procedures, evaluate access strategies, and simulate planned treatments.

In this paper, we will only consider the first two problem areas. However, it is essential to note that in modern medical imaging, we need to integrate these technologies into complete and coherent image guided therapy delivery systems and validate these integrated systems using performance measures established in particular application areas. We should note that in this survey we touch only upon those aspects of the mathematics of medical imaging reflecting the personal tastes (and prejudices) of the authors. Indeed, we do not discuss a number of very important techniques such MATHEMATICAL METHODS IN MEDICAL IMAGE PROCESSING 3 as wavelets, which have had a significant impact on imaging and signal process- ing; see [60] and the references therein. Several articles and books are available which describe various mathematical aspects of imaging processing such as [67] (segmentation), [83] (curve evolution), and [71, 87] (level set methods). Finally, it is extremely important to note that all the mathematical algorithms which we sketch lead to interactive procedures. This means that in each case there is a human user in the loop (typically a clinical radiologist) who is the ultimate judge of the utility of the procedure, and who tunes the parameters either on or off-line. Nevertheless, there is a major need for further mathematical techniques which lead to more automatic and easier to use medical procedures. We hope that this paper may facilitate a dialogue between the mathematical and medical imaging communities.

2. Outline We briefly outline the subsequent sections of this paper. Section 3 reviews some of the key imaging modalities. Each one has certain advantages and disadvantages, and algorithms tuned to one device may not work as well on another device. Understanding the basic physics of the given modality is often very useful in forging the best algorithm. In Section 4, we describe some of the relevant issues in computer vision and image processing for the medical field as well as sketch some of the partial differential equation (PDE) methods that researchers have proposed to deal with these issues. Section 5 is the heart of this survey paper. Here we describe some of the main mathematical and engineering problems connected with image processing in general and medical imaging in particular. These include image smoothing, registration, and segmentation (see Sections 5.1, 5.2, and 5.3). We show how geometric partial differential equations and variational methods may be used to address some of these problems as well as illustrate some of the various methodologies. Finally in Section 6, we summarize the survey as well as point out some possible future research directions.

3. Medical Imaging 3.1. Generalities. In 1895, Roentgen discovered X-rays and pioneered medical imaging. His initial publication [82] contained a radiograph (i.e. an X-ray generated ) of Mrs. Roentgen’s hand; see Figure 3.1(b). For the first time, it was possible to visualize non-invasively (i.e., not through surgery) the interior of the human body. The discovery was widely publicized in the popular press and an “X- ray mania” immediately seized Europe and the United States [30, 47]. Within only a few months, public demonstrations were organized, commercial ventures created and innumerable medical applications investigated; see Figure 3.1(a). The field of radiography was born with a bang1! Today, medical imaging is a routine and essential part of . Pathologies can be observed directly rather than inferred from symptoms. For example, a physician can non-invasively monitor the healing of damaged tissue or the growth of a brain tumor, and determine an appropriate medical response. Medical imaging techniques can also be used when planning or even while performing surgery. For

1It was not understood at that time that X-rays are ionizing radiations and that high doses are dangerous. Many enthusiastic experimenters died of radio-induced cancers. 4 SIGURD ANGENENT, ERIC PICHON, AND ALLEN TANNENBAUM

(a) A “bone studio” commer- (b) First radiograph of cial. Mrs. Roentgen’s hand.

Figure 3.1. X-ray radiography at the end of the 19th century. example, a neurosurgeon can determine the “best” path in which to insert a needle, and then verify in real time its position as it is being inserted.

3.2. Imaging Modalities. Imaging technology has improved considerably since the end of the 19th century. Many different imaging techniques have been developed and are in clinical use. Because they are based on different physical principles [41], these techniques can be more or less suited to a particular organ or pathology. In practice they are complementary in that they offer different insights into the same underlying reality. In medical imaging, these different imaging techniques are called modalities. Anatomical modalities provide insight into the anatomical morphology. They include radiography, ultrasonography or ultrasound (US, Section 3.2.1), computed tomography (CT, Section 3.2.2), magnetic resonance imagery (MRI, Section 3.2.3) There are several derived modalities that sometimes appear under a different name, such as magnetic resonance angiography (MRA, from MRI), digital subtraction an- giography (DSA, from X-rays), computed tomography angiography (CTA, from CT) etc. Functional modalities, on the other hand, depict the metabolism of the underly- ing tissues or organs. They include the three nuclear medicine modalities, namely, scintigraphy, single photon emission computed tomography (SPECT) and positron emission tomography (PET, Section 3.2.4) as well as functional magnetic resonance imagery (fMRI). This list is not exhaustive, as new techniques are being added every few years as well [13]. Almost all images are now acquired digitally and integrated in a computerized picture archiving and communication system (PACS). In our discussion below, we will only give very brief descriptions of some of the most popular modalities. For more details, a very readable treatment (together with the underlying physics) may be found in the book [43].

3.2.1. Ultrasonography (1960’s). In this modality, a transmitter sends high fre- quency sound waves into the body where they bounce off the different tissues and organs to produce distinctive patterns of echoes. These echoes are acquired by a receiver and forwarded to a computer that translates them into an image on a screen. Because ultrasound can distinguish subtle variations among soft, fluid-filled MATHEMATICAL METHODS IN MEDICAL IMAGE PROCESSING 5 tissues, it is particularly useful in imaging the abdomen. In contrast to X-rays, ul- trasound does not damage tissues with ionizing radiation. The great disadvantage of ultrasonography is that it produces very noisy images. It may therefore be hard to distinguish smaller features (such as cysts in breast imagery). Typically quite a bit of image preprocessing is required. See Figure 3.2(a). 3.2.2. Computed Tomography (1970’s). In computed tomography (CT), a number of 2D radiographs are acquired by rotating the X-ray tube around the body of the patient. (There are several different for this.) The full 3D image can then be reconstructed by computer from the 2D projections using the Radon transform [40]. Thus CT is essentially a 3D version of X-ray radiography, and therefore offers high contrast between bone and soft tissue and low contrast between among different soft tissues. See Figure 3.2(b). A contrast agent (some chemical solution opaque to the X-rays) can be injected into the patient in order to artificially increase the contrast among the tissues of interest and so enhance image quality. Because CT is based on multiple radiographs, the deleterious effects of ionizing radiation should be taken into account (even though it is claimed that the dose is sufficiently low in modern devices so that this is probably not a major health risk issue). A CT image can be obtained within one breath hold which makes CT the modality of choice for imaging the thoracic cage. 3.2.3. Magnetic Resonance Imaging (1980’s). This technique relies on the relax- ation properties of magnetically-excited hydrogen nuclei of water molecules in the body. The patient under study is briefly exposed to a burst of radio-frequency energy, which, in the presence of a magnetic field, puts the nuclei in an elevated energy state. As the molecules undergo their normal, microscopic tumbling, they shed this energy into their surroundings, in a process referred to as relaxation. Im- ages are created from the difference in relaxation rates in different tissues. This technique was initially known as nuclear magnetic resonance (NMR) but the term “nuclear” was removed to avoid any association with nuclear radiation2. MRI uti- lizes strong magnetic fields and non-ionizing radiation in the radio frequency range, and according to current medical knowledge, is harmless to patients. Another ad- vantage of MRI is that soft tissue contrast is much better than with X-rays leading to higher-quality images, especially in brain and spinal cord scans. See Figure 3.2(c). Refinements have been developed such as functional MRI (fMRI) that mea- sures temporal variations (e.g., for detection of neural activity), and diffusion MRI that measures the diffusion of water molecules in anisotropic tissues such as white matter in the brain. 3.2.4. Positron Emission Tomography (1990’s). The patient is injected with ra- dioactive isotopes that emit particles called positrons (anti-electrons). When a positron meets an electron, the collision produces a pair of gamma ray photons having the same energy but moving in opposite directions. From the position and delay between the photon pair on a receptor, the origin of the photons can be determined. While MRI and CT can only detect anatomical changes, PET is a functional modality that can be used to visualize pathologies at the much finer molecular level. This is achieved by employing radioisotopes that have different rates of intake for different tissues. For example, the change of regional blood flow

2The nuclei relevant to MRI exist whether the technique is applied or not. 6 SIGURD ANGENENT, ERIC PICHON, AND ALLEN TANNENBAUM

(a) Ultrasound. (foetus) (b) Computed Tomography (brain, 2D axial slice).

(c) Magnetic Resonance Imagery (d) Positron Emission Tomography (brain, 2D axial slice). (brain, 2D axial slice).

Figure 3.2. Examples of different image modalities. in various anatomical structures (as a measure of the injected positron emitter) can be visualized and relatively quantified. Since the patient has to be injected with radioactive material, PET is relatively invas