Biomedical Imaging Informatics 9 Daniel L. Rubin , Hayit Greenspan , and James F. Brinkley

After reading this chapter, you should know the assembled into a pipeline when creating imag- answers to these questions: ing applications? • What makes images a challenging type of data • What is an imaging modality with high spatial to be processed by computers when compared resolution? What is a modality that provides to non-image clinical data? functional information? Why are most imag- • Why are there many different imaging modal- ing modalities not capable of providing both? ities, and by what major two characteristics do • What is the goal in performing segmentation they differ? in image analysis? Why is there more than one • How are visual and knowledge content in segmentation method? images represented computationally? How are • What are two types of quantitative informa- these techniques similar to representation of tion in images? What are two types of seman- non-image biomedical data? tic information in images? How might this • What sort of applications can be developed to information be used in medical applications? make use of the semantic image content made • What is the difference between image regis- accessible using the Annotation and Image tration and image fusion? What are examples Markup model? of each? • What are four different types of image pro- cessing methods? Why are such methods 9.1 Introduction

D. L. Rubin , MD, MS (*) Imaging plays a central role in the health care Departments of Radiology and Medicine , Stanford process. The fi eld is crucial not only to health University , 1201 Welch Road, P285 , care, but also to medical communication and Stanford 94305 , CA , USA education, as well as in research. In fact much of e-mail: [email protected] our recent progress, particularly in diagnosis, can H. Greenspan , PhD be traced to the availability of increasingly Department of Biomedical Engineering, Faculty of Engineering, Tel-Aviv University , sophisticated imaging techniques that not only Tel-Aviv 69978 , Israel show the structure of the body in incredible e-mail: [email protected] detail, but also show the function of the tissues J. F. Brinkley , MD, PhD within the body. Departments of Biological Structure, Biomedical Education and Medical Education, Computer Science and Engineering, University of Washington , 357420 , This chapter is adapted from an earlier version in the third Seattle 98195 , WA , USA edition authored by James F. Brinkley and Robert e-mail: [email protected] A. Greenes.

E.H. Shortliffe, J.J. Cimino (eds.), Biomedical Informatics, 285 DOI 10.1007/978-1-4471-4474-8_9, © Springer-Verlag London 2014 286 D.L. Rubin et al.

Although there are many types (or modalities) The major topics in biomedical imaging infor- of imaging equipment, the images the modalities matics include image acquisition, image content produce are nearly always acquired in or con- representation, management/storage of images, verted to digital form. The evolution of imaging image processing, and image interpretation/ from analog, fi lm-based acquisition to digital for- computer reasoning (Fig. 9.1 ). Image acquisi- mat has been driven by the necessities of cost tion is the process of generating images from reduction, effi cient throughput, and workfl ow in the modality and converting them to digital form managing and viewing an increasing prolifera- if they are not intrinsically digital. Image con- tion in the number of images produced per imag- tent representation makes the information in ing procedure (currently hundreds or even images accessible to machines for processing. thousands of images). At the same time, having Image management / storage includes methods images in digital format makes them amenable to for storing, transmitting, displaying, retriev- image processing methodologies for enhance- ing, and organizing images. Image process- ment, analysis, display, storage, and even ing comprises methods to enhance, segment, enhanced interpretation. visualize, fuse, or analyze the images. Image Because of the ubiquity of images in biomedi- interpretation / computer reasoning is the pro- cine, the increasing availability of images in digi- cess by which the individual viewing the image tal form, the rise of high-powered computer renders an impression of the medical signifi cance hardware and networks, and the commonality of of the results of imaging study, potentially aided image processing solutions, digital images have by computer methods. Chapter 20 is primarily become a core data type that must be considered concerned with information systems for image in many biomedical informatics applications. management and storage, whereas this chapter Therefore, this chapter is devoted to a basic under- concentrates on these other core topics in bio- standing of the unique aspects of images as a core medical imaging informatics. data type and the unique aspects of imaging from An important concept when thinking about an informatics perspective. Chapter 20 , on the imaging from an informatics perspective is that other hand, describes the use of images and image images are an unstructured data type ; as such, processing in various applications, particularly while machines can readily manage the raw those in radiology since that fi eld places the great- image data in terms of storage/retrieval, they can- est demands on imaging methods. not easily access image contents (recognize the The topics covered by this chapter and Chap. type of image, annotations made on the image, or 20 comprise the growing discipline of biomedi- anatomy or abnormalities within the image). In cal imaging informatics (Kulikowski 1997 ), a this regard, biomedical imaging informatics subfi eld of biomedical informatics (see Chap. 1 ) shares much in common with natural language that has arisen in recognition of the common processing (NLP; Chap. 8). In fact, as the meth- issues that pertain to all image modalities and ods of computationally representing and process- applications once the images are converted to ing images is presented in this chapter, parallels digital form. Biomedical imaging informatics is a to NLP should be considered, since there is syn- dynamic fi eld, recently evolving from focusing ergy from an informatics perspective. purely on image processing to broader informat- As in NLP, a major purpose of the methods of ics topics such as representing and processing the imaging informatics is to extract particular infor- semantic contents (Rubin and Napel 2010 ). At mation; in biomedical informatics the goal is the same time, imaging informatics shares com- often to extract information about the structure of mon methodologies and challenges with other the body and to collect features that will be useful domains in biomedical informatics. By trying to for characterizing abnormalities based on mor- understand these common issues, we can develop phological alterations. In fact, imaging provides general solutions that can be applied to all detailed and diverse information very useful for images, regardless of the source. characterizing disease, providing an “imaging 9 Biomedical Imaging Informatics 287

Image Image Acquisition Management/ Storage

Image Content Representation

Image Interpretation and Image Computer Reasoning Processing

Fig. 9.1 The major topics in biomedical imaging infor- image content representation, management/storage of matics follow a workfl ow of activities and tasks com- images, image processing, and image interpretation/com- mencing with include image acquisition, followed by puter reasoning phenotype” useful for characterizing disease, While we seek generality in discussing bio- since “a picture is worth a thousand words.1 ” medical imaging informatics, many examples in However, to overcome the challenges posed by this chapter are taken from a few selected the unstructured image data type, recent work is domains such as brain imaging, which is part of applying semantic methods from biomedical the growing fi eld of neuroinformatics (Koslow informatics to images to make their content and Huerta 1997 ). Though our examples are spe- explicit for machine processing (Rubin and Napel cifi c, we attempt to describe the topics in generic 2010 ). Many of the topics in this chapter there- terms so that the reader can recognize parallels to fore involve how to represent, extract and charac- other imaging domains and applications. terize the information that is present in images, such as anatomy and abnormalities. Once that task is completed, useful applications that pro- 9.2 Image Acquisition cess the image contents can be developed, such as image search and decision support to assist In general, there are two different strategies in with image interpretation. imaging the body: (1) delineate anatomic struc- ture (anatomic/structural imaging), and (2) deter- 1 Frederick Barnard, “One look is worth a thousand mine tissue composition or function (functional words,” Printers’ Ink, December, 1921. imaging) (Fig. 9.2 ). In reality, one does not 288 D.L. Rubin et al.

Radiography

PET-CT

CT MRI

US PET

Spatial resolution (anatomic detail) VLA HLA SA

Planar NM ED

ES

Functional information (tissue composition)

Fig. 9.2 The various radiology imaging methods differ mation depicted (which represents the tissue composi- according to two major axes of information of images, tion—e.g., normal or abnormal). A sample of the more spatial resolution (anatomic detail) and functional infor- common imaging modalities is shown choose between anatomic and functional imag- imaging, recognizing tissue function (e.g., tissue ing; many modalities provide information about ischemia, neoplasm, infl ammation, etc.) is not both morphology and function. However, in gen- the goal, though this is crucial to functional imag- eral, each imaging modality is characterized pri- ing and to patient diagnosis. In most cases, marily as being able to render high-resolution imaging will be done using a combination of images with good contrast resolution (anatomic methods or modalities to derive both structural/ imaging) or to render images that depict tissue anatomic information as well as functional function (functional imaging). information.

9.2.1 Anatomic (Structural) Imaging 9.2.2 Functional Imaging

Imaging the structure of the body has been and Many imaging techniques not only show the continues to be the major application of medical structure of the body, but also the function, where imaging, although, as described in Sect. 9.2.2 , for imaging purposes function can be inferred by functional imaging is a very active area of observing changes of structure over time. In research. The goal of anatomic imaging is to recent years this ability to image function has accurately depict the structure of the body—the greatly accelerated. For example, ultrasound and size and shape of organs—and to visualize abnor- angiography are widely used to show the func- malities clearly. Since the goal in anatomic imag- tioning of the heart by depicting wall motion, and ing is to depict and understand the structure of ultrasound Doppler can image both normal and anatomic entities accurately, high spatial resolu- disturbed blood fl ow (Mehta et al. 2000 ). tion is an important requirement of the imaging Molecular imaging (Sect. 9.2.3 ) is increasingly method (Fig. 9.2 ). On the other hand, in anatomic able to depict the expression of particular genes 9 Biomedical Imaging Informatics 289 superimposed on structural images, and thus can intensity is due to cognitive activity and how also be seen as a form of functional imaging. much is due to background noise. A particularly important application of func- As an example, one approach to fMRI imag- tional imaging is for understanding the cognitive ing is language mapping (Corina et al. 2000 ). The activity in the brain. It is now routinely possible subject is placed in the magnetic resonance to put a normal subject in a scanner, to give the imaging (MRI) scanner and told to silently name person a cognitive task, such as counting or objects shown at 3-s intervals on a head-mounted object recognition, and to observe which parts of display. The actual objects (“on” state) are alter- the brain light up. This unprecedented ability to nated with nonsense objects (“off” state), and the observe the functioning of the living brain opens fMRI signal is measured during both the on and up entirely new avenues for exploring how the the off states. Essentially the voxel values at the brain works. off (or control) state are subtracted from those at Functional brain imaging modalities can be the on state. The difference values are tested for classifi ed as image - based or non - image based . In signifi cant difference from non-activated areas, both cases it is taken as axiomatic that the func- then expressed as t-values. The voxel array of tional data must be mapped to the individual sub- t-values can be displayed as an image. ject’s anatomy, where the anatomy is extracted A large number of alternative methods have from structural images using techniques des- been and are being developed for acquiring and cribed in the previous sections. Once mapped to analyzing functional data (Frackowiak et al. anatomy, the functional data can be integrated 1997 ). The output of most of these techniques is with other functional data from the same sub- a low-resolution 3-D image volume in which ject, and with functional data from other subjects each voxel value is a measure of the amount of whose anatomy has been related to a template activation for a given task. The low-resolution or probabilistic atlas. Techniques for generat- volume is then mapped to anatomy guided by a ing, mapping and integrating functional data are high-resolution structural MR dataset, using one part of the fi eld of Functional Brain Mapping, of the linear registration techniques described in which has become very active in the past few Sect. 9.4.7 . years, with several conferences (Organization for Many of these and other techniques are imple- Human Brain Mapping 2001 ) and journals (Fox mented in the SPM program (Friston et al. 1995 ), 2001 ; Toga et al. 2001 ) devoted to the subject. the AFNI program (Cox 1996 ), the Lyngby tool- kit (Hansen et al. 1999 ), and several commercial 9.2.2.1 Image-Based Functional Brain programs such as Medex (Sensor Systems Inc. Imaging 2001 ) and Brain Voyager (Brain Innovation B.V. Image-based functional data generally come 2001 ). The FisWidgets project at the University from scanners that generate relatively low- of Pittsburgh is developing an approach that resolution volume arrays depicting spatially- allows customized creation of graphical user localized activation. For example, positron interfaces in an integrated desktop environment emission tomography (PET) (Heiss and Phelps (Cohen 2001 ). A similar effort (VoxBox) is 1983 ; Aine 1995 ; Alberini et al. 2011 ) and mag- underway at the University of Pennsylvania netic resonance spectroscopy (MRS) (Ross and (Kimborg and Aguirre 2002 ). Bluml 2001 ) reveal the uptake of various meta- The ultimate goal of functional neuroimaging bolic products by the functioning brain; and is to observe the actual electrical activity of the functional magnetic resonance imaging (fMRI) neurons as they perform various cognitive tasks. reveals changes in blood oxygenation that occur fMRI, MRS and PET do not directly record elec- following neural activity (Aine 1995 ). The raw trical activity. Rather, they record the results of intensity values generated by these techniques electrical activity, such as (in the case of fMRI) must be processed by sophisticated statistical the oxygenation of blood supplying the active algorithms to sort out how much of the observed neurons. Thus, there is a delay from the time of 290 D.L. Rubin et al.

activity to the measured response. In other words Visible light is the basis for an emerging modal- these techniques have relatively poor temporal res- ity called “optical imaging” and has promising olution (Sect. 9.2.4 ). Electroencephalography applications such as cancer imaging (Solomon, (EEG) or magnetoencephalography (MEG), Liu et al. 2011 ). Visible light, however, does not on the other hand, are more direct measures of allow us to see more than a short distance beneath electrical activity since they measure the elec- the surface of the body; thus other modalities are tromagnetic fi elds generated by the electrical used for imaging structures deep inside the body. activity of the neurons. Current EEG and MEG methods involve the use of large arrays of scalp 9.2.3.2 X-Rays sensors, the output of which are processed in a X-rays were fi rst discovered in 1895 by Wilhelm similar way to CT in order to localize the source Conrad Roentgen, who was awarded the 1901 of the electrical activity inside the brain. In gen- Nobel Prize in Physics for this achievement. The eral this “source localization problem” is under- discovery caused worldwide excitement, espe- constrained, so information about brain anatomy cially in the fi eld of medicine; by 1900, there obtained from MRI is used to provide further already were several medical radiological societ- constraints (George et al. 1995 ). ies. Thus, the foundation was laid for a new branch of medicine devoted to imaging the struc- ture and function of the body (Kevles 1997 ). 9.2.3 Imaging Modalities Radiography is the primary modality used in radiology departments today, both to record There are many different approaches that have a static image (Fig. 9.3) as well as to produce a been developed to acquire images of the body. A real-time view of the patient (fl uoroscopy) or a proliferation in imaging modalities refl ects the movie (cine). Both fi lm and fl uoroscopic screens fact that there is no single perfect imaging modal- were used initially for recording X-ray images, ity; no single imaging technique satisfi es all the but the fl uoroscopic images were too faint to be desiderata for depicting the broad variety of types used clinically. By the 1940s, however, television of pathology, some of which are better seen on and image-intensifi er technologies were devel- some modalities than on others. The primary oped to produce clear real-time fl uorescent difference among the imaging modalities is the energy source used to generate the images. In radiology, nearly every type of energy in the elec- tromagnetic spectrum has been used, in addition to other physical phenomena such as sound and heat. We describe the more common methods according to the type of energy used to create the image.

9.2.3.1 Light The earliest medical images used visible light to create photographs, either of gross anatomic structures or, if a microscope was used, of histo- logical specimens. Light is still an important source for creation of images, and in fact optical imaging has seen a resurgence of interest and application for areas such as molecular imaging (Weissleder and Mahmood 2001 ; Ray 2011 ) and Fig. 9.3 A radiograph of the chest (Chest X-ray) taken in the frontal projection. The image is shown as if the patient imaging of brain activity on the exposed surface is facing the viewer. This patient has abnormal density in of the cerebral cortex (Pouratian et al. 2003 ). the left lower lobe 9 Biomedical Imaging Informatics 291 images. Today, a standard procedure for many other hand, since the contrast in images is due to types of examinations is to combine real-time differences in tissue density and atomic number, television monitoring of X-ray images with the the amount of functional information that can be creation of selected higher resolution fi lm images. derived from radiographic images is limited Until the early 1970s, fi lm and fl uoroscopy were (Fig. 9.2 ). Radiography is also limited by rela- the only X-ray modalities available. Recently, tively poor contrast resolution (compared with nearly all radiology departments have shifted other modalities such as computed tomogra- away from acquiring radiographic images on fi lm phy (CT) or MRI), their use of ionizing radia- (analog images) to using digital radiography tion, the challenge of spatial localization due to (Korner et al. 2007 ) to acquire digital images. projection ambiguity, and their limited ability to X-ray imaging is a projection technique; an depict physiological function. As described X-ray beam—one form of ionizing radiation—is below, newer imaging modalities have been projected from an X-ray source through a developed to increase contrast resolution, to patient’s body (or other object) onto an X-ray eliminate the need for X-rays, and to improve array detector (a specially coated cassette that is spatial localization. A benefi t of radiographic scanned by a computer to capture the image in images is that they can be generated in real time digital form), or fi lm (to produce an non-digital (fl uoroscopy) and can be produced using porta- image). Because an X-ray beam is differentially ble devices. absorbed by the various body tissues based on the Computed Tomography (CT) is an important thickness and atomic number of the tissues, the imaging method that uses X-ray imaging to pro- X-rays produce varying degrees of brightness duce cross sectional and volumetric images of and darkness on the radiographic image. The dif- the body (Lee 2006 ). Similar to radiography, ferential amounts of brightness and darkness on X-rays are projected through the body onto an the image are referred to as “image contrast;” dif- array of detectors; however, the beam and detec- ferential contrast among structures on the image tors rotate around the patient, making numerous is the basis for recognizing anatomic structures. views at different angles of rotation. Using com- Since the image in radiography is a projection, puter reconstruction algorithms, an estimate of radiographs show a superposition of all the absolute density at each point (volume element structures traversed by the X-ray beam. or voxel) in the body is computed. Thus, the CT Computed radiography (CR) is an imaging image is a computed image (Fig. 9.4 ); CT did not technique that directly creates digital radiographs become practical for generating high quality from the imaging procedure. Storage phosphor images until the advent of powerful computers replaces fi lm by substituting a reusable phosphor plate in a standard fi lm cassette. The exposed plate is processed by a reader system that scans the image into digital form, erases the plate, and packages the cassette for reuse. An important advantage of CR systems is that the cassettes are of standard size, so they can be used in any equip- ment that holds fi lm-based cassettes (Horii 1996 ). More recently, Digital Radiography (DR) uses charge-coupled device (CCD) arrays to capture the image directly. Radiographic images have very high spatial Fig. 9.4 A CT image of the upper chest. CT images are resolution because a high photon fl ux is used to slices of a body plane; in this case, a cross sectional (axial) produce the images, and a high resolution detec- image of the chest. Axial images are viewed from below the patient, so that the patient’s left is on viewer’s right. tor (fi lm or digital image array) that captures This image shows a cancer mass in the left upper lobe of many line pairs per unit area is used. On the the lung 292 D.L. Rubin et al. and development of computer-based reconstruc- tered and is represented in the image as bright- tion techniques, which represent one of the most ness (more echoes returning to the source is spectacular applications of computers in all of shown as image brightness). The system con- medicine (Buxton 2009 ). The spatial resolution structs two-dimensional images (B-scans) by of images is not as high in CT as it is in radiogra- displaying the echoes from pulses of multiple phy, however, due to the computed nature of the adjacent one-dimensional paths (A-scans). images, the contrast resolution and ability to Ultrasound images are acquired as digital derive functional information of tissues in the images from the outset, and saved on computer body is superior for CT than for radiography disks. They may also be recorded as frames in (Fig. 9.2 ). rapid succession (cine loops) for real-time imag- ing. In addition, Doppler methods in ultrasound 9.2.3.3 Ultrasound are used to measure and characterize the blood A common energy source used to produce fl ow in blood vessels in the body (Fig. 9.5 ). images is ultrasound , which developed from Since the image contrast in ultrasound is research performed by the Navy during World based on differences in the acoustic impedance War II in which sonar was used to locate objects of tissue, ultrasound provides functional infor- of interest in the ocean. Ultrasonography uses mation (e.g., tissue composition and blood pulses of high-frequency sound waves rather fl ow). On the other hand, the fl ux of sound than ionizing radiation to image body structures waves is not as dense as the photon fl ux (Kremkau 2006 ). The basis of image generation used to produce images in radiography; thus is due to a property of all objects called acousti- ultrasound images are generally lower cal impedance. As sound waves encounter dif- resolution images than other imaging modali- ferent types of tissues in a patient’s body ties (Fig. 9.2 ). (particularly interfaces where there is a chance Current ultrasound machines are essentially in acoustical impedance), a portion of the wave specialized computers with attached peripherals, is refl ected and a portion of the sound beam with active development of three-dimensional (which is now attenuated) continues to traverse imaging. The ultrasound transducer now often into deeper tissues. The time required for the sweeps out a 3-D volume rather than a 2-D plane, echo to return is proportional to the distance and the data are written directly into a three- into the body at which it is refl ected; the ampli- dimensional array memory, which is displayed tude (intensity) of a returning echo depends on using volume or surface-based rendering tech- the acoustical properties of the tissues encoun- niques (Ritchie et al. 1996 ).

Fig. 9.5 An ultrasound image of abdomen. Like CT and MRI, ultrasound images are slices of a body, but because a user creates the images by holding a probe, any arbitrary plane can be imaged (so long as the probe can be oriented to produce that plane). This image shows an axial slice through the pancreas, and fl ow in nearby blood vessels ( in color ) is seen due to Doppler effects incorporated into the imaging method