Evaluating Shape Alignment Via Ensemble Visualization," in IEEE Computer Graphics and Applications, Vol

M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70 Feature Article

Evaluating Shape Alignment via Ensemble Visualization

Mukund Raj, Mahsa Mirzargar, J. Samuel Preston, Robert M. Kirby, and Ross T. Whitaker ■ University of Utah

s computational tools for simulation and model is straightforward, and hence we have only data analysis have matured, researchers, the ensemble members themselves from which to scientists, and analysts have become in- gain insight into the originating process. Aterested in understanding not only the determinis- Studying an ensemble in terms of the vari- tic output of these tools, but also ability or dispersion between ensemble members the uncertainty associated with can provide useful information and insight about Visualizing variability in their computations and data col- the underlying distribution of possible outcomes. surfaces embedded in lection. Consequently, there is an Correspondingly, ensemble visualization can be a 3D provides a means of increasing interest in uncertainty powerful way to study this variability. However, understanding the underlying quantification as an integrated a key challenge here is to convey the variability distribution of a collection part of simulation and data sci- among ensemble members while preserving the of surfaces. An expert-based ence in various science and engi- main features they share. Preserving these fea- evaluation of various ensemble neering disciplines. Uncertainty tures is particularly challenging in cases where the visualization techniques quantification views the simula- ensemble members are not fields over which sta- demonstrates the efficacy of tion and data science pipelines as tistical operations such as mean and variance are using a 3D contour boxplot a random process containing pos- well-defined, but instead are derived or extracted sibly both epistemic (reducible) features such as isosurfaces. ensemble visualization and aleatoric (by chance) uncer- In this article, we examine the effectiveness of technique to analyze shape tainty. Quantification efforts in the contour boxplot technique,1 a descriptive sum- alignment and variability in this random process are divided mary analysis and visualization methodology, in atlas construction and analysis. into roughly two categories: the context of a particular medical-data-science application: brain atlas construction and analy- ■■ efforts to understand the uncertainty and/or sis. We conducted an expert-based evaluation of variability of the process by examining instances the visualization of ensembles generated through (samples) of the process and shape alignment using image deformation in the ■■ efforts to determine models (such as probability construction of atlases (or templates) for brain theory) that capture the nature of the process. image analysis. To accomplish this evaluation, we constructed a prototype system for visual- The first of these categories, and the focus of this izing and interacting with ensembles of 3D iso- study, utilizes an ensemble of solutions meant to surfaces through a combination of 3D rendering capture the inherent variability or uncertainty in a (isocontouring) and cut planes (slices through computational or data-science pipeline. Although 3D volumetric fields). In addition, we generalized we assume that the variability seen in the ensem- the algorithm1 to three dimensions as a direct ble can be attributed to some condition or prop- extension of their analysis of isocontours to iso- erty of the generating process, we do not assume surfaces—that is, from codimension-one objects that articulation of the process via a mathematical embedded in 2D to codimension-one objects em-

g3raj.indd 60 4/19/16 2:14 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

bedded in 3D. This generalization lets us compare an ensemble’s contour boxplot summaries to both the full enumeration of the ensemble as well as Image 1 Image 2 other traditional means of atlas evaluation (such as qualitative visual inspection of slices of the atlas image or individual volumetric images used to construct the atlas). In collaboration with domain experts, we used this system to explore the efficacy Atlas of using ensemble visualization techniques for evaluating 3D shape alignment of brain MRI images. The purpose of this article is to study and evalu- Image 3 Image 4 ate the use of contour boxplots in a real-world data- science application: the alignment of 3D shapes or surfaces in a population-based ensemble. Our hypothesis is that the contour boxplot will let users summarize their data in a meaningful way that allows either better or more efficient (faster) assess- Transformation ment of the atlas construction as compared with explicit enumeration of the ensemble (that is, looking at each image individually) or through more Figure 1. An atlas construction scheme involves coarse-grained characterizations, such as examin- deforming and registering all ensemble members to ing the average intensity image or label (segmenta- the atlas. This deformation and registration process is tion) probability maps. called transformation to the atlas coordinate system Our evaluation results show that the contour or the atlas space. boxplot methodology has the potential to significantly benefit the application under study by expected anatomical structure and variability of a providing a visualization of the ensemble’s quan- population and compare different populations in titative summaries. Although we have formulated terms of their group atlases (for example, healthy our hypothesis in the context of a particular appli- and unhealthy groups). Differences in the atlas cation, we believe that our evaluation may provide anatomy can be identified both qualitatively by insight into other arenas where visualization and inspecting unaligned structures (when mapped analysis of shape ensembles are desired. to the atlas space) and quantitatively by analyzing the deformations, quantifying the amount of Brain Atlas Construction change necessary to bring an individual ensemble Constructing an anatomical atlas for a collec- member into alignment. tion of brain images is an important problem in Atlas generation is an automated process, but it medical image analysis. The goal of various atlas is not parameter-free, and the choice of param- construction schemes is to construct a statistical eters can greatly influence the quality of the result. representative image and an associated set of co- In particular, nonlinear deformations computed ordinate transformations (deformations) from an for medical image registration are a trade-off ensemble of images.2 Anatomical atlases provide a between image matching and plausible deforma- common coordinate system (atlas space) in which tions. For example, the deformation should not re- to define reference locations of brain structures. sult in the elimination of anatomical features or As part of the atlas construction process, non- noninvertible transformations. Hence, the defor- linear registration techniques generate deforma- mation is often controlled by tuning parameters tions that can map the anatomies in an individual to find a compromise between the mismatch be- image to the atlas space (see Figure 1). The atlas tween images and the regularity (smoothness) of construction process jointly estimates a represen- the transformation. Because of the regularization tative image defining the atlas space (the atlas of the deformations and the inherent anatomi- image) and the deformations aligning individual cal differences among ensemble members, not all images to this atlas image (that is, it maps the im- features will be perfectly aligned. This imperfect age individually to the atlas space). The atlas image alignment manifests as blurring in the atlas im- generated by these techniques then represents the age where there is disagreement regarding voxel average (or normal) anatomy of this population. intensity among ensemble members when mapped Such atlases help domain experts characterize the to the atlas space.

IEEE Computer Graphics and Applications 61

g3raj.indd 61 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," Feature Article in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

Correct tuning of the regularization parameters scenario is similar in spirit to the feature-space av- lets the deformations account for as much anatom- eraging issue highlighted in earlier work1—that is, ical variability as possible by correctly aligning the the average field’s isosurface (segmentation) is of- corresponding anatomy and not simply matching tentimes not equivalent to a representative of a set similar intensities. This alignment of correspond- chosen from isosurfaces of the individual fields. ing anatomy is essential for an atlas to be effective Thus, the avoidance of feature-space averaging is in later statistical analyses of the population. Op- why we believe the contour boxplot methodology timization convergence can be easily checked, but provides a useful way to summarize the type of the degree to which particular structures align is ensemble data where analyzing feature sets and analyzed qualitatively by observing the amount of their representatives is important.1 Because the blurring in the atlas image and by checking each manual, qualitative evaluation of shape alignment ensemble member’s alignment (deformed to the (as a result of image registration) is challenging, atlas space) with the atlas image. The initial align- quantifying the variability of the shape alignment ment is often unsatisfactory, which results in an and visualizing this variability can facilitate the iterative process of parameter tuning and rerun- domain experts’ ability to effectively validate the ning the atlas-generation process. atlas-construction scheme. In addition, because of problems with image scans, extreme variability among the ensemble Data Preprocessing for Atlases members, or incorrect preprocessing, it may not be The images analyzed in this article are 3D MRI images obtained from the Alzheimer’s Disease Neu- roimaging Initiative (ADNI) database.3 Each brain Deformations computed for image in our ensemble was also provided with a corresponding label map volume with various an- medical image registration are a atomical structures segmented and marked, with each brain region having a unique integer value. trade-off between image matching To analyze a specific structure within the brain and plausible deformations. anatomy, we used the label assigned to that structure to select it and mask out the remaining region in all members of the ensemble. possible to achieve reasonable alignment with the The atlas construction scheme we used is the atlas image for some set of outlier images. Identi- unbiased diffeomorphic atlas,2 which was imple- fying and removing such images is often another mented as part of an open-source medical im- part of the atlas-generation procedure. Automated age atlas-construction package called AtlasWerks measures of global image alignment are available, (www.sci.utah.edu/software/atlaswerks.html). but they do not give insight into why or in which We constructed atlases from MRI image ensembles spatial regions particular ensemble members have using different parameter choices and/or different poor alignment. Depending on the proposed appli- ensembles (subject groups). In each case, after con- cation of the atlas, these insights may be pertinent structing the atlas using the MRI images, the cor- to the decision to prune or keep particular images responding label map images were transformed to (ensemble members). the common (atlas) coordinate space using defor- This manual iteration of parameter tuning/ mation fields calculated during the atlas-construc- pruning and atlas generation eventually yields the tion process we described in the last section. These final atlas to be used in further analysis. We should transformed label maps were then passed as input note two important points about the final atlas im- to the preprocessing pipeline (which we describe in age. First, this representative image/segmentation the next section) for visualization. is not a member of the ensemble itself, but rather For a well-constructed atlas, we can expect an image/segmentation generated through statis- the anatomical structures in the brain to have tical operations on the deformation fields. That a relatively small amount of variability after be- is, it is not a member of the population that best ing transformed to the atlas space. We selected represents the population, but rather an attempt at two anatomical structures in the brain expected statistically characterizing a representative image. to pose different levels of difficulty during atlas Second, the iterative process does not guarantee construction: the left ventricle and the cortex. The that the resulting atlas image will be crisp, that ventricle is often considered to be a distinct struc- there will be no blurry regions in the image. The ture (with high contrast) in the brain image and, image ensemble compared with the atlas image therefore, can be expected to exhibit good align-

62 May/June 2016

g3raj.indd 62 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

ment among ensemble members in the atlas space (if all goes well). The cortex was selected as an example of an anatomical structure with a com- plex shape (see Figure 2), a significant challenge for registration and alignment.

Expert Evaluation Study Details Domain experts use various open-source or commercial packages to visualize slices from individual volumetric images or simply from the average of the aligned images, but to the best of our knowledge, ours is the first attempt to study shape alignment in atlas construction using ensemble visualization techniques. For our evalua- Figure 2. Example anatomical structure: the cortex tion study, we had access to a group of five domain (green) and the ventricle (red). This image shows the experts who work with atlases regularly and who segmentation provided by the label map volume for volunteered to participate in our expert evaluation a typical ensemble member. The coarseness of the study. This group included graduate students, staff segmentation seen in this label map is mitigated by researchers, and faculty who use atlases and medi- smoothing for the final visualization. cal image ensembles in their research projects. We asked the participants to explain their current methodologies for evaluating the atlas construction scheme as well as the quality of the atlases in terms of being a representative of the ensemble. As mentioned earlier, we learned that this process is often performed qualitatively. A visual inspection is carried out to ascertain whether the shapes of the anatomical structures in the atlas space are realistic. Experts also mentioned that, in order for an atlas to be helpful for different medical imaging applications such as the segmentation of a specific brain structure, they need the atlas image and the anatomical structures therein to have sufficient contrast. For example, they expect to see a crisp boundary (in terms of the average combined image (a) (b) intensities) between gray and white matter in the brain. Therefore, the sharpness of the boundaries Figure 3. Atlas image slice constructed using AtlasWerks: (a) Atlas image of the anatomical structures in the atlas image is slice and (b) MRI image slice. The anatomical structures in the atlas another criterion examined qualitatively to evalu- image usually have lower contrast and fuzzier edges compared with the ate the ensemble’s alignment. These qualitative original MRI images. This fuzziness results from performing averaging evaluations are often performed on a subset of the while constructing the atlas. ensemble (in the atlas coordinate system) because visualizing the entire ensemble results in too much line and our design choices to mitigate the chal- clutter and blurriness. lenge of visualizing and rendering an ensemble Figure 3 shows a snapshot of a slice of the brain of 3D isosurfaces. atlas image used as a common (atlas) coordinate system to register individual label maps from the Ensemble Visualization Overview ensemble. Visualization is often data-driven, and therefore uncertainty visualization schemes are typically de- Visualization Pipeline signed to deal with the type of data being visual- To describe our prototype system’s visualiza- ized. For scientific data, users are often interested tion pipeline, we start with a brief summary of in visualizing derived features of their data, such various ensemble visualization strategies that we as transition regions (or edges), critical points, considered and incorporated into our prototype and isosurfaces (of volumetric data), as well as system. We then provide an overview of the pipe- the uncertainty associated with such feature sets.

IEEE Computer Graphics and Applications 63

g3raj.indd 63 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," Feature Article in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70 Region C Figure 4. Three visualizations of ventricles from an Region D ensemble containing 34 images from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset transformed to a common atlas space: (a) 3D contour boxplot visualization, where dark purple indicates the 50 percent volumetric band, the 100 percent band volume is in light purple, the median is in yellow, and outliers are in red (on the cutting plane); (b) direct visualization of the ensemble members (spaghetti plot); and (c) 3D average intensity image.

Region B The focus here is the visualization of isosurfaces in the context of uncertain scalar fields, which has been studied somewhat extensively. Most relevant Region A to the application under study (atlas construc- (a) tion) is the visualization of uncertain isosurface extracted from an ensemble of scalar fields. Region C Region D Here we provide a brief summary of three classes of popular techniques for visualization of uncertain isosurfaces that are extracted from ensembles of scalar fields. The following three techniques were chosen to represent the range of strategies for rep- resenting an ensemble. To analyze the alignment, or lack thereof, of shapes in an ensemble, we incorporated representative members of these technique categories as part of our prototype system.

Enumeration. A widely used approach for ensemble Region B visualization is the direct visualization of all ensemble members. Direct visualization has gained significant interest in applications such as weather forecasting and hurricane prediction.6 Ensemble- Region A Vis is an example of a data analysis tool designed (b) to visualize ensemble data.7 It uses multiple views of fields of interest to enhance the visual analysis of ensembles. We incorporated direct visualization of 3D ensemble members (see Figure 4b) by rendering the curves formed by the region of intersection of each ensemble member’s codimension one isosurface with a cut plane. As long as the isosurface embedded in 3D is closed, closed curves will be generated when the isosurface is sliced for visualization purposes. We refer to this visualization as a spaghetti plot. To facilitate the interpretation of the individual ensemble members, we rendered each of these curves with distinct and random colors. There are (c) a variety of options for rendering the enumeration of all 3D surfaces, including transparency, but clutter is a significant challenge.6 For this (A thorough review of the rich literature on un- work, we present the surfaces of the innermost certainty visualization is beyond the scope of this and outermost volumetric bands formed by all article. Interested readers can consult these works ensemble members. User studies have suggested for further details on recent advancements on the effectiveness of direct ensemble visualization this topic.4,5 ) techniques,6 but such approaches do not provide

64 May/June 2016

g3raj.indd 64 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

Ensemble feature Dual cut planes any quantitative information about the data un- extraction certainty and rely solely on the user for interpret- Image object Filtering, ing data. manipulation feature selection Antialiasing Parametric probabilistic summaries. Many uncertainty visualization schemes use probabilistic modeling to convey quantitative information re- Mesh Surface or volume garding data uncertainty. These techniques often smoothing rendering rely on a certain kind of statistical model such as multivariate normal distributions. Data preprocessingVisualization As a representative of such techniques, we chose to consider the concept of level-crossing probabili- Figure 5. Overview of prototype system designed for shape alignment ties (LCP).8 For visualization, we implemented the evaluation using ensemble visualization. The prototype system consists 3D probabilistic marching cubes algorithms (pro- of data preprocessing and visualization stages. posed based on LCP)9 as part of our initial visualization system. Probabilistic marching cubes rely In our system, we algorithmically extend and on approximating and visualizing the probability implement the contour boxplot analysis for iso- map of the presence of the isosurface at each voxel surfaces embedded in 3D (see Figure 4a) as an location. However, the use of parametric modeling example of visualization techniques based on an can limit the capability of this technique. Approxi- ensemble’s nonparametric descriptive statistical mating the underlying distribution that gives rise to summaries. the ensemble and presenting the user with only aggregated quantities of the inferred distribution can Ensemble Visualization Prototype System be misleading in some applications. For instance, At a high level, our prototype system consists of this approach can often hide or distort structures two stages: data preprocessing and visualization that are readily apparent in the ensemble. (see Figure 5). When visualizing isosurfaces of a binary 3D Nonparametric descriptive summaries. An alterna- segmented image, it is often necessary to perform tive strategy that relies on neither enumeration smoothing to reduce aliasing artifacts and facili- nor parametric modeling of the underlying dis- tate 3D rendering and shading. First, we perform tribution is to form descriptive statistics of an this smoothing in a two-step preprocessing stage. ensemble. Descriptive statistics offer an ensemble In the first step, the binary partitioned image is visualization paradigm for understanding or inter- antialiased using an iterative relaxation process.11 preting uncertainty from an ensemble’s structure. Next, a small amount of mesh smoothing is per- The notion of centrality is a natural approach to formed on the isosurface mesh generated from the understanding an ensemble’s structure. Because antialiased binary image. All visualization prepro- an ensemble is an empirical description of its dis- cessing operations occur on the 3D volume (and tribution, some instances from it are more central corresponding codimension one isosurfaces) prior to the distribution and therefore are more typical to cut-plane extraction. within the distribution. The second stage includes some visualization The notion of data depth provides a formalism strategies to facilitate the perception and naviga- for characterizing how central a sample is within tion of the rendered 3D objects. To improve shape an ensemble. Data depth provides a natural gen- perception in our application, we include interac- eralization of rank statistics to multivariate data.10 tivity with renderings of 3D objects as part of the The univariate boxplot (or whisker plot) is a con- visualization system. In our settings, the user can ventional approach to visualize order statistics. rotate the object displayed on the screen using a Boxplot visualizations provide a visual represen- standard trackball interaction mechanism. The tation of an ensemble’s main features, such as the system lets the user select cutting planes, which most representative member (the median), quar- clip a portion of the volume displayed on the tile intervals, and potential outliers. The notation screen, to render cross-section views of surfaces of data depth has been generalized for ensembles embedded in 3D. The user can also interactively of isocontours.1 Researchers have also proposed orient and translate the cutting plane. Addition- the contour boxplot technique to summarize ro- ally, the system provides the flexibility of having bust and descriptive statistics of ensembles of 2D one or multiple cutting planes and interactively ad- isocontours.1 justing their position and orientation. The system

IEEE Computer Graphics and Applications 65

g3raj.indd 65 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," Feature Article in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

interface lets the user interactively select various ing whether the variability is due to differences in features of interest for rendering in order to focus shape, position, or both. on particular features of interest. For example, the Figure 4 shows the three approaches to visual- user can select specific ensemble members to be izing the aligned ventricles for an ensemble of rendered individually. 34 brains. In Figure 4a, one can immediately In the case of 3D contour boxplots, we per- identify regions of high variability. For example, formed the analysis on the 3D binary segmented in region A, most of the variability is outside the volumetric data (in the preprocessing stage) and 50 percent band, which means that less than rendered the results interactively. Although the half the ensemble members contributed to this analysis was performed on the volumetric data variability. leading to volumetric 50 and 100 percent bands, Looking at the spaghetti plot in Figure 4b, we we rendered the visualization of the statistical see there are in fact only two ensemble members summaries only on chosen cut planes to deal with that significantly differ from the other members the issue of occlusion. For instance, in the absence in region A. These results show that the variability of a cut plane, the 100 percent band entirely oc- is due to overall position as well as shape in this cludes the median shape and the 50 percent band. region. In region B in Figure 4a, we notice that the variability can be attributed to significantly different shapes of the isocontours and that these The contour boxplot, as part of shapes would not easily be aligned through the smooth transformations in this atlas and may re- the atlas construction process, can quire parameter tuning to achieve alignment. By help users tease apart the different observing region C in Figures 4a and 4b, we see that the variability comes mostly from the posi- aspects of variability. tions of the isocontours. Results in region C also show that no particular ensemble member is dis- proportionately responsible for the variability—the Evaluation width of the 50 percent band is nearly that of the To demonstrate the efficacy of using ensemble vi- 100 percent band in this region, and outliers align sualization techniques to study the alignment of well with the median contour. MRI brain images during brain atlas construc- Finally, region D in Figure 4a demonstrates an tion, we gathered feedback in an expert evalu- area of low variability across the ensemble and ation study of the proposed prototype system. provides an example of good alignment of all the We described the prototype system to our expert ventricles, which is confirmed by the spaghetti evaluators (whom we call participants) after a plot in Figure 4b. Figure 4c shows a volume-ren- walk-through presentation of the different en- dered 3D version of the average intensity image semble visualization techniques. The participants for comparison. The average intensity image is an were able to interact with the system and switch essential part of the atlas, but it does not provide through the various visualization. For our study, the same insights for debugging the atlas in a de- we solicited their feedback on the visualization of tailed way. two anatomical structures: the left ventricle and We also showed the participants volume render- the cortical surface. We paid particular attention ings of LCP values, as suggested in earlier work.9 to the participants’ comments concerning the The participants noted that the LCP visualization suitability of ensemble visualization for this ap- shows almost the same information as the aver- plication. As the participants interacted with our age intensity image in Figure 4c, which is already system, we gained useful insights into the atlas used extensively during atlas construction. They data. We describe three examples here. did not feel that further exploration of this form of ensemble uncertainty visualization for evaluat- Local Variability ing atlases would be useful, and therefore we did In our first example, we focus on analyzing the not include comprehensive results from LCP ren- variability within an ensemble of different regions derings in this study. of brain ventricles transformed to a common atlas space using the unbiased, diffeomorphic ap- Overall Variability proach.2 Ensemble visualization not only helps The second example was chosen to evaluate whether general users identify regions that are either well ensemble visualization can also provide insight or poorly aligned, but also provides insight regard- into the overall variability among the members of

66 May/June 2016

g3raj.indd 66 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

(a) (b) (c)

Figure 6. Slices of average intensity atlases for ensembles of 30 brain images: (a) High value of regularization (transformation smoothing), λ = 1.0; (b) the same ensemble with low regularization value, λ = 1/9; and from a different ensemble (subject group) with the regularization/smoothing at λ = 1/9.

an ensemble of aligned shapes. An understanding Figures 6 and 7 shows slices of intensity atlases of the overall variability, as opposed to local vari- and contour boxplot visualizations for each of ability, is useful not only for understanding how the three cases. Figure 6 presents a slice of the well a particular atlas was constructed, but also intensity image for each atlas, and Figure 7 dem- for comparing different atlases. onstrates the 3D contour boxplot visualization of For this example, we constructed three atlases, the cortical surfaces for atlases corresponding to each with an ensemble size of 30. The first atlas the intensity images in Figure 6. was constructed with a high value of regulariza- Using a high value for the regularization param- tion (transformation smoothing), λ = 1.0; a sec- eter enforces high smoothness of the deformation ond atlas was constructed for the same ensemble fields, which in turn makes it harder to arrive at while using a low regularization value, λ = 1/9; a set of deformations that would perfectly align and a third atlas was constructed from a different all the individual images. This lack of alignment ensemble (subject group) with the regularization/ leads to high variability between isosurfaces in smoothing at λ = 1/9. the ensemble. Such high variability is easily visible

Region G

Region E Region E Region F Region F Region H

(a) (b) (c)

Figure 7. Associated contour boxplot visualizations for cortical surfaces in Figure 6: (a) Atlas constructed with high regularization of deformation, (b) atlas constructed with low regularization, and (c) atlas with low regularization using a different ensemble than in the other columns.

IEEE Computer Graphics and Applications 67

g3raj.indd 67 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," Feature Article in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

(a) Region J Region L

Region K Region I

(b) (c)

Figure 8. Visualizations of left ventricles. Crosses mark the correspondence between the images. (a) Left ventricle slice from an intensity image of the atlas. (b) Left ventricle slice of an ensemble member identified as an outlier by data depth analysis. (c) Contour boxplot visualization of an ensemble of 34 ventricles in the atlas space.

by looking at region E in Figure 7a, where the 50 general user how well a particular shape is aligned and 100 percent bands are wider than in the cor- with respect to the rest of the ensemble. Such responding region of the atlas with low regulariza- knowledge is particularly useful in the case of out- tion (see Figure 7b). Better image alignment when lier shapes. Atlas construction is often an iterative the atlas is constructed with low regularization is process, and identifying outlier images that do not also evident in region E by comparing contours of align sufficiently with the atlas is an important the median and outlier shapes rendered on the cut intermediate step in the process. plane in Figures 7a and b. We see that the median In the contour boxplot in Figure 8c, we see a sin- and the outlier shapes are poorly aligned for images gle outlier shape and its alignment relative to the aligned with an atlas constructed with high regu- ensemble. By comparing this visualization with an larization (see Figure 7a), whereas the alignment average intensity image of the left ventricle region is much better when the atlas is constructed with (Figure 8a), we see that an anomaly in region I low regularization. (Figure 8c) shows as a barely perceivable increase Finally, the third atlas in this example (see in intensity in Figure 8a. A similar observation Figures 6c and 7c) demonstrates the effect of in- can be made from the intensity image slice of the herent variability among the ensemble members outlier member shown in Figure 8b. However, the (brain images) on the atlas construction process. anomaly shows up clearly in the contour boxplot, We see that in many regions of Figure 7c, for in- and because it is outside the 100 percent band, stance in region F, the 100 percent band is signifi- we know that the degree of misalignment of this cantly wider than the 50 percent band, indicating shape is rare within the ensemble of ventricles. a significant spread in the distribution of surfaces, Region I also demonstrates the challenges of as- which differs from the variability seen in the cor- sessing geometry in 3D because distances between responding region in Figure 7b, where both bands surfaces can be exaggerated when viewing them nearly overlap. Furthermore, in the third atlas we on a single cut. However, interacting with the vi- see that the outlier is well aligned with the me- sualization by moving and rotating the cut plane dian in some regions (see region G), but it is poorly can help verify the 3D shapes of rank statistics and aligned in others (see region H). This example the surface geometries and separation distances. demonstrates that shape and surface variability in In some cases, aligned shapes can differ in size atlases depends, in addition to construction pa- from the rest of the ensemble. For instance, Fig- rameters, on the inherent variability of shapes in ure 8c shows that the outlier ventricle is notice- the ensemble. Thus, the contour boxplot, as part of ably smaller than the median ventricle in regions the atlas construction process, can help users tease J and K, which is not the case in region L. This apart these different aspects of variability. observation is not possible in the corresponding intensity images. These size differences occur for Member Alignment several reasons. In this example, for instance, the In addition to aiding in the understanding of the outlier ventricle may have been different and ir- general alignment of shapes in an ensemble, the regular to begin with. Another reason could be contour boxplot is also useful in conveying to the mislabeling of the ventricular region during the

68 May/June 2016

g3raj.indd 68 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

segmentation process to generate that image’s la- ■■ The spaghetti plot helped the participants view bels. Finally, the process of generating deforma- the contours of specific ensemble members other tions during the atlas construction might fail, than the median or outliers. leading to irregularities for an ensemble member ■■ The contour boxplot and the spaghetti plot were when mapped onto the atlas space. The contour able to convey important details pertaining to boxplot can provide information that can help the variability in an ensemble, whereas the av- the user decide whether or not any particular erage intensities had limited utility because of outliers need to be removed from the ensemble their general fuzziness. or if further investigation is necessary to identify causes of possible misalignment. The goal of the application we describe here is to evaluate the alignment of 3D shapes, in par- Technique Comparison ticular the alignment of 3D MRI images that have At the conclusion of our study, we asked the par- been transformed to a common atlas space us- ticipants to comment on their experience with the ing various ensemble visualization methods. We system, including the applicability of such a sys- found that the ensemble visualization methods tem if it were integrated into an atlas construction are helpful in characterizing the shape alignment software. We also asked them to compare the ensemble visualizations to the evaluation techniques they currently use. As we mentioned earlier, the The ensemble visualization methods two main techniques currently used for atlas evaluation are inspecting unaligned structures (when are helpful in characterizing the shape mapped to the atlas space) or analyzing the defor- alignment and provide insights that are mations, quantifying the amount of change necessary to bring an individual ensemble member into useful in understanding the variability alignment. We collected the following observations from in alignment. the participants in this study:

■■ Being able to visualize the extent of the varia- and, furthermore, provide insights that are useful tion among the ensemble of aligned shapes in in understanding the variability in alignment. An terms of quantitative percentile information us- understanding of the type or location of the vari- ing the contour boxplot visualization was help- ability can be helpful in tuning parameters used ful for comparing various atlas-construction in atlas construction and/or removal of outliers to schemes (or comparing atlases that were con- achieve better alignment. structed from different ensembles or parameter We also observed that the contour boxplot settings). The participants also mentioned that emerged as a clear favorite of our participants. the contour boxplot has the potential to help One of the contour boxplot’s salient features that reduce the time needed to gain insights regard- makes it distinct from the other ensemble visu- ing the quality of the atlas. alization approaches is its ability to convey an ■■ State-of-the-art techniques for evaluation and aggregated result from the analysis of all shape visualization of atlases provided limited in- regions in the ensemble on any arbitrary cut formation about the variability that remained plane. For example, visualizing a slice of the in- within an ensemble after transforming it to the tensity image, or contours on a cut plane using atlas space. Deformation and image match ener- the spaghetti plot, conveys the variability for only gies (quantities that are optimized during image the region intersecting the cut plane, whereas a registration in atlas construction) are not able to contour boxplot visualization using the same cut provide insight into the geometric discrepancies plane also provides information about the median that are crucial to understanding atlas quality. and outlier contours that are calculated from a ■■ The contour boxplot’s ability to effectively locate global analysis of contours. and characterize different types of variability The contour boxplot, however, has a drawback: was valuable in atlas construction. it does not give the user much information about ■■ An automated and statistically robust way of specific ensemble members, other than the me- identifying and visualizing outliers in an en- dian or the outliers. For such cases, the spaghetti semble can play a major role in constructing plot with interactivity that allows highlighting an atlas. of specific ensemble members can augment the

IEEE Computer Graphics and Applications 69

g3raj.indd 69 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," Feature Article in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

(a) (b)

Figure 9. Contour boxplot visualizations. (a) Visualization for an ensemble of 100 simulated HIV protein. Here, we see the median contour in yellow and the outlier contours in red. (b) Visualization of the isosurface of a pressure field in a fluid flow. The pressure is considered a function of depth to generate a 3D pressure volume. The median contour is in yellow, and the outlier contours are in red.

contour boxplot by providing more detail if the Another application where the study of shape general user wishes to focus on specific anatomical variability and alignment is of significant inter- areas or members of the ensemble. est is fluid mechanics. In fluid mechanics, when developing models of vortex behavior, scientists often study the variability in the shapes of vortex uture work for our system in the context of structures among different simulations (for ex- Fthe current application includes refining the ample, using slightly different parameter settings system in order to address the study participants’ or boundary conditions) to confirm that their ob- suggestions, such as viewing the specific struc- servations are repeatable,13 rather than a numeri- tures in the context of the whole brain and more cal artifact of a particular simulation. The center interaction options. Furthermore, the ensemble of an eddy corresponds to low pressure values in visualization approaches we discussed here can the flow, and hence studying the pressure field of be integrated into an atlas construction package a fluid flow can help detect the position of the ed- to provide users with the ability to interactively dies and regions of high vortices. For this case, we inspect the shape alignments and the variability used the 2D incompressible Navier-Stokes solver as among ensemble members after atlas construc- part of the open source package Nektar++ (www. tion. Motivated by the feedback from the partici- nektar.info) to generate an ensemble of 28 fluid pants, a more comprehensive study is required to flow simulation runs. These simulations were de- examine the applicability of ensemble visualization signed for a steady fluid flowing past a cylindrical to compare different atlas-construction schemes. obstacle. For each of the ensemble members, we Studying shape variability has applications in randomly perturbed the initial conditions such various branches of science. In molecular dynam- as inlet velocity and Reynolds number. For this ics, researchers study different types of molecu- example, the pressure dependence in the third lar structures and the shapes of their potential dimension was computed analytically. Figure 9b fields in solutions (which vary stochastically) shows the contour boxplot visualization of the to understand, for instance, their biochemical isosurfaces of the pressure volume. properties.12 Scientists are also interested in In addition to those we have showcased here, the evolution of the shape of molecules. For ex- there are many possible applications that could ample, the surfaces of 3D molecular chains are benefit from the contour boxplot summary and of significant interest for comparison of various visualization technique. types of protein structures.12 Figure 9a shows the contour boxplot visualization of the surface of an ensemble of simulated HIV molecules. The ensem- Acknowledgments ble members underwent a Procrustes alignment We thank Avantika Vardhan, Clement Vachet, Ben (translation, rotation, scale) using the positions Galvin, and Sarang Joshi for discussions on atlas con- of the underlying molecules. The potential fields struction and help in processing the brain data we use that form these contours are inherently smooth, here. We also thank Lee Makowski and Jaydeep Bard- and thus there was no need to preprocess this han for providing the HIV molecule data and Shireen volume data. Elhabian for help in processing the HIV molecule data.

70 May/June 2016

g3raj.indd 70 4/19/16 2:15 PM M. Raj, M. Mirzargar, J. S. Preston, R. M. Kirby and R. T. Whitaker, "Evaluating Shape Alignment via Ensemble Visualization," in IEEE Computer Graphics and Applications, vol. 36, no. 3, pp. 60-71, May-June 2016. doi: 10.1109/MCG.2015.70

We received valuable feedback from Theresa-Marie 13. C.H.K. Williamson, “Vortex Dynamics in the Rhyne on improving the visualizations. This work Cylinder Wake,” Ann. Rev. Fluid Mechanics, vol. 28, was supported by National Science Foundation (NSF) no. 1, 1996, pp. 477–539. grant IIS-1212806 and by the National Institute of General Medical Sciences of the National Institutes of Health under grant P41 GM103545-17. Mukund Raj is a PhD student in computing and is affili- ated with the School of Computing and the Scientific Com- puting and Imaging institute at the University of Utah. His References research interests include scientific visualization, graph ana- 1. R.T. Whitaker, M. Mirzargar, and R.M. Kirby, lytics, and statistical analysis. Raj has an MS in comput- “Contour Boxplots: A Method for Characterizing ing from the University of Utah. He is a member of SIAM. Uncertainty in Feature Sets from Simulation Contact him at [email protected]. Ensembles,” IEEE Trans. Visualization and Computer Graphics, vol. 19, no. 12, 2013, pp. 2713–2722. Mahsa Mirzargar is a postdoctoral research associate 2. S. Joshi et al., “Unbiased Diffeomorphic Atlas at the Scientific Computing and Imaging Institute at the Construction for Computational Anatomy,” University of Utah. Her research interests include scientific Neuroimage, vol. 23, no. 1, 2004, pp. 151–160. visualization, statistical analysis, uncertainty quantifica- 3. C.R. Jack et al., “The Alzheimer’s Disease Neuroim tion, and multidimensional signal processing. Mirzargar aging Initiative (ADNI): MRI Methods,” J. Magnetic has a PhD in computer engineering from the University of Resonance Imaging, vol. 27, no. 4, 2008, pp. 685–691. Florida. Contact her at [email protected]. 4. K. Brodlie, R.A. Osorio, and A. Lopes, “A Review of Uncertainty in Data Visualization,” Expanding J. Samuel Preston is a PhD student in computing in the the Frontiers of Visual Analytics and Visualization, School of Computing and the Scientific Computing and Springer Verlag, 2012, pp. 81–109. Imaging institute and a software engineer at the Scientific 5. K. Potter, P. Rosen, and C.R. Johnson, “From Computing and Imaging Institute at the University of Utah. Quantification to Visualization: A Taxonomy of His research interests include scientific visualization and Uncertainty Visualization Approaches,” Uncertainty image analysis. Preston has an MS in computing from the Quantification in Scientific Computing, Springer, University of Utah. He is a member of SIAM. Contact him 2012, pp. 226–249. at [email protected]. 6. J. Cox, D. House, and M. Lindell, “Visualizing Uncertainty in Predicted Hurricane Tracks,” Int’l J. Robert M. Kirby is a professor of computer science in the Uncertainty Quantification, vol. 3, no. 2, 2013, pp. School of Computing and the Scientific Computing and Im- 143–156. aging Institute at the University of Utah. His research in- 7. K. Potter et al., “Ensemble-Vis: A Framework for the terests include high-order methods, scientific visualization, Statistical Visualization of Ensemble Data,” Proc. concurrent programming, and high-performance computing. IEEE Int’l Conf. Data Mining Workshops (ICDMW), Kirby has a PhD in applied mathematics from Brown Uni- 2009, pp. 233–240. versity. Contact him at [email protected]. 8. T. Pfaffelmoser, M. Reitinger, and R. Westermann, “Visualizing the Positional and Geometrical Variability Ross T. Whitaker is the director in the School of Comput- of Isosurfaces in Uncertain Scalar Fields,” Computer ing and a professor in the School of Computing and the Sci- Graphics Forum, vol. 30, no. 3, 2011, pp. 951–960. entific Computing and Imaging Institute at the University 9. K. Pöthkow, B. Weber, and H.-C. Hege, “Probabilistic of Utah. His research interests include image analysis, ge- Marching Cubes,” Computer Graphics Forum, vol. 30, ometry processing, and scientific computing. Whitaker has no. 3, 2011, pp. 931–940. a PhD in computer science from the University of North 10. S. López-Pintado and J. Romo, “On the Concept of Carolina at Chapel Hill. He is an IEEE fellow. Contact him Depth for Functional Data,” J. Am. Statistical Assoc., at [email protected]. vol. 104, no. 486, 2009, pp. 718–734. 11. R.T. Whitaker, “Reducing Aliasing Artifacts in Iso-surfaces of Binary Volumes,” Proc. IEEE Symp. Volume Visualization (VVS), 2000, pp. 23–32. 12. X. Zhang et al., “Application of New Multiresolution Methods for the Comparison of Biomolecular Electrostatic Properties in the Absence of Global Structural Similarity,” Multiscale Modeling and Selected CS articles and columns are also available Simulation, vol. 5, no. 4, 2006, pp. 1196–1213. for free at http://ComputingNow.computer.org.

IEEE Computer Graphics and Applications 71

g3raj.indd 71 4/19/16 2:15 PM