REGISTRATIONSHOP an Interactive 3D Medical Volume Registration Tool

REGISTRATIONSHOP an interactive 3D medical volume registration tool

Master’s Thesis

Berend Klein Haneveld

REGISTRATIONSHOP an interactive 3D medical volume registration tool

THESIS

submitted in partial fulﬁllment of the requirements for the degree of

MASTER OF SCIENCE

COMPUTER SCIENCE

Berend Klein Haneveld born in Alphen a/d Rijn, the Netherlands

Computer Graphics and Visualization Group Department of Intelligent Systems Faculty EEMCS, Delft University of Technology Delft, the Netherlands www.ewi.tudelft.nl

REGISTRATIONSHOP an interactive 3D medical volume registration tool

Author: Berend Klein Haneveld Student id: 1217089 Email: [email protected]

Thesis Committee:

Chair: Prof. Dr. E. Eisemann, Faculty EEMCS, TU Delft University supervisor: Dr. A. Vilanova, Faculty EEMCS, TU Delft Daily supervisor: N.N. Smit MSc, Faculty EEMCS, TU Delft Committee Member: Dr. ir. M. Staring, LUMC, dept. Radiology

Contents List of Figures ix

List of Tables xi

1 Introduction 1

2 Related work 2 2.1 Related applications ...... 2 2.2 Registration frameworks ...... 4

3REGISTRATIONSHOP 4 3.1 User interface ...... 4 3.2 Visualizations ...... 4 3.2.1 Color mapping ...... 5 3.2.2 Threshold visualization (THR) ...... 5 3.2.3 Maximum Intensity Projection (MIP) ...... 6 3.2.4 Multi-volume visualization ...... 6 3.2.5 2D comparison tool ...... 7 3.3 Transformation tools ...... 7 3.3.1 Manual transformation tool ...... 8 3.3.2 Landmark transformation tool ...... 8 3.3.3 Automatic transformation tool ...... 9 3.4 Transformation history ...... 10

4 Implementation 11

5 Evaluation 11 5.1 Case study ...... 11 5.2 General remarks ...... 13

6 Conclusions and future work 13

Appendices 17

A User study design and results 17

vii

List of Figures 1 Screenshot of the main interface of REGISTRATIONSHOP displaying two CT lung datasets (taken from the test suite of Elastix). On the left is the baseline data as fixed volume with its visualization parameters and on the right side the follow-up data as moving volume. Both volumes are visualized with the thresholding technique. In the center the two volumes are combined into a single visualization. 5 2 Sequence of MRI thresholding showing the difficulties of visualizing the internals of a whole body dataset. Figures a-e show the results of gradually increasing the lower threshold (0.0, 0.06, 0.24, 0.48 and 0.62, respectively). Figure f shows the volume clipped with a clipping box to show the internals. Figure g shows an image plane on the intersection with the clipping box. Data is courtesy of dr. M. Reijnierse, MD (Department of Radiology, LUMC)...... 6 3 Examples of possible visualizations in REGISTRATIONSHOP. Figure a on the left shows a single volume rendered with MIP. Figure b in the center combines MIP with THR to render two volumes and figure c on the right shows how two volumes are combined in MIP mode. Figures b and c show that the two volumes are not properly aligned. The volumes are CT lung datasets from Elastix’ test suite that have been masked so that surrounding tissue of the lung does not interfere with the visualization. 7 4 2D comparison tool with the lung datasets...... 8 5 Manual transformation tool shows a 3D interactive box. The center sphere can be used to move the blue volume around, the outer spheres are for scaling and clicking anywhere else in the box will rotate the volume. Below the view a transformation matrix can be specified. With every change in the matrix the updated transformation is immediately propagated to the visualization...... 9 6 On the left side the surface picker is shown (red and green cone), together with an active (green) landmark and an inactive (gray) landmark. On the right the two-step picker is shown right after shooting a ray and rotating the volume to be able to see the ray. Beneath the view is the iso-profile along the ray. The current landmarks and their positions are listed in the center beneath the combined visualization. . 10 7 Example of the history tab showing a manual transformation, a landmark transformation and an automatic transformation...... 11 8 On the left in figure a are the default Elastix parameter files offered by REGISTRATIONSHOP. Figure b on the right shows how the parameters can be edited...... 12 9 Registration of two simple blocks, used to give a quick demo of REGISTRATIONSHOP to the experts. . 17 10 Mean value for each applicant that represents the level of agreement with the proposition, resulting from the combination of the results for the related statements of the propositions...... 18

List of Tables 1 Feature comparison of related applications. The feature list is comprised of features that are deemed desirable for a volume registration tool...... 16 2 Statements about REGISTRATIONSHOP with the answers of the experts ...... 22

REGISTRATIONSHOP: an interactive 3D medical volume registration tool

Berend Klein Haneveld Computer Graphics / Medical visualization department Delft University of Technology: TU Delft [email protected]

Abstract Image registration is a technique where two images or volumes are mapped onto each other. In medical imaging this is used to combine and compare images from patients in order to ﬁnd abnormalities or to track a patient’s treatment effects over time. There are registration algorithms and software packages available, but they are cumbersome and hard to use, making image registration a non-trivial and time consuming task. 2D visualization techniques are often used for visualizing 3D datasets and the results of the registration. This is a problem because one dimension is discarded that could give insight in the relative positioning of the datasets. REGISTRATIONSHOP was developed to improve and simplify the process of volume registration with 3D visualizations and simple interactive tools. REGISTRATIONSHOP provides ways of creating basic visualizations of 3D volumetric data. Interactive rigid and non-rigid transformation tools can be used to manipulate the volumes. REGISTRATIONSHOP provides immediate visual feedback for all the rigid transformation tools. A novel 3D comparative visualization technique is introduced as well as a new way of placing landmarks in volumes. An evaluation was performed where experts found REGISTRATIONSHOP to be a promising tool for registration and visualization of the results.

Index terms— Registration, medical, volume, visualization, interaction

1 Introduction ent information which can help diagnosis when properly combined. 3D imaging techniques like Magnetic Resonance Imag- Another use for registration is to create segmen- ing (MRI) and x-ray computed tomography (CT) are tations. By fully registering a dataset to a dataset of increasingly used to study and diagnose diseases of pa- which an atlas1 is available, the segmentation informa- tients [Smith-Bindman et al., 2012]. tion from the atlas can be transferred to the first dataset. Often follow-up scans are acquired to track the pro- Despite the broad potential use of registration, gression of a disease over time. However, comparing there are no easy-to-use tools available to perform reg- datasets from different moments in time is complicated istrations. There are frameworks and visualization tools because the patient will never be in the exact same pos- available that are able to perform registrations and visu- ture. Different scanners and acquisition techniques can alize the results, but most of these tools are not designed be used which complicate comparisons even further. from the ground up to make registration easy and acces- In order to be able to properly compare the image sible for users. Registration functionality comes often volumes, they need to be aligned so that when the im- in the form of an add-on, such as is the case with Slicer ages are fused together, regions from one image cor- [Pieper et al., 2004]. The main user interface of Slicer is respond with regions from the other. The process of set up to be a platform for a set of distinct tools that give aligning images is called registration. actual functionality to Slicer. The result is that all the Registration is not only useful for fusing same required functionality is there, but it is not optimized to modality images, but also for combining multi-modal perform a certain task. In other words, it is a multipur- images like positron emission tomography (PET)/CT with MRI. Each acquisition technique provides differ- 1An atlas is the labeling of the voxels in a volume. pose tool with which multiple problems can be solved This thesis describes the design and evaluation of and therefore it fails to offer a streamlined solution to a REGISTRATIONSHOP. Section 2 focuses on previous specific task such as registration. and related work. Section 3 describes the design of Because registration is a complex topic and time REGISTRATIONSHOP. Section 4 provides an overview consuming to perform, it is not broadly used in clinical of the implementation details. Section 5 contains an ex- settings. Therefore, the medical imaging community pert review of REGISTRATIONSHOP. Section 6 con- will benefit from the development of an easy-to-use, cludes the thesis and discusses the future work. interactive registration tool that will make performing registrations less complex and time-consuming. These improvements to the registration process will make it 2 Related work more appealing to use in clinical settings. The related work is divided into two sections of which Another problem with the current available soft- the first section lists ten related applications in sec- ware is that the visualizations are often limited to 2D tion 2.1 and the second discusses three related frame- slices of the volumes, while registering volumes is an works in section 2.2. intrinsic 3D operation. Discarding this third dimension makes a serious appeal on the user to reconstruct a mental model of the needed changes in alignment, in order 2.1 Related applications to make the image data match. A perfectly aligned result in a 2D slice might be misaligned in 3D. Therefore The following list of applications all have registration a 3D representation is better suited for giving the user functionality, either rigid, non-rigid or both. Each insight into how the data are spatially related in the full application will be reviewed in respect to how this 3D context. functionality is exposed. Some of these applications The main goal of REGISTRATIONSHOP is to make are not freely available, so for those applications all the registration an easier task by improving user interaction information is gathered from online available manuals possibilities and providing visual feedback. By building and videos. a single purpose tool that is specifically designed for AMIDE Mirada Medical performing registrations, the complexity of the whole · · AMILab PMOD Image Fusion (PFUS) process can be reduced and therefore registrations will · · FusionSync Slicer be easier for users to perform. · · Ezys VolView · · The contributions of REGISTRATIONSHOP to the MITK Voreen medical imaging community are as follows: · · A theoretical comparison between all these appli- REGISTRATIONSHOP features interactive trans- cations and REGISTRATIONSHOP has been made by · formation tools that provide users with instant user comparing the available features. The results can be feedback for evaluating the results of a registration. inspected in table 1. (Section 3.3) REGISTRATIONSHOP introduces a novel 3D com- · AMIDE. AMIDE is a Medical Image Data Examiner parative visualization technique where two Maxi- and can be used for viewing, registering and analyz- mum Intensity Projection (MIP) visualizations are ing medical imaging datasets [Loening and Gambhir, combined, inspired by the 2D fused views in- 2003]. The registration tools available are based on troduced by Dzyubackyk et al. [2013]. (Section fiducial markers (landmarks) and maximization of mu- 3.2.4) tual information. Only rigid body transformation is sup- REGISTRATIONSHOP provides a novel way of · ported. Placing fiducial markers needs to be done be- placing landmarks in 3D volumes in two steps with fore starting the “alignment wizard” and corresponding accompanying interactive visualization. (Section markers are required to have the same name. Modify- 3.3.2) ing the transformation matrix for each volume can be REGISTRATIONSHOP makes performing registra- · done in the rotate section of the “modification dialog”. tions easier (Section 5) and serves as the start of The transformation can also be altered by readjusting an open-source registration platform for future en- the markers and running the alignment tool again. 2 deavors to improve the registration process . Single volume rendering as well as multi-volume rendering is supported, but the technique is CPU based 2The source of REGISTRATIONSHOP can be viewed and downloaded at https://github.com/berendkleinhaneveld/ and thus real-time interaction with the volume render is Registrationshop severely limited.

2 AMILab. AMILab is an image processing applica- non-rigid registration methods. Local registration is tion with visualization capabilities [Krissian et al., supported. Mirada Medical contains extensive measure- 2011]. The interface is built around a console view with ment tools for performing detailed studies. toolbars that contain scripts for certain tasks and tools. Some examples of the tools are a calculator, a timer and PMOD Image Fusion (PFUS). PMOD’s fusion tool a torus creator. It is a multipurpose tool and thus it is is a commercial application and offers rigid and non- not focused on performing one task well. Every feature rigid registration methods [PMOD Technologies Ltd.]. seems to be tagged on. Like FusionSync it offers alignment of images by drag- Multi-volume rendering is possible, but the inter- ging with the mouse. The interface is complicated and face leaves much to be desired. For instance, there are offers little explanation of functionality. multiple ways of loading the datasets and assigning it to the ﬁxed and moving volumes, but some methods Slicer. Slicer is a software platform that enables the fail and result in errors. AMILab also supports inter- user to inspect, process and visualize 3D image data active transformation of the volumes, according to the [Pieper et al., 2004]. Most of the functionality comes buttons in the interface, but the transformation box does through the use of modules that implement certain func- not show up. Another problem is that AMILab does not tions and applications for the data. Some of the features provide a way of saving any projects or results. are visualization (2D slices, 3D volume rendering), registration (rigid and deformable) and interactive segmen- FusionSync. FusionSync is a commercially avail- tation. The structure of the program results in fragmen- able visualization and registration application [Chi- tation of the functionality and the user interaction leaves maera], only available for the aycan workstation OsiriX much to be desired. It used to support multi-volume Pro. FusionSync offers 2D visualizations of multi- rendering in version 3.63, but this feature has been re- ple datasets and an interactive rigid registration method moved since version 4.0. where images can be simply dragged around with the mouse. It also features deformable registration meth- VolView. VolView is an open-source, advanced vol- ods. FusionSync is a dedicated application for diagnos- ume visualization tool [Kitware, Inc.]. Scientiﬁc and ing multi-modal and follow-up series of medical data. medical volumes can be loaded next to each other for comparison, but it lacks multi-volume render possibil- Ezys. Ezys is a non-linear 3D medical image regis- ities. It does offer correlation based rigid registration tration program [Gruslys et al., 2012]. It features 2D and multi-modality registration methods. visualizations of multiple datasets. Non-rigid registrations are GPU accelerated and make use of CUDA, so a Voreen. Voreen is an open source rapid application NVIDIA graphics card is required for running this ap- development framework for the interactive visualiza- plication. With Ezys the deformations can be shown as tion and analysis of multi-modal volumetric datasets a 2D overlay and it is possible to animate interactively [Meyer-Spradow et al., 2009]. Multi-volume rendering between the original and deformed data. There are no is possible through the inclusion of a template project, interactive transformation tools included. The interface but the only way of interactively registering multiple is aimed at technical users. volumes is through editing transformation matrices.

The aforementioned tools each have their speciﬁc MITK. MITK is a framework for building medical strengths and shortcomings which can be seen in table 1 imaging applications [Wolf et al., 2004]. It comes with where a feature comparison is provided. The features in MITK workbench which offers functionality in the form the table are deemed as desirable features to have in a of modules. There is a deformable registration module volume registration tool. The table is not an exclusive available that is based on ITK, but this module is not list, but serves as an indication for the state of the cur- distributed with the default builds. In order to use this rently available tools. module, users have to build and run MITK themselves. Of the non-commercial tools, Slicer and Ezys are the most interesting because they offer the most fea- Mirada Medical. Another commercially available tures. But as mentioned before, Slicer is not speciﬁcally application is Mirada Medical [Mirada Medical Ltd.]. 3 The main visualization method makes use of fused 2D Multi-Volume GPU Ray Casting support for Slicer is described on the following web page: https://www. visualizations. Mirada Medical offers a manual interac- slicer.org/slicerWiki/index.php/Modules: tive rigid registration method and automatic rigid and VolumeRendering-Documentation-3.6, February 2014

3 build for performing registrations and therefore is not purposes. easy to use. Ezys is interesting because while it lacks 3D visualization and rigid registration methods, it has Of these frameworks Elastix has been chosen for GPU accelerated non-rigid registration and it features REGISTRATIONSHOP because of its features and the visualizations of the deformation field. Especially the permissive license. More details about this choice can visualization of the deformation field is interesting as it be found in section 4 where some of the implementation is the only application that has this feature. details are discussed. The commercial tools all have the advantage that they have been build as a dedicated registration tool, whereas most non-commercial tools can be described as 3REGISTRATIONSHOP volume visualization tools that are extended with regis- The next section describes the design of REGISTRA- tration functionality, the sole exception being Ezys. TIONSHOP by explaining the main features. The fol- Table 1 makes clear that there is no tool available lowing list gives an overview of the features that will be yet that combines all the desired features. REGISTRA- discussed: TIONSHOP serves as a starting point to fill that gap. User interface ...... 3.1 · 2.2 Registration frameworks Visualizations ...... 3.2 · Color mapping ...... 3.2.1 The following section describes the frameworks that · Threshold visualization (THR) ...... 3.2.2 have been considered for integration into REGISTRA- · Maximum Intensity Projection (MIP) ....3.2.3 TIONSHOP for enabling automatic rigid and non-rigid · Multi-volume visualization ...... 3.2.4 registration methods. · 2D comparison tool ...... 3.2.5 · Transformation tools ...... 3.3 · ITK The Insight Segmentation and Registration Manual transformation tool ...... 3.3.1 · Toolkit (ITK) is an open-source, cross-platform seg- Landmark transformation tool ...... 3.3.2 · mentation and registration framework for multidimen- Automatic transformation tool ...... 3.3.3 · sional data. ITK is implemented in C++ but it is possi- Transformation history ...... 3.4 ble to integrate ITK into Python and Java applications · by creating Python and Java wrappings. From version 4.0 ITK is released with an Apache 2.0 license which is 3.1 User interface a permissive license. This means that anyone can freely REGISTRATIONSHOP is designed to support registra- use and modify the code and can be used for commer- tion of only two volumes at a time. Using this constraint cial purposes. it was a logical step to have two views that each show one of the two volumes: the fixed and the moving vol- Elastix Elastix is another open-source library with ume. Another view should display both datasets at the which rigid and non-rigid registrations algorithms can same time so that the spatial relationship between the be executed and compared [Klein et al., 2010]. It is data can be inspected. based on ITK, so it shares most of its functionality. As Thus the main interface of REGISTRATIONSHOP with ITK, it is possible to include Elastix as a frame- has three main regions: on the left is the fixed volume, work into other applications. Elastix can also be run as on the right the moving volume and in the center is the a command-line tool. Elastix is distributed under the combination of the two volumes. Underneath each of simplified BSD-license which is a permissive license the volume views are tabs that show controls and infor- like the Apache 2.0 license. mation related to the view above. An overview of the interface can be found in figure 1. DROP / MRF Unlike Elastix, DROP is not based on The user can create visualizations on the left and on ITK and uses Markov Random Fields (MRF) to opti- the right side, which are combined in the center view. mize for cost functions [Glocker et al., 2008]. Using The center view has extra parameters for modifying the MRFs has the advantage of not having to compute a combined visualization. derivative of the cost function, which saves computa- tion time. Compared to the gradient descent method 3.2 Visualizations used in Elastix it performs well and is a promising alternative toolkit. The license is not permissive as it REGISTRATIONSHOP offers a selection of visualiza- may be used only for research and non-commercial tion techniques in order to visualize volumetric datasets.

4 Figure 1: Screenshot of the main interface of REGISTRATIONSHOP displaying two CT lung datasets (taken from the test suite of Elastix). On the left is the baseline data as ﬁxed volume with its visualization parameters and on the right side the follow-up data as moving volume. Both volumes are visualized with the thresholding technique. In the center the two volumes are combined into a single visualization.

The next sections describe the available visualization This is used in the 2D comparison tool and in the multi- techniques and discusses how the user can interact with volume visualization when two MIP visualizations are these visualizations. combined.

3.2.1 Color mapping 3.2.2 Threshold visualization (THR) Throughout REGISTRATIONSHOP the fixed volume is The threshold visualization (THR) makes use of Di- shown as orange (1, 1/2,0 in RGB space) and the mov- rect Volume Rendering (DVR). DVR basically traces ing volume is shown as a shade of blue (0, 1/2,1 in RGB rays through the volume and accumulatively builds up space). This helps the user to identify the volumes in all a color value along these rays according to a transfer of the visualizations. These specific colors were cho- function. A transfer function defines the color and the sen because blue and orange can be distinguished even opacity for each voxel. Creating transfer functions is a by most colorblind people4. Another feature is that in non-trivial procedure and can be a time consuming task. RGB color space these colors are complementary col- In order to make it as simple as possible for a user ors, which means that adding these colors results in a to work with volume rendering, only three parameters shade of gray whenever the two colors are of the same of the transfer function are exposed to the user: lower intensity. When the intensity is not equal, the result will and upper thresholds and opacity. Only the voxels that show a shade of blue or orange, making it possible for have values between the lower and upper threshold will the viewer to spot differences in intensity and know- show up in the render and the opacity slider influences ing which volume produced the higher intensity values. how opaque those voxels will appear. Figure 1 shows an example of DVR visualizations 4Only people with monochromatic vision will have trouble dis- cerning the different volumes. Color schemes for colorblind people of thresholded data. On the left side is the fixed volume can be checked with a tool created by Henderson [2000]. data displayed in orange color and on the right side is

5 (a) (b) (c) (d) (e) (f) (g) Figure 2: Sequence of MRI thresholding showing the difficulties of visualizing the internals of a whole body dataset. Fig- ures a-e show the results of gradually increasing the lower threshold (0.0, 0.06, 0.24, 0.48 and 0.62, respectively). Figure f shows the volume clipped with a clipping box to show the internals. Figure g shows an image plane on the intersection with the clipping box. Data is courtesy of dr. M. Reijnierse, MD (Department of Radiology, LUMC). the moving volume data displayed in blue color. tion. This makes the MIP visualization more flexible The number of visualization controls are limited in and useful in some use cases where high iso-values in order to simplify user interaction as much as possible. the data are not the values of interest. By not giving full control over the visualization some datasets will become hard to visualize in a meaningful 3.2.4 Multi-volume visualization way. For example: it is hard to visualize a MRI dataset with just the threshold parameters because iso-values in The aim of the view in the middle is to combine the MRIs do not have a one-to-one correspondence with the visualizations that are shown in the adjacent views. The different tissues in the body which makes it hard to filter available visualization types are THR and MIP, so there with threshold values. Figure 2 gives an example of the are 3 sets of combinations that can be made: THR-THR, problem with thresholding a MRI dataset. THR-MIP, and MIP-MIP. To facilitate viewing the inside of a dataset a clipping box can be used to cut away parts of the data. THR-THR This mode simply copies the transfer Slices can be displayed on the outside of the clipping functions from the adjacent views and applies it to the box to provide a familiar 2D visualization of the inter- volumes. The only parameter that can be adjusted is the nals. In order to prevent the slices from completely oc- opacity for each of the volumes in order to blend them cluding the 3D view, only the pixels are drawn that rep- together. resent visible voxels from the volume. An example of During ray tracing, voxels from both volumes are clipped data with and without a superimposed slice are sampled alternately, starting with the fixed volume. The shown in figures 2f and 2g respectively. technique for compositing the sampled values is the same as the technique used for standard ray tracing. An example is shown in figure 1. 3.2.3 Maximum Intensity Projection (MIP) Another simple way of creating a visual impression of MIP-THR When MIP and THR visualizations are a 3D volume is to use Maximum Intensity Projection combined, values from the two volumes are stored dur- (MIP). This technique shows only the voxels with the ing tracing: the current maximum intensity for the MIP highest intensity along the rays from the camera. The volume and the accumulated color and opacity value for usual parameters are window width and level which ba- the THR volume. These values are composited when sically control brightness and contrast. Most clinicians the opacity value for the THR volume becomes satu- are used to work with window width and level controls rated. An example is shown in figure 3b. on slices by right clicking and dragging horizontally and vertically respectively. Figure 3a shows a MIP visual- MIP-MIP The combination of two MIP visualiza- ization of a masked CT lung dataset. tions uses a technique similar to the image blend- As with THR, it is possible to threshold the data so ing technique introduced by Dzyubackyk et al. [2013]. that some voxels are excluded from the MIP visualiza- Both of the volumes are separately traced with the MIP

6 (a) MIP (b) MIP and THR (c) MIP and MIP

Figure 3: Examples of possible visualizations in REGISTRATIONSHOP. Figure a on the left shows a single volume rendered with MIP. Figure b in the center combines MIP with THR to render two volumes and figure c on the right shows how two volumes are combined in MIP mode. Figures b and c show that the two volumes are not properly aligned. The volumes are CT lung datasets from Elastix’ test suite that have been masked so that surrounding tissue of the lung does not interfere with the visualization. technique. The result of the fixed volume is transformed and expertise. Therefore REGISTRATIONSHOP needs to orange while the result of the moving volume is trans- to offer a familiar visualization method, so that people formed to blue. These values are then added in RGB can get accustomed to the new visualization techniques space. Whenever the values are of similar intensity, a while having the familiar visualization methods avail- gray value is produced. Whenever one of the traces has able at any time to be able to verify the results acquired a higher intensity than the other, that color will shine using the new visualization techniques. through. So when the voxels of the volumes match well, Another reason why 2D visualization needs to be the resulting image will exist of mainly gray values. At present, is to provide a means for detailed inspection of regions with differences in intensity either blue or or- the datasets. 2D visualization provides a direct view of ange will be shown, immediately drawing the attention the data without removing any information, while for of the viewer. This can be seen in figure 3c. a 3D visualization the data needs to be filtered to only This fusion technique is mainly applicable to show the most relevant parts. The detailed parts of the datasets that have the same modality, because the tech- data are often the parts that are filtered out in order to nique is dependent on the similarity of the data. And show the main anatomical structures in the dataset. because this technique is dependent on the maximum Because 2D visualization is not the main focus for values along the ray, differences in lower value ranges the thesis, the 2D comparison tool is limited in visual- are harder to spot. ization and interaction functionality. The layout of the Another problem is that MIP removes any depth views is the same as in the 3D viewer with the fixed vol- perception, removing the ability to perceive the rotation ume on the left, the moving volume on the right and the of the volume. This problem with depth and rotation combined volumes in the center, as can be seen in fig- can make it seem like the volumes overlap well from ure 4. The slices of the volumes are linked as well as the a certain angle while from another angle it is apparent window and level parameters which can be changed by that the volumes do not match well. Interaction with the dragging with the right mouse button. Slicing through visualization is therefore needed to verify the result. the volumes can be done by scrolling.

3.2.5 2D comparison tool 3.3 Transformation tools

Domain experts working with image volumes are used There are three tools in REGISTRATIONSHOP for trans- to inspecting their data with slices through the volume. forming the data: the manual transformation tool, the Providing only 3D volume visualization tools to these landmark transformation tool and the automatic trans- users will discard most if not all of their experience formation tool. What follows is a description of the

7 Figure 4: 2D comparison tool with the lung datasets. functionality of each tool and the interaction possibil- 3.3.2 Landmark transformation tool ities. The landmark transformation tool is meant to obtain a registration that can be more accurate than a manual 3.3.1 Manual transformation tool transformation. By placing landmarks in both volumes a transformation can be calculated that will try to make The purpose of the manual transformation tool is to the landmarks overlap as much as possible using a least coarsely align the data so that the transformation can squares fit. The user can specify whether only trans- be used as initialization for an automatic registration. lation and rotation may occur (Rigid body), or transla- Automatic registration may not always be successful tion, rotation and isotropic scaling (Similarity), or trans- in finding a correct initialization because the automatic lation, rotation, scaling and shearing (Affine). registration might fail in some cases because of the Placing landmarks in a volume is not a trivial pro- lack of overlap between the datasets or in some cases cedure: there are three dimensions in a volume which might become stuck in a local optimum. For this rea- are projected onto a two dimensional screen. The inter- son, manual initialization is needed to provide a means action is possible only in two dimensions as the mouse to present the automatic registration method with a rea- location gives just a 2D coordinate. Due to possible sonable starting point. transparency of voxels, it is not possible to know ex- The manual transformation tool offers a way of actly what voxel the user is pointing at when pointing at simply clicking and dragging the data in order to cre- a 2D image result of DVR. To solve this problem REG- ate a rigid registration. ISTRATIONSHOP offers two ways of helping the user to When a manual transformation is started, an in- place landmarks at the right depth in the volume: the teractive box is shown in the center view where both surface picker and the two-step picker. datasets are shown, as can be seen in figure 5. By dragging the box the moving data can be rotated. Click- Surface picker The surface picker is a tool that traces ing and dragging the spheres on the sides will stretch a ray through the volume originating from where the the data in the corresponding direction and dragging mouse hovers over the volume. It uses the opacity trans- the sphere in the center will move the data around in fer function to evaluate the opacity of the voxels along a plane parallel with the camera view plane. The result- the ray. As soon as the picker encounters a voxel that ing transformation will be shown in the matrix below has a higher opacity value than a certain threshold value the center view. It is also possible to edit the matrix by (specifically 0.05), it will show two cones in the vol- hand. Each adjustment in values will be immediately ume pointing from two opposite sides at that voxel. The propagated to the visualization. two cones are rotated along the estimated normal at that

8 for instance occlusion, the two-step picker can help to place the landmark at its intended depth. The concept of the two-step picker is that a point in three dimensions can be specified by two intersecting lines. So, the user first has to point at the desired location from one perspective, effectively specifying a ray through the volume that intersects with the intended location. Then the user has to view the volume from a different perspective to specify the intended depth along that first ray, using the closest point on the first line to the second line, indicated by the mouse. To help the user in defining the correct depth from the second perspective a line is drawn through the volume along the first ray. A sphere is used to indicate the current depth along the first line. When the user moves the mouse, the sphere follows by relocating to the closest point on the first line to the virtual line specified by the mouse. At times this sphere might be hard to trace in the actual visualization. In order to help the user with specifying the correct depth, a histogram of the iso-values along the first ray is presented below the volume view. The histogram is shown in figure 6 on the right side. The histogram gives the user information about the composition of the tissue along the ray and this additional context can aid the user in specifying the correct depth. The visualization of the sphere in the volume and Figure 5: Manual transformation tool shows a 3D interactive box. The center sphere can be used to move the circle in the histogram are linked together; the user the blue volume around, the outer spheres are for scal- can specify the depth either in the volume or in the his- ing and clicking anywhere else in the box will rotate the togram view. Changes in either widget are propagated volume. Below the view a transformation matrix can be to the other widget. specified. With every change in the matrix the updated transformation is immediately propagated to the visual- Whenever a land- ization. Visualization of the landmarks mark is specified, the location is marked with a sphere and a circle around the sphere. The sphere is drawn point in the volume, so that one of the cones appears in the context of the volume render while the circle is to stick out at the surface while the other cone seems drawn in an overlay, so that it is always visible, no mat- to disappear behind the surface. The cones are ren- ter from which perspective the user looks at the volume. dered in the same context as the volume and thereby The landmarks have to be specified in pairs in order give the user a sense of the depth of the voxel that is be- for the algorithm to determine which points correspond ing pointed at. On the left in figure 6 the surface picker to each other. If no landmark pair is active when the is shown. user specifies a landmark, a new active pair is created The locator will follow the mouse movements of with one empty landmark. As soon as there is a new the user by tracing through the volume. When the lo- complete landmark pair, a new transformation is com- cator points at the desired position, the user can press puted and instantly shown in the multi-volume view. the space bar to place a landmark at that position. The The landmarks that are active are green while the in- space bar is used instead of a mouse click because in active landmarks are gray. The landmarks are displayed this way the mouse can still be used to navigate around in both the single and the multi-volume views. the volume. 3.3.3 Automatic transformation tool Two-step picker Whenever the visualization makes it The automatic transformation tool is a wrapper around hard to use the surface picker successfully because of the command-line version of Elastix. It provides de-

9 Figure 6: On the left side the surface picker is shown (red and green cone), together with an active (green) landmark and an inactive (gray) landmark. On the right the two-step picker is shown right after shooting a ray and rotating the volume to be able to see the ray. Beneath the view is the iso-profile along the ray. The current landmarks and their positions are listed in the center beneath the combined visualization. fault parameter files to start the registration process with When the user applies the parameters, Elastix will which automatic registrations such as affine, b-spline be called in a subprocess. When Elastix successfully and rigid body can be performed. Instead of choos- finishes the registration, the newly created dataset will ing one of the default parameter presets, custom pa- be automatically loaded as moving volume in REGIS- rameter files can be loaded into REGISTRATIONSHOP TRATIONSHOP. so that parameters can be reused. If a rigid transformation has been produced by REGISTRATIONSHOP, it will pass this transformation as an initial transformation 3.4 Transformation history to Elastix. The history tab keeps track of all the transformations by The parameters can be edited right within REGIS- listing all the transformation steps of the manual, land- TRATIONSHOP after loading a parameter file. Before mark and automatic transformation tools. Clicking on a applying the registration the parameters are saved in the transformation will show the result of that transforma- project folder so that the parameters can be checked af- tion in the multi-volume view. This makes it possible to ter an automatic registration has been performed. This compare intermediate registration efforts and evaluate parameter file can also be loaded again in REGISTRA- whether the registration is improved by certain trans- TIONSHOP in order to improve on the last automatic formation steps. An example of the current implemen- registration effort. tation is shown in figure 7. By adjusting the files in the resource folder of REG- At the moment the history section is limited to sav- ISTRATIONSHOP the default parameter files can be ad- ing the current dataset and transformation matrices of justed. Every text file with parameters in that folder will the rigid transformation tools, but the intended func- be listed in REGISTRATIONSHOP. This enables the user tionality also encompasses keeping track of landmark to define a custom set of default parameter files. information. Being able to view all the meta data from

10 it is possible to target all major platforms (OS X, Linux and Windows). Standard VTK does not support rendering of multiple volumes. Therefore a custom extension of VTK has been used that was created by Krissian and Falcon-´ Torres [2012]. The code has been adjusted and extended to support all the possible combinations of DVR and MIP visualizations5.

5 Evaluation Figure 7: Example of the history tab showing a manual A user study has been performed to evaluate if REG- transformation, a landmark transformation and an auto- ISTRATIONSHOP meets its goals of making the regis- matic transformation. tration process easier and more efﬁcacious. Efﬁcacious in this context stands for the ability to produce a de- the registration steps makes it possible to verify the re- sired or intended result. In other words, is REGISTRA- sults of a registration. TIONSHOP more effective than other applications in helping the user to successfully perform registrations?

4 Implementation 5.1 Case study REGISTRATIONSHOP is constructed with the following The case studies design follows the design described languages and libraries: by Yin [1994]. The complete user study design can be found in appendix A. The main study question has been Elastix ...... Registration framework · formulated as follows: “Does RegistrationShop make Python ...... Main programming language · performing registrations easier and more efficacious by Pytest ...... Testing framework · providing 3D visualizations of the datasets in combina- Visualization toolkit (VTK) ...... C++ library · tion with simple interactive tools with instant user feed- PySide ...... Python bindings of Qt · back?” vtkMultiVolRen ...... Multi volume rendering · The main question was decomposed into a number of propositions. These propositions were investigated Elastix was the framework of choice as it entails during four case studies with a medical imaging and ra- all the features of ITK and more. Another reason is diation specialist, a radiologist and two image process- that Elastix can be used as a command-line tool which ing experts, who are all familiar with image registration. makes it straightforward to use from a scripting envi- The next section discusses each proposition and the re- ronment such as Python. REGISTRATIONSHOP incor- sponse of the experts. porates Elastix by offering a user interface for loading and editing parameters that are saved to file and then REGISTRATIONSHOP is easy to learn and use com- passed on to the command-line version of Elastix. pared to other existing tools. The reason to choose Elastix over DROP is that The main layout of the tool was much appreciated Elastix is distributed under a permissive license, which by one of the domain experts as it was deemed to be allows future commercial endeavors by anyone to take clear and concise. The biggest hurdle to overcome for REGISTRATIONSHOP or its components and make it the experts was the navigation in 3D; all of the experts into a commercial application. had little to no previous experience of working with 3D Some other technologies that have been used to visualizations. Nevertheless one of the experts stated construct REGISTRATIONSHOP are Python, the Visual- that he would not need much more time to become ade- ization Toolkit (VTK) and PySide (Python bindings for quately acquainted with the controls. Qt). Python was chosen for its rapid development and prototyping capabilities. Pytest was used for the test The side-by-side views with the available visualiza- driven development of the non-interface classes. VTK tion methods, such as DVR, MIP and clipping planes, was chosen because of its volume rendering support and 5The modifications have been made publicly available its capability for loading medical datasets. PySide was at https://github.com/berendkleinhaneveld/ used for building the user interface because with PySide vtkMultiVolRen/tree/RegistrationShop

11 (a) Default parameter ﬁles (b) Example of parameter edit view

Figure 8: On the left in figure a are the default Elastix parameter files offered by REGISTRATIONSHOP. Figure b on the right shows how the parameters can be edited. help to bring out the internal structure of the datasets The instant feedback helped to indicate whether the and create meaningful visualizations. interactions with the interactive box were helping the All of the experts found the 3D visualizations inter- registration. The opinions about the interactive box of esting because it provided a new way of inspecting the the manual transformation tool were mixed: one of the data. One of the experts remarked that he appreciated experts had trouble operating the box while the others that he could see anatomical structures as 3D objects in- found it quite easy to work with. The experts that found stead of having to construct a mental model from slices. it easy used the box only for translation and scaling. Another expert was able to tell from the 3D visualiza- The other expert tried to rotate the dataset with the box, tions that one of the heads of the whole body datasets but the box did not behave as expected and therefore was tilted backwards, which complicated registering the she was unable to use the manual transformation tool heads onto each other. successfully. Although there were not many problems with ma- Placing landmarks in a 3D volume is made easy nipulating the visualization with the available controls, with the provided tools, because of the 3D visualization creating meaningful visualizations was found to be combined with instant visual feedback. hard. All of the experts are used to looking at their data The surface picker was found to be easy to use with 2D slices. Thus the 3D visualizations were hard to whenever the visualization showed a clear surface. The interpret for the experts, although it was appreciated in two-step picker was appreciated by all experts that had some occasions. the time to use it (three out of four), especially because The 3D visualizations of the datasets combined into of the iso-value profile that was shown underneath the a single view help the user to understand how the two 3D visualization. This made it possible for the experts datasets relate to each other spatially. to place landmarks at locations where the visualization The combined 3D visualization was successful in lacked clarity. showing the spatial relationship of the two datasets to Adjusting the visualization in order to expose the the experts. It enabled them to see the improvements part of the data where a landmark was to be placed, made with the rigid transformation tools. The 3D vi- was experienced as complicated because of unfamiliar- sualization is not sufficient though for a detailed voxel- ity with the 3D visualization and navigation. Most ex- by-voxel comparison, according to the experts. This in- perts would also want an easy way to put landmarks on dicates that the 3D visualization needs to be extended 2D slices for improved precision. One of the experts re- or that a 2D comparison method is required in addition marked that the glyph representing a landmark obstruct to the 3D visualizations. the actual location of the landmark in the volume. This could be overcome by decreasing the size of the sphere The manual transformation tool makes it easy to to have the approximately the same volume as a single coarsely align the datasets by providing instant user voxel representation. The locator of the surface picker feedback. suffers the same problem as the current landmark rep-

12 resentation. Instead of using two cones, a small sphere Another reason why the 2D comparison tool was appre- with a surrounding circle could be used. The sphere ciated is that it provides a familiar way of inspecting the then indicates the location while the circle gives an in- data. dication of the estimated normal at that location. The transformation tools were mostly successful in helping the experts to create registrations. Especially REGISTRATIONSHOP can be used without needing the iso-value profile visualization of the two-step picker detailed technical knowledge of creating visualizations was found to be helpful. and transformations. Some problems with the transformation tools were Two experts indicated that they believed that they encountered during the evaluation. Rotating the data did not need to have any extra technical knowledge with the interactive box of the manual transformation in order to operate REGISTRATIONSHOP. The other tool is not intuitive and the interactive transformation two experts already possessed the necessary technical matrix is regarded as too technical and unnecessary. knowledge. The visualization of the surface picker and the landmarks need to be improved so that they do not obstruct REGISTRATIONSHOP provides all the tools that are needed to perform a registration. the indicated location. One of the experts remarked that REGISTRA- One of the experts missed the ability to visualize TIONSHOP is a great tool for people that are inexpe- the deformation field calculated for non-rigid registra- rienced at performing registration and to those who tions. Another expert missed the ability to make mea- only need to perform registrations irregularly. Another surements and statistical reports of the datasets. A third expert remarked that with previously used registration expert would like to see interactive segmentation as a software he had to work with it for four days before he visualization option. The last expert agreed with the got anywhere, while with REGISTRATIONSHOP he was proposition because Elastix can be used through REG- impressed how quickly he was able to get results. ISTRATIONSHOP, thus it provides all the needed tools. All in all, the case study shows that the experts find REGISTRATIONSHOP makes it easier to work with REGISTRATIONSHOP to be a promising volume regis- Elastix. tration tool that potentially will make registration more accessible. The 3D visualizations are appreciated be- Only two of the experts had experience with cause it enables the visualization of anatomical struc- Elastix. The inclusion of default parameter sets as a tures, yet the unfamiliarity with navigation in 3D cre- starting point was regarded as very useful. One of the ates a hurdle for the experts to overcome. And while experts did not think REGISTRATIONSHOP improved the 3D visualization is suited for displaying anatomical his interaction with Elastix as he was highly experi- structures, it is less suited for detailed voxel-by-voxel enced with the software, but he remarked that REGIS- inspection and comparison of the datasets. TRATIONSHOP would be a great help for people that have little to no experience with performing a registration. The benefit of using REGISTRATIONSHOP over 6 Conclusions and future work Elastix is that the results of an Elastix run can be visu- ally inspected immediately after completion. In this work REGISTRATIONSHOP has been presented as a medical volume registration tool that makes use 5.2 General remarks of 3D visualizations to visualize the internal anatomical structures and contains simple interactive tools that Overall REGISTRATIONSHOP was well received by the provide instant user feedback. experts. The interface was praised by one of the experts A novel 3D comparative visualization was intro- for its simplicity and clear layout. The 3D visualiza- duced to show the user where the data overlaps and how tions of the datasets were regarded as interesting and the registration can be improved by fusing two MIP vi- succeeded in providing a sense of the anatomical struc- sualizations on top of each other. ture of the data. The combined viewer was success- A new way of placing landmarks has been intro- ful in showing the spatial relation between the datasets, duced with the two-step picker. The interactive visual- especially because of the instant user feedback during ization of the iso-values along the ray of the picker was interaction with the transformation tools. The 3D visu- received as very useful by the experts and made it easy alization is not sufficient for analyzing the registration for them to place landmarks at the intended locations. in detail. All experts agree that a 2D visualization is re- REGISTRATIONSHOP shows potential in quired for detailed inspection; consequently, the inclu- making the process of registration less com- sion of the 2D comparison tool was deemed necessary. plex and more compelling to inexperienced

13 users. REGISTRATIONSHOP has been made References publicly available at https://github.com/ berendkleinhaneveld/Registrationshop Chimaera. FusionSync. URL http://www. in order to serve as starting point for future endeavors chimaera.de/chimaera/applications/ to improve the registration process. fusionsync.html. Last visited on 3 March To further strengthen REGISTRATIONSHOP in its 2014. quest to make the registration process easier, the manual O. Dzyubackyk, J. Blaas, C. P. Botha, M. Staring, transformation tool needs to be improved. The current M. Reijnierse, H. Bloem, R. Van der Geest, and tool is sufficient for scaling and translating the data, but B. P. F. Lelieveldt. Comparative Exploration of the interaction for rotating the data needs to be changed Whole-Body MR through Locally Rigid Transforms. or new interaction widgets could be created in the vein International journal of computer assisted radiology of 3D modeling and animation tools such as Maya, and surgery, 8(4):635–647, 2013. Blender and 3ds Max. Such widgets are not presently available in the standard VTK library and would be a B. Glocker, N. Komodakis, G. Tziritas, N. Navab, and welcome addition to the set of interactive widgets. N. Paragios. Dense image registration through MRFs Another important aspect to improve is the visual- and efficient linear programming. Medical Image ization of the landmark glyphs and the surface picker. Analysis, 12(6):731–741, December 2008. The current visualizations occlude the actual location and this can be overcome by rendering a dot in the vol- A. Gruslys, S. Sawiak, and R. Ansorge. 3000 Non- ume with a small circle around it that indicates the es- rigid medical image registrations overnight on a sin- timated normal at that point in the volume. The center gle PC. In IEEE Nuclear Science Symposium Con- dot indicates the actual voxel while the circle around it ference Record, pages 3073–3080, 2012. serves to place the dot in the context of the volume. C. Henderson. Color vision, 2000. URL http:// The user study brought to the attention the limita- www.iamcal.com/toys/colors/. Last vis- tions of the current 3D visualizations in displaying de- ited on 3 March 2014. tails of the datasets. This could be improved by devising a way of visualizing a region of interest in more detail, Kitware, Inc. VolView. URL http://www. like providing a magnifying lens tool. A starting point kitware.com/opensource/volview. could be the work of Wang et al. [2005]. Such a lens html. Last visited on 3 March 2014. could also be used to visualize metrics like mutual information in order to provide the user with extra infor- S. Klein, M. Staring, K. Murphy, M. A. Viergever, and mation about the registration results. J. P. W. Pluim. Elastix: A toolbox for intensity-based medical image registration. IEEE Transactions on At the moment REGISTRATIONSHOP cannot visu- Medical Imaging, 29(1):196–205, 2010. alize the deformation field of a non-rigid registration. This could be a great improvement in helping the user K. Krissian and C. Falcon-Torres.´ GPU volume evaluate the results of a non-rigid registration. High val- ray casting of two volumes within VTK, Decem- ues in the deformation field could indicate regions of in- ber 2012. URL http://hdl.handle.net/ terest where either the registration needs improvement 10380/3350. Last visited on 3 March 2014. or important differences in the data can be seen. The application Ezys already provides 2D visualizations of the K. Krissian, F. Santana-Jorge, D. Santana-Cedres,´ deformation field which could serve as a starting point C. Falcon-Torres,´ S. Arencibia, S. Illera, A. Tru- for a 3D version [Gruslys et al., 2012]. jillo, C. Chalopin, and L. Alvarez. AMILab soft- Another improvement would be to make it possible ware: Medical image analysis, processing and visu- to point at a certain voxel in the volume and automati- alization. volume 173, pages 233–237, 2011. cally perform a local rigid registration to make the vol- A. M. Loening and S. S. Gambhir. AMIDE: a free soft- umes align at that point in the datasets. This idea was in- ware tool for multimodality medical image analysis. troduced successfully by Dzyubackyk et al. [2013] and Molecular imaging, 2(3):131–137, 2003. could help with quickly aligning datasets at points of interest. J. Meyer-Spradow, T. Ropinski, J. Mensmann, and With the mentioned improvements in place, REG- K. Hinrichs. Voreen: A rapid-prototyping envi- ISTRATIONSHOP could serve as a starting point for ronment for ray-casting-based volume visualizations. bringing registration into the clinic by making the pro- IEEE Computer Graphics and Applications, 29(6):6– cess of registration easier and more insightful. 13, 2009.

14 Mirada Medical Ltd. Medical imaging software for radiation oncology and diagnostic radiology. URL http://www.mirada-medical.com. Last visited on 3 March 2014. S. Pieper, M. Halle, and R. Kikinis. 3D Slicer. In Biomedical Imaging: Nano to Macro, 2004. IEEE International Symposium on, pages 632–635 Vol. 1, April 2004. doi: 10.1109/ISBI.2004.1398617. PMOD Technologies Ltd. PMOD Image Fusion. URL http://www.pmod.com/technologies/ products/image_fusion/image_fusion. php. Last visited on 3 March 2014. R. Smith-Bindman, D. L. Miglioretti, E. Johnson, C. Lee, H. S. Feigelson, M. Flynn, R. T. Greenlee, R. L. Kruger, M. C. Hornbrook, and D. Roblin. Use of diagnostic imaging studies and associated radiation exposure for patients enrolled in large integrated health care systems, 1996-2010. JAMA, 307(22): 2400–2409, 2012. L. Wang, Y. Zhao, K. Mueller, and A. Kaufman. The magic volume lens: An interactive focus+context technique for volume rendering. In Proceedings of the IEEE Visualization Conference, page 47, 2005. I. Wolf, M. Vetter, I. Wegner, M. Nolden, T. Bottger,¨ M. Hastenteufel, M. Schobinger,¨ T. Kunert, and H. P. Meinzer. The Medical Imaging Interaction Toolkit (MITK) - A toolkit facilitating the creation of interactive software by extending VTK and ITK. In Pro- ceedings of SPIE - The International Society for Op- tical Engineering, pages 16–27, 2004.

R. K. Yin. Case Study Research: Design and Methods. Thousand Oaks, CA: Sage, 1994.

15 Table 1: Feature comparison of related applications. The feature list is comprised of features that are deemed desirable for a volume registration tool.

HOP S

EGISTRATION Feature R AMIDE AMILab FusionSync Ezys MITK workbench Mirada PMOD - PFUS Slicer VolView Voreen 3D visualization of datasets 3 3 3 7 7 3 3a 7b 3 3 3 Multi-volume render 3 3 3 7 7 3 7 7b 7 7 3 Compare / inspect results 2D / 3D 2D / 3D 3D 2D 2D 7 2D 2D (3Db) 2D 7 3D Clipping of 3D datasets 3 3 3 3 7 7 3 3 3 3 7 Interactive manual transformation tool 3 7 3c 3 7 7 3 3 3 7 3d Interactive landmark transformation tool 3 3e 7 7 7 7 7 3f 3 7 7 Non-rigid registration 3 7 7 3 3 7g 3 3h 3 7 7 7 7 7 7 3 7 7 7 7 7 7 16 Visualization of deformation GPU accelerated registration 7 7 7 3 3 7 7 7 7 7 7 History of transformations 3 7 7 7 7 7 7 7 7 7 7 Dedicated tool for registrations 3 7 7 3 3 7 3 3 7 7 7 Easy / intuitive loading of datasets 3 7 7 3 3 3 3 3 3 3 7 Easy export of registration results 3 7 7 3 3 3 3 3 3 3 7 Save / load projects 3 3 7 3 7 3 3 3 3 3 3 Freely available 3 3 3 7 3 3 7 7 3 3 3 Supported platforms (OS X , Linux , Windows )

aThe only 3D visualization method available for Mirada is MIP. bAvailable in external P3D tool. cFunctionality is broken in current version of AMILab (3.2.1). dVoreen only offers an interactive transformation matrix. eAMIDE only offers a non-interactive landmark transformation wizard. fLimited to only one set of landmarks. gNon-rigid registration module is not included by default in MITK workbench. hNon-rigid registration only available for brain normalization. H REGISTRATIONSHOP makes it easier to work with Appendices Elastix. (Easier / more efﬁcacious)

A User study design and results Note that proposition H is only applicable for the experts who have experience with Elastix. The next section contains the detailed description of Yin’s method specifies that for each separate case, the user evaluation study following the development of these propositions need to be researched. I decided that REGISTRATIONSHOP. The design follows the guide- in order to be able to answer these propositions in de- lines by Yin [1994] for case studies. The main research tail I would have to decompose them further into more question is posed as follows: detailed statements. Another reason to create more detailed statements is that I wanted to have more detailed Does REGISTRATIONSHOP make performing feedback on specific features which I would not have registrations easier and more efficacious by been able to get by providing only the propositions. By providing 3D visualizations of the datasets obtaining the opinion of experts on these statements, the in combination with simple interactive tools propositions can be examined. The derived statements with instant user feedback? are listed in table 2. These propositions were investigated during four The key phrases contained in the question have case studies with a medical imaging and radiation spe- been emphasized and with these key phrases in mind, cialist (JOL), a radiologist (MOR) and two image pro- the following propositions have been constructed: cessing experts (BES and MAS). Two of the experts (JOL and MOR) were given A REGISTRATIONSHOP is easy to learn and use two whole-body MRI datasets to register and the other compared to other existing tools. (Easier / more two experts (BES and MAS) were given lung CT scans efficacious) (baseline and follow-up, taken from the Elastix repos- 6 B The side-by-side views with the available visual- itory ). Each of the experts had 20-40 minutes to use ization methods, such as DVR, MIP and clipping REGISTRATIONSHOP for creating visualizations and planes, help to bring out the internal structure of registering the volumes. Before that a five minute demo the datasets and create meaningful visualizations. was given by me where I quickly showed how to regis- (Easier / more efficacious, 3D visualizations) ter two simple blocks which can be seen in figure 9. After their time with REGISTRATIONSHOP, the ex- C The 3D visualizations of the datasets combined perts were given the chance to express their level of into a single view help the user to understand how agreement to the statements with the help of a 5-point the two datasets relate to each other spatially. (3D 6The lung datasets 3DCT lung baseline.mha and visualizations) 3DCT lung followup.mha can be downloaded from https://github.com/mstaring/elastix/tree/ D The manual transformation tool makes it easy to master/src/Testing/Data. Datasets used in the study coarsely align the datasets by providing instant are from commit 6510952a1a4b02d295b73d9d1b4303b40f6119c3 user feedback. (Simple interactive tools, Instant on November 14, 2013. user feedback)

E Placing landmarks in a 3D volume is made easy with the provided tools, because of the 3D visualization combined with instant visual feedback. (3D visualizations, Simple interactive tools, In- stant user feedback)

F REGISTRATIONSHOP can be used without needing detailed technical knowledge of creating visualizations and transformations. (Easier / more efﬁcacious, 3D visualizations, Simple interactive tools)

G REGISTRATIONSHOP provides all the tools that Figure 9: Registration of two simple blocks, used to are needed to perform a registration. (Easier / give a quick demo of REGISTRATIONSHOP to the ex- more efﬁcacious) perts.

17 A REGISTRATIONSHOP is easy to learn and use compared to other existing tools.

JOL: JOL found the overall interface of REGISTRA- TIONSHOP to be clear and concise. He appreciated that there were not a lot of sub-menus to be found, so he had the feeling that within a few hours he would be accustomed to most of the capabilities of the software. He was not used to navigation in 3D, but felt that he would need just a little more practice to get fully accustomed to the controls.

MOR: MOR did not have trouble with the inter- Figure 10: Mean value for each applicant that repre- face layout of REGISTRATIONSHOP but MOR did have sents the level of agreement with the proposition, result- some trouble with the navigation of the 3D visualization ing from the combination of the results for the related because she was not used to it. The interactive 3D box statements of the propositions. also gave her much trouble, because it did not operate as she expected it to.

Likert-scale, ranging from strongly disagree to strongly BES: BES was quick in learning the interface, but agree. This value serves as an indication of their opin- had trouble with the 3D navigation. He was quick to ion. The experts could also elaborate on the reason for learn how to adopt the visualizations, but mentioned their opinion for a more detailed opinion, because au- that he would appreciate having linked visualizations as dio and screen recordings were made of the complete he had to copy every visualization parameter by hand. evaluation session. He also had some trouble because of some interface el- Table 2 contains the answers of the experts on the ements stealing mouse interaction. For instance, if the statements. Note that because of time constraints not left mouse button is clicked on a slice in the 3D viewer all features of REGISTRATIONSHOP could be tested in it is not possible to rotate the camera anymore. This all cases, so that is why the answers are blank in some hinders navigation and makes the interaction experience spaces. Question 13-16 were Elastix related, so only the inconsistent. two experts who had experience with Elastix (BES and MAS) were in the position to answer those questions. Question 22 was not answered by BES and MAS be- MAS: MAS seemed to adapt quickly to REGISTRA- cause they already have the advanced technical knowl- TIONSHOP and get the idea of the application. He did edge and skills and would not speak for other people but not need a lot of instruction and was quick in discover- themselves. ing the functionality of REGISTRATIONSHOP. Because of the relation between the propositions B The side-by-side views with the available visual- and the statements it is possible to calculate a value ization methods, such as DVR, MIP and clipping from the Likert values for the statements. This value planes, help to bring out the internal structure of then gives an indication of the level of agreement of the datasets and create meaningful visualizations. the experts with the propositions. The statistical signifi- cance of the results is lacking due to the limited number JOL: JOL liked the layout with the views side by side of participants in the study. The values therefore only to the center view. The 3D visualization was new to him serve as a rough indication and are not listed in a table. He had trouble creating a meaningful visualization of Instead figure 10 shows a graph of the calculated values the MRI data with the thresholds. per applicant. 3D navigation was hard for him to get used to. JOL did not have the feeling that he could navigate the vi- Propositions sualizations very well, although he attributed it to his unfamiliarity with tools like REGISTRATIONSHOP that The following section lists the propositions and elabo- have 3D visualizations. He had the feeling that with an rations for every expert. The elaboration is constructed extra 30 minutes he would have got the hang of it. from the audio and screen recording material, the an- He would like to have the possibility of linked swers to the statements and some open questions. views, because he found that it was hard at times to find

18 his way in the datasets. JOL was not familiar with the combination of 2D and 3D in order to become a viable term opacity which made him reluctant to try to adjust tool in professional settings. that parameter. He mentioned that he would appreciate a reset button for the visualization parameters. C The 3D visualizations of the datasets combined into a single view help the user to understand how MOR: During adjusting the visualization MOR the two datasets relate to each other spatially. missed the possibility of resetting the visualization. This made experimenting with the parameters feel less JOL: JOL found that the combined viewer was suffi- safe. MOR only had time to work with the threshold cient in showing the displacement between the two vol- visualization. Because she had to work with the whole umes. But for a more detailed analysis he appreciated body MRI data, she appreciated that there was a clip- that a 2D comparison tool was available. ping box with which the internals of the dataset could be shown. The downside was that the clipping box has the same interaction as the box of the manual transform MOR: MOR found the visualization effective in tool, which she thoroughly disliked. showing whether the volumes were matching or not. But she believed that it could be improved by show- BES: With the visualization options BES managed to ing where the volumes overlap. Only the threshold vi- find vascular intersections for placing landmarks, but he sualization method was used during the experiment in did not prefer the 3D visualization compared to the 2D which overlapping regions are not directly visible. The visualization that he is used to. MIP-MIP combination in which this is possible was not He remarked that if the two volumes are similar shown because of time constraints. that it would be nice to link all the visualization parameters for easier and faster creation of the visualization. BES: BES did not expect that the combined 3D visu- He was also surprised to see that the lower and upper alization would be able to show him much, but while us- threshold settings were not shared between the MIP and ing the manual transform tool he found that it was suc- DVR visualizations. BES wanted to move lower and cessful in showing whether the registration improved upper threshold sliders together so that it is possible to from his transformation efforts. inspect the data by making a window for iso-values and He found it hard to visualize what he hoped to see moving that window from low to high values. with the 3D visualization tools as he was looking for A reset button for the visualization and an interac- detailed sections in the lung dataset. He mentioned that tive orientation cube for the camera would have been for coarsely aligning the datasets the viewer is effective appreciated by BES as he found that he lost himself at and sufficient, but for showing voxel by voxel differ- times because of unfamiliarity with the 3D navigation. ences it is not adequate. He also missed the possibility An orientation cube would be handy for picking a cer- of viewing the deformation field of a non-rigid registra- tain perspective on the data. Another thing that might tion as that could show him which parts of the registra- help him with inspection of the data is to link the cam- tion might need more work or show him actual differ- era settings of all the views. ences in the datasets.

MAS: MAS found the 3D visualization the most interesting part of REGISTRATIONSHOP, because he is MAS: MAS found the 3D visualizations to be very not used to this type of visualization. He especially interesting as it enabled him to view the anatomy in a liked that the 3D visualization gave him a sense of the natural way. But he also stated that thinks that thresh- anatomy in the dataset. He believes that 3D visualiza- olding and MIP are not enough for all use cases. For tion deﬁnitely has its part to play in future medical vi- instance, the most important parts to look at in the lung sualization endeavors and regards the work in REGIS- dataset are small differences which are hard to visualize TRATIONSHOP as a step in the right direction. But at with a 3D technique because often these parts are ﬁl- the moment it still has some problems that need to be tered out as being noise. So 3D helps in understanding overcome before it can be accepted by clinicians and spacial relationships, but 2D visualization is still needed researchers as a viable solution. for detailed analysis. The most jarring problem is that the visualization lacks accuracy so this needs to be overcome or be com- D The manual transformation tool makes it easy to plemented by integrating 2D visualizations in a solid coarsely align the datasets by providing instant way. REGISTRATIONSHOP needs to improve on the user feedback.

19 JOL: JOL found the manual transformation tool to MOR: MOR found the surface picker easy to oper- work well and was successful in using it for quickly ate, but remarked that the current visualization of the aligning the datasets. surface picker obscures the actual place where the landmark is placed which makes it more difﬁcult to accu- MOR: MOR was actively hindered by the 3D inter- rately place landmarks. Also the picker was a little slow active box to do what she intended to do. Rotating the to react during tracing of the volume which hindered her box was found unintuitive and hard to get used to. She with placing landmarks because she had to wait after especially disliked that it was not directly possible to ro- each mouse movement for the locater to catch up. tate in the current view plane or in one of standard view During the experiment there was no time to use the planes (axial, coronal and sagittal). two-step picker.

BES: BES did not find it hard to work with the inter- BES: BES liked working with the two-step picker: active 3D box as it enabled him to make a quick first the line combined with the iso-value histogram were alignment that would have been good enough as initial- much appreciated. After shooting a few rays he found ization for a non-rigid registration. He had no problems that his confidence in correctly placing landmarks grew. with rotation as it was not necessary in his case to rotate BES was looking for certain anatomical points in the data. The instant user feedback helped him in notic- the lungs, namely points where the vessels split. He ing that the registration was improved by his actions. had some trouble finding corresponding splits in both He did not appreciate the possibility of filling in a volumes which caused him to switch focus between the transformation matrix. Not only were there no labels to two views before being certain that he had found the describe the axes, he also believed that it required too right spot for a landmark. This might have been just be- much technical knowledge. cause of unfamiliarity with the 3D visualization, but it might also have been a side effect of using MIP for visu- MAS: MAS believed that the interactive box was easy alization because all depth perception is lost. Therefore to use and found that it was easy to make an alignment rotation of the volume is needed to be certain that you that is accurate enough to be used as initial transforma- are looking at a branch split at the right depth in the tion for an automatic registration. volume. He regarded the interactive transformation matrix as too technical and not helpful for making a registra- MAS: The two-step picker was appreciated by MAS, tion. especially because of the iso-values shown in the ray profile. After setting a few points his confidence in the Placing landmarks in a 3D volume is made easy E tool grew. He disliked the locater of the surface picker with the provided tools, because of the 3D visual- as it blocked his view of what voxel he would place ization combined with instant visual feedback. his landmark on. He also mentioned that the current visualization of a sphere as a landmark also obscured JOL: JOL found the surface picker to be needing the exact location of the landmark. some work. Especially the visualization of the locater because the red cone is often occluding the region of interest. At first he had some trouble with understanding Everyone: Everyone seemed to appreciate the direct the two-step picker, but when he got the idea behind it, updates in the center view. This helped the users to eval- he started to like it. Especially the ray profile with the uate whether enough landmarks were placed or more iso-values along the ray was appreciated as it gave him landmarks were needed for a better registration. confidence that the picked point was at the right depth in the volume. F REGISTRATIONSHOP can be used without needing He found the way of specifying landmark sets to be detailed technical knowledge of creating visualiza- too strict by constraining him to work in a certain way tions and transformations. and would like to see the possibility of adjusting the corresponding points after setting the landmarks. This JOL: JOL found it easy to operate REGISTRA- would add the needed flexibility so that the order of TIONSHOP and reckoned that he needed just a little placing landmarks doesn’t matter. He also was disap- more time to get used to navigation in 3D. He did pointed to notice that landmarks are currently not saved not think he needed more technical knowledge of the with the transform, so editing landmarks after applying visualization methods in order to operate REGISTRA- a transform is not possible yet. TIONSHOP successfully.

20 MOR: MOR found that she did not need extra tech- MOR Not familiar with Elastix. nical knowledge in order to operate the parts of REG- ISTRATIONSHOP that were used during her evaluation BES: BES is familiar with Elastix and especially with session. the available tool in MeVisLab. He deemed it smart to include default parameter sets as a starting point for BES: BES would not answer this question as he al- working with Elastix. ready has the needed technical knowledge and did not want to speculate for people with less technical knowl- MAS: Because of the extended experience of MAS edge. with Elastix, he found that it did not make working with Elastix easier for him, but he reckoned that it would help MAS: MAS would also not answer this question for inexperienced users and irregular users to use Elastix the same reasons as BES. for deformable registrations. He also liked having a standard set of parameter presets to choose from as a G REGISTRATIONSHOP provides all the tools that starting point. are needed to perform a registration. He appreciated that REGISTRATIONSHOP can immediately show the results of the registration performed JOL: JOL found that there was not much missing in by Elastix. order to perform a registration, but he did miss the ability of analyzing the statistical results of registration. In order to make a report on a registration, certain metrics and measurements are used to give a numerical idea of the success of a registration.

MOR: What MOR would like to see in REGISTRA- TIONSHOP is the possibility of interactively segmenting the data for visualization purposes. Clicking a bone for example should only visualize the bones and ﬁlter everything else. And to further help with her work ﬂow automatic measurements could be made of certain seg- mentations such as tumors. Currently she has to take measurements manually in three dimensions of tumors. Automating this task for multiple volumes would help in speeding up the analysis of the progress of a tumor.

BES: BES would like to see the deformation ﬁeld that is produced by non-rigid registrations, because the deformation ﬁeld can give insight into interesting parts of the registrations.

MAS: Because REGISTRATIONSHOP includes Elastix MAS has everything he needs to be able to perform a registration. What he would wish for is that REGISTRATIONSHOP improves on the combination of 2D and 3D visualizations, as that would combine the beneﬁts of both visualization techniques: 3D for main anatomical reference and 2D for detailed inspection of the data.

H REGISTRATIONSHOP makes it easier to work with Elastix.

JOL Not familiar with Elastix.

21 Related Statement JOL MOR BES MAS Mean proposition 1. Navigation through the data is quick and easy. A, F, B 4 2 2 3 2.75 2. The controls for navigation are clear and quick to learn. A, F 4 4 2 3 3.25 3. Inspecting the data is an easy task with the available visualiza- A, B 4 4 2 4 3.5 tion options. 4. The 3D visualizations on the side help to show the internal B 5 1 2 5 3.25 structures and dimensions of the datasets. 5. It is easy to create a meaningful visualization in 3D with the B, A, F 5 1 3 2 2.75 available visualization tools. 6. The interactive box facilitates easy and rapid initial alignment D, A, G, F 4 1 4 4 3.25 of the datasets. 7. The matrix is a useful optional tool for making specific trans- D, F, G 2 1 1.5 ¬ formation adjustments, because it instantly shows result of any modifications. 8. The result of the manual transformation is good enough to be D, H, G 5 4 4 5 4.5 used for initialization of a non-rigid registration. 9. With the surface picker, landmarks can be rapidly placed at the E, G 3 3 2 3.5 2.875 desired locations. 10. The 3D visualization methods (MIP, clipping box, clipping E, G 4 4 4 3 3.75 planes) enable placing landmarks at the desired locations. 11. The immediate updates in the combined viewer show whether E, G, B 4 4 5 5 4.5 the placed landmarks are effective. 12. The Two-step picker enables placing landmarks at points where E 4 5 5 4.67 the surface picker would fail because of the visualization (occlusion, MIP, limitations of thresholding). 13. Creating an initial transform for Elastix was never as easy as H, G 2.5 2.5 with REGISTRATIONSHOP. 14. Loading custom parameter files make it easy to reuse previous H, A 5 5 efforts. 15. The default options are a great starting point for a deformable H, F 4 5 4.5 registration. 16. Editing parameters in REGISTRATIONSHOP is easier than to H, A 4 1 2.5 edit them by hand. 17. The combined 3D visualization makes it possible to evaluate C, G, E 5 4 2 5 4 whether a registration was successful. 18. The 2D comparison tool is an indispensable / crucial / necessary G 5 4 4 5 4.5 tool for evaluating the results of a registration. 19. Because of the 3D visualization there is much less need for a C 2 1 1 1 1.25 2D comparison method. 20. The interactive previews of the transformations in the history G 3 4 4 5 4 list are a useful tool in evaluating whether a transformation makes an improvement to the registration. 21. REGISTRATIONSHOP provides all the tools that are needed to G 2 2 2 5 2.75 perform a registration. 22. No extra technical skills and concepts are needed to be able F, A 4 4 4 operate REGISTRATIONSHOP. 23. I can foresee myself using REGISTRATIONSHOP in the future. A, G 3 3 4 5 3.75 3.9 2.9 3.1 3.7 3.45 Table 2: Statements about REGISTRATIONSHOP with the answers of the experts