Rendering and Interaction in Co-Operative 3-D Image Analysis

Rendering and interaction in co-operative 3-d image analysis

Klaus Toennies, Manfred Hinz, Regina Pohle

Computer Vision Group, ISG, Department of Informatics, Otto-von-Guericke-Universitaet Magdeburg, P.O. Box 4120, D-39016 Magdeburg, Germany

ABSTRACT Most interactive segmentation methods are defined at a 2-d level requiring sometimes ex- tensive interaction when the object of interest is difficult to be outlined by simple means of segmentation. This may become tedious if the data happens to be a 3-d scene of which a part of the differentiating features of a structure is buried in 3-d contextual information. We pro- pose a concept of extending interactive or semi-automatic segmentation to 3-d where coherence information may be exploited more efficiently. A non-analysed scene may be evaluated using volume rendering techniques with which the user interacts to extract the structures that he or she is interested in. The approach is based on a concept of a co-operative system that communicates with the user for finding the best rendition and analysis. Within the concept we present a new method of estimating rendering parameters if the object of interest is only partially analysed.

Keywords: segmentation, image analysis, 3-d rendering, volume rendering, transfer function

1. INTRODUCTION Image segmentation is needed in medical image analysis in applications that range from computer-aided diagnosis to computer-guided surgery. Segmentation often includes part of the analysis because it may focus on extracting a specific object from the background or cate- gorising each object in a two- or three-dimensional scene. In such cases the process is guided by a-priori knowledge with information on the expected appearance of the structures that shall be segmented. As a consequence, a wide variety of segmentation methods based on different models has been developed. Knowledge may be very general such as the homogeneity criterion of a region-based segmentation scheme [1] or rather specific to certain structures such as in an atlas-based anatomy recognition scheme [2]. The former is appropriate if additional information for structure recognition is supplied at a later stage, whereas the latter requires specific adaptation to certain objects to be extracted [3] or a sophisticated learning strategy for acquiring the information for adaptation [4]. In either case, segmentation implies a high cost of adaptation, training, or interaction. It is not an appropriate technique if a medical procedure requiring quantification is carried out routinely, yet it does not justify the high costs. Thus, segmentation in clinical practice is sometimes carried out with inappropriate tools because time and economic constraints do not allow for development of a suitable method or the training of the user. With the recent advent of multi-slice CT scanners as well as with the still increasing use of MRI as tool in diagnosis and treatment planning, the number of images has increased sub- stantially. Quantification through segmentation will support evaluation of such images only if a new class of segmentation tools can be created that are easily adaptable to a wide variety of applications. Adaptation and use of the method should be intuitively understandable to a user who may be unfamiliar with principles and techniques of computer-based image analysis. Furthermore, the adaptation should not require training of the method on large data sets. Any such strategy will include a substantial amount of interaction by which model information is entered into the analysis process. Interaction has to be kept simple and intuitive in order to be easy and efficient to use. This is the case if the interaction follows a metaphor for working with a real structure that is familiar to the user and if the system reacts according to his expectations. For images such as CT or MRI – being the prime target of our work – this analysis has to be carried out on the three-dimensional data set because a single slice often does not reveal sufficient information that can be used for interpretation of the three-dimensional structure in all slices. As information reduction from three to two dimensions during rendering requires prior analysis, a co-operative approach is necessary enabling the user to carefully reduce information and change the data-to-rendition mapping until the structures of interest have been found.

2. CO-OPERATIVE 3-D IMAGE ANALYSIS Co-operative 3-d image analysis allows the user to work with the data without the need for a computer scientist. It extends the scientific visualisation concept. Scientific visualisation for data analysis consists of two parts: • A data representation that may be defined as a vector-valued function on 3-d or 4-d space. • A visualisation technique that assigns a meaning to the data that, in turn, can be mapped onto a rendition. In scientific visualisation, the mapping of the data onto a rendition has to be defined prior to display[5]. A data specialist who knows about the potential interpretation of the data and a visualisation specialist who knows about the mapping from data space to rendering space co- operate in order to decide on the appropriate rendering technique. In co-operative 3-d image analysis, the system replaces the visualisation specialist as a co-op- Co-operative 3-D Image Analysis eration partner. An interaction com- ponent is added for enabling the data interaction specialist to communicate with the changes representation scientific visualisation system (see adds Figure 1). First concepts were pre- information displays sented in [6] for a virtual reality sce- integrates nario. Recent advances in hardware rendition developments for the PC market integration (such as Mitsubishi’s VolumePro board) make it likely that such tech- Figure 1: Co-operative 3-d image analysis adds niques could soon be realised on a interaction to the rendition process in scientific low-cost basis. visualisation. 2.1 Data Representation, Display and Interaction A representation for co-operative image analysis is a superset of the data that is acquired from the imaging device. We previously presented a voxel representation that is able to store this information in an efficient fashion by separating data and display information from each other [7]. Each voxel in this representation may contain a number of function values (from the same or different sources), derivatives of the function values (e.g., gradient) or a number of class labels (potentially associated with probabilities). The representation has a descriptor to describe content of and access to its information. The class labels in the voxel representation are containers for information that is generated during analysis. Known class memberships can be used as part of the display process. Display of the data may be carried out using volume rendering methods [7]. Volume rendering is particularly useful because it takes into account that every point in a 3-d space may potentially contribute to the display. Using emission-absorption volume rendering for structures without explicit surface and combining it with reflection-transmission volume rendering for (partially) extracted surfaces allows for a very flexible tool that may be used for display- ing information at various stages of interpretation showing volumes and surface simultane- ously (see Figure 2). If a suitable mapping between data values and opacities for the volume rendering is chosen then structures will visible even though no explicit segmentation has taken place. However, except for trivial cases the object class membership of a voxel cannot be uniquely determined from the data value. The rendition may provide the user with a quick overview on the data but the final object extraction requires additional interaction between user, data representation and the rendering. Interaction is carried out on the rendition but it influences the data in the representation. It should exhibit a behaviour that is familiar to the user. Earlier approaches mimicked surgical procedures (such as the virtual biopsy in [8]) but we think that much more general concepts may be appropriate for interactive segmentation. Useful metaphors for interaction are: • Desktop metaphor: The user has various trials for segmentation open at the same time. He has tools for integrating, selecting and discarding intermediate results. • Dig and excavate metaphor: The structure of interest is buried in the data. The user has tools for removing superfluous parts of the data. Recognition of structures is guided by information from the data (e.g., gradients indicating boundaries between objects). • Sculpting metaphor: The structure is partially recovered from the data. Tools for

Figure 2: Rendition of the ventricular system from CT using a suitable transfer function (left) and after user interaction (3-d picking and region growing) (right). changing the shape of the structure, for chopping away or gluing together parts of it help to find the final structure. Again, information from the data supports this process. The desktop metaphor is probably the easiest way to create a co-operative analysis system, but it does not make use of human experience on manipulating 3-d objects. Excavating and sculpting require much more computational power eventually integrating visual and haptic feedback in a real-time environment. Within these metaphors different types of interaction may be used: • Binary operations separate parts of a highlighted structure from the remainder of it. Picking followed by region growing, such as the one shown in Figure 2 (right) is an example of such operations. Other operations could be a sculpting procedure or the employment of crop planes for cutting off parts of a structure. • Fuzzy operations assign certainty or uncertainty to class membership. Variations of the transparency at object boundaries through the transfer function is an example of such an operation. Changes of texture or colour for indicating a fuzzy class membership would also constitute a fuzzy operation. If the operation is intended to change a structure’s class membership, the result must be integrated into the representation. Interaction does not always have to happen in real time. If the user expects a discontinuous change such as it is the case for picking a structure, then de- lays between interaction and reaction of the system are acceptable. However, real time operations are necessary if, as it is the case for a sculpting procedure, continuity of action needs to be conveyed to the user. The latter currently poses a big burden on the computing power of a system because a volume rendering method that deals with non-segmented data cannot make use of computation time optimisation measures that rely on separation of the data into impor- tant and background data.

2.2 Computation of the Transfer Function Varying the transfer function enables the user to interact with the data such that he or she may select the structure that he wants to find. Finding the transfer function that maps data values to rendition effects is difficult if it precedes data analysis as required by the co-operative approach. The correctness of the transfer function can be verified only based on the rendition itself. Using an incorrect transfer function may lead to unsatisfactory visualisation that does not give hints regarding changes of the correct rendering function. A technique is needed that makes use of the user’s expectations regarding the content of the data set and fills in information that pertains to image generation technique. Previous methods for finding the transfer function (such as [10]) rely on a statistical evaluation of all grey values in a certain neighbourhood. However, the co-operative concept assumes that the user already knows of what he or she wants to see. Not using this information will increase the number of trials for finding the correct transfer function for the structure of interest. We developed a semi-automatic approach that requests the user to depict a structure that he or she wants to display in a slice of the data. It is assumed that the structure-of-interest is visible in this slice although its appearance may vary over slices. Computation of the transfer function involves computation of an upper and lower boundary that bound the range of grey values for that structure. It is assumed that the object exhibits a certain homogeneity. It may be defined by an unknown variation of the grey values around an unknown mean. The user is not expected to know about mean or variation of grey values in the general case of volume visualisation. We applied a self-adaptive region growing method for estimating the grey level range given the depicted location in the structure [11]. As the mapping is applied to all voxels regardless of whether they belong to the structure or not, the user is also asked to point out a circular region-of-interest so that potentially hiding objects are cropped (see Figure 3). This process generates a homogeneity criterion for a region of unknown size, shape and grey level distribution provided that the grey level distribution deviates from that of adjoining structures (see [12] for details). Homogeneity parameters are acquired during a random walk around the initial seed location with a level of confidence in the current estimate that increases with increasing sample size so that voxels not belonging to the region are recognised by their deviation from this estimate.

3. RESULTS The method was successfully applied for segmenting the kidney from CT and MR images as well as the aorta from contrast-enhanced MRI where density variations of the tissue make defining a transfer function based on a single slice difficult. Interaction consisted of defining a circular region of interest in a single slice that included the object to be reconstructed. A voxel was selected within the region that belonged to the structure in order to compute its homogeneity criterion. The diameter was taken as a diameter of a three-dimensional sphere that enclosed the object of interest. Thus only three mouse clicks were necessary to gather the information for creating renditions of the objects. Results of the rendition, that are seen in Figure 3, show that it is possible to create valid displays of structures based on homogeneity criteria that had not to be specified by the user.

Figure 3: Rendition of the right kidney from CT and of the aorta abdominalis from MRI. The inset shows the slice in which the location and region-of-interest for computing transfer function and rendition were depicted. 4. CONCLUSIONS Co-operative 3-d image analysis is a concept for enabling flexible segmentation of structures in 3-d medical images that employs facilities of fast 3-d rendering algorithms for exploiting 3-d coherence and homogeneity features in combination with intuitively understandable interaction techniques. For being fully useful it requires real-time rendering techniques for true volume rendering that is currently only available on specific computers. How- ever, recent advancements in hardware developments make it likely that low-cost PC-based systems will be available in the near future that will make the concept useful within a routine clinical work environment. It will be necessary, however, to research hardware requirements for a co-operative 3-d analysis environment in order to stipulate or encourage developments that cater to the specific requirements of interactive volume rendering from raw data. Analysis with interactive support on renditions of 3-d data sets can be a powerful concept provided that depictions are interpretable at any stage of the analysis without the support of a visualisation expert. It requires new methods for rendering, data representations, and interaction. First results show that the concept can be filled with life thus opening the field of image analysis for a new technique for versatile analysis methods.

REFERENCES 1. Y. L. Chang, X. Li, Adaptive image region-growing. IEEE Trans. Image Processing, 3(6), 1994, pp.868-872. 2. M. Kaus, S. K. Warfield, F. A. Jolesz, R. Kikinis, Adaptive template moderated brain tumor segmentation in MRI. Bildverarbeitung in der Medizin, Springer, 1999, pp.102-105. 3. T. F. Cootes, A. Hill, and C. J. Taylor. Medical image interpretation using active shape models: Recent advances. 14th Conf. on Information Processing in Medical Imaging, France, June 1995, pp.371-372. 4. J. Weng, A. Singh, and M. Y. Chiu. Learning-based ventricle detection from cardiac MR and CT images. IEEE Trans. Medical Imaging, 16(4), 1997, pp.378-391. 5. S. G. Eick, Visual discovery and analysis. IEEE Trans. Visualization and Computer Graphics, 6(1), 2000, pp.44-58. 6. R. van Teylingen, W. Ribarsky, C. van der Mast, Virtual data analyzer. IEEE Trans. Visualization and Computer Graphics, 3(1), 1997, pp.65-74. 7. K. D. Toennies, S. Bock, Exploration of 3D medical data using combined emission and reflection volume rendering. Proc. SPIE (Medical Imaging 1996), 2707, 1996, pp.44-51. 8. K. D. Toennies, C. Derz, Volume rendering for interactive 3D segmentation. Proc. SPIE (Medical Imaging 1997), 3031,1997, pp. 602-609. 9. K. D. Toennies, U. Tronnier, 3d modeling using an extended cell enumeration representation. ACM Computer Graphics, 24(5), 1990, pp. 13-20. 10. S. Fang, T. Biddlecome, M. Tuceryan, Image-based transfer function design for data exploration in volume visualization. Proc. IEEE Visualization 98, 1998, pp.319-326. 11. R. Pohle, K. D. Toennies, Segmentation of medical images using adaptive region growing. Proc. SPIE (Medical Imaging 2001), 4322, to appear. 12. M. Hinz, R. Pohle, T. Hübner, K. D. Toennies, 3D-Analyse medizinischer Volumendaten unter Nutzung automatisch generierter Transferfunktionen. Bildverarbeitung für die Medizin 2001, Springer, to appear.