SIIM 2016 Scientific Session Image Viewing, Interpretation, and Advanced Thursday, June 301:15 pm – 2:45 pm

Improving the Efficiency of Manual Ground Truth Labeling: Using Automated Segmentation

Hongzhi Wang, PhD, Almaden Research Center (Presenter); Prasanth Prasanna, MD; Jose Morey, MD; Tanveer F. Syeda-Mahmood, PhD

Background

Anatomy segmentation is the problem of delineating anatomical structures in medical images. It is a key operation in automatic detection of anomalies and in most image-guided diagnosis/intervention procedures, as it precisely isolates the anatomical region of interest. Automatic anatomy segmentation can considerably speed up the diagnosis process making it useful in clinical workflows. However, to develop robust automated segmentation methods, they must be compared with manual segmentation done by clinicians or trained experts serving as a “gold standard”. Unfortunately, producing these ground-truth labels by clinicians is currently a laborious process taking well over an hour for annotations of 3D scans even with sophisticated tools such as Amira (FEI Corporate, Hillsboro, Oregon USA). The manual ground truthing effort and the delays associated with this task have made it difficult to develop reliable automated segmentation methods. In this paper we advance a novel approach to speed up the ground-truthing process by aiding it with an automated segmentation method and using the clinicians input for the corrections of the produced segmentations.

Case Presentation

Using automated segmentation combined with manual correction, we developed a semi-automatic ground- truthing system for radiology imaging studies. Specifically, we focused on the problem of improving the labeling speed for the labeling of cardiac CT scans. The cardiac CT studies were acquired by a Siemens CT Scanner (Somatom Definition Flash; Siemens Healthcare, Erlangen, Germany). A set of twenty anatomical structures were studied, namely, sternum, aorta (ascending/descending/arch/root), pulmonary artery (left/right/trunk), vertebrae, left/right atrium, left/right ventricle, left ventricular myocardium, superior/inferior vena cava, and aortic/tricuspid/pulmonary/mitral valve. A first training set of 28 cases were manually segmented for these structures by a clinician using existing commercial software, Amira 5.5.0 to serve as the training reference atlas data for an automatic segmentation method. Specifically, we applied a multi-atlas segmentation method using deformable registration (Advanced Normalization Tools (ANTS) software [1]), joint label fusion of training atlases [2] and corrective learning [3], followed by a post- processing function to smooth the boundaries (see examples in Fig. 1). The post processing function was implemented to encode the geometrical constraints such that the anatomical boundaries between right ventricle and pulmonary artery or between the substructures of aorta do not cross multiple axial slices. For example, if one axial slice contains both aortic root and ascending aorta labels in the result produced by multi-atlas segmentation, the voxels assigned to the two structures are relabeled to either aortic arch or descending aorta, whichever has the most voxels assigned by multi-atlas segmentation. The remaining 53 cardiac CT cases were then automatically segmented using this trained segmentation algorithm. Five of these cases were randomly selected to study the improvement in ground-truthing time by clinician when aided by automatic segmentation. Specifically, these five studies were manually corrected by one clinician after automatic segmentation and the correction time was noted. To avoid the bias created by automatic segmentation, the same set was completely labeled manually after 1 week of gap by the same clinician.

Outcome

For the five cases, the time spent on manual segmentation with/without correcting automatic segmentation was 41.0±6.5 minutes and 66.0±6.5 minutes, respectively. Overall, correcting automatic segmentation saved 37% time for manual segmentation. The results are statistically significant with p<0.0001 on the paired Students t-test. For the 20 anatomical structures, about 5.4% voxels had changed segmentation labels in the manual correction process. However, the voxel correction rates are not uniformly distributed across structures. For structures with less distinctive anatomical boundaries and small volumes, such as the vena cava with right atrium and valves (except mitral valve), more than 20% voxels had to be corrected. For distinctive structures such as sternum and vertebrae, only less than 3% voxels needed to be corrected.

Figure 1

Discussion

Manually correcting automatic segmentation has been applied in several other applications to reduce time cost in manual segmentation, e.g. [4,5]. However, most previous work only focus on segmenting single anatomical structure. To the best of our knowledge, this is the first study that investigates the time saving effect of manually correcting automatic segmentation on a complex segmentation task with 20 neighboring anatomical structures. This is also the first manual correction study conducted using state of the art techniques in automatic anatomy segmentation. The automatic segmentation technique applied in this study is the top performer in several recent grand challenges on anatomy segmentation [6,7]. Hence, our study provides a snapshot of the full potential benefit in time reduction of the manual correction approach. The reduction in time due to automated segmentation was also limited by the annotation software used. The Amira software has a powerful segmentation interpolation function that allows a rater to manually segment 2D slices and then automatically propagates the segmentation to its neighboring non-labeled slices. This function while available for manual segmentation in the tool, was not available as an option for automatic segmentation, due to the somewhat random morphology of errors for a given structure from one slice to the next. Hence, in this study Amira’s segmentation interpolation function was applied in the manual segmentation experiment but not in the manual correction experiment. Even with this disadvantage of using a less powerful editing tool, manual correction still saved 37% time over complete manual segmentation. Hence, we expect more time reduction to be achieved when manual segmentation and manual correction are conducted with the same segmentation editing functions, which will be a more common scenario when other editing tools, such as ITK-Snap, are applied for the manual segmentation task.

Conclusion

We have presented a preliminary study showing that manually correcting the output of automatic segmentation produced by a state of the art multi-atlas segmentation technique can reduce by almost 40% manual segmentation time in a segmentation task of labeling 20 anatomy structures in 3D cardiac CT scans.

References

1. Avants, B., Epstein, C., Grossman, M., Gee, J.: Symmetric diffeomorphic with cross- correlation: Evaluating automated labeling of elderly and neurodegenerative brain. Medical Image Analysis 12(1), 26-41 (2008). 2. Wang, H., Suh, J.W., Das, S., Pluta, J., Craige, C., Yushkevich, P.: Multi-atlas segmentation with joint label fusion. IEEE Trans. on PAMI 35(3), 611-623 (2013). 3. Wang, H., Das, S. R., Suh, J. W., Altinay, M., Pluta, J., Craige, C., Avants B., Yushkevich P.A.. A learning- based wrapper method to correct systematic errors in automatic : consistently improved performance in hippocampus, cortex and brain segmentation. NeuroImage, 55(3), 968-985 (2011). 4. CARS report: Liver segmentation tool allows manual fixes, http://www.auntminnie.com/index.aspx?sec=ser&sub=def&pag=dis&ItemID=86377 5. Daisne, Jean-François, and Andreas Blumhofer. "Atlas-based automatic segmentation of head and neck organs at risk and nodal target volumes: a clinical validation." Radiation Oncology 8.1 : 154 (2013). 6. MICCAI 2012 Grand Challenge and Workshop on Multi-Atlas Labeling, https://masi.vuse.vanderbilt.edu/workshop2012/index.php/Main_Page 7. Asman, A., Akhondi-Asl, A., Wang, H., Tustison, N., Avants, B., Warfield, S. K., & Landman, B. (2013). Miccai 2013 segmentation algorithms, theory and applications (SATA) challenge results summary. In MICCAI Challenge Workshop on Segmentation: Algorithms, Theory and Applications (SATA).

Keywords

Manual Segmentation, Automatic Segmentation, Atlas, Cardiac CT, Amira