CT Image Based 3D Reconstruction for Computer Aided Orthopaedic Surgery

F. Blanco IDMEC-IST, Technical University of Lisbon, Av Rovisco Pais, 1049-001 Lisboa, Portugal P.J.S. Gonçalves Polytechnic Institute of Castelo Branco, Av Empresario, 6000-767 Castelo Branco, Portugal IDMEC-IST, Technical University of Lisbon, Av Rovisco Pais, 1049-001 Lisboa, Portugal J.M.M. Martins & J.R. Caldas Pinto IDMEC-IST, Technical University of Lisbon, Av Rovisco Pais, 1049-001 Lisboa, Portugal

Abstract

This paper presents an integrated system of 3D medical image visualization and robotics to assist orthopaedic surgery, namely by positioning a manipulator robot end-effector using a CT image based femur model. The 3D model is generated from CT images, acquired in a preoperative exam, using VTK applied to the free medical visualization software InVesalius. Add-ons were developed for communication with the robot which will be forwarded to the designated position to perform surgical tasks. A stereo vision 3D reconstruction algorithm is used to determine the coordinate transform between the virtual and the real femur by means of a fiducial marker attached to the bone. Communication between the robot and 3D model is bi-directional since visualization is updated in real time from the end-effector movement.

1. Introduction

In recent years, robotic applications regarding surgical assistance have been appearing more often. Their advantages in precision, accuracy, repeatability and the ability to use specialized manipulators are granting them a place in the operating theatre. By introducing robotic technology in surgery, the goal is not to replace the surgeon but, instead, to develop new tools to assist in accurate surgical tasks. The ability of precise robotic execution from image based models is showing promising results in orthopaedic surgery. Applications such as total hip replacement require an accurate broaching of the femur to match the implant shape as it may influence biomechanics. Medical imagery is widely used in Computer Aided Orthopaedic Surgery (CAOS)[DGN02] to attain data for pre-operative planning, simulation and intra-operative robotic

1 navigation, by the use of sensory systems to track fiducial markers attached to the patient’s bone. This paper presents an experimental robotic system that integrates CT based imaging data and fiducial marker tracking for robot navigation. Figures 1 and 2 summarize the stages for intra-operative robot navigation.

Figure 1 – Operational diagram Figure 2 – Intra-operative system

The 3D model is processed based on CT scans acquired pre-operatively and through it the surgeon can visually position the end-effector. The robotic arm can navigate by tracking the position of fiducial markers and update in real-time the position in the virtual model. Fiducial marker tracking is performed by the use of stereo vision.

Section 2 presents the 3D medical visualization software developed to interact with the robot. Computer Vision is described in section 3 and in section 4 is presented the robotic system. Section 5 describes the overall system and some results obtained. In section 6 some conclusions are presented.

2. 3D Medical Visualization Software

2.1 Introduction

The process of medical image acquisition is an area that is being very developed in recent decades. Computer tomography (CT), magnetic resonance imaging (MRI) and ultrasound imagery revolutionized the diagnostic in medicine. These techniques made possible an internal view of practically every organ and structure of human anatomy, without the resort of evasive techniques, led to more precise diagnosis. This section presents tools for visualization and segmentation of three-dimensional data from a set of medical images. Developed software is presented which is able to merge 2D CT images into a single 3D volume and in a visual and interactive way position the end-effector of the robot in a real-time environment.

2

2.2 Software Development

It was used as platform basis the open-source medical visualization software, InVesalius, and from then on add-on modules were created. InVesalius is a three-dimensional medical imaging software created by a team from Centro de Pesquisas Renato Archer (CenPRA) [MBS07], Brazil, targeted to Brazilian public hospitals. Regarding the system architecture, InVesalius uses Python and C++ programming languages and employs the Visualization ToolKit Library (VTK) The developed software allows the creation of a volumetric rendering based on a set of CT imagery. The user may input data referring to the position and orientation of the end-effector that is sent to the robot and it can then be visually displayed over the volumetric rendering which is updated based on the actual position of the robot’s end-effector. A intersection plane over the end-effector reference frame cuts the volumetric object and the section is shown in another rendering frame. The software, shown on Figure , provides two rendering frames, one with the volumetrical rendering and the intersection plane, and another showing the intersection cut.

Figure 3 – 3D Visualization Software

3. Computer Vision

3.1 Introduction

Computer Vision is used to locate the position of the fiducial marker to bridge the virtual model and the surgical environment. 3D reconstruction algorithm is applied using two uEye high-speed cameras in an Eye- to-Hand configuration, thus allowing them to have a broader view of the robot’s working area and some degree of freedom on the fiducial marker movement. Even though high capture rates

3 were also available, the image acquisition was set at 25 frames per second because the object to be tracked is not a fast moving target and thus providing more time for image processing.

3.2 Fiducial Marker

Many commercially available optical tracker systems use infrared light-emitting diodes or reflective targets to determine the fiducial marker location. However these types of targets were inadequate for the environment conditions available. To resolve this problem, it was used small cold cathode lights. A minor change was necessary; the main vein was too bright and the transparent acrylic housing didn’t convey a homogeneous light distribution. This problem was resolved by giving a matte surface finishing. This procedure gave a homogeneous colour and light intensity distribution both radially and lengthwise.

Figure 4 – Fiducial Marker Prototype

The fiducial marker prototype appears in Figure 4. It is square shaped and has four different colours. The square shape was chosen due to its easy measurable configuration and as a fast computational way to determine orientation. The colours chosen were Red, Green, Blue and Yellow by their distinct histogram traits which facilitate recognition. Black tape was used to cover the extremities of the lights so that the centre of each light may be confined into a small space and enhance precision. The cold cathode lights revealed to be so bright to normal acquisition parameters that the cameras’ exposure time had to be reduced to 1.2 ms. This made visible only the target and blackened all the surrounding environment which turned to be a quite efficient filter

3.3 Software Development

As development basis it was used a software by Silva [S08] that could compute spatial positioning of a single coloured object. From that platform basis it was developed ways to detect the various colours of the marker, compute the marker’s centroid for spatial positioning and through vectorial algebra determines its orientation.

4

The software is written in C++, which provides fast processing capabilities for real time calculations and OpenCV library.

4. Robotics

In the present work a PUMA 560 manipulator robot was used. Previous kinematic equations had been implemented [CFS05][S08] but were only relative to the first 3 joints which confers only correct positioning. Full inverse kinematic equations where implemented using Kang’s solutions [K07]. On behalf of trajectory planning it was also implemented a fifth order polynomial to guarantee a stable motion.

5. System Integration and Results

5.1 System Integration

Figure 5 illustrates how each individual system is interlinked. The core of the system network is the PC Target and is concerned with the Puma controller. All the other systems communicate with it providing and receiving data for its calculations. To manage the kinematic and control algorithms of the PC Target, the PC Host is used to communicate through Xpc Target via UDP. The computer in charge of the computer vision is PC Vision. This computer calculates the spatial location of the fiducial marker and sends the data to PC Target via UDP communication. Three-dimensional bone modelling and visualization are processed in PC 3D and its relative position and orientation are sent to the PC Target via UDP communication. With the information provided by both PC Vision and PC 3D, the PC Target is able to determine the joint coordinates for the desired position to the robot’s controller. Additionally, PC Target computes de current relative position between the marker and the end-effector and sends it back to PC 3D, as real-time update.

Figure 5 – System integration

5

5.2 Results

5.2.1 - 3D Medical Visualization Software

For this work it was used a set of 150 CT images from the pelvis to the knee, from Biomed Town repository [Biomed]. The images are formatted as DICOM files, with 430x430 pixels, 2mm of spacing between images and a 12-bit resolution which represent a scale of 4096 gray levels. Figure 6 shows the rendered femoral structure from the CT images.

Figure 6- Femur 3D Model

5.2.2 – Computer Vision

Proper camera calibration is crucial for correct 3D reconstruction. To perform the calibration of the cameras, it was used the Camera Calibration Toolbox for Matlab [B08]. Once the extrinsic and intrinsic parameters were determined, precision and accuracy tests were applied to determine if the fiducial marker was properly recognized and located. Figure 7 shows graphically the displacement of the marker’s centroid in a fixed position.

1

0.5

Y [cm] 0 -0.2 -0.1

-0.5 0 -0.2 -0.1 0.1 0 0.1 0.2 Z [cm] 0.2 X [ c m ] Figure 7 – Centroid displacement

6

Positioning tends to oscillate at a high frequency which affects the 3D reconstruction. Figures 8, 9 and 10 show the various positions of the fiducial marker along each axis. The coordinates are referent to the left camera coordinate frame.

15

10 0.5 0.5 5

0 0 Y [cm] -5 Y [cm] -10 0 Y [cm] -15 -0.5 0.3 0.5 0.2 -0.5 126 130 0.1 0 125 10 124 5 120 115 0 116.8 0 116.6 116.7 122 110 116.5 -5 105 X [ c m ] -0.1 116.3 116.4 -10 120 X [ c m ] -0.5 116.2 Z [cm] 100 Z [cm] Z [cm] X [ c m ]

Figure 8 - Displacement along Figure 9 - Displacement Figure 10 - Displacement the vertical axis along the horizontal axis along the optical axis

This test revealed that horizontal and vertical displacement was quite accurate, where the maximum error rounded 0.005m at a distance of 1.16m. Depth perception however was being miscalculated reaching errors up to 0.05cm. To attenuate this error a corrective parameter, from depth data interpolation, was added to the calculation. This allowed the displacement error value to diminish up 0.003m in the calibration area.

5.2.3 – Simulated Integrated System

The integrated system was tested on a virtual environment implemented in Simulink. In it, all the information on the fiducial marker location, robot kinematic equations and end-effector positioning were processed.

Figure 11 – Virtual Environment Figure 12 – Physical Environment

In the fiducial position shown above (figure 12), it was sent the end-effector to a determined position, 5cm from the femur head in the femoral axis and a rotation of 50º along the coronal axis.

7

Figure 14 - Desired section cut

Figure 13 – End effector orientation on virtual model

Figure 15 - Obtained section

The intersection plane was mostly accurate but tended to move a little bit around the desired position due to the marker precision error. A snapshot of the section cut reveals the slight displacement mentioned above in figures 14 and 15. It is important to refer the fact that if the actual bone is far from the working area of the robot or if it is oriented in such way that the end-effector can’t reach with the desired orientation, the section cuts do not match at all.

6. Conclusions and Future Work

Firstly it was developed tools for volume rendering from CT imagery and use it for real time manipulation. This tool provided a fair visual structure that enables an in depth perception of the end-effector positioning. The 3D model reacts rapidly in real time processing but a better image segmentation may be needed to evidence the full femur structure, such as the upper and lower extremities. Computer vision was used to determine the spatial location of a fiducial marker created for this thesis. Tests show that the stereo vision computes with precision the positioning of the

8 marker but its accuracy depends on a rigorous calibration. On the region where the calibration was taken place, the system computes accurately horizontal and vertical positioning. However depth perception was not satisfactory for the purpose of this thesis. A corrective parameter was applied thus diminishing the error but not eradicating completely. The marker’s orientation is influenced by the spatial position of the elements of the marker and due to the depth perception error the orientation varies up to 5 degrees in some areas which is not acceptable when positioning the end-effector at a large distance from the fiducial marker. The inverse kinematic equations were properly implemented as well as its home configuration conversion. On trajectory planning, it was implemented a fifth order polynomial.

7. References

[Biomed] Biomed Town Repository 2009 – LHDL Building, http://www.biomedtown.org/biomed_town/LHDL/Reception/datarepository

[MBS07] Martins, T. A. C. P. ; Barbara, A. S. ; Silva, G. B. C. ; Faria, T. V. ; B. Cassaro ; Silva, J. V. L. 2007. InVesalius: Threedimensional medical reconstruction software. In: 3rd International Conference on Advanced Research in Virtual and Rapid Prototyping, Leiria - Portugal. Virtual and Rapid Manufacturing, Taylor & Francis, 2007,. p. 135-142.

[DGN02] DiGioia III A. M. & Nolte L. P., 2002. “The challenges for CAOS: what is the role of CAOS in orthopaedics? Computer Aided Surgery”, vol. 7, pp. 127–128.

[T94] Taylor, R.H, et al., June 1994, “An image-directed robotic system for precise orthopaedic surgery” IEEE Transactions on Robotics and Automation, Vol.10, No.3, pp. 261-275,

[PMM+88] Paul, H.A., Mittlestandt, B.D, Musitis, B.L.Bargar, W.L., Hayes, D.E, 1998, “Application of CT and robotic technology to hip replacement surgery” Proc. First International Symposium on Custom

[S98] Simon, D.A., “What is ‘‘registration’’ and why is it so important in CAOS?” Proc. Computer Aided Orthopaedic Conference, pp. 57-60, 1998

[S98a] Simon D.A.” Intra-operative position sensing and tracking devices”, Proc. Comp. Aided Orthop. Surg. Conf, Pittsburg, pp. 62–64, 1998

[S00] Schroeder, W. ,Avila, L.S. and Hoffman, W. (2000) Vizualizing with VTK: A Tutorial. IEEE Computer Graphics & Applications, Vol. 20, No. 5, pp. 20-27

9

[GW00] Gonzalez, R.C., Woods, R.E. (2000), “Processamento de Imagens Digitais” Edgard Blücher, São Paulo, SP.

[S08] Silva,P., 2008, ” Controlo Visual de Robôs Manipuladores Utilizando um Sistema de Visão Estéreo”, MSc Thesis, Instituto Superior Técnico, Portugal

[K07] Chul-Goo Kang, October 2007, “Online Trajectory Planning for a PUMA Robot”, International Journal of Precision Engineering and Manufacturing Vol. 8, No.4, pp.16-21

[CGL95] Cote, J., Gosselin, C. M. and Laurendeau, D., 1995. “Generalized inverse kinematic functions for the Puma manipulators,” IEEE Transactions on Robotics and Automation, Vol. 11, No. 3, pp 404–408,

[CFS05] Carvalho, André R. D., Faria, Jorge M. P. S., Silva, Pedro A. M., Fevereiro 2005, Controlo por Visão do Puma 560 – Robótica de Manipulação, Instituto Superior Técnico, Lisboa, Portugal,

10