CT Image Based 3D Reconstruction for Computer Aided Orthopaedic Surgery F. Blanco IDMEC-IST, Technical University of Lisbon, Av Rovisco Pais, 1049-001 Lisboa, Portugal P.J.S. Gonçalves Polytechnic Institute of Castelo Branco, Av Empresario, 6000-767 Castelo Branco, Portugal IDMEC-IST, Technical University of Lisbon, Av Rovisco Pais, 1049-001 Lisboa, Portugal J.M.M. Martins & J.R. Caldas Pinto IDMEC-IST, Technical University of Lisbon, Av Rovisco Pais, 1049-001 Lisboa, Portugal Abstract This paper presents an integrated system of 3D medical image visualization and robotics to assist orthopaedic surgery, namely by positioning a manipulator robot end-effector using a CT image based femur model. The 3D model is generated from CT images, acquired in a preoperative exam, using VTK applied to the free medical visualization software InVesalius. Add-ons were developed for communication with the robot which will be forwarded to the designated position to perform surgical tasks. A stereo vision 3D reconstruction algorithm is used to determine the coordinate transform between the virtual and the real femur by means of a fiducial marker attached to the bone. Communication between the robot and 3D model is bi-directional since visualization is updated in real time from the end-effector movement. 1. Introduction In recent years, robotic applications regarding surgical assistance have been appearing more often. Their advantages in precision, accuracy, repeatability and the ability to use specialized manipulators are granting them a place in the operating theatre. By introducing robotic technology in surgery, the goal is not to replace the surgeon but, instead, to develop new tools to assist in accurate surgical tasks. The ability of precise robotic execution from image based models is showing promising results in orthopaedic surgery. Applications such as total hip replacement require an accurate broaching of the femur to match the implant shape as it may influence biomechanics. Medical imagery is widely used in Computer Aided Orthopaedic Surgery (CAOS)[DGN02] to attain data for pre-operative planning, simulation and intra-operative robotic 1 navigation, by the use of sensory systems to track fiducial markers attached to the patient’s bone. This paper presents an experimental robotic system that integrates CT based imaging data and fiducial marker tracking for robot navigation. Figures 1 and 2 summarize the stages for intra-operative robot navigation. Figure 1 – Operational diagram Figure 2 – Intra-operative system The 3D model is processed based on CT scans acquired pre-operatively and through it the surgeon can visually position the end-effector. The robotic arm can navigate by tracking the position of fiducial markers and update in real-time the position in the virtual model. Fiducial marker tracking is performed by the use of stereo vision. Section 2 presents the 3D medical visualization software developed to interact with the robot. Computer Vision is described in section 3 and in section 4 is presented the robotic system. Section 5 describes the overall system and some results obtained. In section 6 some conclusions are presented. 2. 3D Medical Visualization Software 2.1 Introduction The process of medical image acquisition is an area that is being very developed in recent decades. Computer tomography (CT), magnetic resonance imaging (MRI) and ultrasound imagery revolutionized the diagnostic in medicine. These techniques made possible an internal view of practically every organ and structure of human anatomy, without the resort of evasive techniques, led to more precise diagnosis. This section presents tools for visualization and segmentation of three-dimensional data from a set of medical images. Developed software is presented which is able to merge 2D CT images into a single 3D volume and in a visual and interactive way position the end-effector of the robot in a real-time environment. 2 2.2 Software Development It was used as platform basis the open-source medical visualization software, InVesalius, and from then on add-on modules were created. InVesalius is a three-dimensional medical imaging software created by a team from Centro de Pesquisas Renato Archer (CenPRA) [MBS07], Brazil, targeted to Brazilian public hospitals. Regarding the system architecture, InVesalius uses Python and C++ programming languages and employs the Visualization ToolKit Library (VTK) The developed software allows the creation of a volumetric rendering based on a set of CT imagery. The user may input data referring to the position and orientation of the end-effector that is sent to the robot and it can then be visually displayed over the volumetric rendering which is updated based on the actual position of the robot’s end-effector. A intersection plane over the end-effector reference frame cuts the volumetric object and the section is shown in another rendering frame. The software, shown on Figure , provides two rendering frames, one with the volumetrical rendering and the intersection plane, and another showing the intersection cut. Figure 3 – 3D Visualization Software 3. Computer Vision 3.1 Introduction Computer Vision is used to locate the position of the fiducial marker to bridge the virtual model and the surgical environment. 3D reconstruction algorithm is applied using two uEye high-speed cameras in an Eye- to-Hand configuration, thus allowing them to have a broader view of the robot’s working area and some degree of freedom on the fiducial marker movement. Even though high capture rates 3 were also available, the image acquisition was set at 25 frames per second because the object to be tracked is not a fast moving target and thus providing more time for image processing. 3.2 Fiducial Marker Many commercially available optical tracker systems use infrared light-emitting diodes or reflective targets to determine the fiducial marker location. However these types of targets were inadequate for the environment conditions available. To resolve this problem, it was used small cold cathode lights. A minor change was necessary; the main vein was too bright and the transparent acrylic housing didn’t convey a homogeneous light distribution. This problem was resolved by giving a matte surface finishing. This procedure gave a homogeneous colour and light intensity distribution both radially and lengthwise. Figure 4 – Fiducial Marker Prototype The fiducial marker prototype appears in Figure 4. It is square shaped and has four different colours. The square shape was chosen due to its easy measurable configuration and as a fast computational way to determine orientation. The colours chosen were Red, Green, Blue and Yellow by their distinct histogram traits which facilitate recognition. Black tape was used to cover the extremities of the lights so that the centre of each light may be confined into a small space and enhance precision. The cold cathode lights revealed to be so bright to normal acquisition parameters that the cameras’ exposure time had to be reduced to 1.2 ms. This made visible only the target and blackened all the surrounding environment which turned to be a quite efficient filter 3.3 Software Development As development basis it was used a software by Silva [S08] that could compute spatial positioning of a single coloured object. From that platform basis it was developed ways to detect the various colours of the marker, compute the marker’s centroid for spatial positioning and through vectorial algebra determines its orientation. 4 The software is written in C++, which provides fast processing capabilities for real time calculations and OpenCV library. 4. Robotics In the present work a PUMA 560 manipulator robot was used. Previous kinematic equations had been implemented [CFS05][S08] but were only relative to the first 3 joints which confers only correct positioning. Full inverse kinematic equations where implemented using Kang’s solutions [K07]. On behalf of trajectory planning it was also implemented a fifth order polynomial to guarantee a stable motion. 5. System Integration and Results 5.1 System Integration Figure 5 illustrates how each individual system is interlinked. The core of the system network is the PC Target and is concerned with the Puma controller. All the other systems communicate with it providing and receiving data for its calculations. To manage the kinematic and control algorithms of the PC Target, the PC Host is used to communicate through Xpc Target via UDP. The computer in charge of the computer vision is PC Vision. This computer calculates the spatial location of the fiducial marker and sends the data to PC Target via UDP communication. Three-dimensional bone modelling and visualization are processed in PC 3D and its relative position and orientation are sent to the PC Target via UDP communication. With the information provided by both PC Vision and PC 3D, the PC Target is able to determine the joint coordinates for the desired position to the robot’s controller. Additionally, PC Target computes de current relative position between the marker and the end-effector and sends it back to PC 3D, as real-time update. Figure 5 – System integration 5 5.2 Results 5.2.1 - 3D Medical Visualization Software For this work it was used a set of 150 CT images from the pelvis to the knee, from Biomed Town repository [Biomed]. The images are formatted as DICOM files, with 430x430 pixels, 2mm of spacing between images and a 12-bit resolution which represent a scale of 4096 gray levels. Figure 6 shows the rendered femoral structure from the CT images. Figure 6- Femur 3D Model 5.2.2 – Computer Vision Proper camera calibration is crucial for correct 3D reconstruction. To perform the calibration of the cameras, it was used the Camera Calibration Toolbox for Matlab [B08]. Once the extrinsic and intrinsic parameters were determined, precision and accuracy tests were applied to determine if the fiducial marker was properly recognized and located. Figure 7 shows graphically the displacement of the marker’s centroid in a fixed position.