Compact tracking of surgical instruments through structured markers

Supplemental material (published online only)

Title: Compact tracking of surgical instruments through structured markers

Authors: N. Alberto Borghese and I. Frosio

Affiliation: Applied Intelligent Systems Laboratory – Department of Computer Science – University of Milano

Address: Department of Computer Science, Via Comelico 39 – 20135 Milano. Telephone: +39.02.503.16325 – Email: [email protected] – URL: http://borghese.dsi.unimi.it/

1 Compact tracking of surgical instruments through structured markers

Compact tracking of surgical instruments through structured markers N. A. Borghese, I. Frosio Appendix A: Accurate circle function, computed on the observed pixels, is used: f p , R   N  ln t  N  ln ln 1 exp t  R 2  fitting with partial occlusions M 2D    2D  N (A2).  ln 1 exp t   2  R 2 We show here how the centroid of the image of a marker     i 2D  i1 can be computed robustly and in real time even in the presence of partial occlusions. To this aim we apply a slightly enhanced version of an algorithm developed for circle fitting, the RACF (Real-time Accurate Circle Fitting) algorithm introduced in [1]. The original RACF version and the slightly modified version adopted here are reported in this Section.

A.1 Algorithm description

Under the hypothesis that perspective distortion can be neglected, the image of a spherical marker is a circle. However, in practice, the view of the marker can be partially hidden (for instance by the marker support, as in Fig. A1). The marker’s image is therefore often far from a circle and classical circle fitting algorithms, based on algebraic or geometric approaches [2-3], or on fitting the reflectance profile [4] to the grey levels image, fail in computing the true circle center (Fig. A1) that, in the present case, is the projected image of a marker. The Circular Hough Transform (CHT) [5] can reach potentially any accuracy but the computational time increases with the resolution and becomes soon incompatible with real-time [6]. For this reason, a novel algorithm that does determine the center of a circle, in presence of occlusions, has been recently proposed in [1]. The algorithm is based on a principled definition of a likelihood function that explicitly takes into account that only part of the circle can be observed. The base assumption is that a set of pixels belonging to the marker, {pi

= [xi yi]}i=1..N, is extracted from the image through an adequate binarization procedure. For each pi we can write the Fig. A1 (a) the marker center computed with the proposed probability that it belongs to a circle (i.e. its distance from the method is shown as ‘+’; the barycentre of the binarized marker unknown circle center, pM = [xM yM], is smaller than the image is shown as ‘x’. The latter is clearly biased, especially unknown circle radius, R2D). This probability has ideally a for markers A and C. The effect of occlusion is evident in all 2 radial symmetric, step-like, shape and it is 1/(∙R2D ) inside the the markers and it has different effects (b) Minimization of circle and zero outside. For sake of computational efficiency, f(pM) in Eq. (A4) on a simulated marker. Six iterations lead to this discontinuous probability density function is convergence: the circle centre is estimated in [49.99, 50.01], approximated in [26] by a sigmoidal continuous function: the true centre is [50 50]. The algorithm is insensitive to bad initialization: when the starting point is [100, 100] instead than t 1 pp i | p M , R2D   2  2 2 (A1), the barycentre, only one more iteration is required. ln1 expt  R2D  1 expt  i  R2D 

In [1], pM and R2D are computed minimizing (A2) through where i = ||pi–pM||. The scalar parameter t, named transition, regulates the amplitude of the region that separates the simplex algorithm; pM is initialized as the barycentre of {pi the inner and outer area of the circle (experimentally, t = 0.5 = [xi yi]}i=1..N and R2D as the maximum distance of each pi from pM. Iterations are stopped when the norm of the vector of the offers a good compromise between convergence time and -2 accuracy). estimated parameters changes by less than 10 pixels between two consecutive iterations: 0.781ms are required to fit a circle To determine pM and R2D, the negative log of the likelihood

2 Compact tracking of surgical instruments through structured markers of 7 pixels of radius on a Mobile Toshiba Intel Centrino Duo triangle, a line or a circle (Fig. A2a) and varied from 0.25% to @ 2GHz, RAM 2GByte. 50% of the marker’s image surface. Results show that an

The RACF algorithm can be made even faster in the accuracy of 0.01 pixels on PM was obtained for mild occluded markers tracking application, considering the information in images (occlusion from 0.25% to 17%), of 0.13 pixels for the 3D position of the marker associated to the projected medium occluded (occlusion from 17% to 34%) and of 0.54 circle. In particular, we take here into account that a rough pixels for largely occluded (occlusion from 34% to 50%). ˆ Computing the circle center as the centroid of the estimate of the circle radius on the camera image, R2D , can be then derived as: overthreshold pixels or fitting the gray level profiles as carried out by commercial motion capture systems [8, 9] does not ˆ R2D  R3D  f X  fY  2d m  (A3), achieve a good accuracy when occlusions are present: a median error above 1 pixel and errors as large as more than 5 where R is the radius of the marker sphere and d is the 3D m pixels for triangular occlusions of less than 50% of the area, distance between the marker and the perspective center of the do occur. camera, projected over the optical axis and estimated from the A better estimate can be obtained by resorting on algebraic motion history of that marker. d can be determined from an m [2] or, better, geometric [3] circle fitting. However, the estimate of the 3D position of the markers, from their position accuracy of RACF could be reached only by the Circular in the previous frames. Alternatively, as done here, it can be Hough Transform [5]. This is based on the discretization of obtained triangulating an estimate of the center of each marker the range of (pM R ) into a set of so called accumulation cells obtained as barycenter of the marker’s projected image. 2D and by the observation that each cell uniquely identifies a ˆ R2D can be substituted to R2D in (A2) that leads to the circle. The method requires first identifying the pixels at the following simplified cost function: circle edge inside the image and then, for each of these pixels,

N a counter associated to all the cells that are compatible with 2 ˆ 2 f pM   ln1 expt   i  R2D (A4), the pixel, is increased by one. After all the edge points have i1 been considered, the accumulation cell with the highest count

that depends only on pM. This can be efficiently minimized contains the circle centre and radius. To get an accuracy through the Newton’s method which guarantees quadratic comparable with that obtained by RACF, accumulation cells convergence [7]. pM is obtained applying iteratively the of 0.1×0.1 pixels were considered, that lead to a median error following update equation: of 0.35 pixels for low and medium occlusions. The accumulation cells were spread inside a 10x10 pixel area k1 k k 1 k p M  p M  HpM  JpM  (A5), around the true circle center. The computational time of the

k k k T CHT on these images was, on average, 10.24ms (Table A1). where pM = [xM yM ] is the estimated center at the k-th k k Therefore CHT achieves almost the same accuracy of iteration, and J(pM ) and H(pM ) are respectively the gradient k RACF but with more with a computational time more than and the hessian of f(pM) computed in pM , given by: 120 times higher. N N T k  k k k k  Jp M   2tLi xM  ui  Li yM  vi   i1 i1  N N  k k 2 k k k k   M i 2txM  xi   Li  2tM i xM  xi yM  yi  k i1 i1 HpM   2t N N   k k k k k 2 k  2tM i xM  xi yM  yi  M i 2tyM  yi   Li   i1 i1  k k k k k k 2 Li  expi  1 exp i , M i  exp i  1 exp i  , k k 2 k 2 ˆ 2  i  txM  xi   yM  yi   R2D  (A6). Convergence requires less than 10 iterations, at least in all the practical situations experimented: the determination of the circle center, for instance for the circle shown in Fig. A1, through the minimization of (A4) drops from 0.781ms to 0.079ms with a factor 10 of improvement.

A.2 Algorithm evaluation Fig. A2. Panel (a) shows the dataset of 60 partially occluded circles used for the validation of the circle fitting algorithm. In The accuracy of circle fitting procedure was evaluated panel (b), the center of a partially occluded circle (A14 in through adequate simulations. A dataset composed of 60 panel a), centered in [12 12] with radius of 11 pixels, is images of partially occluded markers with a radius of 11 estimated by ACF, GCF, CHT and the proposed method (t = pixels pixels was considered. Occlusions took the shape of a

3 Compact tracking of surgical instruments through structured markers

0.5, R2D = 11). In the legend, the coordinates of the estimated and computational cost. center are also indicated. A critical issue that could affect the accuracy of the circle

fitting algorithm presented here is the computation of R2D in

Table A1. Median distance (± IQR), in pixels, from the true Eq. (A3) that depends on dm. A more precise estimate of dm marker center for CHT and the proposed method, for the and, in turns, of R2D could be obtained through a two-pass dataset in fig. 5. The mean processing time is also reported. procedure. After estimating R2D, the circle is fitted to the Outlier: NO Outlier: YES projection of the marker on the different cameras with such Prop. Method CHT Prop. Method CHT value of R2D. The 3D position of the markers is computed from Radius Error Time Error Time Error Time Error Time the circles center, R2D is computed again and from the new error [pixels] [ms] [pixels] [ms] [pixels] [ms] [pixels] [ms] -20% 1.49±2.32 0.031 3.50±0.03 9.016 1.43±2.31 0.030 3.50±0.03 8.940 markers position and a second fitting is performed. -10% 0.83±1.13 0.035 1.49±0.05 9.656 0.81±1.11 0.034 1.49±0.05 9.738 It is however worth to remark that, for slight overestimate

-5% 0.31±0.75 0.038 0.56±0.01 9.897 0.26±0.76 0.071 0.56±0.01 10.005 of R2D, the accuracy of the algorithm remains high and the 0% 0.07±0.22 0.038 0.41±0.02 10.240 0.07±0.30 0.038 0.41±0.02 10.387 computational cost has only a slight increase (Table A1). A 5% 0.05±0.17 0.080 1.10±0.17 10.616 0.30±0.10 0.071 1.17±0.15 10.754 10% 0.05±0.17 0.083 1.93±0.02 10.859 0.30±0.10 0.085 1.93±0.02 11.158 practical rule for using the proposed circle fitting algorithm is 20% 0.05±0.17 0.086 3.39±0.12 11.562 0.30±0.10 0.085 3.48±0.26 11.739 therefore to estimate R2D as in Eq. (A3) and to adjust this

To make CHT faster, a randomized CHT version (RHT) has estimate with a small increase on R2D before circle fitting. been proposed in [6]. In this case, triplets of points are The circle fitting method may further benefit of the sampled on the circle edge, and from these three points the continuous increase in the resolution of the cameras that are fitting circle is computed and the corresponding cell is used for motion capture, that presently can achieve 16Mpixels updated. However, RHT can find circles that are very far from [9]. A higher resolution allows larger images of the marker the real ones when large occlusions are present unless many with a higher resolution in fitting a circle to them. multiple samples are computed that, again, would make the method slow. Since the estimate of the 2D marker radius in Eq. (A3) is Appendix B – Sensitivity approximate, we also evaluated how an over / under estimate analysis of R2D in (A3) influences the accuracy on pM. To this aim, we have solved (A4) for the simulation data set with a under- In this section, the sensitivity of the axial rotation, i, with ˆ respect to the horizontal pixel position of a projected stripe estimated or overestimated value of R2D . Results are reported in Table A1 and show that RACF is much more robust than point, d/dxS,i, is derived. Similar considerations allow CHT (and other circle fitting methods) as also shown in [26]. deriving d/dyS,i. For notation simplicity, we will omit the i- ˆ index in the following. From (24) we have: Moreover, when R2D is overestimated, the bias in pM is negligible with RACF.  1   Y *  Y * Z * x  Z * Y * x   S   S S S S S S We have also measured how accuracy is influenced by the * * 2  *  * 2 * 2 (B1). xS 1 Y Z  xS  ZS  Z   Y  presence of outliers (white pixels outside the circle area that S S S S * * * * T sometimes occur in the real scenario); to this aim we added an Plugging the expression of P S = [X S Y S Z S] given in (23) outlier pixel to each image of the dataset, connected to the and that of PS,abs in (16), we obtain: main circle area, at a distance of 12 pixels from the true * center. For comparison, we measured the performance of CHT P S T T  T T  R Y R Z R Y R Z TC  v - T on the same dataset, since in case of occlusion CHT generally x S x S (B2). produces an accurate estimate of the circle center; on the other T T  R Y R Z  x S v   v xS  hand, other algorithms like ACF and GCF often fail in this case [1], as shown in Fig. A2b. We are interested only in the first two equations in (B2), as Similar results are obtained in presence of an outlier (see these are the only derivatives in (B1). We notice that the rows T T the right side of Table I). The overall accuracy of the proposed of the matrix R YR Z represent the orientation of the X, Y and Abs circle fitting method in estimating the circle center is similar Z axis of the instrument, rotated by RYRZ with respect to S. to that of RACF [1], but at a computational cost which is 95% Therefore, we define: lower; moreover, fixing a priori the circle radius, some kind of T T T RZY RZY RZY T R ZY  R Y R Z  X Y Z (B3) robustness with respect to the presence of outliers outside the  Instr Instr Instr  circle area was also gained. RZY where XInstr indicates the orientation of the X axis of the Abs instrument, rotated by RYRZ with respect to S. Plugging A.3 Discussion (B2) and (B3) into (B1), we obtain: The accuracy of the circle fitting procedure does depend on  Y *Z RZY  Z * Y RZY   v  the transition parameter t: it does increase with t, but, at the  S Instr S Instr  v    (B4) x * 2 *2  x x  same time, the convergence time also increases. We have set S Z S  YS  S S  t = 0.5 that represents a good compromise between accuracy We can now compute the two derivatives in  xS :

4 Compact tracking of surgical instruments through structured markers

T  T 2 2           RYZ RYZ T   v  2 v  v   xS  c x y S  c y  xS  c x  xS  c y     sin ZInstr  cos YInstr    TC  M0  v   (B12)  R C  1    1        xS R3D cos  xS    xS  xS xS   f x f y   f x  f y           Defining the versor w: 1  2 2   x  c   yS  c y   x  c  1 1  R 0  f  S x     1   S x  v C    x        2 2 w  v xS v xS (B13)  f x   f y   f x  f x  x  c   y  c  0    S x    S y  1     We obtain the final expression of  x :  f x   f y  S (B5)   v RYZ RYZ T  T 2   sin ZInstr  cos YInstr  w TC  M0  v  w x R cos x    and: S 3D S (B14).

T We first notice that the sensitivity is linearly proportional to    T    v  1     v xS R3D cos  : it increases linearly with v xS , which  v M 0  TC      M 0  TC  (B6), xS xS  2   xS  4  xS is proportional to 1/fX (A5). Therefore the sensitivity in the estimate of Rx is larger in the set-ups which use a shorter focal where: length. This function has two singular points: cos  = 0 and 

  2 = 0. These conditions are associated respectively to the 2vT T M 4 T M T T M R 2    C  0    C  0   C  0  3D  marker’s poles and to the points on the circumference (edge) xS xS (B7). T T of the marker image (that determine the polar lines for the  8v TC  M 0 v xS  TC  M 0  sphere). Substituting (B7) into (B6), we obtain: Appendix C – Other tracking T   v  1       M 0  TC    approaches x S  xS  4  xS T T The first approaches to instrument tracking in surgery were  v  1  v      M  T  8v T T  M   T  M   based on electromagnetic trackers [12-13], but their accuracy  x  0 C C 0  x  C 0  S  4   S   and reliability in the operating room, where metallic material (B8), T is present, has been questioned. More recently, inertial  v   2 T     M 0  TC 1 v TC  M 0   tracking, based on MEMS devices, has been exploited.  x   S     However, drifts and integration errors cannot be completely  T  T eliminated and MEMS accuracy remains limited even after  v     2v TC  M 0     M 0  TC   accurate calibration [14-16]. To overcome this, approaches  x    S    based on integrating a geometrical model of the moving object with MEMS measurements have been proposed [17-18], but T and substituting   v M0  TC   2 (Eq. (20)) in (B8), they allow a relatively high accuracy only for a short time. we obtain the final expression of  x S : Haptic devices constitute a different approach that provides also force feed-back [19]. They can be used when instruments  v     v 2 do not undergo complex movements as the instruments are  M  T T   2      T  M T  0 C      C 0  (B9). connected to the device base. This strongly limits the range of x S x S   2  x S    possible movements and the working volume. Plugging (B9) into (B4) we obtain: For these reasons, optical tracking is presently the most used approach. Although several attempts of accurately T * RYZ * RYZ tracking motion by using computer vision techniques [20-21] Y Z  Z Y   v 2 v  TM  S Instr S Instr  T   or 3D cameras like Microsoft Kinect [22] have been     TC  M 0  v    (B10). x * 2 *2 x  x S  Z S  YS   S S  proposed, the classical approach of using markers still guarantees the most reliable results. As from (14), (22) and (23) the following relationship holds:

* RYZ * RYZ RYZ RYZ References YS Z Instr  Z S YInstr R3D sin cos  Z Instr  R3D cos cos  YInstr 2 2  2 2  Z *  Y * R cos cos    R sin cos   1. Frosio I, Borghese NA. (2008) Real-time accurate circle fitting with S S 3D 3D (B11) occlusions. Patt Recogn 41(3):1041-55. RYZ RYZ 2. Ahn SJ, Rauh W, Warnecke H (2001) Least-squares orthogonal  sin Z Instr  cos YInstr  R3D cos   distances fitting of circle, sphere, ellipse, hyperbola, and parabola, Patt We obtain the expression of  x : Recogn 34:2283–2303. S 3. Berman M, Culpin D (1986) The statistical behaviour of some least squares estimators of the centre and radius of a circle. J Royal Statist Soc B 48:183–196.

5 Compact tracking of surgical instruments through structured markers

4. Berman M, Culpin D (1986) The statistical behaviour of some least 15. Frosio, I., Pedersini F. and Borghese N.A. (2009). Autocalibration of squares estimators of the centre and radius of a circle. J Royal Statist MEMS Accelerometers. IEEE Trans. on Instr. & Measurements, Vol. Soc B 48:183–196. 58, Issue 6, June 2009 Page(s):2034 - 2041. 5. Hough PVC (1962) Method and means for recognizing complex 16. I. Frosio, F. Pedersini, N. A. Borghese, “Autocalibration of triaxial Patterns, US Patent, 3069654. MEMS accelerometers with automatic sensor model selection,” IEEE 6. Chen TC, Chung KL (2001) An efficient randomized algorithm for Sensors Journal, 12(6), pp. 2100-8, Jun. 2012. detecting circles. Comput Vision Image Underst 83(2):172–191. 17. Giansanti D, Macellari V, Cappozzo A (2003) Is it feasible to 7. Nocedal J, Wright SJ (2006) Numerical Optimization. Springer Verlag, reconstruct body segment 3-D position and orientation using 2nd Ed. accelerometric data? IEEE Trans Biomed Eng 50(4):476-83. 8. http://www.btsbioengineering.com/ 18. Tanaka M (2007) An industrial and applied review of new MEMS 9. http://www.vicon.com. devices features. Microel Eng MNE 2007, 84(5-8), pp 1341-44. 10. Chernov N, Lesor C (2005) Least squares fitting of circles. J. Math 19. Pettersson J, Palmerius KL, Knutson H (2008) Simulation of Patient Image Vision 23:239–252. Specific Cervical Hip Fracture Surgery with a Volume Haptic Interface, 11. Ahn SJ, Rauh W, Warnecke HJ (2001) Least-squares orthogonal IEEE Trans Biomed Eng 55(4):1255-1265. distances fitting of circle, sphere, ellipse, hyperbola, and parabola. Patt 20. Corazza S, Muedermann L, Chaudhari A, Demattio T, Cobelli C, Recog 34:2283–2303. Andriacchi T (2006) A markerless motion capture system to study 12. Roetenberg D, Slyche PJ, Veltink PH (2007) Ambulatory Position and muscoloskeletal biomechanics: visual hull and simulated annealing Orientation Tracking Fusing Magnetic and Inertial Sensing. IEEE Trans approach. Annals Biomed Eng 34(6):1019-1029. Biom Eng 54(5):883-90. 21. http://www.organicmotion.com/ 13. Plotkin A, Kucher V, Horen Y, Paperno E (2008) A New Calibration 22. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Procedure for Magnetic Tracking Systems. IEEE Trans Magn Kohli P, Shotton J, Hodges S, Fitzgibbon A, KinectFusion: Real-Time 44(11):4525-28. Dense Surface Mapping and Tracking. Siggraph 2011, pp 581-589. 14. Ang WT, Khosla PK, Riviere CN (2004) Kalman Filtering for Real Time Orientation Tracking of Handheld Microsurgical Instrument. IROS 2004 pp 2574-2580.

6 Compact tracking of surgical instruments through structured markers

7