ALGORITHMS FOR INTELLIGENT ROBOTIC SURGICAL SYSTEMS by RUSSELL C JACKSON Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Electrical Engineering and Computer Science CASE WESTERN RESERVE UNIVERSITY January, 2016 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve the dissertation of Russell C Jackson candidate for the degree of Doctor of Philosophy*. Committee Chair M. Cenk C¸avu¸so˘glu PhD Committee Member Wyatt Newman PhD Committee Member Roger D Quinn PhD Committee Member Gregory S Lee PhD Committee Member David Wilson PhD Date of Defense September 24, 2015 *We also certify that written approval has been obtained for any proprietary material contained therein. Copyright c 2015 Russell C Jackson All rights reserved. To my dad, Jay Jackson, who inspired me to become an engineer, and also to my wife, Michele Mumaw, who supported and encouraged me through all the ups and downs of graduate school. Contents Contents i List of Tables v List of Figures vi Acknowledgments viii Abstract 1 1 Introduction 3 1.1 SurgicalSubtaskAutomation . 7 1.2 Contributions ............................... 9 1.3 Dissertation Outline . 9 2 Surgical Automation 11 2.1 RoboticAutomation ........................... 11 2.1.1 SurgicalSubtaskAutomation . 12 2.2 Robot-EnvironmentInteraction . 16 2.2.1 Perception............................. 17 2.2.2 Planning.............................. 21 2.2.3 Control .............................. 23 i CONTENTS 3 Real-Time Visual Tracking of Dynamic Surgical Suture Threads 26 3.1 Introduction................................ 27 3.2 LiteratureReview............................. 28 3.3 SutureDetection ............................. 32 3.3.1 SutureThreadSegmentation. 33 3.4 SutureThreadNURBSModel . 36 3.5 Initializing the Suture Thread . 37 3.6 NURBSCurveIteration ......................... 41 3.6.1 ImageEnergy ........................... 41 3.6.2 EndPointEnergy......................... 42 3.6.3 PointForceAction ........................ 43 3.6.4 PointwiseUpdate......................... 44 3.6.5 3-DimensionalDeprojection . 45 3.7 ExperimentalValidation ......................... 47 3.7.1 QuantativeAccuracy . .. .. 50 3.7.2 CalibratedPatternTracking . 51 3.7.3 Qualitative Tracking Results . 53 3.8 DiscussionandConclusions . .. .. 56 4 Catadioptric Stereo Tracking for Three Dimensional Shape Mea- surement of MRI Guided Catheters 58 4.1 Introduction................................ 59 4.2 Background ................................ 61 4.2.1 CatheterControlandTracking . 61 4.2.2 CatadioptricStereo........................ 62 4.3 CatadioptricHardwareSetUp. 65 4.3.1 Catadioptric Calibration Process . 67 4.4 ExperimentalValidation ......................... 70 ii CONTENTS 4.5 CatheterTracking............................. 71 4.5.1 CatheterModel .......................... 72 4.5.2 CatheterImagingandTracking . 72 4.6 ConclusionsandFutureWork . 74 5 Modeling of Needle-Tissue Interaction Forces During Surgical Su- turing 76 5.1 Introduction................................ 77 5.2 NeedleForceModeling .......................... 78 5.3 SutureNeedleMotionModel. 80 5.3.1 IdealNeedleMotion ....................... 80 5.3.2 NonIdealNeedleMotion. .. .. 80 5.3.3 AreaSweep ............................ 82 5.4 SutureNeedleForces ........................... 85 5.4.1 FrictionForces .......................... 85 5.4.2 AreaForces ............................ 86 5.4.3 CuttingandStiffnessForces . 87 5.4.4 Torque Calculations . 87 5.5 Results................................... 88 5.5.1 ExperimentalMethods . 88 5.5.2 ForceDataPostProcessing . 89 5.5.3 MeasuredForceData....................... 90 5.5.4 ParameterFitting......................... 90 5.6 ConclusionsandFutureWork . 97 6 Needle Path Planning for Autonomous Robotic Surgical Suturing 98 6.1 Introduction................................ 98 6.2 BestPracticesofSuturing . 100 iii CONTENTS 6.2.1 Quantification of the Suturing Guidelines . 101 6.3 Needle Path Planning Algorithm . 103 6.3.1 NeedleApproach ......................... 104 6.3.2 NeedleBite ............................ 105 6.3.3 Needle Reorientation . 105 6.3.4 NeedleRegrasping ........................ 110 6.3.5 Needle Follow Through . 110 6.3.6 NeedlePathInputList. 111 6.4 EmpiricalPathEvaluation . 111 6.4.1 Needle Drive Results . 113 6.5 ResultsandDiscussion .......................... 114 6.6 ConclusionsandFutureWork . 115 7 Conclusions 117 7.1 SutureThreadTracking . .. .. 117 7.2 Catadioptric Stereo Tracking of an MRI Guided Catheter . 118 7.3 Suture Needle-Tissue Interaction Force Modeling . 118 7.4 SutureNeedlePathPlan . .. .. 119 7.5 FutureResearchProblems . 119 Bibliography 122 iv List of Tables 3.1 SutureThreadLengthTable. 50 3.2 CalibrationCurveResultsSummary . 53 4.1 TestPatternTrackingAccuracy . 70 4.2 TrackedCatheterLength. 74 6.1 NeedleDriveParameters . 111 6.2 NeedleDriveForceSummary . 114 v List of Figures 1.1 MISPatientPortalConstraint. 4 1.2 MISSurgicalTool............................. 4 1.3 PrototypeRAMISWrist[10]. ...................... 5 1.4 da Vinci Si System c 2009 Inuitive Surgical Inc.. ........... 6 2.1 SutureNeedleKit............................. 15 2.2 MedicalForceSensor ........................... 18 2.3 StateActionTransition.......................... 24 3.1 SutureNeedleandThread ........................ 30 3.2 ThinFeatureSegmentation . 36 3.3 SegmentedRegionGrowth . .. .. 40 3.4 SutureModelOverlay .......................... 48 3.5 XY-LinearStage ............................. 49 3.6 ThreadCalibrationPattern . 52 3.7 CalibrationPatternOverlay . 53 3.8 Initializing Intersecting Threads . 54 3.9 TrackingaKnotTie ........................... 55 4.1 CatadioptricMirrorGeometry. 64 4.2 CatadioptricImagingSystemComponents . 65 4.3 CatadioptricMRIDiagram. .. .. 67 vi LIST OF FIGURES 4.4 Catadioptric Validation Pattern Tracking . 68 4.5 CatheterLengthMeasurement. 72 4.6 CatheterMRIView............................ 73 5.1 CanonicalNeedleMotion......................... 81 5.2 IdealNeedleMotion ........................... 81 5.3 NeedleMotionAreaSweep........................ 85 5.4 NonIdealNeedleAreaSweep . 86 5.5 ExperimentalSutureApparatus . 91 5.6 ExperimentalForceData. .. .. 92 5.7 LinearForceModel............................ 93 5.8 TorqueModel ............................... 94 5.9 ForceModelBreakdown ......................... 95 5.10 TorqueModelBreakdown . 96 6.1 SutureCrossSection ........................... 101 6.2 NeedleTissueBitePose ......................... 102 6.3 NeedleDepthCalculation . .. .. 103 6.4 Holonomic Needle Reorientation . 104 6.5 Non Holonomic Needle Reorientation . 108 6.6 NeedleDriveTimeLapseComparison. 109 6.7 HolonomicNeedleDrive ......................... 110 6.8 NonHolonomicNeedleDrive. 111 6.9 NeedleDriveForces............................ 112 vii Acknowledgments I would like to thank my advisor Dr. M. Cenk C¸avu¸so˘glu for his guidence and patience. Whatever my new idea was, Cenk always listened and helped me focus my thoughts and mentored me to become a better researcher. Thank you for all of the time, advice, and support that enabled me to grow as a successful robotics engineer. I would also like to thank all of the members of the MeRCIS lab. In particular, I would like to thank Der-Lin Chow, Tipakorn Greigarn, Taoming Liu, and Mark Renfrew who helped me with lab projects or discussing research ideas. I would also like to thank the undergraduate students whom I have mentored during my graduate student career. I hope they learned as much from me as I did from them. Finally, I want to thank my family and friends for their support throughout my academic career. My mom, Barbara Jackson, for her support and encouragement as I trained to become an engineer. My dad, Jay Jackson, while he did not get to see me attend graduate school, passed on invaluable advice that only someone who attended graduate school would know. I would also like to thank Tom and Kathy Mumaw for being supportive of my graduate studies and their thoughtful questions which helped me think about the potential of my research. Most importantly my wife, Michele Mumaw, who was always there for me when I was working late, frustrated by broken software, or needed a friend. Thank you Michele for attending graduate school with me and for your encouragement and wis- dom that enabled me to finish. viii Algorithms for Intelligent Robotic Surgical Systems Abstract by RUSSELL C JACKSON Robotic surgical assistants provide a novel way to improve the versatility and effec- tiveness of minimally invasive surgery but are still a maturing technology. There are many limitations associated with these robots which include a lack of haptic feedback, constrained nonintuitive workspace, and visual distortion. It is difficult to directly address many of the above limitations as the dynamics and kinematics of the robotic arms are significantly different than human anatomy. The research in this dissertation aims to overcome some of the above limitations by addressing problems related to the automation or surgical subtasks such as suturing. Automating subtask completion would overcome many limitations of robotic surgical assistants and allow the surgeon to complete procedures faster and with less fatigue. Ultimately, the surgeon would rely on the robot to perform common surgical subtasks, enabling the surgeon to focus on the overall surgical procedure. This work decomposes the problem of automated surgical subtasks into three pri- mary parts: perception, planning, and control. Advances are made in