Analyzing Perspective Line Drawings using Hypothesis Based Reasoning by Prasanna Govind Mulgaonkar Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of DOCTOROF PHILOSOPHY in Computer Science and Applications APPROVED: Linda G. Shapiro Robert M. Haralick J'ohn W. Roach Roger W. Ehrich Ezra A. Brown August, 1984 Blacksburg, Virginia ANALYSIS OF PERSPECTIVE LINE DRAWINGS USING HYPOTHESIS BASED REASONING by Prasanna Govind Mulgaonkar Committee Chairman: Dr. Linda G. Shapiro Computer Science (ABSTRACT) One of the important is~ues in the middle levels of com- puter vision is how world knowledge should be gradually in- serted into the reasoning process. In this dissertation, we develop a technique which uses hypothesis based reasoning to reason about perspective line drawings using only the const- raints supplied by the equations of perspective geometry. We show that the problem is NP complete, and that it can be solved using modular inference engines for propagating constraints over the set of world level entities. We also show that theorem proving techniques, with their attendant complexity, are not necessary because the real valued attri- butes of the world can be computed in closed form based only on the spatial relationships between world entities and mea- surements from the given image. ACKNOWLEDGEMENTS ...... For they were the best of times and they were the worst of times A dissertation is not created in isolation. It is the product of the interaction between the people surrounding the author, and the times and the places involved. I would like to thank everyone who in one way or the other contri- buted to making my time as a graduate student worthwhile and in some direct or indirect way helped make this dissertation what it is. Dr. Linda Shapiro, and Dr. Robert Haralick guided me throughout my graduate career. All that I have learnt about computer vision, is directly attributable to their time and patience. Dr. Shapiro supported my graduate education by continuous funding on her NSF grant which is what enabled me to survive. I also want to thank Dr. Roger Ehrich for serv- ing on both my masters and my doctoral advisory committees. Dr. John Roach, and Dr. Ezra Brown also provided valuable input to this work. I am deeply indebted to my wife Anjali to whom this work is dedicated. Without her constant encouragement and sup- port, I would not have been able to go through all the hear- taches and headaches that a PhD. requires. Amit, our three iii month old son did his share by making sure that I stayed awake on the nights that I needed to. Indeed this is a fa- mily PhD. rather than an individual accomplishment. To all the friends at the SDA Lab, who provided a friend- ly and stimulating working environment, I wish to say "Thank You". Especially, Ting-Chuen (King of Threshold) Pong, Greg (IBM Spy) Bushman, Greg (Mr. GIPSY) Fabregas, Tom Laffey, Trinh Ho, Einar Ostevold, and all the other folks helped make the last five years something I can look back on. Espe- cially Hyonam Joo, who helped write, debug and often fix on very short notice, the PROLOG interpreter which was used for all the experiments described in this dissertation. Life has facets other than just academic, and we owe a great deal to all our other friends in Blacksburg. Jyothi and Ramesh, and Nileema and Rajendra made extra-curricular life enjoyable and fun. Finally I want to thank my parents for instilling in me the spirit of hard work and striving for academic excel- lence. Their preparatory work led up to this moment. I would also like to acknowledge the help that my in-laws gave. They looked after our baby and our house in the crucial months leading up to my defense so that I could be free to concen- trate on my work and my wife could finish her master's de- gree without any distractions. iv TABLE OF CONTENTS ABSTRACT . ii ACKNOWLEDGEMENTS iii Chapter I. INTRODUCTION 1 II. HYPOTHESIS BASED REASONING FOR VISION. 7 Introduction ........... 7 Hypothesis Generation and Testing 9 The Optimal Hypothesis Problem ...... 11 NP Completeness of the Optimal Hypothesis Problem ..... 15 Formal Definition of CLP ....... 16 Reformulation of the CLP ...... 17 Equivalence of the Two Formulations 17 The Optimal Hypothesis Problem and the Hypothesis Decision Problem 18 Transformation of the CLP to HD 19 Determining Consistency of a Hypothesis . 24 Inference Engines ............... 26 Characterization of Inference Engines . 40 Predicate Calculus with Restrictions .... 41 Inference Engines as a Reasoning Path. 44 Searching for the Maximal Hypothesis. 48 Searching the Lattice. 50 The Search Space as a Binary Tree. 59 Forward Checking. 64 Logical Inconsistency and Completeness for Pruning . • • 68 Applications to Computer Vision . 70 III. CONTROL STRUCTURES FOR REASONING 72 Requirements for the Control. ..... 74 Controlling the Inference Engines . 77 Agenda Based Control Strategy .. 80 The Network Model for Module Control .... 83 Control of Inference Engines using Associative Tables . 88 Conclusion . 92 V IV. KNOWLEDGESOURCES FOR ANALYZING PERSPECTIVE IMAGES 94 Geometry of Perspective Projections of Lines . 97 Vanishing Points ..... 100 Vanishing Trace of a Plane 103 Computing Focal Length 107 Center of the Screen Coordinate System 111 Projection of Points .... 112 Projection of a Rectangle .. 115 Perspective Projection of a Circle 120 Perspective Projection of a Circle 121 Plane of the Circle Parallel to the Screen 124 Inverse of the projection ... 125 Circle on a Z = Constant Plane ..... 126 Inverse of the Projection .... 127 Circle on any Plane Perpendicular to the Screen . 128 Two Solutions for the Plane of the Circle 133 Inverse Projection Properties for General Circles . .. 139 Computing the Focal Length 140 Normal to the Plane of the Circle Given the Focal Length ......... 142 Normal to the Circle Given the Projection of the Center ..... 143 Comments on the Approach of Chu 145 Logical Consistency Rules 148 V. IMPLEMENTATIONAND EXPERIMENTALRESULTS 154 Introduction . 154 Input Data Structure ....... 155 Global Tables and Control of Engines 161 Control of Inference Engines 163 Experimental Results ..... 164 Analysis of Timing and Solutions Generated 164 VI. LITERATURE REVIEW. 192 Vision Systems .. 192 Brooks ACRONYMsystem. 192 Generalized Blob technique 195 The Hanson and Riseman VISIONS system 197 The Levine image segmentations system ... 199 Knowledge Based Systems 201 Iv'I.ACSYMA. 201 DENDRAL . 202 MYCIN ..... 203 PIP . 205 HEARSAY-II .... 205 vi Other Relevant Research 206 VII. CONCLUSION 210 BIBLIOGRAPHY 212 Appendix A. DESCRIPTION OF THE PROLOGAND RATFOR CODE. 217 The top level .... 217 The Backtracking Tree Search. 219 State recording .... 225 Application of Inference Engines. 226 Trace information ........ 232 Efficiency of the Search. 233 B. INPUT DATA FOR TEST EXAMPLES 235 Architectural Scene ..... 235 The Table Scene 250 Image of a rectangle 258 Fountain Image ....... 260 Image of a rectangular parallelipiped 264 Complex Image of the Table Scene. 270 Towers Image ...... 272 C. RESULTS FOR THE SAMPLE DATA 274 Results for the Architectural Scene 274 Results for the Table image . 313 Results for the Table image given the focal length . 323 Results for the rectangle in the Z plane. 328 Results for the fountain scene . 330 Results for the Rectangular Parallelipipid 335 Inconsistent Data . 340 Complex Table Image . 344 Results for the Tower Image 357 D. DATA FOR TIMING EXPERIMENTS ... 368 VITA 371 vii LIST OF FIGURES Figure paae 1. Lattice corresponding to the example .. 54 2. The algorithm for searching a lattice. 57 3. Tree representation for the example . 60 4. A subtree rooted at an internal node . 67 5. The camera coordinate system . 99 6. Computing the focal length. 109 7. The perspective projection of a rectangle 116 8. Rotation of the axes ... 129 9. The cone of rays from th,e origin . 135 10. The right elliptical cone 136 11. The two solutions for the ci~cle. 138 12. Constructing the projection of the center 144 13. An architectural scene 165 14. The drawing of a table scene. 166 15. The base of an ornate fountain 168 16. Number of calls without chunking. 170 17. Average CPU time in seconds 171 18. Efficiency of the process 172 19. Number of consistency checks. 173 20. Average CPU time in seconds per solution. 174 21. Efficiency of subset checking .... 175 22. Number of hypotheses vs. size 176 viii 23. Total CPU time vs. size of hypothesis 177 24. Average CPU time vs. size of hypothesis 178 25. Number of calls vs. size of hypothesis 179 26. Total CPU time vs. size of hypothesis 180 27. Average CPU time vs. size of hypothesis 181 28. Number of hypotheses by type. 182 29. Average CPU time by type ... 183 30. Number of hypotheses by type. 184 31. Average CPU time per hypothesis by type 185 32. Effect of chunking ....... 186 33. Effect of chunking on average CPU time .. 187 34. Effect of forward checking ...... 188 35. Average CPU time with forward checking 189 36. Digitized architectural scene 236 37. The Digitized Table Image 251 38. Synthetic image of a rectangle .. 259 39. Digitized image of a fountain 261 40. Image of a Rectangular Parallelipiped 265 41. Complicated line drawing of the table scene 271 42. Artificial image of two towers .. 273 43. Test data for timing experiments .. 369 ix Chapter I INTRODUCTION The central problem in computer vision, is one of produc- ing an automatic system capable of analyzing unconstrained pictorial information of the same complexity and with the same accuracy as the human visual system. This is currently an unsolved problem. However, we do know that the solution of this problem must involve many diverse knowledge sources and applications of techniques from a wide variety of dis- ciplines.