Real Time Distance Calculation using Stereo Vision Technique Session 2005-2009 Project Advisor Mr. Khan Asmar Azar Submitted by: Ahmed Tassaduq 050620-158 Aisha Ashraf 050620-128 Fatima Zehra Hassan 060820-089 Shajieuddin Hyder Khan 050620-086 Department of Electrical Engineering University of Management and Technology Real Time Distance Calculation Using Stereo Vision Technique Page i A report submitted to the Department of Electrical Engineering In partial fulfillment of the requirements for the Degree Bachelor of Science in Electrical Engineering by Ahmed Tassaduq (050620-158) Aisha Ashraf (050620-128) Fatima Zehra Hassan (060820-089) Shajieuddin Hyder Khan (050620-086) University of Management and Technology October 12, 2009 Real Time Distance Calculation Using Stereo Vision Technique Page ii CERTIFICATE OF APPROVAL It is certified that the work contained in this project report, entitled “Real Time Distance Calculation using Stereo Vision Technique” carried out by Ahmed Tassaduq (050620-158) Aisha Ashraf (050620-128) Fatima Zehra Hassan (060820-089) Shajieuddin Hyder Khan (050620-086) Under the supervision of Mr. Khan Asmar Azar for the partial fulfillment of the degree requirement of Bachelor in Electrical Engineering Approved By ________________ __________________ Dr. Aziz Bhatti Khan Asmar Azar Dean SST Project Advisor Real Time Distance Calculation Using Stereo Vision Technique Page iii Acknowledgements We truly acknowledge the cooperation and help make by our advisor Mr. Khan Asmar Azar, University of Management and Technology. He has been a constant source of guidance throughout the course of this project. We would also like to thank Mr. Saeed-ur-Rehman Turk, Government College University, Lahore for his help and guidance in understanding many important issues. We are also thankful to our friends and families whose silent support led us to complete our project. (Signed) Ahmed Tassaduq (050620-158) Aisha Ashraf (050620-128) Fatima Zehra Hassan (060820-089) Shajieuddin Hyder Khan (050620-086) Date October 12, 2009 Real Time Distance Calculation Using Stereo Vision Technique Page iv TABLE OF CONTENT Chapter 1: Stereo Vision 1 1.0.1 Introduction 2 1.0.2 Background 7 1.1 Why Stereovision? 7 1.1.1 A few possibilities for camera based range sensing exist 8 1.1.2 Key advantages of camera based systems 8 1.2 Methodology 8 1.2.1 Pixel to Distance Calculation 8 1.2.2 Formula 9 1.2.3 Correlation 10 1.3 Camera calibration can be performed by two techniques 10 1.3.1 Photogrammetric calibration 10 1.3.2 Self-calibration 10 1.4 Proof of Concept 11 1.4.1 Calculation of Disparity 11 1.4.2 Disparity and distance inverse relation 12 Real Time Distance Calculation Using Stereo Vision Technique Page v Chapter 2: USB Protocol and Hardware Implementation 13 2.1 The Hardware 14 2.1.1 A4-Tech PK-5 Webcam (ZC0301) 14 2.1.2 Xilinx Spartan 3 starter board 14 2.1.3 A computer 14 2.2 USB Protocol 14 2.2.1 How to use a Core? 15 2.2.2 Issues with pure HDL 15 2.2.3 Embedded development kit 15 2.2.4 Creating a USP IP core 25 Chapter 3: uClinux and Device Integration 51 3.1 Matlab 52 3.2 MicroClinux 52 3.3 Operating System 53 3.3.1 Microblaze & Xilkernel 53 3.3.2 Purpose of an OS 53 3.3.3 Responsibilities of OS 53 3.4 Device Manager 53 3.4.1 Kernel 53 3.4.2 Device drivers 54 3.4.3 Device controller 54 Real Time Distance Calculation Using Stereo Vision Technique Page vi 3.5 Driver Issues 55 3.5.1 Camera Controller: ZC030x 55 3.5.2 Driver GSPCA 55 3.5.3 JPEG decoder 55 3.5.4 Compiler gcc-3.4 55 3.5.5 Additional libraries 55 3.6 Suitable Kernel 55 3.6.1 Why not Xilkernel? 55 3.6.2 Why import another Linux kernel? 56 3.6.3 Suitable kernel for device driver GSPCA 56 3.7 Grabbers 57 3.7.1 Spcagui frame grabber 57 3.7.2 Spcaview frame grabber 57 Chapter 4: Image Processing (C++ and VHDL) 58 4.1 Disparity 59 4.2 Template Matching 59 4.2.1 Methods 60 4.2.1.1 Sum of Absolute Differences (SAD) 60 4.2.1.2 Sum of Square Differences (SSD) 60 4.2.1.3 Normalized Cross Correlation (NCC) 60 4.3 Applied Algorithm 61 Real Time Distance Calculation Using Stereo Vision Technique Page vii 4.3.1 Implementation 61 4.5 Hardware Implementation 61 CONCLUSION 63 Future Projects 63 Appendix A: Correlation (C++) 64 Appendix B : Correlation using State Machine (VHDL) 69 Appendix C: Correlation using For Loop (VHDL) 74 References 78 Real Time Distance Calculation Using Stereo Vision Technique Page viii LIST OF FIGURES Figure 1: Image plane of projection with cameras 3 Figure 2: Overview of the visual pathways from eyes to striate cortex 4 Figure 3: Image rectification 4 Figure 4: Simple lens model 5 Figure 5: Concept of disparity calculation 6 Figure 6: Block diagram of convex lens. 9 Figure 7: two Images taken for correlation calculation with two cameras. 11 Figure 8: the offset calculation in the left and right CCDs. 12 Figure 9: Hardware 50 Figure 10: Independent and dependent parts of the device manager. 54 Figure 11: communication between hardware and software layers. 56 Figure 12: above figure shows the X offset in images 59 Figure 13: Images for template matching 59 Figure 14: Mask 61 Real Time Distance Calculation Using Stereo Vision Technique Page ix Chapter 1 Stereo Vision Real Time Distance Calculation Using Stereo Vision Technique Page 1 Abstract The word "stereo" comes from the Greek word "stereos" which means firm or solid. Stereo vision you can see an object in three spatial dimensions i.e. according to its width, height and depth simply according to its x, y and z-axis. To make stereovision so rich and special in its attributes and qualities concept of depth dimension i.e. Disparity calculation has been added. Stereoscopic vision probably evolved as a means of survival. With stereo vision, we can see where objects are in relation to our own bodies with much greater precision especially when those objects are moving toward or away from us in the depth dimension. We can see a little bit around solid objects without moving our heads and we can even perceive and measure "empty" space with our eyes and brains. According to the web site of the American Academy of Ophthalmology, September, 1996: "many occupations are not open to people who have good vision in one eye only [that means people without stereo vision]".Basically each eye confine its own view and the two separate images are sent on to the brain for progression. These two images turn up simultaneously in the back of the brain, where they are united into one picture. The brain combines the two images by matching up the similarities and adding in the small differences. This small difference between the two images adds up to a big difference in the final picture. Leonardo da Vinci had also realized that “objects at different distances from the eyes project images in the two eyes that differ in their horizontal positions, but had concluded only that this made it impossible for a painter to portray a realistic depiction of the depth in a scene from a single canvas”. Leonardo chose for his near object a column with a circular cross section and for his far object a flat wall. Had he chosen any other near object, he may have discovered horizontal disparity of its features. His column was one of the few objects that projects identical images of itself in the two eyes [1]. Stereopsis (from stereo meaning unadulterated, and opsis meaning vision) is the process in visual perception leading to the sensation of depth from the two slightly different projections of the world onto the retinas of the two eyes. The differences in the two retinal images are called horizontal disparity, or retinal disparity, or binocular disparity. The differences arise from the eye’s different positions in the head. Stereopsis is commonly referred to as depth perception. This is inaccurate, as depth perception relies on many more monocular cues than stereoptical ones, and individuals with only one functional eye still have full depth perception except in artificial cases (such as stereoscopic images) where only binocular cues are present. Stereopsis became popular during Victorian times with the invention of the prism stereoscope by David Brewster. This, combined with photography, meant that tens of thousands of stereograms were produced. Until about the 1960s, research into stereopsis was dedicated to exploring its limits and its relationship to singleness of vision. Researchers included Peter Ludwig Panum, Ewald Hering, Adelbert Ames Jr., and Kenneth N. Ogle. Profundity perception is the visual ability to comprehend the world into three dimensions. Although animals are capable to sense the distance of objects in that environment, the term perception is reserved for humans, who are, as far as is known, the only beings that can tell each other about their experiences of distances. Depth sensation is the ability to move accurately, or to Real Time Distance Calculation Using Stereo Vision Technique Page 2 respond consistently, based on the distances of objects in an environment. With this definition, every moving animal has some sensation of depth. Depth perception arises from a variety of depth cues. These are typically classified into binocular cues that require input from both eyes and monocular cues that require the input from just one eye. Binocular cues include stereopsis , yielding depth from binocular vision through exploitation of parallax.