PHD DISSERTATION Ricardo Ribalda Delgado
Total Page:16
File Type:pdf, Size:1020Kb
Escuela Politecnica´ Superior Departamento de Tecnologıa´ Electronica´ y de las Comunicaciones AN INNOVATIVE VISION SYSTEM FOR INDUSTRIAL APPLICATIONS PHD DISSERTATION Ricardo Ribalda Delgado Madrid, October 2015 Supervised by Javier Garrido Salas PROCEEDING The Dissertation Committee approves, and recommends for acceptance for the degree of Doctor of Philosophy, the Thesis by Ricardo Ribalda Delgado: AN INNOVATIVE VISION SYSTEM FOR INDUSTRIAL APPLICATIONS. Dr. Jesus´ M. Gonzalez´ Dr. Guillermo Gonzalez´ Barahona de Rivera Peces Chairman Secretary Dr. Brian Vinter Dr. Bjarke Jørgensen Dr. Gustavo Sutter Capristo With the grade of: Madrid, 20th. November 2015 i ABSTRACT Despite the fact that computer vision systems place an important role in our society, its structure does not follow any standard. The implementa- tion of computer vision application require high performance platforms, such as GPUs or FPGAs, and very specialized image sensors. Nowadays, each manufacturer and research lab develops their own vision platform independently without considering any inter-compatibility. This Thesis introduces a new computer vision platform that can be used in a wide spectrum of applications. The characteristics of the platform has been defined after the implementation of three different computer vision ap- plications, based on: SOC, FPGA and GPU respectively. As a result, a new modular platform has been defined with the following interchange- ably elements: Sensor, Image Processing Pipeline, Processing Unit, Ac- celeration unit and Computer Vision Stack. This thesis also presents an FPGA synthetizable algorithm for performing geometric transforma- tions on the fly, with a latency under 90 horizontal lines. All the software elements of this platform have an Open Source licence; over the course of this thesis, more than 200 patches have been contributed and accepted into different Open Source projects like the Linux Kernel, Yocto Project and U-boot, among others, promoting the required ecosystem for the creation of a community around this novel system. The platform has been validated in an industrial product, Qtechnology QT5022, used on diverse industrial applications; demonstrating the great advantages of a generic computer vision system as a platform for reusing elements and comparing results objectively. iii RESUMEN A pesar de que los sistemas de vision´ por computadora ocupan un puesto predominante en nuestra sociedad, su estructura no sigue ningun´ estandar.´ La implementacion´ de aplicaciones de vision´ requiere de pla- taformas de alto rendimiento tales como GPUs o FPGAs y el uso de sen- sores de imagen con caracter´ısticas muy distintas a las de la electronica´ de consumo. En la actualidad, cada fabricante y equipo de investigacion´ desarrollan sus plataformas de vision´ de forma independiente y sin ningun´ tipo de intercompatibilidad. En esta tesis se presenta una nueva plataforma de vision´ por computador utilizable en un amplio espectro de aplicaciones. Las caracter´ısticas de dicha plataforma se han definido tras la implementacion´ de tres aplicaciones de vision,´ basadas en: SOC, FPGA y GPU, respectivamente. Como resultado, se ha definido una plataforma modular con los siguientes componentes intercambiables: Sensor, procesador de imagenes´ ”al vuelo”, unidad de procesado prin- cipal, acelerador hardware y pila de software. Asimismo, se presenta un algoritmo para realizar transformaciones geometricas,´ sintetizable en FPGA y con una latencia de tan solo 90 l´ıneas horizontales. Todos los elementos software de esta plataforma estan´ desarrollados con licencias de Software Libre; durante el trascurso de esta tesis se han contribuido y aceptado mas´ de 200 cambios a distintos proyectos de Software Libre, tales como: Linux, YoctoProject y U-boot, entre otros, promoviendo el ecosistema necesario para la creacion´ de una comunidad alrededor de esta tesis.Tras la implementacion´ de la plataforma en un producto comer- cial, Qtechnology QT5022, y su uso en varias aplicaciones industriales se ha demostrado que es posible el uso de una plataforma generica´ de vision´ que permita reutilizar elementos y comparar resultados objetiva- mente. v A mis padres y hermana A Mariana ACKNOWLEDGEMENTS During this (long) journey I have had the pleasure and the chance of being surrounded by an outstanding group of people. The first person I want to thank is my supervisor, Javier Garrido. He has been undoubtedly the pillar that has supported all the pieces to- gether and has lead them to this final composition. He has been a guide both in my academic and personal life. Muchas gracias Javier, esta Tesis es tuya. The second person I want to thank is my friend and current boss, Kristian Madsen. He has given me freedom to work on my Thesis and trust to execute my ideas in his company. During the last five years I have learn the meaning of the word Engineer. Tusind tak Kristian. Then I want to thank Tom Neubert and the rest of the team at Juelich¨ (George, Heinz, Tobias, Anne... ): I have enjoyed every single day of my internship in your lab where I discover my passion, and now, my work: Embedded Design. Vielen dank Tom! I also want to thank the Open Source community for their help and patience, I am particularly thankful to Hans Verkuil and the rest of linux- media. It is a pleasure to mention the place that saw and valued my first steps as an engineer and which I remember always with happiness: HCTLAB and all the people there (past and present): Susana, Guillermo, Angel,´ Pepe, Fernando, Alberto, Javier Tejedor, Victor, Daniel, Antonio, Nico.... I do not forget the rest of Qtechnology, a lot of the work shown here has been done with their support: Alexandre Desnoyers is the author of most of the PCBs of the QT5022, and a walking encyclopedia for anything with a wire. Finn Nielsen has implemented most of the custom FPGA cores for the QT5022. Jean Claude Thibault has designed the rest of the PCBs and cores. Marianna Buschle has implemented the calibration tools of the device. Dimitrios Katsaros has worked on the ix test framework that validate the cameras. Thomas Clausen is the author of the 3D designs. Without their work, this Thesis would be nothing more than another prototype made with a commercial board that has no use outside the Academia. Other friends have reviewed this text with infinite dedication. Juanjo, you have imprinted a very professional level in this text, thank you very much for all the time expend and I wish you all the best with your new child. I should mention also two brave colleagues that have made the tough decision to leave Spain to continue with their careers: Bruno and Car- men, thanks for your help mates ! I want to publicly acknowledge my parents for their support during my entire life. Muchas gracias papas,´ todo en esta vida os lo debo a vosotros. Nunca podre´ daros suficientes gracias por los sacrificios que habeis´ hecho por mi hermana y por m´ı. And of course my wife, for her infinite patience during these years, for being the travel partner of my life and for giving me support and love every single time that I needed it. I have probably forgot to mention a lot of people, I’m sorry about that, as I said, this has been a very long journey and I am getting too old to even remember my own name :). Finally, I want to thank you, yes you! whoever you are, for using your time reading this Thesis. I hope that you can enjoy reading it as much as I enjoyed writing it. x CONTENTS i introduction and state of the art 1 1 introduction 3 1.1 Motivation 5 1.2 Design Goals 7 1.3 History of the Project 8 1.4 Structure of the document 9 2 state of the art 11 2.1 Definition 13 2.2 Industrial Computer Vision 14 2.3 Computer Vision Stages 14 2.4 Computer Vision Computation Architectures 16 2.4.1 Digital Signal Processors 16 2.4.2 Field Programmable Gate Arrays 17 2.4.3 Cluster Computing 17 2.4.4 General Purpose Graphics Processor Unit 17 2.5 Computer Vision Platforms 18 ii custom computer vision systems 21 3 embedded system on chip computer vision 23 3.1 Introduction 25 3.1.1 Classical Biometric Systems 25 3.1.2 X.509 digital certificates 28 3.2 System-on-Token 29 3.2.1 System on Token Architecture 29 3.2.2 Transactions 31 3.3 Implementation 32 3.3.1 Device Selection 32 3.3.2 Biometric Module 33 3.3.3 Certificate Module 35 xi CONTENTS 3.4 Dataflow 35 3.5 Conclusions 36 4 fpgacomputervisionsystem 37 4.1 Introduction 39 4.2 FPGAs with embedded CPU cores 39 4.3 Architecture Description 41 4.3.1 Hardware 41 4.3.2 Software 43 4.4 Verification and Results 45 4.4.1 Hardware Implementation 45 4.4.2 Communication Speed 45 4.4.3 Image processing 46 4.5 Conclusions 47 5 gpucomputervisionsystem 49 5.1 Introduction 51 5.2 Spectrogram acquisition 52 5.3 Processor optimisation 53 5.3.1 Motivation 54 5.3.2 Description of the Operational L0 Processor 54 5.3.3 Performance 56 5.4 GPU acceleration 58 5.5 Conclusion 60 iii modular computer vision system: qt5022 61 6 modular computer vision system 63 6.1 Computer Vision Systems: Design Pattern 65 6.2 Modular Computer Vision System 66 6.3 Interfaces on a Modular Architecture 68 6.4 QT5022: a Modular Computer Vision System 69 6.4.1 Head 70 6.4.2 Body 71 7 data flow 73 7.1 Sensor 75 7.2 Image Processing Pipeline 75 7.2.1 Frame Grabber 76 xii CONTENTS 7.2.2 Pixel Readout 76 7.2.3 Frame bus 83 7.2.4 White Balance 84 7.2.5 XForm 85 7.2.6 Data Packer 89 7.2.7 DMA Engine 92 7.2.8 PCI bridge 94 7.3 Processing Unit 95 7.3.1 Video4Linux2 Input/Output 96 7.3.2 V4L2 Control Interface 98 7.3.3 V4L2 Miscellanea 99 7.4 Acceleration Unit 100