Go-Green or Go-Home: Using computer vision for real-time traffic monitoring Guðjón Björnsson Þórður Friðriksson Thesis of 12 ECTS credits submitted to the School of Computer Science at Reykjavík University in partial fulfillment of the requirements for the degree of Bachelor of Science in Software engineering May 15, 2020 Supervisor: Gylfi Þór Guðmundsson Examiner: Guðmundur Már Einarsson Acknowledgements We want to thank Reykjavík University for providing us facilities while working on this project, even though we did not get a chance to utilize them the whole time due to the Covid-19 pandemic. Thank you Gylfi Þór Guðmundsson for the around the clock guidance and help and special thanks go to Hallgrímur Arnaldsson for proposing the project. Lastly thank you Guðmundur Már Einarsson for your valuable feedback. i [This page is intentionally left blank] ii Contents 1 Introduction1 2 Background2 2.1 Machine learning................................2 2.2 Computer vision................................3 2.3 Object detection................................3 2.4 Neural networks.................................4 2.5 The Icelandic license plate...........................9 2.6 Optical character recognition (OCR)..................... 11 3 Methods 14 3.1 System design overview............................ 14 3.2 Data collection................................. 15 3.3 Choosing and training the object detector(s)................. 15 3.4 Vehicle tracking................................. 16 3.5 Plate reading.................................. 17 3.6 Passenger counting............................... 22 4 Experiments and evaluation 23 4.1 Data used in evaluation............................ 23 4.2 Evaluating passenger counting......................... 24 4.3 Evaluating OCR................................ 26 4.4 Evaluating license plate reading........................ 27 4.5 Performance evaluation of the object detectors............... 28 5 Future work 29 5.1 Hardware changes............................... 29 5.2 Data and training................................ 30 5.3 Different types of DNNs............................ 30 5.4 Cloud solutions................................. 31 5.5 OCR....................................... 31 5.6 Training Tesseract............................... 31 5.7 Improving OCR with explicit rules...................... 32 5.8 Combining YOLO and OCR.......................... 32 5.9 Optimization of the object detector...................... 32 5.10 Improve passenger counting.......................... 32 5.11 Statistical filtering............................... 32 6 Conclusion 33 7 Appendix 36 7.1 Character prediction table........................... 36 iii Abstract With growing population, traffic and air pollution have become a real problem and the need for eco-friendly solutions dire. In our busy world, people tend to make the easy choice instead of the right one. There is therefore an urgency for incentives and penalties to help guide people towards the right and environmentally friendly choices. In this thesis we dive in to the implementation of a system for real-time road-side monitoring of traffic using deep neural networks and computer vision, with the intent of counting a vehicle’s occupants and capturing it’s license plate registration. We could then use the license plate registration to look up a vehicles model, weight and emission of greenhouse gases using the Icelandic department of motor vehicles’ online look up service. With this information we can determine if the vehicle meets Iceland’s standard for a green vehicle. This information can be used to aid in the enforcement of reward systems such as car-pool lane monitoring, automated toll booth charging or priority parking accesses. We evaluate two state of the deep learning networks and show that we can successfully predict the correct number of front seat passengers with an accuracy of 89%, and obtain the correct license plate reading with an accuracy of 81%. Furthermore we propose solutions for improving the system and achieving better accuracy and performance. iv 1 Introduction The world is changing for the worse, both on land and in the sea, and the changes are happening faster then ever before [1], over the last 50 years, nature’s capacity to support us has plummeted. Air and water quality are reducing, soils are depleting, crops are short of pollinators, and coasts are less protected from storms [2] and we as humans are to blame. One of our bigger concerns regarding this problem is our contribution to air pollution via vehicle emissions and with the ever constant growth of the human population combined with the fact that the number of registered vehicles in Iceland is increasing at an even greater rate [3] the prospect of a future with clean air seems bleak, so how can we help relieve this environmental pressure being put on by cars and other motorized vehicles? Some might suggest to abandon car’s altogether, returning to alternative and more eco-friendly means of transportation such as walking or cycling. This is a fine and noble idea but unfortunately it is not a feasible one. As difficult as it is to admit, for society to continue to function with the same productivity, we need fast and reliable means of transportation. So the problem statement is simple, we need to make automotive transportation pollute less all the while keeping society’s throughput equal to current levels. The solution, as with any other problem, might not be as straightforward as we would like to think. There are three main ideas on how to solve this problem. The first is to use public transits like buses, trams or trains which will both reduce air pollution as well as road congestion. The downside being that such modes of transportation will not get everyone exactly where they need to go nor does that transport you as quickly as your own car. The second idea is to drive fuel efficient vehicles with low or even no greenhouse gas emissions. These cars can help the environment while potentially saving you money on fuel costs at the pump. These vehicles can for example be electric, hybrids, hydrogen fuel cell vehicles or even simply cleaner burning gasoline vehicles. The third idea is to carpool with other people who are heading in the same direction and thus share the ride, the cost and the emission of green house gases. In this thesis we will look into ways to better motivate people to utilize the two latter of the three ideas, that is to motivate people to drive more fuel efficient cars and to adopt ride sharing into their daily routines. To achieve this we dive in to the implementation of a traffic monitoring system aimed at distinguishing those who choose environmentally friendly vehicles and ride sharing. If such a system where to exist in a sufficiently general and a small enough form factor it could be deployed in various locations, rewarding individuals who choose a more sustainable way of living. To construct such a system, we research the world of object detection and computer vision to see if it is possible to read and look up registration plates to recognize fuel efficient vehicles, as well as to identify ride sharing by detecting how many passengers are in the front seats. We also research the possibility of doing all this under the constraints of a real time application, which means that our detection needs to be both accurate and fast. 1 The rest of the thesis will go into detail of how this research was conducted, and can be broken up into five chapters. • The background which gives a summary of all the concepts and theory that the research builds upon • The methods chapter that discusses the process and approach that is taken while conducting the research. • The result chapter that displays unbiased raw data from the results of our research. • The discussion and future work chapter that gets our take on the results as well as discusses what work is yet to be done. • Finally the conclusion chapter which is a simple summary and final words of our research and report. Overall, our experiments give us the accuracy and speed at which our implemented system performs as well as to give us an idea of how well it can potentially perform if deployed as a real and usable application in the real world. 2 Background In this chapter we will cover the background material that we build our work upon, starting with a discussion on machine learning. 2.1 Machine learning In most common programming practices the algorithm is written by a programmer, telling the computer exactly how to perform a given task. To solve for advanced tasks, programming can be difficult and very time consuming. Machine learning involves com- puters discovering how they can perform tasks without being explicitly programmed to do so by a programmer. Machine learning algorithms construct a mathematical model from the data given, known as training data. The mathematical model can then be used to make predictions, or take decisions, based on new and previously unseen data. Many machine learning approaches exist but they most commonly fall in to one of the following three categories [4]. • Supervised learning: The computer is presented with example inputs and their desired outputs, given by a “supervisor”, and the goal is to learn a general rule that maps inputs to outputs. • Unsupervised learning: No labels are given to the learning algorithm, leaving it on its own to find structure in its input. Unsupervised learning can be a goal in itself (discovering hidden patterns in data) or a means towards an end (feature learning). 2 • Reinforcement learning: A computer program interacts with a dynamic environ- ment in which it must perform a certain goal (such as driving a vehicle or playing a game against an opponent) As it navigates its problem space, the program is provided feedback that’s analogous to rewards, which it tries to maximise. 2.2 Computer vision Computer vision often abbreviated CV is a scientific field that deals with the study of how a computer sees and understands the content of images and videos. A computer sees an image as a bunch of 1’s and 0’s, those 1’s and 0’s make up pixels which are the smallest element of an image.