A Web-Based Software Platform for Data Processing Workflows and its Applications in Aerial Data Analysis
A thesis submitted to the graduate school of the
University of Cincinnati
in partial fulfilment of the requirements for the degree of
Master of Science in the department of Electrical Engineering and Computer Science of
the College of Engineering and Applied Sciences
June, 2019
by
Niranjan R Krishnan
B. Tech, Cochin University of Science and Technology, Cochin, India
2013
Committe Chair: Dr. Arthur Helmicki, PhD
Committee Members:
Dr. Victor Hunt, PhD
Dr. Nan Niu, PhD Abstract
Given the rapid advances in development of unmanned aerial vehicles (UAV), em- ployment of drones in various business functions becomes reasonable and afford- able. Usage of drones as data collection tools will give us access to a new set of geo-referenced images and videos that were not easily accessible in the past. The ultimate objective of this web based platform for data processing workflows, from here on referred to as the common operating platform, is to enable users to archive, process and visualize aerial data without a need for advanced hardware and software locally. This work details the development of the the common operating platform which consist of a web-based frontend and a backend. The frontend is a web app developed based on Django, Twitter Bootstrap and Javascript where the user authenticates, uploads data, submit processing tasks and visualizes the results. The back end, developed using Python 3, is where data is being stored and various processing tasks are being done based on commercial(Pix4D), open source(OpenDroneMap) and custom(Traffic Monitoring) processing engines. First, the intricacies of the data processing workflow is discussed and this in- cludes diving into detail the various steps related to processing workflows using proprietary, open software and custom software. Second, procedures for integrating commercial processing engines as well as development of the in house traffic param- eter extraction system will be shown and the result of running various case studies and their processing performance will be discussed. And finally, the system archi- tecture design and implementation will be detailed given the scalability, modularity, extensibility and reliablity requirements will be discussed. The idea is to have a secure system, which is accessible to a broad audience, that can receive and service all of their processing requirements. In doing so, it dismisses the need for an unini- tiated audience to install highly specialized software on their personal computers as well as relieves the need for them to maintain sophisticated hardware.
i
Acknowledgements
I wish to acknowledge the expertise and feedback of my professors Dr. Arthur Helmicki and Dr. Victor Hunt. Special thanks to my colleagues, Arjun Chiddarwar and Dr. Mahdi Norouzi, for all their help. I would also like to thank the Ohio De- partment of Transportation(ODOT) for funding the research project. Their support gave us all the resources we could ask for to accomplish this task. I would also like to acknowledge the work of the pilots from the UAV Master Labs at the University of Cincinnati for flying the data collection missions for the uninterrupted traffic flow calculations, Aswin Balasubramaniam for his help with testing and benchmarking the platforms capabilities, Sudhamsh Korrapati for his help with implementing the index page on the website and Ganesh Vathyam and ODOT IT for helping migrate the platform to ODOT’s internal servers. Finally, I would like to thank my Mom and Dad for their love, endless support and encouragement.
ii Contents
1 Introduction 1
1.1 Problem Statement ...... 1
1.2 Motivation of Research ...... 2
1.3 Basic Concepts and Definitions ...... 2
1.3.1 3D Modeling ...... 3
1.3.2 Docker and Containerization ...... 4
1.3.3 Deep Learning Frameworks for Computer Vision ...... 6
1.3.4 Object Detection ...... 8
1.3.5 Object Tracking ...... 10
1.3.6 GPU Accelerated Processing ...... 10
1.3.7 Traffic Metrics ...... 11
1.4 Proposed Approach ...... 13
2 Common Operating Platform 15
2.1 Distributed System Architecture ...... 15
2.1.1 The Web Interface ...... 17
2.1.2 Database Design ...... 21
2.1.3 Backend Systems ...... 24
2.2 Installation ...... 26
2.2.1 System Requirements ...... 26
2.2.2 Special Configurations ...... 27
2.3 Templatizing Workflows ...... 33
iii CONTENTS
2.4 Implementation Issues ...... 36
2.4.1 Changing IT Eco-Systems ...... 36
2.4.2 Maximum file upload size ...... 37
2.4.3 Scalability ...... 38
2.5 Results ...... 38
2.5.1 Register and Login ...... 39
2.5.2 Navigating the Dashboard ...... 39
2.5.3 Processing 3D models ...... 40
2.5.4 Visualize Point Clouds ...... 42
2.5.5 Process Traffic Monitoring Projects ...... 44
2.5.6 Change Password ...... 46
2.5.7 Create API Keys ...... 46
2.5.8 Benchmarks ...... 47
3 Custom Software Design: Traffic Monitoring 50
3.1 Objectives ...... 50
3.2 Region of Interest ...... 51
3.3 Feature Extraction ...... 54
3.4 Lane Detection ...... 59
3.4.1 Detecting Lane Lines using Hough Transforms ...... 59
3.4.2 Tracing Lane Lines ...... 63
3.5 Vehicle Trajectory Analysis ...... 64
3.6 Results ...... 67
3.6.1 Counts of Vehicles ...... 68
3.6.2 Flow Rate, Average Travel Speed and Density ...... 70
4 Conclusion 81
4.1 Future Scope ...... 81
References 83 iv CONTENTS CONTENTS
A Point Clouds on Potree Viewer 91
CONTENTS v List of Tables
2.1 Defintion of Table: UploadedFiles ...... 22
2.2 Defintion of Table: DirectoryStructure ...... 22
2.3 Defintion of Table: APIAuth ...... 23
2.4 Defintion of Table: TaskBucket ...... 23
2.5 Defintion of Table: PostProcessedData ...... 24
2.6 Specifications of Production Server ...... 38
2.7 Specifications of Production and Test Servers ...... 48
2.8 Pix4D vs ODM Benchmarks ...... 48
3.1 Human Counts from April 18, 2018 ...... 69
3.2 Machine Counts from April 18, 2018 ...... 69
3.3 Error(%) from April 18, 2018 ...... 70
3.4 Human Counts from June 20, 2018 ...... 71
3.5 Continued - Human Counts from June 20, 2018 ...... 72
3.6 Machine Counts from June 20, 2018 ...... 73
3.7 Continued - Machine Counts from June 20, 2018 ...... 74
3.8 Error(%) from June 20, 2018 ...... 75
3.9 Continued - Error(%) from June 20, 2018 ...... 76
3.10 Total Counts with Error(%) ...... 76
3.11 Aggregate of human and machine counts for all videos with error(%) 77
3.12 Results of experiment conducted to verify average travel speed . . . . 77
3.13 Average Metric Values for Lane 6 and Lane 4 ...... 79
3.14 Average Metric Values for Lane 5 and Lane 1 ...... 79
vi LIST OF TABLES
3.15 Average Metric Values for Lanes 1, 4, 5 and 6 ...... 80
LIST OF TABLES vii List of Figures
1.1 A point cloud of a building ...... 4
1.2 Docker Architecture[7] ...... 5
1.3 Hypervisor Architecture[7] ...... 6
1.4 Relationships among Speed, Density and Flow Rate on Uninterrupted Flow Facilities [36] [37] ...... 14
2.1 System Architecture Diagram ...... 18
2.2 Entity Relationship Diagram ...... 21
2.3 Register a user account ...... 39
2.4 Login Page ...... 39
2.5 Navigation ...... 40
2.6 Create Pix4D/ODM Project ...... 41
2.7 Upload Image Files ...... 41
2.8 Process Project ...... 41
2.9 Backend Jobs ...... 42
2.10 Create a visualization project ...... 42
2.11 Upload Point Cloud Files ...... 43
2.12 Process Project ...... 43
2.13 Backend Jobs ...... 44
2.14 Create Traffic Monitoring Project ...... 44
2.15 Upload Video Files for Traffic Monitoring Project ...... 45
2.16 Process Traffic Monitoring Project ...... 45
2.17 Status of jobs in Backend Jobs ...... 46
viii LIST OF FIGURES
2.18 Change Password ...... 47 2.19 Create / Delete API Keys ...... 47 2.20 Potree viewer displaying the results of the small project ...... 48 2.21 Potree viewer displaying the results of the medium project ...... 49
3.1 Traffic Monitoring Architecture ...... 51 3.2 Object detection using YOLO ...... 52 3.3 Region of interest scatter plot ...... 52 3.4 Region of interest scatter plot with cluster centers ...... 53 3.5 Region of interest applied to a frame ...... 54 3.6 Intersection over Union of two bounding boxes from consecutive frames 55 3.7 Original Frame ...... 60 3.8 Original Frame in HSV Space ...... 60 3.9 Apply Gaussian Blur ...... 61 3.10 Filter Yellow and White Colors ...... 62 3.11 Canny Edge Detection ...... 62 3.12 Hough Transform Lane Detection ...... 63 3.13 Lane Detection Block Diagram ...... 63 3.14 Tracing Lane Lines ...... 64 3.15 Trajectory of a vehicle recorded by the feature extraction module . . 65 3.16 Lane labels assigned by the lane detection module ...... 68 3.17 Comparison of Lane 6 with Lane 4 ...... 78 3.18 Comparison of Lane 5 with Lane 1 ...... 79 3.19 Comparison of lanes 1, 4, 5 and 6 ...... 80
A.1 Point cloud of ODOT HQ, Columbus, OH ...... 91 A.2 Point cloud of a shed, District 8, OH ...... 92
LIST OF FIGURES ix Listings
2.1 Postgres Installation ...... 27 2.2 Postgres Process Status ...... 28 2.3 Edit postgresql.conf ...... 28 2.4 Edit pg hba.conf ...... 28 2.5 Restart Postgres ...... 28 2.6 Create Database ...... 29 2.7 Build website docker image ...... 29 2.8 Create backend virtualenv and install docker ...... 30 2.9 Website Starter Script Options ...... 30 2.10 Start Point Cloud HTTP Server ...... 31 2.11 Build docker-pix4d ...... 31 2.12 Install Supervisor ...... 32 2.13 Configuration file for backend worker ...... 33 2.14 Pipeline Definition ...... 34 2.15 Pix4D Pipeline ...... 34 2.16 Pix4D Class Implementation ...... 35 3.1 Tracker Class ...... 58 3.2 Feature Extraction Code ...... 59 3.3 Convert Frame to HSV ...... 60 3.4 Apply Gaussian Blur and filter yellow and white colors ...... 61 3.5 Canny Edge Detection ...... 61
x Chapter 1
Introduction
In this chapter, the problem statement, the motivation and the proposed approaches are discussed in detail. Several concepts and definitions regarding the work discussed in upcoming chapters is also discussed.
1.1 Problem Statement
The modern decision making process is a data-centric environment. Data enables informed decision making based on facts, statistics etc. Therefore, the processing of data has become an integral part of the modern workspace. Users may leverage several tools and computational machinery for their everyday needs. But, sometimes the computation required maybe intensive requiring more powerful computers. Or it may involve the use of software that is hard to install and maintain. This work describes a centralized platform that runs computational workflows for users on the cloud. The system consists of a frontend user interface where users can interact and schedule projects for the system to process and a backend framework that carries out the actual computational processing. Such a system would have to be able to support multiple users, scale depending on the workload and above all process data
1 1.2. MOTIVATION OF RESEARCH in a manner that is expected by the end user. In this work, it will be shown how such a system can be used to process 3D models from images acquired by flying unmanned aerial vehicles(UAV) over structures using proprietery(Pix4D Server Engine[1]) and open source(OpenDroneMap[2]) software. The development and integration of a custom software package for processing metrics from videos of uninterrupted traffic flow videos using computer vision methods is also shown.
1.2 Motivation of Research
The role of software-powered monitoring and surveillance systems has seen signif- icant growth in recent years. The same is particularly true for the infrastructure and traffic engineering sectors. The advent of high-quality 3D modeling software and Artificial Intelligence systems has seen huge leaps being made. 3D modeling enables personnel to analyze 3D models of objects without being physically present at the location, and deep learning based algorithms have been instrumental in the emergence of high-quality traffic analysis systems leading to a greater understanding of how to improve road travel. However, for a user who wants to use all of these components, there is no one-stop solution. Contemporary systems that undertake similar tasks are more specialized, handling only the 3D modeling or traffic moni- toring sides, never both together. Due to this reason, the user is forced to subscribe to multiple systems. The Common Operating Platform is a centralized web plat- form that is capable of servicing those users. It enables users to handle all of their processing needs on a centralized server system, have their data available 24x7 over the internet and allow easy access and secure collaboration amongst their peers.
1.3 Basic Concepts and Definitions
This section contains all of the basic concepts and definitions used in this work.
2 CHAPTER 1. INTRODUCTION 1.3. BASIC CONCEPTS AND DEFINITIONS
1.3.1 3D Modeling
3D modeling is the process of developing a mathematical representation of a surface or object in three dimensions using specialized software[3]. 3D models come in a variety of formats. Many applications can be used to generate 3D models. Some are proprietary while others are open source. In this section, we will discuss two applications used for 3D modeling called Pix4DEngine[1] and OpenDroneMap[2] as well as a 3D model format called point clouds.
Pix4DEngine Server
Pix4D Server Engine[1] is a set of programming modules from Pix4D[4], which al- lows for the programmatic generation of 3D models using the proprietary Pix4D[4] software. They are highly customizable and built to be integrated into any enter- prise system. Application Programming interfaces allow developers to automate workflows based on business and feature requirements.
OpenDroneMap
OpenDroneMap[2] is an open source drone mapping software that enables users to generate maps, point clouds, 3D models and DEMs from aerial images. They provide a suite of tools including web-based software to process and visualize models in a browser. They also support a variety of ways to integrate their software into existing workflows including running apps natively to using them inside docker[5] containers.
Point Clouds
A point cloud is a set of data points in space representing a 3D dimensional surface or object. They are generally produced by 3D scanners or 3D modeling software.
CHAPTER 1. INTRODUCTION 3 1.3. BASIC CONCEPTS AND DEFINITIONS
Viewed as a whole, they are a representation of the spatial relationship between different data points in the point cloud. Figure 1.1 is an example of a point cloud. The images were captured using a drone flying around the building. It is rendered using Potree[6], a web-based point cloud rendering software. Point clouds are also good for real-time rendering of point clouds over the web as they allow streaming of data point based on the level of information requested.
Figure 1.1: A point cloud of a building
1.3.2 Docker and Containerization
Docker[5] is a program that performs containerization, an operating system level virtualization technology. It enables shipping of software applications as a container including the code, associated tools and operating system of choice. Docker is also operating system agnostic. It can run on many operating systems including Linux, Microsoft Windows and MacOS. This feature allowed for Linux based applications to be run on a Windows based operating system and vice versa. Docker runs these applications in packages called containers. The software that hosts these containers is called Docker Engine[7].
Containers are a standard, lightweight and secure way to run applica- tions[7]. As seen in figure 1.2, Docker engine runs on top of the host operating
4 CHAPTER 1. INTRODUCTION 1.3. BASIC CONCEPTS AND DEFINITIONS
Figure 1.2: Docker Architecture[7]
system. It shares the OS kernel and runs containers as isolated processes[7]. This is what differentiates Docker from a traditional hypervisor based virtualization tech- nology which runs on top of the infrastructure(figure 1.3).
Software developers create ’images’ of their applications. Containers are merely instances of these images. Docker Engine is capable of running multiple con- tainers at the same time. These containers can also be created from the same image. Images have intermediate layers that increase reusability, decrease disk usage, and speed up the build process by allowing each step to be cached. These intermediate layers are not shown by default[8].
CHAPTER 1. INTRODUCTION 5 1.3. BASIC CONCEPTS AND DEFINITIONS
Figure 1.3: Hypervisor Architecture[7]
1.3.3 Deep Learning Frameworks for Computer Vision
Deep learning frameworks are a programming abstraction which provides generic functionality for neural network computation. Programmers use this as a stan- dard way to build and deploy their applications. The traffic monitoring algorithms discussed in this work use deep neural networks extensively for various purposes. There exist many programming frameworks that may be used to implement these algorithms. For this reason, we conducted a study of different frameworks out there. The factors taken into account are the speed of execution, scalability, simplicity of use, understandability and excellent community support. Also, since the backend server was implemented mainly in python, we limited our comparison to python frameworks only. The study included deep learning frameworks like Pytorch, Ten- sorflow, Caffe, Theano, Chainer, CNTK, DSSTNE, Dynet, Gluon, Keras, Mxnet,
6 CHAPTER 1. INTRODUCTION 1.3. BASIC CONCEPTS AND DEFINITIONS
Paddle, BigDL. From the frameworks listed above, we chose to concentrate on Py- torch, Tensorflow, Caffe, Theano, and Keras were the most popular[9].
Pytorch[10] is the python version of the popular Torch deep learning frame- work. Facebook open-sourced the framework in January 2017. Pytorch offers dy- namic computation graphs, which are useful when working with neural networks. It also contains a lot of modular pieces that are easy to combine. However, spotty documentation and lack of commercial support were deterring factors.
The Google Brain team developed Tensorflow for internal use before being released in November 2015. It provides computational graph abstraction that makes it handy when implementing neural networks. Built on top of a C/C++ engine, it allows the use of multiple GPUs for training models and faster compile times than other frameworks. On top of excellent documentation and widespread community support, Tensorflow also provides a visualization tool called Tensorboard that lets the user visualize their algorithms for greater debugging support. [11]
Caffe[12] lacks the community support that Pytorch and Tensorfow enjoy, but it is an intuitive model that allows users to train their models without writing any code. It is also excellent when it comes to feedforward networks and image processing. However, it lacks out of the box support for GPU training and requires the user to write C++/CUDA for GPU layers.
Theano[13] is a python based deep learning framework, one of the oldest and most widely used. However, in September 2017, it was announced that devel- opment of Theano would cease after the 1.0 release due to competing offerings by strong industrial players[14]. Tensorflow is the framework that is meant to replace it.
Keras[15] is a deep learning framework that can work on top of Theano and Tensorflow. It is a fast-growing framework which provides an intuitive API inspired by Torch and works with Theano and Tensorflow.
CHAPTER 1. INTRODUCTION 7 1.3. BASIC CONCEPTS AND DEFINITIONS
cuDNN is not a framework, but a library of deep learning primitives, which requires special mention. It was designed to help researchers implement their pro- totypes quickly and efficiently. It is specifically designed to work well with parallel processing and graphical processing units based on popular deep learning frame- works like caffe and Tensorflow. Integrating said frameworks was shown to increase the performance of the system by as much as 36%[16].
1.3.4 Object Detection
Object detection is a computer technology that involves detecting instances of se- mantic objects from a particular class in images or videos[17]. Over the years many algorithms have been proposed to accomplish this task with increasing orders of efficiency. Object detection algorithms can be broadly classified as being Machine Learning or Deep Learning based. Machine Learning approaches generally have a feature extraction phase followed by a classification phase. Some popular examples are using HAAR[18], SIFT[19] or HOG[20] for extraction and then using a Support Vector Machine[21] for classification. On the other hand, Deep Learning approaches use deep convolutional neural networks to analyze the pixels in a frame to detect objects. They do not require specific feature definitions as used in machine learning approaches.
The Deep learning approach is the latest and best approach for object de- tection. Chen et al. in their paper Automatic Image Cropping: A Computational Complexity Study[22] proposed a model for automated image cropping. The idea is to hone in on the most striking visual in the image based on the proposed low computationally complex algorithms. The possibility of having multiple cropped sections of the image is also discussed at length. The image is divided in a grid-like fashion and the weights assigned to each sector of the grid based on the signifi- cance of the pixel. The image is finally cropped based on a variety of factors like the average weight of the strongest grid, aspect ratio of the image and other fac-
8 CHAPTER 1. INTRODUCTION 1.3. BASIC CONCEPTS AND DEFINITIONS tors. Herranz et al.[23] tried to use CNN for scene recognition while striving to overcome these shortcomings. The authors identified two related problems to tackle to achieve their objective. Also, unlike previous attempts, they incorporate the scale of the pictures into account, whereby increasing gains dramatically. The paper also proposes techniques to combine information from divergent datasets for CNN learning. Experimental results show near human level recognition performance.
Fu[24] et al. proposed the use of contextual information from images to identify boundaries better. It suggests the use of local structural boundary pat- terns and observations from the surrounding areas as a preliminary analysis for determining the edges of objects in the image. In doing so, they have developed a novel approach based on CNNs and conditional random fields(CRFs) whose results significantly outperform existing F-measures[25] on the CMU benchmark.
Bengio et al. explored unsupervised learning approaches[26] to synthesizing representations in data. The representations can entangle and hide more or less the different explanatory factors of variation behind the data. This can be a hindering factor to effective supervised machine learning. The goal here is to detect patterns in the data that might prove invisible to manual domain representation. The paper reviews work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, manifold learning, and deep learning.
The performance of high scale convolutional neural networks can be im- proved by leveraging efficiently added computation[27]. This is achieved by using methods described as factorizing convolutions and aggressive regularization. Ex- periments described in the publication, Rethinking the Inception Architecture for Computer Vision[28], have shown that they outperform existing deep neural nets by decent odds.
CHAPTER 1. INTRODUCTION 9 1.3. BASIC CONCEPTS AND DEFINITIONS
1.3.5 Object Tracking
Object tracking is the process of following an object of interest from frame to frame in a given video. There are four steps to object tracking: detection of the object, tracking the object, maintaining knowledge of the objects that are currently being tracked [29] and de-registering objects as they move out of the scene. This work uses a multi-object tracker to track vehicles on roadways, and this section will cover topics related to multi-object tracking only. The problem of multi-object tracking is a difficult task of computer vision. Object detection only involves detecting an object from a list of classes. However, tracking requires the system also to maintain knowledge of the objects as it moves around a scene. The knowledge referred to in the previous line is a trajectory of the object’s movement.
A simple algorithm for tracking is the centroid tracking algorithm. It uses Euclidean distance to measure the distance between the centroids of detected objects between frames to track them [30]. This system depends on object detection in every step and assumes that the centroids lie close together between subsequent frames. This idea works well when the objects that are being tracked are a continuous stream of vehicles on a highway, but the process can be computationally taxing since we would then need to calculate the Euclidean distance between a large number of vehicles.
A computationally efficient way of solving this problem is to use the Kuhn- Munkres Algorithm[31] with the Intersection Over Union as the scoring factor.
1.3.6 GPU Accelerated Processing
To study the feasibility of enabling GPU accelerated processing in our system, we explore the use of NVIDIA’s CUDA accelerated application programming inter- face. Karpathy et al. in their paper, Large-scale Object Recognition with CUDA-
10 CHAPTER 1. INTRODUCTION 1.3. BASIC CONCEPTS AND DEFINITIONS accelerated Hierarchical Neural Networks[32] detail methods for training neural net- works to recognize objects in images using Nvidia’s CUDA framework. The idea is to train models on GPUs as opposed to CPUs. The massively parallel implementa- tion on a GPU can be 82 times faster than a single core CPU version of the same system. The labelme[33], MNIST[34], NORB datasets were used to conduct various experiments listed in the publication. The preferred medium for programming was C/C++. The authors mention that there are several differences to parallel CUDA programming as compared to standard sequential programs like mapping loops to threads, blocks, etc. and challenges in using various memory segments. The system used by the authors had an Intel Core i7 940 CPU and an NVIDIA GeForce GTX 285 graphics card on a Linux system with 12GB of RAM.
Based on the works of Karpathy et al., Ciresan et al. presented a fast, fully parameterizable GPU implementation of different convolutional neural network variants[35]. Feature extraction is the process of deriving features which are intended to be informative and non-redundant from an initial set of more extensive features. Determining a set of these primary features is called feature selection[16]. The authors do not use traditional feature extractors that are pre-wired but learn features in a supervised way. Also, the results of their experiments yielded low error rates for object classification using NORB, CIFAR10, and MNIST[34] datasets. Deep neural nets with back-propagation were used to carry out these experiments.
1.3.7 Traffic Metrics
The traffic metrics discussed in this section are focussed primarily on uninterrupted traffic flow. The variables defined in this section are all based on the Highway Capacity Manual[36]. Three basic variables and their relationships are discussed. The variables are flow rate, average travel speed, and density, and they can be used to describe traffic on any roadway. According to the Highway Capacity Manual, flow rate is a parameter common to both interrupted and uninterrupted traffic, but
CHAPTER 1. INTRODUCTION 11 1.3. BASIC CONCEPTS AND DEFINITIONS speed and density apply primarily to uninterrupted flow.
Flow Rate
Flow Rate is the equivalent hourly rate at which vehicles pass over a given point or section of a lane or roadway during a given time interval less than 1 hour, usually 15 min. It is a metric that quantifies demand for facilities on a roadway.
Average Travel Speed
Average Travel Speed is a measure of the quality of traffic service provided to a motorist. It is an important tool used in defining the levels of service for many types of facilities. Speed is defined as a rate of motion expressed as distance per unit time (mi/h). According to the Highway Capacity Manual, there are two recommended ways of calculating average travel speed on a highway. They are:
(i) Space Mean Speed: It is the average speed of vehicles calculated based on the average time it takes for them to travel a segment of a road whose distance is known.
(ii) Time Mean Speed: It is the arithmetic average of the speeds of vehicles seen passing a given segment of the roadway.
This work uses Time Mean Speed since the calculation of Space Mean Speed requires prior knowledge of the length of the highway segment under obser- vation.
12 CHAPTER 1. INTRODUCTION 1.4. PROPOSED APPROACH
Density
According to the Highway Capacity Manual, Density is the number of vehicles oc- cupying a given length of a lane or roadway at a particular instant. It is expressed in vehicles per miles. Direct measurement of density in the field is difficult, so the highway capacity manual defines density using the following equation.
D = V/S (1.1)
where V = Flow Rate (veh/hr), S = Average Travel Speed (mi/hr), and D = Density (veh/mi).
Relationships Among Basic Parameters
The theoretical curves between Speed, Flow Rate and Density as defined in the High- way Capacity Manual, helps us distinguish the levels of congestion on a highway. Based on the level of congestion, they are termed as Unsaturated(green), for low levels of congestion, and oversaturated(red) for higher levels of congestion. The fol- lowing figure represents the theoretical relationships between the various parameters discussed above.
1.4 Proposed Approach
The purpose of this work is to have a single web-based platform to facilitate any of the users processing needs. There are also other motivations for wanting to do this, such as the concentrating the computational overhead required to process resource- intensive operations on a central server eco-system and alleviating the need for an uneducated audience to familiarize themselves with the intricacies of confusing
CHAPTER 1. INTRODUCTION 13 1.4. PROPOSED APPROACH
Figure 1.4: Relationships among Speed, Density and Flow Rate on Uninterrupted Flow Facilities [36] [37] software packages. The proposed approach is to have a workflow management system that can process pre-defined templates. A template is a project that can be processed using a processing engine given valid data. For example, using images of a bridge shot from an unmanned aerial vehicle(UAV) to create a 3D model. The user can create projects using a frontend interface that is accessible over a network through a web browser. Any data associated with the project that is required for processing can be uploaded through the user interface. A scalable backend system that runs on the cloud then processes the data using the processing engine associated with the project. This work demonstrates the use of such a system to process 3D models and traffic monitoring algorithms. The frontend interface was created using HTML, CSS and Javascript with the use of Twitter Bootstrap[38]. The website was created using the Django framework and the backend processing engines were created using Python 3. Docker[5] is used extensively to run different software including the website and the backend processing engines. The processing engines used by the platform are Pix4D Server Engine[1], OpenDroneMap[2] and the custom traffic monitoring software explained in chapter 3.
14 CHAPTER 1. INTRODUCTION Chapter 2
Common Operating Platform
The advancement in Unmanned Aerial Vehicle (UAV) technology over the past years has created many opportunities and possible applications in the transportation industry. UAVs are being used in monitoring and evaluation of construction sites, important construction events, documenting traffic patterns, etc. In this section, a platform to collect and process data from different unmanned aerial systems will be described. The platform allows the user to archive data, share data amongst off- site and on-site personnel, and transform raw data to information, e.g. transform aerial images to 3D models. Using a web-based graphical user interface, users can review the processed data; interact with 3D point cloud rendering system through a computer or an augmented reality goggles. More applications of this system are discussed in [39].
2.1 Distributed System Architecture
The architecture of the common operating platform is based on the microservices software development technique, where the system would consist of lossely coupled services that work together to accomplish a larger task. Each microservice would
15 2.1. DISTRIBUTED SYSTEM ARCHITECTURE essentially specialize in the performance of a small task. This technique is different from the traditional monolithic architecture[40], where the entire system is built as one gargantuan application. The microservices software development technique has a number of advantages[41]. They are:-
1. Fault isolation: Any problems in the system can be isolated to individual modules
2. Eliminate technology lock-in: Microservices provide the flexibility to try differ- ent technology stacks for individual modules. This creates a vibrant software eco-system that consists of applications built with several technologies. This allows the developer to truly use tools that fits the requirement. For exam- ple, in a monolithic architecture written in python, developers will be forced to use tools that are python compatible for all tasks. However, if there is a task that is better solved using Java, they would not be able to do so. In a microservices architecture, this is not a problem. The main program of choice could be python, but if a module needs to be written in Java, it can be easily accomplished.
3. Easy to understand: it is easier to understand because the system is broken into smaller and more managable pieces.
4. Cost effective scalability of individual components: It is possible to cost ef- fectively increase or decrease the scale of resources being used for individual components. This level of configuration is not available in a monolothic frame- work.
With that being said, we will now break down the common operating plat- form into smaller components. From a high level, the common operating platform consists of mainly two components. These components will be further broken down into smaller components in the upcoming sections, but for defining certain termi- nology, we will continue with this definition for the time being.
16 CHAPTER 2. COMMON OPERATING PLATFORM 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
1. A web interface which enables users to upload data to the system. This is the part of the platform that the user interacts with.
2. A backend processing system responsible for processing uploaded data. This layer is not visible to the user.
The website was created using a Django backend with a Postgres database and runs Twitter bootstrap on the frontend. The files uploaded to the server would be stored on the file system with the database keeping tabs on the file location and name along with its associated project. Database design is discussed in more detail in 2.1.2. The backend processing system was implemented using a combination of Python, and Bash scripts. The design is intended to scale for a multi-server architec- ture with the various components intended to work independently as when required. This level of orchestration will support the distribution of the workload among mul- tiple servers and can also be used to save on computation cost as servers can be added or replaced based on demand. The common operating platform also facili- tates API programming. The platform supports HTTP based REST APIs have been used in tandem with the Microsoft HoloLens for visualizing 3D models(processed by the backend). The following sections delve deeper into the design and architecture of the system.
2.1.1 The Web Interface
There are two kinds of web interfaces available. The first one a graphical user interface (GUI)which is accessible using a web browser and the second one is an application programming interface(API)to be used by the devices or programs. The website interface was written in HTML, CSS, and javascript using Twitter’s boot- strap frontend interface library. J-Query and Ajax were used for implementing the functionality.
CHAPTER 2. COMMON OPERATING PLATFORM 17 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
Figure 2.1: System Architecture Diagram
Graphical User Interface
The standard graphical user interface (GUI) has a username/password authentica- tion available on both interfaces. Users may register for an account using a sign-up page. The GUI has several different views which accomplish specific functionality. One such view is the virtual file system interface. It is akin to a PC’s desktop-like view of the different files and folders on a computer. The virtual file system is de- signed after a Unix based file system[42] to emulate operations to store data in an orderly fashion online. The user has options to create files and folders as well as features to handle them like delete. For 3D model processing projects, users are required to create special project directories and upload their images into it. The project may be of two types based on the 3D model processing engine the user wants. They are Pix4DEngine Server[1] and OpenDroneMap[2]. The prior application is proprietary, and the latter is open sourced. Once the files have been uploaded, 3D model processing may be initiated by the user using the options provided on the website.
Once the processing is initiated, the images are processed in the backend
18 CHAPTER 2. COMMON OPERATING PLATFORM 2.1. DISTRIBUTED SYSTEM ARCHITECTURE server using the chosen processing engine. The status of the processing can be seen in the backend jobs tab. This view consists of a history of all the jobs that have been processed by the system. All projects that the system processes will show up here along with its status. The view contains links to potree[6], which is an open source online 3D point cloud viewer where the user can view the finished 3D model once it has been processed. There are also links to the project directory which houses the images for the 3D model processing job.
Apart from these two views, there is a settings view that provides two features. The first is the change password feature. The user is required to type his/her old password, followed by the new password to reset it. The second feature is the API key manager. An API key is a universally unique identifier(UUID) which must be used in the HTTP header of an API request to authenticate access to the backend server. The user may generate API keys for their automated programs or any devices which are required to query the backend server for information. Apart from creating keys, the option to disable API keys is also available.
Application Programming Interface
The common operating platform also provides representational state transfer (REST)[43] application programming interfaces (API) for facilitating interoperability between different computer systems over a network. Responses to API queries are exclu- sively in javascript object notation (JSON)form and uses HTTP methods GET and POST. The API requests are authenticated using a universally unique identifier (UUID) API key. API keys are device specific and may be generated by the user using the options in the settings view. The API key should be attached to the header(X-API-Key)of the HTTP request. The different APIs are:-
1. Uploads API: Upload file to virtual file system
• URL: /api/v1/upload
CHAPTER 2. COMMON OPERATING PLATFORM 19 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
• Request type: POST
• Headers: X-API-KEY
• Request arguments:
∗ docfile: Path to the file
∗ dir name: Virtual file system directory where the file is to be up- loaded
∗ owner: Username of the owner of the file
2. List directories API: List the directories in a specified virtual file system di- rectory
• URL: /api/v1/list/dir/
• Request Type: GET
• Headers: X-API-KEY
• Request arguments:
∗ path: The directory path which the API should list
3. List files API: List the files in a virtual file system directory
• URL: /api/v1/list/files/
• Request type: GET
• Headers: X-API-KEY
• Request arguments:
∗ path: The directory path which the API should list
4. Download file API: Download file
• URL: /api/v1/download/
• Request type: GET
• Headers: X-API-KEY
20 CHAPTER 2. COMMON OPERATING PLATFORM 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
• Request arguments:
∗ file: Full path of the file to download
It is also worthy of mentioning the lbstatus view, which is a kind of API. The lbstatus view is an indicator of the system’s readiness. The load balancer primarily uses it. It accepts an optional argument, status, which can be used to enable or disable lbstatus. The lbstatus of the system can have two values, ’OK’ and ’NOT OK’, with the former indicating that the system is ready to receive requests and the latter indicating the opposite. The system administrator can use this feature as a system health check or when an instance needs to be taken out of rotation.
2.1.2 Database Design
The database for the common operating platform consists of multiple tables to support various operations. The entity relationship diagram is shown in figure 2.2.
Figure 2.2: Entity Relationship Diagram
UploadedFiles
Contains information related to the files uploaded by the user and where it is stored in the file system. The table definition is as shown in table 2.1.
CHAPTER 2. COMMON OPERATING PLATFORM 21 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
Table 2.1: Defintion of Table: UploadedFiles
Entity Type Description file name char(300) Name of the uploaded file file system path char(300) Path to the file in the host server’s file sys- tem vfs path Foreign Key: DirectoryStructure The virtual file system path of the uploaded file owner Foreign Key: User Username of the owner of the file created on timestamp with time zone Timestamp of cre- ation
Table 2.2: Defintion of Table: DirectoryStructure
Entity Type Description dir name char(300) Name of the virtual directory sub directory name char(300) Path to the subdirec- tory owner Foreign Key: User Username of the owner of the file created on timestamp with time zone Timestamp of cre- ation
DirectoryStructure
Contains information related to the virtual file system directory structure. The sub directory name entity is path of the subdirectory. For example, if the dir name is ”/sub dir name/dir name”, then the sub dir name is ”/sub dir name”. Definition is shown in table 2.2.
APIAuth
Contains information related to any API keys created by the user. Definition is shown in table 2.3.
22 CHAPTER 2. COMMON OPERATING PLATFORM 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
Table 2.3: Defintion of Table: APIAuth
Entity Type Description api key char(300) API Key name char(300) Name of the device the API Key belongs to owner Foreign Key: User Username of the owner of the file created on timestamp with time zone Timestamp of cre- ation
Table 2.4: Defintion of Table: TaskBucket
Entity Type Description id integer Primary Key task name char(300) Name of the project type char(100) Type of project. eg: Pix4D, ODM etc. status char(100) Status of project vfs path Foreign Key: DirectoryStructure The virtual file system path of the uploaded file created on timestamp with time zone Timestamp of cre- ation
TaskBucket
Contains information related to projects created by the user. This table is updated when a project is created/modified/deleted. The backend systems access this ta- ble through the application programming interface detailed in section 2.1.1. The definition is shown in table 2.4.
PostProcessedData
Contains information related to the data processed by the projects and their cor- responding location in the file system. After a project has successfully finished processing, the output data is compressed into a zip file and stored on the file sys- tem. The information pertaining to the output data and the project information is stored in this table. The definition is shown in table 2.5.
CHAPTER 2. COMMON OPERATING PLATFORM 23 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
Table 2.5: Defintion of Table: PostProcessedData
Entity Type Description name char(300) Name of the output data file task bucket Foreign Key: TaskBucket Foreign Key to the corresponding task bucket
2.1.3 Backend Systems
The backend server is responsible for serving the website, the APIs and the pro- cessing of the aforementioned 3D models. For the processing of 3D models, we use three software, Pix4D Server Engine[1], OpenDroneMap[2] and a custom software that measures traffic monitoring metrics. The need for the inclusion of these ap- plications in our system will be computationally intensive. It also has to scale to service the needs of many users. The system will need to process multiple 3D mod- els concurrently given limited server capability. So, the objective is to distribute several moving parts amongst multiple servers to spread the load. With this in mind, the backend system was designed to contain multiple blocks which can run independently of each other. Together they create a workflow of numerous inde- pendent systems that work together towards a common objective. So, the following is a high-level overview of the processing workflow starting when a user uploads images/videos for processing, and it gets processed by the backend system.
1. User creates 3D model processing project on website
(i) Creates project directory on the virtual file system
(ii) Uploads images/videos associated with the project
2. User initiates project processing when he/she is ready
3. Django processes the request
(i) Collect data about the processing task from the file system and the database
24 CHAPTER 2. COMMON OPERATING PLATFORM 2.1. DISTRIBUTED SYSTEM ARCHITECTURE
(ii) Queues a processing job
(iii) Queue distributes job to backend controller in a round robin fashion. (There may be more than one backend controller based on demand)
4. Backend controller receives job info and starts processing
(i) Fetches required data from file storage and database
(ii) Initiates processing job with docker containers
(iii) Update database with the status of the processing
5. User is notified on the website when the project has completed processing or if processing encountered an error
We chose Django as the web framework for the site. Django is a free and open-source web framework, written in Python, which follows the model-view- template (MVT) architectural pattern[44]. It enables us to build a high-performance website quickly, while not having to worry about implementing the standard oper- ations required of a modern site. Apart from serving the site, Django also handles setting up 3D model processing tasks in the backend. When the user initiates the processing of a 3D model through the site, Django will collect data about the job and add an entry in a queue with that information. This entry is then distributed to a backend controller which will start processing the project workflow using docker containers.
The backend controller will have several worker processes that will be on standby to receive 3D model processing jobs. Each worker is designed to handle one processing request at a time. While processing a request, it will intimate the queue that is busy processing and is not available to service any requests at this time. It will, however, send a ready to consume request to the queue once it has completed the task at hand. With this design, the system administrator or an automated service can spawn multiple instances of the worker process to service multiple processing requests concurrently.
CHAPTER 2. COMMON OPERATING PLATFORM 25 2.2. INSTALLATION
2.2 Installation
2.2.1 System Requirements
Hardware
1. CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz (12 cores or higher)
2. Memory (RAM): 64GB
3. Disk: 10 TB
Software
This software needs to be installed system-wide.
1. Operating System: RedHat Enterprise Linux 7.6
2. python v2.7.5
3. python-pip2 compatible with python v2.7.5
4. python v3.6.5 or higher
5. python-pip3 v10.0.1or higher
6. Docker v18.x or higher
7. Postgres v10.xor higher(Additional configuration required, see upcoming sec- tions for more information)
8. Supervisord(Additional configuration required, see upcoming sections for more information)
9. python-virtualenv
26 CHAPTER 2. COMMON OPERATING PLATFORM 2.2. INSTALLATION
10. docker python SDK(Additional configuration required, see upcoming sections for more information)
Docker Images
1. opendronemap/opendronemap:v0.3.1 (pull from docker-hub)
2. connormanning/entwine (pull from docker-hub)
3. connormanning/http-server (pull from docker-hub)
4. docker-pix4d(build from source, see upcoming sections for more information)
5. common operating platform uas-website (build from source, see upcoming sec- tions for more information)
Most of these software dependencies can be installed off the shelf as is. Documentation on those processes may be found online.
2.2.2 Special Configurations
Postgres Database
The Postgres database is the first component that needs to be set up during the installation process. It needs to be installed system-wide and requires some configu- ration to make it accessible for the website. The Postgres database can be installed as follows.
# Install the repository rpm yum install https://download. postgresql .org/pub/repos/yum/11/redhat/rhel −7−x86 64 /pgdg−redhat11 −11−2.noarch .rpm # Install the client packages yum install postgresql11 # Install the server packages yum install postgresql11 −s e r v e r # Initialize the database and enable automatic start / u s r / pgsql −11/bin/postgresql −11−setup initdb
CHAPTER 2. COMMON OPERATING PLATFORM 27 2.2. INSTALLATION
systemctl enable postgresql −11 systemctlstart postgresql −11
Listing 2.1: Postgres Installation
Make sure that Postgres is up and running by using the following command. The presence of a host of processes as can be seen below indicates that Postgres is up and running.
$ ps −e f | grep postgres postgres 1147 1 0 13:22 ? 00:00:00 /usr/bin/postgres −D /var/lib/pgsql/data −p 5432 postgres 1152 1147 0 13:22 ? 00:00:00 postgres: logger process postgres 1154 1147 0 13:22 ? 00:00:00 postgres: checkpointer process postgres 1155 1147 0 13:22 ? 00:00:00 postgres: writer process postgres 1156 1147 0 13:22 ? 00:00:00 postgres: wal writer process postgres 1157 1147 0 13:22 ? 00:00:00 postgres: autovacuum launcher process postgres 1158 1147 0 13:22 ? 00:00:00 postgres: stats collector process ninja 24045 10943 0 14:20 pts/3 00:00:00 grep −−color=auto postgres
Listing 2.2: Postgres Process Status
Now, we need to make some configuration changes.The first edit is to make the following change to /var/lib/pgsql/data/postgresql.conf
# Edit /var/lib/pgsql/data/postgresql.conf # Change listen addresses = ’localhost ’ to listen addresses =’∗ ’ l i s t e n addresses =’∗ ’
Listing 2.3: Edit postgresql.conf
Once that is done,make the following changes to /var/lib/pgsql/data/pg hba.conf. Edit the entries for IPv4 connections on localhost and network to use password by changing the entry to md5.
# IPv4 local connections: host all all 127.0.0.1/32 md5 host all all 0.0.0.0/0 md5
Listing 2.4: Edit pg hba.conf
The next step is to restart Postgres and check if it up and running with the method described above.
systemctl restart postgresql −11
Listing 2.5: Restart Postgres
28 CHAPTER 2. COMMON OPERATING PLATFORM 2.2. INSTALLATION
If your database server is up and running, then you are ready to proceed. If it is not, then one of the above steps was improperly completed, and that requires further debugging. The final step is to create a new database user and create the necessary database and privileges to run the common operating platform.
# Change user to postgres sudo su − p o s t g r e s # Login to psql database using the client p s q l # Create database CREATE DATABASE
Listing 2.6: Create Database
Build Website
Once the database is set up, the next step is to set up the site. The site is run inside a and is isolated from the host RHEL server apart from the database. For the time being, we will build the site from source on the RHEL server. This step assumes that docker is already installed and running. Now, go to the folder where you cloned a copy of the source code for the common operating platform website.
cd
Listing 2.7: Build website docker image
Setting up the Backend
Now that we have a working database and the website docker container is built, it is time to set up the backend controller. The backend controller can run on
CHAPTER 2. COMMON OPERATING PLATFORM 29 2.2. INSTALLATION
a Linux operating system with the necessary dependencies installed. Additional storage mounted at /data/ is recommended for storing uploads and other processed data. Clone the GitHub directory on the target machine. The use of a virtualenv is highly recommended but not mandatory. If using a virtualenv, create it(named ”venv”) inside the newly cloned repository. The following command may be used to do this.
v i r t u a l e n v −p python3
Listing 2.8: Create backend virtualenv and install docker
The website can now be started up using the start website.py script as follows.
usage: start website.py [−h ] [−m MOUNT] [− i IMAGE] [−v VERSION ] [− e ENVIRONMENT] [−d PSQL DB ] [−U PSQL USER ] [−P PSQL PASSWD] [−S PSQL HOST ] [−q PSQL PORT ] [−wh HOSTNAME] [−wp WEBSITE PORT] [−g GREYHOUND SERVER] [−gp GREYHOUND PORT]
Script to start UAS Website
optional arguments: −h , −−help show this help message and exit −m MOUNT, −−mount MOUNT Docker volume mount point −i IMAGE, −−image IMAGE Name of docker image −v VERSION, −−v e r s i o n VERSION Version of docker image −e ENVIRONMENT, −−environment ENVIRONMENT Environment of docker container −d PSQL DB , −−p s q l d b PSQL DB Database Name −U PSQL USER , −−p s q l u s e r PSQL USER Database user −P PSQL PASSWD, −−p s q l p a s s w d PSQL PASSWD Database password −S PSQL HOST, −−p s q l h o s t PSQL HOST Database hostname −q PSQL PORT, −−p s q l p o r t PSQL PORT Database port number −wh HOSTNAME, −−hostname HOSTNAME Hostname of website −wp WEBSITE PORT, −−w e b s i t e p o r t WEBSITE PORT
30 CHAPTER 2. COMMON OPERATING PLATFORM 2.2. INSTALLATION
Port number of website −g GREYHOUND SERVER, −−g r e y h o u n d s e r v e r GREYHOUND SERVER Hostname of greyhound server −gp GREYHOUND PORT, −−g r e y h o u n d p o r t GREYHOUND PORT Port number of greyhound server
Listing 2.9: Website Starter Script Options
The http-server which serves point clouds to potree can also be run using the help of the starter script “start greyhound.py”.See if the container is up and running using the “docker ps” command. python start greyhound .py
Listing 2.10: Start Point Cloud HTTP Server
If both instances are up and running, then open up a browser(Google Chrome or Mozilla Firefox is recommended) and try to access the website by typing in the IP Address or the DNS name of the server. You should be able to see the home page now.
docker-pix4d
The docker-pix4d container is responsible for processing pix4d projects on the com- mon operating platform. It needs to be built from source as follows. The first step is to clone a copy of the code to the server. Then, create a folder called “bin” inside and copy the “pix4dengine-0.1.0-py3-none-any.whl” and ”pix4dmapper 4.3.31 amd64.deb” files within. Version updates will require changes to the Dockerfile. Then build it as shown below. docker build −t docker−pix4d .
Listing 2.11: Build docker-pix4d
CHAPTER 2. COMMON OPERATING PLATFORM 31 2.2. INSTALLATION
Supervisord
Once we have the copy of the backend controller installed on the server, it is now time to configure workers to run it periodically. A worker is an instance of the backend controller. The system is capable of running multiple workers based on the availability of computational resources. The first step is to pull necessary docker containers onto the host system. Though this step is not necessary, it is considered a good practice to do this anyway. The backend uses the following containers:
(i) docker-pix4d
(ii) opendronemap/opendronemap:v0.3.1
(iii) connormanning/entwine
(iv) connormanning/http-server
The first one was built in the previous step, but the remaining three can be pulled from docker hub using the “docker pull” command. The supervisor python package will be used to run instances of the backend controller, which processes workflows on the common operating platform. It is installed system-wide or inside a virtualenv using python-pip2. pip2 install supervisor
Listing 2.12: Install Supervisor
The next step is to create an api key worker library. It is used to au- thenticate requests to the website API to gain access to the underlying pending task information. To this, log in to the website. Goto Settings, and create a new API key. Copy the value of API-key and paste it in the settings.py file under ”api key”.Also, make sure the entries for hostname and the username and password for Pix4D are also correct.
32 CHAPTER 2. COMMON OPERATING PLATFORM 2.3. TEMPLATIZING WORKFLOWS
Now, it is time to run the worker instance. All worker instances are run under supervisor. Install supervisor and create a configuration file at /etc/ supervisor/conf.d/worker#.conf, where # is the id of the worker labeled se- quentially. A sample configuration file is shown below.
[program: worker1 ] command =
Listing 2.13: Configuration file for backend worker
To run multiple instances of the worker, create additional configuration files as necessary. The platform is now set up and ready to use.
2.3 Templatizing Workflows
A workflow is defined as the flow of work done by an entity. The Common Oper- ating Platform supports the processing of pre-defined workflows. It is intended to be a system that is capable of automating any computation. In order to perform such operations, templates of such workflows would need to be created. This work demonstrates how the common operating platform is used to process 3D models using proprietary and open source software. It also demonstrates how custom soft- ware(traffic monitoring) can be used to create templates. A template can consist of multiple steps. For example, in the case of 3D modeling, the first step is to create the 3D model itself. Then, the models need to be converted into a format that is suitable for streaming over HTTP for viewing it online on the Potree[6] point cloud visualizer.
To this end, the common operating platform backend framework allows the creation of custom template pipelines. Each step in a template may be defined
CHAPTER 2. COMMON OPERATING PLATFORM 33 2.3. TEMPLATIZING WORKFLOWS programmatically using the Python programming language. These steps would be executed in sequence to process the workflow. The backend framework will handle any error handling, logging and reporting activities. Listing 2.14 shows the definition of a pipeline class. The ’run’ function will process the contents of the pipeline sequentially. c l a s s PipelineProcessor: i n s t a n c e s = list()
d e f i n i t (self , pipeline): self.pipeline = pipeline self.context = dict() self.status = True
d e f set context(self , context): self .context = context
d e f run(self): f o r node in self.pipeline: i f self.status: p i p e l i n e key , pipeline value = node(context=self .context) self.status = pipeline v a l u e [’status’] self .context[pipeline key] = pipeline v a l u e PipelineProcessor . instances .append( self )
Listing 2.14: Pipeline Definition
A context is a set of variables that are used to relay information between two adjoining steps in a pipeline. For example, after Pix4D has finished processing a 3D model, it will relay the information about where the models are stored to the next step using a context variable. Listing 2.15 shows an example of a Pix4D template being processed using the backend pipeline. pipeline = [Pix4d(username=settings .username , password=settings .password , task=task) , Entwine(task=task , entwine directory=settings . entwine directory) , ZipFiles(task=task , target directory=settings . processed d a t a directory) , ProcessedData(task=task , destination=settings . container m e d i a d i r e c t o r y , p o s t p r o c e s s url=settings . post p r o c e s s u r l ) , Greyhound( entwine directory=settings . entwine directory , std logger=task. std l o g g e r ) ] p i p e l i n e processor = PipelineProcessor(pipeline=pipeline) p i p e l i n e processor.set context(context=context) p i p e l i n e processor .run()
Listing 2.15: Pix4D Pipeline
The various steps from listing 2.15 are:
34 CHAPTER 2. COMMON OPERATING PLATFORM 2.3. TEMPLATIZING WORKFLOWS
1. Pix4D: Process the 3D model using the uploaded input images
2. Entwine: Use the point cloud file generated by Pix4D to create point cloud repository for streaming to Potree
3. ZipFiles: Create a zip file of the processed project
4. ProcessedData: Create an entry in the database that the project has finished processing and upload the zip file to the website
5. Greyhound: Restart the Greyhound server which will now stream the pro- cessed point cloud to Potree
Each step in the pipeline is a class, and it has to adhere to a defined framework. The rules of the framework are as follows,
1. Every step must have a class-based implementation
2. The class must implement a ’ call ’ method. The contents of this method will be the logic behind the processing in that step.
3. The ’ call ’ method must return a python tuple of length 2. The first item in the tuple is the name of the step(type: string) and the second item is a dict of context variables for the next step. A mandatory variable in the context is ’status’ which needs to be set to ’True’ or ’False’ based on the state of the processing in this step.
Listing 2.16 shows an example of a Pix4D class implementation. This step uses Docker[5] software development kit(SDK)[45] to run an instance of Pix4D to process a 3D model. import docker c l a s s Pix4d(PipelineMaster): d e f i n i t (self , username, password, task): s u p e r(Pix4d, self). i n i t () self .image =’docker −pix4d’
CHAPTER 2. COMMON OPERATING PLATFORM 35 2.4. IMPLEMENTATION ISSUES
self .username = username self .password = password s e l f . t a s k id = task.task i d s e l f . task name = task.task name s e l f . t a s k path = task.task p a t h self.client = docker.from env ( ) self .container d i r =’/data/pix4d’ s e l f . s t d logger = task.std l o g g e r s e l f . t a s k bucket = task.task b u c k e t self .docker logger = task.docker l o g g e r
d e f c a l l ( s e l f , ∗ args , ∗∗ kwargs ) : s e l f . s t d logger . log(statement=’Starting PIX4D project. Container name={} ’. format( c o n t a i n e r n a m e ) ) self . client .containers.run(image=self .image, name=container name , volumes=
r e t u r n’pix4d’, { ’status’: True, ’input d i r e c t o r y’:
Listing 2.16: Pix4D Class Implementation
2.4 Implementation Issues
2.4.1 Changing IT Eco-Systems
This project was developed for use by the Ohio Department of Transportation’s(ODOT) Unmanned Aerial Systems Center(UASC) with the intention of handing over the platform to ODOT’s IT department to manage. During the design phase of the common operating platform, ODOT IT stipulated that the only operating system environment that they supported was Microsoft Windows. And as a result, the plat- form itself would have to be compatible with a windows system environment. Now, some of the components being used in the common operating platform required a Linux operating system. In order to facilitate this requirement, Docker was used. Docker[5] is a program that performs containerization, an operating system level virtualization technology. It enables shipping of software applications as a container including the code, associated tools and operating system of choice. Docker is also operating system agnostic. It can run on many operating systems including Linux,
36 CHAPTER 2. COMMON OPERATING PLATFORM 2.4. IMPLEMENTATION ISSUES
Microsoft Windows and MacOS. This feature allowed for Linux based applications to be run on a Windows based operating system. And so, even though the common operating platform was developed on an Ubuntu 16.04 LTS operating system, its use of Docker and Python(for the backend) enabled it to run on Windows Server 2016 operating systems. For running the worker nodes of the backend on a windows operating system, windows task scheduler or VisualCron[46] may be used instead of supervisor as mentioned in 2.2.1. supervisor is not available on a windows oper- ating system. The system was also tested in-house on a compatible windows 2016 platform. However, ODOT IT policy did not allow for enabling HyperV technology in the BIOS on its virtual servers which was a pre-requisite for installing Docker on the machine. This policy meant that the platform would not be able to run on a windows operating system hosted on ODOT IT networks. As a result, it was de- cided to create a new RedHat Enterprise Linux based operating system for hosting the common operating platform. Unlike windows, linux operating systems do not require BIOS based configuration to install Docker.
2.4.2 Maximum file upload size
For the proof of concept, the common operating platform was deployed on a single server setup with specifications shown in table 2.6. The frontend interface of the website can be used to upload data to the server for processing. However, tradi- tional file uploads using HTTP POST forms have limitations when it comes to large uploads[47]. This requires data to be chunked into smaller parts and uploaded sep- arately [48]. In order to scale in such a manner, a dedicated apparatus for handling file storage is required and is out of the scope of this work. It may be addressed in future works(see section 4.1). The current version of the common operating platform caps the maximum file upload size to 4GB.
CHAPTER 2. COMMON OPERATING PLATFORM 37 2.5. RESULTS
Table 2.6: Specifications of Production Server
CPU(s) 16 Memory 125G Operating System RedHat Enterprise Linux 7
2.4.3 Scalability
Modern software architectures need to be scalable based on the amount of incoming traffic. The system will need to be fluid and service requests at a fast pace. In order to do so, each component will have to be capable of working on its own in accordance with the black box approach[49]. According to the black box approach, a system is viewed in terms of its inputs and outputs without any knowledge of its internal workings. Doing so, it lets the developer concentrate on scalability by designing a system that will act as a block in a bigger picture. And all of these components work together to complete a particular task.
2.5 Results
The common operating platform supports processing 3D models using Pix4D[1] and OpenDroneMap[2] as well as visualizing them on the website. It can also process videos for traffic monitoring. Users can also process point cloud files(.las and .laz) offline and upload it to the common operating platform for visualization using the onboard Potree[6]viewer. Google Chrome and Mozilla Firefox are the recommended browsers. The platform has been deployed on a production server and is currently in use by the Unmanned Aerial Systems Center of the Ohio Department of Trans- portation(ODOT).
38 CHAPTER 2. COMMON OPERATING PLATFORM 2.5. RESULTS
2.5.1 Register and Login
To register for a user account, simply navigate to http://
Figure 2.3: Register a user account
To login, navigate to http://
Figure 2.4: Login Page
2.5.2 Navigating the Dashboard
Once the user is logged in, navigating the dashboard is easy to use with the overhead taskbar. Click on the available buttons to perform the desired action. Here is a brief
CHAPTER 2. COMMON OPERATING PLATFORM 39 2.5. RESULTS description of the available actions.
(i) Files: Shows the page where all the uploaded files are stored
(ii) Backend Jobs: Displays the page where every 3D modeling projects created on the platform along with their status and associated files
(iii) Settings: Displays the view that has options to change the password and create/delete API keys
(iv) Logout: Logs the user out
Figure 2.5: Navigation
2.5.3 Processing 3D models
3D models can be processed on the Common Operating Platform using Pix4D or OpenDroneMap using the following steps. Users would need to be logged in to do this.
(i) Create a Pix4D/ODM Project: Use the “Create PIX4D Project” or “Create ODM Project” buttons on the lower right side to create your project folders.
(ii) Upload image files into the project folder: After opening your newly created project folder, use the “Upload Files” button on the lower right side to upload your images.
(iii) Start processing: When you are ready, click on the green “Process Project” button on the right side to initiate project processing in the backend server.
(iv) Check the progress of jobs on the “Backend Jobs” tab: This tab has a table of all past and present projects that the system has processed.
40 CHAPTER 2. COMMON OPERATING PLATFORM 2.5. RESULTS
Figure 2.6: Create Pix4D/ODM Project
Figure 2.7: Upload Image Files
Figure 2.8: Process Project
CHAPTER 2. COMMON OPERATING PLATFORM 41 2.5. RESULTS
Figure 2.9: Backend Jobs
The status of your project toggles from ‘created’ → ‘pending’ → ‘queued’ → ‘processing’ → ‘done’ or ‘error’,and it keeps getting updated by the backend processing system.
2.5.4 Visualize Point Clouds
The Common Operating Platform supports the processing of point cloud files (.las and .laz). The user has to upload it using the website using the following steps:
(i) Create a Visualization Project: Use the “Visualize 3D model” button on the lower right side.
Figure 2.10: Create a visualization project
(ii) Upload point cloud file(s) into the project folder: After opening the newly created project folder, use the “Upload Files” button on the lower right side to upload your point cloud file(s). Please note that multiple point cloud files
42 CHAPTER 2. COMMON OPERATING PLATFORM 2.5. RESULTS
associated with different image datasets can be uploaded for visualization. Each point cloud file will be processed independently.
Figure 2.11: Upload Point Cloud Files
(iii) Start Processing: When you are ready, click on the green “Process Project” button on the right side to initiate project processing in the backend server. The backend will create multiple entries to each point cloud file present in the project folder in the backend jobs page. Each point cloud file is processed independently.
Figure 2.12: Process Project
(iv) Check the progress of jobs on the “Backend Jobs” tab: This tab has a table of all past and present projects that the system has processed.
CHAPTER 2. COMMON OPERATING PLATFORM 43 2.5. RESULTS
Figure 2.13: Backend Jobs
The status of your project toggles from ‘created’ → ‘pending’ → ‘queued’ → ‘processing’ → ‘done’ or ‘error’,and it keeps getting updated by the backend processing system.
2.5.5 Process Traffic Monitoring Projects
Users can upload traffic monitoring videos for processing on the common operating platform. The videos will be processed using the methods discussed in chapter 3. However, the website does not support a graphical interface for tracing lane lines(see section 3.4.2), so that feature will be unavailable.
(i) Create a Traffic Monitoring Project: Use the “Traffic Monitoring” button on the lower right side to create your project folders.
Figure 2.14: Create Traffic Monitoring Project
44 CHAPTER 2. COMMON OPERATING PLATFORM 2.5. RESULTS
(ii) Upload image files into the project folder: After opening your newly created project folder, use the “Upload Files” button on the lower right side to upload your images.
Figure 2.15: Upload Video Files for Traffic Monitoring Project
(iii) Start processing: When you are ready, click on the green “Process Project” button on the right side to initiate project processing in the backend server.
Figure 2.16: Process Traffic Monitoring Project
CHAPTER 2. COMMON OPERATING PLATFORM 45 2.5. RESULTS
(iv) Check the progress of jobs on the “Backend Jobs” tab: This tab has a table of all past and present projects that the system has processed.
Figure 2.17: Status of jobs in Backend Jobs
The output of this section will be a zip file with all of the processed in- formation in it. It is available to download in the ”Processed files” column on the table located in the ”Backend Jobs” page.
2.5.6 Change Password
Users can change their passwords in the overhead settings tab. However, if the user forgot their previous password or is not able to login to the platform, they are to contact the system administrator. Please note that this view is only available if you are signed in.
2.5.7 Create API Keys
Users can create API keys for their devices in the settings tab. Please note that the api keys cannot be edited once they have been generated, but they can be removed. The keys are 128 bit randomly generated universally unique identifiers used for authenticating device api requests to the server. More information on how to use APIs can be found in the API documentation.
46 CHAPTER 2. COMMON OPERATING PLATFORM 2.5. RESULTS
Figure 2.18: Change Password
Figure 2.19: Create / Delete API Keys
2.5.8 Benchmarks
In order to benchmark the capabilities of the common operating platform, tests were conducted tests on two different servers running the exact copies of the software. One of them was a production class Microsoft Azure server provided by the sponsors and the other was a powerful desktop Alienware PC. The specifications of the servers are shown in table 2.7.
Two image datasets were compiled for the tests. The first one titled ’small project’ consisted of 46 images and the second one titled ’medium project’ consisted of 199 images. They were processed using Pix4D and OpenDroneMap on both
CHAPTER 2. COMMON OPERATING PLATFORM 47 2.5. RESULTS
Table 2.7: Specifications of Production and Test Servers
ODOT RHEL Server Alienware Server CPU(s) 16 12 Memory 125G 16G Operating System RedHat Enterprise Linux 7 Ubuntu 18.04 LTS
Table 2.8: Pix4D vs ODM Benchmarks
Server Component Small Project Medium Project ODOT RHEL Server Pix4D 15 mins 1 hr ODOT RHEL Server ODM 15 mins 1 hr Alienware Server Pix4D 30 mins 1 hr 5 mins Alienware Server ODM 30 mins 1 hr servers. Table 2.8 shows the results of the testing. It can be seen that the more powerful ODOT RHEL Server outperformed the Alienware server on the ’small project’ but the results are comparable when it comes to the larger ’medium dataset’.
Figure 2.20 and 2.21 shows the Potree viewer displaying the results of the small and medium projects.
Figure 2.20: Potree viewer displaying the results of the small project
48 CHAPTER 2. COMMON OPERATING PLATFORM 2.5. RESULTS
Figure 2.21: Potree viewer displaying the results of the medium project
CHAPTER 2. COMMON OPERATING PLATFORM 49 Chapter 3
Custom Software Design: Traffic Monitoring
This chapter details the work done to create a custom processing system for calcu- lating metrics from videos of uninterrupted traffic flow. The software described in this system runs as a workflow in the common operating platform.
3.1 Objectives
The objective of this section is to demonstrate the use of computer vision algorithms to analyze videos of traffic over a segment of an interstate highway to calculate the average travel speed, flow rate and density per lane of traffic. Previous works in this field have been able to achieve success calculating the cumulative quantity of these metrics for the segment as a whole. This section builds on those works to estimate these metrics for every lane of traffic. It is also the objective to design the framework of this algorithm to be able to run in an automated environment discussed in chapter 2.
50 3.2. REGION OF INTEREST
Figure 3.1: Traffic Monitoring Architecture
3.2 Region of Interest
Calculating a region of interest in the video is a vital data cleaning step. It facilitates the removal of unwanted noise from the video. To extract traffic metrics, we only require those portions of the video frame that involve vehicular traffic. The footage from the drone will almost always contain elements surrounding the roadway like trees, buildings, lamp-posts. The objective of the region of interest extractor is to seek out those areas of the video frame that contain some form of vehicular traffic and cut out those regions that are not required for or will hinder processing.
To detect vehicles, we employ a state-of-the-art, real-time object detection algorithm called You only look once (YOLO) [50]. The idea is to pre-process the entire video and create a scatter plot of all instances of detected vehicles. The object detector is run through the video frame by frame, and for each frame, it returns a list of objects detected, the class of the object and the bounding box coordinates of the object in the frame. The bounding box coordinates make up a rectangle with the coordinates of the upper left and lower right corners known. The mid-point of the diagonal between the two is chosen as the point used in creating the scatter plot
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 51 3.2. REGION OF INTEREST
Figure 3.2: Object detection using YOLO mentioned before. This results in a scatter plot of all the objects ever detected in every frame of the video as shown in figure 3.3.
Figure 3.3: Region of interest scatter plot
52 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.2. REGION OF INTEREST
The next step is to locate the centroids of the dense clusters. An unsu- pervised learning algorithm called the k-means[51] clustering algorithm is used for this purpose. The value of k is configurable by the user. The algorithm partitions the points on the 2D space into k clusters, where each point belongs to a particular cluster based on the nearest mean. The result is a 2D space which is partitioned into Voronoi cells[52].
Figure 3.4: Region of interest scatter plot with cluster centers
The cluster centers form the basis for extrapolating our region of interest based on an adjustable smoothening distance. Centered around the cluster centers, the aim is to mask the areas farthest from it using a bitwise operator in each frame. We use OpenCV[53], which provides inbuilt functions that enable us to do this. A rectangular mask is then applied to the frame using the estimated edge distances. This yields in a frame with just the traffic and the unwanted areas obscured by a black overlay as shown in figure 3.5. In doing so, we remove unwanted detections in the noisy areas, and it enables us to extract a scatter plot of the vehicular traffic
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 53 3.3. FEATURE EXTRACTION efficiently.
Figure 3.5: Region of interest applied to a frame
3.3 Feature Extraction
This step should ideally follow the region of interest phase, but it is not mandatory. Extracting features from the video will be significantly more accurate once the region of interest mask is applied to the frame. While we created a scatter plot from the object detector for the region of interest step, we require more detailed vehicular information for subsequent steps like lane detection. We use an object detector called You only look once (YOLO)[50] and intersection over union(IOU) for tracking.
In the feature extraction phase, we run through the video frame by frame. Several changeable parameters may be used to tweak the performance of the algo- rithm. Object detection is used to identify which targets to track, and this is done for every frame of video. Once the goal has been identified, we assign a tracker to follow it. Tracking consists of mainly two steps:-
1. Initializing the tracker with coordinates obtained from the object detector
54 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.3. FEATURE EXTRACTION
2. Updating the coordinates of the object being tracked
Coordinates are updated using a tracking algorithm based on intersection over union(IOU)[54]. IOU calculates the intersection between two objects detected in consecutive frames as shown in figure 3.6.
Figure 3.6: Intersection over Union of two bounding boxes from consecutive frames
The Object Detector is used to detect bounding boxes in each frame. Each frame will have multiple bounding boxes to compare; the objective is to find a bounding box in the next frame that overlaps a particular bounding box by a pre- determined margin. For this purpose, the Jaccard Index[54] or intersection over union(IOU) is used. The following example shows how intersection over union is used for tracking three objects between 2 frames. The 3x4 matrix shows the coordinates of the three detected objects from the previous frame while the 1x4 matrix is a single detection from the current frame.
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 55 3.3. FEATURE EXTRACTION
xa1 ya1 xa1 ya1 ox1 oy1 ox1 oy1 1 1 2 2 1 1 2 2 xa2 ya2 xa2 ya2(IOU) xb1 yb1 xb1 yb1 ⇒ ox2 oy2 ox2 oy2 1 1 2 2 1 1 2 2 1 1 2 2 3 3 3 3 1×4 3 3 3 3 xa1 ya1 xa2 ya2 ox1 oy1 ox2 oy2 3×4 3×4 oa1 ⇒ oa2 = IntersectionAreas oa3 3×1
The bounding box coordinates(diagonal pmulti-user availabilityoints of the
i i i i rectangle) are given by (xa1, ya1, xa2, ya2) where i is the index out of a total of n j j j j bounding boxes creating an (nx4) matrix. Let (xb1, yb1, xb2, yb2) be a point from the current frame. Then, the intersection over union or IOU operation may be defined as follows.
i i j ox1 = max(xa1, xb1)
i i j oy1 = max(ya1, yb1)
i i j ox2 = min(xa2, xb2)
i i j oy2 = min(ya2, yb2)
Once the overlapping bounding box coordinates are calculated, then the area of that bounding box is calculated in the final step. This can be done multi- plying the length of the bounding box by its breadth. The resultant matrix will be (nx1).
56 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.3. FEATURE EXTRACTION
i i i i i oa = |ox2 − ox1| ∗ |oy2 − oy1|
Similarly, all of the bounding box areas from the previous frame are calcu-
j j j j lated. The co-ordinates (xb1, yb1, xb2, yb2) is in reality, one of many bounding boxes detected in the current frame. The areas of all the bounding boxes detected in the current frame are also calculated. Let them be represented as follows.
|xa1 − xa1| ∗ |ya1 − ya1| 2 1 2 1 P reviousF rameAreas = |xa2 − xa2| ∗ |ya2 − ya2| 2 1 2 1 3 3 3 3 |xa2 − xa1| ∗ |ya2 − ya1| 3×1
|xb1 − xb1| ∗ |yb1 − yb1| 2 1 2 1 CurrentObjectArea = |xb1 − xb1| ∗ |yb1 − yb1| 2 1 2 1 1 1 1 1 |xb2 − xb1| ∗ |yb2 − yb1| 3×1
Then, the intersection over union of the bounding boxes from the previ- ous frame(3x4 matrix) with respect to the selected detection(1x4 matrix) from the current frame is given by,
IntersectionAreas IOU = (P reviousF rameAreas + CurrentObjectArea − IntersectionAreas)
The current object is then assigned to the tracker whose IOU value is high- est. This step is repeated m number of times, where m is the number of bounding boxes detected in the current frame. This gives the algorithm a run-time complex- ity[55] of O(m*j), where j is the number of trackers in the previous frame. This
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 57 3.3. FEATURE EXTRACTION tracker assignment takes place for every vehicle being tracked by the algorithm. Tracker is implemented as a class with every vehicle being an object of that class. Listing 3.1 implements the tracker class. c l a s s Tracker: t r a c k e r counter = TrackerCount() instances = [] f i n i s h e d instances = [] p o s i t i v e trackers = [] n e g a t i v e trackers = [] trajectories = np.array([])
d e f i n i t (self , point, last u p d a t e d f r a m e number , label): s e l f .id = Tracker.tracker counter .increment() self.label = label s e l f . c o ordinates = [] s e l f . f i r s t u p d a t e d f r a m e number = last u p d a t e d f r a m e n u m b e r s e l f . l a s t u p d a t e d f r a m e number = None s e l f . a v g l e n g t h = [ ] self .update(point , last u p d a t e d f r a m e n u m b e r ) self.classified = False
# To be assigned later self .trajectory = None s e l f . l a n e c l a s s e s = [ ] self.slope = None self .intercept = None self.speed = 0.0
d e f update(self , point, frame number ) : s e l f . l a s t u p d a t e d f r a m e number = frame number s e l f . c o ordinates .append(point) s e l f . a v g length .append(np. sqrt((point [2] − p o i n t [ 0 ] ) ∗∗ 2 + (point[3] − p o i n t [ 1 ] ) ∗∗ 2) )
@staticmethod d e f clean trackers(min c o o r d i n a t e s t o track=10, stale t r a c k e r threshold=5): curr frame number = FrameCount. frame c o u n t . g e t c o u n t ( ) to remove = [ ] f o r tracker in Tracker.instances: i f (curr frame number − tracker .last u p d a t e d f r a m e n u m b e r ) >= s t a l e t r a c k e r t h r e s h o l d : i f len(tracker.co o r d i n a t e s ) >= m i n c o o r d i n a t e s t o t r a c k : Tracker. finished instances .append(tracker) Tracker. instances .remove(tracker)
Listing 3.1: Tracker Class
Each tracker has two entities that determine how long an object is tracked. They are a failure counter and a boundary offset. The failure counter counts the number of times a tracker failed to update. These parameters are controlled using the clean trackers function shown in listing 3.1. If the tracker fails to update more than a stipulated number of times, then the object will stop being tracked and the
58 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.4. LANE DETECTION information thus far collected is stored for processing. The boundary offset is used to determine if a tracker has passed beyond the boundaries of the frame. The absence of this parameter causes the tracker to linger on the boundaries even though the object has gone out of the frame. frame bbox matrix, labels list = create b b o x matrix(object detector . predict(frame)) t r a c k e r b b o x matrix = Tracker.get b b o x m a t r i x ( ) v e c t o r c o u n t = 0 f o r vector in frame b b o x m a t r i x : max iou, index = 0, −1 i f tracker b b o x matrix.size != 0: iou = intersection o v e r union(vector , tracker b b o x m a t r i x ) max iou = np . max(iou) index = iou.argmax(axis=0)[0, 0] point = (vector[0, 0], vector[0, 1], vector[0, 2], vector[0, 3]) i f max iou >= i o u t h r e s h o l d : tracker = Tracker.instances[index] tracker .update(point=point , frame number=FrameCount . frame c o u n t . g e t c o u n t ( ) ) e l s e: Tracker. instances .append( Tracker(point=point , last u p d a t e d f r a m e number=FrameCount. frame c o u n t . g e t c o u n t , label=labels list [vector c o u n t ] ) ) v e c t o r c o u n t += 1
Listing 3.2: Feature Extraction Code
3.4 Lane Detection
3.4.1 Detecting Lane Lines using Hough Transforms
In this section, a method to detect lane lines using traditional computer vision transforms is explained. The method is based on the lane detection system discussed in Udacity’s Self Driving Car Engineer Nanodegree[56]. While the method was originally used for detecting lane lines on video taken off a dashboard mounted camera inside a vehicle, it is shown here that the same method can be applied to videos shot using a UAV from the side of the road. The first step is to convert the original frame(figure 3.7) into the HSV(hue, saturation, value) space(figure 3.8), which is an alternate representation of the RGB color model. Listing 3.3 shows code written using OpenCV[53]. Converting the frame into the HSV space helps to
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 59 3.4. LANE DETECTION
Figure 3.7: Original Frame highlight lighter colors like white and yellow, which are the primary colors used to draw lane lines.
@staticmethod d e f t o hsv(image): r e t u r n cv2.cvtColor(image , cv2.COLOR BGR2HSV)
Listing 3.3: Convert Frame to HSV
Figure 3.8: Original Frame in HSV Space
60 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.4. LANE DETECTION
The next step is to apply gaussian blur to the frame(figure 3.9). This reduces the noise in the frame and enables the sharper filtering of yellow and white colors(figure 3.10).
@staticmethod d e f g a u s s i a n blur(img, kernel s i z e =3) : r e t u r n cv2.GaussianBlur(img, (kernel size , kernel s i z e ) , 0)
@staticmethod d e f f i l t e r color(img): y e l l o w min = np.array([20, 100, 100], np.uint8) yellow max = np.array([30, 255, 255], np.uint8) yellow mask = cv2.inRange(img, yellow min , yellow max )
white min = np.array([0, 0, 200], np.uint8) white max = np.array([255, 80, 255], np.uint8) white mask = cv2.inRange(img, white min , white max )
img = cv2.bitwise and(img, img, mask=cv2. bitwise o r ( yellow mask , white mask ) ) r e t u r n img
Listing 3.4: Apply Gaussian Blur and filter yellow and white colors
Figure 3.9: Apply Gaussian Blur
Next Canny edge detection[57] is applied to detect the highlighted edges(figure 3.11). This yields a frame with the lane pixels highlighted.
@staticmethod d e f e d g e detection(img, low threshold , high threshold): r e t u r n cv2.Canny(img, low threshold , high t h r e s h o l d )
Listing 3.5: Canny Edge Detection
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 61 3.4. LANE DETECTION
Figure 3.10: Filter Yellow and White Colors
Figure 3.11: Canny Edge Detection
The final step is to use Hough Line Transform[58] to detect lines in edge images. The result is shown in figure 3.12.
There are false detections in the boundary of the images as can be seen in figure 3.12. This can be minimized using the region of interest method explained in section 3.2. Another method for better detection is also detailed in section 3.4.2.
62 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.4. LANE DETECTION
Figure 3.12: Hough Transform Lane Detection
To summarize, the above process is represented in the form of a block diagram in figure 3.13.
Figure 3.13: Lane Detection Block Diagram
3.4.2 Tracing Lane Lines
Sometimes when the video has a large amount of noise, it becomes impractical to detect lane lines with the methods described in previous sections. Some examples are when the lane lines get degraded over time or during a snowstorm. So, a pre- processing step is employed where the lane lines are traced with human supervision.
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 63 3.5. VEHICLE TRAJECTORY ANALYSIS
The software will open up a dialog where the user can trace a line over the lanes using a mouse. The lane lines are only traced for the first frame of the video. They are then used as a point of reference for detecting lanes or a fall back in case of bad detection. Figure 3.14 shows a user-generated lane trace. The lanes are indexed from 0 to the number of lanes - 1 in the order in which they were drawn.
Figure 3.14: Tracing Lane Lines
Lane Detection using the method described in this section requires a stable video feed. In the absence of a steady feed, it is recommended that the unstable video be processed in smaller durations and aggregate the results.
3.5 Vehicle Trajectory Analysis
The output of the feature extraction module is a trajectory of the vehicles as they drive along the monitored segment of the roadway. The output of the lane detection module is the coordinates of the lane lines. Information from these two modules are fused in this step to calculate lane assignment, Flow Rate, Average Travel Speed and Density(see section 1.3.7 for more information on the metrics). The algorithm will also work in the absence of the data from the lane detection module. The output,
64 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.5. VEHICLE TRAJECTORY ANALYSIS in that case, will be the combination of all the lanes instead of being per lane.
The first step of trajectory analysis is to create a trajectory model for each tracker. After the feature extraction phase, a set of trackers is generated. Each of these trackers holds the coordinate information for every vehicle detected in the video. The blue line in figure 3.15 shows the trajectory of a vehicle recorded by the feature extraction module. As you can see, the edge is very jagged. The instability of the video and the object detector being used causes this.
Figure 3.15: Trajectory of a vehicle recorded by the feature extraction module
It is a hard problem to match the jagged blue trajectory to a lane. So, linear regression is used to fit a line as indicated by the dashed black line in figure 3.15. The lane that this tracker belongs to is determined by this smoothened line. If a tracker has been on multiple lanes, the lane is assigned based on the which lane the tracker traveled through the most. i.e., where the vehicle was detected the most number of times. The three metrics are then calculated once the lane assignment is completed. Each of these metrics is calculated per minute and extrapolated uniformly to the
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 65 3.5. VEHICLE TRAJECTORY ANALYSIS hour.
Flow Rate is the equivalent hourly rate at which vehicles pass over a given point or section of a lane or roadway during a given time interval less than 1 hour[36]. It is expressed in vehicles/hour. Metrics are generated for every minute of video. For example, if the video is 3 minutes long, then three metric data points are created. To do this, the trackers are assigned to one-minute time bins, where the tracker is attached to the bin corresponding to the minute and lane it was detected in. The number of trackers in the bin at the end of the minute defines the flow rate for that minute. Now, we extrapolate the metric to the hour by multiplying the number by 60. This metric is now the number of vehicles detected on that lane for the given hour.
The Average Travel Speed is defined as a rate of motion expressed as distance per unit time (mi/h). This work uses time mean speed(see section 1.3.7 for more details) is used to calculate average travel speed. To calculate the average travel speed for the lane, it is necessary first to calculate the individual speeds of the trackers detected per lane. The average travel speed for the lane will then be the average of those speeds. Chiddarwar et al. [37] proposed a method to calculate the speed of trackers. They map the distance in pixels to the real world measurements like miles. They figure this based on the fact that compact size sedans are on average around 4.8 meters. They use this measure along with the object detector to determine the length of a sedan in pixels. This measure is then used to calculate the distance traveled by a tracker in miles.
Moreover, based on the time for which the tracker was observed for, the speed of the tracker in miles per hour is calculated. This process is repeated for every recorded tracker. The algorithm can be summarized as shown below.
1. Measure the distance in pixels between the point of the first detection and the last point of tracking
66 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
2. Using the known information about the average length of a compact sedan, convert the distance covered by the tracker in pixels to meters
3. Calculate the time of observation between points of first and last detection
4. Calculate the speed in meters per second and convert it to miles per hour by multiplying it by 2.237
5. Repeat above steps for every tracker
Now that the flow rate and average travel speeds have been calculated per lane for every minute of video, these metrics can now be used to determine the corresponding densities with the expression described in section 1.3.7.
3.6 Results
This section describes the results of applying the traffic monitoring algorithms de- tailed in previous sections to videos acquired flying a drone off Exit-6, Mitchell Avenue and St. Bernard on the I-75 South near Cincinnati, OH. The lanes from the highway are labeled as shown in figure 3.16.
The algorithm calculates the following metrics for every lane of traffic: Count of vehicles, Flow Rate, Average Travel Speed and Density. For more infor- mation about these metrics please refer to 1.3.7. An error metric is defined to rate the performance of the lane wise vehicle counts. This metric is given by equation 3.6.
HumanCount − MachineCount Error(%) = ∗ 100 HumanCount CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 67 3.6. RESULTS
Figure 3.16: Lane labels assigned by the lane detection module
3.6.1 Counts of Vehicles
This section lists the vehicular counting results of videos from April 18, 2018 and June 20, 2018. All results are lane wise and use the error metric defined in 3.6 to determine its accuracy. Table 3.1 shows the human counts for April 18, 2018 flights and table 3.2 shows the machine counts for April 18, 2018 flights. Similarly, tables 3.4 & 3.5 shows the human counts for April 18, 2018 flights and tables 3.6 & 3.7 shows the machine counts for June 20, 2018 flights.
A comparison of the human and machine counts is made using the error metric defined in equation 3.6. For the April 18, 2018 flights this information maybe found in table 3.3 and for the June 20, 2018 flights it maybe found in tables 3.8 & 3.9.
68 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
Table 3.1: Human Counts from April 18, 2018
Human Counts Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 0005 1.mp4 16 22 13 4 9 14 78 DJI 0005 8.mp4 20 18 26 7 2 7 80 DJI 0005 2.mp4 19 21 14 4 11 11 80 DJI 005 3.mp4 31 22 24 6 7 4 94 DJI 005 4.mp4 18 22 18 6 9 6 79 DJI 0005 5.mp4 28 26 18 3 5 4 84 DJI 0005 6.mp4 19 16 19 3 1 7 65 DJI 0005 7.mp4 22 21 19 2 9 4 77
Table 3.2: Machine Counts from April 18, 2018
Machine Counts Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 0005 1.mp4 17 24 13 6 5 16 81 DJI 0005 8.mp4 16 16 28 8 3 5 76 DJI 0005 2.mp4 18 21 14 4 12 16 85 DJI 005 3.mp4 32 21 25 6 7 6 97 DJI 005 4.mp4 18 23 17 6 8 8 80 DJI 0005 5.mp4 31 34 19 3 5 3 95 DJI 0005 6.mp4 18 14 21 3 0 6 62 DJI 0005 7.mp4 18 23 23 4 7 7 82
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 69 3.6. RESULTS
Table 3.3: Error(%) from April 18, 2018
Error(%) Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 0005 1.mp4 -6.25 -9.09 0 -50 44.44 -14.28 -3.84 DJI 0005 8.mp4 20 11.11 -7.69 -14.28 -50 28.57 5 DJI 0005 2.mp4 5.26 0 0 0 -9.09 -45.45 -6.25 DJI 005 3.mp4 -3.22 4.54 -4.16 0 0 -50 -3.19 DJI 005 4.mp4 0 -4.54 5.55 0 11.11 -33.33 -1.26 DJI 0005 5.mp4 -10.71 -30.76 -5.55 0 0 25 -13.09 DJI 0005 6.mp4 5.26 12.5 -10.52 0 100 14.28 4.61 DJI 0005 7.mp4 18.18 -9.52 -21.05 -100 22.22 -75 -6.49
All of these trials were conducted independent of each other and collated using Microsoft Excel. Furthermore, the information from both flights is now col- lated to compare it with the works presented in [37]. They have processed the same videos using a different algorithm which calculates the same metrics holistically as opposed to the lane wise segregation proposed here. The total lane wise human and machine counts along with their error metrics can be found in table 3.10. This data is further consolidated to give table 3.11. This is comparable to the findings of Chiddarwar et al. [37] in their work.
As can be seen in table 3.11 the overall error between the human and ma- chine counts on the Interstate highway(lanes 3-6) is 0.22% while on the ramp(lanes 0 and 1) is 9.92%.
3.6.2 Flow Rate, Average Travel Speed and Density
In section 3.6.1, the results of calculating the counts of vehicles on the highway segment were discussed. In this section, the results of calculating the flow rate, average travel speed, and density are discussed. The results are based on the videos
70 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
Table 3.4: Human Counts from June 20, 2018
Human Counts Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 00015 1.mp4 27 23 22 6 9 9 96 DJI 00015 2.mp4 31 25 19 9 10 11 105 DJI 00015 3.mp4 32 25 17 7 13 7 101 DJI 00015 4.mp4 26 15 19 13 8 11 92 DJI 00015 5.mp4 30 22 22 11 10 10 105 DJI 00015 6.mp4 23 25 20 7 3 4 82 DJI 00015 7.mp4 31 20 14 5 16 13 99 DJI 00015 8.mp4 24 23 18 0 10 8 83 DJI 00015 9.mp4 36 23 14 12 3 7 95 DJI 00015 10.mp4 32 24 25 7 10 5 103 DJI 00016 1.mp4 26 18 15 10 7 9 85 DJI 00016 2.mp4 30 21 16 7 10 7 91 DJI 00017 1.mp4 30 24 23 11 6 5 99 DJI 00017 2.mp4 30 25 15 7 12 10 99 DJI 00017 3.mp4 27 20 18 8 10 7 90 DJI 00017 4.mp4 30 18 16 15 4 11 94 DJI 00017 5.mp4 27 18 13 6 7 5 76 DJI 00017 6.mp4 33 20 14 5 10 6 88 DJI 00018 1.mp4 29 25 25 1 8 6 94
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 71 3.6. RESULTS
Table 3.5: Continued - Human Counts from June 20, 2018
Human Counts Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 00019 1.mp4 28 18 17 8 1 6 78 DJI 00019 2.mp4 28 24 24 14 9 12 111 DJI 00019 3.mp4 35 24 26 8 8 8 109 DJI 00019 4.mp4 33 18 17 10 2 14 94 DJI 00019 5.mp4 27 26 16 10 15 12 106 DJI 00019 6.mp4 32 26 16 7 15 10 106 DJI 00020 1.mp4 31 19 16 9 15 10 100 DJI 00020 2.mp4 30 19 12 9 4 8 82 DJI 00020 3.mp4 32 24 16 12 10 6 100 DJI 00020 4.mp4 27 20 13 11 3 7 81 DJI 00020 5.mp4 30 18 12 7 9 9 85 DJI 00020 6.mp4 29 24 14 5 12 8 92 DJI 00020 7.mp4 30 15 18 3 5 3 74 DJI 00020 8.mp4 32 18 19 6 4 12 91 DJI 00020 9.mp4 26 23 17 5 9 8 88 DJI 00020 10.mp4 29 21 12 11 3 6 82 DJI 00020 11.mp4 27 21 14 5 4 9 80 DJI 00020 12.mp4 27 23 16 6 8 7 87 DJI 00020 13.mp4 31 25 18 13 3 5 95 DJI 00020 14.mp4 28 23 16 5 15 6 93 DJI 00020 15.mp4 25 19 16 10 8 6 84 DJI 00020 16.mp4 25 17 14 6 2 10 74
72 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
Table 3.6: Machine Counts from June 20, 2018
Machine Counts Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 0005 5.mp4 31 34 19 3 5 3 95 DJI 0005 6.mp4 18 14 21 3 0 6 62 DJI 0005 7.mp4 18 23 23 4 7 7 82 DJI 00015 1.mp4 25 21 20 6 10 8 90 DJI 00015 2.mp4 32 28 14 8 9 10 101 DJI 00015 3.mp4 28 21 11 8 12 12 92 DJI 00015 4.mp4 25 12 17 9 7 12 82 DJI 00015 5.mp4 31 27 24 14 9 12 117 DJI 00015 6.mp4 21 18 17 7 2 5 70 DJI 00015 7.mp4 30 26 13 3 15 15 102 DJI 00015 8.mp4 24 20 18 1 9 10 82 DJI 00015 9.mp4 34 20 17 13 3 8 95 DJI 00015 10.mp4 36 27 23 9 10 4 109 DJI 00016 1.mp4 27 20 13 10 6 14 90 DJI 00016 2.mp4 30 20 16 8 10 11 95 DJI 00017 1.mp4 28 23 19 10 4 6 90 DJI 00017 2.mp4 30 27 16 7 13 15 108 DJI 00017 3.mp4 30 23 20 8 10 10 101 DJI 00017 4.mp4 32 15 17 20 6 14 104 DJI 00017 5.mp4 29 14 11 5 6 5 70 DJI 00017 6.mp4 32 19 13 5 9 5 83 DJI 00018 1.mp4 27 23 24 2 5 6 87
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 73 3.6. RESULTS
Table 3.7: Continued - Machine Counts from June 20, 2018
Machine Counts Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 00019 1.mp4 30 14 14 6 1 2 67 DJI 00019 2.mp4 30 24 29 11 9 12 115 DJI 00019 3.mp4 36 30 26 7 10 9 118 DJI 00019 4.mp4 34 14 15 11 0 15 89 DJI 00019 5.mp4 31 26 20 11 12 14 114 DJI 00019 6.mp4 33 28 15 6 11 9 102 DJI 00020 1.mp4 38 20 13 9 12 10 102 DJI 00020 2.mp4 28 15 12 8 3 10 76 DJI 00020 3.mp4 32 24 13 12 9 7 97 DJI 00020 4.mp4 26 22 10 11 4 6 79 DJI 00020 5.mp4 25 13 11 7 10 12 78 DJI 00020 6.mp4 23 24 15 5 13 8 88 DJI 00020 7.mp4 32 11 15 3 4 5 70 DJI 00020 8.mp4 39 19 18 5 4 11 96 DJI 00020 9.mp4 29 19 19 6 9 7 89 DJI 00020 10.mp4 22 24 13 11 2 8 80 DJI 00020 11.mp4 27 24 14 8 4 9 86 DJI 00020 12.mp4 27 20 13 5 7 8 80 DJI 00020 13.mp4 33 25 17 16 2 8 101 DJI 00020 14.mp4 25 26 14 5 12 6 88 DJI 00020 15.mp4 24 18 16 12 3 9 82 DJI 00020 16.mp4 23 17 11 7 2 10 70
74 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
Table 3.8: Error(%) from June 20, 2018
Error(%) Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 00015 1.mp4 7.41 8.70 9.09 0.00 -11.11 11.11 6.25 DJI 00015 2.mp4 -3.23 -12.00 26.32 11.11 10.00 9.09 3.81 DJI 00015 3.mp4 12.50 16.00 35.29 -14.29 7.69 -71.43 8.91 DJI 00015 4.mp4 3.85 20.00 10.53 30.77 12.50 -9.09 10.87 DJI 00015 5.mp4 -3.33 -22.73 -9.09 -27.27 10.00 -20.00 -11.43 DJI 00015 6.mp4 8.70 28.00 15.00 0.00 33.33 -25.00 14.63 DJI 00015 7.mp4 3.23 -30.00 7.14 40.00 6.25 -15.38 -3.03 DJI 00015 8.mp4 0.00 13.04 0.00 N/A 10.00 -25.00 1.20 DJI 00015 9.mp4 5.56 13.04 -21.43 -8.33 0.00 -14.29 0.00 DJI 00015 10.mp4 -12.50 -12.50 8.00 -28.57 0.00 20.00 -5.83 DJI 00016 1.mp4 -3.85 -11.11 13.33 0.00 14.29 -55.56 -5.88 DJI 00016 2.mp4 0.00 4.76 0.00 -14.29 0.00 -57.14 -4.40 DJI 00017 1.mp4 6.67 4.17 17.39 9.09 33.33 -20.00 9.09 DJI 00017 2.mp4 0.00 -8.00 -6.67 0.00 -8.33 -50.00 -9.09 DJI 00017 3.mp4 -11.11 -15.00 -11.11 0.00 0.00 -42.86 -12.22 DJI 00017 4.mp4 -6.67 16.67 -6.25 -33.33 -50.00 -27.27 -10.64 DJI 00017 5.mp4 -7.41 22.22 15.38 16.67 14.29 0.00 7.89 DJI 00017 6.mp4 3.03 5.00 7.14 0.00 10.00 16.67 5.68 DJI 00018 1.mp4 6.90 8.00 4.00 -100.00 37.50 0.00 7.45
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 75 3.6. RESULTS
Table 3.9: Continued - Error(%) from June 20, 2018
Error(%) Interstate Ramp File Name Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total DJI 00019 1.mp4 -7.14 22.22 17.65 25.00 0.00 66.67 14.10 DJI 00019 2.mp4 -7.14 0.00 -20.83 21.43 0.00 0.00 -3.60 DJI 00019 3.mp4 -2.86 -25.00 0.00 12.50 -25.00 -12.50 -8.26 DJI 00019 4.mp4 -3.03 22.22 11.76 -10.00 100.00 -7.14 5.32 DJI 00019 5.mp4 -14.81 0.00 -25.00 -10.00 20.00 -16.67 -7.55 DJI 00019 6.mp4 -3.13 -7.69 6.25 14.29 26.67 10.00 3.77 DJI 00020 1.mp4 -22.58 -5.26 18.75 0.00 20.00 0.00 -2.00 DJI 00020 2.mp4 6.67 21.05 0.00 11.11 25.00 -25.00 7.32 DJI 00020 3.mp4 0.00 0.00 18.75 0.00 10.00 -16.67 3.00 DJI 00020 4.mp4 3.70 -10.00 23.08 0.00 -33.33 14.29 2.47 DJI 00020 5.mp4 16.67 27.78 8.33 0.00 -11.11 -33.33 8.24 DJI 00020 6.mp4 20.69 0.00 -7.14 0.00 -8.33 0.00 4.35 DJI 00020 7.mp4 -6.67 26.67 16.67 0.00 20.00 -66.67 5.41 DJI 00020 8.mp4 -21.88 -5.56 5.26 16.67 0.00 8.33 -5.49 DJI 00020 9.mp4 -11.54 17.39 -11.76 -20.00 0.00 12.50 -1.14 DJI 00020 10.mp4 24.14 -14.29 -8.33 0.00 33.33 -33.33 2.44 DJI 00020 11.mp4 0.00 -14.29 0.00 -60.00 0.00 0.00 -7.50 DJI 00020 12.mp4 0.00 13.04 18.75 16.67 12.50 -14.29 8.05 DJI 00020 13.mp4 -6.45 0.00 5.56 -23.08 33.33 -60.00 -6.32 DJI 00020 14.mp4 10.71 -13.04 12.50 0.00 20.00 0.00 5.38 DJI 00020 15.mp4 4.00 5.26 0.00 -20.00 62.50 -50.00 2.38 DJI 00020 16.mp4 8.00 0.00 21.43 -16.67 0.00 0.00 5.41
Table 3.10: Total Counts with Error(%)
Total Counts with Error(%) Interstate Ramp Lane 6 Lane 5 Lane 4 Lane 3 Lane 1 Lane 0 Total Human Count 1369 1047 855 3271 383 383 4406 Machine Count 1366 1037 826 3229 345 345 4393 Error(%) 0.22 0.96 3.39 1.28 9.92 9.92 0.30
76 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
Table 3.11: Aggregate of human and machine counts for all videos with error(%)
Interstate vs Ramp Interstate Ramp Human Counts 1369 383 Machine Counts 1366 345 Error(%) 0.22 9.92 collected on April 18, 2018, and June 20, 2018.
Flow Rate
The flow rate is a metric that is derived from the vehicular counts. We have already proved the results of vehicular counts in section 3.6.1.
Average Travel Speed
The average travel speed is calculated based on the time mean speed discussed in 3.5. In order to verify the accuracy of the average travel speed, we conducted a few tests with vehicles whose speed was known(using odometer readings) and the objective of the exercise was to predict the speed of the vehicle. So, the video of a vehicle being driven on a stretch of road at known speeds was obtained for this experiment. Table 3.12.
Table 3.12: Results of experiment conducted to verify average travel speed
Actual Speed Calculated Speed Error (%) 10 9.75 -2.5 20 21.16 5.8 30 38.34 27.8 54 49.13 9.9 60 54.64 9.8 70 67.47 3.7
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 77 3.6. RESULTS
Density
Density is derived from the flow rate and average travel speed metrics from sections 3.6.2 and 3.6.2. This section shows the metrics plotted against each other as de- scribed in the Highway Capacity Manual[36]. According to the works described in the Highway Capacity Manual, these plots are intended to be compared with each other to give the viewer an idea of the state of traffic flow in the segment. Figure 3.17 shows a comparison of the different metrics between lane 6 and lane 4.
Figure 3.17: Comparison of Lane 6 with Lane 4
Based on the Highway Capacity Manual, traffic can be over-saturated(more vehicles on the road) or under-saturated(fewer vehicles on the road). We chose to compare Lane 6 with Lane 4 because they were the most densely crowded lanes based on visual observations. Moreover, the plots back up the visual observations. As can be seen in the plots, the majority of the points can be found in the over- saturated sections represented by the red regions in the theoretical plots shown in figure 3.17. Table 3.13 shows the average values of different metrics for lane 6 and lane 4.
Similarly, we compare the results of lane five on the interstate to lane one
78 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 3.6. RESULTS
Table 3.13: Average Metric Values for Lane 6 and Lane 4
Lane # Flow Rate Average Travel Speed Density Lane 6 1668.75 24.69 73.54 Lane 4 1007.50 15.89 80.93 on the ramp. Based on visual observations, lane 1 had mostly free-flowing traffic. Therefore, the lane one plots should appear primarily on the undersaturated parts according to the theoretical plot. Figure 3.18 shows the plots for lane 5 vs lane 1.
Figure 3.18: Comparison of Lane 5 with Lane 1
Table 3.14 shows the average values of different metrics for lane 5 and lane 1. It can be inferred that lane 5 had vastly more traffic when compared with lane 1. Moreover, as a result, it was also slower. Lane 1 with less traffic showed faster travel speeds.
Table 3.14: Average Metric Values for Lane 5 and Lane 1
Lane # Flow Rate Average Travel Speed Density Lane 5 1263.75 19.36 73.64 Lane 1 416.25 42.76 9.51
Figure 3.19 shows the comparison of 4 lanes of traffic. When viewed to-
CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING 79 3.6. RESULTS gether, we can see the plots resembling the theoretical curves much better. The average values of different metrics for lanes 1, 4, 5 and 6 are shown in table 3.15.
Figure 3.19: Comparison of lanes 1, 4, 5 and 6
Table 3.15: Average Metric Values for Lanes 1, 4, 5 and 6
Lane # Flow Rate Average Travel Speed Density Lane 1 416.25 42.76 9.51 Lane 4 1007.50 15.89 80.93 Lane 5 1263.75 19.36 73.64 Lane 6 1668.75 24.69 73.54
80 CHAPTER 3. CUSTOM SOFTWARE DESIGN: TRAFFIC MONITORING Chapter 4
Conclusion
A system for processing computational workflows in the cloud was discussed. The challenges of compatibility with different environment and workflow management were also discussed. The process of creating workflows out of proprietary, open source, and custom designed software was described. The performance of the inte- grated processing engines and the data flow management system were reviewed. The system was also shown to be extensible, reliable, modular and scalable. Using deep learning based computer vision algorithms, several traffic parameters like vehicular counts, flow rate, average travel speed, and density were calculated over a segment of the highway for every lane of traffic. The accuracies of these measurements were also shown to be very good in section 3.6.
4.1 Future Scope
In this thesis, a system for processing computational workflows in the cloud was discussed. As the system expands to encompass more use cases, it will require additional work. Here are some of the tasks that can be considered for future enhancements.
81 4.1. FUTURE SCOPE
1. The traffic monitoring algorithm uses an object detection model, trained on Microsoft’s COCO dataset[59]. Because of the generalistic nature of this dataset, we have seen misclassifications, and missed detections occur when using the model. Training a more specialized model with a vehicular dataset can yield better results with the framework of the current algorithm.
2. Some data processing tasks require specialized hardware for speeding up com- putation. These resources can be expensive. So, upgrading all the nodes in a multi-server setup may be considered extravagant. The ideal solution would be to have an optimal number of nodes with specialized capabilities and for the backend to allocate them as and when required. A prime example of such a scenario is the use of a graphical processing unit(GPU) for processing 3D models with Pix4D. It is recommended that the open source Kubernetes[60] engine be used in tandem with Docker[5] to facilitate the orchestration and allocation of resources in such a manner.
3. The current version of the common operating platform caps the maximum file upload at 4GB. This is because in a limited cluster setup, uploads are handled using a single HTTP POST request which has a limited time to live with respect to server wait times. In order to overcome this handicap, the data will need to split into multiple chunks/requests and shipped to the destination in place of the single request. A scalable storage system such as Microsoft Azure Blob Storage can be the alternate that we are looking for. They provide end points where data can be uploaded with higher times to live and are optimized for such tasks. It will also eliminate the need for large local storage and is more cost effective when it comes to storing large amounts of data.
4. Migrate postgres database to Oracle or Microsoft SQL Server according to ODOT IT requirements.
5. Integrate additional features like control points and manual tie points into the Pix4D processing workflow.
82 CHAPTER 4. CONCLUSION References
[1] Pix4D SA. Pix4DEngine Server. 2019. url: https : / / www . pix4d . com / enterprise/pix4dengine.
[2] Drone Mapping Software - OpenDroneMap. url: https://www.opendronemap. org/ (visited on 02/28/2019).
[3] What is Solid Modeling? 3D CAD Software. Applications of Solid Modeling.
url: https://www.brighthubengineering.com/cad-autocad-reviews- tips/19623-applications-of-cad-software-what-is-solid-modeling/ (visited on 05/02/2019).
[4] Professional photogrammetry and drone mapping software — Pix4D. url: https://www.pix4d.com/ (visited on 05/28/2019).
[5] Inc. Docker. Docker - Build, Ship, and Run Any App, Anywhere. 2016. url: https://www.docker.com (visited on 02/25/2018).
[6] Entwine & Potree. url: http://potree.entwine.io/ (visited on 02/28/2019).
[7] What is a Container? — Docker. url: https://www.docker.com/resources/ what-container (visited on 06/06/2019).
[8] docker images — Docker Documentation. url: https://docs.docker.com/ engine/reference/commandline/images/ (visited on 06/06/2019).
[9] Deep Learning Comp Sheet: Deeplearning4j vs. Torch vs. Caffe vs. Tensor- Flow vs. MxNet vs. CNTK - Deeplearning4j: Open-source, Distributed Deep
83 REFERENCES
Learning for the JVM. url: https://deeplearning4j.org/compare-dl4j- tensorflow-pytorch{\#}torch (visited on 10/15/2017).
[10] Adam Paszke et al. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. 2017.
[11] Mart´ınAbadi et al. “TensorFlow: Large-Scale Machine Learning on Hetero-
geneous Distributed Systems”. In: (). url: https://arxiv.org/pdf/1603. 04467.pdf.
[12] Yangqing Jia et al. “Caffe: Convolutional Architecture for Fast Feature Em-
bedding”. In: (2014). arXiv: 1408.5093. url: http://arxiv.org/abs/1408. 5093.
[13] James Bergstra et al. Theano: Deep Learning on GPUs with Python. Tech. rep.
2011, pp. 1–48. url: http://github.com/pascanur/DeepLearningBenchmarks.
[14] Pascal Lamblin and Yoshua Bengio. MILA and the future of Theano - Google
Groups. 2017. url: https://groups.google.com/forum/{\#}!topic/ theano-users/7Poq8BZutbY.
[15] Fran¸coisChollet et al. “Keras: Deep learning library for theano and tensor- flow”. In: URL: https://keras. io/k 7.8 (2015).
[16] Sharan Chetlur et al. “cuDNN: Efficient Primitives for Deep Learning”. In: ().
url: https://arxiv.org/pdf/1410.0759.pdf.
[17] Yali Amit and Pedro Felzenszwalb. Object Detection. url: https : / / cs . brown.edu/people/pfelzens/papers/detection.pdf.
[18] Paul Viola and Michael Jones. “Rapid object detection using a boosted cas- cade of simple features”. In: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. Vol. 1. IEEE. 2001, pp. I–I.
[19] David G Lowe. “Distinctive image features from scale-invariant keypoints”. In: International journal of computer vision 60.2 (2004), pp. 91–110.
84 REFERENCES REFERENCES
[20] Navneet Dalal and Bill Triggs. “Histograms of oriented gradients for human detection”. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE. 2005, pp. 886–893.
[21] Corinna Cortes and Vladimir Vapnik. “Support-vector networks”. In: Machine learning 20.3 (1995), pp. 273–297.
[22] Jiansheng Chen et al. “Automatic Image Cropping : A Computational Com-
plexity Study”. In: (). url: http://www.cv-foundation.org/openaccess/ content{\ _ }cvpr{\ _ }2016 / papers / Chen{\ _ }Automatic{\ _ }Image{\ _ }Cropping{\_}CVPR{\_}2016{\_}paper.pdf.
[23] Luis Herranz, Shuqiang Jiang, and Xiangyang Li. “Scene recognition with
CNNs: objects, scales and dataset bias”. In: (). url: http : / / www . cv - foundation.org/openaccess/content{\_}cvpr{\_}2016/papers/Herranz{\_ }Scene{\_}Recognition{\_}With{\_}CVPR{\_}2016{\_}paper.pdf.
[24] Huan Fu et al. “Occlusion Boundary Detection via Deep Exploration of Con-
text”. In: (). url: http://www.cv-foundation.org/openaccess/content{\_ }cvpr{\_}2016/papers/Fu{\_}Occlusion{\_}Boundary{\_}Detection{\_ }CVPR{\_}2016{\_}paper.pdf.
[25] David M W. “EVALUATION: FROM PRECISION, RECALL AND F-MEASURE TO ROC, INFORMEDNESS, MARKEDNESS & CORRELATION”. In: Jour-
nal of Machine Learning Technologies 2.1 (2011), pp. 37–63. url: http:// dspace.flinders.edu.au/dspace/ http://www.bioinfo.in/journalcontent. php?vol{\_}id=115{\&}id= http://www.bioinfo.in/contents.php?id= 51.
[26] Yann Lecun, Yoshua Bengio, and Rm 4g332. “Convolutional Networks for Images, Speech, and Time-Series LeCun & Bengio: Convolutional Networks for Images, Speech, and Time-Series LeCun & Bengio: Convolutional Net-
works for Images, Speech, and Time-Series”. In: (). url: https : / / www . researchgate.net/profile/Yann{\_}Lecun/publication/2453996{\_
REFERENCES 85 REFERENCES
}Convolutional{\_}Networks{\_}for{\_}Images{\_}Speech{\_}and{\_ }Time-Series/links/0deec519dfa2325502000000.pdf.
[27] Rafael Uetz and Sven Behnke. “Large-scale object recognition with CUDA- accelerated hierarchical neural networks”. In: 2009 IEEE International Con- ference on Intelligent Computing and Intelligent Systems. IEEE, 2009, pp. 536–
541. isbn: 978-1-4244-4754-1. doi: 10.1109/ICICISYS.2009.5357786. url: http://ieeexplore.ieee.org/document/5357786/.
[28] Christian Szegedy et al. “Rethinking the Inception Architecture for Computer
Vision”. In: (). url: https://arxiv.org/pdf/1512.00567.pdf.
[29] Mubarak Yilmaz, Alper; Javed, Omar; Shah. “Object Tracking: A Survey”.
In: ACM Computing Surveys2 38 (2006). url: http : / / delivery . acm . org / 10 . 1145 / 1180000 / 1177355 / a13 - yilmaz . pdf ? ip = 129 . 137 . 96 . 7{\&}id=1177355{\&}acc=ACTIVE SERVICE{\&}key=1D8E1CA5B8D7D8DD. E3810009149F8453 . 4D4702B0C3E38B35 . 4D4702B0C3E38B35{\ & }{\ _ }{\ _ }acm{\_}{\_}=1556557472{\_}c23505e4b7cfcf47b04bc134bca5c48d.
[30] Adrian Rosebrock. Simple object tracking with OpenCV - PyImageSearch. url: https://www.pyimagesearch.com/2018/07/23/simple-object-tracking- with-opencv/ (visited on 04/29/2019).
[31] Samuel Murray. “Real-Time Multiple Object Tracking A Study on the Impor-
tance of Speed”. In: (). arXiv: 1709.03572. url: https://arxiv.org/pdf/ 1709.03572.pdf.
[32] Andrej Karpathy et al. “Large-scale Video Classification with Convolutional
Neural Networks”. In: (). url: http://cs.stanford.edu/people/karpathy/ deepvideo.
[33] Bryan C. Russell et al. “LabelMe: A Database and Web-Based Tool for Im- age Annotation”. In: International Journal of Computer Vision 77.1-3 (2008),
pp. 157–173. issn: 0920-5691. doi: 10 . 1007 / s11263 - 007 - 0090 - 8. url: http://link.springer.com/10.1007/s11263-007-0090-8.
86 REFERENCES REFERENCES
[34] Li Deng. “The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]”. In: IEEE Signal Processing Magazine
29.6 (2012), pp. 141–142. issn: 1053-5888. doi: 10.1109/MSP.2012.2211477. url: http://ieeexplore.ieee.org/document/6296535/.
[35] D. Ciresan, U. Meier, and J. Schmidhuber. “Multi-column deep neural net- works for image classification”. In: 2012 IEEE Conference on Computer Vision
and Pattern Recognition. IEEE, 2012, pp. 3642–3649. isbn: 978-1-4673-1228-8. doi: 10.1109/CVPR.2012.6248110. url: http://ieeexplore.ieee.org/ document/6248110/.
[36] Roger P Roess and Elena S Prassas. “The Highway Capacity Manual: A Con- ceptual and Research History Springer Tracts on Transportation and Traffic
STTT Volume 1: Uninterrupted Flow”. In: (). url: https://link-springer- com.proxy.ohiolink.edu:9100/content/pdf/10.1007{\%}2F978-3-319- 05786-6.pdf.
[37] Arjun Chiddarwar. “Application Of Computer Vision Algorithms For Unin- terrupted Traffic Monitoring Based On Aerial Images And Videos”. In: June (2019).
[38] Bootstrap · The most popular HTML, CSS, and JS library in the world. url: https://getbootstrap.com/ (visited on 06/03/2019).
[39] Niranjan Krishnan et al. A Common Operating Platform for Employing UAVs
in Infrastructure Monitoring. 2018, pp. 221–226. isbn: 978-1-57117-454-3. url: https://ndtlibrary.asnt.org/en/2018/ACommonOperatingPlatformforEmployingUAVsinInfrastructureMonitoring. aspx.
[40] Microservices vs Monolithic Architecture — MuleSoft. url: https://www. mulesoft.com/resources/api/microservices-vs-monolithic (visited on 06/06/2019).
REFERENCES 87 REFERENCES
[41] Microservices Architecture: Advantages and Drawbacks. url: https://cloudacademy. com/blog/microservices-architecture-challenge-advantage-drawback/ (visited on 06/06/2019).
[42] Yukun Liu, Yong Yue, and Liwei Guo. “UNIX File System”. In: UNIX Operat- ing System. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 149–185.
doi: 10.1007/978-3-642-20432-6_6. url: http://link.springer.com/ 10.1007/978-3-642-20432-6{\_}6.
[43] Mark Masse. REST API Design Rulebook: Designing Consistent RESTful Web
Service Interfaces. url: https://books.google.com/books?hl=en{\&}lr= {\&}id=eABpzyTcJNIC{\&}oi=fnd{\&}pg=PR3{\&}dq=rest+api{\&}ots= vzPE3{\_}egHz{\&}sig=5DFT3ISVAtEOR646SsNgp84EKEc{\#}v=onepage{\& }q=rest api{\&}f=false.
[44] Nigel George. Mastering Django: Core (The Django Book). 2nd Edition.
[45] docker/docker-py: A Python library for the Docker Engine API. url: https: //github.com/docker/docker-py (visited on 05/09/2019).
[46] VisualCron - Windows Scheduling Software and Task Automation — Task
Scheduler Alternative. url: https : / / www . visualcron . com/ (visited on 06/06/2019).
[47] Upload limits for Internet Eplorer, Mozilla firefox, Google Chrome, Opera, IIS
and ASP. Article. url: https://www.motobit.com/help/scptutl/pa98.htm (visited on 06/03/2019).
[48] System design Dropbox or Google drive. – Narendra L – Medium. url: https: //medium.com/@narengowda/system-design-dropbox-or-google-drive- 8fd5da0ce55b (visited on 06/03/2019).
[49] Andreas Holzinger et al. “A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop”. In: (2017). arXiv:
1708.01104. url: http://arxiv.org/abs/1708.01104.
88 REFERENCES REFERENCES
[50] Joseph(University of Washington) Redmon. “YOLOv3: An Incremental Im-
provement”. In: (). url: https://pjreddie.com/media/files/papers/ YOLOv3.pdf.
[51] J. A. Hartigan and M. A. Wong. “Algorithm AS 136: A K-Means Clustering
Algorithm”. In: Applied Statistics 28.1 (1979), p. 100. issn: 00359254. doi: 10. 2307/2346830. url: http://www.jstor.org/stable/10.2307/2346830? origin=crossref.
[52] Franz Aurenhammer and Franz. “Voronoi diagrams—a survey of a funda- mental geometric data structure”. In: ACM Computing Surveys 23.3 (1991),
pp. 345–405. issn: 03600300. doi: 10 . 1145 / 116873 . 116880. url: http : //portal.acm.org/citation.cfm?doid=116873.116880.
[53] Itseez. Open Source Computer Vision Library. https://github.com/itseez/ opencv. 2015.
[54] University of Konstanz) Kosub, Sven (Department of Computer & Information Science. “A note on the triangle inequality for the Jaccard distance”. In: ().
arXiv: 1612.02696. url: https://arxiv.org/pdf/1612.02696.pdf.
[55] Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Ap-
proach. 1st. New York, NY, USA: Cambridge University Press, 2009. isbn: 0521424267, 9780521424264.
[56] Self Driving Car Engineer Nanodegree — Udacity. url: https://www.udacity. com/course/self-driving-car-engineer-nanodegree--nd013 (visited on 05/06/2019).
[57] JOHN CANNY. “A Computational Approach to Edge Detection”. In: Read-
ings in Computer Vision (1987), pp. 184–203. doi: 10.1016/B978- 0- 08- 051581- 6.50024- 6. url: https://www.sciencedirect.com/science/ article/pii/B9780080515816500246.
REFERENCES 89 REFERENCES
[58] Richard O. Duda and Peter E. Hart. “Use of the Hough Transformation to Detect Lines and Curves in Pictures”. In: Commun. ACM 15.1 (Jan. 1972),
pp. 11–15. issn: 0001-0782. doi: 10.1145/361237.361242. url: http:// doi.acm.org/10.1145/361237.361242.
[59] Lin, Tsung-Yi; Maire, Michael; Belongie, Serge; Hays, James; Perona , Pietro ; Ramanan, Deva; Dollar, Piotr;Zitnick, C. Lawrence. “Microsoft COCO: Com-
mon Objects in Context”. In: (). url: http://vision.ucsd.edu/sites/ default/files/coco{\_}eccv.pdf.
[60] Production-Grade Container Orchestration - Kubernetes. url: https://kubernetes. io/ (visited on 06/03/2019).
90 REFERENCES Appendix A
Point Clouds on Potree Viewer
Potree[6] is an open source web based point cloud visualization tool. It has been used in the common operating platform for visualizing point clouds generated by the 3D model processing engines as well as user uploaded point cloud files. .las and .laz are the supported formats.
1. Figure A.1 shows a point cloud of ODOT Headquarters, Columbus, Ohio. The point cloud file was generated elsewhere and uploaded to the common operating platform for visualization.
Figure A.1: Point cloud of ODOT HQ, Columbus, OH
91 2. Figure A.2 shows a point cloud of a Shed on ODOT premises at District 8, Ohio. The point cloud file was generated using Pix4D Server Engine [1] on the common operating platform and visualized on the online Potree viewer.
Figure A.2: Point cloud of a shed, District 8, OH
92 APPENDIX A. POINT CLOUDS ON POTREE VIEWER