Graph-theoretic data modeling with application to neuromorphology, activity recognition and event detection A Dissertation Presented to The faculty of the School of Engineering and Applied Science University of Virginia In partial fulfillment of the requirements for the degree Doctor of Philosophy (Electrical and Computer Engineering) by Tamal Batabyal April 2019 Approval Sheet This dissertation is submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Electrical and Computer Engineering) Author: Tamal Batabyal This dissertation has been read and approved by the examining committee: Scott T. Acton, Dissertation Adviser Zongli Lin, Committee Chair Stephen G. Wilson, Committee Member Laura Barnes, Committee Member Daniel S. Weller, Committee Member Barry Condron, Committee Member Accepted for the School of Engineering and Applied Science: Dean, School of Engineering and Applied Science April 2019 Acknowledgement I would like to express my deepest gratitude to my advisor, Dr. Scott T. Acton to give me the opportunity and freedom to pursue my research. He provided support and encour- agement during my graduate studies, especially while pursuing out-of-the-box solutions of problems. I would like to thank my committee members, Dr. Zongli Lin, Dr. Stephen G. Wilson, Dr. Laura Barnes, Dr. Dan Weller and Dr. Barry Condron for their time and consideration. My sincere thanks to my recommenders - Dr. Scott T. Acton, Dr. Dan Weller, Dr. Andrea Vaccari, and Dr. Barry Condron for their important career advice during my post-doctoral applications. I would also like to thank my Master's advisor, Dr. Dipti Prasad Mukherjee, for his help and support. Thanks are given to the members of VIVA Laboratory for their help and encouragement. I will definitely miss discussing research problems and socializing with them. Sincere thanks to Dr. Suvadip Mukherjee, Dr. Rituparna Sarkar, Dr. Shruba Gangopadhyay, Dr. Anirudhha Dutta for their advice and suggestions regarding my career opportunities. Friends have been a big part in my life during my graduate studies. I would like to acknowledge Neel Samanta, Justin Cavanaugh, Ivan Shabalin, Tiffany Ly, Paul Bon- czek, Magdalena Woi´nska, Anna Borowska, Rachel Smith, Bassem Tossun, Sourav Maji, Deepak Kosanam and others. Special thanks to Izabela Hawro for her hospitality and generosity. I would like to mention Megan Evans and Deiziane Viane for their limitless enthusiasm in the squash court. Also, special thanks to the salsa club at University of Virginia. My sincere thanks to Archan Ray and Kaushik Chakraborty for having weirdest and funniest discussions since we met at ISI Kolkata. I would also like to acknowledge Nilan- jan Datta, Avik Chakraborti, Subhabrata Samajdar, and Sanjay Bhattacharjee for their support while I was at ISI Kolkata. Lastly, nothing can I achieve without my parents, Samar Kumar Batabyal and Suvra Batabyal. I simply have no words to describe their continuous support, love, affection and sacrifice. Abstract Recently the world is witnessing an explosion of data, demanding scientific and application- specific models for data analysis. Unarguably, data is the new oil. It is well-known that a graph is the most abstract and discrete representation of data that originated from real world problems. These problems range from the seven bridges of K¨onigsberg in 1736 to monolithic integrated circuits to biology, food web, internet, transportation, computer systems, social networks and countless others. Depending on the problem at hand, such complex data can be broadly classified into two groups. The first group accounts for problems where the data inherit distinct and invariant graph structures due to several factors that include the nature of data acquisition, data sampling, and intrinsically dis- crete structures of the data sources. A noteworthy example is 3D reconstructed neuron cells, where each neuron emulates a tree-structure branching topology. The second group of problems involves datasets that are deficient of fixed graph structures. Unlike the first category, the graph in this case should either be estimated based on optimization criteria or be initialized using some ad hoc constraints. Image or video based data is one example of this category, where the graph could be drawn from the pixels, the patches, or the frames depending on the nature of the problem. The primary objective of this thesis is to investigate the above two categories of graph structured data with varying degrees of structural complexity in specific problems. We perform novel image analysis, build graph models, develop graph theoretic tools and scalable algorithms for two major purposes - informatics and categorization. In the first group, we consider activity recognition and neuromorphology. In the second group, we consider geomorphological event detection and event monitoring from videos. In activity recognition, we use 3D reconstructed skeleton models of subjects performing a predefined set of activities. In such a model, the number of vertices and edges are fixed, connoting low structural complexity. We leverage such simple structures and propose Unified Graph Signal Processing (UGraSP) to integrate activity recognition and person identification using the same graph features. With additional graph structures, UGraSP is improved to make Unified Graph based Activity Detection (UGrAD) to encode the ac- tivity dynamics in terms of graph features. The central aspect of this thesis is the graph theoretic modeling of neuronal arbors. Feature-based representation and categorization of neurons, a topic that correlates with neuronal functionality, is still an open problem in neuromorphology. We use 3D reconstructed neurons (fixed graph) from stained im- ages, where the number of vertices is not fixed for every neuron but the edges between the vertices are structurally fixed. We propose NeuroPath2Path that utilizes path-based mod- eling of neuron anatomy and provides a visualization tool for the continuous deformation between a pair of neurons. NeuroPath2Path offers several advantages. Decomposition of a neuron into paths can be viewed as an assembly of individual circuits from the terminals to the soma, integrating semi-local features that act as path descriptors. Next, instead of subgraph matching, NeuroPath2Path provides a full-graph matching algorithm. The matching algorithm presents several biological factors, including fractality and decaying importance of features along the path. NeuroPath2Path also precisely investigates the fea- sibility of algorithmic constraints on the structural repertoire of neuronal arbors, thereby enforcing criteria, such as hierarchy mismatch. NeuroPath2Path can be extended to two major domains - morphological analysis and structural transformation of microglia cells, and in progressive degradation of neuronal paths in neurodegenerative diseases. In addi- tion, the knowledge of neuronal functions, such as potentiation and co-adaptive spiking can be translated to neural networks in order to achieve better performance and emulate neurons by way of neural networks. In geomorphological event detection, we are interested in three major application areas, which are road health estimation, sinkhole detection and monitoring, and rock-slope fault detection, from InSAR (Interferometric Synthetic Aperture Radar) data. The structural complexity of the data is elevated compared to the previous two problems. It is because there is no predefined connectivity among the vertices. We first model the data with a graph using K-nearest neighborhood approach. Then, we develop Laplacian Weighted Covariance (LaWeCo) for spatial localization of geomorphological events. Later, using a temporal bipartite graph model and iterative prior estimation, we propose DDT to detect as well as monitor such events. Simultaneous detection and tracking of events from videos is another challenging prob- lem with more complexity of graph structures. Here, there are multiple options for the selection of vertices and edges. By blending the recursive estimation of a spatial graph and temporal graphs of patches over time into a dictionary learning framework, we pro- pose Graph based Dictionary learning for Event Detection (GraDED) to accomplish both the tasks. Lastly, we consider the problem of accelerating the convergence of LMS filters. The problem apparently does not require graph modeling of data and combinatorial fea- ture extraction. However, the imposition of a graph structure to the data is shown to improve the convergence of LMS. Contents Contents vi List of Figures ix List of Tables xvi 1 Introduction 1 1.1 Objectives and contributions . .6 1.2 Thesis outline . .9 2 Background 10 2.1 Graph theory . 10 2.2 InSAR & ArcGIS . 12 2.2.1 Coherent motion analysis toolbox . 13 2.2.2 Road smoothness analysis toolbox . 13 3 Invariant or unique graph structure 15 3.1 Activity recognition . 16 3.1.1 Unified graph signal processing (UGraSP) . 17 3.1.1.1 Proposed method . 17 3.1.1.2 Projection on extended Laplacian eigenvector basis . 18 3.1.1.3 Feature construction for activity recognition . 19 3.1.1.4 Feature construction for person identification . 20 3.1.1.5 Results . 21 3.1.1.5.1 Activity recognition . 22 3.1.1.5.2 Person identification . 23 3.1.2 Unified graph based activity detection (UGrAD) . 25 3.1.2.1 Action boundary determination . 26 3.1.2.2 Bipartite flow construction . 27 3.1.2.3 Polynomial fitting . 29 3.1.2.4 Datasets . 31 3.1.2.5 Results . 32 3.2 Neuromorphology . 34 3.2.1 NeuroBFD . 38 vi Contents vii 3.2.1.1 Feature construction . 38 3.2.1.2 Results . 41 3.2.2 Neuron solver using Laplacian (NeuroSoL) . 43 3.2.2.1 Vertex labeling . 44 3.2.2.2 Feature extraction . 47 3.2.2.3 Optimization . 48 3.2.2.4 Results . 49 3.2.3 What is Path2Path and its variants? . 53 3.2.4 ElasticPath2Path . 54 3.2.4.1 Neuron as a graph . 55 3.2.4.2 Elastic morphing and SRVF . 56 3.2.4.3 Path-to-Path matching . 58 3.2.4.4 Datasets and Results . 58 3.2.5 NeuroPath2Path . 62 3.2.5.1 Path modeling of a neuron .