Flock Heterogeneity and its Applications by Geoff Nagy M.Sc., University of Manitoba, 2016 B.Sc., University of Manitoba, 2013 Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in the School of Computing Science Faculty of Applied Science © Geoff Nagy 2021 SIMON FRASER UNIVERSITY Summer 2021 Copyright in this work rests with the author. Please ensure that any reproduction or re-use is done in accordance with the relevant national copyright legislation. Approval Name: Geoff Nagy Degree: Doctor of Philosophy (Computer Science) Title: Flock Heterogeneity and its Applications Examining Committee: Chair: Mo Chen Assistant Professor Richard Vaughan Senior Supervisor Professor Richard Zhang Supervisor Professor Hang Ma Supervisor Assistant Professor Nick Sumner Internal Examiner Associate Professor Computing Science Simon Fraser University Dylan Shell External Examiner Associate Professor Computer Science and Engineering Texas A&M University Date Defended: August 31, 2021 ii Abstract Although the vast majority of synthetic flocking models assume that agents in a flock are homogeneous, biologists have shown that this is not the case in real flocks. Mixed-species flocks, for example, are quite common, as are differences in interaction behaviours between agents in a flock. This heterogeneity presents a barrier to our understanding of flocking behaviour, and must be taken into consideration when developing models of such behaviour. This thesis makes the following contributions. First, it describes a software tool for gener- ating photo-realistic images of synthetic flocks for the purpose of training a neural network to learn individual-level attributes such as bird species, position, depth, and flapping phase. Although a quantitative evaluation of this tool is not available at the time of writing, qual- itative analysis shows that the output from this tool shares many features in common with photos of real flocks. The second contribution of this thesis is to describe and evaluate the performance of an en- gineered flocking controller that exploits the pairwise flocking phenomenon—the tendency for birds of certain species to flock in stable monogamous pairs. Advantages of this engi- neered controller include more stable flock formations that are less likely to break up in the presence of obstacles, as well as a reduced number of tracking interactions overall between agents. This controller is evaluated using both simulated and real drones (on a small team of novel low-cost drones called the µBee). The aforementioned advantages are found to be present in both settings at multiple scales. These results show that an engineered pairwise flocking controller may have useful real-world applications in settings where flock stability or limited computational resources available for tracking flock mates are important factors. Keywords: biologically-inspired robotics, drones, flocking iii Dedication This thesis is dedicated to my father, Steve Nagy. Thanks, Dad, for always being there and encouraging me to achieve great things. iv Acknowledgements Acknowledgements are hard to write—inevitably, I’ll end up forgetting someone! If your name is not listed below, and you believe that it should be, you can reach me at [email protected]. But since that email address will likely be rudely yanked out from underneath me by the time I’m no longer a student, I wouldn’t hold my breath. First and foremost, I would like to thank my advisor, Richard Vaughan. I will miss our travels, particularly those in the UK. I fondly remember repeated instances of let’s try this pub on nights where you, Sepehr, and I were already stumbling back to the Nightingale House. I’ve learned a lot from you and I am immensely grateful for your guidance over these past five years. By some strange twist of fate, it might be possible that you’ll still be my boss somewhere after I graduate, but that’s probably not very likely. Wait a second... Also, a very big thank you to both Richard Zhang and Hang Ma for supervising my thesis progress, Nick Sumner for his role as internal examiner, and Dylan Shell as my external examiner. Thank you all for taking the time to support me. My graduate student colleagues (including those long-graduated), too numerous to men- tion here, are also owed significant gratitude. Thank you very much for comments on early drafts of my papers, NSERC applications, and other work. However, some of you I am grateful I will never travel with again, and you know who you are. On a serious note, to Suzie, I owe immeasurable gratitude. Everything I do becomes easier with your love and support. Thank you for your endless encouragement and for always being there for me. It means the world to me. Lastly, thank you to my family and friends for putting up with my anti-social behaviours during my graduate school career. My studies actually had nothing to do with that, so you can expect more of the same. v Table of Contents Approval ii Abstract iii Dedication iv Acknowledgements v Table of Contents vi List of Tables ix List of Figures x 1 Introduction 1 1.1 Flocks and Heterogeneity . 1 1.2 Studying Flocks with Computer Vision . 2 1.3 Pairwise Flocking . 3 1.4 Thesis Structure . 5 2 Previous Work 6 2.1 Introduction . 6 2.2 Filming Flocks . 6 2.3 Biologically-Inspired Algorithms and Robots . 9 2.4 Flocking Algorithms . 12 2.5 Conclusion . 13 3 The BirdGen Tool 15 3.1 Introduction . 15 3.2 Motivation . 16 3.3 Design . 18 3.3.1 Virtual Environment . 18 3.3.2 3D Models and Flocking Characteristics . 18 3.3.3 Flocking Controller . 20 vi 3.4 Output . 22 3.5 Comparison Against Real Flocks . 24 3.5.1 Visual Comparison . 24 3.5.2 Training Performance on a Neural Network . 26 3.6 Future Work and Conclusion . 27 4 Pairwise Flocking Simulations 28 4.1 Introduction . 28 4.2 Flocking Controller . 28 4.3 Evaluation . 29 4.3.1 Simulator and Environment . 29 4.3.2 Experiments and Performance Criteria . 31 4.4 Results . 33 4.5 Discussion . 35 4.5.1 Applications and Limitations . 37 4.6 Future Work and Conclusion . 39 5 The µBee Drone 40 5.1 Introduction . 40 5.2 Main Specifications and Development . 42 5.2.1 3D Printing the µBee Frame . 42 5.2.2 Electronic Components and Hardware . 42 5.3 Support for Motion Capture Systems . 44 5.4 Flight Control Loop . 46 5.5 Flight Panel Application . 50 5.6 Wireless Protocol . 50 5.7 Future Work and Conclusion . 52 6 Pairwise Flocking with the µBee 53 6.1 Introduction . 53 6.2 Drone Command Center . 54 6.2.1 Interface and Modules . 54 6.2.2 Simulation Accuracy . 56 6.3 Verification of Pairwise Flocking Correctness . 58 6.3.1 Pairwise Flocking Controller Port . 58 6.3.2 Results . 59 6.4 Physical µBee Experiments . 60 6.4.1 Flying Multiple µBees . 60 6.4.2 Experiment Setup . 64 6.4.3 Experiment #1 . 66 vii 6.4.4 Experiment #2 . 68 6.4.5 Overall Discussion . 71 6.4.6 Limitations and Future Work . 72 6.5 Evalution of the µBee . 72 6.6 Conclusion . 73 7 Conclusion 75 Bibliography 77 Appendix A Code 83 viii List of Tables Table 3.1 Individual bird parameters used by BirdGen. 20 Table 4.1 Experiment #1 numbers of paired and unpaired agents. 32 Table 5.1 Relevant 3D printing parameters used in Cura 4.8.0 to slice the µBee frame for export to an AnyCubic Mega S 3D printer. 43 Table 5.2 The µBee PCB’s fabrication parameters. 46 Table 5.3 Message types defined by the µBee wireless communications protocol. 51 Table 5.4 Flight parameters used to tune the µBee’s flight, by flight parameter group. All values are 4-byte floating-point. 52 Table 6.1 Breakdown of the number of trials for the experiments used to verify the correctness of the pairwise flocking behaviour implemented in the DCC. 59 Table 6.2 Average number of sub-flocks at the conclusion of each trial. 69 Table 6.3 Average number of tracking connections during all trials, grouped by configuration. 70 Table 6.4 Average length of each trial, grouped by configuration. 70 ix List of Figures Figure 1.1 The main forces involved in a classical Boids controller. Adapted from http://www.red3d.com/cwr/boids/............... 2 Figure 1.2 Paired agents A and B using the pairwise flocking controller. Agent A tracks the neighbours on A’s side of the formation, plus Agent B. Agent B tracks agents on the opposite side, plus Agent A. Both agents travel closer together than other non-paired agents. 4 Figure 1.3 High-resolution photographs of corvid flocks by Jolles et al. [26] re- veal that such flocks can be partially composed of pairs. 4 Figure 2.1 Top-down illustration of the 3-camera setup used by Kelly et al. [28]. 7 Figure 2.2 Pairs are present in this flock of corvids. Used with permission from Guillam McIvor. 8 Figure 2.3 The main forces involved in a classical Boids controller. Adapted from http://www.red3d.com/cwr/boids/. 12 Figure 3.1 Sample output image generated by BirdGen. Ground truth 3D poses of all simulated birds are known and can be used for training a neural network. 15 Figure 3.2 Representation of a 4-camera setup for imaging flocks of birds. 16 Figure 3.3 The sketches provided by Guillam McIvor and the resulting 3D mod- els and rigging. 19 Figure 3.4 Sample output image generated by BirdGen. 21 Figure 3.5 A photograph of a real flock of jackdaws.