FASTSWARM: A Data-driven FrAmework for Real-time Flying InSecT SWARM Simulation Wei Xiang1 Xinran Yao1 He Wang2 Xiaogang Jin1,3∗ 1 State Key Lab of CAD&CG, Zhejiang University, Hangzhou 310058, China 2 School of Computing, University of Leeds, Leeds LS2 9JT, United Kingdom 3 ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab, Hangzhou 310058, China Abstract Insect swarms are common phenomena in Keywords: insect swarm simulation, data- nature and therefore have been actively pur- driven, optimization, collective behavior, real sued in computer animation. Realistic insect time swarm simulation is difficult due to two challenges: high-fidelity behaviors and large scales, which make the simulation practice 1 Introduction subject to laborious manual work and excessive Insects are ubiquitous in both the real and virtual trial-and-error processes. To address both worlds, and many of them present collective be- challenges, we present a novel data-driven haviors for efficient and collaborative work. In framework, FASTSWARM, to model complex the real world, flying insect swarms can exhibit behaviors of flying insects based on real-world a great variety of behaviors such as aggrega- data and simulate plausible animations of flying tion, mating, migration and escaping [1], where insect swarms. FASTSWARM has a linear time the individual behaviors are often correlated in complexity and achieves real-time performance various ways, from collaborative to competitive for large swarms. The high-fidelity behavior or even adversarial. Simulating realistic insect model of FASTSWARM explicitly takes into swarms are in the interest of many areas. In consideration the most common behaviors of robotics, research of insect swarms has led to flying insects, including the interactions among new algorithms for robots collective jobs on in- insects such as repulsion and attraction, the self- formation transfer, decision-making, task par- propelled behaviors such as target following and titioning or transport [2, 3]. In computer ani- obstacle avoidance, and other characteristics mation, insect swarms have been used to create such as the random movements. To achieve wondrous natural phenomena and interesting vi- arXiv:2007.11250v1 [cs.GR] 22 Jul 2020 scalability, an energy minimization problem sual effects [4, 5]. However, simulating scalable is formed with different behaviors modelled collective behaviors of insect swarms with high- as energy terms, where the minimizer is the fidelity remains challenging. desired behavior. The minimizer is computed Existing methods for simulating flying insect from the real-world data, which ensures the swarms mainly fall into two categories: empir- plausibility of the simulation results. Extensive ical and data-driven. Empirical methods aim simulation results and evaluations show that to abstract swarm behaviors into mathematical FASTSWARM is versatile in simulating var- models and deterministic systems, such as the ious swarm behaviors, high fidelity measured field-based methods [4], or a combination of by various metrics, easily controllable in the field-based with the force-based methods inducing user controls and highly scalable. [5]. One limitation of such methods is that ∗Corresponding author. E-mail: [email protected] the simulated trajectories are often too regular and lack of visual diversity, due to their deter- fore optimize for the velocity to update the mo- minism nature. In contrast, data-driven meth- tion states of the agents. In addition, we use an ods tend to rely on real-world data, such as us- implicit Euler scheme to improve the stability. ing computer vision techniques to capture 3D Formally, the contributions of the paper in- trajectories of swarms [6, 7, 8] for simulation clude: [9, 10, 11]. However, due to the intrinsic limi- tations of optical sensors (e.g. occlusions), the • A novel data-driven 3D swarm simulation motion capture is set up in massively simplified framework which captures a variety of bio- laboratory environments, and there are still ex- logically important behaviors. cessive tracking errors where only short track- • An optimization method that maximally lets can be relatively reliably obtained. This cre- makes use of real-world data to ensure the ates tremendous difficulties in simulating flying simulation fidelity for flying insect swarms. insect swarms with the desired high-fidelity and scalability. First, the captured trajectories can- • A scalable model for large swarm simula- not be relied upon to extract all behaviors of fly- tion with straightforward user control. ing insects. Second, the generalizability of the model based on simple data is limited by both The remainder of this paper is organized as the environment complexity and the swarm size. follows. After briefly introducing related work in Section 2, we give a pipeline overview of our In this paper, we propose a novel data-driven approach in Section 3. In Section 4, we explain framework (FASTSWARM) to address the chal- our optimization-based data-driven model. We lenges for simulating flying insect swarms. Our show simulation results and evaluations of our framework models insects as agents, and the method in Section 5, and discuss the limitations swarm behavior computation as an energy min- and future work in Section 6. imization problem. A variety of important be- haviors identified in numerical analysis [12] and empirical observations are captured by different 2 Related Work energy terms, including the interaction among agents, the self-propulsion of agents and the mo- Data-Driven Simulation. In graphics, data- tion noise of agents, so that the minimizer leads driven methods have been proposed to simulate to realistic behaviors. Besides, our framework behaviors of crowds and traffics. Given trajec- also model user-defined behaviors by employ- tories (or tracklets) extracted from crowd data, ing user-control energy terms. The total energy example-based methods can blend them to gen- function is constructed in the way that it can erate new animations [14], use a “clone and be optimized quickly to achieve scalability and paste” technique to generate larger crowds [15], real-time performance. or cluster them into groups and update the mo- During optimization, instead of seeking a tion of an agent based on the actions of its near- minimizer by pure mathematical optimizations est patch or associated group [16, 17, 18]. which would make the minimizer only ideal in In data-driven traffic simulation, Chao et al. theory, we seek the minimizer by referencing [19] present a video-based approach to learn the a motion characteristic dataset generated from specific driving characteristics of drivers to re- the real-world data, so that the simulated behav- construct or simulate traffic flows. By taking iors mimic the real data. However, this means the spatio-temporal information of traffic flows that the motion characteristics we reply on in as a 2D texture, a texture synthesis technique the reference dataset has to be reliable. In the is developed to populate virtual road networks real-world data, although excessive noises ex- with realistic traffic flows [20]. In addition, ist and whole trajectories can rarely be obtained, deep learning can also be used to learn the latent velocity is much more reliable as it can be esti- patterns of vehicle trajectories for intersectional mated from short tracklets [13]. In our simula- traffic simulation and editing [21]. Recently, tion framework, both velocity and acceleration an interactive data-driven optimization approach are regarded as the motion characteristics for [13] has been proposed to simulate traffic sce- generating the reference dataset, and we there- narios with heterogeneous agents. Real Trajectories Noise Interaction among Agents Response to environment Attraction Motion Danger Scenario Repulsion Reference Dataset Zone User-Defined Control Motion States Attraction-Agent of Agents Zone Data Processing Initialization Optimization Result Figure 1: Overview of our data-driven approach for simulating flying insect swarms. Although these methods can generate plausi- tied to specific data. Recently, data-driven noise ble crowd or traffic animations, they focus on models and force-based models are introduced 2D simulation and cannot be easily extended to in [10, 11] to generate biologically plausible an- 3D flying insect swarms because the motion dy- imations of flying insect swarms. However, they namics of insects are significantly different. do not generalize well to more complex scenar- Insect Swarm Simulation. There has been ios because they are prone to numerical errors an interdisciplinary effort in the research of col- during generalization. lective behaviors of insects. Researchers in agri- Except for data-driven methods, a hybrid culture proposed an insect migration trajectory model combining potential fields and curl-noise simulation method to accurately predict the des- [29] is developed in [4] to simulate various be- tinations of insect migration and achieve effec- haviors of flying insect swarms. However, such tive early warning to reduce the impact of pests a forced-based and field-based method often on agriculture [22]. Public health researchers generates trajectories that look too regular be- proposed an indoor flight behavior model of cause all agents share similar motion patterns. host-seeking mosquitoes for selecting bed nets For the special effect simulation application, that can effectively reduce the spread of the Chen et al. present a flock morphing method virus from mosquitoes