A Data-Driven Framework for Real-Time Flying Insect SWARM Simulation

FASTSWARM: A Data-driven FrAmework for Real-time Flying InSecT SWARM Simulation

Wei Xiang1 Xinran Yao1 He Wang2 Xiaogang Jin1,3∗ 1 State Key Lab of CAD&CG, Zhejiang University, Hangzhou 310058, China 2 School of Computing, University of Leeds, Leeds LS2 9JT, United Kingdom 3 ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab, Hangzhou 310058, China

Abstract Insect swarms are common phenomena in Keywords: insect swarm simulation, data- nature and therefore have been actively pur- driven, optimization, collective behavior, real sued in computer animation. Realistic insect time swarm simulation is difficult due to two challenges: high-fidelity behaviors and large scales, which make the simulation practice 1 Introduction subject to laborious manual work and excessive Insects are ubiquitous in both the real and virtual trial-and-error processes. To address both worlds, and many of them present collective be- challenges, we present a novel data-driven haviors for efficient and collaborative work. In framework, FASTSWARM, to model complex the real world, flying insect swarms can exhibit behaviors of flying insects based on real-world a great variety of behaviors such as aggrega- data and simulate plausible animations of flying tion, mating, migration and escaping [1], where insect swarms. FASTSWARM has a linear time the individual behaviors are often correlated in complexity and achieves real-time performance various ways, from collaborative to competitive for large swarms. The high-fidelity behavior or even adversarial. Simulating realistic insect model of FASTSWARM explicitly takes into swarms are in the interest of many areas. In consideration the most common behaviors of robotics, research of insect swarms has led to flying insects, including the interactions among new algorithms for robots collective jobs on in- insects such as repulsion and attraction, the self- formation transfer, decision-making, task par- propelled behaviors such as target following and titioning or transport [2, 3]. In computer ani- obstacle avoidance, and other characteristics mation, insect swarms have been used to create such as the random movements. To achieve wondrous natural phenomena and interesting vi-

arXiv:2007.11250v1 [cs.GR] 22 Jul 2020 scalability, an energy minimization problem sual effects [4, 5]. However, simulating scalable is formed with different behaviors modelled collective behaviors of insect swarms with high- as energy terms, where the minimizer is the fidelity remains challenging. desired behavior. The minimizer is computed Existing methods for simulating flying insect from the real-world data, which ensures the swarms mainly fall into two categories: empir- plausibility of the simulation results. Extensive ical and data-driven. Empirical methods aim simulation results and evaluations show that to abstract swarm behaviors into mathematical FASTSWARM is versatile in simulating var- models and deterministic systems, such as the ious swarm behaviors, high fidelity measured field-based methods [4], or a combination of by various metrics, easily controllable in the field-based with the force-based methods inducing user controls and highly scalable. [5]. One limitation of such methods is that ∗Corresponding author. E-mail: [email protected] the simulated trajectories are often too regular and lack of visual diversity, due to their deter- fore optimize for the velocity to update the mo- minism nature. In contrast, data-driven meth- tion states of the agents. In addition, we use an ods tend to rely on real-world data, such as us- implicit Euler scheme to improve the stability. ing computer vision techniques to capture 3D Formally, the contributions of the paper in- trajectories of swarms [6, 7, 8] for simulation clude: [9, 10, 11]. However, due to the intrinsic limitations of optical sensors (e.g. occlusions), the • A novel data-driven 3D swarm simulation motion capture is set up in massively simplified framework which captures a variety of bio- laboratory environments, and there are still ex- logically important behaviors. cessive tracking errors where only short track- • An optimization method that maximally lets can be relatively reliably obtained. This cre- makes use of real-world data to ensure the ates tremendous difficulties in simulating flying simulation fidelity for flying insect swarms. insect swarms with the desired high-fidelity and scalability. First, the captured trajectories can- • A scalable model for large swarm simula- not be relied upon to extract all behaviors of fly- tion with straightforward user control. ing insects. Second, the generalizability of the model based on simple data is limited by both The remainder of this paper is organized as the environment complexity and the swarm size. follows. After briefly introducing related work in Section 2, we give a pipeline overview of our In this paper, we propose a novel data-driven approach in Section 3. In Section 4, we explain framework (FASTSWARM) to address the chal- our optimization-based data-driven model. We lenges for simulating flying insect swarms. Our show simulation results and evaluations of our framework models insects as agents, and the method in Section 5, and discuss the limitations swarm behavior computation as an energy min- and future work in Section 6. imization problem. A variety of important behaviors identified in numerical analysis [12] and empirical observations are captured by different 2 Related Work energy terms, including the interaction among agents, the self-propulsion of agents and the mo- Data-Driven Simulation. In graphics, data- tion noise of agents, so that the minimizer leads driven methods have been proposed to simulate to realistic behaviors. Besides, our framework behaviors of crowds and traffics. Given trajec- also model user-defined behaviors by employ- tories (or tracklets) extracted from crowd data, ing user-control energy terms. The total energy example-based methods can blend them to gen- function is constructed in the way that it can erate new animations [14], use a “clone and be optimized quickly to achieve scalability and paste” technique to generate larger crowds [15], real-time performance. or cluster them into groups and update the mo- During optimization, instead of seeking a tion of an agent based on the actions of its near- minimizer by pure mathematical optimizations est patch or associated group [16, 17, 18]. which would make the minimizer only ideal in In data-driven traffic simulation, Chao et al. theory, we seek the minimizer by referencing [19] present a video-based approach to learn the a motion characteristic dataset generated from specific driving characteristics of drivers to re- the real-world data, so that the simulated behav- construct or simulate traffic flows. By taking iors mimic the real data. However, this means the spatio-temporal information of traffic flows that the motion characteristics we reply on in as a 2D texture, a texture synthesis technique the reference dataset has to be reliable. In the is developed to populate virtual road networks real-world data, although excessive noises ex- with realistic traffic flows [20]. In addition, ist and whole trajectories can rarely be obtained, deep learning can also be used to learn the latent velocity is much more reliable as it can be esti- patterns of vehicle trajectories for intersectional mated from short tracklets [13]. In our simula- traffic simulation and editing [21]. Recently, tion framework, both velocity and acceleration an interactive data-driven optimization approach are regarded as the motion characteristics for [13] has been proposed to simulate traffic sce- generating the reference dataset, and we there- narios with heterogeneous agents. Real Trajectories Noise Interaction among Agents Response to environment Attraction

Motion Danger Scenario

Repulsion Reference Dataset Zone User-Defined Control Motion States Attraction-Agent of Agents Zone

Data Processing Initialization Optimization Result

Figure 1: Overview of our data-driven approach for simulating ﬂying insect swarms.

Although these methods can generate plausi- tied to specific data. Recently, data-driven noise ble crowd or traffic animations, they focus on models and force-based models are introduced 2D simulation and cannot be easily extended to in [10, 11] to generate biologically plausible an- 3D flying insect swarms because the motion dy- imations of flying insect swarms. However, they namics of insects are significantly different. do not generalize well to more complex scenar- Insect Swarm Simulation. There has been ios because they are prone to numerical errors an interdisciplinary effort in the research of col- during generalization. lective behaviors of insects. Researchers in agri- Except for data-driven methods, a hybrid culture proposed an insect migration trajectory model combining potential fields and curl-noise simulation method to accurately predict the des- [29] is developed in [4] to simulate various be- tinations of insect migration and achieve effec- haviors of flying insect swarms. However, such tive early warning to reduce the impact of pests a forced-based and field-based method often on agriculture [22]. Public health researchers generates trajectories that look too regular be- proposed an indoor flight behavior model of cause all agents share similar motion patterns. host-seeking mosquitoes for selecting bed nets For the special effect simulation application, that can effectively reduce the spread of the Chen et al. present a flock morphing method virus from mosquitoes [23]. In swarm robotics, of flying insect swarms with pre-defined shape swarm behaviors can be used to control collab- constraints [5]. Instead of pre-defined morph- orative robots, such as the insect-based biobots ing, our framework can generate plausible ani- for search and rescue [24], unmanned aerial ve- mations based on real-world motions to satisfy hicle quadrotors emulating insect swarm behav- user-defined constraints. iors [25], and the task allocation algorithm based on ant colonies [26]. 3 Methodology Overview In graphics, methods have been proposed to simulate collective behaviors of ants [27, 28]. As illustrated in Figure 1, FASTSWARM can However, it is non-trivial to extend these meth- be conceptually described as a three-phase pro- ods to simulate flying insect swarms because of cess: the data processing stage, the initializa- their significant differences in motion dynam- tion stage, and the real-time simulation stage. ics. Data-driven methods have been developed Our simulation algorithm makes full use of the to simulate flying insect swarms. Li et al. [9] dataset extracted from the data processing stage present a framework to simulate flying insect to compute the trajectories of the agents in the swarms by sampling the statistical information swarm. At the data processing stage, the raw from real datasets. However, it is not general data are noisy tracklets [6, 8]. For each type of enough because a new local steering model is insects, we generate a reference dataset that con- needed whenever the data changes. Different tains the motion characteristics specific to that from the prior work, we present a general frame- type. Each data sample consists of two items: work which is maximally data-driven and auto- velocity and acceleration, estimated using for- matically adaptive to data, and therefore is not ward differencing. We organize the data by the speed, similar to [13], for fast indexing. At Ei,user is the user-control term to enforce user- the initialization stage, we initialize the motion defined constraints. scenario, the number of insects, and the initial For each agent i, the minimization of E aims motion state of each agent. The motion state to update vi,t+1 by selecting a v ∈ dv ∈ D. For of an agent includes its position, velocity, mo- simplicity, we use the selected velocity v and ˆv tion randomness, and control direction. During to represent vi,t+1 and its direction in the fol- simulation, we employ a data-driven approach lowing definitions of the energy terms. to update the motion state for each agent. For every time step, our model selects a velocity 4.1 Interaction among Agents from the reference dataset that minimizes an objective function which models the interactions, Insects move in coordination with each self-propulsion, and user-specified controls. other and aggregate without collisions. In FASTSWARM, such coordination is modeled as interactions which include repulsion for 4 Data-Driven Optimization collision avoidance and attraction for aggrega- Model tion. The interaction energy Ei,int is therefore defined as:

The reference data is denoted as D = ∪vdv, Ei,int = Ei,rep + Ei,att, (3) where dv = [v, a], v is the velocity, and a is the acceleration. For a swarm with N agents at time where Ei,rep and Ei,att represent the repulsion t, the state of agent i (i = 1, ..., N) is denoted n cd 12 energy and the attraction energy, respectively. as si,t = [pi,t, vi,t, vî,t, vî,t], si,t ∈ R , where 3 3 pi,t ∈ R is the current position, vi,t ∈ R is the n 3 4.1.1 Repulsion current velocity, vî,t ∈ R is the noise direction, cd and vî,t is the user-defined control direction. We An agent avoids collisions with other agents that further use St = ∪isi,t to represent the motion are too close to it. Based on the zonal interaction state of the whole swarm at t. Then, the motion model, which dictates that an insect has a short- dynamics of agent i is formulated as: range repulsion within a circular distance-based zone [12], we describe the repulsion zone using vi,t+1 = arg min E(i, v, a, St), a spherical kernel function, centered at the agent vi,t+1∈dv∈D with a predefined cut-off radius d (see Fig- p = p + v ∆t, rep i,t+1 i,t i,t+1 (1) ure 1 for the 2D representation), and an agent vˆn = f (s ), i,t+1 N i,t will avoid collisions with its neighboring agents vˆcd = f (s ,ENV ), i,t+1 CD i,t t within the repulsion zone. Assuming that agent i will move in a timestep using velocity v in where the velocity v ∈ d ∈ D minimizes i,t+1 v the data term d and it’s repulsive neighbor j the objective function E, a is the acceleration v will hold its current velocity v in a timestep, in d , ∆t is a timestep, and p is the posi- j,t v i,t+1 then the distance-based repulsion energy E tion at t + 1. After updating the velocity and i,rep is calculated as follows: position of each agent, we update the noise direction vˆn and the user-defined control direc- 0 i,t+1 pi,t+1 = pi,t + v∆t, cd 0 tion vî,t+1 using fN and fCD, respectively. The pj,t+1 = pj,t + vj,t∆t, objective function E is defined as: Ei,rep =wrep· kp0 −p0 k ρ 1− i,t+1 j,t+1 2 1 X d E(i, v, a, St) = Ei,int + Ei,sp + Ei,n + Ei,user, e rep , |RN| (2) j∈RN where Ei,int is an interaction term to model (4) the interactions among agents, Ei,sp is an self- where wrep ≥ 0 is the weight of the repulsion propulsion term to control the motion of agents, energy, RN is the set of neighbor agents, |RN| Ei,n is an noise-induced control term to model is the number of the repulsive neighbors, ρ > 0 the randomness of agents’ movements, and is a constant used for the scaling of the energy, 0 0 and pi,t+1 and pj,t+1 are the predicted positions drive the agent to react to external stimuli: of agent i and its neighbor j. Ei,sp = Ei,vel + Ei,env. (8) 4.1.2 Attraction 4.2.1 Internal Drive Attraction exists between agents, and also between every agent and the center of the swarm We assume that the internal drive ensures the motion smoothness, which is formulated con- [12]. The attraction energy Ei,att therefore consists of both the attraction from the neighbors sidering the first-order and second-order deriva- and the attraction from the swarm center: tives of motions:

Ei,vel = Ei,dir + Ei,acc, (9) Ei,att = Ei,oa + Ei,cos, (5) where E = w · eρkvˆ−vî,tk2 is the direc- where E is the energy of attraction from the i,dir dir i,oa tional continuity energy, and w ≥ 0 is the neighbors, and E is the attraction energy dir i,cos weight. We also consider the continuity of its from the center of the swarm. As shown in acceleration, Figure 1, the attraction zone of an agent to its neighbors is also formulated as a spherical ker- Ei,acc = wacc · (Ei,adir + Ei,amag) , (10) nel function with two cut-off radii, one inner 0 datt,1 and one outer datt,2: The attraction energy where wacc ≥ 0 is the weight. We use a = from the other agents Ei,oa is defined as: v − vi,t to compute the predicted acceleration so that if a dv is selected, and a is the 0 kp −p k2 ρ i,t+1 j,t −1 corresponding acceleration in dv. In Equation 1 X datt,1 0 E = w · e , ρkaˆ −aˆk2 i,oa oa |AN| 10, Ei,adir = e is designed for the j∈AN continuity of the direction of acceleration, and (6) E = |ka0k − kak| is designed for the con- where w ≥ 0 is the weight of the energy term, i,amag oa tinuity of the magnitude. Overall, E min- AN is the set of the attraction neighbors of agent i,dir imizes the direction changes and E mini- i, and |AN| is the number of attraction neigh- i,acc mizes the acceleration changes compared to the bors. The attraction energy on agent i is normal- real data. The regularization on both motion ized by the total number of the neighbor agents. derivatives leads to smooth motions. For the attraction to the swarm center on agent i, we use vî,center, the direction that points from the agent’s current position to the swarm center, 4.2.2 Response to Environment as the desired direction of the chosen velocity Insects can react quickly to external stimuli, cos vˆ. The attraction from swarm center Ei is de- e.g. following a target, or escaping from an ap- fined as: proaching predator. To model one stimulus, the energy of response to it is defined as: ρkvˆ−vî,centerk2 Ei,cos = wcos · e , (7) ρΨ Ei,env = wenv · e , (11) where wcos ≥ 0 is the weight of the energy term. where wenv ≥ 0 is the weight, and the function 4.2 Self-Propulsion Ψ is defined as:  Besides being reactive, insects are also self-  kvˆ − vî,stik2, attracted,  0 propelled. We therefore define a self-propulsion Ψ = kpi,t+1 − psti,tk2  1 − , startled, energy term, not only to drive the agents to keep  ddanger moving similar to the reference trajectories, but (12) also to response to external stimuli in environ- where psti,t is the stimulus’ position, vî,sti is the ments. For agent i, the self-propulsion energy direction pointing from the agent’s current po- includes the internal propulsion term Ei,vel to sition to the stimulus. In addition, as Figure 1 drive the agent to generate new trajectories sim- shows, an agent is startled if a potential danger ilar to the data, and the reaction term Ei,env to is within a range defined by ddanger. 4.3 Noise As the movements of swarms show strong randomness, we introduce a noise-induced control term to generate plausible trajectories. We introduce the curl noise [29], which has been used in force-based models to generate more accurate simulation results [10, 11], to model the motion (a) 37 insects (b) 300 insects randomness. The function fN denotes the curl noise function for generating a noise direction Figure 2: Aggregation behavior of insect n vî,t. Similar to the calculation of the energy of swarms in different scales. direction continuity in Equation 9, the energy of noise-induced direction control is defined as:

n user-controlled behaviors, the former showing ρkvˆ−vˆ k2 Ei,n = wn · e i,t , (13) the high visual realism and the latter showing controllability. where wn ≥ 0 is the weight of the energy term.

4.4 User-Defined Motion Control 5.1.1 Collective Behaviors Besides the behavioral modeling of swarms, it is We use the reference dataset from [6] to generate aggregation mating escaping necessary to induce user-control for the purpose the , , and behaviors of animation. We model user-control as a direc- of flying insect swarms. The results show that tion control signal and introduce an energy term FASTSWARM is capable of simulating a variety to constrain an agent to follow predefined user- of natural behaviors with good visual quality. defined trajectories. For agent i, the user-control Aggregation. Our approach can generate the energy is calculated as: aggregation behavior of insect swarms in different scales (see Figure 2). Figure 2(a) and Figure cd ρ||vˆ−vˆ ||2 Ei,user = wuser · e i,t , (14) 2(b) respectively shows two swarms of different sizes in the aggregation scenario of the reference cd where wuser ≥ 0 is the weight and vî,t is the dataset with an unchanged swarm center. control direction of the agent for generating spe- Mating. Figure 3 shows the mating behavior cific trajectories. of insects generated by our approach. 100 male agents (the cyan ones) are attracted by a female 5 Results and Evaluations (the red one) insect and run after the female. Escaping. Our approach can also generate The implementation is done in C++ and the the escaping behavior of insect swarms. In Fig- experiments were run on a PC with an In- ure 4, 100 flying insects are startled by a sud- tel (R) Core (TM) i7 4.00 GHz CPU, 32 GB den obstacle, and try to escape from the ob- RAM, and an NVIDIA Geforce GTX 1060 stacle. During simulation, the obstacle will GPU. We provide both qualitative and quantita- pass through the swarm, and the insects in the tive evaluations to demonstrate the performance swarm escape from the danger and then aggre- of FASTSWARM. Due to the space limit, we gate again. only show representative results and refer the readers to the supplementary materials for more 5.1.2 User-Defined Shape Constraint details. In all our experiments, we set ρ = 2.5, and the weights of the energy terms for the test To evaluate the controllability, we ask the users scenarios are shown in Table 1. to draw different 3D shapes which are used as constraints. Figure 5 shows two shape- 5.1 Qualitative Results constrained examples to constrain the agents to move following different 3D curves. We regard We first show qualitative results. The results are the 3D curve as a bidirectional curve with sev- divided into two parts: natural behaviors and eral keypoints (Figure 1: User-Defined Control). Figure 6 shows the time performance of our method against different swarm scales. Theo- retically, the time complexity is O(kN) where N is the number of insects and k is data sample size in the reference data. In our experiments, k = 300 is enough to generate all behaviors. The linear time complexity guaran- tees the high-performance of FASTSWARM. In Figure 3: Mating Figure 4: Escaping practice, we further accelerate the computation behavior. behavior. in three ways:

• Fast indexing. We discretize the space Each agent is randomly initialized near a key into a 3D grid and index the neighbor- point with a random velocity selected from the hood region of every agent, and approxi- referenced dataset. Then, it is assigned a task mate the repulsion and attraction regions to traverse the keypoints. During simulation, we using cubes whose dimension is the dis- update the control direction vˆcd of agent i as the i,t tance thresholds in their respective energy direction from the agent’s position pi,t to the po- goal terms. During simulation, only local search sition of the goal keypoint pi,t : and computation is done.

V = pgoal − p , • Reduced search space. We sort the refer- i,goal i,t i,t (15) cd ˆ ence dataset by the speed and divide the vˆi,t = Vi,goal, dataset into several groups. To choose a ˆ where Vi,goal is the directional vector of Vi,goal. new velocity for an agent, our algorithm The goal keypoint will be updated to the next only traverses the corresponding group and keypoint when the agent reaches its current goal. the groups with similar speed. • Parallelization. As Equation 1 is computed for each agent and is therefore highly parallelizable. We also parallelize FASTSWARM to concurrently compute the updates for multiple agents.

(a) 500 ﬂying insects form (b) 500 ﬂying insects form a heart shape. a star shape.

Figure 5: Two shape-constrained ﬂocking examples. There are 11 key points in (a) and 10 key points in (b). Figure 6: Time performance of our approach. The computation cost is linear w.r.t the number of insects 5.2 Quantitative Evaluations 5.2.1 Time Performance 5.2.2 Comparisons To quantitatively test the scalability, we evaluate FASTSWARM with an increasing number We compare our method with the method in of insects in an aggregation scenario, similar [10] and the ground-truth under different met- to Section 5.1.1. Table 2 shows the time per- rics. Here, we take the real tracklets [8] as the formance of the test scenarios in Section 5.1. ground-truth. In both comparisons, the scenes Scenario wrep woa wcos wdir wacc wenv wn wuser Aggregation 1 1 0.05 1 1 0 0.2 0 (N = 37 / 300) Mating 1 1 0.05 1 1 1 0.5 0 Aggregation 1 0.2 Escaping 1 1 1 2 0.3 0 In danger 0 0 Shape Constraint 1 0.5 0 1 1 0 1 1 (Heart / Star)

Table 1: The weights for our implementation in Section 5.1.

Scenario N Time (s/f) ter in controlling the density of the swarm to be 37 0.0003 Aggregation similar to the reference dataset. 300 0.0025 Mating 100 0.0159 5.2.3 Trajectory Synthesis Escaping 100 0.0107 Shape Constraint 500 0.0037 Given only trackelets (instead of whole trajec- (Heart / Star) tories) are avaiable, FASTSWARM provides a possible avenue to recover them for animation Table 2: Time performance of the simulation re- applications. Although it is difficult to exactly sults shown in Section 5.1 recover the trajectories or even compute their numerical accuracy, due to the lack of ground- are the same aggregation scenario in a box as truth data, our recovered trajectories can achieve the ground-truth [8], and the swarms are initial- global visual similarities. As shown in Figure 8, ized by randomly selecting one frame from the we extract the trajectory segments of 37 agents real data. Since directly comparing individual from the real-world dataset of [6] (Figure 8(a)). trajectories is not possible, the evaluation met- By taking these trajectory segments as the ini- rics include distributions of density, velocity and tial trajectories of a swarm with 37 agents, our acceleration, as they capture both the state and method can be used to predict the subsequent the motion dynamics. Similar to previous work trajectories (Figure 8(c)). [13], we use minimum-distance to describe the Our method can also synthesize plausible in- density. Results are shown in Figure 7. The sect swarm animations by mixing the trajecto- distributions of velocity, minimal distance and ries from the real-world dataset with the syn- acceleration of FASTSWARM are much closer thetic trajectories. As Figure 8(d) shows, in to those in the ground-truth than [10]. This is the real-world trajectories in 8(b), we add 100 true for both motion dynamics (velocity in Fig- agents that are simulated by FASTSWARM. ure 7(a) and acceleration in Figure 7(c-d)) and states (density in Figure 7(b)). The better mo- 6 Conclusion tion dynamics shows that FASTSWARM is su- perior in capturing realistic behaviors. We have presented a general and scalable data- For comparing the density, in [10], the density driven optimization framework to simulate fly- of the swarm is controlled simply by a distance- ing insect swarms in real time. Our optimiza- based attraction force, and the homogeneous tion method is capable of generating natural col- distance control results in the relatively uniform lective behaviors of flying insects by utilizing density of the swarm as the agents aggregate a motion characteristic dataset extracted from around the attraction boundary. Addressing this real data. The generated animations are plau- issue, we control the aggregation of agents by sible and have high visual realism. We have combining the distance-based attraction among validated our approach using extensive experi- agents with a direction control that drives the ments through qualitative and quantitative anal- agents to move towards the swarm center, and ysis. Our method also provides a means to gen- Figure 7(b) shows that our method performs bet- erate interesting shape-constrained swarm be- 0.40 0.14 0.7 0.6 0.8 Real Data 0.35 0.12 0.6 0.7 Our Method 0.5 0.30 0.10 0.5 0.6 [Ren et al. 2016] 0.4 0.25 0.5 0.08 0.4 0.20 0.3 0.4 0.06 0.3 0.15 0.3 0.2 Probability 0.10 0.04 0.2 0.2 0.1 0.05 0.02 0.1 0.1 0 0 0 0 0 0 500 1000 1500 2000 2500 0 50 100 150 200 250 300 -600 -400 -200 0 200 400 600 -2000 -1000 0 1000 2000 3000 -600 -400 -200 0 200 400 Velocity (pixel/s) Distance (pixel) Acceleration - x (pixel/s2) Acceleration - y (pixel/s2) Acceleration - z (pixel/s2) (a) (b) (c) (d) (e)

Figure 7: Comparisons of the distributions of the velocity (a), minimum distance (b), and acceleration (c-e for x, y and z direction) between our method, the real data, and [10].

tured trajectories from real world or created by animators. We are also planning to employ the Long Short-Term Memory (LSTM) networks to explore the latent representations of trajectories to further enhance our results. (a) (b) References

[1] Michael Sinhuber and Nicholas T Ouellette. Phase coexistence in insect swarms. Physical review letters, 119(17):178003, 2017. (c) (d) [2] Christian Blum and Xiaodong Li. Swarm intel- Figure 8: Results of trajectory synthesis. (a) ligence in optimization. In Swarm intelligence, shows the captured intermittent trajec- pages 43–85. Springer, 2008. tories of 37 agents. (b) shows the [3] Jose´ Halloy, Gregory´ Sempo, Gilles Caprari, real trajectories of the following 500 Colette Rivault, Masoud Asadpour, Fabien frames where exist lots of intermittent Tache,ˆ Imen Sa¨ıd, Virginie Durier, Stephane segments. (c) shows our prediction Canonge, Jean Marc Ame,´ et al. Social inte- result of (a) by extending the initial gration of robots into groups of cockroaches to control self-organized choices. Science, trajectories to 500 frames. (d) shows 318(5853):1155–1158, 2007. the blending trajectories of 100 virtual agents with (b). [4] Xinjie Wang, Xiaogang Jin, Zhigang Deng, and Linling Zhou. Inherent noise-aware insect swarm simulation. Computer Graphics Forum, 33(6):51–62, 2014. haviors controlled by users, which benefits computer animation. Moreover, our method can be [5] Qiang Chen, Guoliang Luo, Yang Tong, Xi- used to predict the missing trajectories in the aogang Jin, and Zhigang Deng. Shape- constrained flying insects animation. Computer captured dataset and augment the dataset with Animation and Virtual Worlds, 30(3-4):e1902, different number of insects while maintaining 2019. the global visual similarity to the real data. [6] Douglas H. Kelley and Nicholas T. Ouel- Limitations and Future Work. Our method lette. Emergent dynamics of laboratory insect relies on the quality of the reference dataset. swarms. Scientific reports, 3:1073, 2013. However, as our reference dataset is extracted [7] James G Puckett, Douglas H Kelley, and from imperfect trajectories captured using com- Nicholas T Ouellette. Searching for effective puter vision techniques, which contain noises forces in laboratory insect swarms. Scientific and erroneous data, our result may replicate the reports, 4:4766, 2014. defects in the input data. In the future, we are [8] Hai Shan Wu, Qi Zhao, Danping Zou, and interested in exploring a more general frame- Yan Qiu Chen. Automated 3d trajectory mea- work so that it can simulate any types of mo- suring of large numbers of moving particles. tion (not restricted to insects) with limited cap- Optics express, 19(8):7646–7663, 2011. [9] Weizi Li, David Wolinski, Julien Pettre,´ and [19] Qianwen Chao, Jingjing Shen, and Xiaogang Ming C. Lin. Biologically-inspired visual sim- Jin. Video-based personalized traffic learning. ulation of insect swarms. Computer Graphics Graphical Models, 75(6):305–317, 2013. Forum, 34(2):425–434, 2015. [20] Qianwen Chao, Zhigang Deng, Jiaping Ren, [10] Xinjie Wang, Jiaping Ren, Xiaogang Jin, and Qianqian Ye, and Xiaogang Jin. Realistic data- Dinesh Manocha. Bswarm: biologically- driven traffic flow animation using texture syn- plausible dynamics model of insect swarms. thesis. IEEE Transactions on Visualization and In Proceedings of the 14th ACM SIG- Computer Graphics, 24(2):1167–1178, 2018. GRAPH/Eurographics Symposium on Com- puter Animation, pages 111–118, 2015. [21] Huikun Bi, Tianlu Mao, Zhaoqi Wang, and Zhi- gang Deng. A deep learning-based framework [11] Jiaping Ren, Xinjie Wang, Xiaogang Jin, and for intersectional traffic simulation and editing. Dinesh Manocha. Simulating flying insects us- IEEE Transactions on Visualization and Com- ing dynamics and data-driven noise modeling puter Graphics, 2019. to generate diverse collective behaviors. PloS one, 11(5):e0155698, 2016. [22] Yixuan Wang, Rui Wang, Kai Cui, Weim- ing Tian, Dongli Wu, and Shuqing Ma. In- [12] Ryan Lukeman, Yue-Xian Li, and Leah sect trajectory simulation method based on Edelstein-Keshet. Inferring individual radar observation. The Journal of Engineering, rules from collective behavior. Proceed- 2019(21):7365–7369, 2019. ings of the National Academy of Sciences, 107(28):12576–12580, 2010. [23] Jeff Jones, Gregory D Murray, and Philip J McCall. A minimal 3d model of mosquito [13] Jiaping Ren, Wei Xiang, Yangxi Xiao, Ruigang flight behavior around the human baited bed Yang, Dinesh Manocha, and Xiaogang Jin. net. bioRxiv, 2020. Heter-sim: Heterogeneous multi-agent systems simulation by interactive data-driven optimiza- [24] Alper Bozkurt, Edgar Lobaton, Mihail Sichi- tion. IEEE Transactions on Visualization and tiu, Tyson Hedrick, Tahmid Latif, Alireza Di- Computer Graphics, 2019. rafzoon, Eric Whitmire, Alexander Verderber, Juan Marin, and Hong Xiong. Biobotic in- [14] Eunjung Ju, Myung Geol Choi, Minji Park, Je- sect swarm based sensor networks for search hee Lee, Kang Hoon Lee, and Shigeo Taka- and rescue. In Signal Processing, Sen- hashi. Morphable crowds. ACM Transactions sor/Information Fusion, and Target Recogni- on Graphics (TOG), 29(6):1–10, 2010. tion XXIII, volume 9091, pages 498–503, 2014. [15] Yi Li, Marc Christie, Orianne Siret, Richard Kulpa, and Julien Pettre.´ Cloning crowd [25] Argel A Bandala, Elmer P Dadios, Ryan motions. In Proceedings of the ACM SIG- Rhay P Vicerra, and Laurence A Gan Lim. GRAPH/Eurographics Symposium on Com- Swarming algorithm for unmanned aerial ve- puter Animation, pages 201–210, 2012. hicle (uav) quadrotors–swarm behavior for aggregation, foraging, formation, and tracking. [16] Kang Hoon Lee, Myung Geol Choi, Qyoun Journal of Advanced Computational Intelli- Hong, and Jehee Lee. Group behavior from gence and Intelligent Informatics, 18(5):745– video: a data-driven approach to crowd sim- 751, 2014. ulation. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Com- [26] Yara Khaluf, Seppe Vanhee, and Pieter puter animation, pages 109–118, 2007. Simoens. Local ant system for allocating robot swarms to time-constrained tasks. Journal of [17] Mingbi Zhao, Stephen John Turner, and Wen- Computational Science, 31:33–44, 2019. tong Cai. A data-driven crowd simulation model based on clustering and classification. [27] Shihui Guo, Meili Wang, Gabriel Notman, Jian In 2013 IEEE/ACM 17th International Sympo- Chang, Jianjun Zhang, and Minghong Liao. sium on Distributed Simulation and Real Time Simulating collective transport of virtual ants. Applications, pages 125–134, 2013. Computer Animation and Virtual Worlds, 28(3- 4):e1779, 2017. [18] Panayiotis Charalambous and Yiorgos Chrysanthou. The pag crowd: A graph [28] Wei Xiang, Jiaping Ren, Kuan Wang, Zhigang based approach for efficient data-driven crowd Deng, and Xiaogang Jin. Biologically inspired simulation. Computer Graphics Forum, ant colony simulation. Computer Animation 33(8):95–108, 2014. and Virtual Worlds, 30(5):e1867, 2019. [29] Robert Bridson, Jim Houriham, and Marcus Nordenstam. Curl-noise for procedural fluid flow. ACM Transactions on Graphics (TOG), 26(3):46, 2007.