COMPUTER ANIMATION AND VIRTUAL WORLDS Comp. Anim. Virtual Worlds 2016; 27:262–270 Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/cav.1712 SPECIAL ISSUE PAPER Predictable behavior during contact simulation: a comparison of selected physics engines Se-Joon Chung1* and Nancy Pollard2 1 Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, USA 2 Robotics Institute, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, USA ABSTRACT Contact behaviors in physics simulations are important for real-time interactive applications, especially in virtual reality applications where user’s body parts are tracked and interact with the environment via contact. For these contact simula- tions, it is ideal to have small changes in initial condition yield predictable changes in the output. Predictable simulation is key for success in iterative learning processes as well, such as learning controllers for manipulations or locomotion tasks. Here, we present an extensive comparison of contact simulations using Bullet Physics, Dynamic Animation and Robotics Toolkit (DART), MuJoCo, and Open Dynamics Engine, with a focus on predictability of behavior. We first tune each engine to match an analytical solution as closely as possible and then compare the results for a more complex simulation. We found that in the commonly available physics engines, small changes in initial condition can sometimes induce differ- ent sequences of contact events to occur and ultimately lead to a vastly different result. Our results confirmed that parameter settings do matter a great deal and suggest that there may be a trade-off between accuracy and predictability. Copyright © 2016 John Wiley & Sons, Ltd. KEYWORDS predictability; physics engine comparison; physics simulation; virtual worlds *Correspondence Se-Joon Chung, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, USA. E-mail: [email protected] 1. INTRODUCTION aim to shed some light on [6–9]. These studies compare various physics engines with regard to their speed, accu- Physics simulations are becoming more important for racy, and stability. Our work differs from those works real-time interactive applications, especially for virtual primarily by focusing on predictability of the motion, in reality applications where user’s body parts are tracked and particular predictable variation in behavior over simulation interact with the environment. Thankfully, we have many tests having smoothly varying parameters. capable physics engines available for use such as Bullet In our comparisons, we tune individual engine’s param- Physics [1], Dynamic Animation and Robotics Toolkit (DART) [2], MuJoCo [3], and Open Dynamics Engine eters to best match the analytical solution for a simple (ODE) [4]. simulation. Then, we compare results for a more complex However, it is often difficult to decide which one will simulation where an analytical solution is hard to derive. In be best for one’s use case with so many choices of physics this way, we aim to compare all engines under the settings engines. A recent survey in the robotics community shows that produce the most accurate performance, because one that even robotics researchers, who depend on physics sim- engine’s optimal settings may not be suitable for another. ulations all the time, are divided among their choice of In terms of predictability and smooth behavior, we found primary physics engine [5]. The survey also shows that that while MuJoCo performed best in our first experiment, robotics researchers are constantly seeking for better solu- the same parameter setting caused it to not do well in our tions, with many researchers trying out other engines to test their performance and some researchers abandoning second experiment. Even though they did not produce sim- previously used engines in favor of others. ulations close to ground truth in the first experiment, ODE This difficulty in choosing which physics engine to use and DART produced the most predictable simulations in is precisely what many physics engine comparison studies our second experiment. 262 Copyright © 2016 John Wiley & Sons, Ltd. S.-J. Chung and N. Pollard Predictable behavior during contact simulation 2. RELATED WORKS to specific physics engines. In this light, Drumwright and Shell have evaluated different methods of modeling con- A number of researchers have performed physics engine tact in multibody simulation [11] and performance of linear comparisons, and this line of research is constantly evolv- complementary problem (LCP) solvers for rigid body con- ing. Seugling and Rölin compared Newton Dynamics, tact problems [12]. Our work is complementary to their ODE, and NovodeX (now PhysX) with the goal of finding works in that we consider each engine as a whole when a suitable physics engine to use in Virtools, a 3D author- evaluating contact behaviors. ing tool [6]. They compare and rank the three engines In order to derive an analytical solution to use as refer- under nine-benchmark testing accuracy of friction force, ence in our parameter tuning, we apply the rimless wheel accuracy of gyroscopic force, conservation of energy, con- model in McGeer’s work [13] to a cube that we use straint handling, and contact handling. Then, based on the throughout our comparisons. We then build up on their sum of rankings in each criterion, they concluded that model to derive the relationship between angular velocity NovodeX had the best overall performance. In another of the cube and its rolling behavior. comparison work by Boeing [7], more physics engines are compared with an additional focus on gaming technol- 3. OVERVIEW OF PHYSICS ENGINES ogy. They compared seven physics engines (PhysX, Bullet, JigLib, Newton Dynamics, ODE, Tokamak, and True Axis) We utilized a modified version of the physics simulation using integrator, material, constraint, collision, and stack- engine comparison framework used in the work of Erez ing tests. They concluded that no one engine performed et al. [10] for our comparisons. We made extensions to best at all tasks, but of the open-source engines, Bullet the framework to be able to apply active external forces. provided the best overall results for gaming. For simula- Unlike in the previous work, Havok and PhysX were not tion purposes, however, the most important property of the available to us, but we added DART, which is another pop- simulation should be determined first in order to select the ular physics engine. In the succeeding texts, we provide an best engine. overview of each engines included in our comparisons. Most of the physics engines compared in the afore- mentioned studies are still undergoing active develop- Bullet. Bullet Physics [1] is an open-source physics library ment and call for continued efforts in evaluating them. developed by Erwin Coumans. The original Bullet is In a more recent study, Hummel et al. compare Bullet, based on maximal coordinates and uses a sequential Newton Dynamics, Havok, ODE, and PhysX with focus impulse (SI) solver for constraint resolution, but the on high accuracy at interactive rates for assembly sim- recent version provides options to use various mixed lin- ulations [8]. They performed six benchmarks that test ear complementarity problem (MLCP) solvers [14]. We collision, constraint stability, interpenetration, and friction. explored using these MLCP solvers as well as the default They concluded there is no general physics engine that per- SI solver. We used Bullet Physics version 2.83.7, which forms best for any given task. Furthermore, they suggest was released on 8 January 2016. that physics engines optimized for games such as PhysX Bullet Multibody (Bullet MB). Bullet Physics includes a and Havok may provide less accurate simulations because generalized coordinate approach based on Feather- of the optimizations they make for stability and perfor- stone’s algorithm [15] called multibody since version mance. On the other hand, Bullet, Newton, and ODE were 2.82. We also include this multibody version of Bul- found to be good candidates for their assembly simulations. let in our comparisons. Currently, the multibody version In another study by Peters and Hsu, four physics engines only supports SI solver that is an optimized implementa- (Bullet, DART, ODE, and Simbody) included in Gazebo tion of projected Gauss–Seidel (PGS) method. The same robot simulator are compared under four benchmarks [9]. 2.83.7 version of Bullet is used. They found certain engines may perform worse on specific DART. DART [2] is an open-source physics library cre- benchmarks according to the contact model, joint damp- ated by the Georgia Tech Graphics Lab and Humanoid ing, or coordinate system. However, for the most complex Robotics Lab. Like Bullet MB, DART uses the general- robot walking simulation, all of the engines produced ized coordinate Featherstone approach. Although other similar trajectories. MLCP solvers seem to exist in DART, the default solver Some have taken a different approach to physics engine used in this version of DART was Dantzig LCP solver. comparison and focused more on benchmarks with com- Because DART did not provide an interface to switch plex models that can stress-test the engines. Erez et al. solvers at runtime, we used the default Dantzig LCP compared the self-consistency of Bullet, Havok, MuJoCo, solver. We used DART version 5.1.1, which was released ODE, and PhysX under identical simulation parameters on 6 November 2015. in order to measure their integration error [10]. They MuJoCo Euler. MuJoCo [3], which stands for Multi-Joint found MuJoCo performed best on robotics-related tests, dynamics with Contact, was developed by Emo Todorov which it was designed for, while gaming engines won the for Roboti LLC. MuJoCo is the only closed-source gaming-related tests without a clear leader among them. engine in our comparisons. MuJoCo is unique in the fact A different branch of works sought to compare the that it does not use an LCP solver to resolve constraints underlying contact algorithms directly without associations but formulates it as a convex optimization problem Comp.