Simulation Tools for Model-Based Robotics: Comparison of Bullet, Havok, MuJoCo, ODE and PhysX Tom Erez, Yuval Tassa and Emanuel Todorov. Abstract— There is growing need for software tools that can many of these engines are aimed at graphics and animation, accurately simulate the complex dynamics of modern robots. where it is often sufficient to achieve visually-plausible While a number of candidates exist, the field is fragmented. simulation, reducing the motivation to pursue the more It is difficult to select the best tool for a given project, or to predict how much effort will be needed and what the elusive goal of physically-accurate simulation. Simulating ultimate simulation performance will be. Here we introduce contact dynamics with a velocity-stepping method is in itself new quantitative measures of simulation performance, focusing problematic because it calls for solving NP-hard problems at on the numerical challenges that are typical for robotics as each simulation step. Consequently much recent effort in this opposed to multi-body dynamics and gaming. We then present area has focused on developing convex approximations that extensive simulation results, obtained within a new software framework for instantiating the same model in multiple engines yield similar contact behavior while being more tractable and running side-by-side comparisons. Overall we find that computationally [8], [9], [10], [11], further complicating the each engine performs best on the type of system it was designed question of physical accuracy. and optimized for: MuJoCo wins the robotics-related tests, The notion of a physics engine in multi-body dynamics while the gaming engines win the gaming-related tests without and gaming dates back to MathEngine, and indeed many of a clear leader among them. The simulations are illustrated in the accompanying movie. the engines used today can trace their origins back to it, in terms of methodology if not code. These engines use a I. INTRODUCTION Cartesian representation, maintaining 6 degrees of freedom As robots become more complex and dynamically- (DOF) for each body, and imposing joints as equality con- capable, scripted movements and hand-tuned servo con- straints in 6N-dimensional space where N is the number trollers are no longer sufficient to achieve high levels of of bodies. This approach is natural for systems with few performance, especially in uncertain environments. Instead or no joints constraints. However a robot with N links has there is growing need for model-based approaches in mul- configuration space whose dimensionality is much closer tiple areas of robotics. These include testing and validation to N that 6N. Thus the preferred approach in robotics is of candidate control strategies before executing them on the to work in generalized/joint coordinates – which is both robot; dynamically-consistent state estimation and model- more efficient computationally, and more accurate because predictive control which rely on faster-than-realtime inter- joint constraints cannot be violated. This is one reason why nal simulation; system identification which is by definition SD/FAST, which is actually older than MathEngine and its model-based; and emerging machine learning techniques that descendants, remains popular in robotics. It does not handle need large datasets of state transitions which are difficult to contact dynamics, and so a number of research groups have obtain from the physical system. developed their own contact solvers around the smooth multi- Despite this need for fast and accurate simulation and the joint simulation provided by SD/FAST. availability of raw computing power to make it possible, The inadequacy of the Cartesian approach to heavily- the existing simulation tools remain a limiting factor. Early constrained systems such as robots is now well recognized. work in robot dynamics gave rise to a range of efficient This is currently prompting a new wave of simulators that recursive algorithms [1], implemented perhaps most notably aim to combine the best of the earlier approaches: efficient in SD/FAST [2] as well as the MATLAB Robotics Tool- recursive algorithms in joint space, and modern velocity- box [3]. That phase focused on smooth multi-joint dynamics, stepping methods for contact dynamics. This category in- largely leaving contact dynamics for future work. Yet con- cludes MuJoCo [12] and DART [13] (formerly RTQL8), as tacts are the primary means by which robots interact with well as additions to PhysX [14], Bullet [15] and Havok [16] their environment. Accurate contact modeling and simulation that utilize some form of joint-space representations. ODE remains an active area of research. Spring-damper models of [17] is the only engine in our comparison that does not yet contact dynamics were replaced by impulse-based velocity- have such functionality. stepping methods [4], [5], [6], [7], different flavors of which Are any of these engines suitable for wide adoption lie at the core of most physics engines used today. However in robotics, not just for kinematics and visualization, but also for dynamics? A recent survey [18] found this field University of Washington, Departments of Applied Mathematics to be rather fragmented: while high-level visualization and and Computer Science & Engineering. fetom,[email protected], modeling packages such as Gazebo and V-Rep are reasonably [email protected] This work was supported by the NSF and DARPA. Thanks to Erwin popular, none of the physics engines surveyed stood out. Coumans, John Hsu and Kenneth Bodin for suggestions and criticism. The most extensive usability test in this regard was the DARPA Virtual Robotics Challenge (VRC), where a real- II. SOFTWARE FRAMEWORK time simulation of the Atlas robot was implemented in ODE and accessed by the research teams through Gazebo. While In order to compare different physics engines, we created the VRC was eventually a success, it is notable that for at a basic interface that has three functions: least 6 months out of the 9-month program the simulation ● Create a simulation instance in a specified state. was borderline usable. Grasping with a high-DOF hand ● Advance the simulation one time step forward, given an remained problematic even longer. And this was done by an input of joint torques. experienced and well-funded development team, using one ● Output the current state in a uniform format. of the more mature physics engines available, with extensive We implemented this interface using the API of the help from a number of robotics experts participating in the different engines; see Appendix for the engine-specific im- VRC. Hopefully the experience gained in the VRC will plementation details. We record the position (translation and transfer to other robots and tasks. Still, one has to wonder, rotation) and velocity (linear and angular) of every rigid how much effort is required for an individual research group body. The data is then used to compute features such as to develop a realistic dynamic simulation of their robot. momentum, energy, joint dislocation, joint limit violation and The main goal of this paper is to compare multiple physics contact penetration. The data is also used for visualization. engines on identical model systems and characterize their In order to allow models to be edited once and used in speed, accuracy and stability. We hope that our tests will all engines, we chose to specify the models in XML. Since help roboticists assess the performance they are likely to most of the engines we consider do not include a built- achieve if they were to invest time in developing a detailed in parser for any model definition markup languages, we simulation, and to choose the engine that best suits their took a different route: we parse the model description in needs. Such comparisons are very much needed, which was one engine, extract the resulting body-joint configuration, also one of the conclusions of another comparison study [18]. and use this information to create the simulation world in Earlier work along these lines [19] is becoming outdated every other engine, using the respective APIs to place bodies given the rapid development of physics simulation software. in their global position, attach them with properly-oriented More recent engine comparisons [20], [21] were done from constraints, and connect collision elements to the different the viewpoint of multi-body dynamics, and the tests there are bodies. We chose MuJoCo and its native MJCF file format mostly complementary to ours. These papers also describe to provide this service since it has a simple and compact data physics abstraction frameworks whose goals are similar to representation format. But a different format such as URDF our framework described in SectionII. The Open Source could have also been used to obtain identical results. Robotics Foundation (OSRF) is currently working on a In order to create a uniform playing field for the com- comparison of ODE, Bullet and DART – which are the parison, we identified a limited set of common features that engines integrated in Gazebo [22]. Accuracy tests of the are supported by all engines and are relevant for a robotics Vortex engine (but not other engines) are described in [23]. application. In particular, we restricted our models in the Even though all these tests are related, what distinguishes the following ways: present work is the emphasis on more challenging model Joints: systems that can stress-test the engines. In contrast, most Only hinge joints are allowed between parent and previous comparisons focused on simpler systems where child. This excludes other common joints such as slide, ball, analytical benchmarks are available. The rationale for our and universal joints. For engines that use joint coordinates, benchmarks is presented in Section III. we also allow a free joint that represents a floating body It should be noted that we are the developers of the (which is the default in Cartesian coordinates).