Making Games Sound as Good as They Look Real-time Geometric Acoustics on the GPU Sound in Modern Games • Commercial middleware (FMod, Wwise) • Hardware or software rendering (DirectSound 3D, OpenAL, EAX) • Some additional extensions to improve immersion – ex: calculate occlusion (per source) and filter if occluded • Existing systems are parametric, rather than simulations Parametric vs. Simulation • Parametric model: parameters are derived from game state (reverb wet/dry, room size, source distance, direction, etc..) • Parameters then map to DSP variables which interact with raw source (recorded samples) • Game state never directly interacts with signal processing Parametric vs. Simulation • Simulation model: game state directly generates audio • Advantages: audio experience is more directly influenced by game world, can be more immersive if done right • Disadvantages: can easily sound “broken” if model is used outside limitations - not simple to fix by ‘fudging’ parameters S.T.A.L.K.E.R.™ : Call of Pripyat • Basic positional audio (rendered through OpenAL) • Almost no environment dependence • Limited immersion, high performance • Typical for games where focus is on graphics S.T.A.L.K.E.R.™ : Call of Pripyat ARMA 2: Operation Arrowhead • 2010 (PC only), continuously updated (version from late- 2012) • Simulation focus, HRTF and “software” occlusion • Rendering through OpenAL (although OpenAL cannot process geometry) • Focuses on sound as a gameplay mechanic ARMA 2: Operation Arrowhead Quake 3 Implementation Q3dm1 • Acoustic version shown (3186 triangles) • 4096 rays to generate reflections • 3 orders of reflection - specular & diffuse; HRTF per-reflection (12k) • IIR based material descriptions Example: Used in Game Engine (Quake 3 Arena) For video: https://www.youtube.com/watch?v=TXUTgEmnD6U (please use headphones if possible!!) Geometry Engine “Backwards” Ray-tracing • Start at listener • Use specular and diffuse reflection approximation at each bounce • Generate impulse response • 1 ray per thread Geometry Engine • Finding exact paths vs. sampling sound field • Impulse response acts as filter • Each ray has different frequency characteristics due to HRTF (different incident direction) • Unique among real-time systems Example: Corner Bass Reinforcement • Situation on right, both listeners are facing source • Listener 1 is in around center of room, frequency response is fairly flat • Listener 2 is in corner, frequency response has low frequencies boosted • Why is this? Example: Corner Bass Reinforcement Example: Corner Bass Reinforcement Example: Corner Bass Reinforcement Performance • Computationally expensive (scales to Order*Sources*Triangles*Rays) • Triangles count can be fairly low (use collision mesh instead of displayed mesh) • Single order BVH (bounding volume “hierarchy”) is sufficient Geometry Engine Geometry Engine Geometry Engine Optimizations (Geometry) • Rather not tolerate linear scaling to sources (reasoning: everything else can be mitigated in design by reducing quality) • Solution: Dynamically allocate rays to sources • Psychoacoustic justification: more chaotic sound scene = less ability to discern individual sounds Dynamic Ray Allocation • Dynamic ray allocation: • First problem: amplitude changes as rays are re- assigned • Second problem: slower ray-tracing (warp divergence between bounding volumes) Ray Sorting • Re-sort rays by assigned source in each block (inspired by Garanzha & Loop, 2010) • Advantage: Gain spatial coherence when performing occlusion testing • Disadvantage: Ray source positions and directions are less coherent (remember each source still needs omnidirectional sampling of rays) GPU Audio Processing • Each ray generates audio stream for mix-down (4096 in example) • Delayed and filtered due to reflected materials and HRTF position • Parallel mix-down (optimal performance depends on architecture) Behavior at Reflection • Typical commercial software uses per-frequency band simulation • Reflection (1.0-absorption) coefficient is a scalar for each band (may use separate sets of coefficients for diffuse reflections) • Approximate multi-band simulation by representing reflection behavior as bi-quad filter (loss of some degrees of freedom) GPU Mix-down • Atomic operations on Kepler GPUs (easy) • Round-robin method on pre-Kepler GPUs (similar to parallel reduction) • Only part of the work is done on GPU (CPU needed to mix down shared memory regions) HRTF and Diffraction • At low frequencies (below 1KHz) HRTF essentially flat • These are the frequencies which diffract the most around architecture (wavelength @ 250Hz ~ 1.3m) • Precise positioning of diffracted sources is not very important • HRTF IIR filter coefficients derived from FIR experiments (can use Prony’s method, but easier just to match approximate amplitude and cutoff) Integrating Artificial Reverb • Simulation is useful for first 3 or 4 orders (for reference, on GTX Titan using q3dm1 map, 3 orders simulation takes ~30% real-time performance) • 3 orders generates about 500ms of reverb time, but in practice, T60 for a comparable room can be several seconds (proportional to volume and inversely proportional to absorption) • Additional problem in that simulated reverberation gets grainier as reverb time gets longer Integrating Artificial Reverb • Can estimate artificial reverb length with mean free path • Intuitive to apply reverb at end of DSP chain, but problematic in some situations • Resolve by applying reverb at front of DSP chain (before HRTF filtering) Design Considerations… • On a parametric system, easy to have creative input • Make the sound more ‘exciting’ or more ‘chaotic’ • We don’t want the sound designer to become level designer • Simulation may only be suitable for certain types of games (realistic, rather than cinematic) Thanks for Listening! • For more information, look for dissertation entitled: Design of a Real-Time GPU Accelerated Acoustic Simulation Engine for Interactive Applications (University of Illinois Press, 2014) .