DOCSLIB.ORG

Scalable Immersive Audio for Virtual Environments

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Scalable Immersive Audio for Virtual Environments Scalable Immersive Audio for Virtual Environments By Mirza Beig A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Science University of Ontario Institute of Technology Supervisors: Dr. Bill Kapralos and Dr. Pejman Mirza-Babaei May 2018 Copyright © Mirza Beig, 2018 P a g e | 2 Abstract This thesis discusses the history of game sound and how it developed to become increasingly more immersive and realistic by accounting for the spatial aspects of sound. It then compares several 3D sound technologies before continuing on to present a custom spatial sound system that uses the highly parallel nature of modern graphics processing units (GPUs) to process large amounts of data for simulated sound propagation. In a real-time virtual reality game environment, the system is able to handle dynamic geometry and movable sound sources in full 3D at interactive rates. To test the implementation, I developed a VR-compatible video game for desktop and mobile platforms with an emphasis on play-by-sound gameplay mechanics. Results demonstrated low overhead on all devices tested when used with the appropriate pathfinding backend (CPU/GPU). Additional testing yielded that the GPU part of the system is capable of processing several thousand 3D paths at interactive rates. P a g e | 3 Acknowledgements I would like to gratefully acknowledge the support of both my parents who helped me not only all the way through my masters, but all throughout my life as well. I would also like to thank my exceptionally helpful supervisors, Dr. Bill Kapralos and Dr. Pejman Mirza-Babaei, who constantly gave me direct input and were available for advice when I needed it. I would not have gotten into this program and had the remarkable support that I did if it wasn’t also for additional professors such as Dr. Andrew Hogue, Dr. Lennart Nacke, Dr. Karen Collins, Dr. Richard Pazzi, and James Robb. I’d also like to thank the committee members Dr. Alvarro Joffre Uribe Quevedo and Dr. Loutfouz Zaman who gave me feedback after reviewing my thesis. I’d like to thank all my friends who made my journey more fun. A special shout-out goes to Naeem Moosajee, who advised me to get involved with doing my masters early on, as well as Albert Bushara, Lucas Lupinski, and Oliver Pollard. They all showed up when I was sick with yummy chocolates. That was pretty cool. Finally, the financial support of the Social Sciences and Humanities Research of Council of Canada (Insight grant led by Dr. Karen Collins) is gratefully acknowledged. P a g e | 4 Table of Contents Abstract ......................................................................................................................................................... 2 Acknowledgements ....................................................................................................................................... 3 Table of Contents .......................................................................................................................................... 4 List of Figures ................................................................................................................................................ 8 List of Tables ............................................................................................................................................... 10 List of Code.................................................................................................................................................. 11 Acronyms .................................................................................................................................................... 12 Chapter 1: Introduction .............................................................................................................................. 14 1.1 Overview ........................................................................................................................................... 14 1.2 Motivation ......................................................................................................................................... 14 1.3 Thesis Statement ............................................................................................................................... 16 1.4 Thesis Outline .................................................................................................................................... 17 1.5 Summary ........................................................................................................................................... 17 Chapter 2: Literature Review ...................................................................................................................... 19 2.1 Overview ........................................................................................................................................... 19 2.2 Video Games ..................................................................................................................................... 19 2.3 The Importance of Game State Feedback......................................................................................... 20 2.4 Localizing Sounds and Establishing Presence in Virtual Environments ............................................ 22 2.5 Sound Source Localization: Direction ............................................................................................... 24 2.5.1 Overview .................................................................................................................................... 24 2.5.2 Binaural Cues: ITD and ILD ......................................................................................................... 24 2.5.3 Monaural Cue: Spectral Cue / HRTFs ......................................................................................... 26 2.6 Sound Source Localization: Distance ................................................................................................ 27 2.7 Sound Source Localization: Acuity .................................................................................................... 28 2.8 Sound Source Localization: The Environment .................................................................................. 29 2.8.1 Reflections and Reverb .............................................................................................................. 29 2.8.2 Parametric Reverb ..................................................................................................................... 32 2.8.3 Convolution Reverb ................................................................................................................... 33 2.8.4 Occlusion, Obstruction, Transmission, and Diffraction ............................................................. 35 2.9 Acoustical Modelling ......................................................................................................................... 37 P a g e | 5 2.9.1 Overview .................................................................................................................................... 37 2.9.2 Distance, Velocity, and Directionality ........................................................................................ 38 2.9.3 Impulse Response ...................................................................................................................... 39 2.9.4 Wave-Based Acoustics ............................................................................................................... 40 2.9.5 Geometrical Acoustics-Based Acoustics .................................................................................... 42 2.9.6 Geometrical Acoustics: The Image-Source Method .................................................................. 42 2.9.7 Geometrical Acoustics: Ray Tracing ........................................................................................... 43 2.9.8 Geometrical Acoustics: Beam Tracing........................................................................................ 44 2.9.9 Geometrical Acoustics: Diffraction ............................................................................................ 44 2.10 Summary ......................................................................................................................................... 45 Chapter 3: The History of Sound in Video Games ....................................................................................... 47 3.1 Overview ........................................................................................................................................... 47 3.2 The Early Years .................................................................................................................................. 47 3.3 The First Generation: Silence, Bleeps, and Bloops ............................................................................ 48 3.4 The Second Generation: Programmable Game Sound and Synths .................................................. 48 3.5 Home Computers and the Fourth Generation: DSP and Stereo for Everyone ................................. 49 3.6 Enter the CD-ROM: Massive Storage = Quality Sampled Audio ....................................................... 50 3.7 First-Person Shooters: Play-by-Sound Mechanics ............................................................................ 50 3.8 Surround Sound and 3D Audio: Maximum Localization ..................................................................
Load more

Recommended publications
  • Audio Middleware the Essential Link from Studio to Game Design
    AUDIONEXT B Y A LEX A N D E R B R A NDON Audio Middleware The Essential Link From Studio to Game Design hen I first played games such as Pac Man and GameCODA. The same is true of Renderware native audio Asteroids in the early ’80s, I was fascinated. tools. One caveat: Criterion is now owned by Electronic W While others saw a cute, beeping box, I saw Arts. The Renderware site was last updated in 2005, and something to be torn open and explored. How could many developers are scrambling to Unreal 3 due to un- these games create sounds I’d never heard before? Back certainty of Renderware’s future. Pity, it’s a pretty good then, it was transistors, followed by simple, solid-state engine. sound generators programmed with individual memory Streaming is supported, though it is not revealed how registers, machine code and dumb terminals. Now, things it is supported on next-gen consoles. What is nice is you are more complex. We’re no longer at the mercy of 8-bit, can specify whether you want a sound streamed or not or handing a sound to a programmer, and saying, “Put within CAGE Producer. GameCODA also provides the it in.” Today, game audio engineers have just as much ability to create ducking/mixing groups within CAGE. In power to create an exciting soundscape as anyone at code, this can also be taken advantage of using virtual Skywalker Ranch. (Well, okay, maybe not Randy Thom, voice channels. but close, right?) Other than SoundMAX (an older audio engine by But just as a single-channel strip on a Neve or SSL once Analog Devices and Staccato), GameCODA was the first baffled me, sound-bank manipulation can baffle your audio engine I’ve seen that uses matrix technology to average recording engineer.
    [Show full text]
  • THINC: a Virtual and Remote Display Architecture for Desktop Computing and Mobile Devices
    THINC: A Virtual and Remote Display Architecture for Desktop Computing and Mobile Devices Ricardo A. Baratto Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2011 c 2011 Ricardo A. Baratto This work may be used in accordance with Creative Commons, Attribution-NonCommercial-NoDerivs License. For more information about that license, see http://creativecommons.org/licenses/by-nc-nd/3.0/. For other uses, please contact the author. ABSTRACT THINC: A Virtual and Remote Display Architecture for Desktop Computing and Mobile Devices Ricardo A. Baratto THINC is a new virtual and remote display architecture for desktop computing. It has been designed to address the limitations and performance shortcomings of existing remote display technology, and to provide a building block around which novel desktop architectures can be built. THINC is architected around the notion of a virtual display device driver, a software-only component that behaves like a traditional device driver, but instead of managing specific hardware, enables desktop input and output to be intercepted, manipulated, and redirected at will. On top of this architecture, THINC introduces a simple, low-level, device-independent representation of display changes, and a number of novel optimizations and techniques to perform efficient interception and redirection of display output. This dissertation presents the design and implementation of THINC. It also intro- duces a number of novel systems which build upon THINC's architecture to provide new and improved desktop computing services. The contributions of this dissertation are as follows: • A high performance remote display system for LAN and WAN environments.
    [Show full text]
  • THE 2003 Editionlinux
    SUBSCRIBE or renew your subscription to APC for your chance to WIN the new Alfa 156 JTS, valued at over $54,000 Only $65 for 12 issues THE 2003 edition linux POCKETBOOK Subscribe ... www.apcmag.com Online at magshop.com.au or Call 13 61 16 Authorised under NSW Permit No. L02/09075 VIC: 02/2531 SA: T02/3553 ACT: TP02/3650 NT: NT02/3286 For terms and conditions refer to www.xmas.magshop.au. Expiry date: 24/12/02 Contents CHAPTER 1 Customising Gnome 57 CHAPTER 6 Editorial INTRODUCTION 11 Exploring KDE 60 WORKING WITH WINDOWS 131 The origins of the Customising KDE 64 What about Windows? 132 Welcome back to The Linux Pocketbook 2003 edition! penguin 12 Windows connectivity 138 Many of you will probably remember the original print ver- CHAPTER 4 sions of The Linux Pocketbook on newsstands across the country. Why Linux? 18 Basic security 145 The original versions sold so well that we ran out of copies. We’ve The ways of the world 20 USING LINUX 67 had countless requests for reprints, so we’ve decided to bundle the Connecting to the Net 68 CHAPTER 7 entire book into this single resource. This version of the pocketbook relies heavily on Mandrake Linux 9.0 or Red Hat 8.0. Both were CHAPTER 2 Applications 71 PLAYING WITH LINUX 151 released late in 2002, and can be easily found for sale at www.everyth INSTALLING LINUX 21 Conjuring Linux 75 Linux multimedia 152 inglinux.com.au, or for download from either mandrakelinux.com or First published December 2000.
    [Show full text]
  • You Door Know Vvhor You've
    The Independent l _. NewsletterNewslefter for Ensoniq Users r T You Don'tDoor Know vvhorWhat You've In This Issue .. Got Till It's Gone: Got Till l’r's Gone: Articles:Articles. : The MRD-lMRD-101 a1 MR Disk from Ensoniq ZR-76ZR-76 Specifications Pat FFinniganinnigan Ensoniq ................................................................................................. ........... 3 ~14-f-1'1-I-' :1.‘-:1.’-:i":!.'-2117:‘. :» ~ ~ _ _ August. Mac owners need Stuffitt to unun-binhex-binhex NNAMMAMM News: Ensoniq Makes Keyboards! ~¢. the file; PC’ersPC'ers need WinZip or other unZIP Sam Mims ....................................................... ............................................. 5 app to decompress these filesfiles.. Go to w3.en-w3.en­ soniq.com,soniq.corn, click on the "Download"“Download” link, The X Sounds select the "Musical“Musical Instrument Files"Files” link, and Garth HHjeltejelte .............................................................................................. 6 select the "MR61171"“MR6l/71” link. Click on the 1 MRDMRD-101-IOI link and specify a folder to save it'it in. Once you'veyou’ve pulled it down, decodeldecode/ decompress it, copy to a 1.4 Mb floppy, stuffsniff ReviewsRGVIGWSZ: it into your MR-61176MR-61/76 and press that load buttonbutton... ... Ensoniq MRDMRD-101-IO 1 For: MR-6I,MR-61, MRMR-76,-76, MR Rack. Pat FFinniganinnigan .............................................................................. cover Product: MRDMRD--l101 0 I Sound & Sequence Disk. The Roses Price: FreeFree.. This is
    [Show full text]
  • Game Audio Via Openal
    Game Audio via OpenAL Summary In the Graphics For Games module, you learnt how to use OpenGL to create complex graphical scenes, improving your programming skills along the way, and learning about data structures such as scene graphs. In this workshop, you'll see how to add sounds to your game worlds using the OpenAL sound library. New Concepts Sound in games, OpenAL, PCM audio, binary file formats, FourCC codes, WAV files, limited resource management Introduction Audio has played an important part in gaming almost as long as there have been games to play - even Pong back in 1972 had simple sound effects. We've moved a long way from then; the 80s brought dedicated audio hardware such as the Commodore 64's SID chip that could play 3 simultaneous syn- thesised sounds, and later the Amiga brought the ability to play audio samples to the home gaming market. In modern gaming hardware, we can expect to hear many simultaneous sounds and music tracks, often in surround sound. Game developers now employ dedicated sound engineers that will carefully adjust the sounds in each game release to create an immersive aural experience - making sure that each individual sound is uniquely identifiable and correctly equalised, and that every music track suits the situation they will be played in. At the heart of a game's audio experience is the code that plays back the game sounds, and cal- culates which speakers they should use - the sound system. Although we can't hope to compete with the complex sound systems of AAA games, we should still be able to make a robust, simple system for the addition of sound in our 3D games, and that's what this workshop will assist you in creating.
    [Show full text]
  • A Software-Based Remote Receiver Solution
    Martin Ewing, AA6E 28 Wood Rd, Branford, CT 06520; [email protected] A Software-Based Remote Receiver Solution Need to get your receiver off site to avoid interference? Here is how one amateur connected a remote radio to a club station using a mixture of Linux, Windows, and Python. The local interference environment is an station computer would run control software details are available at www.aa6e.net/wiki/ increasing problem for ham radio operation, to manage the remote operation. W1HQ_remote. Hardware construction is making weak signal operation impossible This article focuses mostly on the straightforward for anyone with moderate at some times and frequencies. The ARRL software — how we developed a mixed C experience. Staff Club Station, W1HQ, has this problem and Python (and mixed Linux and Windows) in spades when W1AW, just across the project with audio and Internet aspects. System Overview parking lot, is transmitting bulletins and Interested readers will probably want to Figure 1 shows the overall system design. code practice on seven bands with high consult the code listings. These are provided The BeagleBoard XM or “BBXM” is an power. Operating W1HQ on HF in the prime at www.aa6e.net/wiki/rrx_code and are inexpensive (~$150) single board computer evening hours is not possible. There is no also available for download from the ARRL with a 1 GHz ARM processor, Ethernet problem transmitting in this environment QEX files website.1 Additional project and USB connections, and on-board audio — it’s a receiver problem. We could solve 2 1 capability. It has 512 MB of RAM and the problem by setting up a full remote base Notes appear on page 6.
    [Show full text]
  • Microsoft Directshow: a New Media Architecture
    TECHNICAL PAPER Microsoft Directshow: A New Media Architecture By Amit Chatterjee and Andrew Maltz The desktop revolution in production and post-production has dramatical- streaming. Other motivating factors are ly changed the way film and television programs are made, simultaneously the new hardware buses such as the reducing equipment costs and increasing operator eficiency. The enabling IEEE 1394 serial bus and Universal digital innovations by individual companies using standard computing serial bus (USB), which are designed with multimedia devices in mind and platforms has come at a price-these custom implementations and closed promise to enable broad new classes of solutions make sharing of media and hardware between applications difi- audio and video application programs. cult if not impossible. Microsoft s DirectShowTMStreaming Media To address these and other require- Architecture and Windows Driver Model provide the infrastructure for ments, Microsoft introduced Direct- today’s post-production applications and hardware to truly become inter- ShowTM, a next-generation media- operable. This paper describes the architecture, supporting technologies, streaming architecture for the and their application in post-production scenarios. Windows and Macintosh platforms. In development for two and a half years, Directshow was released in August he year 1989 marked a turning Additionally, every implementation 1996, primarily as an MPEG-1 play- Tpoint in post-production equip- had to fight with operating system back vehicle for Internet applications, ment design with the introduction of constraints and surprises, particularly although the infrastructure was desktop digital nonlinear editing sys- in the areas of internal stream synchro- designed with a wide range of applica- tems.
    [Show full text]
  • Windows Phone API Quickstart
    Windows Phone API QuickStart Fundamental Types and Threading and cont. cont. Wallet▲ Date / Time Synchronization .NET Microsoft.Phone.Maps.Controls Microsoft.Devices Map, MapLayer, MapOverlay, .NET Microsoft.Phone.Maps.Services Microsoft.Phone.Tasks Windows Runtime PhotoCamera, CameraButtons, CameraVideo- ♦♣ Windows Runtime + GeocodeQuery, ReverseGeocodeQuery, Route- AddWalletItem Windows.Foundation ♦ BrushExtensions Windows.System.Threading Microsoft.Phone Query Microsoft.Phone.Wallet DateTime, Uri ThreadPool, ThreadPoolTimer Microsoft.Phone.Tasks Wallet, Deal, WalletTransactionItem, WalletAgent ♦♣ ♦ PictureDecoder Windows.Foundation.Collections Windows.UI.Core MapsTask, MapsDirectionsTask, MapDownload- Microsoft.Phone.Tasks ▲ IIterable<T>, IVector <T>, IMap<TK, TV>, IVec- CoreDispatcher, CoreWindow, erTask Multitasking torView <T> MediaPlayerLauncher, CameraCaptureTask, ♦ Note: You can get the current dispatcher from PhotoChooserTask, ShareMediaTask, SaveRing- System.Device.Location Windows.Storage.Streams CoreWindow.GetForCurrentThread() GeoCoordinateWatcher .NET Buffer toneTask Microsoft.Xna.Framework.Audio Microsoft.Phone.BackgroundAudio .NET BackgroundAudioPlayer, AudioTrack, AudioPlay- Microphone, SoundEffect, DynamicSoundEffec- ▲ .NET System tInstance erAgent, AudioStreamingAgent ♦ + VoIP System WindowsRuntimeSystemExtensions Microsoft.Xna.Framework.Media Microsoft.Phone.BackgroundTransfer ■ Object, Byte, Char, Int32, Single, Double, String, System.Threading MediaLibrary, MediaPlayer, Song Windows Runtime BackgroundTransferService,
    [Show full text]
  • 5.1-Channel PCI Sound Card
    PSCPSC705705 5.1-Channel PCI Sound Card • Play all games in 5.1-channel surround sound, including EAX™ 2.0, A3D™ 1.0 and even ordinary stereo games! • Full compatablility with EAX™ 1.0, EAX™ 2.0 and A3D™ 1.0 games • Hear high-impact 3D sound from games, movies, music, and external sources using two, four, or six speakers. • 96 distinct 3D voices, 256 distinct DirectSound voices & 576 distinct synthesized Wavetable voices. • Included software: Sonic Foundry® SIREN Xpress™, Acid XPress™, QSound AudioPix™. 5.1-Channel PCI Sound Card PSCPSC705705 Arouse your senses. Make your games come to life with the excite- Technical Specifications ment of full-blown home cinema. Seismic Edge supports the latest multi-channel audio games and DVD movies, and can transform stereo Digital Acceleration sources into deep-immersion 5.1-channel surround sound. (5.1 refers • 96 streams of 3D audio acceleration including reverb, obstruction, and occlusion to five main speakers – front left, right and center, and rear left and • 256 streams of DirectSound accelerations and digital mixing right – and one bass subwoofer.) All this without straining your com- • Full-duplex, 48khz digital recording and playback puter’s resources, because complex audio demands are handled on- • 64 hardware sample rate conversion channels up to 48khz board by Seismic Edge’s powerful computing chip.You’ve never experi- • Wavetable and FM Synthesis enced games like this before! • DirectInput devices Experience state-of-the-art 360º Surround Sound. An embed- Comprehensive Connectivity ded, patented QSound algorithm extracts complex and distinct 5.1- • 5.1-channel (6 channel) analog output channel information from stereo or ProLogic sources.
    [Show full text]
  • Soundfont Player™ 1.0 Operation Manual
    SoundFont Player™ 1.0 Operation Manual E-MU World Headquarters E-MU / ENSONIQ P.O. Box 660015 Scotts Valley, CA 95067-0015 Telephone: (+1) 831-438-1921 Fax: (+1) 831-438-8612 www.soundfont.com www.emu.com SoundFont Player™ 1.0 Operation Manual E-MU World Headquarters E-MU / ENSONIQ P.O. Box 660015 Scotts Valley, CA 95067-0015 Telephone: (+1) 831-438-1921 Fax: (+1) 831-438-8612 Internet: www.soundfont.com www.emu.com SoundFont Player Operation Manual Page 1 This manual is © 2001 E-MU / ENSONIQ. All Rights Reserved Legal Information The following are worldwide trademarks, owned or exclusively licensed by E-mu Systems, Inc, dba E-MU / ENSONIQ, registered in the United States of America as indicated by ®, and in various other countries of the world: E-mu®, E-mu Systems®, the E-mu logo, Ensoniq®, the Ensoniq logo, the E-MU / ENSONIQ logo, Orbit The Dance Planet, Planet Phatt The Swing System, Proteus®, SoundFont®, the SoundFont logo, SoundFont Player,. Sound Blaster and Creative are registered trademarks of Creative Technology Ltd. Audigy, Environmental Audio, the Environmental Audio logo, and Environmental Audio Extensions are trademarks of Creative Technology Ltd. in the United States and/or other countries. Windows is a trademark of Microsoft Corporation in the United States and/or other countries. All other brand and product names are trademarks or registered trademarks of their respective holders. SoundFont Player Operation Manual Page 2 Table of Introduction ...................................................................................6
    [Show full text]
  • Creative® Sound Blaster® Audiopci TM
    ® ® TM Creative Sound Blaster AudioPCI 128 MODEL CT5801 Creative’s Industry-Standard 1373-Based Sound Card Delivers High Quality Digital Audio The model CT5801 Sound Blaster® AudioPCITM 128 for Hewlett-Packard, is an outstanding sound card solution that delivers excellent audio quality and features at an affordable price. The Sound Blaster AudioPCI 128 is driven by Creative’s industry-standard ES1373 DSP engine, coupled with AC97 version 2.1 CODEC, providing the next generation of audio performance while maintaining full Sound Blaster 16 legacy compatibility. Audio - Primary Features The Sound Blaster AudioPCI 128 is a complete digital audio recording and playback system, capable of 16- bit/48KHz fidelity. The Sound Blaster AudioPCI 128 delivers 128-voice polyphony for wavetable audio, real- time DSP effects including reverb, chorus and spatialization, plus real-time bass and treble equalization control. In addition, the Sound Blaster AudioPCI 128 supports 3D Positional Audio, Microsoft’s DirectSound, DirectSound 3D, Environmental Audio Extensions (EAX) for DirectSound3D, and support for the Aureal A3D API, which allows applications written to the A3D API to run on the Sound Blaster AudioPCI 128. A stereo 2w/channel amplifier is also included. Compatibility The Sound Blaster AudioPCI 128 is fully Plug and Play compliant for ease of use in either genuine DOS, DOS Box, Windows Millenium, Windows 95/98/98SE, Windows NT, and Windows 2000 applications, and is fully Sound Blaster 16 compatible in MS-DOS utilizing a patented method of Sound Blaster emulation developed by Creative. In addition, the AudioPCI 128 offers full General MIDI capability, and also supports Microsoft’s PC98, PC99, and Multimedia PC Level II and III specifications.
    [Show full text]
  • Linux Games Page 1 of 7
    Linux Games Page 1 of 7 Linux Games INTRODUCTION such as the number of players and the size of the map, then you start the game. Once the game is running clients may Hello. My name is Andrew Howlett. I've been using Linux join the game. Clients connect to the game using TCP/IP, since 1997. In 2000 I cutover to Linux for all my projects, so it is very easy to play multi-player games over the except I dual-booted Windows to play games. I like to play Internet. Like many Free games, clients are available for computer games. About a year ago I stopped dual booting. many platforms, including Windows, Amiga and Now I play computer games under Linux. The games I Macintosh. So there are lots of players out there. If you play can be divided into four groups: Free Games, native don't want to play against other humans, then Freeciv linux commercial games, Windows Emulated games, and includes some nasty AIs. Win4Lin enabled games. This presentation will demonstrate games from each of these four groups. BZFlag Platform BZFlag is a tank combat game along the same lines as the old BattleZone game. Like FreeCiv, BZFlag uses a client/ Before I get started, a little bit about my setup so you can server architecture over TCP/IP networks. Unlike FreeCiv, relate this to whatever you are running. This is a P3 900 the game contains no AIs – you must play this game MHz machine. It has a Crystal Sound 4600 sound card and against other humans (? entities ?) over the Internet.
    [Show full text]