Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
August 28, 2014
arrowPress Releases
August 28, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
6 Ways 3D Audio Can Expand Gaming Experiences
by Michel Henein on 11/01/13 06:24:00 am   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

“Sound is that which does not simply add to, but multiplies the effect of the image.”  Akira Kurosawa. 

Intro:

Over the past few years, we’ve witnessed sound for cinema being transformed from 5.1/7.1 to the high channel-count and object-based sound formats available for today's film-makers to extend 'surround sound' to the next level of immersion for audiences (i.e. Dolby ATMOS, DTS Multi-Dimensional Audio, and Auro Technologies’ Auro 3D). These formats add speakers all around, including above the listener to emit 'elevation cues', allowing sound to pan around in all three dimensions.

Games can also use elevation cues to further expand audio in all three dimensions (without the need for additional speakers by using stereo headphones, for example); through the use of digital filters (HRTFs derived from dummy-head measurements or modeling of the auditory cortex), a convincing 3D 'effect' can be created by leveraging  a sound object's XYZ position in game space to simulate a sound being heard at a corresponding point in a 3D sound-field.  

Yes, 3D audio technologies and solutions have been around for quite some time but the majority of today's game developers do not implement true 3D positional audio into their games; instead developers tend to stick with standard stereo and 5.1 surround sound as the de facto standard for the vast majority of titles being produced today. Keep in mind that not all 3D audio solutions are alike and really compelling 3D audio typically requires quite a lot of processing resources — precious resources that aren't always made available for audio processing. With the explosion of multi-core processors for mobile gaming, powerful next-gen consoles arriving later this year, and dedicated audio DSP resources being offered on new GPUs (i.e. AMD's TrueAudio), the conditions are ripe for the mass adoption and standardized use of compelling and powerful 3D audio solutions by game developers for their game titles across many different platforms.

Here is what 3D sound can provide game developers:

Elevation: placement above and to some degree, below

Azimuth: placement in front, behind, and to the sides

Distance: using elevation and azimuth to define a position for a sound inside a 3D sound-field (i.e. a spherical sound-field wrapped around you) the sound can be pushed out in space with distance to allow improved depth perception (using distance cue processing, 3D room simulation, for example.)

These parameters expand the sound-field beyond what stereo, 5.1, and 7.1 can produce (which are 2D formats). For developers looking to offer a bit more realism and immersion for their players, 3D audio may be the 'lowest-hanging fruit' available to do so.

Here are 6 ways developers can use 3D audio to expand gaming experiences:

  1. Enhanced immersion for mobile games:

There are tons of fun games on the small screen (for your mobile phone or tablet) however, 3D audio can be used to create the illusion of a larger game world through the use of an immersive sound-field that envelops the player. While there may be a bit of a disconnect between a small screen and an enveloping, spherical sound-field, 3d audio for mobile gaming can heighten the feeling of being "in the game."

Ear Monsters, by Ear Games, is a forward-thinking iOS game that employs the use of 3D audio to drive gameplay, rather than visuals.  The use of 3D auditory cues to drive gameplay serves to extend the boundaries of the game world beyond the small screen of the mobile device, for example, the player can tap in the general direction of the sound being heard in 3d space (for example, tapping the top of the screen if an attack is heard coming from above.)

  1. Sound-centric games:

Using 3D audio, sound-centric games can:

  • Help the visually impaired to enjoy gaming. Ear Monsters allows players to play without needing to see what’s happening on screen.
  • Create interactive stories using voice over and sound effects positioned in a 3D sound-field for an immersive story-telling experience.
  • Use darkness and lack of visibility to increase reliance on 3D sound cues can make more compelling and immersive horror games.

This list barely scratches the surface with all the possibilities that sound-centric games can offer players.

  1. Improve situational awareness in 3D games (i.e. 3D FPS):

3D audio can improve situational awareness in FPS games by relaying sound cues from certain directions correctly to the player. For example, in most modern combat FPS games that rely use stereo, 5.1, or 7.1 audio delivery, when a player on the ground level is being shot at by a sniper who's perched high in a tower, the sound of the sniper’s shot does is not heard from above. With 3D audio, the sound of the sniper’s shot would be heard from above, like it's supposed to. 

  1. Reduction of UI graphics:

It is well known that using sound to substitute for UI (i.e. running out of ammo or player health indication using sound prompts) can help remove display clutter. 3D sound can be used to extend the sound cues by using space to indicate more information (i.e. using a sound cue with elevation to indicate a rainstorm is coming.)

  1. Positional 3D audio for multiplayer voice chat:

By leveraging 3D audio, a '3D radio communications' of sorts can be created to hear team member communication based on actual position in the game (i.e. if a member of a squad is up on a hill, the radio communication from that player would be elevated.)

  1. 3D audio compliments the VR experience:

Conventional stereo, 5.1, or 7.1 audio playback limits the sound field to a two-dimensional plane creating a disconnect with the visual field offered by VR. 3D audio eliminates this problem by allowing a full 3D sound-field to perfectly compliment the 3D stereoscopic visual field; head-tracking (with the Oculus Rift, for example) coupled with 3D audio allows the player to move their head around and expect to hear sound all around them correctly, including sounds from above.

How can developers implement 3D audio in their games?

Games typically use audio middleware solutions (FMOD and Wwise, for example) for their run-time sound engines so developers can utilize 3D audio technologies made available from Dolby, DTS, GenAudio, Auro 3D, and Iosono (check with your audio middleware provider for available solutions.) Encourage your teams to explore using either of the various 3D audio solutions available today to enhance the aural experience for your title(s) beyond what stereo, 5.1, and 7.1 can offer. 

 


Related Jobs

Cloud Imperium Games
Cloud Imperium Games — SANTA MONICA, California, United States
[08.27.14]

Senior Producer
Yoh
Yoh — Vancouver, British Columbia, Canada
[08.27.14]

Rendering Engineer Job
Yoh
Yoh — Vancouver, British Columbia, Canada
[08.27.14]

Multiplayer Designer Job
Raven Software / Activision
Raven Software / Activision — Madison, Wisconsin, United States
[08.27.14]

Network Engineer






Comments


Robert Allen
profile image
Very nice post! I've been looking at 3D audio and how it is much more effective at immersing players than other sound formats (stereo, 5.1 etc). A reason that I think that it hasn't been nearly as embraced as it could be is that the best results of using 3D sound processing come from complementing it with binaurally captured audio, which would limit players to using headphones for the full effect (unless they have a crazy speaker set up!). Whichever sounds were to be recorded binaurally would have to be captured again with more 'usual' mic techniques for players not wishing to use headphones. I do wish that 3D audio was used a lot more!

David Fisk
profile image
I'm sorry, but I think the idea of 3D sound is a waste of time. People have to want to spend the money on a good sound system to support it, and I just don't see that happening. For mobile apps? That's just plain stupid.


...and I'm a sound designer.

Michel Henein
profile image
(Sorry, I forgot to click 'reply' to David's comment): David, I believe what your saying is that using a 3D sound implementation that requires many speakers for an immersive audio experience would be impractical for games (I agree with this) however, some filter-based 3D audio solutions (which can produce elevation cues, for example) only require a set of stereo headphones or speakers for some decent 3D effects. It may not be as immersive, as say a theater with a 64 speaker Dolby ATMOS system, but as far as I'm concerned, it's better than just standard stereo, 5.1, etc. :) For mobile gaming, 3D audio can be a great value add by allowing developers to use, for example, spatial auditory cues as a game mechanic (like the Ear Monsters game I mentioned in my blog.)

Nathan Mates
profile image
I implemented DirectSound3D (version 8) in games, then Microsoft knifed DirectSound3D in the back starting with Vista. Not everyone has the budget to pick up (or rewrite for) middleware just to work around Microsoft's limitations.

I assume that OpenAL is effectively dead, with its last published version in 2005. Do you have a list of middlewares that are free to use that work with Windows? Indie licenses for FMOD/WWise probably don't cut it because I'm working (in my spare time, for free) on patching Battlezone II, which came out in 1999.

James McDermott
profile image
Your first paragraph makes a good point, Nathan. As you said, not everyone's going to have the budget required to rewrite middleware to work around Microsoft's limitations. I would add this is likely to be true of adapting middleware to XAudio 2, the audio API which, as far as computer OSes go, Windows Vista, Windows 7, and possibly Windows 8 and 8.1 use. With all the hours spent changing code to work with the new middleware, training on how the middleware works in terms of audio design, debugging, testing...all this combined with licencing fees and potentially more (debugging, approval, etc.) could make it not worth it.

Of course, you may not have a choice given that DirectSound3D is depreciated and, in my experience of playing games using DirectSound3D on modern Windows OSes, can cause problems ranging from minor, like not being able to use hardware-accelerated sound for better effects, to moderate, like choppy sound or occasionally-inaccurate sound reproduction, to major, like the game not running at all or causing a BSOD (Blue Screen of Death).

To address your second paragraph, OpenAL, as far as Creative's support of it is concerned, seems to be all but dead. Thankfully, there is an ongoing, multi-platform, and open-source project to improve upon the final revision of OpenAL's open-source code called OpenAL Soft. It supports many EAX features, like air absorption, occlusion, and environmental reverb, runs on Windows, Mac, and Linux (not sure about mobile OSes), and runs all supported features directly through software - meaning no dedicated sound card is required. You can find out more at http://kcat.strangesoft.net/openal.html

Also, as far as other free-to-use sound middlewares which can be used on Windows, I'm, unfortunately, not familiar with any.

David Richardson
profile image
For anyone interested in HTML5, the Web Audio API includes HRTF. If you're already using it for sound, 3D audio becomes trivial implement.

Support is still spotty, but is improving.

Gregg Wilkes
profile image
Gregg Wilkes - Further to developments in 3D Audio for gaming...

I am working with VisiSonics, a spinout using technology based on research from the University of Maryland Computer Science department. The research there focused on capturing and reproducing over headphones the full panoramic sound environment of a street corner, so that newly blinded soldiers could learn to trust their hearing in a lab environment before moving to the world outside. This research resulted in unique IP related to the development of patented Spherical Array capture devices, development of HRTF models and individualizations. Continuous development and improvements have allowed us to bring to market an array of 3D sound offerings such as stereo headsets for gaming, headphone surround sound for video (laptops, tablets, smartphones), and up-mixed stereo presented in studio 7.1 over headphones. I’ll address only the gaming application here.

Since launching VisiSonics two years ago, we have been working on an efficient plug-in for gaming. The resulting “RealSpace 3D Audio” technology allows developers to map sound to objects in 3D with pinpoint precision and creates incredible immersion over stereo headsets. Our beta plug-in for the current version of Unity 3D Engine provides the following features:
- Process multiple sound sources with very low overhead per sound

- Our 3D sound reproduces azimuth, elevation and range. Elevation and range are typically not modeled effectively in many of today's 3D sound systems

- Much more realistic 3D sound experience incorporating reverberation from room walls and dampening by partitioning walls

- Introduces notion of "Sound Material" for modeling reverberation properties of different textures. This “RealSpace Texture” technology enables rooms with different wall textures to reflect sound in a more natural, "expected" manner.

- An extensive, well documented C++ API. For Unity, the API is exposed in C# as well, making it possible for developers to call RealSpace 3D Audio functions from their game scripts.

- Available for PC/Mac/iOs/Android/Win Phone 8


We recently demonstrated the basic features of RealSpace 3D at Unity User Group meetings in Washington D.C. (WAUUG) and in Baltimore (Bully Entertainment). Suffice it to say attendees were very impressed, and we are working with several of them via a free license to test the RealSpace 3D Plug-in and add 3D sound to their current projects and titles.

Michel Henein
profile image
Hi Gregg,

Would appreciate not using the comments section to advertise your product - there are other advertising methods available on Gamasutra :)

Thanks,
Michel

Gregg Wilkes
profile image
Michael,

Since we are in the Beta stage, I didn't see it as an advertisement, but having re-read it again, I see your point. My apologies.

Gregg


none
 
Comment: