Web Audio and 3D Soundscapes: Introduction

In this tutorial we will be taking a close look at the fundamental Web Audio elements that are used to construct 3D soundscapes for immersive interactive applications including, but not limited to, 3D games.

The Web Audio API and the terminology it uses can sometimes be confusing but this tutorial aims to take away the complexity and provide a simpler explanation of the Web Audio elements and how they work together to form a 3D soundscape.

Demonstration

This demonstration contains three sounds that are rotating around a listener, the listener's direction is indicated by the arrow. If you imagine looking down on a game character (the listener), the rotating sounds could be easily represent friends or foes circling the character.

The demonstration source code and resources are attached to this tutorial.

AudioContext

The AudioContext interface is the heart and soul of Web Audio, it provides the functions required to create various Web Audio elements as well as providing a way to send all of the audio to hardware and onwards to someone's speakers or headphones.

var audioContext = null

if (window.AudioContext !== undefined) {

    audioContext = new AudioContext()

}

It's important to make sure the AudioContext interface is available because Web Audio is still fairly new and it might not be available in some web browsers.

As well as providing the functions required to create various Web Audio elements, theAudioContext interface has two important properties; destination and listenerwhich are both read-only. The destination property can be thought of as the connection to audio hardware, it's where all of the generated audio will eventually end up. The listener property (we will look as this in more detail later) represents thething that is listening to all of the audio, e.g. a character, or more precisely a camera, in a game.

Buffers

The AudioBuffer and AudioBufferSourceNode interfaces allow us to play audio.AudioBuffer objects contain the raw audio (sound samples) that are tweaked, crunched, and crushed as they make their way through Web Audio before reaching someone's speakers or headphones. AudioBufferSourceNode objects are used to start and stop the audio contained in AudioBuffer objects.

The standard way to load audio into an AudioBuffer object is to use aXMLHttpRequest object with its responseType set to arraybuffer. When the audio file has been loaded the array buffer is then sent to the AudioContext object for decoding and, if the decoding is successful, we will be provided with an AudioBufferobject.

var loader = new XMLHttpRequest()

loader.open("GET", "massive-explosion.ogg")

loader.responseType = "arraybuffer"

loader.onload = whenLoaded

loader.send()

function whenLoaded(event) {

    var data = loader.response

    if (data === null) {

        // There was a problem loading the file.

        return

    }

    // Decode the data.

    audioContext.decodeAudioData(data, whenDecoded)

}

function whenDecoded(audioBuffer) {

    // "audioBuffer" is an AudioBuffer object.

}

The decodeAudioData() function also has a third parameter that accepts a second callback, that callback is called when the loaded audio file cannot be decoded.

decodeAudioData(data, whenDecoded, whenFailed)

Not all web browsers support the same audio formats, a good table of supported formats can be found here, so you might want to use the second callback to fallback to an alternative audio format if needed. For example, Internet Explorer doesn't support OGG Vorbis but it does support MP3. The only real problem with MP3 is it doesn't allow seamlessly looped audio like OGG Vorbis does.

When you have an AudioBuffer object available you can play it using anAudioBufferSourceNode object.

var source = audioContext.createBufferSource()

// Attach an AudioBuffer object.

source.buffer = audioBuffer

// Connect the "source" object to the "destination" object.

source.connect(audioContext.destination)

// Optionally, tell "source" to loop the audio continuously.

source.loop = false

// Start the audio.

source.start()

It's important to remember AudioBufferSourceNode objects are single-shot audio players, in other words you can only use the start() function once. You will need to create an AudioBufferSourceNode object and connect it (directly or indirectly) to thedestination object exposed by the AudioContext object whenever you want to play audio from an AudioBuffer object.

You could make life a little simpler by creating a small utility function that creates, connects, and starts an AudioBufferSourceNode object for you.

function play(audioBuffer, audioContext) {

    var source = audioContext.createSourceBuffer()

    source.buffer = audioBuffer

    source.connect(audioContext.destination)

    source.start()

}

play(audioBuffer01, audioContext)

play(audioBuffer02, audioContext)

play(audioBuffer03, audioContext)

When an AudioBufferSourceCode object finishes playing, and if you have no references to the object anywhere (e.g. you don't have them stored in an array), then Web Audio will automatically disconnect the object for you. This is extremely handy when you only need to fire-and-forget short sound effects etc.

If you decide to loop the audio, using the AudioBufferSourceNode loop property, you will need to keep a reference to the AudioBufferSourceNode object somewhere so you can stop() the audio playing.

source.stop()

So at this point we are using buffers to play audio, but the audio is being played directly without any panning or spatialization being applied to it. This is where PannerNodeobjects come into play.

Panners

PannerNode objects allow us to position audio in 3D space, within a cartesian coordinate system. This is where most of the 3D magic happens.

A PannerNode object has quite a few properties that allow us to fine-tune the behavior of the audio but for this tutorial we are only interested in two of them; maxDistance andpanningModel. The maxDistance property is the distance from the listener at which point the audio volume will be zero. This is an arbitrary value and will only have meaning within your application but it defaults to 10000. The panningModel tells Web Audio how to process the audio passing through a PannerNode object. For 3D soundscapes you will probably want to set the value to HRTF (head-related transfer function).

To set the position of an AudioBufferSourceNode we use the setPosition() function exposed by a PannerNode object.

var panner = audioContext.createPanner()

panner.panningModel = "HRTF"

// Set the 3D position (x, y, z).

panner.setPosition(1, 2, 3)

// Connect the "source" object to the "panner" object.

source.connect(panner)

// Connect the "panner" object to the "destination" object.

panner.connect(audioContext.destination)

// Start the audio.

source.start()

To make things a little clearer let's update the utility function we created previously.

function play(audioBuffer, x, y, z, audioContext) {

    var source = audioContext.createSourceBuffer()

    source.buffer = audioBuffer

    var panner = audioContext.createPanner()

    panner.panningModel = "HRTF"

    panner.setPosition(x, y, z)

    source.connect(panner)

    panner.connect(audioContext.destination)

    source.start()

}

play(audioBuffer01, 1, 2, 3, audioContext)

play(audioBuffer02, 4, 5, 6, audioContext)

play(audioBuffer03, 7, 8, 9, audioContext)

At this point we are playing audio and positioning it in 3D space, but there is one more important element we need to look at; the audio listener.

W3C PannerNode specification.

The Audio Listener

Every AudioContext object exposes a listener object that represents the position and orientation of the thing that's listening to the audio. Usually the thing would be a virtual camera that's attached to a game character's head, the bumper of a car, the tail of an aircraft, or anything else that makes sense to the viewer from their perspective.

The listener object has a setPosition() function and a setOrientation()function. The setPosition() function places the listener somewhere within the 3D space, and the setOrientation() rotates the listener (imagine a camera panning and tilting).

The setPosition() function works in exactly the same way as the PannerNodesetPosition() function and accepts three coordinates.

audioContext.listener.setPosition(x, y, z)

The setOrientation() function is a bit more complex, it accepts two unit vectors. The first vector represents the listener's rotation (the direction the camera is pointing), and the second vector represents the listener's up direction (it points out of the top of the camera).

audioContext.listener.setOrientation(x1, y1, z1, x2, y2, z2)

If you only need to rotate the listener around one axis the vector calculations are relatively simple. For example, if you are using the same coordinate system that WebGL uses where positive x points to the right of the screen, positive y points to the top of the screen, and positive z points out of the screen, then you can rotate the listener around the y axis (pan the camera) using one cos() function call and onesin() function call.

// The listener's position (could be anything).

var x = 50

var y = 100

var z = 0

audioContext.listener.setPosition(x, y, z)

// Calculate the rotation vector.

// rad = rotation, in radians

var rad = 0.10

var v1 = Math.cos(rad) // x

var v2 = 0             // y

var v3 = Math.sin(rad) // z

// The "up" vector

var v4 = 0 // x

var v5 = 1 // y

var v6 = 0 // z

audioContext.listener.setOrientation(v1, v2, v3, v4, v5, v6)

The demonstration for this tutorial (source code is attached) does a similar thing and rotates the PannerNode objects around a single axis.

W3C AudioListener specification.

Conclusion

In this tutorial we took a look at the fundamental Web Audio elements that are used to construct 3D soundscapes for immersive interactive applications including, but not limited to, 3D games. Hopefully this tutorial has been of use to you and has provided enough information for you to have an understanding of how audio buffers, panners, and listeners work together to produce 3D soundscapes.

If you have any feedback or any questions please feel free to post a comment below.

Resources

The Next Step: Implementation

In the next tutorial, Web Audio and 3D soundscapes: Implementation, we will be take all of the above (and more) and wrap it in a simplified API. The main focus of the next tutorial will be 3D games but the API will be generic enough for use in various immersive interactive applications.

posted by: http://webdesign.tutsplus.com

Blog Tutorial