With the rise of 360, virtual, mixed and augmented reality experiences, it’s apparent that immersive, dynamic, and reactive spatial audio is necessary to deliver cutting-edge content, and preserve or exaggerate the realism they deserve. Major streaming video platforms such as Facebook and YouTube already support spatial audio play back, and a number of tools and plug-ins in development for the past few years arm sound designers with the tools required to deliver such mixes easily.
Spatial audio is a powerful way to fully immerse a user, add realism, and direct attention within VR or 360 video experiences. While audio cues can help direct a user’s attention, spatial audio makes what we hear match what we see and experience. It realistically triggers audio cues from any direction, draws in the user and provides a believable VR experience.
What is Spatial Audio?
Spatial audio in virtual reality is the manipulation of audio signals so they mimic real world acoustic behavior. It not only serves as a mechanism to complete the immersion, but is also a very effective interface. Audio cues can call attention to various focal points in a narrative, or draw the user into watching specific parts of a 360 video, for example.
The user can experience spatial audio with a normal pair of headphones, with no need to invest in special speakers, hardware or multi-channel headphones.
Check out these examples of spatialized audio on Facebook. Make sure you have your headphones on!
Pro Tools | HD + Facebook 360 Spatial Workstation
In Pro Tools | HD 12.8.2, Facebook 360 Spatial Workstation plug-ins are included so dynamic and immersive spatial audio mixes can be created right inside Pro Tools | HD. 12.8.2 also supports Ambisonics (4, 9, and 16-channel tracks & busses), which adds flexibility when working with immersive audio. Ambisonics is a technology to create, capture, and render 3D sound fields in a spherical format around a particular point in space. It’s conceptually similar to 360 video, except the entire spherical sound field is audible and responds to changes in head rotation. There are many ways of rendering to an Ambisonic field, but all of them rely on decoding to a binaural stereo output to allow the user to perceive the spatial audio effect over a normal pair of headphones.
Ambisonic audio can be of n-orders comprising various channels. More channels results in higher spatial quality, although there is a limit to the perceived difference in sound quality with higher orders. Regardless of the number of channels used for encoding the original signal, the decoded binaural audio output will always be to two channels. As the listener moves their head, the content of the decoded output stream shifts and changes accordingly, providing a 3D spatial effect.
While third-party plug-ins have worked with Pro Tools | HD before this update, various methods had to employed to work around a pre-existing 8-channel limitation. The new updates supports Ambisonic audio, tracks and busses natively, making the workflow infinitely easier.
Facebook 360 Spatial Workstation
The Facebook 360 Spatial Workstation is an end-to-end pipeline that allows sound designers to drop in audio sources, pan and sync to scene elements, and render to a single Ambisonic file to play back on Facebook, Oculus Video, and other platforms. Originally developed by Two Big Ears, Facebook 360 Spatial Workstation is now a free tool provided by the Audio 360 team at Facebook.
Facebook 360 Spatial Workstation is a collection of plug-ins for DAWs that include a Spatialiser, 360 VR Video Player, Encoder, and rendering SDK, just to name a few. These plug-ins help authors create and position spatial audio content, encode it with platform-specific metadata (for Facebook, YouTube, etc.), and publish.
The Facebook 360 Spatial Workstation is a collection of plug-ins for creating interactive spatial mixes for 360 videos.
The Spatialiser plug-in allows the sound designer to place a sound source in space. The source could be mono, an Ambisonics recording, or multi-channel such as a surround recording or reverb. Non-mono sources act as a “bed,” while diegetic mono sources, such as dialogue and sound effects, are usually placed in a scene. Non-diegetic audio, such as narration or background music, is usually routed to the head-locked stereo bus, which does not rotate as the user looks around.
The Control plug-in acts as the command center controlling how all audio is routed for real-time binaural playback through headphones. This plug-in also manages global settings of features such as early reflection modelling and mix focus.
The 360 Video Player is “slaved” to the DAW timeline, and allows the sound designer to preview the mix with the 360 video in real-time, either in VR or on the desktop. Desktop mode allows designers to rotate the video with the keyboard or mouse, which will rotate the sound field instantly, providing instant feedback during the authoring stage.
The Facebook 360 Encoder application takes a video file and combines the audio files into the video container, suitable for playback on Facebook and other supported platforms. It is also possible to convert between audio formats, such as getting a 1st order ambiX output from a 2nd order ambiX input and adding metadata to the file describing values for the Mix Focus feature. This process also injects relevant spatial metadata into the tool, making the final asset ready for upload to supported platforms.
The Converter plug-in is a utility that can rotate a mix after it has been created, or output to other formats such as 4-channel ambiX.
The Loudness meter provides an overview of the loudness of the entire spatial mix. Loudness for spatial mixes is considerably different than legacy channel-based formats. Spatial audio for 360 videos is considerably more complex, and this meter provides useful data that will prevent the final published content from distorting when played back on the target device.
Supported Platforms for Playback
- Facebook on Desktop (Chrome browser) and mobile apps
- Oculus Video on Gear VR and Rift
- .tbe format: Apps with Rendering SDK
- YouTube 360
- Other platforms with support for Ambisonics or quad-binaural format. Note that these platforms have specific instructions for preparing assets.
Here are some key resources to get started with Facebook 360 Spatial Workstation.