The past couple of years have been monumental for the growth of virtual reality (VR) as a medium. VR has given rise to new ways of creating and experiencing stories, both visually and aurally, allowing users to be fully transported into a virtual world. With the unique perspective VR offers, creators are no longer fully relying on a visually-heavy narrative, which at times can be overwhelming. Content creators have begun to leverage spatial audio as a form of sensory stimulation that can guide viewers through a storyline.
In this article, I will share several lessons learned and challenges faced while working on top VR experiences for major film franchises, including: Warner Bros.’ IT: Float; IT: Escape From Pennywise; Annabelle Creation: Bee’s Room; Conjuring 2: Experience Enfield; Conjuring 2: Visions and experiences for The Greatest Showman from 20th Century Fox.
Unique Challenges of Spatial Sound
One of the biggest challenges in spatial audio mixing, as opposed to traditional sound mixing, is ensuring the mix is reactive and responds to the movement of the headset. Mix engineers cannot control where the user will be looking, making it difficult to predict what their experience will sound like at any given moment. Trying to crescendo to the climax of a scene or heighten a jump scare can fall flat if the viewer is not facing the action or is distracted by what is around them. It’s important for the entire team, both audio and visual, to constantly work toward the same goal of guiding the user through the experience, while still allowing them the freedom to explore their surroundings. In a CGI world, timing can be tweaked and camera angles adjusted after the fact to place the viewer exactly where the storyteller wants them. This flexibility isn’t possible when shooting a live-action story, making it essential that directors talk to their sound team early in the creative process while they are still in the storyboard phase. Everyone must be sure that the sights and sounds will work together to service the story.
Equally Unique Advantages
A huge advantage of spatial sound is knowing how the end user will savor the experience. Headphones are the preferred playback system for VR audio and are the only way to experience accurate and truly immersive sound. Audio producers no longer have to wonder what type of audio system the consumer will be listening on. With traditional media, there is a constant comparison of mixes between large and small speakers, as well as stereo and 5.1 surround sound setups, to ensure that the mix properly translates onto as many different audio playback systems as possible. In mixing for VR, there is the opportunity to focus the mix solely on one format to ensure the experience created in the studio sounds just as good for the viewer on their headphones.
Spatial Audio And The Creative Approach
Working with spatial audio has influenced many audio professionals’ creative approach to mixing, where they are able to create soundscapes that aren’t possible with traditional sound formats. Even with surround mixing, we are tied to a flat screen in front of us. The tendency is to shy away from using the entire soundscape and not draw too much attention away from what is happening on screen. VR experiences allow us to break free from a static 2D viewing model. Audio producers are more creatively involved and encouraged to completely immerse the viewer by creating interesting, dynamic sound that completes the picture and works in all directions.
Audio professionals are not the only ones whose creative approach has been altered by spatial audio. Filmmakers are now realizing that they can use spatial cues to steer the audience in a certain direction that allows them to properly experience a narrative. While music and sound in film have always been used to manufacture emotion and elicit audience reaction, creators are now leaning on sound to guide viewers through a narrative and even “stand in” for visuals when there is not enough footage to work with. While it can be challenging to always direct users where to look, creators are able to use the entire field of view for both picture and sound, allowing the audio to take on an even more critical role than before. There has been a reawakening for directors, cinematographers, and even actors as they realize the potential of 360-degree content. Audio professionals like myself have become energized by the opportunity to help filmmakers tell their stories and unlock the full potential of the VR medium.
Demand For Audio Problem Solving
Exploring a new medium comes with its fair share of difficulties and obstacles to overcome. The greatest challenge to working with spatial audio is the complicated delivery formats required by the various viewing platforms. There is no clear standard agreed upon and the major platforms each support a host of different formats with different specs and varying levels of quality. This bottleneck can curb creativity as certain formats unexpectedly may not deliver the immersive experience created in the studio. The end result of this uncertainty can lead creators to become frustrated when their vision doesn't translate the same across different platforms, unnecessarily impacting the consumer’s experience. Audio experts must remain solution-oriented at the forefront to find ways to work around platform translation issues.
Understand The Past To Form The Future
I’ve learned that in order to fully understand a format and continue to push it forward, you must understand how and why it was adopted and why its continued use is important for the community. Many people don’t know that Ambisonics, the basis for spatial audio technology, has been around since the ‘70s. Other technologies and delivery systems had to advance around spatial audio to help support its growth before it could reach the average consumer. In a sense, VR could be considered the savior of spatial audio. Immersive audio is best experienced using headphones and with much of the population now investing in higher quality electronics, there is a good match between the available VR hardware and software. With the necessary tech already in the hands of most consumers and a straightforward ease of use, spatial audio has all the right ingredients for a bright, long-lasting future.
To continue the growth of spatial audio design and mixing, the industry should unify around a standard delivery format that provides impeccable sound quality, is cross-compatible and works seamlessly for the end user. The ambisonics format is easily expandable and it is expected that the industry will continue to push VR development to deliver a vast amount of aural realism within 360-degree experiences.
In the end, the user will gravitate to what is easiest. The VR industry will survive by creating exceptional content with uncompromised quality, delivering an experience that users will want to share with others.
Matthew Bobb is the CEO of the full-service audio company Spacewalk Sound (https://spacewalksound.com) in Pasadena, CA, sister company of SunnyBoy Entertainment. He is a spatial audio expert whose work can be seen in top VR experiences.