You’re a detective checking out a creepy farmhouse that’s the scene of the owner’s murder. You search for clues by exploring different rooms, and while nothing ‘jumps’ out to scare you, this overwhelming feeling of unease escalates until you’re running for your life, trying to escape and hide from the killer. Robot Invader’s mystery-adventure VR game Dead Secret, available now for Samsung Gear VR, combines two things that VR does really well for games — makes exploration interesting, and immerses the player into the game experience. And Robot Invader does it very successfully, with visuals and sounds working seamlessly together.
Designing and implementing multidirectional, spatialized sound for an interactive 3D environment seems like a daunting task, but game studio Robot Invader (http://robotinvader.com) has got it down to a science. Chris Pruett, chief taskmaster/co-founder of Robot Invader, based in Menlo Park, CA, walks Post through the workflow on Dead Secret, explaining their approach to VR sound.
Starting with the sound design on Dead Secret, Pruett explains that each room the player moves through has its own subtle visual and aural theme that helps remind the player of the feeling of that room. “It was pretty tricky to achieve without being in-your-face,” says Pruett, who worked with freelance sound designer Jordan Fehr. He created subtle, aural themes by using ambient background sounds and very specific reverbs that are directly related to the size and content of each room. For certain rooms, there were very specific emotions that Pruett wanted to evoke. “We did it with art as well, but a lot of the emotion comes from the sound. One of the most common feelings was foreboding. Maybe you enter a room and there isn’t anything particularly threatening about the room visually but we wanted the player to feel uneasy. That’s almost entirely a sound issue.” Adding to the feeling of unease is composer Ben Prunty’s score.
As the game progresses, the player is put under more significant stress while trying to hide from the killer. During those sections, Pruett wanted the sound to be abrasive. “We did several iterations of sounds that were abrasive yet weren’t going to hurt the player’s ears. We wanted to increase the stress levels but not make it uncomfortable,” he says.
In addition to the stereo ambiences and abstract sounds, Pruett had sound designer Fehr create specific sound assets, like a variety of footsteps for numerous surfaces, and the hum for a 1950’s refrigerator, all delivered as high-quality mono files. “In virtual reality, where we are spatializing sounds, stereo sound files don’t make sense,” says Pruett. One important feature of VR sound is spatialization, which Oculus explains as the ability to play a sound as if it is positioned at a specific point in three-dimensional space. That includes above and behind the player’s position. Using the refrigerator hum as an example, Pruett can put the hum at exactly the same spot as the refrigerator is in the visuals. The sound of the refrigerator changes as the player gets closer or further away, and the sound changes as the player turns his/her head.
Pruett used the Oculus Audio SDK, a free tool written by Oculus that helps developers add spatialization processing to their audio assets by employing HRTF (head-related transfer function) filtering. “Essentially, we take a sound that we specify and play it back from a point in space that we specify, and do it in a way that what you hear in your left ear and your right ear aren’t going to be the same exact sound because the sound has to flow across the shape of your head. Under the hood, it’s a very complicated transformation of the sound wave but in terms of actually implementing it, the Oculus Audio SDK makes it pretty simple,” says Pruett. The Oculus Audio SDK works with game engines like Unity3D, and middleware like FMOD and Wwise.
Robot Invader uses the Unity game engine for which they designed their own custom audio infrastructure that allows them to quickly implement sounds into their games. It also allows them to control reverb on a per sample basis, move sounds around in a 3D space, and reuse audio sources to save memory. To that infrastructure, Pruett added code to leverage the Oculus Audio SDK for playing spatialized sound.
“That was pretty much a drop-in addition to our system,” says Pruett. “The hard part wasn’t the technology. The hard part is figuring out how to put sounds in a world in a way that would help us control the mood and the feeling of the game.”
Dead Secret will also be available on other VR platforms, including the Oculus Rift, and even non-VR platforms. “Both are supported by Oculus’s SDK. For the non-VR application, if you play while wearing headphones, you’ll get nice spatialized audio,” says Pruett. Other VR platforms, like PlayStation VR, which doesn’t support the Oculus Audio SDK, provide their own, similar SDKs. “We are not finished with the work yet, but at the moment, it looks like our existing audio infrastructure, plus Unity’s infrastructure, will work very nicely across the various platforms.”