Anyone doing spatial audio design for VR?

[BKK-OZ]

I am working a bit in VR lately, taking some of my art and re-creating it as VR experiences. Some interesting challenges with the audio side of things, just wondering if anyone else is starting to work in spatial/VR audio?

[labman]

I dont work in VR BKK, but I imagine SPAT by flux is something that would be killer for your work. I can't say enough positive things about it for moving and placing things in space.

[BKK-OZ]

Thanks for that labman.

I am familar with SPAT (I have Flux's HEar plug).
But neither of those help with the workflow side of things.
They are both pretty useful in terms of traditional sound design workflows, but not VR.

Two Big Ears have developed a DAW-VR plugin, but it isn't supported in DP, and Google have some tech that one can use within Unity as part of their VR developer's kit. Would love to use the former but can't, and the Google tech means working out audio from within Unity which is not ideal.

Still searching...

[dix]

I've been looking into this myself. Thanks for the info about Two Big Ears. That's the only viable Mac option that I've seen. Seems like there's a lot more options for PCs.

Whole lotta info here: https://developer.oculus.com/documentat ... dk/latest/

[BKK-OZ]

Thanks for that link dix, will check it out.

I wrote to Two Big Ears (and pestered them via Twitter) about supporting DP.
They wrote to me and said 'not yet'. I tried their AU plug anyway, but it didn't pass the DP validation routine.

- maybe if others also said they are interested, it might help move things along a bit?

Edit:
I just re-checked, and the Two Big Ears plug has loaded, but my trial license has expired, so I am going to try to get a new one.

Also found this binaural, HRTF plugin, but still after that elusive workflow: http://www.dklingler.com/projects/spatializer

Edit/Edit:
The Two Bigs Ears set up requires two plugins, one to sit on your audio channel, the other as a 'Control', where you route your signals/mix, etc. The first loads OK in DP, but the Control won't pass validation for me.

I should say I am on DP 8+, so I dunno if this would still be the case in DP 9+.

[dix]

Okay, I'll do my part to encourage DP support from TBE in the future.

Thanks for the updates. VR ain't going away. It'd be cool if MOTU got ahead of it by offering Spatial Sound Design tools in DP like they do with Surround.

[billf]

I am working a bit in VR lately, taking some of my art and re-creating it as VR experiences. Some interesting challenges with the audio side of things, just wondering if anyone else is starting to work in spatial/VR audio?

It's an entirely different area from traditional audio because you have to work in X, Y, and Z. There is also distance, doppler, effect of rooms/walls/material etc. to take into account in order to create realism. Also a big challenge will be to make it work in real time as not all VR will be limited to static game environments.

You can experience some of this in the virtual world Second Life by going to a live music venue. That platform allows you to control your camera such that you can move it in the 3d space and the sound will adjust accordingly. For example, if you go to a live music venue, you can set it up so that if you move the camera close to the performer so that the sound is "in your face," but as you move the camera back the sound moves back as well. That happens in real time (live), and while that platform is a decade old now, it can give you a sense of some of the challenges that exist in VR sound.

Perhaps some tools emerge from Apple, as Tim Cook has been talking up VR lately, and Apple has patents in this area. Whether tools emerge in the form of additions to DAW's (either as plugins or stock capabilities), or are new tools remains to be seen. The audio solutions will also likely need to work with mesh design tools (Maya, Blender, etc.) as well as with VR headsets. So, it will be interesting to see how it develops.

[BKK-OZ]

It's an entirely different area from traditional audio because you have to work in X, Y, and Z.

So true.

And for games, and other VR experiences, there is really no 'mixing' per se, because you put the sounds into the VR environment, and the observer in effect mixes them via their movements - closer=louder, etc. It is more like setting up for a live concert where the audience can freely move around, than it is like mixing for traditional stereo, or even 5.1 playback.

Interestingly (at least to me) is that no one really seems to be talking about audio for VR video. In a VR video, as distinct from a VR experience, the observer can move their head around, but not their body (the camera might move, but all the observer can do is swivel their head. VR video are typically location shots and the like, but that is the format I am currently experimenting with.

VR is as big a jump as silent-talkies, black/white-color, if not bigger.

No one has (AFAIK) cracked what that means for VR audio, because there doesn't appear to be any way to set up the audio so that it tracks with the user.

[billf]

No one has (AFAIK) cracked what that means for VR audio, because there doesn't appear to be any way to set up the audio so that it tracks with the user.

The closest I've seen so far is the stuff from Linden Lab and Philip Rosedale's new venture, High Fidelity. Linden Lab have Second Life (which as mentioned above is a decade old and incorporates some 3d audio capabilities), and an alpha for a new VR technology they call Sansar which aims to bring VR to the web. High Fidelity (also in alpha) is associated with Linden Lab (Rosedale founded both companies), and has working sites (one a Pink Floyd museum), so 3d audio is on the radar, although in the really early phases.

Here is part of a recent post from High Fidelity:

Eric’s fireworks system combines a cartoon-like industrial button mechanism with an elegant little script to detect interaction. Using Javascript, the system instantiates and launches an entity skyward using physics, creates and parents a particle system to the entity for a smoke trail, and pops out a another, final particle system for the crescendo. When combined with sound effects, audio injectors and High Fidelity’s excellent HRTF implementation, the fireworks system is an audiovisual delight, best shared with friends making obligatory oooh & aaah sounds.

[BKK-OZ]

The closest I've seen so far is the stuff from Linden Lab and Philip Rosedale's new venture, High Fidelity.

Had a look at that, thanks for the info.

Their thing seems to be kind of like a Second Life+, or a Unity online experience, which is still different from VR video. In VR video, it is a bit like the old Quicktime VR thing, where you have a panoramic image (motion video now) wrapped around the observer. In a thing like High Fidelity, or a standard VR experience, it is relatively straight-forward to place sounds spatially. But in a VR video, everything except rotation is in the can from the start, and so giving spatial audio cues becomes problematic.

Not sure if I am explaining myself well here...

[BKK-OZ]

Good article with quick history of spatial audio and an update on the current situation: https://t.co/hPV1HgYkLT