Mixed Reality Music Making: AR DAW & Reactive Audio Tools
Next generation technology has been visibly disrupting the music industry this year. AI voice changers and beat making apps have drummed up fresh controversy regarding copyright law and fair use of artist identity. With all the AI drama, parallel innovations in mixed reality are barely making headlines.
Spatial Labs has recently announced what could arguably become the most exciting development in music and augmented reality. In this article we'll share some big updates in the world of mixed reality music making, from Spatial Labs' new Light Field software to smaller projects that combine interactive music with augmented reality holograms.
What is mixed reality music?
Mixed reality music is a broad label for any audio-reactive technology blending physical and digital worlds in a single experience. These could range from live performances by celebrity holograms to mixed reality DAWs that take your audio workflows off-screen.
Major labels started experimenting with augmented reality music videos and live holograms as early as 2012. Rhianna famously collaborated with Doritos to bring her track Who's That Chick? into an augmented reality format. Fan used their webcams to scan an AR marker located on the bag of chips, triggering a custom performance of the song that changed depending on how the bag was held and moved.
2012 was also the year that deceased rapper Tupac made an appearance with Snoop Dogg at Coachella, marking one of the first ever major debuts of a 3D hologram live on stage. Mixed reality has remained a popular attraction at music festivals ever since.
In the ten years that have followed, consumer access to mixed reality technology has continued to advance. Hardware solutions like Microsoft Hololens remain cost prohibitive for most, ranging from $3,500 - 5,000, but that hasn't stopped software companies from jumping in to contribute to the app ecosystem.
A team of Japanese software developers created and promoted a mixed reality piano teacher called Teomirn in 2017. You can watch a clip of that experience below:
A separate group of students from Carnegie Mellon University released their own AR piano experience called Music Everywhere in 2017. It offered a different take on piano lessons, seeking to create a colorful gamified experience with tiny musician holograms that sit atop the player's electric keyboard.
Six years have passed since these initial efforts were made to enhance the piano playing experience. Many new developments have since taken place. It was only one week ago that the first major mixed reality DAW appeared.
Spatial Labs presents Light Field's AR DAW
Digital music creatives and producers are notorious for putting in long hours at the studio. Ear fatigue is a well documented audio editing problem that can impact your performance as you try to perfect a mix.
It's far less common to hear about eye fatigue among music producers, despite the risks that screen time may pose. Secondary health problems like inhibited melatonin production are also associated with blue light exposure during long computer sessions.
Spatial Labs recently announced their efforts to create a viable alternative to making music on LCD screens, potentially reducing the strain on your eyes as you work on a track. Their mixed reality music software, called Light Field, has not been released commercially yet.
The impressive promo video above dropped just one week ago, without much clarity regarding their release timeline. This lack of transparency has attracted skepticism on internet forums, where one user speculated the company could be hyping a product that doesn't exist, to attract investors.
It's too early to know whether that's the case, but we'll give Spatial Labs the benefit of the doubt. Comparable technology already exists, like the laser keyboard niche that replaces physical QWERTY keyboards with a flat holographic projection of keys onto your desk.
So despite the impressive set of features that Light Field claims to offer, there's no reason to jump to conclusions about fakery. It's reasonable to believe that their team has in fact made headway on this tool, even if it's not publicly available yet.
Lucas Rizzotto's mixed reality synthesizer
Spatial Labs aren't the only ones cooking up new mixed reality music experiences. Futurist Lucas Rizzotto has been breaking new ground year after year, from his AI Beatles covers (back when AI music was "still underground") to current experiments with holographic synthesizers.
A video published recently on LinkedIn showed Lucas manipulating a granular synthesizer using a mixed reality hologram. He explained to one of his fans that each particle in the block represents a variation of a sample - the cube detects the location of your hand relative to the holographic cube and activates the particles as they are touched. Check that out below:
The Y-Z dimensions of the cube change pitch and other attributes in the synthesizer. Lucas designed scripts to record the movement of his hands as he performs live, so that he can play back his performance in Unity. From there, he places the video in Unity using their timeline system, lines up the virtual elements with his camera footage and replays the hand data from his recorded performance.
Lucas has also designed an interactive musical body suit that acts as a kind of MIDI controller. He's not the first to attempt this - artists like Red Letter J have built physical, interactive audio outfits like the one below.
The key innovation we're witnessing today is the option to strip away physical sensors in favor of lidar and machine vision. Lucas shares his inventions on LinkedIn and answers questions about how everything works. We encourage you to reach out to him and show support for all the progress he's made in this field.
Audio reactive elements in Adobe Aero & Spark AR
Artists have been experimenting with reactive audio for decades, but Brian Eno was one of the first to push 2D music visualizers into 3D space. He launched his mobile app Bloom back in 2008, correctly predicting that users would enjoy making ambient music from the manipulation of simple shapes. Ten years later, in 2018, Eno leveled up the Bloom experience with a mixed reality music exhibit of the same name.
If you're itching to get into the mixed reality music space yourself, there are at least two popular options available. Augmented reality artist Heather Dunaway Smith has been leading the way as one of the most accomplished creators in this space. As a product evangelist, she also educates consumers on how to build their own AR music experiences.
In the video above, she shows how anyone can create audio-reactive elements in augmented reality using Meta's Spark AR. The tutorial includes step-by-step instructions for constructing nodes in the patch editor. It's refreshing to see this kind of transparency in the artist process, considering this same AR mirage became a featured exhibit at Coachella this year.
Meta's SparkAR is not the only option available to musicians who want to expand their creative horizons. Adobe Aero also offers extensive resources for creating immersive experiences with audio.
So if you've been feeling exhausted by all the AI music generation happening this year, treat yourself to a completely different facet of musical tech. It's a great way to build new skills and differentiate yourself from other music producers.
As an aside, we would love to see a creative application of AudioCipher to an augmented reality experience. Our article on musical synesthesia might give you some ideas. Have fun and let us know if you come up with something - we'll be happy to promote it!
You can check out our previous article on VR music experiences for more information on VR DAWs and immersive music technology.