#1 AI Music Podcast 2023: Holly Herndon Interviews Dadabots
Holly Herndon is an accomplished musician and techno-futurist. Her songwriting centers on bold vocal arrangements with glitchy, ethereal and avant-garde music. She is the creator of an AI vocal character named Spawn, who appears on some of her songs, and the broader AI voice transfer software called Holly+.
Holly earned a PhD in music and computer research from Stanford a few years ago. Since then, she's continued to explore the AI music space with her partner Mat Dryhurst. They co-host Interdependence, a show about AI music, decentralized economies, and the current state of machine learning as professional artists.
Holly's Interview with Dadabots
In this article, I'll be sharing highlights from an Interdependence interview between Holly, Mat, and generative cyborg duo, Dadabots.
What will happen to musicians once computers can create thousands of hours of listenable music in any style, with minimal effort or cost?
The episode, Playing with fire and extreme AI music, aired on January 10th, 2023 and is packed full of insights about the current state of AI music.
Scroll to continue reading or use this table of contents to jump to a section.
Introducing Dadabots
Dadabots are a duo of machine learning experts who specialize in generative audio. Their aim is to eliminate humans from music. Or at least, that's their tagline. They might be joking.
A decade ago, Dadabots was already involved with the 2014 Echo Nest Remix, where other non-AI concatenative mixer tools like Wubmachine made their debut as well. Echo Nest was later acquired by Spotify.
Since then, they have entered the International AI Song Contest, partnered with Harmonai, and joined with Compressorhead to make AI music robots. We'll cover all of that in this article.
Timbre Transfer vs. Generative AI Music
Listen to the podcast here: Playing with fire and extreme AI music
Holly specializes in vocal AI and tone transfer while Dadabots are more focused on generative music. These different approaches to AI give each of them a unique vantage point on the conversation.
Mat Dryurst hosts the conversation, while Holly pops in occasionally to share her thoughts. To understand where she's coming from with her art, check out this TED talk and performance from 2022.
Holly+ and Spawn: The AI vocals of Holly Herndon
During the presentation, Holly speaks about her vocal software Holly+ and explains that she's licensing the AI voice for commercial use. To do this, she has opted into an NFT royalty system, using a DAO that she trusts.
Her website does mention that she worked with Never Before Heard Sounds to implement the voice transfer, but so far NBHS has not released their software publicly. They received a round of press in July 2021, but as of March 2023 the app is still not available. Here's a demo of what it can reportedly do:
Since NBHS has not released their software, we're not able to test Holly+ within the context of stem splitting and modifying a song. Guess we'll just have to be patient!
8 Best AI Music ideas from Holly's Dadabots interview
Listen to the podcast here: Playing with fire and extreme AI music
Holly's guests are Dadabots, some of the most ambitious ML music people I've seen. Here are eight of our favorite ideas they shared on the podcast.
1. AI music faucets: Infinite music generators
John Cage famously wrote a piece for organ that takes 639 years to complete, but it's hard to imagine anyone carrying out that performance.
AI music faucets seems to be a new concept that emerged from the ability to train a model on music and produce unlimited variations. Above is a Bach Faucet. Developers have joked that you could push a button and generate hundreds of thousands of sonatas with it.
2. The $2M cost of training OpenAI Jukebox
I thought it was interesting to hear Dadabots estimate OpenAI's costs for a full round of training on all the music in Jukebox was roughly $2M. That just goes to show how pricey machine learning still is and how far we are from making AI music generation readily available.
OpenAI's Jukebox API was trained on hundreds of artists and thousands of songs. To my knowledge, the data set included lots of music protected by copyright law. It's unclear to me whether they had to pay any licensing fees to include it in their data, since anyone was allowed to use the API to generate original music.
3. The Black Wall: A private zone for AI music bots
One of my favorite ideas presented by Dadabots was the black wall, borrowed from the game Cyberpunk 2077. It's a place tasked with keeping rogue AIs from breaking through into the rest of cyberspace and wreaking havoc.
My impression of the black wall is that it conceals a layer of the deep web where AI music bots train privately on copyrighted audio files, to learn and possibly even create music without interruption, curbing the potential for commercial misuse.
Digital Rights Management enforces copyright law by preventing channels from republishing sampled audio without permission. If the fabled music from behind this AI music black wall did flood the internet, it could be struck down and removed automatically by DRM if it was too similar to source material.
4. The Dadabots Synthesizer Genre Knob
Here's another concept Dadabots brought to the conversation with Holly. What if you could interact with music in real time, switching between genres to change the entire arrangement at the drop of a hat. Dadabots are currently working on a prototype for this concept using generative AI and I'm excited.
The closest thing I've heard to this musically is music from John Zorn, like the classic Naked City recordings, Cobra music games, and tracks like the one below called This Way Out:
Zorn has been interviewed about his own experiments with composing in shifting blocks of sound, where multiple genres of music fit into one song. It's a fascinating technique that few artists have attempted, but could be achievable in the future if Dadabots can pull off their genre knob!
5. Generating AI Tablature for Compressorhead
Dadabots are taking generative AI music tools into the physical dimension, through a partnership with the musical animatronics group Compressorhead. We've previously covered the fascinating world of AI music robots and were happy to hear Compresshorhead was involved.
As they explain in the interview, the robot band was designed by engineers who don't have as much experience in music or programming. Dadabots stepped in to help improve the team's music pipeline.
The robot guitarist is being reprogrammed to understand tablature and perform it accurately in a live setting. Dadabots are also working with a machine learning engineer to put together a guitar-tab generating neural network.
If everything comes together as planned, Compressorhead will be able to generate the tablature for metal riffs with AI and feed them into this animatronic guitarist, to play it perfectly without having to rehearse.
Once the entire band is hooked up to automation like this, the dream is to use the Dadabots genre knob to change what the band performs in real time.
6. Fixing Dillinger Escape Plan's Predictability Bug
Dadabots talk about early experiments training their neural networks on the album Calculating Infinity by Dillinger Escape Plan. It's a timeless metalcore album packed with angular, odd-meter guitar riffs, insane drums, and iconic vocals.
There's just one problem. The more you listen to an album like Calculating Infinity, the less it surprises you with its twists and turns.
Dadabots wondered whether this loss of novelty was a kind of "bug" that could be fixed for music lovers. By training on mathcore bands like Dillinger Escape Plan, they were able to fine tune the output and customize their model to generate thousands of hours of similar music. Problem solved?
The original:
Dadabots version:
7. AI Explores the latent space between songs
The image above shows a vinyl record overlayed on a solar system, with the record player needle dragging through an orbital path of the planets.
Holly, Mat and Dadabots speak about the notion of latent space in machine learning. It's a technical topic steeped in mathematics and also a way of addressing where AI music actually comes from. After all, if it's not simply reusing existing material, how is the computer inventing music out of thin air?
Dadabots explain that they are tapping into the dark matter of deep learning space. The songs between the songs, like the space between the planets, if you will.
8. AI Music: Similarities to P2P file sharing
Dadabots aren't exactly tiptoeing around the problems of AI music generation. If anything, as Mat jokes during the podcast, they're pouring gasoline on the fire and running straight into the flames.
It's a touchy subject, as musicians already struggle to collect a fair share of royalties from streaming platforms. Dadabots don't shy away from discussing the impact of machine learning on artist streaming revenue. They don't come bearing much in the way of solutions either.
They do point to peer to peer file sharing networks as a clue about what could happen with AI music. When apps like LimeWire and Napster flooded the internet with free music, it disrupted CD and record stores, impacting artist revenue as a whole.
AI music could similarly lead to an over-abundance of music. It really depends on how streaming platforms manage their content and whether they can adapt AI music copyright law to include fair remunerations for artists.
Conclusion
This decade will be remembered as the time when artificial intelligence roared into center stage. Music will lag behind the text and image generation considerably, due to the high cost, time, and expertise to generate anything of quality.
I enjoyed hearing all of the insights and experiences from Dadabots and appreciate Holly and Mat for having them on Interdependence.
Listen to the podcast here: Playing with fire and extreme AI music