Spotify has been quietly positioning itself as a player in the AI music space. Until recently, these efforts were focused on making better music recommendations with machine learning. Their audio collection and user metadata is a goldmine for this kind of work.
In this article we'll cover Spotify's most recent AI playlist and AI remix features. We'll explore how these efforts compete directly with Amazon and TikTok, before zooming out to look at the bigger picture. What is Spotify's endgame?
Table of Contents
Spotify AI Playlist vs Amazon Maestro, TikTok Remix
Last year, in the beginning of 2023, Spotify introduced an AI DJ that speaks to you, narrating decisions as it serves up songs that fit your tastes and listening habits. You can watch a promotional trailer for that experience below:
On April 7 2024, Spotify announced a new AI playlist feature that builds on the same premise, but with more user input and control. Instead of passively receiving songs from the DJ, premium subscribers can now input any kind of text and retrieve sounds that match that mood.
Within a little over one week, Amazon Music fired back and announced its own AI playlist feature called Maestro. The app operates on the same premise, translating any kind of written input (or even emojis) into playlists for that theme.
Both services have made it clear that they won't allow offensive language or inappropriate prompts. This may come as a disappointment to some, who would prefer an uncensored experience.
Spotify is already one step ahead of Amazon with a second AI feature called Remix, announced April 12th. In a recent interview with the Verve, CTO Gustav Soderstrom explained that Spotify was redesigning the app to compete with TikTok.
Remix will allow users to speed up, mash-up, and edit existing tracks. When creators do this on TikTok, record labels have a difficult time capturing royalties. Spotify will combat this by ensuring that any modified track can be traced back to the original license holders.
Universal Music Group went so far as to remove their entire artist catalog from TikTok in February 2024, as a show of force. However, many of the artists themselves were upset by this decision, as it represented a net loss for them in terms of discoverability and revenue.
Will Spotify return to music production tools?
People tend to think of Spotify as a streaming service. But the company has been investing in other sectors of the music industry. In 2013 they published Soundtrap Studio, a free and mobile-friendly music creation app, to help people write and record songs quickly.
A digital audio workstation may seem strange coming from a playlist app, but their mission statement clarifies things a bit:
“Our mission is to unlock the potential of human creativity—by giving a million creative artists the opportunity to live off their art and billions of fans the opportunity to enjoy and be inspired by it.” - Spotify
Spotify wants to deliver you that highly personalized playlist. But instead of hiring millions of music experts to do it manually, they've acquired and developed machine learning algorithms that can do it automatically (or at least, try to).
In 2014, Spotify acquired The Echo Nest, a music intelligence and data platform. According to LinkedIn, Francois Pachet left Sony in 2017 to join Spotify’s team as the director of their Creator Technology Research Lab. That same year, they acquired another AI audio processing startup called Niland.
In 2021, Spotify announced the acquisition of Podz for their podcast recommendation system. In June 2022, they released Spotify Karaoke and bought Sonantic, an AI Voice company. Forbes ran an article at the end of that month, hinting that AI music creation was coming down the pipeline.
Basic Pitch: Spotify’s Audio-To-MIDI Converter
In June of 2022, Spotify announced that their Audio Intelligence Lab partnered with the SoundTrap team on an open source audio-to-MIDI application called Basic Pitch.
This software uses machine learning to transcribe audio recordings into MIDI notes. Since its release, we've seen higher quality products come from companies like Samplab and RipX, but Basic Pitch is the only free web app that offers this service.
If you’re a musician who’s been trying to figure out what the heck your favorite artist played on some song, now’s your opportunity to find out.
Benefits of Basic Pitch
Transcribe any instrument: Unlike some of the other note-detection algorithms, Basic Pitch is trained to convert audio from any instrument.
Works with chords (polyphonic): For a long time, audio-to-midi apps have been limited to single-note melodies. They had a hard time processing multiple notes at the same time. Basic Pitch lets you transcribe full chords and melodies.
Pitch bend detection: Live music has expressive subtleties, like pitch bending, that are easily lost in MIDI. Basic Pitch captures and conveys these pitch bends in its MIDI output.
Faster than most tools: Basic Pitch requires minimal resources to run, so its output is almost instantaneous.
Mobile Friendly: Works on iphone and android devices.
Problems with Basic Pitch
Difficulty with delay effects: As you can imagine, the system has some trouble with effects like delay. Are they separate notes or a single note with an effect? To the trained ear it’s obvious, but not for Basic Pitch. It's going to record each delay articulation as its own note.
Limited vocabulary for percussion: The MIDI output from Basic Pitch is not advanced when it comes to percussion. We’ll let this go, since pitch values are easier to map that percussion sound libraries, but it does impact usability.
Struggles with dense arrangements: The more instruments you have in a mix, the harder it will be for Basic Pitch to pull them apart and represent them accurately. Instruments that share a similar timbre and pitch frequency may intrude on one another.
If you're a musician who wants to peer behind the curtain of a track and find out what chords they're using, now's your chance!
While Basic Pitch isn't flawless yet, it’s a step in the right direction. I personally find it exciting to see this kind of collaboration between departments happening at Spotify.
The Spotify Enhance Feature
Most premium Spotify accounts have access to a playlist building feature called enhance. Spotify can enhance a collection of tracks and generate personalized recommendations that expand on an existing theme of your choice. The best part is that you can use playlists of any size. So if there’s one song that’s really hitting the spot and you want to hear more like it, enhance lets you do it.
The example above showcases how a single track can be turned into its own playlist. When you hit the enhance button, all of the other tracks appear moments later. Prepare to be impressed by the quality of their suggestions. I am regularly surprised by new artists that I discover with this tool.
Enhance your own songs: If your music is published on Spotify, you can expand it and hear other artists similar to you. Even if your music is brand new and has never been added to other user playlists, you’ll still get accurate pairings thanks to Spotify's AI audio feature detection.
Publishing tip: Finished your music but don’t know how to get it online? Check out digital distribution services like Distrokid or CD Baby. These are two great entry points for starting your career in the music business.
There are two major components at play in Spotify's music recommendation engine; content-based and collaborative filtering.
Content-based filters evaluate a track on twelve audio features; acousticness, liveness, speechiness, instrumentalness, energy, loudness, danceability, valence, duration, tempo, key, and mode.
Collaborative filters take a look at Spotify’s user-generated assets (like playlists) to find songs with a similar audience.
Together, these filter sets allow the Spotify AI to make predictions about what you might want to hear next. The illustrations below come from a piece written by Ashrith Shetty, titled What Makes a Song Likeable?
The changing slides of the animations depict six Spotify audio features across a hundred individual Spotify tracks.
The Dark Side of Spotify and AI Music
If you've heard complaints about Spotify creators not being paid fair royalties, you might find this documentary interesting. It covers Spotify's revenue sharing model and how bad actors set up phone farms to exploit the system.
The dark side of Spotify could be summarized as a scenario where the platform strips artists of fair and meaningful compensation for their work.
As a company that leveraged the culture of digital piracy in its early days, it's yet to be see whether they'll make similar moves to train a generative model on their audio collection, the way Suno, Riffusion and Udio have.
The company already blew the whistle on AI music spam in 2023, after detecting signs of streaming fraud coming from bad actors the generative audio platform Boomy. It's unclear whether they'd have the same objections if the AI music was being created in house.
We'll be watching this space closely and will continue to update this page as the situation develops.