AI music reporter Kyle Wiggers (at TechCrunch) announced in October 2023 that Riffusion, a generative AI music startup, has successfully raised $4M in a seed round. That same week, a new Riffusion website rolled out with improvements to the user dashboard. Several new AI music features were introduced.
The company has been online for close to a year now. In its first incarnation, Riffusion was a free web app that offered an instrumental AI text-to-music service. No user sign up or login was required to get started, so anyone could jump right into the music making experience.
October's version of the site represents a big step forward. Users have private libraries to store their creations and can bookmark their favorite tracks.
The web app generates instrumentals in a given BPM and key signature. Then it creates a second, compatible layer of vocals on top of it. The end user receives a single track with the two combined. I've heard a senior developer at an AI music company refer to this technique as AI vocal overpainting.
Riffusion includes the option to isolate audio stems for any track you create. Download the drum, bass, vocal and harmony layers as needed!
In this article, I'll walk you through Riffusion's new features in detail. We'll also provide an overview of the company's brief history, and their biggest competitors.
Table of Contents
Exploring new features in Riffusion Version 2.0
To follow along, visit Riffusion's website and sign up for a free account. Once you're registered, this will be the first dashboard you see:
Trending riffs on Riffusion's Explore page are a collection of AI music generated by other users. Hover over a track to read the lyrics like karaoke and hear them performed by an AI voice.
Users can click the heart icon to improve that song's rank in the community. Click through the creator's profile name to see a library with other riffs they've made.
We've time stamped a walkthrough of Riffusion below, so it takes you straight to the Explore community page. Have a listen to what the music sounds like and see how the app works.
How do you generate an AI song in Riffusion?
To create an AI song in Riffusion, click the "plus" icon located at the top left corner of your dashboard. You'll be encouraged to add lyrics and describe the sound of music you want, based on features like genre, vocal style, or overall vibe.
Type in lyrics or use a microphone to speak words directly into the app. If you don't have any idea where to start, try a super one or two word concept. Then click write lyrics and let the app generate something for you. You'll find a similar prompt genius button that helps you write descriptions of music with an AI assistance.
Once your lyrics are ready and the sound has been described, hit Riff to generate some music. Within a minute, you'll have three tracks to choose from.
Click the play button on any card to hear the song played back with AI vocals and instrumentals. In a moment we'll show you where to access the audio stems.
Hover over a card to reveal a few options represented by icons.
The remix feature refreshes a track and delivers a variation.
The favorite icon will stash the track in your User Library.
The share icon copies a link to a public URL where anyone can listen.
The ellipsis includes several additional options. You can open the riff in a dedicated window, save as audio or video, and split the audio into stems.
The Split out stems option will separate your track into vocals, drums, bass, and "other" layers. Download any individual audio layer and use them in a DAW.
Click on Open riff to navigate to a full page view of the track. You can access most of the same features, but stem separation is no longer available and instead there's a control over the riff's visibility (Public vs private "Link only" mode).
Now that you're up to speed on their updates, let met take you back to the company's roots, as a two-person indie hacker team who landed on the front page of sites like Hacker News and TechCrunch. within days of their initial release.
Riffusion V1: The First AI Text-to-Music Generator
Riffusion’s early history traces back to December 2022, at the tail end of a year ripe with AI text-to-image generation. Midjourney and Dalle were becoming near household words. Then OpenAI released ChatGPT3 and suddenly the world had access to conversational artificial intelligence.
Text prompts had become a hot topic and the internet wanted to know when AI music generation would have its moment in the sun. Who would be the first to drop a service that converted descriptions of music into raw audio?
At the time, AudioCipher’s text-to-midi plugin was turning words into melodies and chord progressions, but without AI. It gave users control over key signature, scale, chord extensions and rhythm automation. The app was intended for DAW musicians to beat writer's block. It wasn't an AI tool for the general public.
Riffusion entered the ring in December 2022, delivering on “text-to-music” in a free, open source web app. Suddenly, users didn’t need any knowledge of music theory to get started. They could type in the genre of music or instruments they wanted, and produce short riffs in a matter of seconds.
The creators were a pair of indie programmers, Seth Forsgren and Hayk Martiros, who started it as a hobby project. Indie hackers, entrepreneurs and musicians all seemed to appreciate the effort and the website became a viral hit overnight.
Riffusion AI: Spectrogram Generator to Music
Wondering what Riffusion sounded like when they first launched? Have a listen in the video demo above.
Riffusion was built with help from a Stable Diffusion model called img2img. It trained on a multimodal dataset of labeled spectrograms. Each short audio clip was captured in an image and paired with a caption describing features like genre, instrument, speed, vibe, etc.
When a user requested music in a particular style, Riffusion's model generated new spectrograms in that style. It interpolated and stitched the short clips together, sonifying the spectrograms and turning them into audio files. Denoising options were available in an effort to improve the final output quality.
Riffusion remained in this state for most of 2023. It was the first web app to offer AI generated text-to-music as a free service. There were no other products stepping up to the plate to rival them during the first trimester of 2023. It took nearly 6 months for the big tech companies (Google, Meta, and Stability AI) to make their move.
An AI music team at Google responded to Riffusion' December 2022 release with a TechCrunch press statement in January 2023. They linked to a demo of their MusicLM model, showing off advanced use cases that Riffusion didn't have.
Mid-2023: Google MusicLM, Meta MusicGen
A working demo of MusicLM didn’t reach the public until May 2023. It delivered higher quality audio and compositions, but lacked the exciting audio upload and style transfer features that had been promised by their demo page in January.
Meta swooped in a month later, making a press release in June about their own generative AI text-to-music model, called AudioCraft (MusicGen). They one upped Google by making AudioCraft open source, with a Github repo and API that developers could use to embed the service into their own interactive AI tools.
MusicGen was soon available on Hugging Face, and developers surfaced the same style transfer feature MusicLM had claimed to have but failed to deliver.
As of November 2023, MusicLM and MusicGen lack any AI voice features.
Late-2023: Stable Audio, Splash, and Chirp
Three months later, in September 2023, Stability AI announced a text-to-music web application called Stable Audio. Around this same time, companies like Chirp and Splash were surfaced with AI lyric-to-song generators. Riffusion's new website and features put them firmly in the running with the other rising stars. It helps that the service is completely free!
Visit Riffusion's website here: https://www.riffusion.com/