Deep neural networks like Google Magenta and OpenAI’s Musenet are using machine learning to attempt something similar, but with music. Instead of turning words into pictures, they use MIDI files as their prompts and compose new sections with artificial intelligence. The final render is in MIDI format, so you can import it back to your DAW for further refinement.
On January 26th 2023, Google announced the release of an upcoming text-to-music generator called MusicLM. This seems to imply that advanced AI music generation is right around the corner.
In this article, I’ll show you a simple method for making music using artificial intelligence. We’ll go over some of the recent history of ai generated music, with a focus on Google and OpenAI. I’ll share a quick tutorial on how to use Musenet, augment the experience with AudioCipher VST, and scale up your output with Steven Waterman’s open source MuseTree app.
Please keep in mind that this article was written prior to Musenet being turned off. Some of the statements will be made as if the software works, which to our knowledge it currently does not. Musetree is not usable for this same reason.
What is AI generated music?
Google has been flirting with ai generated music for years. They launched Magenta in 2016, taking initial steps toward a space that OpenAI’s Musenet later joined. An AI music Doodle was published in March 2019, letting Google homepage visitors write a short melody in their browser. The application’s artificial intelligence generates a four-part Bach chorale based on your tune.
When people sit down to make a song, they draw from sounds they’ve heard and enjoyed in the past. This is not so different from the way deep learning algorithms create their music compositions, albeit a faster, more rigorous training process.
Musenet is built on a deep neural network with transformer kernels from Sparse Transformer. It analyzes MIDI files and builds an internal musical vocabulary to model its training data. With similar unsupervised technology as GPT-2, you could say that the music generation algorithms have their own maps of harmonic theory that were not directly programmed by a human.
Musenet’s dataset is somewhat diverse, with sixteen musical “styles” in total. It leans into classical music like Mozart, Bach, Chopin or Rachmaninoff. If you want something more modern, you can choose from pop artists like The Beatles and Lady Gaga. There’s a video game option too - but is that a style of music?
Generating MIDI Music with MuseNet
To kick off your AI music composition journey, I recommend popping open the Musenet website and scrolling down to this strip. You’ll want to enter advanced mode by clicking on show advanced settings:
From here, select a style (genre), intro (song snippet), and some different instruments for the AI to compose with. You can also increase or decrease the number of tokens with the slider, to change the length of the final musical output. When you’re ready, click the arrow next to your MIDI snippet to run the app:
Musenet works best when your instrument selection matches the style and intro you picked. Within about a minute, you’ll have four expanded versions to listen to.
Turning AudioCipher melodies into full songs
Imagine if Musenet could turn everyday phrases into music, the way Dalle 2 does with words. It's not exactly possible yet, but there's a creative workaround you can try. For this technique you’ll need a copy of the AudioCipher MIDI plugin.
Turn AudioCipher melodies into full songs with three simple steps
Type a phrase into AudioCipher, choose your key signature and rhythm settings, and drag it onto a virtual instrument. Make any changes you want to the rhythm, notes, etc.
Export the MIDI track to a folder on your hard drive
Import the MIDI file with Musenet’s Custom MIDI Upload feature
Musenet’s output for your AudioCipher track will be shaped by the style and instruments that you select. Here's what the generated music could look like:
Once you hear something you like, click the download icon to grab the MIDI file. Bring this back into your DAW to make any edits, from quantization to melodic changes. This chord progression only exists because of your initial snippet, which means it's yours to use and transform however you'd like!
MuseTree: An alternative interface
If you want to go deeper with these tools, I recommend the free browser application MuseTree. Running on the same neural network as Musenet, their custom interface lets you generate and manage a higher volume of tracks at once, with way more genres to choose from. You can create variations in bulk, pick ones you like and build a whole tree of MIDI sequences.
Along with the nice interface, here are some benefits you get from MuseTree:
Full tree:
Generate more than 4 nodes at a time
Develop multiple MIDI variations at once
More generation options:
604 genres available
Generate up to 1000 tokens at a time
Full persistence options:
Save/Load the full tree to work on later
Export wav audio
Import/Export midi
The option to save Musenet projects in MuseTree is a game changer. It means you can generate music iteratively, over some period of days or weeks, and come back to it any time. It's like having a peculiar MIDI DAW that works in your browser.
Of course, the best use of this technology is to create your own songs. So here's a step by step tutorial on adding your own MIDI files to MuseTree.
How to add custom MIDI to MuseTree:
The types of prompts you can use in MuseTree are limitless. From AudioCipher MIDI files, songs you wrote and recorded in a DAW, or MIDI files you find on the internet from some other artist. Really, anything is fair game for your experiments.
Your first project will be empty, aside from a single root node.
Right click a node to access the add MIDI option. You'll want to upload a short clip (8-12 seconds) in order to avoid long load times in the app.
On the import screen, select Upload MIDI and pull in a file. You'll see some lines of MIDI code appear, which you can ignore. Hit the import button to assign the MIDI track to that node.
Now that your MIDI track has been assigned to the node, right click it again and hit the Request More option.
This will allow you to generate a set of four new nodes. You can use the request more option several times in a row to rapidly iterate through as many variations as you want.
Each version will follow the guidelines of your settings in the right panel, however they're usually different from one another to be interesting.
If you enjoy the idea of making music with artificial intelligence and want to keep exploring, check out this article on the best AI music apps of 2022. These apps have advanced interfaces with controls for tempo, key signature, and more.
Update (4/3/2023): The Musenet API went down around the same time ChatGPT went public and we are waiting to find out whether it has been temporarily or permanently decommissioned.
A new AI Music DAW called WavTool recently dropped, so if you're interested in generating MIDI with AI, that's the best tool we can recommend currently.