Text-to-music systems differ in detail, but most share the same basic ideas.
Turning audio into something a model can learn
Raw audio is a very long list of numbers, tens of thousands per second. Most systems first compress audio into a more compact representation, either a sequence of discrete "audio tokens" produced by a neural audio codec, or a compressed continuous representation called a latent space.
Learning from examples
During training, the model sees large numbers of audio examples paired with text descriptions, such as genre tags, captions or lyrics. It learns which sounds tend to go with which words: that "lo-fi" often comes with soft drums and dusty keys, or that "orchestral" comes with strings and brass.
Two common ways to generate
- Autoregressive transformers predict audio tokens one after another, a bit like predicting the next word in a sentence. Each new token depends on the prompt and everything generated so far.
- Diffusion models start from noise and gradually refine it into audio that matches the prompt, over many small steps.
Some systems combine both, for example generating structure first and detail second.
Why prompts matter so much
The model only knows what you tell it. A prompt with genre, tempo, instruments and structure narrows down the possibilities, so the output is more likely to match what you imagined. That is why prompt writing makes such a difference.
Limits to expect
- Long-range structure is harder than short-range sound; very long tracks can drift.
- Exact timing requests are followed approximately, not to the frame.
- Rare genres and instruments are harder because the model has seen fewer examples.
Where training data comes from
The source and licensing of training data is one of the most important questions in AI music. We cover it in AI music training data explained.
Written by the Sonvela editorial team with AI drafting assistance, then checked against the sources linked in the text. Spotted an error? Tell us and we will correct it.