Music and audio generation
Models can now make songs, sound effects and background music from a few words. Learn how they work in general, and the hard questions about artists, consent and copyright.
In 60 seconds
Music and audio generation
Models can now make songs, sound effects and background music from a few words. Learn how they work in general, and the hard questions about artists, consent and copyright.
How it works, in general
| Approach | Idea | Links to earlier lessons |
|---|---|---|
| Audio tokens | A neural audio codec turns sound into a sequence of tokens. A transformer predicts the next audio token, like an LLM predicts the next word. The codec turns tokens back into sound. | Like a language model, but the "words" are pieces of sound |
| Diffusion on audio | Start from noise in a compressed audio space (or a spectrogram) and clean it step by step, guided by the text prompt. | Like image diffusion, but the "picture" is sound over time |
What is hard about music
- Long structure: a song has verses, a chorus that returns, and a build-up. Keeping this over minutes is harder than making a few good seconds.
- Several things at once: melody, rhythm, harmony, instruments and voice must all fit together.
- Styles with less data: Carnatic ragas, folk forms and regional instruments may be less well learned than globally popular styles.
- Singing words clearly: lyrics can be slurred or mispronounced, especially in less-represented languages.
The consent and copyright questions
- 1
Training data
Were the songs licensed, or collected without permission? Were artists asked? - 2
Style vs copy
Sounding "like" a genre is normal in music. Reproducing a recognisable melody or recording is a different matter. - 3
Voice and likeness
A model that sings in the voice of a real singer raises personality and consent questions. Indian courts have, in some cases, protected well-known people's voice and likeness from unauthorised AI use. - 4
Ownership of output
Who owns an AI-made track? Rules vary by country, and many say a human author is needed for copyright. - 5
Fair pay
Should artists whose work trained a model share in what it earns?
- "Make a new song in the voice of [famous singer]"
- Uploading an AI track as your own work with no label
- Asking for "the tune of" a specific film song
- Original background music for your own video, under a tool licence you have read
- Labelling AI-made music honestly when you share it
- Working with a musician who agrees and is credited
Watch and read more
Lab
A checklist for deciding whether a piece of AI music is fine to use, tested on three cases.
The problem
You run social media for a small café in Pune and want background music for short videos.
- Write a checklist of 5 to 7 yes-or-no questions to ask before using an AI-made track. Cover: the tool's licence, commercial use, voices of real people, recognisable tunes, labelling, and keeping records.
- Apply your checklist to three cases: (a) an original instrumental track made with a tool whose licence allows commercial use on your plan; (b) a track made by asking for "a song in the voice of a famous playback singer"; (c) a track that closely follows the tune of a popular film song.
- For each case, write your decision (use, do not use, or ask a professional) and one line of reasoning.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why is "it only learned the style" not a complete answer to artists' concerns?Reveal
Because artists' concerns are wider than copying one song. They include whether their recordings were used to train the model without consent or payment, whether the model can imitate their recognisable voice, and whether cheap AI tracks will replace paid work in areas like advertising jingles and background music. Even if no single output copies a song, these questions about consent, credit and fair pay remain.
Q2How could an AI music tool be designed to respect artists better?Reveal
Train on licensed or public-domain music and music from artists who opted in, and say so publicly. Block prompts that name real artists' voices unless the artist has joined a consent programme. Check outputs for close matches with known recordings before release. Share revenue with contributing artists. Add a watermark or content credentials so AI tracks can be identified.
Questions people ask
Can I use AI music in my YouTube video?
It depends on the tool's licence terms and the platform's rules, which change. Read both. Some tools allow commercial use on some plans and not others. Keep a record of how you made the track.
Is it copying if the AI music sounds like a famous style?
Styles and genres are generally not owned by anyone. But closely reproducing a specific melody, lyrics or recording can raise real copyright issues, whether a person or a machine does it.
How do musicians use these tools well?
For sketching ideas, making demo tracks, generating sound effects, separating instruments from a mix, or practising with a backing track. Many see AI as one more instrument, and keep creative control themselves.
Can AI-made music be detected?
Sometimes, through watermarks added by the tool or through detection models, but neither is fully reliable. Honest labelling by the person sharing it matters most.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.