> ## AI Agent Context
> Get the complete documentation index at: https://aistudio.google.com/llms.txt
> Use this file to discover all available pages.
> For the full documentation in a single file, fetch: https://aistudio.google.com/llms-full.txt

# Real-time music generation using Lyria RealTime

{% callout type="note" %}
  Lyria RealTime is an [experimental model](/docs/models#experimental).

{% /callout %}

The Gemini API, using
[Lyria Realtime](https://deepmind.google/technologies/lyria/realtime/),
provides access to a state-of-the-art, real-time, streaming music
generation model. It allows developers to build applications where users
can interactively create, continuously steer, and perform instrumental
music.

Lyria Realtime music generation uses a persistent, bidirectional,
low-latency streaming connection using
[WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API).

To experience what can be built using Lyria Realtime, try it on AI Studio
using the [Prompt DJ](https://aistudio.google.com/apps/bundled/promptdj) or the
[MIDI DJ](https://aistudio.google.com/apps/bundled/promptdj-midi) apps.

{% callout type="note" %}
  Looking for non-streaming music generation? See
  [Generate music with Lyria 3.5](/docs/music-generation).

{% /callout %}

## Generate and control music  {% id="generate-music" %}

Lyria Realtime works similarly to the [Multimodal Live API](/docs/live-api)
in that it uses Websockets to maintain real-time communication with the model.

The following code demonstrates how to generate music:

{% tabs %}
  {% tab filename="sample.py" label="Python" %}
  This example initializes the Lyria Realtime session using
    `client.aio.live.music.connect()`, then sends an
    initial prompt with `session.set_weighted_prompts()` along with an initial
    configuration using `session.set_music_generation_config`, starts the music
    generation using `session.play()` and sets up
    `receive_audio()` to process the audio chunks it receives.

  ```python
    import asyncio
    from google import genai
    from google.genai import types
  
    client = genai.Client(http_options={'api_version': 'v1beta'})
  
    async def main():
        async def receive_audio(session):
          """Example background task to process incoming audio."""
          while True:
            async for message in session.receive():
              audio_data = message.server_content.audio_chunks[0].data
              # Process audio...
              await asyncio.sleep(10**-12)
  
        async with (
          client.aio.live.music.connect(model='models/lyria-realtime-exp') as session,
          asyncio.TaskGroup() as tg,
        ):
          # Set up task to receive server messages.
          tg.create_task(receive_audio(session))
  
          # Send initial prompts and config
          await session.set_weighted_prompts(
            prompts=[
              types.WeightedPrompt(text='minimal techno', weight=1.0),
            ]
          )
          await session.set_music_generation_config(
            config=types.LiveMusicGenerationConfig(bpm=90, temperature=1.0)
          )
  
          # Start streaming music
          await session.play()
    if __name__ == "__main__":
        asyncio.run(main())
  ```

  Objective: For a more complete code sample, refer to the
    [Lyria RealTime - Get Started](https://colab.sandbox.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_LyriaRealTime.ipynb)
    file in the cookbooks repository.

  {% /tab %}

  {% tab filename="sample.js" label="JavaScript" %}
  This example initializes the Lyria Realtime session using
    `client.live.music.connect()`, then sends an
    initial prompt with `session.setWeightedPrompts()` along with an initial
    configuration using `session.setMusicGenerationConfig`, starts the music
    generation using `session.play()` and sets up an
    `onMessage` callback to process the audio chunks it receives.

  ```javascript
  import { GoogleGenAI } from "@google/genai";
  import Speaker from "speaker";
  
  const client = new GoogleGenAI({
    apiKey: GEMINI_API_KEY,
      apiVersion: "v1beta" ,
  });
  
  async function main() {
    const speaker = new Speaker({
      channels: 2,       // stereo
      bitDepth: 16,      // 16-bit PCM
      sampleRate: 44100, // 44.1 kHz
    });
  
    const session = await client.live.music.connect({
      model: "models/lyria-realtime-exp",
      callbacks: {
        onmessage: (message) => {
          if (message.serverContent?.audioChunks) {
            for (const chunk of message.serverContent.audioChunks) {
              if (chunk.data) {
                const audioBuffer = Buffer.from(chunk.data, "base64");
                speaker.write(audioBuffer);
              }
            }
          }
        },
        onerror: (error) => console.error("music session error:", error),
        onclose: () => console.log("Lyria RealTime stream closed."),
      },
    });
  
    await session.setWeightedPrompts({
      weightedPrompts: [
        { text: "Minimal techno with deep bass, sparse percussion, and atmospheric synths", weight: 1.0 },
      ],
    });
  
    await session.setMusicGenerationConfig({
      musicGenerationConfig: {
        bpm: 90,
        temperature: 1.0,
      },
    });
  
    await session.play();
  }
  
  await main();
  ```

  Objective: For a more complete code sample, try the
    [Prompt DJ](https://aistudio.google.com/apps/bundled/promptdj)
    and [MIDI DJ](https://aistudio.google.com/apps/bundled/promptdj-midi) apps on
    AI Studio

  You can then use `session.play()`, `session.pause()`, `session.stop()` and
    `session.reset_context()` to start, pause, stop or reset the session.

  {% /tab %}

{% /tabs %}

## Steer music in real-time  {% id="steer-music" %}

You can steer music generation in real-time by sending prompts and updating
generation parameters in real time.

### Prompt Lyria Realtime  {% id="prompting-lyria" %}

While the stream is active, you can send new `WeightedPrompt` messages at any
time to alter the generated music. The model will smoothly transition based
on the new input.

The prompts need to follow the right format with a `text` (the
actual prompt), and a `weight`. The `weight` can take any value except `0`. `1.0`
is usually a good starting point.

{% tabs %}
  {% tab filename="sample.py" label="Python" %}
  ```python
    from google.genai import types
  
    await session.set_weighted_prompts(
      prompts=[
        {"text": "Piano", "weight": 2.0},
        types.WeightedPrompt(text="Meditation", weight=0.5),
        types.WeightedPrompt(text="Live Performance", weight=1.0),
      ]
    )
  ```

  {% /tab %}

  {% tab filename="sample.js" label="JavaScript" %}
  ```javaScript
    await session.setWeightedPrompts({
      weightedPrompts: [
        { text: 'Harmonica', weight: 0.3 },
        { text: 'Afrobeat', weight: 0.7 }
      ],
    });
  ```

  Note that the model transitions can be a bit abrupt when drastically changing
    the prompts so it's recommended to implement some kind of cross-fading by
    sending intermediate weight values to the model.

  {% /tab %}

{% /tabs %}

### Update the configuration  {% id="steer-config" %}

You can steer music generation by updating the music generation parameters in
real time. You can't just update a parameter, you need to set the whole
configuration otherwise the other fields will be reset back to their default
values.

Since updating the bpm or the scale is a drastic change for the model you'll
also need to tell it to reset its context using `reset_context()` to take the
new config into account. It won't stop the stream, but it will be a hard
transition. You don't need to do it for the other parameters.

{% tabs %}
  {% tab filename="sample.py" label="Python" %}
  ```python
    from google.genai import types
  
    await session.set_music_generation_config(
      config=types.LiveMusicGenerationConfig(
        bpm=128,
        scale=types.Scale.D_MAJOR_B_MINOR,
        music_generation_mode=types.MusicGenerationMode.QUALITY
      )
    )
    await session.reset_context();
  ```

  {% /tab %}

  {% tab filename="sample.js" label="JavaScript" %}
  ```javaScript
    await session.setMusicGenerationConfig({
      musicGenerationConfig: {
        bpm: 120,
        density: 0.75,
        musicGenerationMode: MusicGenerationMode.QUALITY
      },
    });
    await session.reset_context();
  ```

  {% /tab %}

{% /tabs %}

## Prompting Lyria RealTime  {% id="prompting-lyria-realtime" %}

Lyria RealTime uses weighted prompts to blend musical genres, instruments, and moods dynamically. To explore prompt steering strategies, keyword tag vocabularies, and full prompt examples, see the [Lyria prompt guide](/docs/lyria-prompt-guide#realtime-prompting).

## Best practices  {% id="best-practices-music" %}

* Client applications must implement robust audio buffering to ensure smooth
  playback. This helps account for network jitter and slight variations in
  generation latency.
* Effective prompting:  * Be descriptive. Use adjectives describing mood, genre, and instrumentation.
  * Iterate and steer gradually. Rather than completely changing the prompt,
    try adding or modifying elements to morph the music more smoothly.
  * Experiment with weight on `WeightedPrompt` to influence how strongly a new
    prompt affects the ongoing generation.


## Technical details  {% id="technical-details-music" %}

This section describes the specifics of how to use Lyria Realtime music
generation.

### Specifications  {% id="audio-specs" %}

* Output format: Raw 16-bit PCM Audio
* Sample rate: 48kHz
* Channels: 2 (stereo)

### Controls  {% id="controls" %}

Music generation can be influenced in real time by sending messages containing:

* `WeightedPrompt`: A text string describing a musical idea, genre, instrument,
  mood, or characteristic. Multiple prompts can potentially be supplied to blend
  influences. See [above](/docs/realtime-music-generation#steer-music) for more details on how to best prompt
  Lyria Realtime.
* `MusicGenerationConfig`: Configuration for the music generation process,
  influencing the characteristics of the output audio.). Parameters
  include:  * `guidance`: (float) Range: `[0.0, 6.0]`. Default: `4.0`.
    Controls how strictly the model follows the prompts. Higher guidance
    improves adherence to the prompt, but makes transitions more abrupt.
  * `bpm`: (int) Range: `[60, 200]`.
    Sets the Beats Per Minute you want for the generated music. You need to
    stop/play or reset the context for the model it take into account the new
    bpm.
  * `density`: (float) Range: `[0.0, 1.0]`.
    Controls the density of musical notes/sounds. Lower values produce sparser
    music; higher values produce "busier" music.
  * `brightness`: (float) Range: `[0.0, 1.0]`.
    Adjusts the tonal quality. Higher values produce "brighter" sounding
    audio, generally emphasizing higher frequencies.
  * `scale`: (Enum)
    Sets the musical scale (Key and Mode) for the generation. Use the
    [`Scale` enum values](/docs/realtime-music-generation#scale-enum) provided by the SDK. You need to
    stop/play or reset the context for the model it take into account the new
    scale.
  * `mute_bass`: (bool) Default: `False`.
    Controls whether the model reduces the outputs' bass.
  * `mute_drums`: (bool) Default: `False`.
    Controls whether the model outputs reduces the outputs' drums.
  * `only_bass_and_drums`: (bool) Default: `False`.
    Steer the model to try to only output bass and drums.
  * `music_generation_mode`: (Enum)
    Indicates to the model if it should focus on `QUALITY` (default value) or
    `DIVERSITY` of music. It can also be set to `VOCALIZATION` to let the
    model generate vocalizations as another instrument (add them as new
    pompts).

* `PlaybackControl`: Commands to control playback aspects, such as play, pause,
  stop or reset the context.

For `bpm`, `density`, `brightness` and `scale`, if no value is provided, the
model will decide what's best according to your initial prompts.

More classical parameters like `temperature` (0.0 to 3.0, default 1.1), `top_k`
(1 to 1000, default 40), and `seed` (0 to 2 147 483 647, randomly selected by
default) are also customizable in the `MusicGenerationConfig`.

#### Scale Enum Values  {% id="scale-enum" %}

Here are all the scale values that the model can accept:

| Enum Value | Scale / Key |
| :--- | :--- |
| `C_MAJOR_A_MINOR` | C major / A minor |
| `D_FLAT_MAJOR_B_FLAT_MINOR` | D♭ major / B♭ minor |
| `D_MAJOR_B_MINOR` | D major / B minor |
| `E_FLAT_MAJOR_C_MINOR` | E♭ major / C minor |
| `E_MAJOR_D_FLAT_MINOR` | E major / C♯/D♭ minor |
| `F_MAJOR_D_MINOR` | F major / D minor |
| `G_FLAT_MAJOR_E_FLAT_MINOR` | G♭ major / E♭ minor |
| `G_MAJOR_E_MINOR` | G major / E minor |
| `A_FLAT_MAJOR_F_MINOR` | A♭ major / F minor |
| `A_MAJOR_G_FLAT_MINOR` | A major / F♯/G♭ minor |
| `B_FLAT_MAJOR_G_MINOR` | B♭ major / G minor |
| `B_MAJOR_A_FLAT_MINOR` | B major / G♯/A♭ minor |
| `SCALE_UNSPECIFIED` | Default / The model decides |

The model is capable of guiding the notes that are played, but does
not distinguish between relative keys. Thus each enum corresponds both to the
relative major and minor. For example, `C_MAJOR_A_MINOR` would correspond to all
the white keys of a piano, and `F_MAJOR_D_MINOR` would be all the white keys
except B flat.

### Limitations  {% id="limitations" %}

* Instrumental only: The model generates instrumental music only.
* Safety: Prompts are checked by safety filters. Prompts triggering the filters
  will be ignored in which case an explanation will be written in the output's
  `filtered_prompt` field.
* Watermarking: Output audio is always watermarked for identification following
  our [Responsible AI](https://ai.google/responsibility/principles/) principles.

## What's next  {% id="next-steps" %}

* See the [Lyria prompt guide](/docs/lyria-prompt-guide) for prompting strategies and keyword vocabularies,
* Generate full songs and vocal tracks with [Lyria 3.5](/docs/music-generation),
* Instead of music, learn how to generate multi-speakers conversation using
  the [TTS models](/docs/speech-generation),
* Discover how to generate [images](/docs/image-generation) or [videos](/docs/video),
* Instead of generation music or audio, find out how to Gemini can
  [understand Audio files](/docs/audio),
* Have a real-time conversation with Gemini using the
  [Multimodal Live API](/docs/live-api).

Explore the [Cookbook](https://github.com/google-gemini/cookbook) for more
code examples and tutorials.