Audio

This content is for v0.4. Switch to the latest version for up-to-date documentation.

The engine mixes a project’s audio as it renders its pictures, from the same document. This page follows the signal from a clip to the master bus, then shows how to hit a loudness target and how the browser feeds and plays the mix. After reading it you can balance dialogue and music in JSON.

Where sound comes from

An audio source on an audio track.
A video source with "has_audio": true. Its clips on a video track add their sound to the mix. Without the flag they are silent.

A clip’s time_map applies to its sound too: reverse plays it backwards, freeze is silent, and a clip at a speed other than 1 is resampled, so its pitch changes with its speed. A composition’s media nodes add no sound.

The signal path

clip: gain, pan, fades, effects  →  track bus: gain, pan, effects, mute, solo, ducking  →  master: gain, effects, limiter

Level	Field	Controls
Clip	`audio`	`gain`, `pan`, `fade_in`, `fade_out`, `fade_shape`, `effects`, `muted`
Track	`audio`	`gain`, `pan`, `effects`, `muted`, `solo`, `ducking`
Project	`audio`	`gain`, `effects`, `limiter`

Gain is linear amplitude, from 0 to 16, and animatable: 1 leaves the level as it is, 0.5 halves it.
Pan is the stereo position, from -1 (left) to 1 (right), and animatable.
Fades are durations in ticks, fade_in from the clip’s start and fade_out back from its end. fade_shape is linear (two linear fades over one overlap dip by 6 dB) or equal_power (a quarter sine; two of them crossfade at constant power).
Solo on any track restricts the mix to soloed tracks.

Gain and pan keyframes on a clip or track are values like any other; see Animation.

Effects

Each chain holds at most 16 effects, applied in order:

eq: 1 to 16 biquad bands, each peaking, low_pass or high_pass, with frequency_hz, q and (for peaking) gain_db.
compressor: lookahead compression with threshold_db, ratio, attack_ms, release_ms and lookahead_ms.
reverb: Schroeder reverberation with room_size, damping and mix.

Ducking

Ducking lowers one track while another plays. It is written on the track that is attenuated: sidechain_track names the track whose mixed output drives the envelope, threshold (linear) is where attenuation begins, depth_db how far it goes, and attack_ms and release_ms how fast. A track cannot duck itself, and ducking chains must not form a cycle.

The limiter

The master bus ends in a peak limiter. When you leave limiter out, it is { "threshold": 1, "release_ms": 50 } (a ceiling at full scale). An explicit "limiter": null leaves the mix unlimited.

An example mix

Dialogue from an interview video, a music bed that ducks under it, and a compressor and limiter on the master:

{
  "schema_version": 1,
  "id": "audio-mix",
  "name": "Audio mix",
  "settings": { "width": 1280, "height": 720, "frame_rate": { "num": 30, "den": 1 }, "sample_rate": 48000 },
  "sources": [
    {
      "id": "interview",
      "kind": "video",
      "locator": "file:///media/interview.mp4",
      "has_audio": true
    },
    { "id": "music", "kind": "audio", "locator": "file:///media/music.m4a" }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "dialogue",
        "name": "Dialogue",
        "kind": "video",
        "clips": [
          {
            "id": "interview-clip",
            "source_id": "interview",
            "timeline_range": { "start": 0, "end": 2400000 },
            "source_range": { "start": 600000, "end": 3000000 },
            "audio": {
              "gain": { "mode": "constant", "value": 1.2 },
              "fade_in": 30000,
              "effects": [
                {
                  "type": "eq",
                  "bands": [{ "kind": "high_pass", "frequency_hz": 80, "q": 0.7 }]
                }
              ]
            }
          }
        ]
      },
      {
        "id": "music-bed",
        "name": "Music",
        "kind": "audio",
        "clips": [
          {
            "id": "music-clip",
            "source_id": "music",
            "timeline_range": { "start": 0, "end": 2400000 },
            "source_range": { "start": 0, "end": 2400000 },
            "audio": {
              "fade_in": 120000,
              "fade_out": 240000,
              "fade_shape": "equal_power",
              "pan": { "mode": "constant", "value": -0.2 }
            }
          }
        ],
        "audio": {
          "gain": { "mode": "constant", "value": 0.5 },
          "ducking": {
            "sidechain_track": "dialogue",
            "threshold": 0.05,
            "depth_db": -12,
            "attack_ms": 20,
            "release_ms": 300
          }
        }
      }
    ]
  },
  "audio": {
    "gain": { "mode": "constant", "value": 1 },
    "effects": [
      {
        "type": "compressor",
        "threshold_db": -18,
        "ratio": 3,
        "attack_ms": 10,
        "release_ms": 150
      }
    ],
    "limiter": { "threshold": 0.9, "release_ms": 50 }
  }
}

The interview clip skips the first 5 s of its file, gains 1.2, fades in over a quarter second, and runs through a high-pass filter at 80 Hz.
The music fades in over 1 s and out over 2 s with equal-power curves, sits slightly left, and its track plays at half gain, ducking by up to 12 dB when the dialogue track rises above 0.05.
The master compresses gently and limits at 0.9.

Loudness

alpha2/loudness measures loudness as ITU-R BS.1770-4 and EBU R128 define it: integrated loudness (LUFS), loudness range, true peak and sample peak, and momentary and short-term maxima. It also plans the static gain that brings a mix to a target.

Native exports take a loudness target and do both steps for you: loudness: { targetLufs, truePeakCeilingDbtp } in @alpha2-sh/node, loudness={"target_lufs": -14, ...} in Python, and --loudness-target <LUFS> with --true-peak-ceiling <dBTP> on the command line. The target is -60 to 0 LUFS, the ceiling -20 to 0 dBTP.

In the browser, you run the three steps yourself:

Call engine.setLoudnessMetering(true) and render the audio once with renderAudio.
Pass JSON.parse(engine.loudnessReport()) to planLoudnessNormalization(wasm, report, targetLufs, ceilingDbtp).
Render the export with engine.setNormalizationGainDb(plan.gainDb).

The plan keeps the mix under the master limiter’s threshold, so the limiter does not change the planned loudness.

Audio in the browser

In the browser, your app decodes audio and uploads it; the engine mixes.

// Decoded samples of one source, resampled to the project's sample rate, planes concatenated.
engine.uploadAudioBlock("music", 0n, 48000, 2, samples, true /* this upload reaches the end */);

const block = engine.renderAudio(0n, 4800, true); // up to engine.audioMaxBlock samples per channel
const missing = JSON.parse(engine.audioMissingSources()); // sources this block still needs

uploadAudioBlock(sourceId, startSample, sampleRate, layout, samples, endsSource) takes planar audio at the project’s sample rate. startSample is the source sample of the first sample, and layout is a channel count or the speaker of each plane.
Missing samples mean audio that hasn’t arrived yet, not silence, so a mix needs every sample of its block. Upload zeros past a source’s end, or pass endsSource.
audioNeeds(startSample, frames) lists the source intervals a block needs. A block that lacks some is reported by audioMissingSources rather than mixed with a gap.
renderAudio returns the mixed channels concatenated.

For preview, alpha2/audio-timeline moves mixed blocks to the audio thread through a ring in shared memory, and alpha2/audio-worklet plays them in an AudioWorklet. Shared memory needs a cross-origin isolated page; canShareAudioMemory() tells you whether you have one. For export, alpha2/audio renders audio offline into a sink. alpha2/waveform computes waveform summaries for a timeline view.

Schema: audioEvery audio field, limit and default.

Browser supportCross-origin isolation and the worklet.

DeterminismThe mix is the same natively and in WebAssembly.