The project model

This page walks through the parts of a project document and how they refer to each other. After reading it you can read any project and know where each field belongs.

The shape of a document

Directoryproject
- schema_version always 1
- id, name
- settings canvas size, frame rate, sample rate, background, color
- sources[] media, generated or composition sources
- compositions[] graphs of image operations
- effect_resources[] LUTs that effects read
- Directorytimeline
  - Directorytracks[] video, audio or caption
    clips[] a source over a time range
    transitions[] between overlapping clips
    captions cues and style, on a caption track
- audio the master bus

schema_version, id, name, settings, sources and timeline are required. Every other field has a default and can be left out. Unknown fields are refused.

An annotated example

This project has three sources, a composition and three tracks of different kinds:

{
  "schema_version": 1,
  "id": "model-tour",
  "name": "Model tour",
  "settings": { "width": 1280, "height": 720, "frame_rate": { "num": 30, "den": 1 }, "sample_rate": 48000 },
  "sources": [
    { "id": "coast", "kind": "image", "locator": "asset://coast" },
    { "id": "music", "kind": "audio", "locator": "asset://music" },
    {
      "id": "label",
      "kind": "generated",
      "generator": {
        "type": "text",
        "text": "Coast",
        "style": { "font_family": "Hanken Grotesk", "font_size": 0.08 }
      }
    }
  ],
  "compositions": [
    {
      "id": "soft",
      "nodes": [
        { "id": "in", "op": { "type": "clip" } },
        {
          "id": "blur",
          "op": {
            "type": "effect",
            "input": "in",
            "effect": { "id": "alpha2.effect.gaussian-blur", "version": 1 }
          }
        }
      ],
      "output": "blur"
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "picture",
        "name": "Picture",
        "kind": "video",
        "clips": [
          {
            "id": "coast-clip",
            "source_id": "coast",
            "timeline_range": { "start": 0, "end": 480000 },
            "source_range": { "start": 0, "end": 480000 },
            "composition": { "id": "soft" }
          },
          {
            "id": "label-clip",
            "source_id": "label",
            "timeline_range": { "start": 480000, "end": 720000 },
            "source_range": { "start": 0, "end": 240000 }
          }
        ]
      },
      {
        "id": "subtitles",
        "name": "Subtitles",
        "kind": "caption",
        "captions": {
          "style": { "font_family": "Hanken Grotesk" },
          "cues": [
            {
              "timeline_range": { "start": 60000, "end": 360000 },
              "text": "Waves on the coast."
            }
          ]
        }
      },
      {
        "id": "score",
        "name": "Score",
        "kind": "audio",
        "clips": [
          {
            "id": "music-clip",
            "source_id": "music",
            "timeline_range": { "start": 0, "end": 720000 },
            "source_range": { "start": 1200000, "end": 1920000 }
          }
        ]
      }
    ]
  }
}

Reading it top to bottom:

settings: a 1280x720 canvas at 30 fps, mixing audio at 48 kHz. The frame rate is a fraction (num/den), so 29.97 fps is { "num": 30000, "den": 1001 }.
sources: coast is an image and music an audio file; both have a locator the host resolves. label is generated: the engine draws it from its generator, so it has no locator.
compositions: soft is a graph with two nodes: the clip’s own picture, then a Gaussian blur of it. Its output names the node the clip shows.
Track picture (kind video): coast-clip shows the photo for four seconds through soft. label-clip follows it and shows the text for two seconds.
Track subtitles (kind caption): one cue, from 0.5 s to 3 s. A caption track has cues instead of clips.
Track score (kind audio): music-clip plays the music from 10 s into the file (source_range), starting at the timeline’s start (timeline_range).

Settings

ProjectSettings are output-wide: width and height in pixels, frame_rate, sample_rate, plus background (the opaque color under every layer, black by default), blend_space (where blend modes are evaluated, linear_light by default) and color (how HDR sources are treated; see Color and HDR).

Sources

A source is a resource that clips refer to by id. Its kind says what it is:

`kind`	What it is	Who supplies it
`video`	A video stream, possibly with audio. Set `has_audio: true` for its clips on a video track to add to the mix.	The host decodes it.
`audio`	An audio stream.	The host decodes it.
`image`	A still image.	The host decodes it.
`generated`	A fill, shapes, text or layout picture, from its `generator`.	The engine draws it. See Generators.
`composition`	One of the project’s compositions, used as a source, drawn with the source time as its own time.	The engine draws it; the host supplies the media its nodes read.

The locator is an opaque URI the engine never opens. In the browser your app uploads frames under the source’s id; natively you map ids to files (--asset coast=coast.jpg).

A source can also carry a provider (SVG, animated image, Lottie, OpenEXR, image sequence), naming the extension that supplies its frames when the host’s own decoders don’t read the format, a fingerprint of the media the project recorded, and color_metadata that overrides the color tags of a video.

Timeline, tracks and clips

The timeline is a list of tracks. Visual tracks are drawn back to front in list order. A track has one kind:

video: composited pictures. Clips may show video, image, generated or composition sources.
audio: mixed sound.
caption: timed text, drawn over the tracks before it and under the ones after. It has captions and no clips. See Captions.

A clip shows a source over two ranges:

timeline_range: where it sits on the project timeline.
source_range: which part of the source it reads.

Both are half-open tick ranges, [start, end). When their lengths differ, the clip plays faster or slower; time_map can also play it in reverse, freeze one instant or loop. See Time and ticks.

A track shows one clip at a time. Clips on one track may overlap only where a transition joins them. For layers, use one track per layer, or one composition that merges them.

Beyond its ranges, a visual clip can have visual (position, anchor, scale, rotation, opacity), composition, masks, a matte, a blend_mode and a cache hint. Every clip has audio controls.

Compositions

A composition is an acyclic graph of image operations with one output. A clip uses one through "composition": { "id": ... }; without one, a clip shows its source as it is. Nodes can take the clip’s picture, read another source, draw a generator, apply effects and transitions, place, merge, matte and mask. See Effects and compositions.

Effect resources

effect_resources are non-media data that effects read. The one kind today is lut3d, a 3D LUT normally authored as a .cube file. An effect names it in its resources, and the host supplies the file (--effect-asset on the command line, effectAssets in Node.js).

Master audio

The top-level audio is the master bus: gain, effects and a final peak limiter applied to the sum of every track. See Audio.

Time and ticksTicks, ranges, the frame grid and time maps.

AnimationConstant and keyframed values, and whose time they use.

Schema: documentEvery field of the document and its settings.

Schema: timelineTracks, clips, ranges, masks and mattes.