Captions

Captions are timed text on their own track, drawn by the engine with the same text renderer as text generators. This page explains caption tracks, cues, styles and word timing, and how to import SubRip and WebVTT files. After reading it you can subtitle a video and highlight words as they are spoken.

A caption track

A track with "kind": "caption" has no clips. Its captions hold:

cues: each a CaptionCue with a timeline_range and a text. Cues are in timeline order and don’t overlap; a track shows one cue at a time.
style: the TextStyle every cue starts from, the same style a text generator uses. By default, white text at 5% of the canvas’s shorter side, centred near the bottom.
styles: named patches of that style, at most 32. Each field a patch sets replaces the base’s value.

A caption track draws over the tracks before it and under the ones after it, so put it last to draw it on top.

{
  "id": "subs",
  "name": "Subtitles",
  "kind": "caption",
  "captions": {
    "style": {
      "font_family": "Hanken Grotesk",
      "font_size": 0.06,
      "background": { "color": { "red": 0, "green": 0, "blue": 0, "alpha": 0.6 }, "padding": 0.3 }
    },
    "styles": {
      "Narrator": {
        "fill": { "type": "solid", "color": { "red": 1, "green": 0.85, "blue": 0.3, "alpha": 1 } }
      },
      "karaoke": {
        "animators": [
          {
            "selector": { "type": "timing", "state": "upcoming" },
            "properties": { "opacity": { "mode": "constant", "value": 0.45 } }
          }
        ]
      }
    },
    "cues": [
      {
        "timeline_range": { "start": 0, "end": 240000 },
        "text": "The tide comes in twice a day."
      },
      {
        "timeline_range": { "start": 240000, "end": 480000 },
        "text": "Every word lands on time.",
        "speaker": "Narrator"
      },
      {
        "timeline_range": { "start": 480000, "end": 720000 },
        "style": "karaoke",
        "text": [
          { "text": "One ", "time": { "start": 0, "end": 60000 } },
          { "text": "word ", "time": { "start": 60000, "end": 120000 } },
          { "text": "at ", "time": { "start": 120000, "end": 168000 } },
          { "text": "a time.", "time": { "start": 168000, "end": 240000 } }
        ]
      }
    ]
  }
}

Choosing a style per cue

A cue picks one of the track’s styles in two ways:

style names it directly. The third cue uses karaoke.
speaker says who speaks (a WebVTT voice, for example). When a cue names no style, the track style with the speaker’s name applies. The second cue’s speaker is Narrator, so it is drawn in the Narrator style’s yellow.

Runs and karaoke

A cue’s text is a string, or a list of runs whose texts joined are the text. A run can carry class names and a time span, measured from the cue’s start and inside it.

A timing selector in a text animator selects characters by their run’s time against the current time: upcoming (not said yet), active (being said) or spoken. In the example, the karaoke style dims upcoming words to 45% opacity, so each word lights up when its time comes. ramp fades the change in over a duration instead of switching, and sweep spreads a run’s span over its characters so the highlight moves through a word letter by letter.

Animator keyframes in a caption style are in each cue’s own time, from its start.

Importing SRT and WebVTT

parseCaptions in alpha2/captions reads SubRip ("srt") or WebVTT ("webvtt") text with the engine’s own parser, and captionTrack builds a track from the cues:

import init, * as wasm from "alpha2/wasm";
import { captionTrack, parseCaptions } from "alpha2/captions";

await init();
const { rows, skipped, notes } = parseCaptions(wasm, vttText, "webvtt");
const tracks = rows.map((cues, index) =>
  captionTrack(`subtitles-${index}`, "Subtitles", cues, { font_family: "Hanken Grotesk" }),
);

Timestamps are exact: 1 ms is 120 ticks.
rows holds one caption track’s cues per row. Cues that overlap in the file go to different rows, since a track shows one cue at a time, and a row holds at most 4096 cues, the most a track holds.
A WebVTT voice sets the cue’s speaker. Voices, <c.class>, <b>, <i> and <u> become run classes, and inline timestamps split the cue into timed runs. Other markup is dropped.
skipped counts unreadable blocks, and notes counts markup the import read but could not keep.

For example, this WebVTT cue becomes four timed runs, the third cue of the example above:

00:00:04.000 --> 00:00:06.000
One <00:00:04.500>word <00:00:05.000>at <00:00:05.400>a time.

splitCue and trimCue split and narrow a cue, keeping and rebasing its run times. In the editor, importCaptions imports caption files in one undoable step. In Python, alpha2.compose.Captions takes SubRip, WebVTT, cues or word timestamps.

GeneratorsText styles and animators, shared with captions.

Schema: captionsCaptions and CaptionCue.

AudioMixing the sound under your captions.