Captions from an SRT file

After this recipe you can import an .srt or .vtt file into a caption track, style it, and highlight each word as it is spoken. The engine parses the file, so the browser, Node.js and Python read the same cues from it.

Import the file

A subtitle file like this one:

talk.srt

1
00:00:00,300 --> 00:00:02,000
Every frame is rendered on the GPU.

2
00:00:02,200 --> 00:00:04,000
The same project, in every target.

parseCaptions from alpha2/captions parses it with the engine’s parser in WebAssembly, and captionTrack wraps the cues in a track:

import init, * as wasm from "alpha2/wasm";
import { captionTrack, parseCaptions } from "alpha2/captions";

await init();
const text = await (await fetch("/talk.srt")).text();
const { rows, skipped, notes } = parseCaptions(wasm, text, "srt"); // or "webvtt" for a .vtt file

const style = {
  font_family: "Hanken Grotesk",
  font_size: 0.055,
  background: { color: "#000000b3", padding: 0.3 },
};
const tracks = rows.map((cues, i) => captionTrack(`subs-${i}`, "Subtitles", cues, style));
project.timeline.tracks.push(...tracks);

• rows holds one track’s worth of cues per row. Cues in a row never overlap, so a file with overlapping cues gives more than one row. Add one caption track per row.
• skipped counts blocks the parser could not read. notes counts markup it read but could not keep.
• Timestamps are exact: 1 ms is 120 ticks. 00:00:00,300 becomes 36000.

The track this builds, in a project over a photo:

{
  "schema_version": 1,
  "id": "captions-srt",
  "name": "Captions from SRT",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": {
      "num": 30,
      "den": 1
    },
    "sample_rate": 48000
  },
  "sources": [
    {
      "id": "forest",
      "locator": "asset://forest",
      "kind": "image"
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "picture",
        "name": "Picture",
        "kind": "video",
        "clips": [
          {
            "id": "photo",
            "source_id": "forest",
            "timeline_range": {
              "start": 0,
              "end": 540000
            },
            "source_range": {
              "start": 0,
              "end": 540000
            }
          }
        ],
        "transitions": []
      },
      {
        "id": "subs",
        "name": "Subtitles",
        "kind": "caption",
        "clips": [],
        "captions": {
          "style": {
            "font_family": "Hanken Grotesk",
            "font_size": 0.055,
            "background": {
              "color": "#000000b3",
              "padding": 0.3
            }
          },
          "cues": [
            {
              "timeline_range": {
                "start": 36000,
                "end": 240000
              },
              "text": "Every frame is rendered on the GPU."
            },
            {
              "timeline_range": {
                "start": 264000,
                "end": 480000
              },
              "text": "The same project, in every target."
            }
          ]
        }
      }
    ]
  }
}

A caption track has kind: "caption", no clips, and a captions object with a style and the cues. The style fields you leave out take their defaults: centred, anchored at its bottom centre (anchor 0.5, 1) at position 0.5, 0.92, white, font_size 0.05 of the picture’s shorter side, wrapped at max_width 0.9. Here the style adds a font, a slightly larger size and a background, a box behind each line whose padding is a fraction of the font size.

Note

Captions draw with the fonts the renderer has, never system fonts. Give the export the same font file as the preview (--font on the CLI, fonts in Node.js and Python), or the text draws in another font and may break onto other lines.

Highlight the active word

SRT has one time per cue. For word-by-word highlighting, each word needs its own time. WebVTT carries word times as inline timestamps:

talk.vtt

WEBVTT

00:00:00.300 --> 00:00:02.000
Every <00:00:00.700>frame <00:00:01.100>is <00:00:01.300>rendered <00:00:01.700>live.

parseCaptions(wasm, text, "webvtt") splits such a cue into runs, each with a time relative to the cue’s start. A style animator with a timing selector then changes the characters of runs by their state: upcoming (not said yet), active (being said) or spoken (said).

{
  "schema_version": 1,
  "id": "captions-karaoke",
  "name": "Karaoke captions",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": {
      "num": 30,
      "den": 1
    },
    "sample_rate": 48000
  },
  "sources": [
    {
      "id": "wall",
      "locator": "asset://wall",
      "kind": "image"
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "picture",
        "name": "Picture",
        "kind": "video",
        "clips": [
          {
            "id": "photo",
            "source_id": "wall",
            "timeline_range": {
              "start": 0,
              "end": 300000
            },
            "source_range": {
              "start": 0,
              "end": 300000
            }
          }
        ],
        "transitions": []
      },
      {
        "id": "subs",
        "name": "Subtitles",
        "kind": "caption",
        "clips": [],
        "captions": {
          "style": {
            "font_family": "Hanken Grotesk",
            "font_size": 0.08,
            "font_weight": 700,
            "position": {
              "x": 0.5,
              "y": 0.85
            },
            "strokes": [
              {
                "color": "#000000",
                "width": 0.08
              }
            ],
            "shadows": [
              {
                "color": "#00000099",
                "offset": {
                  "x": 0,
                  "y": 0.06
                },
                "blur": 0.15
              }
            ],
            "animators": [
              {
                "selector": {
                  "type": "timing",
                  "state": "upcoming"
                },
                "properties": {
                  "opacity": {
                    "mode": "constant",
                    "value": 0.55
                  }
                }
              },
              {
                "selector": {
                  "type": "timing",
                  "state": "active",
                  "ramp": 3000
                },
                "properties": {
                  "fill": {
                    "mode": "constant",
                    "value": "#ffd040"
                  },
                  "scale": {
                    "mode": "constant",
                    "value": {
                      "x": 1.12,
                      "y": 1.12
                    }
                  }
                }
              }
            ]
          },
          "cues": [
            {
              "timeline_range": {
                "start": 36000,
                "end": 240000
              },
              "text": [
                {
                  "text": "Every ",
                  "time": {
                    "start": 0,
                    "end": 48000
                  }
                },
                {
                  "text": "frame ",
                  "time": {
                    "start": 48000,
                    "end": 96000
                  }
                },
                {
                  "text": "is ",
                  "time": {
                    "start": 96000,
                    "end": 120000
                  }
                },
                {
                  "text": "rendered ",
                  "time": {
                    "start": 120000,
                    "end": 168000
                  }
                },
                {
                  "text": "live.",
                  "time": {
                    "start": 168000,
                    "end": 204000
                  }
                }
              ]
            }
          ]
        }
      }
    ]
  }
}

The two animators:

• The first selects upcoming runs and sets their opacity to 0.55, so words not yet said are dimmed.
• The second selects the active run and sets its fill to amber and its scale to 1.12. ramp: 3000 (25 ms) fades the change in and out instead of switching it.

The text has a strokes outline and a soft shadows entry, both in fractions of the font size, so it stays readable over any picture. Add "sweep": true to the active selector to move the highlight through each word character by character.

Runs can also carry a class (WebVTT <c.name>, <b>, <i>, <u> become classes). A class selector styles them, for example { "type": "class", "class": "key" } to lift key words.

Export it

CLI

alpha2-host export --project captions.json \
  --asset forest=forest.jpg \
  --font HankenGrotesk.ttf \
  --out captions.mp4

The default range ends where the last thing on the timeline ends, caption cues included.

Node.js

import { readFile } from "node:fs/promises";
import { exportProject } from "@alpha2-sh/node";

const project = JSON.parse(await readFile("captions.json", "utf8"));
await exportProject({
  project,
  assets: { forest: "forest.jpg" },
  fonts: ["HankenGrotesk.ttf"],
  startTicks: 0,
  endTicks: 540_000, // 4.5 s
  frameRate: { num: 30, den: 1 },
  output: "captions.mp4",
});

Python

alpha2.compose reads a subtitle file and lays it over a video in one node. CaptionedVideo fits the video to the canvas (cover by default) and puts the captions over it. Captions takes a path to an .srt or .vtt file and a style: outline (the default), boxed, bar-bottom, bold-center, bold, karaoke or pop. The last two highlight words and need word times.

import alpha2
from alpha2 import compose as c

video = c.Video(c.CaptionedVideo("talk.mp4", c.Captions("talk.srt", style="boxed")))
compiled = video.compile(fonts=["HankenGrotesk.ttf"])
alpha2.export(compiled, "talk-captioned.mp4")

Compiling asks the host’s conform for the video’s size and length (no GPU needed). For word-level karaoke from a speech-to-text tool, pass word times instead of a file:

words = [
    {"start": 0.0, "end": 0.5, "text": "Sing"},
    {"start": 0.5, "end": 1.0, "text": "along"},
]
captions = c.Captions(words=words, style="karaoke", highlight="#ffd040")

See also

• Captions: cues, runs, styles and speakers.
• Captions schema: every caption field.
• Python: alpha2.compose and its other nodes.
• Animated lower third: text in a layout box.