Captions from an SRT file

This content is for v0.4. Switch to the latest version for up-to-date documentation.

After this recipe you can import an .srt or .vtt file into a caption track, and style it. The engine parses the file, so the browser, Node.js and Python read the same cues from it.

Import the file

A subtitle file like this one:

1
00:00:00,300 --> 00:00:02,000
Every frame is rendered on the GPU.

2
00:00:02,200 --> 00:00:04,000
The same project, in every target.

parseCaptions from alpha2/captions parses it with the engine’s parser in WebAssembly, and captionTrack wraps the cues in a track:

import init, * as wasm from "alpha2/wasm";
import { captionTrack, parseCaptions } from "alpha2/captions";

await init();
const text = await (await fetch("/talk.srt")).text();
const { rows, skipped } = parseCaptions(wasm, text, "srt"); // or "webvtt" for a .vtt file

const style = {
  font_family: "Hanken Grotesk",
  font_size: 0.055,
  background: { color: "#000000b3", padding: 0.3 },
};
const tracks = rows.map((cues, i) => captionTrack(`subs-${i}`, "Subtitles", cues, style));
project.timeline.tracks.push(...tracks);

rows holds one track’s worth of cues per row. Cues in a row never overlap, so a file with overlapping cues gives more than one row. Add one caption track per row.
skipped counts blocks the parser could not read. Markup (<b>, <i>, WebVTT tags) is dropped.
Timestamps are exact: 1 ms is 120 ticks. 00:00:00,300 becomes 36000.

The track this builds, in a project over a photo:

{
  "schema_version": 1,
  "id": "captions-srt",
  "name": "Captions from SRT",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": {
      "num": 30,
      "den": 1
    },
    "sample_rate": 48000
  },
  "sources": [
    {
      "id": "forest",
      "locator": "asset://forest",
      "kind": "image"
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "picture",
        "name": "Picture",
        "kind": "video",
        "clips": [
          {
            "id": "photo",
            "source_id": "forest",
            "timeline_range": {
              "start": 0,
              "end": 540000
            },
            "source_range": {
              "start": 0,
              "end": 540000
            }
          }
        ],
        "transitions": []
      },
      {
        "id": "subs",
        "name": "Subtitles",
        "kind": "caption",
        "clips": [],
        "captions": {
          "style": {
            "font_family": "Hanken Grotesk",
            "font_size": 0.055,
            "background": {
              "color": "#000000b3",
              "padding": 0.3
            }
          },
          "cues": [
            {
              "timeline_range": {
                "start": 36000,
                "end": 240000
              },
              "text": "Every frame is rendered on the GPU."
            },
            {
              "timeline_range": {
                "start": 264000,
                "end": 480000
              },
              "text": "The same project, in every target."
            }
          ]
        }
      }
    ]
  }
}

A caption track has kind: "caption", no clips, and a captions object with a style and the cues. The style fields you leave out take their defaults: centred, anchored at its bottom centre (anchor 0.5, 1) at position 0.5, 0.92, white, font_size 0.05 of the picture’s shorter side, wrapped at max_width 0.9. Here the style adds a font, a slightly larger size and a background, a box behind each line whose padding is a fraction of the font size.

Outline instead of a box

A box keeps captions readable over any picture. An outline does the same with less of the picture covered: a band around the glyphs, drawn behind them, whose width is a fraction of the font size (at most 0.5).

{
  "schema_version": 1,
  "id": "captions-outline",
  "name": "Outlined captions",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": {
      "num": 30,
      "den": 1
    },
    "sample_rate": 48000
  },
  "sources": [
    {
      "id": "wall",
      "locator": "asset://wall",
      "kind": "image"
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "picture",
        "name": "Picture",
        "kind": "video",
        "clips": [
          {
            "id": "photo",
            "source_id": "wall",
            "timeline_range": {
              "start": 0,
              "end": 300000
            },
            "source_range": {
              "start": 0,
              "end": 300000
            }
          }
        ],
        "transitions": []
      },
      {
        "id": "subs",
        "name": "Subtitles",
        "kind": "caption",
        "clips": [],
        "captions": {
          "style": {
            "font_family": "Hanken Grotesk",
            "font_size": 0.08,
            "font_weight": 700,
            "position": {
              "x": 0.5,
              "y": 0.85
            },
            "outline": {
              "color": "#000000",
              "width": 0.12
            }
          },
          "cues": [
            {
              "timeline_range": {
                "start": 36000,
                "end": 144000
              },
              "text": "Every frame"
            },
            {
              "timeline_range": {
                "start": 144000,
                "end": 240000
              },
              "text": "is rendered live."
            }
          ]
        }
      }
    ]
  }
}

Here the style sets a bold weight, a larger size, a black outline 0.12 of the font size wide, and moves the text up a little (position 0.5, 0.85). Every cue of the track draws in this one style: a caption track has a single style.

Export it

alpha2-host export --project captions.json \
  --asset forest=forest.jpg \
  --font HankenGrotesk.ttf \
  --out captions.mp4

The default range ends where the last thing on the timeline ends, caption cues included.

import { readFile } from "node:fs/promises";
import { exportProject } from "@alpha2-sh/node";

const project = JSON.parse(await readFile("captions.json", "utf8"));
await exportProject({
  project,
  assets: { forest: "forest.jpg" },
  fonts: ["HankenGrotesk.ttf"],
  startTicks: 0,
  endTicks: 540_000, // 4.5 s
  frameRate: { num: 30, den: 1 },
  output: "captions.mp4",
});

alpha2.compose reads a subtitle file and lays it over a video in one node. CaptionedVideo fits the video to the canvas (cover by default) and puts the captions over it. Captions takes a path to an .srt or .vtt file and a style: outline (the default), boxed, bar-bottom or bold-center.

import alpha2
from alpha2 import compose as c

video = c.Video(c.CaptionedVideo("talk.mp4", c.Captions("talk.srt", style="boxed")))
compiled = video.compile(fonts=["HankenGrotesk.ttf"])
alpha2.export(compiled, "talk-captioned.mp4")

Compiling asks the host’s conform for the video’s size and length (no GPU needed). For word times from a speech-to-text tool, pass words instead of a file. They are grouped into cues of up to max_chars characters:

words = [
    {"start": 0.0, "end": 0.5, "text": "Sing"},
    {"start": 0.5, "end": 1.0, "text": "along"},
]
captions = c.Captions(words=words, style="bold-center")