Extensions

Extension providers supply frames for formats the ordinary video and image decoders do not read. After reading this page you can declare such a source in a project, export it with the native host, and render it in the browser with the extensions module.

The same Rust code reads these formats everywhere: alpha2-host has it built in, and the browser loads it as its own WebAssembly module (alpha2_extensions_bg.wasm) only when a page opens such a source.

The providers

A source names its provider in provider. The provider’s format decides the source’s kind:

provider.format	Reads	Source kind	Settings
svg	A static SVG document, rasterized at its shown size	image
animated_image	A GIF, animated WebP or APNG: one cycle of frames, each with its own duration. Repetition is the clip’s.	video
lottie	A Lottie animation, as a JSON file or a dotLottie package, evaluated at its own frames	video	animation: the dotLottie animation to show, by id. Absent means the package’s first.
exr	A still OpenEXR image: linear-light RGB(A), read at half precision	image	primaries: overrides the file’s chromaticities
image_sequence	PNG or OpenEXR stills played in order at a fixed rate	video	frames, frame_rate, start_number, missing, primaries

primaries is one of bt709, bt2020, p3_d65, aces_ap0 or aces_ap1. Without it, an OpenEXR file’s own chromaticities apply, or BT.709 when it has none. See color and HDR.

Image sequences

An image sequence lists each frame’s locator in play order:

• frames: the locators. null marks a missing number, allowed only when missing says how to show it.
• frame_rate: the rate the frames play at, { "num", "den" }.
• start_number: the number of the first frame, as its file was numbered (default 0).
• missing: reject (the default), hold (show the previous frame) or blank.

A project with every provider

project.json

{
  "schema_version": 1,
  "id": "extensions",
  "name": "Extensions",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": { "num": 30, "den": 1 },
    "sample_rate": 48000
  },
  "sources": [
    { "id": "logo", "kind": "image", "locator": "logo.svg", "provider": { "format": "svg" } },
    { "id": "sticker", "kind": "video", "locator": "sticker.gif", "provider": { "format": "animated_image" } },
    { "id": "intro", "kind": "video", "locator": "intro.lottie", "provider": { "format": "lottie", "animation": "main" } },
    { "id": "plate", "kind": "image", "locator": "plate.exr", "provider": { "format": "exr", "primaries": "p3_d65" } },
    {
      "id": "render",
      "kind": "video",
      "locator": "sequence://render",
      "provider": {
        "format": "image_sequence",
        "frames": ["render_0001.png", "render_0002.png", null, "render_0004.png"],
        "frame_rate": { "num": 24, "den": 1 },
        "start_number": 1,
        "missing": "hold"
      }
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "back",
        "name": "Background",
        "kind": "video",
        "clips": [
          { "id": "render-clip", "source_id": "render", "timeline_range": { "start": 0, "end": 20000 }, "source_range": { "start": 0, "end": 20000 } },
          { "id": "plate-clip", "source_id": "plate", "timeline_range": { "start": 20000, "end": 120000 }, "source_range": { "start": 0, "end": 100000 } }
        ]
      },
      {
        "id": "front",
        "name": "Graphics",
        "kind": "video",
        "clips": [
          { "id": "intro-clip", "source_id": "intro", "timeline_range": { "start": 0, "end": 60000 }, "source_range": { "start": 0, "end": 60000 } },
          { "id": "logo-clip", "source_id": "logo", "timeline_range": { "start": 60000, "end": 120000 }, "source_range": { "start": 0, "end": 60000 } }
        ]
      },
      {
        "id": "top",
        "name": "Sticker",
        "kind": "video",
        "clips": [
          { "id": "sticker-clip", "source_id": "sticker", "timeline_range": { "start": 0, "end": 120000 }, "source_range": { "start": 0, "end": 120000 } }
        ]
      }
    ]
  }
}

A source’s locator is opaque to the engine: it never opens it. The host decides what a locator means, so the same project works with files on disk natively and with bytes the page fetched in the browser.

Export natively

alpha2-host reads provider sources from the files you give, the same way it reads video and images.

What	CLI	Request field (Node.js, Python, protocol)
The file of an SVG, animated image, Lottie or OpenEXR source	--asset <source-id>=<path>	assets
The frame files of an image sequence, in order; an empty path for a gap	--frames <source-id>=<path>,<path>,...	frames (null for a gap)
An image an SVG <image href> or a Lottie image asset references, by the locator the source writes	--resource <source-id> <locator> <path>	resources
Fonts for SVG text	--font <path>	fonts

alpha2-host export --project project.json \
  --asset logo=art/logo.svg \
  --asset sticker=art/sticker.gif \
  --asset intro=art/intro.lottie \
  --asset plate=plates/plate.exr \
  --frames render=render/render_0001.png,render/render_0002.png,,render/render_0004.png \
  --font fonts/HankenGrotesk.ttf \
  --out extensions.mp4

The same export from Node.js:

await client.exportProject({
  project,
  assets: {
    logo: "art/logo.svg",
    sticker: "art/sticker.gif",
    intro: "art/intro.lottie",
    plate: "plates/plate.exr",
  },
  frames: {
    render: ["render/render_0001.png", "render/render_0002.png", null, "render/render_0004.png"],
  },
  resources: { logo: { "art/badge.png": "/media/badge.png" } }, // an <image href> in logo.svg
  fonts: ["fonts/HankenGrotesk.ttf"],
  startTicks: 0,
  endTicks: 120_000,
  frameRate: { num: 30, den: 1 },
  output: "extensions.mp4",
});

A referenced image that no resources entry names is not drawn, and the export reports missing_image. memory.providerSourceBytes (default 512 MiB) bounds the memory open provider sources hold; past it, sources not shown in the last two frames are closed and reopened when needed.

Render in the browser

In the browser, alpha2/extensions reads these formats, and you upload the frames it renders to the engine like any other frames (see Supply frames).

1. Load the module once per page with loadExtensions().
2. Register the fonts SVG text draws with: setFonts.
3. Open each provider source from its file’s bytes: open(provider, bytes).
4. For each need frameNeeds reports, render the source at that source time and upload the result.

import { loadExtensions } from "alpha2/extensions";

// Pass the URL of alpha2/wasm/alpha2_extensions_bg.wasm when your bundler moves it.
const extensions = await loadExtensions();
extensions.setFonts([fontBytes]); // the same fonts, in the same order, as a native export

const bytesOf = async (url) => new Uint8Array(await (await fetch(url)).arrayBuffer());
const opened = new Map();
for (const source of project.sources) {
  if (!source.provider || source.provider.format === "image_sequence") continue;
  opened.set(source.id, {
    kind: source.kind,
    source: extensions.open(source.provider, await bytesOf(`/media/${source.locator}`)),
  });
}

const { width, height } = project.settings;
const time = 60_000n;
for (const need of JSON.parse(engine.frameNeeds(time))) {
  const entry = opened.get(need.sourceId);
  if (!entry) continue; // not a provider source
  const ticks = BigInt(need.mediaTimeTicks);
  const frame = entry.source.render(ticks, { width, height });
  const still = entry.kind === "image";
  if (frame.precision === "linear_f16") {
    // OpenEXR: premultiplied linear-light half floats in BT.709 primaries.
    if (still) engine.uploadPresentedLinearImage(need.sourceId, frame.width, frame.height, frame.pixels, width, height);
    else engine.uploadPresentedLinearFrame(need.sourceId, ticks, frame.width, frame.height, frame.pixels, width, height);
  } else if (still) {
    engine.uploadPresentedRgbaImage(need.sourceId, frame.width, frame.height, frame.pixels, true, width, height);
  } else {
    engine.uploadPresentedRgbaFrame(need.sourceId, ticks, frame.width, frame.height, frame.pixels, true, width, height);
  }
}
engine.renderAt(time);

• render(ticks, fit) rasterizes vector sources (SVG, Lottie) to fit fit, keeping their aspect, and renders raster sources at their own size. The uploadPresented* methods place the frame into the output as the engine places every source.
• An 8-bit frame is premultiplied sRGB RGBA8 (precision: "rgba8"). An OpenEXR frame is precision: "linear_f16", and values outside 0 to 1 are kept.
• frame.diagnostics lists notices worth showing, such as missing_font, missing_image or unsupported_feature.
• source.reader() makes an independently positioned reader over the same decoded data, for two clips that read one animation at distant times. source.release() frees it.

Image sequences and referenced images

Image sequences take no bytes of their own: their frames, like the images an SVG references, are resources registered by locator. withResources registers exactly what one call needs, and releases it afterwards, so a sequence enters the module one frame at a time:

const provider = project.sources.find((source) => source.id === "render").provider;
const read = (locator) => bytesOf(`/media/render/${locator}`);

const sequence = await extensions.withResources(
  read,
  () => extensions.openNeeds(provider),
  () => extensions.open(provider),
);
const frame = await extensions.withResources(
  read,
  () => sequence.needs(ticks),
  () => sequence.render(ticks, { width, height }),
);

registerResource(locator, bytes) and releaseResource(locator) manage resources by hand instead. Use one mechanism or the other: withResources also releases locators you registered yourself.

Probe a file

extensions.probe(bytes) asks the providers whether they read a file. It returns { provider, report }, or null when no provider claims the bytes, so your own decoders can read them:

const found = extensions.probe(bytes);
if (found) {
  const { format, width, height, timing, loop_count, items, color } = found.report;
  // timing: { kind: "static" }, { kind: "frames", durations }, or { kind: "rate", rate, frames }
}

report.items lists the animations of a dotLottie package; pass one as provider.animation.

Recover from a module failure

The module runs Rust code, which can abort on a broken file. ExtensionModuleError means the module aborted during a call; it has restarted, so later calls work, but the source involved stays failed. StaleExtensionSourceError means a source was opened before a restart: register its resources and open it again.

Note

@alpha2-sh/editor does all of this for you, and can run it in a worker (extensionsWorkerFactory).

Next

• Browser: frame supply and the upload methods.
• CLI: alpha2-host export and its flags.
• Color and HDR: how OpenEXR’s linear light enters the working space.