Browser (alpha2)

This content is for v0.4. Switch to the latest version for up-to-date documentation.

The alpha2 package is the engine compiled to WebAssembly, plus the TypeScript that drives it. After reading this page you can load the engine in a page or a worker, render a project on a canvas, feed it the frames and audio your application decodes, and export the result as an MP4, MOV, MKV or WebM file.

The engine never opens a media file. Your application decodes video, images and audio and uploads them; the engine renders what the project shows and hands back frames and mixed audio. For a ready-made editor with import, undo and an export bridge, use @alpha2-sh/editor, which is built on this package.

Install

npm install alpha2

pnpm add alpha2

yarn add alpha2

mediabunny is an optional dependency. It provides the decoders and the MP4, MOV, MKV and WebM writer in alpha2/mediabunny.

Load the engine

alpha2/wasm exports init (the default export) and WebEffectRegistry, the engine object. init() loads alpha2_wasm_bg.wasm from beside the module:

import init, { WebEffectRegistry } from "alpha2/wasm";

await init();

If your bundler moves the .wasm file, pass its URL, a Response, its bytes or a compiled WebAssembly.Module as module_or_path. The package exports three binaries:

File	What it is	When it loads
`alpha2/wasm/alpha2_wasm_bg.wasm`	The engine	`init()`
`alpha2/wasm/alpha2_audio_worklet.wasm`	The audio worklet’s DSP	When you start audio playback
`alpha2/wasm/alpha2_extensions_bg.wasm`	SVG, animated images, Lottie, OpenEXR and image sequences	Only for such a source (extensions)

With Vite, for example, import the URL with ?url:

import init from "alpha2/wasm";
import engineUrl from "alpha2/wasm/alpha2_wasm_bg.wasm?url";

await init({ module_or_path: engineUrl });

Create a renderer

A renderer is a WebEffectRegistry bound to a canvas. Choose the constructor by where the canvas lives:

Constructor	Canvas	Draws with
`WebEffectRegistry.createForHtmlCanvas(canvas)`	An `HTMLCanvasElement` on the page	WebGPU, else WebGL2
`WebEffectRegistry.createForCanvas(canvas)`	An `OffscreenCanvas`, usually in a worker	WebGPU, else WebGL2
`WebEffectRegistry.createCpu(canvas)`	An `OffscreenCanvas`, a page canvas, or `undefined`	The engine’s CPU renderer

const canvas = document.querySelector("canvas");
const engine = await WebEffectRegistry.createForHtmlCanvas(canvas);
console.log(engine.backend); // "webgpu" or "webgl2"

The CPU renderer

When the browser has neither WebGPU nor WebGL2 (Chrome with hardware acceleration off, a blocklisted GPU), createCpu makes the engine’s CPU renderer. It takes the same calls, draws in WebAssembly on one thread, and puts the preview on the canvas’s 2D context. It draws exactly what the native CPU renderer draws, needs no canvas to export, and refuses runtime WGSL effects with EFFECT_ERROR.

Check the browser

detectBrowserCapabilities() in alpha2/robustness probes the browser without making a renderer. Its renderer field is what a render worker’s renderer: "auto" draws with: "gpu" when the browser has WebGPU or WebGL2, else "cpu". wasm says whether the engine can load at all, so you can check it before fetching the engine:

import { detectBrowserCapabilities } from "alpha2/robustness";

const caps = await detectBrowserCapabilities();
// { webgpu, webgl2, renderer, webcodecs, offscreenCanvas, wasm,
//   diagnostics: ["webgpu-unavailable", ...] }

Build and load a project

A project is the engine’s JSON document (see the project model). You can write it by hand or with ProjectBuilder from alpha2/builder, which takes times in seconds and generates ids:

import { ProjectBuilder } from "alpha2/builder";

const builder = new ProjectBuilder({ name: "Tutorial project", width: 1280, height: 720, frameRate: 30 });
const image = builder.source("image", "asset://image-1", { id: "image-1" });
builder.videoTrack({ name: "Main video" }).clip({ source: image, at: 0, duration: 1 });
const project = builder.build(); // throws a RangeError listing every problem

new ProjectBuilder({ name, width, height, frameRate, id?, sampleRate?, background?, ids? }). frameRate is a number or an exact rational such as { num: 30000, den: 1001 }. sampleRate defaults to 48000 and id to "project".
source(kind, locator, { id?, hasAudio? }) adds a video, audio or image source; generated(generator) adds a generator; composition(nodes, output) adds a composition graph.
videoTrack(), audioTrack() and captionTrack() return track builders. clip({ source, at, duration, sourceIn?, sourceDuration?, effects?, with? }) places a clip; effects chains effects through a composition, and with sets any other clip field. transition(fromClipId, toClipId, effect, easing?) joins two overlapping clips.
ticks(n) gives an exact time in ticks wherever the builder takes seconds.

validateProject(project) in alpha2/project returns the problems the engine would refuse, as strings, without loading anything; validateProjectJson(text) does the same for JSON text. alpha2/project also has the document’s TypeScript types, its limits, and tick conversions (secondsToTicks, ticksToSeconds, samplesToTicks, ticksToSamples). See validation and errors.

import { validateProject } from "alpha2/project";

const problems = validateProject(project);
if (problems.length > 0) throw new Error(problems.join("\n"));
engine.loadProject(JSON.stringify(project)); // checks again and throws what it refuses

loadProject keeps the frames and audio uploaded for every source whose id and locator are unchanged, so an edit needs no re-decode. Media that changes under an unchanged locator needs a new source id.

Size the canvas

The renderer previews at half the project size by default. After loading, size the canvas to previewSize and tell the renderer:

const [width, height] = engine.previewSize;
canvas.width = width;
canvas.height = height;
engine.resizeSurface(width, height);

engine.setPreviewScale(scale) changes the scale (above 0, at most 8). At 1, preview text and effects match an export pixel for pixel. Resize the surface to the new previewSize afterwards.

Render your first frame

Load the engine and make a renderer.
Load the project.
Upload a frame for each source the time needs. A still needs one upload: premultiplied sRGB RGBA8 pixels.
Call renderAt with a time in ticks.

import init, { WebEffectRegistry } from "alpha2/wasm";
import { ProjectBuilder } from "alpha2/builder";

await init();
const canvas = document.querySelector("canvas");
const engine = await WebEffectRegistry.createForHtmlCanvas(canvas);

const builder = new ProjectBuilder({ name: "Tutorial project", width: 1280, height: 720, frameRate: 30 });
const image = builder.source("image", "asset://image-1", { id: "image-1" });
builder.videoTrack({ name: "Main video" }).clip({ source: image, at: 0, duration: 1 });
engine.loadProject(JSON.stringify(builder.build()));

const [width, height] = engine.previewSize;
canvas.width = width;
canvas.height = height;
engine.resizeSurface(width, height);

// One opaque white image, premultiplied sRGB RGBA8, at the project's size.
engine.uploadRgbaImage("image-1", 1280, 720, new Uint8Array(1280 * 720 * 4).fill(255));

const result = JSON.parse(engine.renderAt(0n)); // draws the frame at time 0
console.log(result.missingSources); // []

Text in captions, text generators and layouts draws with the fonts you register: engine.addFont(family, bytes, faceIndex) for one face, or engine.addFontFile(family, bytes) for every face of a file. Give a native export the same font files and preview text matches exported text.

Supply frames

The engine decides which frame of each source a time needs; you decode and upload those frames. This is the frame-supply model.

What `renderAt` returns

renderAt(ticks) returns JSON:

{
  "requestedTimeTicks": 0,
  "presentedTimeTicks": null,
  "missingSources": ["video-1"],
  "missingFrames": [{ "clipId": "clip-1", "sourceId": "video-1", "mediaTimeTicks": "48000" }],
  "repeated": false,
  "unchanged": false
}

missingSources names the sources it had no frame for. missingFrames gives the exact source time each one was needed at, which differs from timeline time for trimmed or retimed clips.
presentedTimeTicks is null until a frame has been presented. repeated means the previous picture was shown again. unchanged means the frame equals the last one, so nothing was rendered.

Ask before you render

frameNeeds(ticks) lists what a time needs before you render it, as JSON from the same snapshot renderAt uses:

const needs = JSON.parse(engine.frameNeeds(48_000n));
// [{ clipId: "clip-1", readerId: null, sourceId: "video-1", mediaTimeTicks: "48000" }]

Tick values are decimal strings so they keep the full 64-bit range. Use them as given: never derive a source time yourself (for example by rounding timelineSeconds - clip.start). Keep one decoder cursor per clip (and per readerId, a composition step that reads the source itself).

Upload

Method	For
`uploadVideoFrame(sourceId, mediaTimeTicks, frame)`	A decoded `VideoFrame`. The engine takes ownership and closes it. Fitted upright to the project size. HDR frames are refused.
`uploadRgbaFrame(sourceId, mediaTimeTicks, width, height, pixels)`	Tightly packed premultiplied RGBA8 at a source time.
`uploadRgbaImage(sourceId, width, height, pixels)`	A still shown at every time.
`uploadPresentedRgbaFrame` / `uploadPresentedRgbaImage`	RGBA8 at its own size, placed into the output as the engine places every source. Takes a `premultiplied` flag and the output size.
`uploadYuvFrame(...)`	Planar YCbCr samples as `VideoFrame.copyTo` writes them. This is how HDR (PQ or HLG) frames reach the engine.

Upload, then render the same time again. Release memory with releaseFrame, trimFramesBefore(sourceId, ticks) as playback moves on, or clearFrames. setUploadedFrameBudget(bytes) bounds what the renderer keeps (default 1 GiB).

`FrameSupply`

FrameSupply in alpha2/frame-supply runs this loop for you. It reads frameNeeds, calls your decoder once per clip cursor, uploads the results, shares one upload between needs that resolve to the same decoded frame, and keeps retired uploads within a byte budget.

import { FrameSupply } from "alpha2/frame-supply";

const supply = new FrameSupply(engine); // { maxRetiredBytes, maxCachedImageBytes, maxDecodeConcurrency, ... }
const layers = new Map([["image-1", { id: "image-1", kind: "image" }]]);

const presented = await supply.render(0n, {
  width: 1280, // the project's size
  height: 720,
  liveSourceIds: new Set(layers.keys()),
  getLayer: (id) => layers.get(id),
  // Resolve with RGBA8 at the project size for a still, or { kind: "video-frame", frame } for video.
  pixelsFor: async (layer, seconds, options) => decode(layer, seconds, options),
  isCurrent: () => true, // return false once a seek or edit makes this request stale
});

render supplies, presents and retires in one call, and rejects when a source is still missing. supply does only the uploads, for a host that renders itself; pass its uploaded receipts to retire once rendered. The defaults are 256 MiB of retired frames, 256 MiB of cached stills and 2 concurrent decodes.

Decode with WebCodecs

alpha2/media defines the decoder contract (BrowserVideoDecoder: refill, flush, close) and two pieces that implement it:

WebCodecsVideoDecoder(sourceId, source, options?) decodes elementary chunks from any demuxer. source gives a VideoDecoderConfig and chunksFrom(mediaTimeTicks, signal), chunks in decode order from the keyframe at or before a time.
MediabunnyVideoDecoder.open(sourceId, blobOrBytes, options?) in alpha2/mediabunny demuxes the file with mediabunny as well. It plays reversed clips by decoding windows of frames, and replaces a failing hardware decoder once with a software one.

A refill request names the source, the exact source time, a reason (prefetch, seek, evicted, device-recovery), a generation, and the consumer that owns the cursor:

import { decoderConsumerId } from "alpha2/media";
import { MediabunnyVideoDecoder } from "alpha2/mediabunny";

const decoder = await MediabunnyVideoDecoder.open("video-1", file); // a File or Blob
const [need] = JSON.parse(engine.frameNeeds(48_000n));
const decoded = await decoder.refill({
  sourceId: need.sourceId,
  mediaTimeTicks: BigInt(need.mediaTimeTicks),
  reason: "seek",
  generation: 0,
  consumerId: decoderConsumerId(need.clipId, need.readerId),
});
engine.uploadVideoFrame(decoded.sourceId, decoded.requestedTimeTicks, decoded.frame);
engine.renderAt(48_000n);

The decoder options are byte and frame limits: maxCachedFrameBytes and maxReadyFrameBytes (64 MiB each), maxCachedFrames (2, the minimum), and stallTimeoutMs (5000).

DecoderRefillCoordinator(open, deliver, options?) runs refills over many sources. It opens one decoder per source and consumer, runs one refill per source at a time and at most maxConcurrency (2) overall, and beginGeneration() aborts requests from before a seek.

Export

Export renders offscreen at export quality on the project’s exact frame grid, and never touches the canvas. Each frame comes back as planar I420 (BT.709, limited range) with padded rows.

Step by step

beginExport(startTicks, endTicks, rateNum, rateDen, codec) starts an export over a half-open range. codec (h264, hevc, vp9, av1) only places the chroma samples the way that codec expects. Then ask for frames with beginExportFrame(), which returns a ticket:

import { videoFrameFromI420 } from "alpha2/video-frames";

engine.beginExport(0n, 120_000n, 30, 1, "h264"); // [0, 1 s) at 30/1 fps
for (;;) {
  const ticket = engine.beginExportFrame();
  if (ticket.kind === "complete") break;
  if (ticket.kind === "waiting") {
    // ticket.missingFrames: decode and upload these, then ask again.
    throw new Error(`upload ${ticket.missingSources} first`);
  }
  const frame = await ticket.readback;
  const video = await videoFrameFromI420(frame.pixels, {
    format: "I420",
    codedWidth: frame.width,
    codedHeight: frame.height,
    layout: frame.layout,
    colorSpace: frame.colorSpace,
    timestamp: Math.round(Number(frame.timeTicks) / 0.12), // ticks to microseconds
  });
  video.close(); // or hand it to a VideoEncoder
}
engine.endExport();

A waiting ticket names the frames the next export frame needs. Upload them and call beginExportFrame again: the export continues at the same frame.
You can call beginExportFrame again before a readback resolves, so the GPU renders the next frame while the last one is read back.
exportProgress() reports completed and total frames, the range widened to the frame grid (effectiveRange), and the audio window the export covers.
videoFrameFromI420 builds a VideoFrame over the planes, repacking them where the browser ignores layout.

Into a file with `exportToSink`

exportToSink(exporter, sink, refill, options?) in alpha2/exporter runs that loop into an encoder sink, keeping at most maxInFlightFrames (3) frames and maxInFlightBytes (128 MiB) of readbacks in flight. MediabunnySink in alpha2/mediabunny is a WebCodecs sink that writes MP4, MOV, MKV or WebM.

The exporter talks to a render backend. With a render worker, WorkerExportBackend is that backend:

import { Alpha2Exporter, exportToSink } from "alpha2/exporter";
import { WorkerExportBackend } from "alpha2/worker";
import { decoderConsumerId } from "alpha2/media";
import { canEncodeVideoTrack, MediabunnySink, MediabunnyVideoDecoder } from "alpha2/mediabunny";

// `backend` is an initialized WorkerRenderBackend; `file` holds source "video-1".
const decoder = await MediabunnyVideoDecoder.open("video-1", file);
const rateControl = { bitrate: 8_000_000, bitrateMode: "variable" };
const encoding = await canEncodeVideoTrack("avc", { width: 1280, height: 720, rate: rateControl });
if (encoding === null) throw new Error("this browser has no H.264 encoder");

const exporter = new Alpha2Exporter(
  new WorkerExportBackend(backend, "h264"),
  { num: 30n, den: 1n },
  0n,
  240_000n,
);
const sink = new MediabunnySink({
  container: "mp4",
  videoCodec: "avc",
  videoRateControl: rateControl,
  ...(encoding.codecString === undefined ? {} : { videoCodecString: encoding.codecString }),
  frameRate: { num: 30, den: 1 },
  destination: { kind: "memory", maxBytes: 256 * 1024 * 1024 },
});

const result = await exportToSink(exporter, sink, async (waiting) => {
  for (const { clipId, readerId, sourceId, mediaTimeTicks } of waiting.missingFrames) {
    const decoded = await decoder.refill({
      sourceId, mediaTimeTicks, reason: "prefetch", generation: 0,
      consumerId: decoderConsumerId(clipId, readerId),
    });
    await backend.uploadVideoFrame(decoded);
  }
});
console.log(result.frames, result.fps);
const output = sink.output; // { kind: "memory", buffer: ArrayBuffer }

Aborting options.signal rejects with an ExportAbortedError coded cancelled; a lost GPU rejects with one coded device-lost. Either way sink.abort runs once.
A frame still waiting after four refills fails with RESOURCE_LIMIT.
destination: { kind: "stream", writable } writes through a seekable stream as it muxes, for outputs too large for memory.
resolveVideoRateControl and resolveAudioBitrate in alpha2/rate-control turn a quality level (very-low to very-high) into the same numbers a native export uses. alpha2/containers lists which codecs each container takes.

Audio

The engine mixes the project’s audio: clip gain, pan and fades, effects, track mute and solo, ducking and the master bus (see audio). You upload each source’s decoded audio, resampled to the project’s sample rate, and ask for mixed blocks.

// Planar float samples at the project rate, the planes end to end.
engine.uploadAudioBlock(
  "music",          // source id
  0n,               // first sample's index in the source, at the project rate
  48_000,           // sample rate
  2,                // a channel count, or FFmpeg channel names such as ["FL", "FR"]
  samples,          // Float32Array
);

const block = engine.renderAudio(0n, 1024, true); // planar output, channels end to end
const missing = JSON.parse(engine.audioMissingSources()); // sources the block lacked

audioNeeds(startSample, frames) lists the source intervals to decode before mixing a range, including read-ahead and the history effects replay after a seek.
A block is at most audioMaxBlock samples per channel. A block that needs audio the engine does not have is reported by audioMissingSources, not mixed with a gap.
MediabunnyAudioDecoder.open(sourceId, file, wasm) in alpha2/mediabunny decodes a source and resamples it with the engine’s AudioResampler (wasm is the alpha2/wasm module, imported with import * as wasm). Its refill returns a block with startSample, sampleRate, speakers and channels, ready for uploadAudioBlock.
trimAudioBefore(sourceId, ticks) drops audio playback has passed.

Play it

For real-time preview, the mixed blocks reach the audio thread through a ring in shared memory:

alpha2/audio-timeline has AudioRingBuffer (the ring), AudioPump (keeps the ring filled from a mixer), and AudioPlaybackClock, which counts the samples the audio thread has played. Present clock.timelineTicks each animation frame and the picture follows the audio device without drift.
alpha2/audio-worklet has Alpha2AudioWorkletHost.create(context, wasm, options), which adds the processor module and starts the DSP, and useAudioRing(node, ring.transfer), which points the node at the ring. loadAlpha2AudioWorkletWasm() fetches the packaged alpha2_audio_worklet.wasm.

alpha2/loudness measures loudness (ITU-R BS.1770-4, EBU R128) and plans normalization; alpha2/waveform computes waveform summaries; renderOfflineAudio in alpha2/audio renders a range into a sink for export.

Preview in a worker

alpha2/worker runs the engine in a dedicated worker that owns the GPU device and an OffscreenCanvas, so rendering never blocks the page. The worker loads one project when it starts.

import init, { WebEffectRegistry } from "alpha2/wasm";
import { createWasmRenderWorkerHost, installRenderWorker } from "alpha2/worker";

await init();
const project = await (await fetch("/project.json")).text();
installRenderWorker(
  self,
  createWasmRenderWorkerHost(WebEffectRegistry, project, {
    renderer: "auto", // "auto" (default), "gpu" or "cpu"
    // fonts: [{ family: "Hanken Grotesk", bytes }],
  }),
);

On the page, WorkerRenderBackend drives the worker and Alpha2Player in alpha2/player presents times through it:

import { Alpha2Player } from "alpha2/player";
import { WorkerRenderBackend } from "alpha2/worker";

const worker = new Worker(new URL("./render-worker.js", import.meta.url), { type: "module" });
const player = new Alpha2Player(new WorkerRenderBackend(worker), async (requests) => {
  // Decode each DecoderRefillRequest and upload it with player.uploadVideoFrame(decoded).
});

const canvas = document.querySelector("canvas");
canvas.width = 1280;
canvas.height = 720;
await player.initialize(canvas.transferControlToOffscreen());
await player.seek(0n);
await player.present(4_000n);

new Alpha2Player(backend, onRefill?). The player keeps no clock: present(ticks) shows the time it is given. It serializes presentations, asks onRefill for the frames a presentation lacks, and presents the same time again once they arrive. seek drops frames decoded for earlier requests.
The worker’s renderer option is "auto" (WebGPU, else WebGL2, else the CPU renderer when the browser offers neither), "gpu" (never falls back) or "cpu". WorkerRenderBackend.rendererChoice says which renderer the worker made and, on the CPU, why, as the native host’s renderer and rendererReason do.
After a device loss, the player makes the device again on the next presentation.

Edit a project

alpha2/edit has the timeline edits as pure functions from one valid project to another. Each returns { ok: true, project } or { ok: false, error } and never changes its input. Times are whole ticks.

import { split, rippleDelete, trim } from "alpha2/edit";

const result = split(project, "clip-1", 60_000, "clip-1b");
if (!result.ok) throw new Error(`${result.error.code}: ${result.error.message}`);
engine.loadProject(JSON.stringify(result.project));

The operations are split, trim, rippleTrim, rippleDelete, rippleDeleteSpan, rollTrim, slip, slide, retime, insert, overwrite, append, move, moveToTrack and moveClips. Ripple edits take syncTracks to move content on sync-locked tracks too. deferValidation(run) checks a batch of edits once, as a whole. Undo, selection and snapping are your application’s job, or the editor’s.

Subpaths

Import	What it has
`alpha2/wasm`	The engine: `init`, `WebEffectRegistry` (rendering, export, audio mixing), and functions for rate control, captions, loudness and waveforms.
`alpha2/project`	The project document’s types and limits, tick conversions, `validateProject`.
`alpha2/builder`	`ProjectBuilder`: documents from seconds, with generated ids.
`alpha2/edit`	Split, trim, ripple, roll, slip, slide, retime, insert, overwrite, append, move.
`alpha2/captions`	SRT and WebVTT import into caption tracks.
`alpha2/geometry`	Shape geometry for editors: bounds, hit tests, nearest points, path lengths, trims, resampling, motion paths.
`alpha2/layout`	Layout generators: `measureLayout` and a builder from CSS-like props, flexbox and grid. See layout.
`alpha2/effects`	Effect definitions, the built-in catalog’s types, and `RuntimeEffectRegistry` for WGSL effects registered at run time.
`alpha2/extensions`, `alpha2/extensions-wasm`	SVG, animated image, Lottie, OpenEXR and image sequence sources (`loadExtensions`).
`alpha2/text`	`WasmTextRasterizer` (the engine’s, the same pixels as a native export) and `OffscreenTextRasterizer` (the browser’s).
`alpha2/assets`	Media identity, proxies and conform reports.
`alpha2/frame-supply`	`FrameSupply`.
`alpha2/media`	Decoder contracts, `WebCodecsVideoDecoder`, `DecoderRefillCoordinator`.
`alpha2/mediabunny`	Decoders and an MP4, MOV, MKV and WebM sink on mediabunny.
`alpha2/hdr-frames`, `alpha2/video-frames`	Uploading planar and HDR `VideoFrame`s; `VideoFrame`s from exported I420 planes.
`alpha2/player`, `alpha2/worker`	`Alpha2Player` and the render worker.
`alpha2/exporter`	`Alpha2Exporter` and `exportToSink`.
`alpha2/export-destination`	Where an export’s bytes go: a file, or memory.
`alpha2/export-failure`, `alpha2/export-retry`	How export failures are classified, and a retry policy over them.
`alpha2/containers`	The export containers and the codecs each offers.
`alpha2/rate-control`	Output bitrates for the engine’s quality levels.
`alpha2/audio`, `alpha2/audio-worker`	Offline audio rendering into a sink, in a worker or not.
`alpha2/audio-timeline`, `alpha2/audio-worklet`, `alpha2/audio-worklet-processor`	Real-time audio: the shared ring, its pump, and the AudioWorklet and its processor module.
`alpha2/loudness`, `alpha2/waveform`	Loudness measurement and normalization; waveform summaries.
`alpha2/robustness`	`detectBrowserCapabilities` and degradation reports.
`alpha2`	Most of the above in one import: `assets`, `audio`, `audio-timeline`, `audio-worker`, `audio-worklet`, `containers`, `effects`, `exporter`, `extensions`, `frame-supply`, `hdr-frames`, `loudness`, `media`, `player`, `project`, `rate-control`, `robustness`, `text`, `waveform` and `worker`.

Each module’s TypeScript declarations document its full API.

React: a canvas component over Alpha2Player.
Editor: documents, commands, undo, and a ready preview and export bridge.
Extensions: SVG, Lottie, animated images and OpenEXR in the browser.
Time and ticks and determinism: why browser and native exports draw the same frames.