Picture in picture

After this recipe you can show one clip as an inset over another: scaled down, placed in a corner, with rounded corners and a drop shadow. This is FFmpeg’s overlay and scale in one document.

Scale, place and round the inset

Every visual clip has a visual object: position, anchor, scale, rotation_degrees and opacity. A clip’s masks limit where it is visible. Put the inset on a track above the main picture, since later tracks draw on top.

{
  "schema_version": 1,
  "id": "pip",
  "name": "Picture in picture",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": {
      "num": 30,
      "den": 1
    },
    "sample_rate": 48000
  },
  "sources": [
    {
      "id": "coast",
      "locator": "asset://coast",
      "kind": "image"
    },
    {
      "id": "forest",
      "locator": "asset://forest",
      "kind": "image"
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "main",
        "name": "Main",
        "kind": "video",
        "clips": [
          {
            "id": "main-clip",
            "source_id": "coast",
            "timeline_range": {
              "start": 0,
              "end": 480000
            },
            "source_range": {
              "start": 0,
              "end": 480000
            }
          }
        ],
        "transitions": []
      },
      {
        "id": "inset",
        "name": "Inset",
        "kind": "video",
        "clips": [
          {
            "id": "inset-clip",
            "source_id": "forest",
            "timeline_range": {
              "start": 60000,
              "end": 480000
            },
            "source_range": {
              "start": 0,
              "end": 420000
            },
            "visual": {
              "anchor": {
                "x": 1,
                "y": 1
              },
              "position": {
                "mode": "constant",
                "value": {
                  "x": 0.96,
                  "y": 0.93
                }
              },
              "scale": {
                "mode": "constant",
                "value": {
                  "x": 0.32,
                  "y": 0.32
                }
              },
              "opacity": {
                "mode": "keyframes",
                "keyframes": [
                  {
                    "time": 60000,
                    "value": 0,
                    "interpolation": {
                      "type": "cubic_bezier",
                      "x1": 0.2,
                      "y1": 0,
                      "x2": 0,
                      "y2": 1
                    }
                  },
                  {
                    "time": 108000,
                    "value": 1,
                    "interpolation": {
                      "type": "hold"
                    }
                  }
                ]
              }
            },
            "masks": [
              {
                "shape": {
                  "type": "rectangle",
                  "center": {
                    "mode": "constant",
                    "value": {
                      "x": 0.5,
                      "y": 0.5
                    }
                  },
                  "size": {
                    "mode": "constant",
                    "value": {
                      "x": 1,
                      "y": 1
                    }
                  },
                  "corner_radius": {
                    "mode": "constant",
                    "value": 0.1
                  }
                }
              }
            ]
          }
        ],
        "transitions": []
      }
    ]
  }
}

• Size. A source is first fitted inside the canvas, keeping its aspect ratio. scale multiplies that, so 0.32 draws the inset at 32% of the canvas width for a 16:9 photo on a 16:9 canvas.
• Place. anchor is the point of the source that position puts on the canvas, both normalized (0 to 1). With anchor at the bottom-right corner (1, 1) and position at (0.96, 0.93), the inset’s corner sits 4% from the right edge and 7% from the bottom, whatever its scale. The default anchor is the centre.
• Round. The mask is a rectangle the size of the whole source (center 0.5, 0.5 and size 1, 1) with a corner_radius. Mask geometry is in the clip’s source space, before its transform, so the mask scales and moves with the inset. corner_radius is a fraction of the source’s shorter side.
• Fade in. opacity is keyed from 0 at 0.5 s to 1 at 0.9 s. A clip’s visual is keyed in timeline time, so these keys sit inside the clip’s timeline_range, which starts at 0.5 s.

Tip

Use an ellipse mask of size 1, 1 instead for a round inset, as video calls show a speaker. Add feather to a mask to soften its edge.

Add a drop shadow

A shadow has to fall outside the inset, so it has to be drawn after the inset is placed. A clip’s visual and masks are applied last, so do the work in a composition instead: a graph that builds the clip’s picture. This one masks the clip, places it with a transform node, and runs alpha2.effect.drop-shadow on the result.

{
  "schema_version": 1,
  "id": "pip-shadow",
  "name": "Picture in picture with a shadow",
  "settings": {
    "width": 1280,
    "height": 720,
    "frame_rate": {
      "num": 30,
      "den": 1
    },
    "sample_rate": 48000
  },
  "sources": [
    {
      "id": "coast",
      "locator": "asset://coast",
      "kind": "image"
    },
    {
      "id": "forest",
      "locator": "asset://forest",
      "kind": "image"
    }
  ],
  "compositions": [
    {
      "id": "inset",
      "output": "shadow",
      "nodes": [
        {
          "id": "clip",
          "op": {
            "type": "clip"
          }
        },
        {
          "id": "rounded",
          "op": {
            "type": "shape_mask",
            "input": "clip",
            "masks": [
              {
                "shape": {
                  "type": "rectangle",
                  "center": {
                    "mode": "constant",
                    "value": {
                      "x": 0.5,
                      "y": 0.5
                    }
                  },
                  "size": {
                    "mode": "constant",
                    "value": {
                      "x": 1,
                      "y": 1
                    }
                  },
                  "corner_radius": {
                    "mode": "constant",
                    "value": 0.1
                  }
                }
              }
            ]
          }
        },
        {
          "id": "placed",
          "op": {
            "type": "transform",
            "input": "rounded",
            "transform": {
              "anchor": {
                "x": 1,
                "y": 1
              },
              "position": {
                "mode": "constant",
                "value": {
                  "x": 0.96,
                  "y": 0.93
                }
              },
              "scale": {
                "mode": "constant",
                "value": {
                  "x": 0.32,
                  "y": 0.32
                }
              }
            }
          }
        },
        {
          "id": "shadow",
          "op": {
            "type": "effect",
            "input": "placed",
            "effect": {
              "id": "alpha2.effect.drop-shadow",
              "version": 1,
              "params": {
                "opacity": {
                  "mode": "constant",
                  "value": {
                    "type": "float",
                    "value": 0.8
                  }
                },
                "angle": {
                  "mode": "constant",
                  "value": {
                    "type": "float",
                    "value": -90
                  }
                },
                "distance": {
                  "mode": "constant",
                  "value": {
                    "type": "float",
                    "value": 0.02
                  }
                },
                "softness": {
                  "mode": "constant",
                  "value": {
                    "type": "float",
                    "value": 0.04
                  }
                }
              }
            }
          }
        }
      ]
    }
  ],
  "timeline": {
    "tracks": [
      {
        "id": "main",
        "name": "Main",
        "kind": "video",
        "clips": [
          {
            "id": "main-clip",
            "source_id": "coast",
            "timeline_range": {
              "start": 0,
              "end": 480000
            },
            "source_range": {
              "start": 0,
              "end": 480000
            }
          }
        ],
        "transitions": []
      },
      {
        "id": "inset",
        "name": "Inset",
        "kind": "video",
        "clips": [
          {
            "id": "inset-clip",
            "source_id": "forest",
            "timeline_range": {
              "start": 0,
              "end": 480000
            },
            "source_range": {
              "start": 0,
              "end": 480000
            },
            "composition": {
              "id": "inset"
            }
          }
        ],
        "transitions": []
      }
    ]
  }
}

The graph, from the clip to the output:

1. clip is the inset’s own picture.
2. rounded is a shape_mask node with the same rounded rectangle. Its geometry is in its input’s normalized space.
3. placed is a transform node. It takes the same fields as a clip’s visual, so the scale and position are the ones above.
4. shadow runs the drop shadow over the placed picture, so the shadow follows the rounded outline. angle is in degrees, and distance and softness are normalized lengths (the catalog defaults are 0.01).

The inset clip then uses the composition with "composition": { "id": "inset" } and keeps its default visual. Effect parameters are typed values ({ "type": "float", "value": 0.8 }). Parameters you leave out keep their defaults, here the black color. Composition keyframes are in composition time, which starts at the clip’s start.

Export it

The sources are coast and forest, so the export needs a file for each.

CLI

alpha2-host export --project pip.json \
  --asset coast=coast.jpg --asset forest=forest.jpg \
  --out pip.mp4

Node.js

import { readFile } from "node:fs/promises";
import { exportProject } from "@alpha2-sh/node";

const project = JSON.parse(await readFile("pip.json", "utf8"));
await exportProject({
  project,
  assets: { coast: "coast.jpg", forest: "forest.jpg" },
  startTicks: 0,
  endTicks: 480_000, // 4 s
  frameRate: { num: 30, den: 1 },
  output: "pip.mp4",
});

With video files instead of photos, change the sources’ kind to video and point --asset at the video files. The clips’ source_range then chooses which part of each video plays.

See also

• Effects and compositions: node types and composition time.
• Timeline schema: visual and masks on a clip.
• Effects reference: drop-shadow, outline and the rest.
• Vertical video: scale to fill a 9:16 frame.