Skip to content
Kevin Liu's WikiSearch

ui-recording-timeline

Generated source view for the actual executable engineering/ui-recording-timeline skill. The durable routing article is Agent Engineering Skills. Source: skills/engineering/ui-recording-timeline/SKILL.md

Runtime Source

Field Value
Category engineering
Origin unknown
Slug ui-recording-timeline
Source slug ui-recording-timeline
Family Agent Engineering Skills
Source skills/engineering/ui-recording-timeline/SKILL.md

Description

Turn a UI screen recording (.mp4/.mov) into an interactive paint timeline with each paint's time, duration and painted or moved regions. Use for "what painted when", a filmstrip or a load breakdown.

Skill Source

---
name: ui-recording-timeline
description: Turn a UI screen recording (.mp4/.mov) into an interactive paint timeline with each paint's time, duration and painted or moved regions. Use for "what painted when", a filmstrip or a load breakdown.
---

# UI recording timeline

Produces a self-contained folder with `index.html` and the video. The video fills the window. Below it are a one-line caption and a thin timeline (phase colors, paint ticks, hover thumbnails, scrubbing). Change regions flash on the video at each paint, and a paint list drawer opens with `l`.

Everything needed is in this skill directory:

- `scripts/timeline.py`: one CLI with all the steps. Dependencies are declared inline, and `uv run` installs them.
- `assets/template.html`: the page. `build` injects the data into it.

Requirements: `ffmpeg`/`ffprobe` and `uv` on PATH. Chrome or Chromium is needed only for `preview`.

Below, `T` means `uv run <this skill dir>/scripts/timeline.py`. Define it as a shell function, not a variable: zsh doesn't word-split variables. Use the absolute path of the skill directory:

```bash
T(){ uv run "/abs/path/to/ui-recording-timeline/scripts/timeline.py" "$@"; }
```

 `WD` is the output folder. If the user's project has conventions for where research or collateral goes, follow them; otherwise put `WD` next to the video as `<video-stem>-timeline/`.

## Workflow

### 1. Extract and find candidate paints

```bash
T extract /path/to/recording.mp4 WD     # copies video, writes meta.json (frame timestamps; VFR is fine)
T candidates WD                         # table of frames that differ from the previous frame
```

Each row has the frame index, time, % changed, kind (`change` / `full` / `cursor?`), and the normalized bbox. Small cursor-like changes are hidden (use `--all` to show them).

### 2. Look at the frames, then decide what counts as a paint

Never label from the table alone. Look at the frames:

```bash
T sheet WD --frames 0,4,14,26,61 --name overview          # labelled contact sheets -> WD/sheets/*.jpg
T sheet WD --frames 18,19 --crop 0,0,0.25,0.15 --name zoom # zoom into a small change
```

Read the sheet images with the image viewer. Make sure to include the frame before each candidate, so you can see what changed.

Rules for choosing paints:
- **Is a paint:** a visible content or layout change. That covers skeletons appearing, real content replacing skeletons, navigation to a blank page, a new page's first paint, a label or button-state change, and browser chrome that signals load state (the reload icon replacing the stop X).
- **Is not a paint:** cursor movement, hover highlights, spinner rotation, caret blink, or a gradual shimmer on skeletons that are already there. Look at the bbox: tiny regions that wander across consecutive frames are the cursor.
- **Animations:** a multi-frame animation (a check-mark animating, a panel sliding in) is one paint at its final frame. Give it `"base": <frame before the animation started>` so the diff compares the whole animation.
- **Minor paints:** set `minor: true` on small in-place updates (a label, a button state, browser chrome). They get shorter ticks and muted text.
- **Phases:** group paints into phases by screen or URL (for example: app shell, SSO gate, IdP, blank redirect, signed-in app, idle). The address bar in the frames usually tells you. Phases must cover 0 to the recording's duration with no gaps.
- **Idle phases:** set `"idle": true` on stretches where nothing on screen changes. That always includes the tail after the last paint, and can include long waits on one unchanged screen. Idle phases are drawn hatched grey, not in a palette color: a saturated color would suggest activity, and dark grey makes the bar look like it ends early. Don't trim the idle tail; it's real time, and the video spans it.
- **Metrics:** the first two appear in the caption, so put the most important first (for example "main content" and "visually complete"). Others, like time on the IdP or total blank time, go in the drawer.
- **Caveat:** t=0 is the first frame of the recording, not navigation start. If the first frame already shows UI, say so in `notes`.

### 3. Write `WD/paints.json`

```json
{
  "title": "vercel.com/vercel-labs with an SSO redirect",
  "notes": "Recording starts mid-navigation: the dashboard shell is already on screen at t=0.",
  "metrics": [
    {"value": "6.50 s", "label": "main content"},
    {"value": "8.37 s", "label": "visually complete"},
    {"value": "2.53 s", "label": "on Okta"}
  ],
  "phases": [
    {"start": 0.0,   "end": 0.802,  "name": "Dashboard (unauthenticated)"},
    {"start": 0.802, "end": 2.002,  "name": "Vercel SSO gate"},
    {"start": 2.002, "end": 8.468,  "name": "Okta and back"},
    {"start": 8.468, "end": 10.935, "name": "Idle", "idle": true}
  ],
  "paints": [
    {"frame": 0,  "label": "Dashboard shell: sidebar nav and header skeleton (recording start)"},
    {"frame": 4,  "label": "Projects and usage skeleton cards"},
    {"frame": 19, "label": "Team switcher changes to \"Select Team\"", "minor": true},
    {"frame": 87, "label": "Okta Verify check badge (animates 2.668 to 3.068 s)", "minor": true, "base": 74}
  ]
}
```

- `frame` is the decoded frame index from `candidates`. The time comes from the frame's real timestamp, so don't hand-compute it. Phase boundaries are in seconds: use the `t` of the paint that starts each phase, and the duration from `extract` for the last one.
- Labels are short and concrete. Say what appeared, changed or disappeared, and quote on-screen text.
- Optional per phase: `"color": "#hex"`. By default colors come from a palette.

### 4. Compute and check the change regions

```bash
T regions WD     # move-aware diff for every paint -> regions.json
T check WD       # boxes drawn on every paint frame -> WD/sheets/check-*.jpg
```

Look at every check sheet. Pink means painted, blue means moved, and thin grey shows where the content moved from. How the diff works (details in `compute_regions`):
- Changed layout shapes are snapped to whole elements: words, icons, inputs, cards. Faint shimmer ends are included.
- A shape found nearby in the previous frame, whose old spot was vacated, counts as moved. For example, when a sidebar item is inserted, the rows below it slide down. Same-offset moves are grouped.
- Ambiguous repeated shapes (identical chevrons, rows) count as moved when they line up with an offset already found in the same frame.
- The cursor is learned, not guessed from size. Cursor sprites (arrow, hand, text cursor) come from the `candidates` frames where only something small moved on its own. Only boxes containing a known sprite, in either frame, are dropped. Small real paints like a logo, an icon or a toolbar button are kept. So run `candidates` before `regions`.
- Everything else is painted. That includes flat fills like a new backdrop.

If a box looks wrong:
- First check the paint choice. A wrong `base` or a hover state inside the paint frame is the usual cause.
- Only then consider tuning the constants at the top of `compute_regions`. They're tuned at 1710 px wide and scaled automatically: `EDGE`/`CHG` thresholds, `SEARCH` distance, the cursor size, and the grow tolerance.

### 5. Build, preview, and open

```bash
T build WD                    # paint thumbnails + WD/index.html
T preview WD --t 6.502        # headless screenshot with boxes frozen -> WD/preview.png
open WD/index.html
```

Read `preview.png` to verify the layout and that the overlay lines up with the video. Re-run `build` after any change to `paints.json` or `regions.json`. Run `regions` again only when the paint frames or bases change.

## Page reference

- Keys: `←`/`→` step between paints, `space` plays, `d` toggles the change flashes, `l` opens the paint list drawer, `f` goes fullscreen. Clicking the video plays or pauses. Clicking or dragging the timeline scrubs, and hovering shows a thumbnail.
- URL options: `#t=6.502` deep-links to a time, and `?hold` freezes the boxes, which helps with screenshots.
- "Shown" is the time until the next paint.
- The folder is portable: `index.html`, `recording.*`, `paints/`. `meta.json`, `paints.json`, `regions.json` and `sheets/` are inputs and debug output, safe to keep.

## Reporting back

Summarize the phases, the key metrics, and anything notable (blank screens, layout shifts that the diff shows as moves, long waits). Give the path to `index.html`. Mention the t=0 caveat, and that times are accurate to ±1 frame.

Timeline