A structured analysis of the issues reported in the GTM kickoff → typographic social cut creative brief, verified against the codebase, with a minimal, consistently-scoped proposal for each root cause.
“Zero network spend” is not a saving and it is not six missing features. It is one contract failure wearing six masks.
Almost every primitive the brief needed already exists as compute — hyperframes-render composites HTML/CSS typography over video; ffmpeg-overlay can place a PiP; the supervision tracker already emits a per-frame sidecar. The agent could not reach them because the capability cards don’t publish enough to plan (which price is real?) or call (what are the parameters?). The dominant fix is to finish the capability contract already half-built on this branch — not to build a compositor.
The report is an unusually good field test: nothing crashed, the deliverables shipped, and the author did their own root-cause pass. This analysis does three things on top of it:
Every claim below was checked in the working tree. Where the report and the code disagree, §7 records the correction.
The report lists six things to fix. They collapse to four root causes — and the top two are contract/metadata, not missing compute.
| Reported symptom | Verified? | Root cause | Real nature |
|---|---|---|---|
#4 · ffmpeg-overlay metering contradictory — per-second in list_capabilities vs per-call in describe_capability (~1000×) |
Yes | RC-1 | Two un-reconciled price sources (live orch vs static mirror) |
#5 · hyperframes-render / -caption / ffmpeg-overlay publish no input schema — cannot call without a probing round-trip |
Yes | RC-1 | Input schema is an optional field the golden gate ignores |
#1 · overlay can’t express a timed, positioned PiP — no start/end, no x/y → 14 scenes for a 17 s brief |
Yes | RC-2 | Thin primitive missing 2 params; capable one (hyperframes) unreachable |
| #2 · No tracking primitive an overlay can bind to — sidecar produced but never consumed as an anchor | Yes | RC-2 | Missing binding contract between two things that already exist |
#3 · transcribe(burn) is paragraph-granular, no word split, no type styling |
Yes | RC-3 | Shared upstream ASR ceiling + no styled-caption compositor wired |
| #6 · No colour-grading capability among the 34 tool caps | Yes | RC-4 | Genuinely absent; lowest priority; adjacent family exists |
Colour code — RC-1 contract RC-2 composition RC-3 captions RC-4 missing cap. RC-1 accounts for two symptoms and de-risks RC-2 and RC-3, which is why it leads the sequence.
Metering is off by ~1000×; capability cards publish identity, price and latency but no input schema, so an agent “cannot call it without a probing round trip.”
Both are real, and both are the same class of defect: the card an agent reads is assembled from several hand-maintained sources that are never reconciled, and the one field that would let an agent call a tool cap is optional.
describe_capability reads price from the committed static mirror lib/capabilities/registry.json via getRegistrySnapshot() → for ffmpeg-overlay: unit_kind:"call", $0.00013. describe-capability.ts:67,104–105list_capabilities reads price from the live orchestrator /capabilities payload and spreads it raw — whatever the VM meters (e.g. per-second). list-capabilities.ts:156,266–269get_pricing is a third path: live-first, static fallback. pricing/estimate.ts:252–276CAPABILITIES_JSON, not in this repo; the repo mirror is live-first-overridden and can lag on any VM rebuild or canary rollback. Nothing cross-checks the two.usage.inputs) is optional. Only 8 caps carry one — all AI media caps. lib/capabilities/usage.ts:45–119ffmpeg-*, every hyperframes-*, and the five newly-added supervision-* caps have no usage block → describe_capability returns inputs: undefined and only a guessed example. describe-capability.ts:88–119run_capability validates nothing — inputs is z.record(z.string(), z.any()) passed straight to /inference. run-capability.ts:24–38ffmpeg-overlay (base_url, overlay_url, position, scale, opacity, padding) exist only hardcoded inside overlay.ts:112–119 — undiscoverable through any card.descriptor.ts with an io block (a real per-cap input schema), a two-tier self-validating ToolCapability price shape, and the draft_capability / cap_add authoring path that validate both. The supervision caps missed the input schema precisely because they were added the old way — hand-edited maps + registry.json — bypassing cap_add. The golden gate didn’t catch it because usage is optional. The fix is to finish and enforce what exists, not to invent anything.
#1 — overlay has five fixed anchors and no start-time or x/y, so a single 8.55 s PiP forces slicing the A-roll into 14 scenes. #2 — nothing binds a tracker to an overlay; “an overlay stuck to a moving subject… was 30 lines of OpenCV locally.”
Both confirmed. overlay is thin wiring; its image branch forwards only corner/center anchor + scale/opacity/padding — no time, no x/y. overlay.ts:21–62,112–119. And the supervision sidecar is produced and even persisted onto the scene blob, but no code reads it back to drive a position.
The gap is not “we can’t composite.” hyperframes-render takes arbitrary HTML/CSS over a background video (create-media.ts:2670–2687) — it can express timed (CSS animation) and absolutely-positioned typography and PiP. The real gap is: the cheap, discoverable overlay primitive can’t do timed PiP, and the capable one is undiscoverable because it publishes no schema (RC-1). So RC-1 already removes half of RC-2’s pain.
#3 — transcribe(burn) returns paragraph-granularity styling: no word-level split, no font/fill/stroke/position. “Word timings came from local faster-whisper again. Second confirmation — see LAT-27.”
transcribe takes only source_url, burn, language and forwards wizper’s segment chunks 1:1 to ffmpeg-burn-subtitles with zero styling params. transcribe.ts:25–43,101–115,199. hyperframes-caption is registered but unimplemented (name only, no handler).
clean_speech not buildable for one reason: the live ASR (wizper) rejects word-level timestamps, and that restriction lives in the VM-side payment-critical byoc inference-adapter, out of this repo. So word-level captions (#3) and word-level filler removal (PR-1) are blocked by a single upstream limitation. The report’s “word timings came from local faster-whisper again” is the third independent confirmation of the same ceiling. Fix it once and two features unblock.
.cube / eq / tone-curve capability in registry.json or capabilities.ts. Adjacent deterministic ffmpeg-* caps exist, so it is a natural one-cap addition — or an explicit capture-time non-goal. The report’s real ask is that it be stated rather than discovered.analyze_video_tracks in track mode requires a detections_url from a prior yolo-detect, because the container ships detector-less for MIT-license preservation (YOLO-NAS bundling is a deferred phase). analyze-video-tracks.ts:67–73,206–215. That is a real ergonomics tax, addressed under S2c.The report is accurate on substance. Three details should be corrected so the fixes target the right thing:
analyze_video_tracks is availability: experimental.” There is no availability field in the registry — availability is computed (lib/capabilities/availability.ts:26–36) and the supervision caps are status:"active", so they compute to available/registered, never experimental. The real blocker is the detector-less two-step dependency, not an experimental flag. Fix the dependency, not a label.transcribe.” The bottleneck is upstream ASR granularity at the VM adapter (see RC-3), shared with clean_speech PR-1 — not merely an unwired client param. Treat it as one upstream unlock, not a per-tool patch.hyperframes-render and the existing ASR path rather than building new engines.
Reconcile price, then guard it. Make the committed mirror’s display_price_usd/unit_kind provably consistent with the live VM CAPABILITIES_JSON, and add a drift check (same class as the existing golden test, but across the static↔live boundary) that fails CI when they disagree. Have describe_capability label its price with source and surface the live figure when it differs. Finishes the two-tier ToolCapability enforcement — no new pricing system. → closes #4.
Make the invocation schema a required, golden-gated field. Promote usage.inputs / descriptor.io from optional to required for every live tool cap, ideally mirrored from the VM container’s @tool(schema=…) decorator so there is one source. Backfill ffmpeg-overlay, hyperframes-render, hyperframes-caption, supervision-* now — the param names already exist hardcoded and just need lifting. Then describe_capability publishes real params and run_capability can validate. → closes #5; de-risks S2 and S3 by making hyperframes-render callable.
Route all new caps through cap_add / draft_capability, which already validate io + offering, so a schema-less cap can’t be registered again. The supervision caps bypassed this — that bypass is the regression to close.
Extend overlay’s image branch with start/end (or start+duration) and optional normalized x/y, forwarded to ffmpeg-overlay’s overlay=x:y:enable='between(t,…)'. A bounded param addition, not a compositor. → closes #1; kills the 14-scene workaround.
Define the anchor contract: let overlay accept track_sidecar_url + track_id, read the per-frame coordinates the supervision sidecar already produces, and emit a time-varying position (ffmpeg sendcmd, or delegate to hyperframes-render with a keyframed absolute position). → closes #2 — the “overlay stuck to a moving subject” demo. Reuses an output that exists today.
✓ SHIPPED ffmpeg-overlay gained a track keyframe param (ffmpeg sendcmd), and overlay accepts track or a track_sidecar_url (+ track_class) that reduces any detector/track sidecar to the path. Deployed + verified live: the overlay follows a moving subject.
Halve the tracking friction: have analyze_video_tracks auto-chain yolo-detect when detections_url is absent (one call, not two) and correct the availability labeling. Small, honest ergonomics.
Unlock word-level timestamps at the VM byoc inference-adapter — the single upstream blocker shared with clean_speech PR-1. It is out-of-repo and payment-adjacent, so scope it carefully, but it unblocks two features, which is why it beats per-feature hacks.
Implement hyperframes-caption as the styled word-level path (font/fill/stroke/position via HTML/CSS through the existing hyperframes-render compositor), fed by the word timings from S3a, and add a granularity + style passthrough to transcribe(burn). Reuses the compositor — no new typography engine. → closes #3.
If S3a can’t land soon: document hyperframes-render as the interim type path (as the report suggests) and state honestly that word timings still require local ASR. Don’t mark the gap closed while the ceiling stands.
Add ffmpeg-colorgrade (LUT / eq / curves) to the ffmpeg-* tool family — deterministic, cheap, per-call, fits the existing pattern — or explicitly document colour grading as a capture-time non-goal. Either way it becomes stated, not discovered. → closes #6.
hyperframes-render in S2/S3.clean_speech PR-1, then wire the existing compositor.| Existing artifact | Relationship to this proposal |
|---|---|
| docs/editing-gaps-plan.md · PR-2 (overlay) | This is the overlay the report critiques. S2a/S2b extend it with the timed/tracked parameters it deliberately kept out — a continuation, not a rewrite. |
| docs/editing-gaps-plan.md · PR-3 (tracked reframe, deferred) | The field test elevates this from “deferred over-engineering” to top thesis value. Re-open it scoped to the anchor contract only (S2b), which is much smaller than the general reframe PR-3 was avoiding. |
| docs/editing-gaps-plan.md · PR-1 (clean_speech) | Shares RC-3’s upstream ASR ceiling. S3a is the common unlock; sequence them together. |
| Capability standardization (descriptor.ts, cap_add, ToolCapability) | Already solves the price half of RC-1. S1a enforces it across the static↔live boundary; S1b extends the same standard to the invocation schema, which it does not yet cover. Pure continuation of work in flight. |
docs/editing-gaps-plan.md · stale ffmpeg-cutlist reference | Minor: the no-regression section still names a cap the spike killed. Worth a one-line cleanup when PR-1 is next touched. |
hyperframes-render for all HTML/CSS-over-video work (typography, tracked PiP, styled captions).overlay stays thin wiring. Two bounded params (start/end, x/y) + one sidecar input — not a timeline or an NLE.Bottom line. Six reported gaps → four root causes → one dominant fix. The compute to satisfy this brief mostly exists on the network today; the agent couldn’t reach it because the capability cards don’t publish enough to plan or call. Finish the contract already half-built on this branch, add two small overlay parameters and one anchor binding, make one upstream ASR unlock, and add one small cap — and “agentic video editing” goes from $0.00 because nothing was reachable to reachable end-to-end, without building a compositor.
Analysis verified against storyboard-a3 @ origin/main 3980aa10 · worktree branch worktree-gtm-editing-gaps-proposal · 2026-07-27.