This page measures orchestration tokens — the LLM cost of deciding, plumbing and re-stating the work around a render. It deliberately does not measure GPU inference, which is billed per pixel or per second and is identical in every configuration. If a page tells you an agent framework made your renders cheaper, it is measuring the wrong thing.
360 trials · 3 configurations · 6 workloads · 3 briefs. Cost units = billed input + 10 × output (output is ~10× input price). Every constant is calibrated against production measured on 2026-08-10 — provenance table at the bottom. Regenerate with npm run bench:skills.
Every token in this bench is booked to exactly one of these terms. That is what makes the result actionable rather than promotional: you can see which term a given workload is dominated by, and what in the implementation removes it.
This is the most useful result on the page. Baseline cost split by term, at a 192-model catalog. Generation work is dominated by model discovery and retry. Long-video editing is dominated by state re-hydration — the transcript. Same product, different mechanism.
| Workload | Floor | State re-hydration | Model discovery | Step handoff | Retry | Saving |
|---|---|---|---|---|---|---|
| 7-step generation · 24 scenes | 12% | 13% | 31% | 6% | 39% | 7.8× |
| 19-step long-form · 24 scenes | 7% | 7% | 39% | 8% | 38% | 7.2× |
| 9-step edit · 10 min source | 5% | 19% | 33% | 5% | 38% | 10.0× |
| 9-step edit · 60 min source | 3% | 52% | 4% | 1% | 41% | 45.9× |
| 14-step mixed · 60 min source | 5% | 45% | 8% | 1% | 40% | 17.2× |
Bars are stacked by cost term, all scaled to the same axis within a section, so the shape of the saving is visible, not just its size.
Generate a multi-scene piece from a thin brief. Every scene runs a full key-frame → animate → grade → voice → mix chain. This is the reference generation workload.
flux-devtopaz-upscalepixverse-i2vffmpeg-colorgradechatterbox-ttssonilo-t2m · once per projectffmpeg-muxPer-scene steps run once per scene; the music score is generated once for the project.
What this says: Model discovery and retry together account for roughly 70% of the baseline. Both go to zero in the agent path, because create_media receives an intent and the capability layer resolves the model, adapts the params and re-dispatches on failure — all server-side, none of it in the context window.
What a real long-form piece actually ships through: 12 per-scene steps (key frame, character lock, upscale, animate, extend, grade, stabilize, reframe, voice, lip-sync, SFX, mux) plus 7 project-level finishing steps (score, concat, transcribe, subtitles, brand overlay, LUFS mix, export). At 24 scenes that is 295 distinct model and tool invocations.
gpt-imagekontext-edittopaz-upscalepixverse-i2vflux-3-extendffmpeg-colorgradeffmpeg-stabilizeffmpeg-reframechatterbox-ttsveed-lipsync-v2mirelo-sfx-v2vffmpeg-muxsonilo-t2m · once per projectffmpeg-concat · once per projectwhisper-word · once per projectffmpeg-burn-subtitles · once per projectffmpeg-overlay · once per projectffmpeg-audio-mix · once per projectffmpeg-export · once per projectThis is the pipeline the brand-film and luxury-product playbooks in this repo run, expressed in capabilities that are live on production today.
What this says: Note the saving ratio DROPS as scene count rises (10.1× at 6 scenes → 7.2× at 24). That is honest and worth understanding: the baseline's dominant term is model discovery, which amortizes across scenes — you pick the i2v model once and reuse it. The agent's cost, meanwhile, grows linearly with scenes because each scene is a real turn. Longer pipelines widen the gap; more scenes narrow it.
Cut an existing video: transcribe, analyse shots, pick moments, strip silence, trim, reframe vertical, grade, burn word-level captions, export. Nothing is generated. The scale driver here is not scene count — it is source duration, because the artifact the model must reason over is a word-level transcript.
whisper-word · once per projectmarlin-video · once per projectfind_moments · once per projectffmpeg-silence-cut · once per projectffmpeg-trim · once per projectffmpeg-reframe · once per projectffmpeg-colorgrade · once per projectffmpeg-burn-subtitles · once per projectffmpeg-export · once per projectEvery step is project-scoped. A word-level cue serializes to ~34 chars, and speech runs ~150 wpm — so a 60-minute source is ~306,000 chars ≈ 76,500 tokens of transcript, per round that touches it.
What this says: This is where the largest saving in the bench lives, and it comes from a completely different term than generation does. At 60 minutes, state re-hydration alone is 52% of baseline cost: the transcript and edit-decision list get re-pasted on every decision round. Livepeer Agent never puts the transcript in the context window — transcribe and find_moments produce and hold it server-side and return a compact moments list. The saving therefore grows with video length: 7.2× at 3 minutes, 45.9× at 60.
The most common real request: “here is my footage, cut it down, add B-roll and a voiceover, make it vertical.” Pays the editing chain's duration-driven state cost AND a 5-step generation chain per inserted shot. Insert count scales with source length.
whisper-word · once per projectmarlin-video · once per projectfind_moments · once per projectffmpeg-silence-cut · once per projectffmpeg-trim · once per projectffmpeg-reframe · once per projectffmpeg-colorgrade · once per projectffmpeg-burn-subtitles · once per projectffmpeg-export · once per projectflux-devseedance-mini-i2vffmpeg-colorgradechatterbox-ttsffmpeg-concatThe 9 editing steps run once per project; the 5 insert steps run once per generated insert (≈ one insert per 4 minutes of source).
What this says: Mixed work sits between the two pure cases, and that is the practical point: as a workload shifts from generation toward editing, the saving migrates from model discovery to state re-hydration without shrinking. At 60 minutes, state is 45% of baseline and the saving is 17.2×.
192 is measured, not projected: it is what GET /capabilities returns on production today (157 AI models + 35 deterministic tool containers). 300 is the forward bar for a saturated marketplace. Only the model-discovery term moves — which is exactly the point: the baseline gets worse as the ecosystem grows, and the agent path does not.
| Workload | saving @ 192 | saving @ 300 | Δ |
|---|---|---|---|
| 7-step generation · 24 scenes | 7.8× | 9.2× | +1.4 |
| 19-step long-form · 24 scenes | 7.2× | 8.8× | +1.7 |
| 9-step edit · 60 min | 45.9× | 46.9× | +1.1 |
| 14-step mixed · 60 min | 17.2× | 18.0× | +0.9 |
A thin brief becomes N quality scene prompts, and nothing else. No model selection, no multi-step chain. Isolates the cost of the authoring pipeline (plan → develop → anchor → optimize) that the baseline runs by hand and the agent absorbs into one cached context.
What this says: 3.2× at 6 scenes rising to 5.8× at 24, driven entirely by state re-hydration growing O(N²) — every scene-development pass re-pastes every prior scene. This is the floor of the agent's advantage: it holds even with a single model and no pipeline at all.
If you have already written every prompt yourself and want exactly one render each, the agent path costs more, not less:
The cause is concrete and measured: a creative agent session carries a 48,264-char tool surface (the 10 verbs a gen+edit session touches, measured on production). Caching amortizes it across a real session, but a one-shot render with a finished prompt never gets the chance. There is a mitigation in the product — the lean tool profile (STORYBOARD_LEAN_TOOLS, 12 verbs) — and this bench does not credit it. The number stays on the page because a benchmark that only reports wins is not a benchmark. The honest summary: the agent earns its overhead the moment there is more than one step, more than one scene, or more than one model choice to make.
create_media takes an action and an intent, not a model id. resolveCapability plus a 184-entry normalized registry (mean 553 chars per descriptor, vs ~900 for a raw provider doc) select server-side. Provider documentation never enters the context window.applyCapabilityAdaptation normalizes each capability's parameter contract before dispatch — the same layer whose absence made raw run_capability run a 42% error rate against create_media's 7.5%.input-requirements guards ensure a fallback matches input modality, not just output kind. A failed render re-dispatches server-side; the model is never asked to diagnose it.transcribe and find_moments run ASR and shot analysis server-side and return a compact moments list — the word-level transcript is produced, used, and stored without ever being pasted into a prompt.Measured against production on 2026-08-10. The previous revision of this bench used a 3,000-char tool surface and a 40/100-model catalog; both were far below reality, which understated the baseline's architecture cost and overstated its catalog cost.
| Constant | Value | Source |
|---|---|---|
| Live capabilities | 192 (157 AI + 35 tool) | GET /api/capabilities |
| MCP verbs / full surface | 187 verbs · 260,451 chars ≈ 65,100 tok | POST /api/mcp tools/list on production |
| Agent tool surface (cached) | 48,264 chars ≈ 12,070 tok | the 10 verbs a gen+edit session touches |
| create_media schema alone | 16,539 chars ≈ 4,130 tok | same tools/list response |
| Normalized descriptor | mean 553 chars (median 364) | lib/capabilities/registry.json, 184 entries |
| Raw provider doc | 900 chars | midpoint of upstream fal / Luma / xAI docs |
| Per-call param repair rate | 35% | run_capability 42% vs create_media 7.5% error rate |
| Prompt cache discount | 90% of input rate | Anthropic / Gemini caching |
| Output price multiple | 10× input | Claude Sonnet + Gemini Flash |
| Word-level cue size | 34 chars @ 150 wpm | whisper-word output shape |
Sources: token-model.ts · run-sim.ts · bench README · plan §6