Agent-Stack Benchmark
Models × platforms × real agentic tasks — run through one open-source chat-agent harness with real sandboxes and actually-accounted cost per run. Not another “which model is smartest” leaderboard: this measures the stack you'd actually use (model + provider + tool loop), and what a solved task costs.
last run: 2026-07-09 210 runs 10 tasks × 7 models repeats per cell: 3
Leaderboard — cost per solved task
| model (stack) | success | median latency | mean $/run | $ per solved task |
|---|---|---|---|---|
| grok | 30/30 (100%) | 5s | $0.0024 | $0.0024 |
| gpt | 30/30 (100%) | 3s | $0.0051 | $0.0051 |
| gpt-5.5 | 30/30 (100%) | 4s | $0.0100 | $0.0100 |
| sonnet | 30/30 (100%) | 7s | $0.0119 | $0.0119 |
| claude-fable-5 | 30/30 (100%) | 8s | $0.0428 | $0.0428 |
| haiku | 27/30 (90%) | 8s | $0.0144 | $0.0160 |
| gptoss | 26/30 (87%) | 4s | $0.0009 | $0.0010 |
Cost vs success
Task matrix
| task | claude-fable-5 | gpt | gpt-5.5 | gptoss | grok | haiku | sonnet |
|---|---|---|---|---|---|---|---|
| bench-bash-arithmetic | 3/3 10s · $0.0555 | 3/3 4s · $0.0062 | 3/3 5s · $0.0124 | 3/3 4s · $0.0011 | 3/3 7s · $0.0032 | 3/3 10s · $0.0124 | 3/3 10s · $0.0115 |
| bench-code-run | 3/3 15s · $0.0585 | 3/3 5s · $0.0069 | 3/3 6s · $0.0145 | 3/3 4s · $0.0012 | 3/3 8s · $0.0033 | 3/3 10s · $0.0171 | 3/3 10s · $0.0130 |
| bench-data-wrangle | 3/3 10s · $0.0589 | 3/3 6s · $0.0074 | 3/3 6s · $0.0118 | 2/3 4s · $0.0010 | 3/3 7s · $0.0025 | 3/3 14s · $0.0149 | 3/3 10s · $0.0129 |
| bench-json-transform | 3/3 4s · $0.0266 | 3/3 1s · $0.0036 | 3/3 2s · $0.0080 | 3/3 1s · $0.0006 | 3/3 3s · $0.0020 | 3/3 6s · $0.0098 | 3/3 6s · $0.0080 |
| bench-multistep-pipeline | 3/3 11s · $0.0679 | 3/3 5s · $0.0063 | 3/3 5s · $0.0112 | 2/3 4s · $0.0012 | 3/3 7s · $0.0025 | 3/3 10s · $0.0175 | 3/3 10s · $0.0120 |
| bench-reasoning-sheep | 3/3 4s · $0.0261 | 3/3 1s · $0.0036 | 3/3 2s · $0.0073 | 3/3 2s · $0.0006 | 3/3 2s · $0.0022 | 2/3 7s · $0.0302 | 3/3 7s · $0.0289 |
| bench-regex-extract | 3/3 5s · $0.0269 | 3/3 1s · $0.0037 | 3/3 2s · $0.0079 | 3/3 3s · $0.0006 | 3/3 2s · $0.0012 | 3/3 6s · $0.0093 | 3/3 6s · $0.0081 |
| bench-string-reverse | 3/3 5s · $0.0263 | 3/3 1s · $0.0036 | 3/3 2s · $0.0083 | 1/3 2s · $0.0006 | 3/3 2s · $0.0021 | 1/3 7s · $0.0091 | 3/3 7s · $0.0081 |
| bench-tricky-reasoning | 3/3 5s · $0.0263 | 3/3 1s · $0.0036 | 3/3 2s · $0.0078 | 3/3 1s · $0.0006 | 3/3 3s · $0.0020 | 3/3 6s · $0.0112 | 3/3 6s · $0.0080 |
| external-tool-loop | 3/3 11s · $0.0551 | 3/3 5s · $0.0061 | 3/3 5s · $0.0105 | 3/3 4s · $0.0011 | 3/3 6s · $0.0033 | 3/3 10s · $0.0125 | 3/3 6s · $0.0091 |
Methodology
- Every run goes through the same production chat pipeline (prompts, tool loops, sandboxes) of an open-source Slack AI-teammate app — Claude models via Anthropic Managed Agents (cloud sandbox), other models via the Vercel AI Gateway (Vercel Sandbox microVMs for shell) or Cloudflare Workers AI.
- Deterministic checkers only (expected program output / regex on the final reply) — no LLM judges. Tasks that need execution really execute in a sandbox.
- Cost is the harness's real accounting: gateway-reported request costs where available, token×rate estimates otherwise (Claude sessions include platform runtime).
- Latency = wall-clock from user message to final reply, including sandbox cold starts — because that's what a user experiences.
- Caveats: small task set (v0), few repeats, prompts not tuned per model, harness differs between Claude (native agent) and others (OpenAI-style loop) — deliberately, since that's the stack each model actually gets.