In 27,280 coding agent runs, 64% of inference spend went to runs that failed
FrontierHarness Eval v1.1 ran 22 coding-agent harnesses with 5 models on the same 62 tasks, 4 times each: 27,280 runs. The best-value harness was different with each model, and cost ranks moved far between models. With the model fixed, the most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest.
In 27,280 agent runs, 64% of inference spend went to failed runs, highlighting inefficiencies in model harnesses.
27,280 agent runs were analyzed with 64% of inference spend on failed runs.
Different models required different optimal harnesses for cost and pass rate.
Four harnesses showed top performance in cost or pass rate across models.
Summarised automatically by AI from the original article by Hacker News. AI can make mistakes, so check the original for details.
We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?
Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.
V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.
Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.
Read the full story on Hacker NewsThat's the opening of the story. The full piece is published by Hacker News.
To celebrate Mistral Vibe + Le Chonk becoming the world’s best open model-harness duo, we’re making Vibe free in Europe for the first 3 months. (Yes, it’s actually better than…
The highly classified network that the National Security Agency uses for its internal communications is down, according to six people familiar with the matter.