Tool call accuracy is an important part of serving open-weight AI models. When your tool calls aren’t accurate, the model builds up on nonsensical context. We’ve been carrying our own fork of vLLM to hit 100% in tool call schema accuracy with Kimi and DeepSeek. Fortunately, vLLM 0.31 added support for constrained decoding, a technique to guarantee schema-adherence. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.
vLLM 0.31 introduces constrained decoding to improve tool call accuracy, achieving 100% schema adherence for some models.
Tool call accuracy is critical for agentic workloads and model quality.
vLLM 0.28 had 92% schema accuracy, with 8% failures in tool calls.
Constrained decoding, now in vLLM 0.31, ensures full schema adherence.
Summarised automatically by AI from the original article by Hacker News. AI can make mistakes, so check the original for details.
When I use an open-weight model, I see results that are nowhere close to what the quality benchmarks indicate.
Tool call accuracy is an important part of serving open-weight models. When your tool calls aren’t accurate, the model builds up on nonsensical context. This leads to low quality results, particularly for agentic workloads where multi-turn sessions are common.Sadly, many providers and widely-used inference engines don’t fully pass tool-call accuracy tests. For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.Fortunately, vLLM recently added support for constrained decoding, a technique to guarantee schema-adherence. This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.In this blog post, we first describe tool call schema accuracy and why it’s important in serving agentic workloads. We then talk about constrained decoding and how it fixes the schema accuracy problem.
Read the full story on Hacker NewsThat's the opening of the story. The full piece is published by Hacker News.
"I’ve spoken with 60 air traffic controllers tonight about what Sean Duffy said at the White House today," said one journalist. "None of them are buying what he said."
Rachelle Hampton is a writer and co-owner at Defector Media, and the host of Normal Gossip, a podcast of anonymous listener-submitted stories about everyday drama. Whether that's…
AI summary TechCrunch recommends the X3 and X4 Pro e-readers for their portability and features, with the X4 Pro offering a backlight for better versatility.
