At Neuralwatt, we can see how the agent harness calling a model affects the energy it uses. I've been looking through a month of traffic on our cloud, and the harness seems to decide a surprising share of how much work our GPUs do. We usually talk about inference efficiency in terms of the model (how many active parameters, its attention mechanism) or the hardware (H200 vs B200 vs B300). The client deserves a closer look.
Almost all of our traffic is agentic, with far more input than output. A typical coding-agent turn sends 100k tokens or more of conversation and gets back a few hundred tokens. Most of that input is the previous turn's prompt with a little added at the end. If the earlier text is unchanged, the server already has its key-value (KV) cache and only has to process the new part. This is prompt caching, our biggest source of efficiency. Serving a cached token is far cheaper than computing it again.
