Why AI Agent Token Costs Keep Climbing, and How to Cut the Overhead
A Hermes analysis measured 73% of every agent API call as fixed overhead. Here's why it's structural, why caching only gets you part way, and what actually reduces your agent token bill.

If your agent's API bill keeps climbing faster than the work it produces, the tokens you're paying for probably aren't the ones doing the reasoning. A public analysis of a production Hermes deployment (issue #4379) put a hard number on it: roughly 73% of every API call in that deployment was fixed overhead, about 13,900 tokens that shipped on every single request regardless of what the agent was being asked to do.
Two things to get straight up front. 73% is the raw, uncached overhead: the full weight of the payload before any provider-side discount. Tool schemas are static, and providers cache them via prompt caching, so on repeated workloads a large share of that overhead is discounted after the first call. That caveat matters, and we'll come back to it. But the raw number is still the right place to start, because caching is a discount on the overhead, not a removal of it. Everything you do to shrink the overhead in the first place makes the cached footprint smaller too. And the number is pinned, not current — which is the second thing.
Read, not run, and pinned to where the number came from. Issue #4379 — titled "Token overhead analysis: 73% of each API call is fixed overhead (~13.9K tokens) — data + suggestions" — was opened 2026-04-01 and read at source on 2026-10-11 as closed, state_reason: completed, closed 2026-09-12. The figure is that report's reading of that configuration, and the configuration pre-dates tool-schema deferral, which Hermes has since merged — quantified in the merged pull request's own title as −49% schemas and −6.5K tokens per call on desktop. We have not re-derived 73% against a current release and are not restating it as one: nothing was installed or run for this article, and the issue, pull-request and release state was read. Read the proportion as the shape of the problem, and measure your own stack for your own number.
That number matters because it reframes the whole cost problem. Most teams try to cut their agent bill by switching to a cheaper model or trimming conversation history. Both help at the margin. Neither one touches the structural 73%.
Why the overhead is structural
Overhead is not waste in the sloppy sense. It is the price of making a stateless API call behave like a stateful agent. Every time a framework calls the model, it has to re-send the context the model needs to act correctly:
- The system prompt and role instructions that define how the agent behaves.
- The full JSON schema for every tool the agent might call.
- Governance, safety, and formatting rules the runtime injects.
- Enough conversation history for the model to stay coherent across turns.
The model keeps no memory between calls, so all of this gets shipped again on every turn. In the Hermes breakdown, tool definitions and the system prompt dominate the overhead. When an agent loads tools across shell, filesystem, web, browser, delegation, and other capabilities, the tool schemas alone can account for thousands of tokens per call before the agent has read a single word of the task.
Then it compounds. A real agent task is rarely one call. It is a loop: read, act, observe, act again. Twenty turns on a moderate task means that fixed overhead gets paid twenty times over. The variable part of the payload, the part carrying the actual work, is often the smallest line on the invoice. Long-running agents run a lot of turns, and every one of them re-pays the toll.
The evidence is no longer anecdotal
For most of the last year, "agents are expensive" was a feeling. This quarter it became a set of numbers, from two very different vantage points.
The ground-level view came from operators posting their own token traces and converging on tool-schema bloat as the primary culprit (issue #4379): engineers showing their working, not a vendor making a claim. That thread ran from April to September 2026 and closed as completed, its final comment reporting that tool search had landed — which is the whole arc of this article in one line, and the reason the numbers above are pinned rather than live.
The enterprise view landed at the same time. When Meta moved to cap internal AI token spending, the Hacker News thread that followed made the scale concrete. Meta employees reportedly burned 73.7 trillion tokens in about thirty days. Uber exhausted its 2026 AI coding budget in four months and capped spend at $1,500 per employee per tool. ServiceNow moved to daily per-employee usage monitoring. These are companies with serious budgets discovering that agent token consumption scales in a way headcount never did.
The two signals point at the same root cause from opposite ends. The Hermes issue explains the per-call mechanics. The enterprise caps show what happens when you run those mechanics at scale without anyone controlling the overhead.
What operators are actually trying
The workarounds discussed across these threads fall into a few buckets, roughly ordered by how much they move the number.
Let the provider cache the static parts. This is the first line of defence and, for repeated workloads, the highest-leverage one you can turn on today. Because tool schemas and the system prompt are the same on every call, providers can cache them and charge a fraction of the price on subsequent hits. If you're running the same agent configuration across many calls and you haven't confirmed prompt caching is active, that's the first thing to check: it directly discounts the biggest slice of the 73%.
Route the easy turns to a cheaper model. This cuts the per-token price but not the token count. Useful, and limited. Matching each agent to the right adapter and model tier is where that choice gets made.
Trim history aggressively. Summarising or truncating old turns keeps the variable part of the payload small. That is real savings on long tasks, and most mature setups already do it.
Shrink the fixed part itself. Caching discounts the overhead; it doesn't remove it, and it doesn't help the first call or workloads where the tool set keeps changing. The remaining lever is to send fewer tool schemas in the first place, which both cuts the un-cached cost and shrinks the footprint that gets cached. That is where the community's attention has moved next.
Caching and history-trimming are table stakes now. Reducing the schema footprint is the newer front.
Lazy tool loading: the lever the ecosystem actually pulled
The pattern for that last lever is lazy tool loading, and when this article first ran we described it as still under discussion rather than shipped. Two months later it shipped. Issue #6839, "Feature: Lazy Tool Schema Loading — Two-Pass Tool Injection to Reduce Token Overhead", is closed as completed, and the pull request implementing it is merged. It went at exactly the portion of overhead that caching leaves on the table, and it is the reason the headline number above describes a configuration the upstream has since made smaller.
Read, not run. Read at source on 2026-10-11: issue #6839 is closed with state_reason: completed, closed 2026-09-02, and pull request #97979 — "feat(tool-search): core-tool deferral — curated event-triggered set behind the bridge (−6.5K tok/call desktop, −49% schemas)" — is merged into main, merged 2026-09-02. That merge falls inside the window of release v0.21.1 (tag v2026.9.7, published 2026-09-07), whose notes describe the tag as rolling up main since v0.21.0 of 2026-08-31 while saying they do not enumerate every feature in the window — so the notes do not name it, and the window is what places it. The latest release on the same read was v0.21.6, published 2026-10-08. Nothing was installed or run: issue state, pull-request state and release notes were read. The −49% and −6.5K figures are the merged pull request's own title, not a measurement of ours.
The idea is simple to state. Instead of injecting the full schema for every tool on every call, the runtime hands the model a lightweight list of what exists. If the turn needs no tools, the model answers and the heavy schemas never ship. If the model asks for a specific tool by name, the runtime makes a second pass with just that tool's definition attached. You pay for the schema of the tool you use, not for the forty you don't. The schemas that never ship are schemas the provider never has to cache either.
For an agent with a wide toolset, this is the difference between hauling the entire toolbox to every job and grabbing the one wrench the job needs. The savings land exactly where the overhead concentrates. Cheap, frequent tools like shell and filesystem stay always-loaded, while heavy, occasional ones like browser control, code execution, and web search load only on demand.
It isn't free as a design. It can add a round trip on the turns that do need a tool, and it asks the runtime to manage a two-phase call cleanly. That cost shows in the shape of what shipped: by its own title the change defers a curated, event-triggered set rather than making every schema lazy, which is the conservative version of the idea and not the whole of it. But for the overhead-heavy, many-tool agents that dominate real deployments, it goes straight at the largest line item instead of nibbling the edges, which is why it got built.
Where Paperclip fits
The reason this matters for how you choose a runtime is that overhead is easiest to control when it's a setting, not a re-engineering project.
Paperclip exposes a per-agent token budget, a ceiling on how much an agent may spend before the runtime steps in. Setting that budget, along with the turn and concurrency caps that stop a runaway run, is part of wiring the company after import. For a cost-conscious operator that's the lever that turns "agent token consumption scales in a way headcount never did" from a surprise on the invoice into a number you set on purpose. It's the same discipline the enterprise teams above are now imposing by hand, available as a default you choose once rather than a cap you bolt on after the bill lands.
On the schema-footprint side, lazy tool loading has gone from proposal to merged upstream, and Paperclip's runtime-level architecture, where tools and budgets are configuration rather than per-agent code, is the right shape to carry that discipline rather than something each agent has to re-implement. A Blueprint can ship with cost-lean defaults already chosen, so the overhead awareness the Hermes thread is reaching for by hand is simply how the setup starts.
The takeaway
That 73% reading is worth internalising because it changes where you look, and it keeps that value even though the upstream it came from has since shrunk its own number. Caching gets you a discount on the overhead; it doesn't erase it, and it doesn't touch the first call or a shifting tool set. The cheapest structural win in an agent stack is usually not a cheaper model — it's refusing to build thousands of tokens of unused tool schemas into every one of a thousand calls in the first place. The frameworks moved to lazy loading and token-budget control because that is where the structural cost sits, and the enterprise teams getting throttled are the proof of what happens when nobody owns that number.
Subscribe below for more on running AI-native operations. Practical field notes on agents in production, straight to your inbox.


