新闻详情

新闻详情

首页 / 资讯中心 / 详情

Agent: The Washing Away of Wrongs — Prefix Caching: How MCP Ordering Affects Model Latency and Cost

发布时间:2026/11/18 18:43:14来源:尧图网络
Agent: The Washing Away of Wrongs — Prefix Caching: How MCP Ordering Affects Model Latency and Cost
Agent: The Washing Away of Wrongs — Prefix Caching: How MCP Ordering Affects Model Latency and CostAbout this seriesThe Washing Away of Wrongs(《洗冤集录》) is a work of forensic medicine by Song Ci of the Southern Song dynasty. It brings together earlier knowledge of postmortem examination and his own experience investigating judicial cases.“Do not take an accusation at face value. Examine it carefully, and make every effort to establish the facts.”The book repeatedly stresses examining the evidence in person and checking it against multiple sources:“Do not shrink from foul smells and filth … go to the body yourself … inquire widely, reconcile the accounts, and never accept something as true on the word of just one or two people.”These are our translations of the excerpts quoted in the Chinese article.Agent: The Washing Away of Wrongs, or「Agent 洗冤集录」, borrows its name from this commitment to examination and evidence. It is also a reminder to ourselves: be less impatient, and find out what happened before reaching a conclusion.As developers and maintainers of an agent harness, we often hear user feedback that boils down to two words:slow (a performance problem)anddumb (a capability problem). In our experience investigating these reports, most of the causes turn out to be bugs in the harness’s design or implementation.Missing context, incorrectly relayed tool results, and broken state transitions can all make an agent look “dumb.”Repeated execution, cache misses, and poorly designed interaction loops can make it slow.Often, improving an agent’s capabilities and performance means fixing these specific bugs.The user’s experience is the starting point. The cause still needs to be examined, step by step: what did the model actually receive, what did the tools return, and how did the system handle those results?This series documents those investigations and fixes. We start with original requests, logs, and code, then use experiments to test our inferences. The reference to “washing away wrongs” expresses a simple aim: whenever we attribute a failure to an agent, we should be able to produce evidence others can inspect and explain the mechanism behind it.It started with MCP orderingWhen an agent is slow, it is easy to suspect the model first. Is it doing too much reasoning? Is the context too long? As input tokens keep accumulating, “slow and expensive” starts to seem like the inevitable price of a complex task.Sometimes, though, the reason an agent spends more time recomputing old content is an unremarkable piece of string assembly code.We encountered exactly this bug. After adding MCP support, our harness collected service descriptions from a Go map and joined them into the system prompt.The descriptions did not change by a single word, but their order could change on every request.A person could read those paragraphs in any order and understand the same thing. For a prompt cache that depends on an identical prefix, however, the reordering could affect reuse of the entire conversation history that followed.Using one real continuation sample, we ran six paired measurements comparing two request variants: one that retained the reordering and one that kept the service descriptions in a fixed order. With fixed ordering, the cache hit rate rose from39.01% to 99.15%, and mean model-call latency fell from7.31 seconds to 4.24 seconds, a reduction of about 42%. Total input tokens stayed the same; more computation over existing content could be reused. Where cache reads cost less, that also creates an opportunity to reduce input charges. Actual billing and end-to-end task latency still need separate validation.This article uses that MCP ordering bug to explain how prefix caching affects model latency and cost. The investigation began with an easily overlooked question:when unchanged content is sent to the model again, does it really stay unchanged?Why ordering affects latencyAn agent usually needs several model calls to complete a task. Each time a tool finishes, the harness appends its result to the conversation history and sends the updated input to the model. As the task progresses, more and more content appears repeatedly in requests.Prompt caching can reuse computation over that repeated content. OpenAI’s documentation explains that reuse requires an exact prefix match. This makes it useful to put stable instructions first and append new messages at the end.First, consider how requests grow.One user turn can contain multiple model calls.The model may ask to read a file, receive the tool result, and then answer. A follow-up from the user begins the next turn.Request contentTurn 1: first callTurn 1: after tool resultTurn 2: user follow-upStable instructions, tool definitions, MCP descriptionsFirst includedUnchangedUnchangedUser message U1First includedUnchangedUnchangedAssistant tool call T1 and tool result R1Not yet producedAppendedUnchangedAssistant answer A1 and user follow-up U2Not yet producedNot yet producedAppendedRelationship to the preceding requestAssume no usable cache yetPrevious input is a prefix of the new inputPrevious input remains a prefix of the new inputBlue blocks mark the prefix that matches the preceding input. Yellow blocks mark newly appended input. The diagram shows why cached computationcanbe reused. The actual number of cache hits also depends on cache lifetime, server-side processing, and other conditions, so it must be checked in the reported usage. This is a teaching example, with details such as reasoning items omitted; it is not the original conversation used in the experiment below.Our investigation was inspired byUnrolling the Codex agent loop. The article describes how unstable MCP tool ordering once caused cache misses. When we inspected our own requests, we found a similar problem:MCP service descriptions were assembled in an unstable order.Interestingly, even the developers of Codex had fallen into this trap. The relevant fix, PR #2611, was merged on August 25, 2025, more than a year before our investigation. I had read that blog post long before, yet failed to recognize the same problem while building our own system. Reading about someone else’s mistake does not necessarily stop us from repeating it. That is another reason for writing this account: if it helps even one more developer avoid the same pitfall, it has served a purpose. As the Chinese phrase “后人复哀后人” warns, otherwise, later generations may repeat the same mistakes and in turn become the subject of others’ lament.Using three anonymous services, we can compare normal appending with reordering:The service descriptions have not changed, but they appear before the conversation history. Once they are reordered, the history that follows no longer has the same complete prefix as the preceding request. The earlier portion may still hit the cache, while reuse of the later portion is affected.The cause in our code was simple: we read service descriptions from a Go map and concatenated them directly. Go map iteration order is not fixed, and the assembly function did not sort its input.We fixed this at the shared assembly entry point: first copy the valid service descriptions, then sort them by service name. Both initialization and continuation use this entry point, so the same set of descriptions now produces the same text. The description content and tool permissions remain unchanged.It is useful to distinguish the protocol from its implementation here. MCP’stools/listreturns an array. The Codex bug described in the article was in its Rust implementation: the fix collected tools from an internal HashMap into a list and sorted them by fully qualified tool name; see PR #2611. JavaScript’s nativeMap, as used in TypeScript, iterates in insertion order, but does not automatically sort by name. If items are inserted in a different asynchronous completion order each time, the final sequence can still change. What matters is that identical content produces an identical sequence.Testing with real requestsIn a session log containing 99 requests, we found 32 changes that affected only the order of these descriptions. We selected one continuation for the experiment: the number of tools stayed at 41, the history grew from 76 to 79 entries, and the reordered MCP service-description block contained 276 characters.Usingqwen3.8-maxthrough our existing gateway, we compared “reordered” and “stable order” variants. Apart from the ordering and the markers used to isolate the two variants’ caches, tools, history, and model parameters were identical.We interleavedsix paired experiments, making 24 real calls: 12 warm-up calls and 12 measurement calls. For each variant in each pair, we first sent the preceding input to establish the cache, then measured the continuation request. The output limit was fixed at 64 tokens, and the business tools were not actually executed.The table includes only measurement calls: six per variant, with values averaged per call.MetricReorderedStable orderChangeTotal input tokens62,99862,998UnchangedCache-read tokens24,57662,464Up by 37,888Uncached input tokens38,422534Down 98.61%Cache hit rate39.01%99.15%Up 60.14 percentage pointsMean time to first reasoning chunk5.896 s2.769 sDown 53.03%Mean model-call latency7.311 s4.241 sDown 41.99%Calls with stable ordering were faster in all six pairs, with a mean reduction of3.07 seconds.Two quantities must be kept separate:total input did not decrease; uncached input fell by 37,888 tokens per call.Each request still contained the full history. More of that history could simply reuse existing computation.“First reasoning chunk” in the table means the model’s firstthinking_delta, not the first text shown to the user. The latency results cover this particular model call with a long history and a short output. They do not establish that complete tasks, or all users’ requests, become 42% faster.Cached input can also cost lessCaching also affects input charges. The following examples use selected GPT and Claude prices checked onSeptember 29, 2026, alongside the Qwen model used in this evaluation. Prices are in USD per million input tokens.ModelStandard inputCache readUnit-price reduction for cache hitsQwen3.8-Max (this evaluation)$2.00$0.2587.5%GPT-6 Astra$10.00$1.0090%GPT-6 Sol$2.00$0.2090%Claude Fable 5.1$10.00$0.2597.5%Claude Opus 5.5$4.00$0.2095%Claude Sonnet 5$2.00$0.2090%Pricing scope: Qwen uses implicit-cache pricing for the international deployment in Singapore; GPT uses the Standard short-context tier; Claude uses standard pricing with default global routing. Batch pricing, acceleration charges, and regional surcharges are excluded.Cache reads are cheaper, but writes also count. GPT-6 cache writes are charged at1.25 timesthe standard input rate. Claude charges1.25 timesfor a five-minute cache write and2 timesfor a one-hour write. These are the applicable rates for tokens written to the cache, not an additional charge of the same amount on top of standard input pricing.This explains why costs can fall even when total input tokens remain unchanged: more input is billed at the cache-read rate. However, the percentages in the table describethe input unit-price reduction for cache hits. A complete request also incurs charges for uncached input, cache writes, and output. Our experiment ran through an internal gateway, so public prices do not represent its actual bill. GPT and Claude did not participate in the latency experiment above.Lessons for agent developersWhether prompt caching works well depends in part on how the harness organizes input. An ordering change that seems trivial to a human can prevent reuse of a large amount of computation when it appears before a long conversation history.We fixed the ordering of MCP service descriptions and added regression checks: different permutations must produce identical text, while real content changes must still appear in the prompt. Production validation still needs to cover tool execution, long outputs, and multi-turn tasks, with attention to how long users actually wait.When an agent seems “slow” or “dumb,” first examine your own implementation.Blaming the model can end an investigation at “that’s a model problem; agent developers cannot control it.” Yet an apparent limitation in model performance or capability may still be an agent bug. Poorly organized context, incorrectly relayed tool results, and repeated execution are all things developers can inspect and fix. This cache failure is one example. Check the actual requests, tool results, and execution path before deciding which issues are implementation defects and which reflect the model’s limits.We enhanced our Cache Token Trace implementation and recorded two lessons in the repository’sAGENTS.mdas conventions for future development.Consider caching whenever you change the system prompt.Alongside deciding whether content is necessary, check how often it changes and whether it alters an existing prefix. Keep identical content in a deterministic order, and append dynamic information where semantics and permissions allow. Constraints that need updating must still be updated; caching must never take priority over correctness.Treat input caching as an important agent performance metric.Record total input tokens, cached input tokens, uncached input tokens, and the cache hit rate (cache-read tokens / total input tokens), then interpret them alongside response latency. This separates “we sent less input” from “we reused more computation over the same input,” and helps catch performance regressions caused by prefix changes earlier.Experiment notes: September 17, 2026; one real continuation sample. We first ran two pairs, then expanded to six. The paired bootstrap 95% interval for the mean reduction in model-call latency was 2.136–4.247 seconds. Token counts came from the gateway’s raw usage. Requests, outputs, and timings for all 24 calls were retained and verified. Earlier mocks were used only to screen for the issue and are not included in the table. The code fix and this pre-deployment experiment are recorded separately; production impact remains to be validated.References: OpenAI, Unrolling the Codex agent loop; OpenAI, Prompt caching. The performance data in this article comes from our own measurements through the Qwen integration.
网站建设高端定制企业官网
RELATED

相关资讯

更多精彩内容,欢迎继续阅读

较早相关资讯

最新相关资讯

【外设】之大彩串口显示屏 2026/11/17 7:04:08

【外设】之大彩串口显示屏

大彩串口屏初步使用 1 .官网下载 STM32 屏幕 GUI 设计资料 http://www.gz-dc.com/category/typeid/4112 找到 STM32 Keil 工程,移植相关代码因项目而异进行移植,由于项目简单,本人只对用到的指令接口进行修改。 比如:注意事项&…

阅读更多 →
无法下载Windows系统iso文件 2026/11/18 14:20:46

无法下载Windows系统iso文件

当我遇到这个问题的时候,我打开了一个网站: 登录 然后我打算下载的时候: 突然那个官方的连接就可以下载了:

阅读更多 →
【清华代码熊】DeepSeek V4.1 Flash 后训练详解 2026/11/17 19:43:34

【清华代码熊】DeepSeek V4.1 Flash 后训练详解

📌 上期解析了 DeepSeek V4.1 Flash 模型架构改进,本期解析 DeepSeek V4.1 Flash 预训练/后训练技术: 🌟 预训练:45T 文本 多模态混合语料、直接训练 sparse attention(取消 DeepSeek V4 的 dense 冷启动&…

阅读更多 →
Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic ... 2026/11/18 11:04:06

Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic ...

文章主要内容和创新点 主要内容 本文聚焦于多模态大语言模型(MLLM)强化学习(RL)训练中的效率问题,提出了一个名为Shuffle-R1的框架。研究发现,当前RL训练存在两个关键缺陷: 优势值坍缩(Advantage Collapsing):批次中大多数优势值集中在零附近,导致有效梯度信号被淹…

阅读更多 →
PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction 2026/11/18 15:55:22

PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction

一、文章主要内容总结 本文聚焦于利用大型语言模型(LLMs)实现个人身份信息(PII)脱敏的研究,旨在解决传统脱敏方法(如基于规则的系统、领域特定命名实体识别(NER)模型)泛化能力差、跨格式/跨语境适应性弱的问题。 研究通过全面评估多种LLM架构(包括密集型LLM(D-LLM…

阅读更多 →
LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model 2026/11/17 19:43:32

LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model

文章主要内容和创新点 主要内容 本文聚焦于二进制图像-文本相关性评估任务(判断图像与文本“相关”或“不相关”),针对该任务中文本格式多样、相关性定义随场景变化等挑战,提出了基于多模态大语言模型(MLLM)的解决方案LLaVA-RE。 模型设计:LLaVA-RE基于LLaVA 1.5架构,…

阅读更多 →

今日资讯

本周资讯

本月资讯

看完文章仍有疑问?

联系尧图顾问,获取一对一建站咨询

立即免费咨询 📞 400-888-8888
📞 ✉