Skip to content
NLEN
Illustration: Workers AI KV cache leak: GLM-5.3 exposed other customers' data

Workers AI KV cache leak: GLM-5.3 exposed other customers' data

By Ivo Donker · — with AI assistance (DeepSeek V4.1 Flash)

Cloudflare has published an incident report about a tenant isolation issue affecting exactly one model on Workers AI: @cf/zai-org/glm-5.3, in other words GLM 5.3. Between August 27 and September 9, 2026, a small number of customers using that model through Workers AI received responses that contained data from requests other customers had made to the same model shortly before theirs. In total 155 responses were affected, all of them to prompts longer than 681,280 tokens. According to Cloudflare the problem has been fixed, and other Workers AI models were not affected.

The report is on Cloudflare's own blog, was written by Kevin Flansburg and Michelle Chen and appeared on October 26, 2026 as a coordinated disclosure together with the SGLang team. The report came in on September 8, 2026 through Cloudflare's HackerOne program, from security researcher Gadi Evron of Knostic. Cloudflare says it found no evidence of deliberate exploitation. The most instructive part of the report is not the incident itself but the two causes behind it: a configuration mismatch and a bug in disaggregated prefill/decode. Cloudflare therefore also gives advice to other operators running SGLang this way. What inference does at a basic level is explained in the explainer on inference, where you see prefill, decode and memory use in relation to each other.

What Cloudflare reports

The core of the report: customers of the GLM 5.3 model on Workers AI could, with very long prompts, receive text that came from earlier sessions of other users. The researcher reported that very long prompts sometimes returned text belonging to other users' sessions. Cloudflare investigated and was able to reproduce it. After excluding Cloudflare's own test traffic and the researcher's, the requests identified were consistent in timing and volume with ordinary long-context use. Cloudflare therefore found no evidence of deliberate exploitation.

Cloudflare confirms in the report that this concerned one model. Other models on Workers AI were not affected. Cloudflare also does not state a number of affected customers; it remains "a small number of customers". Just as important: the leak only occurred with prompts longer than 681,280 tokens. That is a threshold you are unlikely to reach with ordinary chat use, but which does occur within agentic coding workflows — the kind of work GLM models are built for, according to Cloudflare. If you want to follow how Cloudflare Workers is developing for AI builders, the overview of Cloudflare Workers signals is a practical starting point.

The exposure was present from the moment the model launched on Workers AI on August 27, 2026. That matters for anyone reconstructing their own usage: the window in which responses could contain other customers' data runs from that launch date until the model was taken offline on September 9, 2026. Cloudflare's report does not describe the incident as an attack or as misuse, and the requests it identified look like normal long-context workloads.

Scope: only GLM-5.3 on Workers AI

It is tempting to read this as a broad platform problem, but it is not. According to Cloudflare, GLM 5.3 was the only model affected, and every other disaggregated deployment on Workers AI has KV pools that are larger than its context length, on both prefill and decode. Put differently: the fault lay in the combination of one specific configuration and one specific code path in the engine, not in the shared infrastructure as a whole.

That scoping is not merely a precautionary statement. It also determines what you need to do as a user. If you use other models on Workers AI, this report gives you no reason to change anything. If you used GLM 5.3 with extremely long prompts, it is sensible to review your own situation — not because the problem is still occurring, but because you want to know whether you were inside the window from August 27 to September 9, 2026. For layered applications with multiple customers on one infrastructure, building multi-tenant LLM applications is where isolation and separation questions come together.

It is worth being precise about what the report does and does not say about the data involved. Cloudflare describes responses containing data from other customers' requests to the same model: prompts and partial generations that were still present in KV cache pages. The report does not characterise the data any further than that, and it draws no legal conclusions. For operators, the practical consequence is that GLM 5.3 output from that window to prompts above 681,280 tokens may have contained content that did not originate from your own request.

How the KV cache works in this system

To understand the cause, it helps to know what a KV cache is. A language model processes a prompt in steps and stores key and value tensors per layer as it goes: the KV cache. On every subsequent token the model does not have to recompute those intermediate results. That is why a long conversation runs faster than a series of separate short prompts. The cache takes up GPU memory, and how much room is left for it plays a central role in this incident. If you want to know how to handle it in practice, the guide on the KV cache and how to configure it in llama.cpp and vLLM is a good place to go deeper.

Cloudflare runs GLM 5.3 with the open-source inference engine SGLang in a disaggregated configuration. Prefill GPUs process the prompt and build up the KV cache; that cache is transferred via RDMA to decode GPUs, which then generate the response. Cloudflare uses Mooncake for this. RDMA here simply means that memory on one machine goes directly to memory on another, without a detour through the CPU and the ordinary network path. That saves latency when moving large amounts of KV data.

The cache in this system is a shared pool of fixed-size pages. Those pages are not zeroed between requests — according to Cloudflare this is common among inference engines and necessary for the best performance — so the engine has to track precisely which page belongs to which request. As soon as that bookkeeping is wrong somewhere, a request can read pages that still contain the data of a previous request. That is exactly what happened here.

The design assumes three things hold at once: the pool is large enough for the configured context length, the engine refuses work it cannot place, and page ownership is respected on the receiving side. The incident is instructive precisely because all three assumptions failed together, and each of them is something you can check in your own deployment.

Cause one: a KV pool smaller than the context window

GLM 5.3 was deployed on Workers AI with a context window of 1,048,576 tokens, but the KV cache pool on the prefill GPUs had room for 681,280 tokens. Those two values are configured separately in SGLang: the pool size follows at startup from the GPU memory remaining after the model weights are loaded, while the context length is a separate configuration value. Nothing in the engine forces them to agree.

Cloudflare was aware of the mismatch before launch and expected the engine to return an error when a prompt was too long. In the long-context tests, however, successful HTTP responses came back. Cloudflare's testing looked at HTTP response codes and performance at long contexts, not at the content and coherence of the model output. That is how the mismatch went unnoticed: the system appeared to work, and at the level of status codes it did work.

That is a pattern you also encounter in your own stack. A health check that only looks at status codes and latency tells you nothing about whether the output still relates to the input. As soon as you configure context lengths above the size of your KV pool, you are inviting problems that such a check does not catch. The mismatch by itself, though, was not enough to produce the leak; it needed a second fault in the engine to turn a capacity limit into a data exposure.

Cause two: an SGLang bug in disaggregated prefill/decode

The mismatch alone did not lead to the leak. There was also a bug in the disaggregated prefill/decode mode of SGLang. A prompt that did not fit in the pool was internally marked as rejected and replaced with a one-token placeholder. Instead of aborting and returning a client error, prefill generated KV cache for that placeholder and transferred it to the decode node. That decode node received "transfer complete" and then, in the attention step, read stale KV cache pages that still held data from earlier requests: prompts and partial generations from other users.

The client received HTTP 200 instead of HTTP 400. The reply was coherent and contained content from other people's KV pages. That makes this type of fault hard to recognise: nothing broke, no error message appeared, and the response looked normal. The bug had already been reported to SGLang by another user as issue #30233 on July 6, 2026.

Note what the two causes do together. The configuration mismatch creates the situation in which a prompt cannot be placed. The engine bug determines what happens next: the request is not refused, but converted into a placeholder and pushed through the disaggregation pipeline. On the decode side, the transfer completion signal is taken at face value, and attention runs over pages that belong to someone else. You need both faults for the exposure; either one alone would have produced an error or a refusal rather than a leak.

Timeline

Cloudflare publishes the timeline in UTC. That is useful to know if you want to map it back to your own logs, which are often in local time or another zone.

If you are reconstructing your own exposure, the relevant window is August 27 to September 9, 2026, and the relevant requests are those with prompts above 681,280 tokens. The model was offline for a few hours on September 9 between 14:00 and 19:40 UTC; requests in that period received HTTP 400 rather than a model response.

Remediation and hardening

The GLM 5.3 configuration has been changed: the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length. That means a prompt that fully uses the context window fits comfortably within the pool. In addition, the engine has been patched so that a request with a pending rejection is retired at the disaggregation admission queue with its original error, before any KV is transferred. The upstream SGLang pull requests #36988 and #38935 address this incident.

More hardening has followed since the incident. Deferred release has been extended so that prefill source pages are held and every failure type is covered. Mooncake transfer batches are bounded. The load balancer has been changed so that a client disconnecting no longer aborts a request mid-transfer. And every change to a disaggregated deployment now includes a check: KV pool size against context length, deferred release on both roles, transfer batch bounds and the required patches. The engine fix and the hardening have been applied to every model Cloudflare serves with SGLang.

For the longer term, Cloudflare is working with the SGLang team on integrity checks inside the engine itself, with page-level tags so that a request cannot read a KV page it does not own. A prototype of that cost under 1% throughput and tail latency in tests, but was disabled because of reliability issues. Cloudflare is also applying formal verification methods to the RDMA protocol for KV cache transfer.

Advice for operators running SGLang themselves

The recommendations in the report are directly applicable, even if you are not a Cloudflare customer. On every prefill and decode node, measure the KV pool size against the context length, and do so from the engine's startup log, after every change to weights, quantization, parallelism or memory fraction. If the pool is smaller than the context length, lower the context length. Do not rely on a proxy to enforce the limit: by definition it sits on the outside of the problem.

Upgrade to SGLang v0.5.22 or later; that version includes #36988 and #38935. Run deferred KV release on both roles and bound Mooncake transfer batches. Turn on corruption detection where it is supported: --disaggregation-enable-kv-checksum and --kv-canary. Cloudflare does note that these options do not yet cover every model architecture, so they are a supplement to the configuration check, not a replacement for it. Finally, make sure your proxy does not generate unnecessary aborts.

What this means for Workers AI users

For most Workers AI users, nothing changes. This concerned one model, GLM 5.3, and that model was taken offline on September 9, 2026 and restored the same day with a corrected configuration and an engine patch. Other models were not affected. If you use GLM 5.3 with very long prompts, it is worth establishing whether you were in the period from August 27 to September 9, 2026 and whether you submitted prompts above 681,280 tokens in that period.

More interesting is the broader lesson: in inference, isolation is not a property you set up once, but something that is either correct or incorrect with every configuration change. Every time you raise a context window, adjust a quantization or increase parallelism, the relationship between what you promise and what your pool can handle shifts. If you want a grip on that, you can use AI security for businesses as a framework: not as a certificate, but as a set of recurring checks.

The timing of the report is also a practical detail. Cloudflare published on October 26, 2026, after coordinating with the researcher and RadixArk, and refers to hackerone.com/cloudflare for reporting security issues in Workers AI. If you run inference yourself, it helps to walk through your incident procedure once out loud before you need it; setting up an incident protocol for failures caused by language models describes how to approach that without drifting into legal conclusions. And if you are choosing between running it yourself and buying it in, the overview of inference hosting and serverless APIs lays out the trade-offs.

Checks for your own inference stack

What the report makes clear above all is how narrow the path was along which this went wrong: one model, one configuration in which the pool was smaller than the window, and one code path in the engine that turned a rejection into a placeholder instead of an error. All three were needed at the same time. That is also why the fix consists of several layers — a larger pool, an engine patch, and checks that make a next mismatch visible sooner.

For developers using other Workers AI models, the report gives no reason for action; if you used GLM 5.3 with extremely long prompts in the stated window, review your own usage. For anyone running SGLang in disaggregated mode themselves, the recommendations in the report are the shortest route to a safer setup: measure, bound the context length, upgrade and enable deferred release. The full report is at Cloudflare; the references below indicate which parts come from where.

Checked on 2026-10-29.

This article was compiled with AI assistance and editorially reviewed; sources are listed below.

Sources