Tenant isolation in AI inference: lessons from a cache incident

On 26 October, Cloudflare published a technical account of a tenant-isolation issue in one Workers AI model. It reported that 155 responses to very long prompts contained data from other customers’ earlier requests to the same model. Cloudflare said the affected interval ran from 27 August to 9 September 2026, the affected model was GLM-5.3, and it found no evidence of deliberate exploitation.

The incident is useful to study because it was not a conventional database query returning the wrong tenant’s row. It involved inference memory: a KV cache that stores intermediate model state so generation can run efficiently. The failure happened when an oversized request was rejected by one part of the system but continued through another part as if its cache had been correctly initialized.

How a rejected request left stale state behind

Cloudflare’s deployment configured a context window of 1,048,576 tokens while the prefill KV cache pool could hold 681,280. The expected behavior was an error for a request that did not fit. Instead, the inference engine’s rejection path replaced the oversized prompt with a one-token placeholder. Prefill then generated cache data for that placeholder and signaled that the cache transfer was complete.

The decode node treated that signal as proof that the full request had overwritten its cache pages. Most of the pages still held prior request state. Generation could therefore attend to stale cache content and return another customer’s data. The problem was not simply that two users shared hardware; it was that the system’s ownership and completion assumptions diverged across a failure path.

Cloudflare said all affected responses were for prompts longer than 681,280 tokens. It disabled the model, corrected the pool sizing, patched the inference engine to return an error for rejected requests, and restored the model after validation. It also described broader hardening across its SGLang deployments.

What engineering teams should take from it

The central lesson is that rejection paths deserve the same isolation review as successful requests. An error can occur after memory has been allocated, after work has moved between machines, or after one service has told another that a handoff succeeded. If those components disagree about which request owns a buffer, “failed” does not necessarily mean “no output” or “no data access.”

For a multi-tenant inference system, review these properties explicitly:

• Admission: reject a request before partially initializing shared state, or guarantee that partial state cannot be read.
• Ownership: tie each cache allocation and transfer to a request identity and tenant boundary; verify that identity at the receiver.
• Completion: define exactly what a successful transfer acknowledgement proves. A control message should not stand in for validation of the data it refers to.
• Failure cleanup: exercise oversized prompts, cancellations, timeouts and node restarts. Confirm that pages are released or overwritten before reuse.
• Isolation tests: run two-account tests with unique harmless canary strings. Check successful and failed requests, not just normal short prompts.
• Observability: log admission failures and handoffs with request-level identifiers that do not expose prompt contents. Alert on invariant violations and unexpected cross-request markers.

These controls are not a claim that every inference provider has the same defect. They are review questions for any service that reuses GPU memory, cache pages or intermediate state across requests.

Capacity limits should agree across the whole pipeline

A configured context window is an advertised capability. The memory pool, scheduler, transfer layer and decoder must all support the same admitted request. Validate that relationship at deploy time rather than relying on the request path to discover a mismatch under load.

Also test the exact outcome when a request is too large. It should fail closed: a clear client error, no generation from stale state, and cleanup that has been verified before memory is reused. Where the inference engine spans multiple processes or machines, test both sides of the handoff and inject failures between them.

The published incident does not show evidence that attackers repeatedly exploited the issue, according to Cloudflare. It does show why security reviews need to follow data through resource limits and error handling, especially in fast, shared AI infrastructure. Isolation is a property of the whole request lifecycle, including the path where the request should stop.

Sources: Cloudflare · SGLang