KV Cache Offload vs. Object Storage: What Each Actually Does

KV Cache Offload vs. Object Storage: What Each Actually Does

Table of Contents

KV cache offload cut mean time to first token from 60 seconds to 10 seconds in one test, according to a Red Hat benchmark of tiered offloading on H100 GPUs published on 2 November 2026. The result came from one model on one GPU configuration, and it is why storage teams are being pulled into a conversation they were not ready for. Engineering teams now ask for KV cache offload in the inference storage plan. Few storage architects can say which tier that maps to or whether it changes the budget.

KV cache offload moves the attention key-value data that a language model builds during inference out of scarce GPU memory. It places that data in larger, slower tiers such as CPU memory or local NVMe flash. The model can then serve longer contexts and more users without recomputing the data. The Modular inference handbook describes it as moving attention data “from GPU memory to lower-cost storage like CPU memory or disk.”

The two terms in the title sound like competing answers to the same storage question. They are not. One is disposable working memory that an inference engine manages. The other is the durable system of record for models, datasets and embeddings. This article explains what each does, where each belongs, and what an enterprise should specify for both.

What KV Cache Offload Does Inside an Inference Engine

Every time a language model reads a prompt, it computes attention keys and values for each token and stores them so it does not repeat the work for the next token. That store is the key-value (KV) cache. The KVSwap paper by Huawei Zhang, Chunwei Xia and Zheng Wang notes that the cache grows linearly with sequence length and batch size, and often exceeds the size of the model itself.

The cache lives in GPU high-bandwidth memory (HBM), which it shares with the model weights. Longer prompts, longer conversations and more concurrent users all enlarge the cache. GPU memory then becomes the limit on how many requests a server can hold at once. Offload relieves that limit. It moves idle cache blocks to a lower tier and loads them back when a session or prefix returns.

What Is KV Cache Offload?

KV cache offload is an inference-engine feature that spills attention key-value data from GPU memory to CPU memory, local flash or a shared tier, then reloads it when needed. The SemiAnalysis glossary says it parks attention state the chips cannot hold in host memory, so a long session can be resumed instead of recomputed.

The point of the feature is reuse. If a request shares a prefix with an earlier one, such as a long system prompt, a retrieved document or the earlier turns of a conversation, the engine loads the stored blocks instead of rebuilding them.

One property matters more to a storage buyer than any benchmark. The KV cache is derived data. If it is lost, the model recomputes it, and the cost is time, not information. This is the fact that separates the offload tier from every other tier in the stack.

Four-card diagram of inference tiers: GPU memory holds live KV cache, CPU memory and local NVMe hold rebuildable offloaded cache, shared object storage holds model weights, checkpoints, datasets and embeddings
The four tiers of an inference stack and the kind of data each one holds.

Where KV Cache Offload Sits in the Storage Hierarchy

A practical way to read KV cache offload is as a ladder. The top rung is GPU memory, which is fastest and smallest. Below it sit CPU memory, local NVMe flash, and finally a shared or remote tier that several servers can reach. The llm-d documentation on its KV offloader describes the same ladder: CPU memory for low latency on one node, local disk, and shared or remote storage for cross-node sharing and persistence. It also notes that writes to lower tiers happen in the background without blocking inference.

Each step down trades speed for capacity. CPU memory comes first because, in the Red Hat test, CPU DRAM gave the first and most efficient extension beyond GPU HBM. Models whose KV working set fit in the CPU allocation captured about 90 percent of the available gain. A second tier, such as local NVMe or a distributed file system, nearly doubled the improvement in that test, from 3.3 times to 5.8 times throughput over the baseline.

Several independent sources report the benefit, always tied to cache reuse and a specific setup:

  • Red Hat measured up to 5.8 times higher throughput on Nemotron-Ultra-253B with roughly 50,000 input tokens per request on H100 GPUs. This is a best case for that model and workload.
  • A Google Cloud engineering post on multi-node offloading reported a 95 percent cache hit rate and roughly 40 percent better time to first token (TTFT) in its test configuration, using Llama-3.3-70B and a 50,000-token prompt.
  • The llm-d project states that loading cached blocks is typically faster than recomputing them, and up to 16 times faster for long prompts.
  • An AWS engineering post on offloading with vLLM reports a 2 to 22 times TTFT reduction and up to 9 times higher throughput with high cache hit rates under concurrent load.

The caveat is that gains depend on hit rate, and the same post attributes cache hit ratios of approximately 50 percent in production retrieval-augmented generation (RAG) and multi-turn scenarios to LMCache research. A workload with little reuse will see a small gain.

The technique is also moving from research into products. The Ray Serve LLM guide documents offloading built on automatic prefix caching, and the vLLM documentation has a usage page for KV offloading configuration. At CES 2026, NVIDIA introduced an Inference Context Memory Storage Platform to push inference context onto NVMe flash, and Blocks and Files reported that a dozen storage partners back it.

Two cautions apply to disk tiers. Research on disk-aware designs, including the KVSwap work on on-device inference, shows that reading cache back from flash needs care with access patterns. And these tiers are performance infrastructure. They need throughput and low latency. They do not need snapshots, retention or immutability, because there is nothing in them worth recovering.

What Object Storage Does That KV Cache Offload Cannot

Object storage is the durable store for the data an AI program is built from: training and fine-tuning datasets, RAG source documents, model weights, checkpoints and generated output. Applications reach it over the AWS-compatible S3 REST API, one whole object at a time. Buckets act as a policy boundary, so access, retention, immutability and lifecycle rules apply per bucket.

The priorities are the reverse of the cache tier. Raw speed matters less here, and protection matters more. Object Lock, immutable snapshots, replication and encryption decide whether a model repository survives. The threat may be ransomware, an operator error, or a poisoned dataset that nobody notices until the model misbehaves. A deleted or tampered model is not recomputed from anything. It is lost or compromised.

Is Object Storage Fast Enough to Hold KV Cache?

For the live cache tier, it is the wrong tool in its usual form. Reloading cache blocks is a fine-grained, latency-sensitive operation in the path of a user request. Object access works with whole objects over an API. That mismatch is a design caution, not a measurement, and it is why engines target memory and local flash first.

The picture is not absolute. The engineering roadmap for KV cache offloading at scale describes three stages: local offloading, peer-to-peer sharing between servers, and remote persistent storage. In the peer-to-peer stage, cached context is lost when the cluster restarts. The third stage closes that gap by persisting to durable shared storage. A shared tier can hold cache, but that is a deliberate design decision made in the serving stack. It does not turn object storage into a replacement for the primary data.

Governance bodies treat the durable category as protected data. The NIST control overlay work for AI systems names training and test data, model weights and configuration settings as system components that need specific controls. The KV cache does not appear on that list.

Two-card comparison: KV cache offload holds derived, short-lived data where loss costs recompute time; object storage holds long-lived weights, datasets and checkpoints where loss means lost work
KV cache offload serves speed with rebuildable data, while object storage serves durability for data that must survive.

What Enterprises Need Beyond KV Cache Offload

The practical outcome is a two-tier requirement set. The performance tier serves the inference engine. The durable tier serves everyone else, including security, backup and audit.

What Storage Does a Private LLM Deployment Need?

A private LLM deployment needs a fast, rebuildable tier for KV cache and a protected, durable tier for models, datasets, embeddings and checkpoints. The cache tier is sized for throughput and latency, and it is not backed up. The durable tier is sized for capacity and recovery. It adds immutability, retention, access control and encryption.

Each tier has its own requirements:

  • Cache tier: low latency and high throughput on CPU memory and local NVMe, sized against the working set of active sessions. No backup, snapshots or retention are needed.
  • Durable tier: capacity for datasets and model repositories, immutable copies, tested recovery, role-based access and an audit trail. Speed matters, but protection decides the design.
  • Boundary between them: a clear rule that nothing irreplaceable lives only in the cache tier, and that the serving stack can rebuild any cache after a restart.

The durable tier also deserves the security practices that government agencies set out for AI data. The joint CISA, NSA and FBI guidance on securing AI data, published in May 2025, recommends encryption at rest, in transit and during computation, digital signatures, provenance tracking and trusted infrastructure. It is foundational guidance, not new research. Its recommendations map directly onto how model and dataset storage should be run.

Shared GPU clouds add a governance question that engineering pages rarely address. Models, embeddings and prompts may persist on a provider’s default storage. The enterprise should know where, who holds the keys and how tenants are separated. Data control, data sovereignty and egress charges are why many enterprises keep model and dataset storage on premises. On-premises storage reduces exposure to those issues. It does not remove risk, and it does not by itself deliver compliance.

Where StoneFly Fits Beside KV Cache Offload

StoneFly does not provide KV cache offload. That capability belongs to the inference engine and is configured with the serving stack. StoneFly addresses the two things around it: the on-premises server platform that runs inference, and the durable tier that holds the data.

StoneFly AI Servers powered by NVIDIA GPUs are configurable GPU systems built with NVIDIA L40S, A100 and H100 families for inference, private large language models and RAG. They support integrated NVMe and SAS storage inside the server. They also support disaggregated shared multi-tier storage that multiple servers reach, so compute and storage scale independently. The shared multi-tier design is described as providing near-zero latency between storage and AI processing, and it keeps datasets, models and checkpoints close to the GPUs.

For the durable tier, StoneFly AWS-compatible S3 object storage runs on StoneFly appliances in the enterprise’s own data center, with applications using the AWS-compatible S3 REST API. It is not attached to an Amazon S3 bucket and does not require cloud storage. Immutable Object Lock, the patented Air-Gapped Vault®, encryption at rest and in transit, role-based access control and multi-factor authentication protect AI and machine-learning datasets, model weights and checkpoints. Local deployment gives direct control, data sovereignty and predictable local performance. It also avoids public-cloud egress charges for local object access.

Organizations with several departments or projects sharing one AI platform can use StoneFusion MSP Edition to give each tenant a dedicated child storage environment, with its own buckets, quotas, access rules and protection policies. That is storage-level separation, and it works alongside the isolation controls in the compute and network layers.

A Buyer’s Checklist for KV Cache Offload and Inference Storage

Use these questions with the platform team, the storage vendor and any GPU cloud provider:

  1. Which data is cache that can be rebuilt, and which data is durable? Insist on a written classification for models, checkpoints, embeddings, datasets and prompts.
  2. Where do weights, checkpoints, embeddings and datasets persist, and who controls the encryption keys?
  3. Is the durable tier immutable and recoverable, and has recovery been tested against a deleted or tampered model?
  4. How are tenants separated, and what evidence can the provider show?
  5. What happens to cached or persisted prompt data when a session ends?
  6. What are the egress and data-location terms for datasets and model repositories?

Conclusion: Keeping KV Cache Offload and Object Storage Distinct

KV cache offload is working memory that an inference engine spills to cheaper tiers so it can serve longer contexts and more users. Object storage is the system of record for the models and data the inference depends on. Offload tiers the first. It does not replace, protect or substitute for the second.

For an enterprise, that means two sets of requirements. Size the cache tier for speed and accept that it can be rebuilt. Size the durable tier for capacity, immutability and recovery. Losing it means lost work, not a slower response. As KV cache offload matures and shared tiers become common, the line between the two will stay in the same place: if the data cannot be recomputed, it belongs in protected storage.

To review your inference data classes and storage tiers, talk to a StoneFly storage specialist, or read about StoneFly AWS-compatible S3 object storage and StoneFly AI Servers.

Related Products

StoneFly DR365V Veeam Ready Backup & DR Appliance

Unified Storage and Server (USS™) Hyperconverged Infrastructure (HCI)

Unified Scale-Out (USO™) SAN, NAS, and S3 Object Storage Appliance

Subscribe To Our Newsletter

Join our mailing list to receive the latest news, updates, and promotions from StoneFly.

Please Confirm your subscription from the email