How to Back Up AI Training Data and Checkpoints Affordably

How to Back Up AI Training Data and Checkpoints Affordably

Table of Contents

Most AI storage budgets pay for GPUs and the fast path that feeds them. Very little pays to protect what that path produces. According to the 2026 IBM Cost of a Data Breach analysis, roughly one in five organizations reported an AI-related breach, and of those, 92% lacked proper AI access controls. The assets at risk are the datasets, weights and checkpoints that took weeks of GPU time to build.

An AI checkpoint is a saved snapshot of a model’s weights and training state that lets a run resume; it is not a backup unless an independent, protected copy exists elsewhere. A checkpoint on the same storage as the training run protects against a crashed job. It does not protect against a failed array, a mistaken deletion, a poisoned dataset or an attacker with admin access.

A practical method follows. It ranks each asset by what it costs to recreate and how often it changes. It then assigns a copy type and a retention rule. The independent copy goes on a lower-cost immutable tier instead of flash.

What Is Expensive to Recreate in an AI Pipeline?

Five kinds of assets are worth protecting: training datasets, model weights and optimizer state, checkpoints and experiment artifacts, embeddings and vector stores, and the pipeline code and configuration that tie them together. Each one costs GPU time, engineer time or data that cannot be collected twice.

Start with training datasets. A curated, labelled and cleaned dataset takes many engineer hours to build. Some of its source data may no longer exist. Losing it means repeating that work, if repeating it is even possible.

Model weights and optimizer state come next. Weights are the result of the training run. Optimizer state lets a run continue rather than restart. Checkpoints capture both at intervals, along with experiment artifacts such as evaluation results and configuration. Embeddings and vector stores are the indexes that retrieval applications depend on. They can be rebuilt, but a rebuild uses compute and leaves the application slow or offline.

Integrity matters as much as availability. The joint government guidance on securing data used to train and operate AI systems, issued by CISA, the NSA, the FBI and international partners, stresses that data security protects the accuracy, integrity and trustworthiness of AI outcomes. NIST describes the attacker side in its taxonomy of adversarial machine learning attacks, which includes data poisoning. A backup taken after poisoning keeps the poison. Retention must therefore include clean, earlier versions.

Confidentiality adds a second cost. IBM reports that model inversion averaged USD 6.07 million per incident, and describes it as a technique in which adversaries infer or reconstruct sensitive training data by analysing model outputs. Training data is therefore a target even when it never leaves the building.

Finally, checkpoint writing is itself a performance problem. A December 2025 research paper on large language model checkpoint and restore I/O reports up to 3.9 times higher write throughput than one prior system and 7.6 times higher than another in the authors’ tests. AI checkpoints are tuned for speed on fast storage, which is why they cannot double as the protected copy.

What to Protect, and How Often?

Protect every asset whose recreation cost exceeds the cost of keeping a second copy, and set the copy interval by how much work you can afford to lose. Checkpoint frequency and backup frequency are separate decisions made for separate reasons.

Checkpointing frequency is a training resilience choice. It balances the cost of writing a checkpoint, which pauses or slows GPUs, against the cost of redoing work after a failure. Backup frequency answers a different question: how much of the finished work could vanish if the primary system and its checkpoints were lost together. A team can checkpoint every few minutes and still have no independent copy.

A useful rule is to tie the backup interval to the cost of the lost work. If a run consumes expensive GPU hours every day, losing a day of checkpoints is a budget event, and the independent copy should be refreshed accordingly. If a dataset changes quarterly, a snapshot after each change is enough. Universal numbers do not exist, because they depend on model size, cluster cost and risk appetite.

Retention is where affordability starts. Keep the latest checkpoint for fast resume, keep milestone checkpoints that mark validated model versions, and prune intermediates once a later milestone exists. Pruning loses no safety as long as the milestones are protected.

The matrix below turns the method into a working inventory.

Asset Cost to recreate Change rate Copy type Retention approach
Curated training datasets High: engineer time, possibly unrepeatable Low to moderate Immutable, air-gapped copy Keep every approved version
Model weights and optimizer state Very high: GPU time Per run Immutable copy of milestones Keep validated releases
Checkpoints High while a run is live, lower afterwards Very high Latest local, milestones copied off-system Prune intermediates after a milestone
Embeddings and vector stores Moderate: rebuild compute and downtime Moderate to high Snapshots plus replication Keep recent generations
Pipeline code and configuration Low to moderate High Version control plus backup Keep full history
Three tier cards: NVMe active tier for datasets in use and current checkpoint, capacity tier for older checkpoints and finished datasets, and an immutable air-gapped copy for approved datasets, validated releases and milestone checkpoints.
Each AI asset has a working tier and an independent protected copy on an immutable air-gapped tier.

The NIST practice guide on identifying and protecting assets against ransomware and other destructive events begins with identifying assets across devices, data and applications. Applying that discipline to AI workloads means the ML team and the backup team agree on one list.

Why Most AI Storage Plans Leave a Gap

Most plans leave a gap because different teams design the training path and the protection path on different budgets. The result is checkpoints that share a failure domain with the data they protect. The backup system has never been told those assets exist.

Is a checkpoint a backup? No. A checkpoint is a resume point stored where the training job can reach it quickly. A backup is an independent, retained and protected copy that survives the loss of the primary system. If one deletion, one storage failure or one compromised credential can remove both, the checkpoint was never a backup.

Two cards comparing an AI checkpoint, which protects against a crashed job, with a backup, which survives loss of the primary system.
A checkpoint resumes a training run, while a backup is an independent protected copy that survives loss of the primary system.

Four patterns recur in the plans that fail.

  • Protection is a closing item. The storage design is sized for throughput, and backup is added in the last slide, if at all.
  • Backup teams do not see AI assets. Datasets and checkpoints sit on GPU-adjacent storage that the backup catalogue never scans.
  • Backup repositories stay online. The CISA and MS-ISAC guide to stopping ransomware warns that many ransomware variants find and subsequently delete or encrypt accessible backups.
  • Restores are never tested. Nobody knows how long it takes to bring back a model and resume a run.

The case for investing in the independent copy is stronger than it was a year ago. The Sophos State of Ransomware 2026 survey, reported by Help Net Security and covering 2,158 IT and security leaders whose organizations were hit in the past year, found that two-thirds of victims with encrypted data restored files from backup, up from just over half the year before. The report summarizes the shift in one line: “Backups carried more of the recovery this year.” When backups carry the recovery, attackers have every reason to go after them first.

Keeping an Immutable Copy Without Paying Flash Prices

You keep an immutable copy affordably by matching each asset to the cheapest tier that meets its recovery need. You also reduce what has to be stored. Flash belongs where GPUs read and write. The protected copy does not need to live there.

Tiering is the first lever. NVMe holds active datasets and the current checkpoint. Capacity media holds older checkpoints, finished datasets and the independent copy. Automated tiering moves data as it cools, so the expensive tier stays small.

Data reduction is the second lever. Deduplication removes repeated blocks across checkpoints that differ only slightly, and thin provisioning avoids reserving capacity that is not yet used. Retention policy is the third, because every checkpoint that is pruned on schedule is capacity that is never bought.

Location is the fourth. An independent analysis of egress costs for AI and machine learning data notes that there is no special AI rate: training pulls, model transfer and inference streaming bill at the same per-gigabyte egress as every other workload. A protected copy held in a public cloud therefore charges every time it is read back, including during a restore. Keeping the copy on local object storage avoids public-cloud egress charges for local access.

Immutability and air-gapping solve different problems, and the best designs use both. A guide to S3 Object Lock for compliance and security covers the retention modes in more detail. Isolation reduces network exposure. Immutability stops protected data from being changed or deleted during its retention period. CISA advises teams to maintain offline, encrypted backups of critical data and to test their availability and integrity. The same guidance adds a caution. Immutable storage may not meet compliance criteria for certain regulations, and a misconfiguration can be costly. Review retention settings before you apply them. The NIST publication on security guidelines for storage infrastructure is a useful reference for the surrounding design.

Four-stage flow: write on NVMe, tier to capacity, lock the copy with Object Lock and an air gap, then test the restore.
Data moves from NVMe to capacity, is locked in an immutable air-gapped copy, and is proven restorable by a test.

Restore testing turns the design into evidence. A short routine is enough.

  • Restore a milestone checkpoint to a clean system, not to the original array.
  • Resume a training run from it and confirm the loss curve continues as expected.
  • Record the elapsed time and treat it as your measured recovery time objective (RTO).
  • Repeat after any change to the storage, the retention rules or the checkpoint format.

Until that routine has run once, the recovery time in your plan is a guess.

Four-step cycle: restore a milestone checkpoint to a clean system, resume the run, record the measured RTO, repeat after changes.
The restore test cycle: restore a milestone checkpoint, resume the run, record the elapsed time, and repeat after changes.

How StoneFly Protects AI Data and Model Checkpoints

StoneFly builds the training platform and the protected copy so that both come from one design. It addresses the problem this article describes. AI datasets, model weights, checkpoints, embeddings, source data and generated results are hard or costly to reproduce, yet most plans protect none of them independently.

StoneFly AI Servers powered by NVIDIA GPUs are GPU-accelerated systems for generative AI, large language models, training, fine-tuning and inference. They are offered with NVIDIA L40S, A100 and H100 GPUs. Storage is either integrated or disaggregated and shared, and is engineered for near-zero latency between storage and AI processing. The multi-tier design uses NVMe for active datasets and SAS for capacity data, with local model and checkpoint storage and scratch space. Networking of up to 100 Gb in applicable configurations carries datasets, checkpoints, replication and backup traffic. SourceConnect integrates data from more than 300 sources and includes replication and secure backups.

The independent copy lives on StoneFly Object Storage, an AWS-compatible S3 object storage platform that stores data on StoneFly appliances rather than in an Amazon bucket. Immutable Object Lock protects selected objects from modification, overwrite or deletion during the retention period. The patented Air-Gapped Vault® adds air-gapped and immutable object data, so isolation and immutability work together. Deduplication, thin provisioning and automated tiering between performance, capacity and optional cloud tiers address the cost levers described above. Local deployment avoids public-cloud egress charges for local object access. The platform also provides immutable snapshots, replication and encryption at rest and in transit. Integrated threat detection and response flags abnormal object writes and mass deletion attempts. DCIG named it a Top 5 Midmarket SDS Object-Based Backup Target for 2025 to 2026.

Teams that already run Veeam can use StoneFly DR365V, a Veeam Ready air-gapped and immutable backup and disaster recovery appliance that includes the patented Air-Gapped Vault®. It adds automated recovery testing and on-demand isolated sandbox testing, which support the restore routine above. Shared file datasets and model repositories can also sit on StoneFly SSO NAS, which offers file-level WORM and the Air-Gapped Vault® where configured. This setup guide to immutable file-level WORM explains how to configure it.

The inventory, the matrix and the restore test come first; the products are one way to implement the tiers.

Treating AI Checkpoints as Assets Instead of Scratch Files

The gap in most AI storage plans is not a technology gap. It is an ownership gap. Training storage is funded and tuned, while the datasets and checkpoints it produces are handled as scratch files until the day one goes missing.

The method in this article closes that gap in three steps.

  • Inventory the assets. List datasets, weights, checkpoints, embeddings and pipeline code, with the cost to recreate and the change rate of each.
  • Tier the copies. Keep flash for active work, place the independent copy on a lower-cost tier, and make that copy immutable and air-gapped.
  • Test the restore. Resume a run from a protected checkpoint, record the time and repeat it on a schedule.

Retention rules, deduplication, tiering and local access each reduce what must be stored and what must be paid when data is read back. The evidence supports the effort: two-thirds of ransomware victims in the Sophos survey restored from backup, and the joint government guidance on AI data security treats integrity as central to trustworthy AI.

If your team is planning or expanding AI workloads, the right moment to add the protection line is before the first long training run, not after the first loss. You can review the plan with a StoneFly storage architect to map your own assets to tiers and build a tested restore into the schedule.

Related Products

StoneFly DR365V Veeam Ready Backup & DR Appliance

Unified Storage and Server (USS™) Hyperconverged Infrastructure (HCI)

Unified Scale-Out (USO™) SAN, NAS, and S3 Object Storage Appliance

Subscribe To Our Newsletter

Join our mailing list to receive the latest news, updates, and promotions from StoneFly.

Please Confirm your subscription from the email