Backing up a self-hosted AI agent before it deletes your work

Most teams treating AI agent backups as an afterthought discover the gap at the worst possible moment: when the agent, operating with valid write credentials, has already reached the recovery data. A self-hosted agent is not a standard web application. It accumulates state across four distinct layers, and it routinely holds runtime permissions that span all of them. Getting AI agent backups right requires a specific strategy per layer, isolation of the backup destination from the agent's write path, and restore drills run before an incident forces the issue.

The four layers a self-hosted agent backup must cover

Most teams back up the application database and stop there. For a self-hosted AI agent, that leaves three more layers unprotected.

Application database. Postgres, SQLite, or equivalent. Stores task state, run history, and configuration references. Standard pg_dump or WAL archiving applies without modification.

Vector store indexes. Qdrant, Chroma, or self-hosted Weaviate. These hold embedded representations of your retrieval corpus. Change frequency is high (every ingestion run) and size commonly runs 2-20 GB, growing with corpus size. Rebuilding from scratch requires re-embedding all source documents, which can take hours even with GPU acceleration. Do not treat vector indexes as disposable caches.

Conversation and episodic memory. Frameworks such as LangGraph, Mem0, and MemGPT persist agent memory separately from the main database. Lose this layer and the agent reverts to stateless mode with no session continuity. Change frequency is per interaction; size is usually under 1 GB but the content is structurally irreplaceable.

Agent code and prompt templates. The smallest layer by bytes, but the most critical for deterministic replay. Replaying a historical run against a restored database with a different prompt template version produces different outputs. Include a tagged snapshot of prompt templates in every backup set, not just in version control.

A sound agent checkpoint strategy treats all four layers as co-dependent. Restore the database without the matching vector index version and queries return results, but similarity scores are meaningless because the query embeddings live in a different vector space than the stored representations.

AI agent backups and the blast-radius problem

The April 2026 eon.io postmortem documented an agent that deleted production data and its own backup store in under 10 seconds, using valid API credentials. This is not a freak incident. It is the default configuration for most self-hosted setups: the agent needs write access to operate, and the backup destination is typically provisioned under the same identity.

Why agent credentials must never reach the backup destination

The agent runtime identity must have zero permissions on the backup storage target. Not read-only: zero. An agent with list or read access to its own backups can, under a sufficiently broad instruction set, corrupt or delete those backups.

The minimum isolation model is two IAM roles: one for the agent process, one for the backup job. The backup destination has a resource policy that explicitly denies writes and deletes from the agent's role. The backup job role is never mounted into the agent container. This is two roles and one deny policy, with no architectural complexity.

Immutable storage options and costs

Immutable object storage enforces isolation at the storage layer rather than relying on IAM correctness alone.

S3 Object Lock in Compliance mode prevents any identity from deleting objects before the retention period expires. As of Q1 2026, S3 Glacier Instant Retrieval costs approximately $0.023 per GB per month. For a 20 GB snapshot set with 90-day retention, storage cost is roughly $1.40 per month.

Backblaze B2 with Object Lock costs approximately $0.006 per GB per month as of Q1 2026, roughly four times cheaper. B2 is S3-compatible, so Restic, Kopia, and rclone work against it without modification. Agent data protection at this price point is a configuration question, not a budget question.

Snapshot schedules matched to agent workload type

Three workload types with concrete schedules:

Batch agents (run on a schedule, process a queue, stop): daily full snapshot plus hourly incremental. Target RPO: one hour. The primary risk is losing in-progress work on a batch that runs for several hours.

Continuous agents (always-on, memory accumulates across sessions): per-session checkpoint after each completed conversation cycle, plus a nightly full snapshot. Target RPO: end of last completed session. This is the workload type where losing conversation memory is most damaging, because users interact with the agent expecting continuity across sessions.

Event-driven agents (triggered by external events, chaining multiple tool calls): snapshot after each completed tool-call cycle, plus nightly full. Target RPO: last completed tool-call cycle.

A concrete RPO target for production autonomous agents is under 15 minutes, achievable with the schedules above provided the snapshot write path does not block the agent's main execution thread.

One important detail for embedded databases: full nightly exports are safer than WAL streaming for SQLite-backed memory stores. WAL streaming requires continuous network connectivity and a compatible receiver. A nightly SQLite export to immutable storage is simpler, more portable, and restores in seconds.

What a real agent restore looks like

Recovery is the section most backup guides skip entirely. Restoring components out of order produces failures that look like data corruption but are sequencing errors.

Correct restore order for a LangGraph plus Postgres plus Qdrant stack:

1. Config and API key references. Restore environment variables and secrets manager references first. Nothing else starts without valid credentials pointing at active endpoints.
2. Postgres database. Restore from the snapshot matching the target timestamp. Apply WAL segments if using point-in-time recovery.
3. Qdrant vector index. Restore the snapshot tarball, restart Qdrant, verify collection availability. Expected time: 10-40 minutes depending on corpus size, as Qdrant rebuilds internal segment indexes on startup after snapshot restore.
4. Conversation memory store. If stored separately from Postgres, restore next before starting the agent.
5. Agent process. Start only after all backing services report healthy.

Two failure modes that only surface at restore time: embedding model version mismatch (the index was built with a different model than the config now references, producing silent retrieval degradation where queries return results with meaningless scores) and stale tool endpoint configs (the restored config references a webhook URL or API endpoint that was decommissioned after the snapshot was taken). Mitigation for both is asserting model identifier and endpoint health as mandatory steps in the restore verification sequence.

Automated restore drills: how to know your backup works before you need it

A backup that has never been restored is a hypothesis. Self-hosted agent recovery in a real incident is not the time to discover that a vector index snapshot was corrupt or that the restore script had a broken dependency.

Monthly automated restore drills in an isolated environment should restore all four layers from the most recent production snapshot, start the agent process, run a fixed deterministic prompt set and assert that outputs fall within measurable tolerance of the expected results, verify that all tool integrations return semantically valid responses, and log the result along with restore duration per component. The whole thing should feel like running a comprehensive test suite, except you're testing against real production data instead of fixtures.

Restic and Kopia both support scripted restore verification as of their 2025 releases. Kopia's snapshot verify command checks integrity without a full restore; pairing that with a periodic live restore drill covers both corruption detection and sequencing validation. The "restore-first" validation pattern, where teams run a live restore before the first production use of a new backup configuration, has gained traction in 2026 among teams that have been burned by untested agent rollbacks.

Retention windows and what 2026 compliance frameworks expect

The EU AI Act's operational requirements for high-risk autonomous systems began enforcement in August 2025, with broader provisions phasing in through 2026. Draft guidance from the European AI Office points toward minimum retention of 90 days for audit trails and 7 years for decision records in regulated financial or medical contexts. The European Commission's AI Act resource page is the primary reference for current technical annexes.

For a self-hosted agent deployment: tool-call records and decision logs should be retained for a minimum of 90 days for any deployment touching EU user data, regardless of risk classification. For agents operating in regulated domains (insurance, healthcare, financial advice), decision records affecting individual users require 7-year retention. Conversation memory follows the 90-day floor, subject to GDPR right-to-erasure obligations that can override the AI Act's retention minimum.

As of 2026, most indie-hosted agent stacks have no documented retention policy at all. That is a compliance gap and an audit gap: if a user disputes an agent decision, there is nothing to review. A tiered automated workflow protection structure (hot storage for 30 days, cold immutable storage for the retention remainder) with automated lifecycle transitions addresses both the backup requirement and the retention requirement in one configuration pass.

Key takeaways

AI agent backups require coverage across four co-dependent layers: application database, vector index, conversation memory, and agent code with prompt templates. The single most dangerous default is an agent whose runtime credentials can reach its own backup destination. Immutable object storage with a dedicated backup IAM role eliminates that risk at under two dollars per month for most indie deployments. Run monthly restore drills and log the results. Retention policies are a compliance requirement in 2026 for any EU-facing deployment, not an optional practice.

---

Self-hosted agents operate across four layers, and most teams back up only one. The production CLAUDE.md template in the kit covers the isolation rules that keep your agent's write credentials away from its own backups, so a compromised agent can't reach the recovery data.

→ Get the welcome kit