> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/production-operations/backup-restore.md). # Backup and restore Backups protect control-plane state; configuration snapshots protect a narrower configuration workflow. They are complementary and are not interchangeable. ## Data ownership * **MongoDB:** source of truth for tenants, gateways, routes, policies, identities, audit records, configuration versions, regional state, and other control-plane data. * **Redis:** shared counters, sessions, caches, runtime coordination, and the durable job stream/index when Redis Streams jobs are enabled. * **Data-plane storage:** signed last-known-good config and an anti-rollback trust anchor. These support offline continuity but do not replace control-plane backup. * **Release and environment artifacts:** signed chart, image digests, values digest, external Secret versions, and infrastructure definitions needed to recreate service. If durable jobs are enabled, do not classify all Redis data as disposable. Losing an accepted stream entry can lose work; replaying uncertain work can repeat an external side effect. Define Redis persistence, backup, and idempotency around the workload RPO. ## Backup automation The repository backup helper and optional chart CronJob run `mongodump`, create a gzip archive, and write an adjacent JSON manifest with SHA-256 and size. The helper redacts URI credentials from its manifest and dry-run output. The generated archive covers MongoDB. Operators must transfer the archive and manifest to encrypted immutable storage, enforce replication and retention, test restores, and protect the manifest as sensitive metadata. Back up Redis, Kubernetes Secret versions, DNS, certificates, and infrastructure through their owning platform services. Restore target selection and authorization always remain controlled operator actions. For strict point-in-time recovery, pair application archives with provider-supported MongoDB snapshots or PITR and verify their transaction-consistent recovery behavior. ## Define objectives Record per environment: | Item | Required decision | | ------------- | ---------------------------------------------------------------------------------------------- | | RPO | Maximum acknowledged data and job interval that may be lost | | RTO | Time from declaration to restored, verified service | | Retention | Daily, weekly, monthly copies and legal hold | | Copies | At least one encrypted, immutable, off-cluster copy | | Authority | Who can start backup, decrypt, restore, and delete | | Drill cadence | Daily verification, monthly staging restore, quarterly regional drill are a practical baseline | ## Create and verify an archive Run from a restricted host with MongoDB Database Tools installed. Supply the connection through the process environment or a projected file, not a command argument recorded in shell history: ```bash test -n "${MONGODB_URL:-}" export MONGODB_DATABASE=kangaldb python scripts/backup_restore.py \ --database "$MONGODB_DATABASE" \ backup \ --output-dir /secure/backups/kangal ``` The command prints the archive and manifest paths. Verify before transfer: ```bash python scripts/backup_restore.py verify \ /secure/backups/kangal/.archive.gz ``` Copy the archive and manifest as one unit to encrypted immutable storage. Independently record the object version, retention lock, encryption key reference, checksum, database cluster identity, release tag, completion timestamp, and backup job identity. The manifest contains connection metadata and should be access-controlled. ## Scheduled Kubernetes backups The chart can render one CronJob that writes to an existing PVC: ```yaml backup: enabled: true schedule: "0 2 * * *" concurrencyPolicy: Forbid database: kangaldb existingSecret: kangal-runtime mongoUriKey: MONGODB_URL persistence: existingClaim: kangal-backups networkPolicy: enabled: true ``` Provide the PVC before enabling the CronJob and arrange off-cluster transfer for each completed archive. Run only one authoritative schedule against a cross-region database to avoid duplicate, ambiguous archives. Trigger and inspect a one-off run without exposing the connection string: ```bash CRONJOB=$(kubectl -n "$NAMESPACE" get cronjob \ -l app.kubernetes.io/instance="$RELEASE",app.kubernetes.io/component=backup \ -o jsonpath='{.items[0].metadata.name}') JOB="kangal-backup-$(date +%s)" kubectl -n "$NAMESPACE" create job --from="cronjob/$CRONJOB" "$JOB" kubectl -n "$NAMESPACE" wait "job/$JOB" --for=condition=Complete --timeout=1h kubectl -n "$NAMESPACE" logs "job/$JOB" ``` A completed Job proves command completion, not restorability. Mount the archive read-only in a restricted verification workload, recalculate SHA-256, and run a staging restore on schedule. ## Restore drill Always restore to an isolated staging database first: 1. Open a change record and identify archive, manifest, target, RPO/RTO, and approver. 2. Verify object retention metadata and SHA-256 before download and after transfer. 3. Confirm the target connection points to staging and contains no required data. 4. Run a dry-run to inspect the redacted command. 5. Restore with checksum verification and `--drop` only after target approval. ```bash python scripts/backup_restore.py \ --database "$MONGODB_DATABASE" \ restore /secure/backups/kangal/.archive.gz \ --verify \ --drop \ --dry-run python scripts/backup_restore.py \ --database "$MONGODB_DATABASE" \ restore /secure/backups/kangal/.archive.gz \ --verify \ --drop ``` `--drop` deletes matching target collections before restore. Never run it until the target URI, database name, and change authorization have been independently checked. ## Production recovery 1. Declare recovery, stop traffic changes, and preserve timestamps and current state. 2. Fence administrative writes and scale API producers and workers to zero. Keep data planes serving only if their verified last-known-good policy allows it. 3. Preserve the failed database and provider logs; do not restore over a primary that may still recover or participate in an election. 4. Create a new recovery target when possible. Verify topology and transaction support. 5. Verify and restore the selected archive or provider PITR point. 6. Restore Redis according to the declared job/session RPO. Review pending jobs and external idempotency records before replay. 7. Start API replicas, verify `/ready`, then start compatible workers. 8. Compare tenant, gateway, route, policy, consumer, certificate, audit, and config version counts with pre-loss evidence. 9. Verify data-plane config version/checksum convergence, routed traffic, login, background-job health, metrics, and audit delivery. 10. Reopen writes and traffic gradually; record measured RPO/RTO and follow-up actions. Label-based workload fencing can be prepared as follows, but the incident commander must approve execution and the original replica counts must be recorded first: ```bash kubectl -n "$NAMESPACE" get deploy \ -l app.kubernetes.io/instance="$RELEASE" kubectl -n "$NAMESPACE" scale deployment \ -l app.kubernetes.io/instance="$RELEASE",app.kubernetes.io/component=api \ --replicas=0 kubectl -n "$NAMESPACE" scale deployment \ -l app.kubernetes.io/instance="$RELEASE",app.kubernetes.io/component=background-jobs \ --replicas=0 ``` ## Acceptance evidence A successful drill includes archive and manifest checksum, source/target cluster IDs, release and schema versions, start/end timestamps, restored counts, representative tenant reads, one routed request, worker and queue status, data-plane convergence, measured RPO/RTO, and cleanup proof. Store evidence without connection strings, tokens, private keys, or customer payloads. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.kangal.dev/production-operations/backup-restore.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.