> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/production-operations/operations.md).

# Operations overview

This section describes how to run Kangal as a production service. It covers deployment, secret delivery, backup and recovery, regional failover, and incident triage. Use it together with your platform runbooks for Kubernetes, MongoDB, Redis, DNS, certificates, and global traffic management.

## Runtime ownership

Kangal has four operational component types:

| Component         | Production role                                                         | Persistent dependency                                                   |
| ----------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| API               | Control-plane API, administration, configuration, and readiness         | MongoDB and Redis                                                       |
| Frontend          | Operator console                                                        | API                                                                     |
| Background worker | Durable integration and asynchronous jobs                               | Redis and MongoDB                                                       |
| Data plane        | Request matching, policy execution, upstream proxying, and config cache | Control plane while online; signed last-known-good config while offline |

MongoDB is the source of truth for control-plane configuration. Redis provides shared runtime state and, when durable jobs are enabled, the work queue and job indexes. Treat both as production dependencies even though some counters can be rebuilt.

## What Kangal automates

The production chart can render API, frontend, worker, and optional data-plane workloads with probes, resource controls, non-root security contexts, HPA, pod disruption budgets, network policies, ingress, and certificate resources. Kangal also provides:

* `/health` liveness and `/ready` dependency readiness endpoints;
* checksum-bearing MongoDB backup archives through the backup helper and optional scheduled Kubernetes job;
* tenant-scoped configuration validation, diff, snapshot, sync, and rollback;
* durable Redis Streams jobs with retry, lease takeover, dead-lettering, and operator-visible health;
* regional health, replication evidence, promotion fencing, and audit records;
* immutable release artifacts, security scans, signatures, and release evidence through the supported release workflow.

## Operator responsibilities

Production operation requires operators to provide and maintain:

* production MongoDB and Redis topology, authentication, encryption, persistence, replication, capacity, and provider-level backups;
* Kubernetes cluster upgrades, ingress controllers, storage classes, DNS, global load balancers, firewall policy, and external secret delivery;
* backup copying to immutable off-cluster storage, retention enforcement, restore approval, and recovery drills;
* regional database replication, old-primary writer fencing, or traffic switching;
* alert routing, incident command, change approval, and capacity planning.

Assign a named owner and a tested runbook to every responsibility before marking a deployment production-ready.

## Minimum production baseline

Before accepting traffic:

1. Deploy immutable image tags from one signed release.
2. Use an external MongoDB replica set or sharded cluster with multi-document transaction support.
3. Use authenticated, persistent Redis. Run a shared failover-capable Redis endpoint for durable jobs in multi-region installations.
4. Inject runtime secrets from a secret manager; do not store values in Helm files.
5. Run at least three API replicas and two background workers, then size from measured load and queue behavior.
6. Enable TLS, explicit CORS origins, security headers, network policy, HPA, and pod disruption budgets after validating the cluster policy model.
7. Send application and audit logs to retained storage and scrape `/metrics`.
8. Define RPO/RTO, verify a backup, and complete a staging restore.
9. Run a release smoke test through the same ingress path used by clients.

## Operational signals

Monitor at least:

* `/health`, `/ready`, pod restarts, unavailable replicas, and rollout duration;
* request rate, error rate, p95/p99 latency, upstream errors, and in-flight requests;
* MongoDB primary availability, transaction capability, replication lag, storage, and backup age;
* Redis availability, latency, persistence, memory, evictions, stream pending count, queue depth, retries, and dead letters;
* data-plane heartbeat age, desired/applied config version, checksum, NACK count, and last-known-good age;
* certificate expiry, ingress TLS errors, and DNS health;
* regional evidence age, replication lag, pending promotion phase, and drill age.

## Change discipline

For every production change, retain the release tag, commit SHA, chart version, rendered-values digest, image digests, Helm revision, pre-change configuration snapshot, backup reference when required, approver, and verification evidence. Workload rollback and data rollback are separate decisions; never assume a Helm rollback reverses data or configuration changes.

Continue with [Deployment](/production-operations/deployment.md), [Secrets](/production-operations/secrets.md), [Backup and restore](/production-operations/backup-restore.md), [Multi-region](/production-operations/multiregion.md), and [Troubleshooting](/production-operations/troubleshooting.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.kangal.dev/production-operations/operations.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.