> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/production-operations/multiregion.md).

# Multi-region

Kangal supports tenant-scoped, active-passive disaster-recovery control. It records regions, federation membership, gateway policy, health and replication evidence, promotion generations, fencing confirmation, audit events, and drill results.

The supported operating model combines Kangal's promotion controls with an approved external orchestrator. The orchestrator replicates data, fences the previous database writer, switches global traffic, and submits fresh, verifiable evidence. Configure `automatic_failover` as `false`; promotion requires the two-phase procedure below.

## Supported safety model

A promotion is a two-phase operation:

1. **Begin:** Kangal verifies tenant scope, target membership, current generation, target health, replication status, evidence age, and configured lag threshold. It creates a new generation and fence token but does not change the effective region.
2. **External action:** controllers fence the previous writer, switch health-checked traffic, retain immutable evidence, and sign the exact confirmation envelope.
3. **Confirm:** Kangal verifies the pending generation, fence token, target, evidence, expiry, previous-primary-fenced assertion, controller allowlist, HMAC signatures, and quorum. Only then does the effective region change.

Stale or incomplete prerequisites return `412`. Concurrent, expired, or mismatched operations return `409`. Idempotency keys may be retried only with identical input.

## Infrastructure topology

Use a separate release and namespace in each region. Every control plane must use:

* the same externally managed, cross-region MongoDB replica set or sharded cluster;
* a regional authenticated Redis endpoint for regional runtime state;
* one shared, failover-capable Redis endpoint for durable background jobs;
* the same federation definition and controller allowlist;
* a unique `KANGAL_REGION_ID`, regional API URL, ingress, DNS, and labels;
* immutable images from the same compatible release.

Do not enable bundled MongoDB or Redis for multi-region. Run one authoritative backup schedule against the global database, not one schedule per region.

## Responsibility boundary

| Kangal enforces                                           | External operators/controllers enforce          |
| --------------------------------------------------------- | ----------------------------------------------- |
| Tenant isolation and region membership                    | Database replication and write concern          |
| Evidence schema, digest, age, and lag limits              | Real health probes and replication checkpoints  |
| Promotion generation and idempotency                      | Old-primary writer fencing                      |
| Pending confirmation expiry                               | DNS/load-balancer traffic switch                |
| Controller allowlist, unique keys, signatures, and quorum | Controller key custody and rotation             |
| Effective region state and immutable audit                | Evidence object retention and incident approval |

An API status of `stable` is not proof that global traffic and database writer state remain correct. Continuously reconcile all three views.

## Controller key delivery

Supply `KANGAL_DR_FENCING_CONTROLLERS_JSON` from a runtime Secret to every regional API deployment. Each controller ID and key must be unique, and each key must contain at least 32 bytes of entropy. Do not store the map in values, source control, tickets, or release artifacts. Set a quorum that can still be met during one controller outage but cannot be satisfied by a single shared credential.

## Bootstrap resources

Use an admin token with the intended tenant scope. Both unversioned and versioned admin base paths expose the same resource operations.

1. Create at least two regions with `POST /admin/regions`.
2. Submit health evidence with `POST /admin/regions/{region_id}/health-evidence`.
3. Create a federation with `POST /admin/federations` and list allowed controller IDs.
4. Set an active-passive policy with `PUT /admin/gateways/{gateway_id}/dr-policy`.
5. Submit source-to-target replication evidence with `POST /admin/gateways/{gateway_id}/replication-evidence`.
6. Read readiness and blockers from `GET /admin/gateways/{gateway_id}/dr-status`.

Example policy body:

```json
{
  "tenant_id": "acme",
  "federation_id": "<federation-id>",
  "primary_region_id": "<primary-region-id>",
  "failover_order": ["<secondary-region-id>"],
  "mode": "active_passive",
  "max_replication_lag_seconds": 30,
  "evidence_max_age_seconds": 300,
  "require_healthy_target": true,
  "require_caught_up_replication": true,
  "automatic_failover": false
}
```

Evidence must reference an immutable monitoring event, database checkpoint, signed object, or retained controller log. `digest_sha256` is the digest of that object. `observed_at` and `expires_at` must describe the real observation window. Never invent a digest from the API request itself.

## Promotion runbook

### 1. Declare and prepare

* Open an incident/change record and name incident commander, database, traffic, application, and security owners.
* Freeze unrelated deployments and configuration rollouts.
* Record current generation, effective region, traffic distribution, database writer, replication checkpoint, error rate, and backup reference.
* Confirm that the target can handle full capacity and that rollback criteria are clear.

### 2. Refresh evidence

Submit fresh target-health and source-to-target replication evidence. Read DR status immediately before beginning. Do not proceed with any blocker, unknown writer state, or evidence that will expire before confirmation can finish.

### 3. Begin promotion

```bash
curl -fsS -X POST \
  "$API/admin/gateways/$GATEWAY_ID/promotions" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d @promotion-request.json
```

The request must include `tenant_id`, target region, current `expected_generation`, a unique `idempotency_key`, and an approved reason. Handle the returned fence token only inside the secure orchestration workflow. Do not print it to CI logs or tickets.

At this stage the phase is `awaiting_external_confirmation`; the effective region has not changed.

### 4. Fence and switch

The external workflow must:

1. reject or stop writes in the previous primary;
2. prove the new database writer and required replication checkpoint;
3. switch global traffic using health-checked, bounded changes;
4. verify representative read and write traffic in the target;
5. retain one immutable evidence object;
6. have distinct allowlisted controllers sign the canonical confirmation envelope.

The signature covers controller, tenant, gateway, source, target, generation, fence token, previous-primary-fenced state, and the full evidence object. Signing only the evidence digest is insufficient.

### 5. Confirm

Submit `POST /admin/gateways/{gateway_id}/promotions/confirm` before the pending confirmation expires. Acceptance requires `phase=stable`, matching effective and desired region IDs, unchanged generation, rejected writes at the old primary, healthy traffic at the new region, and one immutable audit entry per promotion stage.

## Failback

Failback is another promotion, not an undo button. Repair and catch up the preferred region, collect new evidence in the reverse direction, use a new idempotency key, fence the current writer, switch traffic, and confirm through the same quorum. Never reuse stale failover evidence.

## Drills and alerts

Create drill records under `/admin/dr-drills`. Completed `passed` or `failed` records require external evidence and a completion timestamp. Record measured RPO/RTO, traffic impact, writer-fencing proof, and cleanup.

Alert on stale or unhealthy region evidence, replication lag above policy, unknown replication state, a pending confirmation approaching expiry, a region without enough controller keys to meet quorum, shared job Redis failure, missed backup, and overdue restore/failover drills.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.kangal.dev/production-operations/multiregion.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.