> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/production-operations/secrets.md).

# Secrets

Deliver production secrets at runtime from a managed secret system. Do not commit secret values to Git, Helm values, Compose overrides, CI logs, tickets, chat, release notes, configuration exports, or support bundles.

## Secret inventory

Maintain an owner, source, consumers, rotation period, and last rotation date for each secret class:

| Class                | Typical runtime key                                    | Consumers                                  |
| -------------------- | ------------------------------------------------------ | ------------------------------------------ |
| Token signing        | `SECRET_KEY`                                           | API and workers                            |
| Field encryption     | `FIELD_ENCRYPTION_KEY`                                 | API and workers that read encrypted fields |
| MongoDB connection   | `MONGODB_URL`                                          | API, workers, and backup/restore jobs      |
| Redis connection     | `REDIS_URL`                                            | API and workers                            |
| Durable-job Redis    | `KANGAL_BACKGROUND_JOBS_REDIS_URL`                     | Every regional API producer and worker     |
| OAuth and SSO        | Provider client secrets and signing material           | API                                        |
| Email                | `SMTP_USERNAME`, `SMTP_PASSWORD`                       | API or mail worker                         |
| Provider access      | Cloud, observability, AI, and integration credentials  | API and workers as required                |
| Data-plane trust     | Node token, signing key, cache key, client certificate | Control plane and selected data plane      |
| DR controller quorum | `KANGAL_DR_FENCING_CONTROLLERS_JSON`                   | Every regional control plane               |

Use different values for development, staging, and production. Separate unrelated trust boundaries instead of reusing one value for token signing, field encryption, data-plane signing, and external providers.

## Kubernetes delivery

Create the runtime Secret with an external secret controller, CSI driver, or an equivalent platform integration. Reference the existing Secret from the chart:

```yaml
api:
  extraEnvFrom:
    - secretRef:
        name: kangal-runtime

backgroundJobs:
  enabled: true
  inheritApiExtraVolumes: true
```

Use projected files through `api.extraVolumes` and `api.extraVolumeMounts` for private CA bundles, workload-identity tokens, or provider credential files. Workers inherit those API mounts by default; disable inheritance and declare worker-only projections when the worker needs a smaller trust set.

Avoid `api.secretEnv` and `credentials.secrets[].stringData` in a committed values file. The chart can render them, but Helm release state would then contain the supplied values. Prefer references to pre-existing Secrets.

Verify names and keys without reading values:

```bash
kubectl -n "$NAMESPACE" get secret kangal-runtime \
  -o go-template='{{range $key, $value := .data}}{{$key}}{{"\n"}}{{end}}'
kubectl -n "$NAMESPACE" get pod \
  -l app.kubernetes.io/instance="$RELEASE" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].ready}{"\n"}{end}'
```

Do not run `kubectl get secret -o yaml` as routine evidence.

## Startup and fail-closed behavior

In production, the API requires both a token-signing secret and an independent `FIELD_ENCRYPTION_KEY`; do not reuse either value for the other purpose. `/ready` returns `503` when MongoDB or Redis is unavailable. Durable jobs configured to require shared Redis refuse to start when the dedicated queue URL is absent. Independently verify credential privileges, certificate expiry, backup access, and provider reachability during deployment acceptance.

## Stored secret behavior

Kangal encrypts recognized secret-bearing fields before persisting supported plugin, vault, integration, protocol, identity, and AI configuration. Public admin responses redact those values. Vault containers redact every scalar because their key names are user-defined.

`FIELD_ENCRYPTION_KEY` is mandatory. Every replica that reads the same encrypted data must receive the same active value, and that value must be independent from token signing and every other trust boundary. Provision and rotate it only through the organization's approved field-encryption procedure.

## External secret references

Vault-aware configuration can store references instead of provider values:

```
vault://hashicorp/tenants/acme/apps/orders#token
vault://aws/tenants/acme/apps/orders#password
vault://gcp/apps/orders#api_key
vault://azure/tenants/acme/apps/orders#client_secret
vault://conjur/tenants/acme/apps/orders#password
```

Resolution is tenant-scoped and cached in Redis for a bounded period. Grant the runtime identity access only to the mapped tenant path. Use workload identity or projected short-lived credentials where possible. Keep custom provider endpoints on approved HTTPS origins and restrict outbound network access.

References are not secret values, but they reveal provider and object structure. Treat configuration exports and diagnostic artifacts as sensitive metadata.

## Rotation procedures

### Token signing

1. Generate a new value in the secret manager.
2. Move the current key into the previous-signing-key list and make the new key current.
3. Restart API and worker replicas gradually.
4. Verify new access and refresh tokens, then monitor signature failures.
5. Remove the previous key only after the maximum accepted token lifetime and any incident-specific grace period have elapsed.

### Field encryption

The runtime uses one active field-encryption key. Follow the approved rotation procedure; do not directly replace the key. Take a verified backup, test the procedure in staging, control secret writes for the change window, complete the approved re-encryption step, validate reads, and then deploy the new key to every replica.

### Database and Redis credentials

Use overlapping credentials when the provider supports them: create, grant least privilege, update the runtime Secret, roll workloads, verify `/ready` and worker health, then revoke the old credential. Confirm backup and restore identities separately.

### Provider, data-plane, and DR credentials

Rotate each trust boundary independently. Re-test the specific integration, re-register or roll data-plane nodes when node trust changes, and prove that the DR controller quorum can still sign the exact promotion envelope. Never paste controller keys or fence tokens into an incident ticket.

## Logging and evidence

Kangal redacts known secret fields and release-check output, but operator commands and third-party tools may not. Configure log collectors to remove authorization headers, cookies, connection strings, private keys, and arbitrary Kubernetes Secret data. Evidence should contain Secret names, versions, timestamps, workload revisions, and test results, never the values themselves.

If exposure is suspected, revoke first, preserve redacted evidence, rotate every derived or reused credential, invalidate sessions where applicable, and review audit records for use after the exposure time.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.kangal.dev/production-operations/secrets.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.