> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/production-operations/troubleshooting.md). # Troubleshooting Start with read-only evidence. Record UTC time, environment, release tag, Helm revision, incident/change ID, affected tenants and routes, first symptom, and the last known-good time before restarting or scaling anything. ## First five minutes ```bash kubectl config current-context helm status "$RELEASE" -n "$NAMESPACE" kubectl -n "$NAMESPACE" get deploy,pod,svc,ingress,hpa,pdb kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp | tail -n 50 curl -sS -o /dev/null -w 'health=%{http_code}\n' "$PUBLIC_URL/health" curl -sS -o /dev/null -w 'ready=%{http_code}\n' "$PUBLIC_URL/ready" ``` Then collect component logs by release and component label. Do not add `--previous` unless the container restarted, and do not paste raw logs into support channels until authorization headers, cookies, connection strings, private keys, and customer data have been removed. ## Understand the probes * `/health` returns `200` when the API process can serve HTTP. It does not check MongoDB, Redis, upstreams, or provider integrations. * `/ready` pings MongoDB and Redis. It returns `503` when either dependency check fails. * A healthy but unready pod should remain alive and leave service endpoints. Restarting it repeatedly does not repair the dependency. ## Symptom matrix | Symptom | Likely boundary | Check first | | --------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------- | | `/health` fails | Process, pod, Service, ingress, or TLS | Pod state, logs, endpoints, ingress events | | `/health` is 200; `/ready` is 503 | MongoDB or Redis | DNS, TLS/auth, network policy, primary/Redis availability | | Frontend loads but API calls fail | Frontend API URL, ingress paths, CORS | Browser request URL, API Service, configured origins | | API is ready; login fails | Signing key, Redis sessions, identity provider | Auth logs, Secret version, callback URL, provider status | | Queue depth grows | Workers, shared Redis, provider outage | Worker replicas, worker health, pending/retry/dead-letter counts | | Config rollout stalls | Data-plane connectivity or NACK | Heartbeats, desired/applied version, checksum, rollout phase | | Rollback/snapshot returns 503 | MongoDB lacks transactions | Topology, writable primary, replica-set URI options | | Ingress returns 502/504 | Service endpoints, network policy, timeout | Endpoints, pod readiness, ingress logs, upstream latency | | TLS errors | Certificate, SNI, DNS, or trust chain | Served certificate, Certificate conditions, DNS target | | Helm upgrade times out | Scheduling, probes, PDB, image pull, hooks | Events, Pending reason, image status, readiness, PDB | ## API crash loop 1. Read the terminated container reason and previous logs. 2. In production, confirm `SECRET_KEY` exists without reading its value. 3. Check MongoDB/Redis environment key names and Secret projection. 4. Verify writable `/tmp` and any projected CA/token paths when the root filesystem is read-only. 5. Confirm image architecture and immutable tag match the cluster. 6. Check memory limit and `OOMKilled` before increasing replicas. Do not replace the production signing key with a temporary value to make the pod start; that invalidates sessions and may break encrypted data access. ## MongoDB failures Use provider tools or read-only `hello`/replica status to identify the writable primary, majority, member health, and last committed point. Confirm the URI includes the correct replica-set or sharded topology and TLS/auth options. Pause configuration rollout, snapshot, rollback, and restore writes while there is no writable majority. Never force replica reconfiguration, remove members, downgrade to a standalone database, or restore while an election may complete. Exact gateway rollback and transaction-consistent configuration capture fail before the first write when transactions are unavailable. Restore topology support, compare live and snapshot checksums, then start a new fenced rollout. ## Redis and background jobs `/ready` depends on Redis. Durable job producers return `503` rather than execute work inline when Redis cannot accept or read a job. Workers stop acknowledging and retry; pending entries are reclaimed after the lease when Redis returns. Check: * API and worker use the intended Redis endpoint; * multi-region producers and workers use the same durable-job endpoint and prefix; * Redis persistence, memory, eviction, latency, stream group, pending entries, and dead-letter growth; * at least one healthy worker exists for every registered payload version. Do not switch production to the in-memory backend or flush Redis to clear an incident. Before replay, verify whether the external side effect completed and use the job or workload idempotency key. ## Stale or offline data plane An offline data plane may continue on a verified signed last-known-good bundle. A stale bundle is not permission to run indefinitely. 1. Compare control-plane reachability, DNS, TLS, node token, and client-certificate fingerprint. 2. Record desired/current version and checksum, bundle issue time, last heartbeat, pending reports, and NACK reason. 3. Preserve cache and anti-rollback storage; never edit the cache envelope or delete trust storage to bypass rollback protection. 4. Restore connectivity and publish a correctly signed newer bundle. 5. Verify online mode, cleared stale state, ACK, checksum convergence, and unchanged plus newly introduced routes. Pause promotion when selected nodes lack a valid last-known-good snapshot or exceed the rollout policy's failure threshold. ## NetworkPolicy and ingress Render the active policies and compare them with required flows. API and workers may need DNS, MongoDB, Redis, identity, email, observability, provider, and upstream egress. Frontend needs API egress. Data plane needs control-plane and upstream access. Test from a short-lived diagnostic pod using the same namespace, ServiceAccount, DNS, and policy labels where your security policy permits it. Do not solve a policy issue by opening unrestricted internet egress; add the required destination and retain the test. For mTLS, the TLS terminator must strip client-supplied verification/certificate headers before injecting trusted values. Direct pod access must be blocked or a caller can bypass the trust boundary. ## Certificate failure 1. Inspect Certificate, CertificateRequest, issuer, DNS challenge, and Secret reference. 2. Record current certificate serial, validity, and Secret resource version. 3. Keep the last valid TLS Secret while repairing issuance. 4. Verify the certificate actually served through ingress with `openssl s_client`. 5. Check `/health` over the public TLS path. Do not delete the live TLS Secret as the first recovery action. ## Backup or restore job failure * Confirm the existing PVC is bound, writable for backup, and read-only for restore. * Confirm the Secret contains `MONGODB_URL` without printing it. * Check database DNS, TLS/auth, NetworkPolicy, image pull, deadline, and storage space. * Verify archive and manifest names are adjacent and SHA-256 matches. * Never retry `--drop` against an uncertain target. Reconfirm database identity first. ## Helm rollout failure Inspect `helm status`, release history, events, pod scheduling, image pulls, probes, PDB, HPA, and hooks. `--atomic` may already have restored the previous revision. Confirm the active revision and image digest before issuing another rollback. If smoke tests fail after workloads become ready, preserve logs and evidence, then use the previous known-good revision. Handle database/configuration rollback separately. ## Incident stop conditions Stop a recovery or drill on tenant leakage, unexpected mutation, wrong restore target, unknown database writer, error-budget breach, evidence expiry, failed cleanup, or impact outside the approved scope. Escalate with redacted logs, request/correlation IDs, versions and checksums, topology state, timeline, and actions already taken. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.kangal.dev/production-operations/troubleshooting.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.