> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/observability/slos-audits.md). # SLOs and audits Service-level objectives answer whether observed gateway traffic meets an availability or latency target. Audit records answer who changed administrative state and what changed. They are separate product records with independent query and retention contracts. ## SLO permissions and tenant scope SLO endpoints use the standard admin role and Admin API scope rules: * Read calls require a readable admin role or `admin:read` token scope. * `POST`, `PUT`, and `DELETE` require an admin writer or `admin:write` scope. * Regular administrators are restricted to their tenant. * A super administrator without an assigned tenant must provide `tenant_id`. All examples use: ```bash export KANGAL_URL='https://kangal.example.com' export KANGAL_TOKEN='kangaladm_...' export TENANT_ID='tenant-acme' ``` ## SLO object An SLO supports two objective types: * `availability`: a bad event is a response with status `500..599`. Client 4xx responses do not consume this error budget. * `latency`: a bad event has latency strictly greater than `latency_threshold_ms`. Create an availability objective: ```bash curl -fsS -X POST \ -H "Authorization: Bearer $KANGAL_TOKEN" \ -H 'Content-Type: application/json' \ "$KANGAL_URL/api/v1/admin/slos" \ -d '{ "tenant_id": "tenant-acme", "gateway_id": "gw-production", "name": "Production availability", "description": "Successful gateway responses over 28 days", "objective": { "type": "availability", "target_percent": 99.9 }, "evaluation_window_minutes": 40320, "alert_policy": { "enabled": true, "warning_burn_rate": 1.0, "critical_burn_rate": 2.0 }, "enabled": true }' | jq ``` Create a latency objective: ```bash curl -fsS -X POST \ -H "Authorization: Bearer $KANGAL_TOKEN" \ -H 'Content-Type: application/json' \ "$KANGAL_URL/api/v1/admin/slos" \ -d '{ "tenant_id": "tenant-acme", "gateway_id": "gw-production", "name": "Orders latency", "objective": { "type": "latency", "target_percent": 99.0, "latency_threshold_ms": 750 }, "evaluation_window_minutes": 10080, "alert_policy": { "enabled": true, "warning_burn_rate": 1.0, "critical_burn_rate": 3.0 } }' | jq ``` Validation limits: | Field | Constraint | | --------------------------------- | ---------------------------------------------------------------------------- | | `name` | 1..120 characters | | `description` | At most 500 characters | | `gateway_id` | At most 128 characters | | `objective.target_percent` | Greater than 0 and less than 100 | | `objective.latency_threshold_ms` | Required for latency objectives and greater than 0; invalid for availability | | `evaluation_window_minutes` | 5..43,200; default 10,080 (7 days) | | `alert_policy.warning_burn_rate` | Greater than 0; default 1 | | `alert_policy.critical_burn_rate` | Greater than warning; default 2 | ## SLO endpoints | Method and path | Purpose | | --------------------------------------------------------------- | ------------------------------------------------- | | `GET /api/v1/admin/slos?tenant_id=...&gateway_id=...` | List SLO definitions. | | `POST /api/v1/admin/slos` | Create an SLO. | | `GET /api/v1/admin/slos/{slo_id}?tenant_id=...` | Get one SLO. | | `PATCH /api/v1/admin/slos/{slo_id}?tenant_id=...` | Update selected SLO fields. | | `DELETE /api/v1/admin/slos/{slo_id}?tenant_id=...` | Delete an SLO. | | `POST /api/v1/admin/slos/evaluate?tenant_id=...&gateway_id=...` | Evaluate all matching SLOs now. | | `POST /api/v1/admin/slos/{slo_id}/evaluate?tenant_id=...` | Evaluate one SLO now. | | `POST /api/v1/admin/slos/{slo_id}/acknowledge` | Acknowledge that SLO's warning or critical alert. | | `GET /api/v1/admin/slos/{slo_id}/timeline?tenant_id=...` | Read that SLO's timeline events. | Use the generated OpenAPI document for the complete request schema for update and evaluation filters: ```bash curl -fsS "$KANGAL_URL/openapi.json" \ | jq '.paths | with_entries(select(.key | startswith("/api/v1/admin/slos")))' ``` ## Evaluation model Evaluation calculates: ``` allowed_bad_events = total_events * (100 - target_percent) / 100 burn_rate = bad_events / allowed_bad_events ``` State is `critical` when burn rate is at or above the critical threshold, `warning` when it is at or above the warning threshold, and `healthy` otherwise. With no usable samples, state is `no_data`. A disabled objective or disabled alert evaluates healthy when data exists; it does not turn `no_data` into healthy. Evaluation uses the retained observations available for the requested tenant, gateway, and window. The result reports its source, sample counts, observed time range, and `evaluated_at`; use these fields to judge freshness and coverage. Observations can arrive late, and configured retention may cover less than the requested window. With no usable samples, the state is `no_data`. An SLO result is a diagnostic alerting signal, not an exact billing ledger, compliance archive, or proof of complete request capture. Before acting on a result, confirm that the observed time range and sample count are suitable for the objective. Evaluation runs when an operator, the console, or an external scheduler calls an evaluate endpoint. For unattended alert refresh, schedule the all-SLO operation: ```bash curl -fsS -X POST \ -H "Authorization: Bearer $KANGAL_TOKEN" \ "$KANGAL_URL/api/v1/admin/slos/evaluate?tenant_id=$TENANT_ID&gateway_id=gw-production" \ | jq ``` Choose an interval short enough for the alerting objective, but avoid evaluating faster than the underlying traffic and persistence can provide meaningful new samples. ## Alert state, acknowledgement, and timeline Only `warning` and `critical` incidents can be acknowledged. A state transition resets the acknowledgement so the new condition is visible. Timeline entries include SLO creation/update, alert transitions, and acknowledgements. ```bash curl -fsS \ -H "Authorization: Bearer $KANGAL_TOKEN" \ "$KANGAL_URL/api/v1/admin/slos?tenant_id=$TENANT_ID&gateway_id=gw-production" \ | jq '.items[] | {id, name, evaluation, alert}' curl -fsS -X POST \ -H "Authorization: Bearer $KANGAL_TOKEN" \ -H 'Content-Type: application/json' \ "$KANGAL_URL/api/v1/admin/slos/SLO_ID/acknowledge" \ -d '{"tenant_id":"tenant-acme","note":"Investigating under incident INC-8472"}' \ | jq curl -fsS -G \ -H "Authorization: Bearer $KANGAL_TOKEN" \ --data-urlencode "tenant_id=$TENANT_ID" \ --data-urlencode 'limit=100' \ "$KANGAL_URL/api/v1/admin/slos/SLO_ID/timeline" | jq ``` Timeline `limit` defaults to 100 and is capped at 500. ## Audit records Administrative writes covered by Kangal's audit policy create immutable audit records. An entry can include: * Actor ID and username, action, entity type, and entity ID. * Tenant ID, request ID, source IP, timestamp, and metadata. * Before and after values for changed state. * Schema version, `immutable: true`, previous hash, and audit hash. The hashes form a per-tenant chain, with a separate global chain. Export records to the organization's protected evidence store and perform retention and integrity verification according to its compliance procedure. The audit list route requires a user/session access JWT. Obtain one through login: ```bash ACCESS_TOKEN=$( curl -fsS -X POST \ -H 'Content-Type: application/json' \ "$KANGAL_URL/auth/login" \ -d '{"username":"admin@example.com","password":"REDACTED","tenant_id":"tenant-acme"}' \ | jq -r '.access_token' ) curl -fsS -G \ -H "Authorization: Bearer $ACCESS_TOKEN" \ --data-urlencode "tenant_id=$TENANT_ID" \ --data-urlencode 'actions=create' \ --data-urlencode 'actions=update' \ --data-urlencode 'entity=service' \ --data-urlencode 'skip=0' \ --data-urlencode 'limit=50' \ --data-urlencode 'sort_direction=desc' \ "$KANGAL_URL/api/v1/admin/audits/" \ -D /tmp/kangal-audit-headers.txt | jq grep -i '^x-total-count:' /tmp/kangal-audit-headers.txt ``` `skip` defaults to 0. `limit` defaults to 50 and is capped at 200. `sort_direction` accepts `asc` or `desc`; repeat `actions` to match multiple actions. A regular admin's query is forced to their own tenant, while a super administrator can query globally or filter by tenant. Audit coverage focuses on administrative changes. It is not a record of every read. SLO state changes have their own incident timeline, and configuration snapshots are separate records. Do not assume that the absence of an audit event proves an operation did not occur. Set the audit storage lifecycle, backup, export, and legal-hold policy at the database and platform layers. ## Troubleshooting | Symptom | What to check | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | SLO remains `no_data` | Check tenant/gateway IDs, the observed time range, sample counts, observability retention, ingestion health, and write-drop metrics. | | Availability SLO ignores 4xx | This is expected; only 5xx responses are bad availability events. | | Latency event at the threshold is good | This is expected; bad latency is strictly greater than the threshold. | | Acknowledgement fails | Only warning or critical states are acknowledgeable; refresh the incident because a transition may have replaced its state. | | Alerts never refresh | Run or schedule `POST /api/v1/admin/slos/evaluate`; evaluations occur on explicit calls. | | Audit returns `401` with an Admin API token | Use a user access JWT from `/auth/login` for the audit route. | | Audit list looks truncated | Read `X-Total-Count` and paginate with `skip`/`limit`, up to 200 per request. | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.kangal.dev/observability/slos-audits.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.