> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/ai-gateway/security-observability.md). # Security and observability AI Gateway treats client identity, provider credentials, and provider egress as separate trust boundaries. Client-supplied tenant, gateway, and consumer values are checked against stored state before configuration is loaded. ## Runtime authentication ### Consumer API key Send: ``` X-Tenant-ID: X-Gateway-ID: X-API-Key: ``` `X-Tenant-ID` is a lookup selector, not trusted identity. The credential's tenant, consumer tenant, and gateway tenant must all match. Optional `X-Consumer-ID` or `X-Consumer-Username` claims must match the verified consumer. ### Consumer bearer JWT Send `Authorization: Bearer ` and `X-Gateway-ID`. The token must be a valid Kangal token containing a stored `consumer_id`; tenant and consumer claims are rechecked against the database. Typical failures: | Status | Meaning | | ------ | ------------------------------------------------------------- | | `400` | Missing `X-Tenant-ID` for API key or missing `X-Gateway-ID` | | `401` | Missing/invalid consumer credential | | `403` | Spoofed tenant/consumer/gateway claim or cross-tenant gateway | | `404` | Gateway does not exist | | `429` | AI policy or plan quota denied the request | ## Provider secrets Provider plaintext `api_key` and secret-capable header values are encrypted at rest and redacted from API and audit responses. Prefer a tenant-scoped `vault://...` reference: ```json { "api_key_ref": "vault://ai/providers/openai#api_key" } ``` The resolver requires the verified tenant at request time. It accepts a scalar secret or a single scalar field; use the URL fragment to select a field from a structured secret. `env://NAME`, `env:NAME`, and `api_key_env` are disabled by default. They work only when `AI_GATEWAY_ALLOW_ENV_SECRET_REFS=true`, and the variable name must match a safe environment identifier. This is process-global, not tenant-scoped, so Vault references are safer for shared deployments. The direct runtime never forwards the inbound Kangal authorization header to a provider. ## Outbound request policy Before credentials are attached, provider and external embedding URLs are validated with these rules: * HTTPS is required. * URL credentials are not allowed. * Loopback, private, link-local, multicast, reserved, and cloud metadata destinations are blocked. * DNS results must remain within the allowed public address policy. * Redirects are not followed and 3xx responses are rejected. ## Guardrails Request guardrails enforce: * `enabled`; * `max_prompt_chars` across textual message content; * case-insensitive substring matches in `blocked_terms`. Response guardrails enforce case-insensitive `response_blocked_terms` and regex values in `response_blocked_patterns`. A blocked normal response returns `403`. A protected stream is buffered in full and returns no provider bytes if a rule matches. ## PII replacement When an enabled `redaction` record is active, Kangal replaces email, phone-like, and US SSN-like values in string or text-part message content before the provider call: ``` [REDACTED_EMAIL] [REDACTED_PHONE] [REDACTED_SSN] ``` Redaction counts are recorded in response metadata. Enabling redaction applies all three built-in detectors. Validate these detectors against your traffic: the replacement pass covers string and text-part message content, not arbitrary nested JSON, images, audio, tool payload semantics, or every locale-specific identity format. ## Telemetry and audit governance Use payload-free operational telemetry and immutable audit events for governed monitoring. Restrict access by tenant and role, define retention, and export records only to approved monitoring and audit systems. ## Telemetry endpoints All telemetry endpoints require an admin credential and verify that the gateway belongs to the requested tenant. ``` GET /admin/gateways/{gateway_id}/ai-gateway/telemetry/summary GET /admin/gateways/{gateway_id}/ai-gateway/telemetry/requests GET /admin/gateways/{gateway_id}/ai-gateway/telemetry/live ``` ### Summary Query parameters: | Parameter | Default | Limits | | ----------- | -------------- | ------------------- | | `tenant_id` | Admin's tenant | Cross-tenant denied | | `hours` | `24` | 1-720 | The response includes request/provider-call counts, availability, fallback and cache rates, normalized tokens, estimated cost and savings, average/p95 latency, and observed provider health. Health is outcome-based, not a synthetic probe: * `healthy`: all observed provider calls succeeded; * `degraded`: at least one failed; * `unavailable`: at least half failed; * `no_data`: no provider calls in the window. Policy/client failures do not count as provider outages. Summary scans are bounded to the newest 5,000 events and return `truncated: true` when exceeded. ### Request drilldown `hours` is 1-720 and `limit` is 1-100, default 25. Optional filters are `provider`, `status_code`, `fallback_used`, and `cache_hit`. Items include IDs, scope, route/consumer references, status, outcome, latency, attempts, usage, cost, cache/fallback state, error type, and stream completion. Prompts, responses, and credentials are not included. ### Live tail `limit` is 1-100, default 25. Persist and resend `next_cursor`. When `has_more` is true, fetch immediately; otherwise wait `poll_after_ms` (normally 2,000 ms). The cursor is versioned, HMAC-signed, bound to tenant/gateway and retention, and supports configured previous signing secrets during key rotation. Expect: * `400` for malformed, unsigned legacy, expired-signature, or tampered cursors; * `403` for a cursor from another tenant/gateway; * `backpressure`, `gap_detected`, or `cursor_expired` when a bounded consumer or retention window was overtaken. The dashboard keeps only its newest 100 local rows. ## Retention and failure behavior `ai_gateway_operations` has a TTL index. Retention defaults to 720 hours and is clamped to 24-720 by `AI_GATEWAY_TELEMETRY_RETENTION_HOURS`. Runtime telemetry writes are fail-open and bounded by `AI_GATEWAY_TELEMETRY_PERSIST_TIMEOUT_MS`, default 100 ms and clamped to 1-1,000 ms. A slow database does not hold open the provider response. Dropped writes increment the payload-free Prometheus counter `kangal_ai_gateway_telemetry_persistence_dropped_total{store,reason}`. Immutable audit events are stored separately from the TTL operation collection. ## Security checklist 1. Issue tenant-scoped consumer credentials and test spoofed-scope rejection. 2. Store provider keys in Vault and restrict outbound HTTPS destinations. 3. Keep client-provided provider auth and environment references disabled. 4. Test request `blocked_terms` and response terms/patterns with representative normal and streaming payloads. 5. Validate PII detectors against the actual languages and data formats. 6. Apply tenant-scoped access and retention controls to telemetry and audit data. 7. Alert on provider failure classes, telemetry drops, budget denials, and unusual cache savings. 8. Treat cost values as estimates and reconcile against provider billing. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.kangal.dev/ai-gateway/security-observability.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.