> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/observability/metrics-logs-traces.md). # Metrics, logs, and traces Kangal provides several related but deliberately separate signal paths. Prometheus metrics are optimized for alerting, gateway analytics provide bounded operational summaries, recent traffic provides request diagnostics, and correlated events connect request and trace identifiers over the configured retention window. ## Prometheus metrics Prometheus scrapes `GET /metrics`: ```yaml scrape_configs: - job_name: kangal metrics_path: /metrics static_configs: - targets: ['kangal-api:8000'] ``` ```bash curl -fsS "$KANGAL_URL/metrics" | grep '^kangal_' ``` The route has no application-level authentication dependency. Keep it on a private network, protect it with an ingress policy or proxy authentication, and avoid exposing it directly to the internet. Core series include: | Metric | Type and use | | ------------------------------------------- | --------------------------------------------------------------------------------------- | | `kangal_requests_total` | Request counter labeled by method, path, status, tenant, gateway, workspace, and route. | | `kangal_request_latency_seconds` | End-to-end request latency histogram. | | `kangal_route_latency_seconds` | Route latency histogram for route-level SLO diagnostics. | | `kangal_inflight_requests` | Current request gauge. | | `kangal_rate_limit_hits_total` | Requests rejected by rate limiting. | | `kangal_plugin_errors_total` | Plugin failures. | | `kangal_plugin_latency_seconds` | Plugin execution latency. | | `kangal_upstream_errors_total` | Upstream failures. | | `kangal_observability_writes_dropped_total` | Correlated events dropped with `reason="timeout"` or `reason="failure"`. | Some deployments also expose background-job metrics when those workers are enabled. Inspect the live `/metrics` payload rather than assuming optional collectors are present. Example PromQL: ```promql # Request rate by gateway sum by (tenant_id, gateway_id) (rate(kangal_requests_total[5m])) # 5xx percentage 100 * sum(rate(kangal_requests_total{status=~"5.."}[5m])) / clamp_min(sum(rate(kangal_requests_total[5m])), 1) # p95 request latency histogram_quantile( 0.95, sum by (le, tenant_id, gateway_id) ( rate(kangal_request_latency_seconds_bucket[5m]) ) ) # Correlated-event persistence failures sum by (reason) (rate(kangal_observability_writes_dropped_total[5m])) ``` Labels such as path, tenant, gateway, workspace, and route can create high cardinality. Aggregate in recording rules, keep user-generated IDs out of route templates, and set retention and remote-write limits in Prometheus according to tenant count. ## Gateway analytics `GET /api/v1/admin/gateways/{gateway_id}/overview` includes an `analytics` block with seven-day request counts, error rate, average and p95 latency, rate-limit hits, and slow routes/plugins. ```bash curl -fsS \ -H "Authorization: Bearer $KANGAL_TOKEN" \ "$KANGAL_URL/api/v1/admin/gateways/gw-production/overview" \ | jq '.analytics' ``` The error counter includes responses with status 400 or higher, and p95 is estimated from configured latency buckets. The analytics block is a diagnostic seven-day summary: recent requests can appear after ingestion delay, retained samples may not cover every event, and the result is not an exact event-by-event rolling window. Use a source with documented exact query semantics when completeness is required. ## Recent traffic logs The supported tenant-aware API is: ```http GET /api/v1/admin/gateways/{gateway_id}/traffic/recent?limit=50 ``` `limit` must be between 1 and 200. The endpoint verifies that the caller can see the gateway and returns the newest available diagnostic entries up to that limit. ```bash curl -fsS \ -H "Authorization: Bearer $KANGAL_TOKEN" \ "$KANGAL_URL/api/v1/admin/gateways/gw-production/traffic/recent?limit=100" \ | jq '.logs[] | {timestamp, method, path, status_code, latency_ms, upstream}' ``` Traffic results can include timestamp, method, path, route path and ID, tenant and gateway IDs, status, latency, upstream, plugins, client IP, and user identifier. The recent-traffic API does not return captured headers. External logging destinations can have different fields, so restrict access and sanitize upstream-sensitive values there. Recent traffic is a bounded, best-effort diagnostic view. Entries can be delayed, omitted, or unavailable because of retention, ingestion health, service changes, or the requested limit. The endpoint does not guarantee complete capture or durable retention; do not use it as a billing ledger, compliance archive, or proof that an unlisted request did not occur. ## Application logs Kangal writes structured JSON to standard output. Common fields are `ts`, `level`, `msg`, and `logger`, with request, tenant, gateway, and admin context added where available. ```bash kubectl logs -n kangal deploy/kangal-api --since=30m \ | jq 'select(.level == "ERROR")' kubectl logs -n kangal deploy/kangal-api --since=30m \ | jq 'select(.request_id == "support-case-8472")' ``` An optional best-effort Elasticsearch log handler is controlled by `ELASTIC_HOST`, `ELASTIC_PORT`, `ELASTIC_SCHEME`, and `ELASTIC_INDEX`. Delivery failure must not be treated as proof that application work failed; monitor the destination independently. ## Logging plugins Gateway policies can emit request logs to HTTP, file, TCP, UDP, Kafka, Datadog, or an OpenTelemetry-oriented HTTP receiver. Treat each destination as part of the request-path performance budget. Operational defaults and cautions: * Header capture defaults to disabled. Enable it only with explicit allow-listing and redaction requirements. * Sink failures default to non-fatal (`fail_on_error: false`). Turning failures into request failures should be an intentional availability decision. * Network sinks execute during request handling; benchmark timeouts and receiver failure behavior before production use. * Kafka delivery requires the Python Kafka client in the runtime unless a producer is injected by the deployment. * File logging writes to the API process filesystem. Use durable volumes and rotation, or prefer a centralized destination. ## Trace propagation Kangal accepts W3C `traceparent`, creates trace/span identifiers when it is absent or invalid, and forwards correlation context upstream. The OpenTelemetry logging plugin emits a JSON span envelope over HTTP with defaults such as service name `kangal-gateway` and span name `kangal.gateway.request`. Configure a receiver that accepts the plugin's HTTP JSON span envelope and operate the collector/tracing backend required by your environment. Sampling policies accept rates from 0 through 1 and support random or hash-based decisions; a decision header can expose whether a request was sampled. Trace an individual request across signals: 1. Capture `X-Request-ID` from the gateway response. 2. Query `/api/v1/admin/observability/requests/{request_id}` for trace and span IDs. 3. Search structured application logs by `request_id`. 4. Search the configured tracing receiver by `trace_id`. 5. Compare the event latency with Prometheus and recent traffic around the same timestamp. ## Security and data handling The correlated observability event path is payload-free: request/response bodies, credentials, and headers are not stored there. Traffic logs and external logging plugins are separate and may carry more detail depending on policy configuration. Before enabling a sink: * Classify paths, query strings, headers, client IPs, and user identifiers under the organization's data policy. * Keep `Authorization`, cookies, API keys, and credential material out of log payloads. * Encrypt transport, authenticate the receiver, and restrict tenant access. * Align receiver retention with Kangal retention and deletion obligations. * Send a synthetic request and verify both delivery and redaction. ## Troubleshooting | Symptom | Checks | | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Prometheus cannot scrape | Test `/metrics` from the Prometheus network, then inspect ingress and network policy. The route does not use a bearer token. | | Dashboard and overview disagree | They use different stores and aggregation semantics. Compare time windows and scrape gaps. | | Recent traffic misses requests | Check the requested time and limit, telemetry ingestion health, retention, dropped-write metrics, and recent service changes. Missing entries are possible in this diagnostic view. | | Trace backend has no span | Verify sampling, receiver compatibility, endpoint reachability, and plugin error metrics/logs. | | Request succeeds but sink has no log | With `fail_on_error: false`, sink failure is non-fatal. Inspect the destination and plugin errors. | | Metrics cardinality grows quickly | Aggregate dynamic labels, use normalized route paths, and add recording rules. | | Correlated request search is sparse | Check `kangal_observability_writes_dropped_total` and tenant retention. | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.kangal.dev/observability/metrics-logs-traces.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.