> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/ai-gateway/providers-models.md). # Providers and models AI Gateway runtime configuration is stored as records under one verified `tenant_id + gateway_id` scope. Each record has these common fields: | Field | Rules | | ------------- | ------------------------------------------------------------------------------------ | | `tenant_id` | Required on create; cross-tenant values are rejected | | `kind` | `provider`, `model_route`, `policy`, `pricing`, `guardrail`, `redaction`, or `cache` | | `name` | Required, 1-120 characters; unique within the kind and scope | | `enabled` | Defaults to `true` | | `config` | Kind-specific JSON object | | `description` | Optional operator text | | `labels` | Optional JSON metadata; not interpreted unless documented | ## Provider fields Create a `provider` record for each independently authenticated upstream. | `config` field | Behavior | | ------------------------- | ------------------------------------------------------------------------------------------ | | `base_url` | Required. Runtime requires a public HTTPS destination and rejects redirects | | `provider_type` or `type` | `openai`, `openai_compatible`, `azure_openai`, or `anthropic` | | `models` | Optional exact or suffix-`*` membership patterns used when no explicit model route matches | | `api_key` | Plaintext input accepted, encrypted at rest, redacted on reads | | `api_key_ref` | Preferred `vault://...` reference; `secret_ref` and `vault_ref` are aliases | | `api_key_env` | Environment variable name; disabled unless explicitly allowed at runtime | | `headers` | Additional upstream headers. Values are treated as secret-capable fields | | `chat_completions_path` | Defaults to `/chat/completions` | | `embeddings_path` | Defaults to `/embeddings`; used only by external cache embeddings | | `timeout_seconds` | Non-negative request timeout; defaults to 60 seconds | | `prompt_cost_per_1k` | Non-negative input-token price | | `completion_cost_per_1k` | Non-negative output-token price | | `api_version` | Azure OpenAI `api-version` query value | | `anthropic_version` | Anthropic header; defaults to `2023-06-01` | Example: ```bash curl -sS -X POST \ "$KANGAL_ADMIN_URL/admin/gateways/$GATEWAY_ID/ai-gateway/configs" \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H 'Content-Type: application/json' \ -d "{ \"tenant_id\": \"$TENANT_ID\", \"kind\": \"provider\", \"name\": \"openai-primary\", \"enabled\": true, \"description\": \"Primary public OpenAI endpoint\", \"config\": { \"provider_type\": \"openai\", \"base_url\": \"https://api.openai.com/v1\", \"models\": [\"gpt-4.1*\"], \"api_key_ref\": \"vault://ai/openai#api_key\", \"timeout_seconds\": 45 } }" ``` ## Provider behavior | Type | Translation and authentication | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `openai` / `openai_compatible` | Sends the OpenAI-shaped payload to `{base_url}/{chat_completions_path}` with `Authorization: Bearer ...` | | `azure_openai` | Uses `/openai/deployments/{deployment}/chat/completions`, puts the target model in the deployment path, removes `model` from the body, and sends `api-key` | | `anthropic` | Uses `/v1/messages` by default, converts system/user/assistant messages for the provider, and sends `x-api-key` | Anthropic streaming remains provider-native. Only non-streaming Anthropic payloads are normalized to the OpenAI chat-completion response shape. For Azure, set `base_url` to the resource origin, for example `https://example.openai.azure.com`, and set the route target's `model` to the deployment name. For any OpenAI-compatible service, confirm that its endpoint accepts the OpenAI chat-completions shape and bearer authentication. Provider and embedding destinations must use public HTTPS; localhost, private, link-local, metadata, plain-HTTP, and redirecting destinations are rejected. ## Model routes A `model_route` maps the model requested by a client to one or more provider targets. `config.model` defaults to the record name suffix when omitted. ```json { "tenant_id": "acme", "kind": "model_route", "name": "chat:support", "enabled": true, "config": { "model": "support", "strategy": "ordered", "targets": [ {"provider": "openai-primary", "model": "gpt-4.1-mini", "priority": 20, "weight": 80}, {"provider": "anthropic-fallback", "model": "claude-3-5-haiku-latest", "priority": 10, "weight": 20} ] } } ``` Target fields: | Field | Meaning | | ----------------------------------------------------------- | ----------------------------------------------------------------------- | | `provider` or `name` | Required configured provider name | | `model` or `upstream_model` | Optional model/deployment rewrite; omit to preserve the requested model | | `priority` | Integer. Higher values are selected first by `priority` strategy | | `weight` | Integer >= 1 for weighted rotation | | `latency_ms` | Optional non-negative seed used until live EWMA exists | | `semantic_text`, `description`, or `prompt_hint` | Text used by semantic target ordering | | `semantic_keywords`, `keywords`, `tags`, or `semantic_tags` | Additional semantic hints | Requested model routes support exact names and trailing wildcard patterns such as `gpt-4.1-*`. A wildcard route should omit target-level `model` unless every request is intentionally rewritten to one fixed upstream model. ## Pricing records Pricing can be embedded in a provider or supplied as a `pricing` record: ```json { "tenant_id": "acme", "kind": "pricing", "name": "openai-primary:support", "config": { "provider": "openai-primary", "model": "support", "currency": "USD", "input_per_1k": 0.00015, "output_per_1k": 0.0006 } } ``` Cost-estimate selection uses the configured provider name and reports USD. When models on one upstream have different prices, use distinct provider records so each route resolves to the intended rates. Treat the result as an estimate and reconcile it with provider billing. ## Updating safely `PATCH` replaces the supplied `config` object; it does not deep-merge omitted keys. Read the record, preserve required fields, then submit the complete new configuration. A returned `[redacted]` marker can be sent back to preserve the stored encrypted value. ```bash curl -sS -X PATCH \ "$KANGAL_ADMIN_URL/admin/gateways/$GATEWAY_ID/ai-gateway/configs/$CONFIG_ID" \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H 'Content-Type: application/json' \ -d '{ "enabled": true, "config": { "provider_type": "openai", "base_url": "https://api.openai.com/v1", "api_key_ref": "[redacted]", "timeout_seconds": 30 } }' ``` ## Troubleshooting | Symptom | Check | | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `422 Invalid AI Gateway config` | Required kind-specific field, list element, numeric minimum, strategy name, or regex syntax | | `503 AI provider outbound URL was rejected` | Public HTTPS, DNS result, no redirect, no private/link-local/metadata destination | | `503 AI provider credential is unavailable` | Vault tenant scope, secret fragment, provider access, or environment-reference policy | | `503 No AI provider configured for model` | Enabled provider, route pattern, target name, provider model membership, and default provider | | Azure path contains `openai/deployments` twice | Use the Azure resource origin as `base_url` | | OpenAI-compatible provider returns auth or payload errors | Confirm bearer authentication, chat-completions path, and OpenAI request/response compatibility | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.kangal.dev/ai-gateway/providers-models.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.