> For the complete documentation index, see [llms.txt](https://docs.kangal.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kangal.dev/ai-gateway/routing-resilience.md).

# Routing and resilience

AI Gateway resolves a requested model to an ordered target list, applies consumer policy before any provider call, and moves to another target only for eligible failures.

## Target selection

Set `strategy` on an enabled `model_route` record.

| Strategy        | Selection behavior                                                                                         |
| --------------- | ---------------------------------------------------------------------------------------------------------- |
| `ordered`       | Uses target array order                                                                                    |
| `priority`      | Highest numeric `priority` first; equal values preserve array order                                        |
| `weighted`      | Stateful weighted round-robin determines the first target, then deduplicates the remaining fallback order  |
| `least_latency` | Lowest live provider/model latency EWMA first, then configured latency, priority, and array order          |
| `semantic`      | Hash-vector similarity between prompt text and target hints; falls back to array order if no target scores |

Aliases such as `weighted_round_robin`, `weighted_rr`, `latency`, `least_latent`, `semantic_routing`, and `semantic_router` are accepted. `latency_ewma_alpha` must be greater than 0 and at most 1; runtime defaults to 0.3 and clamps live values to 0.01-1.0.

{% hint style="info" %}
For `priority`, larger numeric values are selected first. Review every target's value before enabling the strategy.
{% endhint %}

Semantic routing uses deterministic hash-vector similarity over configured keywords and hints. Its vector dimensions are clamped to 8-512.

## Fallback rules

The built-in retryable provider statuses are:

```
408, 409, 425, 429, 500, 502, 503, 504
```

Transport errors continue to the next target. A successful status with a non-object or non-JSON body is classified as `provider_malformed_response` and can also fall back. A valid non-retryable provider error is returned immediately with its status and parsed body. Kangal does not repeatedly retry one target; each configured target is attempted at most once per request.

When target exhaustion consists of transport or malformed-response failures, Kangal returns a gateway error, normally `502`, with a safe summary and attempt metadata. If the final target returns a valid provider HTTP error, Kangal can pass through that provider status and parsed body. Provider response bodies from malformed responses are not exposed.

Streaming fallback can occur while opening a provider stream. Once response headers have been returned and chunks are flowing, a later stream failure is recorded as `provider_stream_error`; it cannot transparently restart on another provider without duplicating or reordering client output.

## Consumer policy precedence

Policy lookup is deterministic:

1. Exact `consumer_id` policy.
2. First policy matching the consumer's stored group order.
3. Consumer policy named `default` or `*`.
4. No AI policy if none matches.

Example default policy:

```json
{
  "tenant_id": "acme",
  "kind": "policy",
  "name": "default",
  "enabled": true,
  "config": {
    "allowed_models": ["support", "gpt-4.1-*"],
    "denied_models": ["*-preview"],
    "token_budget": 250000,
    "rate_limit": 60,
    "rate_window_seconds": 60
  }
}
```

For a group policy, set `consumer_group` or `group` in `config`. For an exact consumer policy, set `consumer_id` or use the consumer ID as the record name. Model patterns support exact values, `*`, and trailing `*` prefixes.

## Budget and rate-limit behavior

* `token_budget` sets the estimated usage ceiling for the selected policy.
* A request is preflighted using roughly one token per four prompt characters plus `max_tokens` or `max_completion_tokens`.
* Final provider-reported usage is added after a successful response or final streaming usage event.
* `rate_limit` is enforced per selected policy subject and requested model in a rolling request window.
* A zero budget or rate limit disables that check.

## Exact and semantic cache

Cache is used for non-streaming responses. Exact keys include the request model, messages, temperature, top-p, max-token fields, stop values, and tenant/consumer scope by default. Streaming requests return `X-Kangal-AI-Cache: bypass`.

```json
{
  "tenant_id": "acme",
  "kind": "cache",
  "name": "default",
  "enabled": true,
  "config": {
    "enabled": true,
    "mode": "hybrid",
    "ttl_seconds": 300,
    "similarity_threshold": 0.86,
    "vary_by_tenant": true,
    "vary_by_consumer": true,
    "vector_backend": "mongodb_atlas",
    "vector_index_name": "ai_gateway_semantic_cache",
    "vector_path": "embedding",
    "vector_dimensions": 64,
    "vector_num_candidates": 100,
    "search_limit": 50,
    "embedding_provider": "openai-primary",
    "embedding_model": "text-embedding-3-small",
    "embedding_fail_open": true,
    "allow_scan_fallback": false
  }
}
```

Supported vector backend aliases are `memory`, `scan`, `mongo_scan`, `auto`, `atlas`, `mongodb_atlas`, `mongodb_vector`, and `native`. Search limit is clamped to 1-100. Vector dimensions are clamped to 8-512 at runtime; the Admin API accepts positive values.

Embedding choices:

* Built-in deterministic hash vectorizer when no provider is configured.
* A named AI provider plus `embedding_model`.
* An object with a provider name or public HTTPS `base_url`, model, path, credential/reference, and timeout.
* Azure OpenAI embedding path translation.

If external embedding fails and `embedding_fail_open` is true, the provider request continues without semantic search/write. If false, cache embedding failure fails the AI request. For Atlas, provision a compatible index on `ai_gateway_semantic_cache`; with `allow_scan_fallback: false`, missing or incompatible vector search fails instead of scanning.

## Resilience workflow

1. Configure two independently authenticated providers.
2. Map the public alias to explicit upstream models.
3. Start with `ordered` and a 20-60 second provider timeout.
4. Exercise retryable status, timeout, malformed response, and total-failure cases in a provider sandbox.
5. Verify `X-Kangal-AI-Fallback`, the response attempt list, and telemetry.
6. Enable least-latency only after real traffic has produced representative EWMA observations.
7. Enable semantic cache only after validating tenant/consumer scope and the production vector index.

## Troubleshooting

| Symptom                                    | Cause or action                                                                                          |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| Fallback never occurs                      | The first error is non-retryable, only one target exists, or streaming failed after output began         |
| Wrong provider wins priority               | Higher numeric priority is selected first                                                                |
| Weighted traffic looks uneven              | Weight controls the rotating first attempt, not a stateless random distribution; observe a larger sample |
| Least-latency ignores configured values    | Live EWMA takes precedence after the first observation                                                   |
| Cache misses across consumers              | `vary_by_consumer` defaults to true                                                                      |
| Cache returns `502`/`503` during embedding | Disable fail-closed embedding or fix the provider/index configuration                                    |
| Stream latency increases sharply           | Response guardrails buffer the complete SSE body before release                                          |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.kangal.dev/ai-gateway/routing-resilience.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.