Skip to content

AI Model Gateway

Editions: OSS, Cloud, Enterprise. Unless stated otherwise, everything on this page ships in OSS.

The Preloop Model Gateway centralizes model traffic from managed runtimes. Instead of handing provider API keys to every local agent container, Preloop keeps credentials in its secret store and issues short-lived gateway tokens to enrolled runtimes.

Why Use the Gateway

Direct provider access Through Preloop Gateway
Raw API keys on every machine Short-lived runtime tokens
Spend scattered across providers Canonical ApiUsage ledger with attribution
No per-agent model limits Subject-scoped allowed-model lists
Hard to audit prompts/responses Normalized gateway events and session drill-down

The gateway is the companion to the MCP Safety Layer: tools go through /mcp/v1, model calls go through /openai/v1, /anthropic/v1, or /gemini/v1.

Supported Endpoints

Preloop exposes provider-compatible ingress on the gateway service:

Path Purpose
GET /openai/v1/models Model listing for OpenAI-compatible clients
POST /openai/v1/chat/completions Chat completions with SSE streaming
POST /openai/v1/responses Responses API with SSE streaming
POST /anthropic/v1/messages Anthropic-format clients such as Claude Code and Claude Desktop
POST /anthropic/v1/messages/count_tokens Token counting; forwarded upstream, not recorded or charged
GET /anthropic/v1/models Model listing in Anthropic's shape, filtered by allowed models
Gemini-compatible routes under /gemini/v1beta Google/Gemini clients when configured

Authentication reuses the same runtime bearer tokens issued during managed agent onboarding. Preloop preserves API-key context and runtime-principal metadata for attribution.

The OpenAI-compatible surface also serves clients you point at the gateway by hand with an account API key. Cursor is the main example: its Override OpenAI Base URL setting routes the AI panel's third-party model calls (Ask/Plan and Agent modes) through /openai/v1, with alias resolution serving Claude and Gemini models over the same path. Tab autocomplete, inline edit, and Cursor-billed bundled models stay on Cursor's backend and cannot be routed through any gateway. See Cursor model routing.

Subscription OAuth Passthrough

If you use Claude Code with a Pro/Max subscription instead of an API key, Preloop can still route that traffic through the gateway. The credential type is a Claude Code OAuth token stored on the AI model, and the gateway treats it specially:

  • Byte-faithful proxying. Anthropic requires the exact Claude Code request shape (the sentinel system block first, cache_control markers intact) and rejects anything else. So the gateway forwards system, messages, and nested cache_control blocks untouched, only the model identifier and streaming flag are set by Preloop, and SSE streams are relayed verbatim.
  • Still governed and budgeted. Budget preflight, governance tool-stripping, attribution, and usage recording run exactly as on the normal path. Only the message bytes are sacred.
  • No context optimization on this traffic. Message-level context optimizations are deliberately not applied: rewriting blocks would break byte-fidelity and destroy the prompt-cache prefix. See Session Optimization.
  • The credential does not leak to other agents. The OAuth token lives on one AI model in your account, and gateway calls resolve models per authenticated subject with per-agent attribution: another agent cannot ride a Claude Code subscription credential.
  • Validation is surfaced honestly. Onboarding sends a direct gateway route/accounting probe using managed configuration; it does not launch the application. If the provider throttles it or refuses on billing/quota grounds, the agent stays enrolled but shows an unverified badge in the console (and a Reason in preloop agents list/onboarding summaries) until preloop agents validate <agent> --live passes. A successful probe verifies the direct route and accounting; application behavior remains unverified.
  • Brief provider overloads are retried. When Anthropic answers an explicit 502, 503, 504 or 529 that the gateway classifies as transient, the gateway retries the same request (same body, credential and headers) with jittered exponential backoff, honouring Retry-After up to MODEL_GATEWAY_UPSTREAM_RETRY_AFTER_CAP_SECONDS. A provider x-should-retry: false or a terminal classification stops it. Nothing else is retried: not 429, auth or revoked-token errors, validation errors, quota or policy denials, and not transport failures or timeouts (Anthropic may already have processed the request). A stream is never reopened once Anthropic has accepted it. The budget is MODEL_GATEWAY_UPSTREAM_RETRY_MAX_ATTEMPTS total attempts (default 3); set it to 1 to turn retries off. If every attempt fails, the client gets the last attempt's error, mapped exactly as a single failed attempt would be (a final 504 is reported as 502, like any other unclassified provider 5xx), with that attempt's error message and x-should-retry / rate-limit headers. The request still records one usage row, with meta_data.upstream_retries holding the number of extra attempts. Claude Code runs its own retries on top of this, so one client request can reach Anthropic up to three times by default, and Claude Code's retries multiply that. These retries absorb short overloads; they cannot hide a long Anthropic outage.

Claude Desktop and the Claude apps gateway

Claude Desktop and Claude Code can reach Preloop's Anthropic surface in two ways.

Desktop direct. Set Desktop's inferenceProvider to gateway, inferenceGatewayBaseUrl to https://<your-preloop>/anthropic, and inferenceGatewayApiKey to a Preloop API key. Add inferenceCustomHeaders: {"X-Preloop-Client": "claude-desktop"} so usage rows record client: claude_desktop. Desktop's model picker reads GET /anthropic/v1/models. Attribution and budgets follow the API key, as for any other client.

Claude apps gateway upstream. A customer-run Claude apps gateway can use Preloop as its provider: anthropic upstream:

upstreams:
  - provider: anthropic
    base_url: https://<your-preloop>/anthropic
    forward_user_identity: true
    auth:
      api_key: ${PRELOOP_UPSTREAM_KEY}
    headers:
      x-preloop-upstream-secret: ${PRELOOP_UPSTREAM_SECRET}

The key must be a trusted upstream key: an API key with the scope model_gateway:trusted_upstream. Only account admins can create one:

curl -X POST https://<your-preloop>/api/v1/auth/api-keys \
  -H "Authorization: Bearer <admin token>" -H "Content-Type: application/json" \
  -d '{"name": "claude apps gateway",
       "scopes": ["model_gateway:trusted_upstream"],
       "trusted_upstream_secret": "<at least 16 random characters>",
       "per_subject_budget": {"period": "monthly", "hard_limit_usd": 50}}'

On a trusted upstream key Preloop reads the identity headers the apps gateway adds with forward_user_identity: true:

Header Meaning
x-claude-gateway-user-id IdP sub; required to identify the developer
x-claude-gateway-user-email Developer email, when the IdP supplies one
x-litellm-end-user-id Email fallback
  • Each developer becomes a gateway subject, keyed on the IdP sub and scoped to the key. When the email matches an existing member of the key's account, the subject is linked to that member, so the member's user budgets apply. Preloop never creates users, logins or permissions from these headers.
  • On any other credential the identity headers are ignored, because any client can forge them.
  • The secret is optional but recommended, because the key sits in the gateway's config file. When trusted_upstream_secret is set, a request without the matching x-preloop-upstream-secret header gets 401. Preloop stores only a sha256 hash.
  • Budgets: a gateway_subject budget policy limits one developer; per_subject_budget on the key is the default for every developer without their own policy; the key's api_key budget still caps the gateway as a whole. Subject-scoped allowed_models (governance subject type gateway_subjects, keyed by the subject id) apply to messages, count_tokens and GET /anthropic/v1/models. There is no REST writer for gateway_subjects yet; until there is, per-developer model limits are set in the account's subject governance store, and the key's own allowed_models covers every developer behind it.
  • Usage rows carry meta_data.gateway_source (claude_apps_gateway or direct), meta_data.client (claude_desktop, claude_code or unknown), gateway_subject_id and gateway_subject_email. Audit entries name the subject.
  • Sessions keep Claude Code's x-claude-code-session-id. A request without it is grouped as gw:<gateway_subject_id>:<UTC date>.

Why denials are 429 on this path. The apps gateway fails over to its next upstream on 403, which would let a developer bypass a Preloop budget. It relays a 429 on an email-carrying request to the developer as-is. So on a trusted upstream key with identity headers, budget denials return:

HTTP/1.1 429 Too Many Requests
retry-after: 86400
x-should-retry: false

{"type":"error","error":{"type":"billing_error","message":"Preloop budget exceeded for dev@example.com: ..."}}

retry-after is whole seconds until the budget period resets. Rate limits return 429 with "type": "rate_limit_error". The account kill switch, allowed_models denials, model authorization denials (model_not_authorized) and content policy denials (content_policy_denied, including require_approval holds that were not approved) return 429 with "type": "permission_error" and x-should-retry: false on this path too, so failover cannot route around them. Requests without a trusted upstream identity get 429 for budget denials too, and keep 403 for policy denials (see Status codes). POST /anthropic/v1/messages/count_tokens is not budgeted, but it honours the kill switch: a halted account forwards nothing upstream. A developer whose IdP sends no email gets the same 429, but the apps gateway treats it as a capacity error and fails over; that cannot be fixed on Preloop's side.

Passthrough. On the byte-faithful subscription OAuth path, anthropic-version, anthropic-beta (any value) and every other anthropic-* request header are forwarded verbatim, as are unknown body fields and the system array. Models with a provider API key still go through the gateway's translation layer. Streams stay text/event-stream. Upstream error bodies and the anthropic-ratelimit-unified-* and x-should-retry response headers are relayed to the client.

What Gets Recorded

Every gateway request can persist:

  • token usage and estimated cost in ApiUsage
  • provider, model alias, managed agent, runtime session, flow, and API key attribution
  • normalized model_gateway_call events with redaction-aware payload capture (see Redaction for what is stored and how to store less)
  • provider-neutral conversation previews for operator review

The console Cost, Audit > Sessions and Models views read from this ledger.

A rejected API key marks its model

When a provider answers 401 for a model that uses your own API key, the gateway records the provider, its sanitized sentence and the first and latest rejection times on the stored credential. The model reads credentials_status: error with credentials_last_error_code: upstream_auth_rejected. An upstream billing 402 records upstream_billing_rejected instead. The API, CLI and Models view expose the same status and detail.

The first rejection in an episode creates an audit event (ai_model.credential_rejected), queues an owner email (subject to each owner's email preference) and emits the account webhook event of the same name to subscribed endpoints. Repeated rejections send no further notice until acceptance or key replacement clears the mark. The notice names the model, provider, sanitized provider sentence and first rejection time, and explains that the provider rejected the model's key, not the owner's Preloop login. For authentication, replace or rotate the API key for this model; for billing, add credits or replace the key. Email delivery is best effort and deferred off the gateway request; delivery failure does not re-arm the episode.

A flow selecting a model marked with an authentication rejection tests that stored key before creating an execution, clone lease or container. OpenRouter uses its free authenticated key endpoint; other providers receive a one-token completion, with no retries and a five-second timeout. Probe calls are metered and audited through the gateway. A successful probe clears the mark and starts the run. A fresh authentication rejection refuses the trigger with HTTP 409, code model_credentials_rejected, and the model name and sanitized provider sentence. Network errors, timeouts, 429 and 5xx do not refuse the run. Healthy models receive no extra probe, query or write on their gateway success path. Billing marks never refuse runs because credits can be added at the provider.

A mark remains visible until an accepted request or key replacement/rotation clears it. Subscription OAuth logins keep their existing refresh-failure tracking; 403 answers, system models and deployment-provided hosted keys are not tracked. No key material is included in notices, audit entries or webhook payloads.

Every flow response (GET /api/v1/flows, GET /api/v1/flows/summary, GET /api/v1/flows/{id}) carries model_health for the flow's own model: {"status": "error", "code": "upstream_auth_rejected", "model_id": ...} while that model's credential is marked, {"status": "missing", "code": "model_deleted", ...} after a forced delete removed it, ok otherwise, and null for a flow with no model. One list call answers "which of my flows will fail". The Flows list shows a quiet Model key failing or Model deleted chip on those rows.

What uses a model

GET /api/v1/ai-models/{model_id}/used-by lists what is configured to use a model, independent of traffic:

  • agents: managed agents bound to the model.
  • flows: flows that select it as their own model, in a routing or label rule, or as the retry model, enabled or disabled, with is_enabled, last_execution_at and last_execution_status.
  • ci_identities: restricted CI identities bound to those flows, with is_active, active_key_count and the soonest key_expires_at. No key material.
  • shared_credential_models: other models of the account that use the same stored credential, with their credentials_status.
  • enabled_flow_count and disabled_flow_count: how many listed flows can run, and how many are disabled. The console header reads "2 enabled flows · 1 disabled".

Only resources the caller can see are listed; restricted CI keys cannot call it. The model page in the console shows the same list.

For account admins and the model's editors, the single-model read (GET /api/v1/ai-models/{model_id}) adds credentials_expires_at: when a stored subscription login (Claude Code, Codex) expires. It is null for API keys and for other readers. credentials_last_verified_at is when Preloop last wrote the credential, on import, CLI push or server-side refresh. The model page shows both next to Upstream credentials. No token material is returned.

Rotating a credential (POST /api/v1/ai-models/{model_id}/credentials/rotate) returns affected_models (every model sharing the secret) and affected_flows (the flows that select any of them).

Before you retire a model or key

  1. Read GET /api/v1/ai-models/{model_id}/used-by. Repoint every listed flow, and every CI identity's flow, at the replacement model.
  2. If the key is shared (shared_credential_models is not empty), rotate it rather than removing it, and check affected_flows in the rotation response.
  3. Delete the model. While any enabled flow of the account still selects it, DELETE /api/v1/ai-models/{model_id} answers 409 with {"detail": {"code": "model_in_use", "flow_count", "disabled_flow_count", "flows"}} and changes nothing. Disabled flows cannot fail now, so they do not block the delete, but they still lose the model. ?force=true deletes anyway. Either way, the flows that had it as their own model are left without one, report model_health.status: missing, and the audit entry for the delete lists every one of them, enabled or disabled, under orphaned_flows. Routing, label and retry rules that named the model are listed in the audit entry too but are not rewritten.

Which models an agent sees

A managed agent's credential is offered exactly the models that belong to it, on GET /openai/v1/models, GET /anthropic/v1/models and in the enumeration of a model_not_authorized error:

  1. Explicit allowlist first. When the agent (or its API key, flow or gateway subject) has allowed_models set — the checked models under Models & spend on the agent page — that list is what the agent sees and may call.
  2. Otherwise its configured models. With no allowlist, an agent that has configured models — the models preloop agents onboard bound for it plus any sibling the gateway registered when the agent picked a model in its own picker — sees those and nothing else. Account-wide BYOK and Preloop-hosted models are not offered to it just because they exist in the account.
  3. Agents with no configured models (custom runtimes enrolled through a runtime session token, MCP-only enrollments), plain API keys and console user tokens keep seeing every BYOK, API-key and hosted model in the account.

Subscription OAuth models (Claude Code, Codex) are additionally bound to one managed agent and never appear for any other credential, whatever the allowlist says. A request for a model that exists in the account but is outside the agent's set returns 400 model_not_authorized; the message says whether the model belongs to another agent's subscription or is simply not configured for this agent, and lists the models the credential can use.

Budget and Allowed Models

Before dispatching upstream, Preloop evaluates:

  1. Account-level gateway budgets
  2. Flow-level budgets when the call originates from a flow execution
  3. Subject-scoped allowed-model lists on the active API key or managed agent

Soft limits can annotate responses; hard limits deny the call before it reaches the provider.

OSS includes account, flow, API-key and managed-agent budgets with soft and hard limits, plus spend summaries and budget-health alerts.

Cloud and Enterprise

Per-user and per-team budgets, soft and hard limit notifications, and negotiated price overrides come from the RBAC and billing plugins. See Enterprise Billing & FinOps.

Status codes

Every gateway router (/openai/v1, /anthropic/v1, /gemini/v1beta) uses one status per condition, with the body in the router's own provider format.

Condition Status Anthropic error.type OpenAI error.type / error.code Gemini error.status Headers
Budget hard limit (account, flow, user, team, API key, gateway subject, per-model, per-execution ceiling, trial or free hosted-model allowance, pricing required for enforcement) 429 billing_error insufficient_quota / insufficient_quota RESOURCE_EXHAUSTED retry-after: <seconds>, x-should-retry: false
Rate limit (Preloop or upstream) 429 rate_limit_error rate_limit_error / upstream value RESOURCE_EXHAUSTED retry-after when known; never x-should-retry: false
Policy denial (kill switch, model allowlist, model authorization, content policy) 403 permission_error permission_error PERMISSION_DENIED none
Hosted credit exhausted (Cloud and Enterprise hosted billing) 402 unchanged unchanged unchanged unchanged

Details for a budget denial:

  • retry-after is an integer number of seconds until the budget window resets, or 3600 when the reset is unknown.
  • x-should-retry: false tells the Anthropic and OpenAI SDKs not to retry. Claude Code also stops retrying when retry-after is above 60 seconds.
  • The OpenAI body keeps the Preloop machine code in error.preloop_code (budget_limit_exceeded or execution_budget_exceeded).
  • Anthropic and Gemini messages keep the Model gateway budget exceeded: ... prefix (the per-execution ceiling says Execution budget exceeded: ...).
  • The Gemini body adds google.rpc.ErrorInfo (domain: preloop.ai) and google.rpc.RetryInfo details. Gemini CLI treats a retryDelay above five minutes as a terminal quota error; a shorter one makes it wait for the window to reset before it retries.
  • Usage and audit rows record status 429, audit error_type budget_limit_exceeded and error_class budget_exceeded. Budget denials do not count toward the rate limit report.

Requests from a trusted upstream (the Claude apps gateway) that carry a developer identity also get 429 for policy denials, because the apps gateway fails over to its next upstream on 403.

Migrating from 403 budget denials

Before this change a budget hard limit returned 403 (permission_error), the same status as a policy denial. Integrations that match 403 to detect an exhausted budget must match 429 with billing_error (Anthropic), insufficient_quota (OpenAI) or RESOURCE_EXHAUSTED (Gemini) instead. To tell a Preloop budget denial from an upstream rate limit, check x-should-retry: false, error.preloop_code on the OpenAI router, or the Model gateway budget exceeded message prefix.

Who produced an error

An error from the gateway can come from Preloop itself (for example a missing or rejected Preloop key) or from the upstream model provider (for example the provider rejecting the credential configured for the model). The HTTP status is the same in both cases, so the error body says which one it was:

Field Meaning
origin gateway when Preloop decided, upstream when the model provider did
provider The upstream provider configured for the model, for example openrouter
category The upstream error class, the same value as the X-Preloop-Error-Class header
upstream_status The HTTP status the provider reported
request_id The gateway request id recorded on the run's usage row
cause On a 424 repeated_upstream_failure: the first failure of the run (origin, provider, category, upstream_status, message, request_id), so a later error never stands in for the original cause

The OpenAI and Anthropic routers put these keys inside error, next to message, and in the error / response.failed streaming events. The Gemini router carries them as google.rpc.ErrorInfo metadata (string values, domain: preloop.ai) in error.details.

A provider-reported 401, 402, 403 or 404 names the provider in the message and quotes its own sentence unchanged, for example:

Upstream provider openrouter rejected the provider credential configured for this model (HTTP 401). Provider message: User not found. This is not a rejection of your Preloop credentials.

The provider's wording is passed through as the provider's message; it does not by itself say which account or key the provider evaluated.

Managed Agent Onboarding

When you run preloop agents onboard openclaw or preloop agents onboard hermes, the CLI attempts to rewrite local model configuration so traffic targets the Preloop gateway URL instead of direct provider endpoints. MCP configuration is narrowed to the managed Preloop MCP entry at the same time.

If model rewriting is unsupported for a particular runtime build, MCP governance can still succeed while model traffic needs a manual follow-up edit.

Latency

Measured hop vs calling the model directly (self-hosted OSS, latency ping, not a soak): Gateway overhead. HTTP to the gateway port is the Preloop process. Public HTTPS adds the reverse proxy and TLS.