Skip to content

Session Optimization

After this page you can take a runtime session that burned more tokens than it should have, get concrete recommendations for it, apply one with a single click, and verify the saving with a measured replay — all on the open-source stack, using your own model keys.

Session optimization shipped in the open-source core in 0.12.0. The session_optimization feature is always advertised by GET /api/v1/features, and the Optimize tab is visible in every console.

The loop

Optimization is a three-step loop on one runtime session:

1. Analyze   POST /api/v1/billing/cost/runtime-sessions/{id}/optimizations
2. Apply     POST /api/v1/billing/cost/runtime-sessions/{id}/optimizations/apply
3. Verify    POST /api/v1/billing/cost/runtime-sessions/{id}/replay

In the console, open Audit → Sessions, pick a session, and switch to the Optimize tab — it drives the same three endpoints.

The Optimize tab in the session observer showing an optimization suggestion with evidence
The Optimize tab: suggestions with evidence, a context profile, and the applied-action history for one session.

1. Analyze

POST /api/v1/billing/cost/runtime-sessions/{runtime_session_id}/optimizations (requires the view_cost permission) inspects the session's captured gateway traffic and produces a context profile — cache behavior, retry waste, oversized tool outputs, tools that were advertised but never invoked — plus actionable suggestions.

Two analysis modes, chosen automatically:

  • Deterministic — always available. Pure analysis of the recorded events; the response reports "generated_by": "local". Costs nothing, needs no model.
  • LLM-assisted — when your account has a default active AI model configured (your own key), the service additionally runs an evidence-grounded model pass. That internal call goes through your own gateway with purpose="session_optimization", so it is budgeted and attributed like any other spend — charged to the account, not to the session being analyzed. You can pin a specific model by passing model_id.

If no model is configured, analysis does not fail — you get the deterministic result. If the model call errors out or hits the daily cap, the response falls back to deterministic and tells you why in llm_skipped_reason.

2. Apply

POST .../optimizations/apply (requires edit_ai_models) applies a recommended action in one click. Applying writes governance and budget configuration only — it never spends model budget itself. Applied actions are listed at GET .../optimizations/actions.

3. Replay-verify

POST .../replay (requires edit_ai_models) re-runs the session's recorded requests with a candidate optimization — for example, with specific tools removed or tool-output fields filtered — and measures the difference instead of estimating it:

{
  "consented": true,
  "n_runs": 3,
  "candidate": {
    "removed_tool_names": ["search"],
    "filtered_output_fields": {}
  }
}
  • consented: true is required — a replay spends real model budget to measure real savings, so the API refuses to run without explicit consent (HTTP 400 otherwise).
  • n_runs is 1–10 (default 3); more runs tighten the confidence band.
  • The response reports input_delta_tokens, input_pct_saved, a banded end-to-end delta (median/low/high), the cost_spent on the replay itself, and an inconclusive flag when the band straddles zero.

BYOK vs hosted analysis — the honest version

The split is simple:

Analysis Who pays for the model Availability
Deterministic Nobody — no model call Always, every edition
LLM-assisted on your own model (BYOK) You, through your own gateway budget Always, every edition
LLM-assisted on a hosted built-in model The operator (Preloop Cloud) Metered on Cloud

Self-hosted open source never gates anything: there is no hosted model to meter. On Preloop Cloud, hosted-model analysis is compute the operator pays for, so it can be metered — the server exposes an authorizer hook (optimization_gating) that a billing plugin can register; a denial comes back as HTTP 402 at request time. The UI is never hidden and BYOK analysis is never touched by that hook.

What optimization cannot do: subscription OAuth traffic

If you route Claude Code Pro/Max subscription traffic through the gateway (see AI Model Gateway), that traffic is proxied byte-faithfully — Anthropic requires the exact Claude Code request shape, so Preloop forwards it untouched apart from governance tool-stripping.

Consequences for optimization, stated plainly:

  • Message-level context optimizations are not applied to subscription-OAuth requests. Rewriting blocks would break byte-fidelity and destroy the prompt-cache prefix, so the gateway deliberately leaves them alone.
  • Analysis still works — the traffic is recorded, attributed, and budgeted, so the Optimize tab can still profile it and recommend changes you make on the agent side.
  • Replay-verify candidates that rewrite message content do not apply to this traffic either.

Try it

TOKEN=...   # a console user's bearer token
SESSION=... # a runtime session id from GET /api/v1/runtime-sessions

curl -X POST \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  http://localhost:8000/api/v1/billing/cost/runtime-sessions/$SESSION/optimizations \
  -d '{}'

On a fresh install with no AI model configured you will get a "generated_by": "local" deterministic result — that is the expected baseline, not an error.