Event-Driven Agentic Flows

Editions: OSS. Contributor documentation for this repository.

This chapter covers the Flow subsystem: the Trigger Service, Flow Orchestrator, NATS queue, agent infrastructure, and data flows. Remote runners, matrix/batch fan-out, eval result artifacts, and evidence packs are included here.

Private runners are the default. resolve_runner_pool picks a pool in this order: trigger _runner override, flow.runner_pool, account.default_runner_pool, any online private runner (auto), then the hosted executor. The literal server at any explicit level opts into hosted compute. A resolved private pool uses RemoteRunnerExecutor to lease work to a matching self-hosted runner. An explicit pool can name a runner by id, name or label, but a personal runner (registered by a member who is not an account admin) only takes the lease when the flow author may use it for flows: its registrant, an account admin, or a runner:flow_use grant from the authorizer, the same rule as pinning agent_config.runner_id (#2176). Without the RBAC plugin every member is an account admin, so the gate only narrows routing when roles are enabled. Admin-registered and registrant-less runners stay routable by pool, and flows with no recorded author (legacy rows, or an author who was hard deleted, since created_by_user_id is set to NULL on delete) keep the runner's flow consent as their gate; a deactivated author is refused. The same gate feeds harness_routing_reason, so a refused runner reports owner_has_no_runner and lets harness_fallback: fallback_server take over. If no runner in that pool has a free slot the job queues for 15 minutes (DEFAULT_QUEUE_TIMEOUT) and then fails. A runner holds min(owner ceiling, what the process reports) executions at once (default 2), one row per lease in flow_runner_assignment, and dispatch prefers the runner with the most free slots. Runners maintain an outbound WebSocket control plane for leases, heartbeats, status, logs, completion, and halt requests; durable lease metadata stays in PostgreSQL so temporary disconnects can recover without persisting short-lived API tokens. The CLI (preloop runner fg) redials that socket on unexpected close and keeps an in-flight Docker job, sending complete/logs after reconnect. Register/connect/disconnect also publish runner_updated on account-updates.{account_id} so the console Runners page updates without a refresh. When agent_config.execution_path is persistent, create_executor_for_execution selects AgentControlExecutor before that pool resolution and delivers the rendered prompt as one audited Agent Control send_message to target_agent_id; see Persistent flow execution.

Private Docker jobs carry a versioned transient launch specification generated by the hosted Codex/OpenCode adapters. Reconnect delivery rebuilds credentials and rejects changed leased configuration; scripts and credential environments are excluded from persisted lease JSON. Docker runs an explicit bootstrap, clears stale reports, and requires both exit zero and a recognized structured result.json verdict. The API checks this again before accepting success. The private protocol exports only the bounded report, not workspace archives. Unsupported harnesses fail explicitly. See the runner image contract.

The default private-runner job is still docker run of the flow image (or a custom image / docker_image). An opt-in host execution profile is a distinct private-only path: the runner advertises named local CLIs and executes a fixed local command under the operator's login. Hosted compute rejects these profiles. Native success requires completion_protocol: host_exec, a recognized structured harness result and exit zero; Docker launch v1 is a separate protocol. Profiles map requested model identifiers to local aliases through model_map. Remote setup commands, custom commands, workspace seeds and isolated publication fail closed. Checkout requires the local profile's allow_checkout; managed publication and continuation are limited to their supported Copilot path. The workspace root controls directory placement, not filesystem isolation. Preloop governance and native CLI permissions both apply, and per-job MCP credentials do not turn the host into a sandbox. Explicit Codex, OpenCode and DeepSeek profiles use locally configured CLIs and initially support fresh flows only. Codex and Claude Code host runs send their gateway requests with a signed per-execution marker (X-Preloop-Execution, delivered as host_exec_gateway), so usage, run limits and gateway_metered are per execution; a run with model output and no gateway request fails gateway_bypassed (see services/host_exec_gateway.py). See native profiles.

Review publication receipts

A pull request review flow can require that its review was delivered (EE #202): git_clone_config.review_publication.required: true on the flow itself (the Pull Request Reviewer preset carries the block for flows created from it; an existing flow without the block is never gated, whatever its source preset says). The MCP update_pull_request tool writes a review_publication_receipt row only after the code host accepted a review (approve, request_changes or comment), bound to the account, the execution named by the calling credential, provider, repository, PR number, the reviewed head SHA when the provider reports one (GitHub commit_id), action and provider review id. Reactions and metadata edits write nothing. At the end of the run FlowExecutionOrchestrator._apply_review_publication_gate looks for a receipt for this execution and the triggering PR and head, and stores the outcome in result.review_publication (published, already_reviewed, unverified (see version skew below) or not_published with a reason). A SUCCEEDED run with not_published becomes FAILED with review_publication_incomplete: <reason> and the publication_missing category; the agent's report and blocker stay in the result. The agent's review_posted claim, the success sentinel and a zero exit code never satisfy the gate. A deliberate no-post run (review_action: null, no blocker) passes as already_reviewed only when an earlier run of the account holds a receipt for the same PR and head. A receipt that fails to persist is logged at error level and the run fails for want of it, unless the same execution's tool activity shows the delivered review; the gate cannot tell a failed write from version skew, so that case also passes as unverified (see version skew below), never as published.

scripts/init_db.py refreshes the global preset catalog after migrations on native, Compose and Helm deployments, without propagating prompt or tool changes to account flows. The requirement is a per-flow opt-in: a catalog refresh never gates existing derived flows (production incident 2026-11-04, where a refresh switched the gate on for every derived flow while the API still ran a build without the receipt-writing tool, and every run failed with no_receipt after delivering its review). To require publication on an existing flow, set review_publication.required: true on that flow once the API build that writes receipts is serving; required: false opts out.

Manual preset and CLI PR runs carry the fetched head in the same provider fields as webhook events. A trigger naming a PR without its head fails with trigger_head_missing; an older-head receipt cannot prove publication for that run. Providers that do not report the reviewed head on their receipt (GitLab and Bitbucket) can prove delivery for the current execution and PR, but cannot satisfy the head-exact already_reviewed skip.

Version skew (#2208). The gate runs in the worker and the receipt is written by the API, so during a rolling deploy (or while a new API image fails to start) a worker can find no receipt for a review the code host accepted. When the execution holds no receipt at all, the gate reads the server-recorded tool activity (crud_runtime_session_activity.list_tool_calls_for_flow_execution, the rows _describe_incomplete_review also reads). A succeeded update_pull_request call whose classified work_ref is a review (action: reviewed) on the triggering repository and PR number yields {"status": "unverified", "reason": "receipt_unavailable", "activity_id", "review_state", "head_sha"}; the run is not failed and a warning is logged. A bare PR number with no repository, a failed or refused call, or a call on another PR does not match. A receipt for another PR or head is still not_published, and activity is not read when any receipt exists. unverified is never reported as published: no surface maps it to a delivered review.

The receipt's repository is the path the agent passed, except that a numeric project id (or any value equal to the resolved project's tracker identifier) is replaced by the resolved project's slug, which is path_with_namespace on GitLab, so it matches the path in the merge request URL (mcp._review_receipt_repository).

Ad-hoc preset runs

POST /api/v1/flows/run-preset runs a catalog preset on a GitHub or GitLab issue without waiting for a tracker webhook. confirm_create=false only resolves the account flow (409 flow_missing when none exists). confirm_create=true clones the preset on first use, then triggers it. The auto-created flow has empty trigger_event_types and no trigger_event_source, so tracker events do not start extra runs until the operator edits the flow. Other tracker types (for example Jira) return 400.

Jira repository binding

A Jira-triggered flow with git clone enabled and no repositories of its own clones the code-host repository bound on the flow or the Jira project, and writes the opened pull request back to the issue as a comment and a remote link. See Jira project repository binding.

Prompt placeholders

Flow prompt_template strings are resolved before the agent starts. Besides {{project.*}}, {{account.*}}, and {{trigger_event.*}}, an execution resolver provides:

• {{execution.id}}: this run's id.
• {{execution.url}}: console URL {PRELOOP_URL}/console/flows/executions/{id}.
• {{execution.resume_from}}: prior execution id when this run was started from a human comment on a PR this flow opened. Empty otherwise.
• {{execution.ci_failure}}: when this run was started because GitHub CI failed on a PR this flow opened: provider, job name, and check URL. Empty otherwise.
• {{flow.review_instructions}}: blocking review rules stored on the flow. Empty when unset. The Pull Request Reviewer reads this the same way it reads .preloop/review-policy.md, and truncates it at 16 KiB.

Trigger-payload free text (a placeholder read from the trigger event whose last segment is title, body, description, note, text, message, comment, summary or content) is not substituted in place. The template keeps a reference such as [see field "trigger_event.payload.issue.title" in the trigger_data block below], and the values follow the template in one <trigger_data> block that tells the agent the contents are data, not instructions. A placeholder that names a whole object or list from the trigger event, such as {{trigger_event}}, {{trigger_event.payload.issue}} or a label array, goes to the block as JSON for the same reason. Tags in a value that could close the block are escaped. Scalar identifiers, numbers, URLs and states are still substituted inline, and a template with no such field gets no block. See backend/preloop/utils/prompt_data.py.

Any placeholder may take a |truncate(N) filter, for example {{trigger_event.payload.object_attributes.description|truncate(16384)}}. N is a byte cap (not a character count) so it matches the launch-payload and tokenizer costs that actually grow. The cut is on a UTF-8 boundary. {{name|truncate}} without N uses 16 KiB. When the resolved value is longer, the injected text is the prefix plus:

[truncated by Preloop: showing the first N bytes of TOTAL; fetch the full text
with the tool that owns this object, for example get_pull_request]

Unbounded webhook fields are capped because they dominate context window, reviewer attention, and the kernel execve string limit (one argv element or one NAME=value entry cannot exceed 128 KiB). Templates without the filter are unchanged. Operator-facing grammar also lives in Webhook Triggers.

Agent launch payload (container environment)

Custom agent images and private runners must not assume the rendered prompt arrives as one environment variable. Linux caps a single execve string (one argv element or one NAME=value entry) at MAX_ARG_STRLEN (131072 bytes). The control plane delivers large text as base64 chunks and reassembles files inside the container:

Variable	Meaning
PRELOOP_AGENT_PROMPT_0..N	Base64 chunks of the rendered prompt (96 KiB each).
PRELOOP_AGENT_PROMPT_CHUNKS	Chunk count (0 when the prompt is empty).
PRELOOP_AGENT_PROMPT_BYTES	Decoded byte length, used to abort a truncated reassembly.
AGENT_PROMPT_FILE	Path the chunks materialize at (/tmp/preloop/prompt.txt).
AGENT_PROMPT	Whole prompt, only when it is <= 64 KiB. Absent above that.
PRELOOP_INNER_SCRIPT_0..N	Base64 chunks of the Kubernetes inner agent script.
PRELOOP_INNER_SCRIPT_CHUNKS / PRELOOP_INNER_SCRIPT_BYTES	Count and decoded size for that script.
PRELOOP_INNER_SCRIPT	Legacy whole-script value. Still honoured by the artifact wrapper so an old control plane can drive a new image.

A new control plane can drive an old image: small prompts still set AGENT_PROMPT, and the wrapper still accepts a whole PRELOOP_INNER_SCRIPT. An old control plane can drive a new image the same way. Custom images that read the prompt themselves should prefer AGENT_PROMPT_FILE after the launch script's materialization block, or reassemble PRELOOP_AGENT_PROMPT_* the same way (base64 -d, then wc -c against _BYTES). Do not require AGENT_PROMPT for prompts above 64 KiB.

OpenHands (the default agent_type) uses this prompt transport on Docker and Kubernetes. Gemini and OpenCode feed the materialized file to the CLI on stdin, so the prompt is not an argv element (issue #692). Residuals that still expand the file into argv are aider (--message "$(cat ...)"), OpenHands (-t "$(cat ...)"), and Codex nudge/recovery ("$(cat ...)" as a positional). Those are not part of this contract.

Model stream recovery

Hosted Docker/Kubernetes agent scripts and supported private Docker launches keep provider retries separate from session recovery. Codex uses four request retries and five stream retries (explicit for the gateway provider, matching its native defaults); OpenCode 1.18.29 already retries a model request up to five times; Gemini enables transport retries with four total chat-model attempts. After a transient CLI failure, the container can resume the captured parent conversation twice, with two- and four-second backoffs and a 600-second timeout per resume (forced termination after another five seconds). The execution's overall timeout still applies. These retry layers can multiply upstream attempts; they are bounded and do not guarantee recovery from a sustained outage.

Recovery requires an explicit captured session identifier. It does not start another conversation if capture fails, and it skips cancellation, terminal authorization/quota failures, and a terminal completion report written during this invocation. A content/stat fingerprint captured before the CLI starts preserves old result.json evidence from restored workspaces without mistaking it for completion of the new turn. The continuation prompt preserves conversation/tool results and asks the agent to inspect uncertain external actions before writing again. Initialization and the container's publication block execute once; model-controlled effects are not guaranteed exactly once. PRELOOP_STREAM_RECOVERY, PRELOOP_STREAM_RECOVERY_RESULT, and PRELOOP_STREAM_RECOVERY_UNAVAILABLE remain visible in agent logs. OpenCode/Gemini terminal JSON errors override a misleading zero CLI exit; truncated output without a terminal completion cannot count as success.

Private native-host execution also accepts explicit Codex, OpenCode and DeepSeek CLI (dsh) profiles. These new adapters support fresh flows only; native resume and publication are unavailable. Their existing Docker defaults remain available without a host profile. See native profiles. Gemini is not currently a supported private Docker runner harness.

PR-comment resume

create_pull_request (and GitLab merge-request creation) records the opened HTML URL and source branch on flow_execution.result (flow_pr_binding). When a flow listens to both issue events (issue_labeled or issue_opened) and comment_created, a later human comment on that PR starts a new execution of the same flow with _resume in the trigger payload. The container clones and pushes the existing PR branch. Unmatched comments do not start a run. Native CLI --resume is a separate follow-up.

Wrapper-opened PRs bind too. Not every PR comes from the MCP tool: when git_clone_config.create_pull_request is set, the container's post-execution step opens the PR itself with a plain curl. Title and body are JSON-encoded in the container (json.dumps) so quotes and newlines in a commit message or a multi-line pull_request_description cannot produce a 400. The wrapper prefers pr_title / pr_body from /workspace/result.json, then interpolated git-config fields, then a flow-attribution fallback: an execution link plus the last commit body, or [Preloop] {flow_name} with a **Commits:** list when more than one commit landed. The create response is written to /workspace/evidence/pr.json, with a lookup by head branch (GitHub) or source branch (GitLab) when the PR already existed, and one log line PRELOOP_PR_OPENED {"url": ..., "branch": ..., "provider": ...} per publication. The orchestrator reads every line off the log stream and rescans the output summary after reconnects. It retains the first URL in result.pr_url for existing follow-up behavior and records every distinct wrapper publication in result.published_prs with URL, source branch, provider, and repository. The execution view links each publication; a second repository's PR does not implicitly gain continuation. The list is capped at 32 entries per execution; published_prs_truncated records an overflow rather than allowing unbounded writes.

Agent decisions. An agent can print PRELOOP_DECISION {"summary": ..., "reason": ..., "alternatives": [...]} on one line for each non-trivial choice it made; the Automated Issue Implementation and Pull Request Reviewer presets ask for one per decision. The orchestrator (live stream and output-summary rescan) and the runner log channel parse the line and append it once to result.decisions. A line that is not a JSON object with a summary and a reason (neither empty nor a placeholder such as "..." or "<why>") is logged and ignored. Text is bounded per field, the list is capped at 50 entries with decisions_truncated on overflow, and both keys survive the agent's own result.json overwrite. Decisions are the agent's claims, kept for reports to cite; nothing branches on them.

Change brief. The Automated Issue Implementation preset writes a <!-- PRELOOP_BRIEF --> ... <!-- /PRELOOP_BRIEF --> block into its PR body, after the description and before the Closes line, with What changed, Why, Assumptions, Not verified, Rollback and Decisions sections; its Decisions bullets restate the run's own PRELOOP_DECISION markers. When the run finishes, the orchestrator parses the block from the body the publisher selects from the agent's result.json (same aliases and validation) and stores it as result.brief, with a missing list of absent sections and a truncated flag. Only the first 60,000 bytes are read, a section keeps at most 50 bullets of up to 1000 characters, and an agent-authored brief key is always dropped. The Pull Request Reviewer's PRELOOP_SUMMARY block adds Not verified and Must know lines after the Overview. Like decisions, a brief is the author's claim.

Implement, review, resume. The PR reviewer flow posts as the Preloop bot, and bot-sent events are dropped by the loop guard, so its findings never reached the implementer. A comment_created event from a known Preloop identity is now allowed through that guard only when the body carries the reviewer marker <!-- preloop-review:flow-id:<id>[:severity:<S>] -->; every other bot comment stays dropped. Three guards bound the resulting loop:

• Self-loop: a comment whose marker flow id names the receiving flow (its UUID or its name slug) never resumes it, so a flow cannot restart itself on its own output.
• Cap: max_resumes_per_pr (flow-level, agent_config.max_resumes_per_pr, default 5) limits how many resumes one PR can start. The running count lives on the opening execution's result (resume_count) and each resume carries its resume_index; comments past the cap are logged and skipped.
• Queue one: a correlated comment that arrives while a run for the same PR is still going does not start a competing execution. It sets pending_followup (with pending_followup_comment_url) on that running execution; further comments during the same run are coalesced onto the same flag. When the run reaches a terminal status (succeeded or failed), the orchestrator clears the flag and starts exactly one follow-up resume for the PR.

CI-failure resume

Failing GitHub check_run, check_suite, and workflow_run events on a PR this flow opened resume the same way as a review comment, with {{execution.ci_failure}} filled in. The skip/resume filter is GitHub-only: existing GitLab pipeline / job flows keep firing on every event. GitLab payload extractors stay in flow_ci_feedback so an opt-in can reuse them later.

Matrix / Batch Fan-Out

One flow definition can drive an agent-harness × model evaluation grid without cloning the flow per combination:

• Trigger: POST /api/v1/flows/{flow_id}/trigger reserves the top-level matrix key in the trigger body. When present, it must be a list of up to 25 {"agent_type"?, "ai_model_id"?} cells; the trigger fans out to one execution per cell (an empty object {} runs the flow defaults). Validation is all-or-nothing (allowed keys, agent types from the agent factory registry, account-visible models), and all execution rows are committed before any cell is dispatched. Note for existing users: matrix is now a reserved key in trigger bodies, it is stripped before template-variable resolution and never reaches {{trigger_event...}} placeholders; rename any pre-existing matrix field in custom trigger payloads.
• Batch identity: all cells share a batch_id (indexed column on flow_execution). Per-cell overrides are persisted on each execution under the reserved _matrix key of trigger_event_details, making every cell self-describing. All paths that (re)build an agent executor (initial run, resume, background monitor, crash recovery) resolve the effective agent type through resolve_execution_agent_selection (matrix cell, then routing record, then flow default), so an interrupted cell is always handled by its own harness rather than the flow default.
• Observation: GET /api/v1/flows/batches/{batch_id}/executions lists a batch (account-scoped, sorted by matrix index) with a rollup of status counts, tokens, tool calls, and estimated cost, so an eval matrix can be observed as a unit.
• Response shape: non-matrix triggers are wire-identical to before; matrix triggers return batch_id plus per-cell execution references.

Execution lineage

Executions expose parent_execution_id, root_execution_id, and delegation_depth in detail and lightweight list responses. Existing and root runs have null parent/root IDs and depth 0. Later delegation writers supply these values at creation; this storage contract does not start child runs.

The parent is an indexed self-reference with ON DELETE SET NULL, so removing a parent preserves its children. The indexed root ID has no foreign key and remains a grouping label after root deletion; recorded depth is also preserved. A whole tree consists of the root row plus executions whose root ID points to it. CRUDFlowExecution.get_children requires an account, returns direct children only, and orders them by start time then execution ID. The list query loads the lineage fields and existing parking timestamps with the row so serializing a page adds no per-row reads.

Label-based model routing

A flow can optionally store ordered routing rules in agent_config.model_routing (no extra column). Each rule has a stable id, labels.any and/or labels.all against the issue's current labels, and an account-owned ai_model_id plus compatible agent_type. The first matching rule selects the model and harness for that execution. If none match, or the key is absent, the flow's selected model and harness are used. Examples in docs use operator-defined labels such as documentation or bug; there is no built-in taxonomy.

The controller writes the chosen rule or default onto the execution under reserved _model_routing after validating ownership and _model_usable_for_agent. Webhook bodies, tracker payloads, and authenticated trigger JSON cannot supply ai_model_id, _matrix, _model_routing, or _resume as authorized overrides. Retries pin from the persisted source execution (retry_of_execution_id); native continuation pins from a controller-loaded prior execution after account and lineage checks. A live rule or default change does not swap the model mid-conversation. Every new execution records its selected defaults, including flows without rules. Durable feedback copies this identity before reserving a repair turn. A missing legacy identity or changed native selection blocks continuation and requires an explicit new execution; no automatic model handoff occurs. New matrix cells also materialize their effective defaults. Authorized matrix eval cells still win over routing. Account budgets still apply. Operator guide: model routing.

Eval / observe runs (structured result artifact). The Observe / Eval preset (backend/presets/003-observe-eval.yaml) establishes a run → capture → serve contract: the agent writes its final report to /workspace/result.json (preloop.eval.result/v1: status, summary, metrics, checks, optional artifacts), the orchestrator captures it first-class on every terminal path (success, failure, stop, timeout) via the Docker archive API (no log scraping or sentinels) and persists it to flow_execution.result (JSONB, size-capped; malformed or unfetchable artifacts are recorded as wrapped {"error": ...} objects so failures stay visible). It is served on GET /api/v1/flows/executions/{id} and GET /api/v1/flows/executions/{id}/result (404 when the run reported nothing); list payloads exclude it. The artifact is also an active success-confirmation channel alongside FLOW_EXECUTION_SUCCESS: success confirms the flow; error, failed, and failure fail it; and eval fail remains a subject verdict rather than a flow failure. Preloop transports and meters the artifact; scoring is owned by the customer's own verifier. On the Kubernetes backend a completed pod's filesystem is unreachable through the API, so the agent script is wrapped to emit result.json (and the evidence pack, see below) into the pod's log stream as base64 between structured PRELOOP_ARTIFACT_* marker lines; the runner parses the emission back out of the pod log with the same validation and error surfaces as the Docker archive path, and the marker lines are filtered out of operator-facing logs and summaries.

Completion-confirmation recovery ladder. An exit-0 run must confirm success through one of the two channels above (the exact-line FLOW_EXECUTION_SUCCESS sentinel, or a recognized result.json status/verdict); that requirement is unchanged for generic flows. When neither channel fired, the agent script has usually already reminded itself in place (see below) and the orchestrator walks a recovery ladder before failing closed: (1) post-exit log rescan: it refetches the complete runtime logs (docker logs / K8s pod logs, no tail limit) and rescans for the sentinel, armed at the PRELOOP_AGENT_EXEC_START marker, covering the edge where a late stream reconnect loses the log tail (logged as the post_exit_log_rescan milestone); (2) confirmation nudge: it re-invokes the agent once, at most one round per execution across retry attempts, with a minimal follow-up prompt (head of the original prompt plus a quoted output tail) asking it to confirm via the originally instructed channel or state failure plainly (logged as the confirmation_nudge_used milestone with its outcome). The nudge is side-effect free (git_clone_config and custom_commands are stripped) and only runs on runtimes that opt in via AgentExecutor.supports_confirmation_nudge (currently codex and opencode); all other runtimes skip straight to fail-closed. Two settings bound the round: FLOW_CONFIRMATION_NUDGE_MAX_TOKENS (default 4096) is a hard output-token ceiling (a larger limit inherited from the flow's model is clamped down; a tighter one is respected) and also bounds the embedded prior-context excerpt at ~4 chars/token, and FLOW_CONFIRMATION_NUDGE_TIMEOUT_SECONDS (default 300) bounds its wall clock. If the ladder ends without confirmation, or the nudge states failure, the execution is marked FAILED with the existing missing-confirmation message, which says so explicitly when the agent was already reminded in its own container.

In-place completion nudge (the cheap rung). no_confirmation was the largest single failure class on staging (33 of 100 failed executions, median duration 0.8 minutes): runs that exited 0 and simply never used a completion channel. The cheapest sound way to get the confirmation is to ask the agent that is still standing there, so the generated agent script now checks the completion contract itself and, when nothing confirmed it, re-invokes the SAME harness on the SAME session in the SAME container and workspace (opencode run --continue, codex exec resume --last) with a short reminder: do not redo or extend the task, do not take new side effects, write /workspace/result.json and print the sentinel for success, or report the failure plainly, then stop. The reminder's output is judged by exactly the same contract, because it lands on the same stdout and writes the same result artifact. Safety comes from placement rather than policy: the block is emitted before the container's post-execution git block and runs only when the harness exited 0, so a nudge can never re-run a push or a PR creation; it runs at most once; and its exit code is discarded, so a failed reminder leaves the run exactly where it was. Three marker lines make the round observable (PRELOOP_COMPLETION_NUDGE, PRELOOP_COMPLETION_NUDGE_RESULT exit=<n>, PRELOOP_COMPLETION_NUDGE_UNSUPPORTED <runtime>); the orchestrator reads them from the live stream and again from the post-exit log refetch, publishes them as the completion_nudge and completion_nudge_result timeline events, and then stands down from its own second-session nudge (skipped_inplace_nudge_used), which would be a second model call for the same answer from a session that knows strictly less. It also stands down when any action is already on the timeline (skipped_actions_recorded): a re-invoked model that decides to finish the job would repeat that posted comment. Runtimes that opt in declare AgentExecutor.supports_inplace_completion_nudge (currently codex and opencode). Settings: FLOW_COMPLETION_NUDGE_ENABLED (default true, flowExecution.completionNudge.enabled) is the fleet-wide kill switch, FLOW_COMPLETION_NUDGE_TIMEOUT_SECONDS (default 300, floor 30) bounds the round.

Wider completion signals for runtimes that cannot be resumed. Gemini, Aider, OpenHands and remote runners have no resume mechanism, so there is nobody left to ask. For those (and only those: the check is a strict identity test against supports_inplace_completion_nudge is False, so an unknown capability keeps failing closed) a non-empty result.json that the agent actually wrote is accepted as the completion signal even when its vocabulary is not one Preloop recognizes, logged as the completion_signal_accepted milestone with signal: result_artifact_present. An explicit failure status is decided earlier and still fails the run, and a runtime that was asked directly and declined to confirm never reaches this rung.

No-progress guard (a run that never edits anything). Some implementation runs read, plan, summarize and end with an empty workspace, having spent the full budget on nothing. Two rungs catch that, both driven by one piece of evidence: the workspace itself. (1) Live, on the poll loop: when agent_config.no_progress_after_seconds is set and that many seconds have passed with the checkout provably clean (git status --porcelain empty and no local commit ahead of the remote, probed in the running container at most once a minute), the orchestrator delivers ONE short reminder into the session that is still running, through the same resume mechanism the in-place completion nudge uses (codex exec resume --last, started detached so the loop is not blocked), logged as the no_progress_nudge milestone and published as a timeline event. After a further no_progress_grace_seconds (default 600) still clean, the run is stopped and classified agent_no_progress (no_progress_stop). A probe that cannot answer (Kubernetes, an unreadable checkout) returns "no answer" and never "clean": a stop always rests on a probe that actually ran, and a run that has any diff is never nudged and never stopped by this guard. Runtimes without an in-place resume get the stop without the reminder. The key is null by default, which disables both rungs. (2) Terminal: when the agent itself reports failure through result.json and the container's post-execution git block found no commit to push (PRELOOP_NO_COMMITS <branch>), the failure is classified agent_no_progress rather than the generic agent error, so "tried and could not" is separable from "never started". agent_config.retry_on_no_progress ({"enabled": true, "ai_model_id": ..., "reasoning_effort": ...}, off by default) then creates exactly one retry through the normal manual-retry path with retry_of_execution_id set, optionally escalating to a stronger model; a retry is never itself retried.

Per-flow timeout budget. flow.timeout_seconds (nullable, 60..86400, on the create/update API and settable from a preset YAML) is the wall-clock budget for one execution of that flow; NULL keeps the deployment default FLOW_EXECUTION_MAX_WAIT_SECONDS (3600, flowExecution.maxWaitSeconds). The orchestrator resolves and clamps it once per run, records it on the agent_monitoring_started milestone (timeout_seconds, timeout_source), enforces it in the monitor loop, and on expiry writes agent_execution_timeout and a message that names the budget that ran out ("this flow's timeout budget" vs "the default timeout budget"), still classified timeout. This is what separates "genuinely long" from "stuck": with one global ceiling, all 7 staging timeouts sat on exactly 3600 seconds and said nothing about which was which. A PR review that should finish in minutes (002-pull-request-reviewer.yaml, 1800) and a release audit that legitimately runs for hours (006-release-security-audit.yaml, 7200) now carry their own budgets.

The budget is one deadline from launch. The deadline is launch_requested_at (the moment admission lets the runtime start; start_time when it is missing) plus the budget, and the monitor checks it against the wall clock on every poll. Everything after admission counts: starting the container or Job (image pull, scheduling), time spent inside status, log and metrics calls, every retry attempt and the backoff between them. A retry is not started when the remaining budget cannot cover its backoff plus 30 seconds (execution_retry_skipped_deadline milestone), and a worker that resumes monitoring after a handoff reads the same anchor from the row, so nothing restarts the clock. agent_monitoring_started records the deadline and the elapsed_since_launch at that point, agent_status events carry elapsed as wall-clock seconds since launch, and on expiry agent_execution_timeout carries elapsed_seconds and the message reports the real seconds ("timed out after 374 seconds (budget 300 seconds from launch)") when they differ from the budget. Context preparation before admission (clone, custom commands) is not in the budget. Runtime backstop: a Kubernetes agent Job gets activeDeadlineSeconds = remaining budget + 300 seconds of grace, so the Job ends even if no orchestrator is left to stop it; the grace leaves the monitor time to stop the Job itself and keep the logs and result artifact. Docker containers and private runners have no runtime-side deadline and rely on the monitor. The completion-confirmation nudge still runs on its own timer after the agent exits; its Kubernetes Job backstop is that nudge timeout plus the same 300 seconds of grace, not the main run's remaining budget.

Stream idle bound. A custom-provider Codex call (including the Preloop gateway) waits stream_idle_timeout_ms for the next byte of a streamed response before it treats the stream as dropped, then reconnects up to stream_max_retries (5) times inside the same call. agent_config.stream_idle_timeout_seconds (30..3600, validated on create/update) sets that wait per flow; unset keeps 600. Either way preloop.services.stream_stall.resolve_stream_idle_timeout_seconds caps it at half the run's timeout budget (the orchestrator passes flow_timeout_seconds in the execution context), because a wait as long as the budget can never fire: the run is stopped before Codex reconnects and the log never names the stall. The cap only changes an unconfigured flow when its budget is under 1200 seconds (a 900 second flow gets 450). Native OpenAI writes no provider block and keeps Codex's built-in wait. The Codex script also sets RUST_LOG so codex_core::responses_retry logs the reason for each reconnect (idle timeout waiting for SSE); without it the Reconnecting... n/m lines carry no reason. When a run times out and its log shows an idle signal with no model output (codex, exec, thinking, tokens used) after the last one, the message says it timed out "while waiting on a silent model stream", failure_category is model_stream_idle, result.stream_stall carries the evidence (idle_reconnects, retries_exhausted, stream_idle_timeout_seconds, last_signal), and an agent_stream_stalled milestone is written. A run that stalled, recovered and then ran long stays timeout (issue #872).

Hosted-monitor concurrency. Each flow-execution worker process may babysit FLOW_EXECUTION_MAX_INFLIGHT hosted monitors at once (default 10, flowExecution.maxInflight). The cap is a process-wide semaphore shared by execute_flow and resume_flow_execution. Other worker pools stay serial: they were not sized for fan-out. Helm injects a dedicated flowExecution.databasePool (size 10 / overflow 4) on the flow-execution pool so those short-lived sessions are not squeezed through the generic worker 2+4 pool.

Per-account admission cap. Admission is also bounded per account, so one account cannot fill every worker. FLOW_EXECUTION_MAX_RUNNING_PER_ACCOUNT (default 5, Helm flowExecution.maxRunningPerAccount) is enforced inside claim_execution. The cap bounds hosted compute only: an execution assigned to one of the account's private runners is bounded by that runner's own capacity, so it is neither counted nor held back. An account may override the cap through account.meta_data["flow_execution_max_running_per_account"]. A refused execution stays PENDING with queued_reason=account_concurrency_cap and its message is nacked for later redelivery. Independently, a flow gets at most one active execution per tracker object (issue, pull request, or merge request): a later matching trigger is recorded as skipped rather than queued. Comment and CI deliveries are exempt so they can still feed a live run. A labels condition on a labeled/unlabeled delivery tests the label the event carries, not the issue's full label list.

Stale-claim reaper. Every FLOW_EXECUTION_RECLAIM_INTERVAL_SECONDS (default 30, Helm flowExecution.reclaimIntervalSeconds) the reaper re-publishes active executions that are unclaimed or whose claim heartbeat went stale. That is the deploy-handoff safety net: an execution whose owning worker died is adopted by another worker inside one stale window. Three bounds keep it from becoming a storm, because "nobody has claimed this" and "nobody has a free slot for this" look identical from the database. (1) The pass runs under a session-level advisory lock (pg_try_advisory_lock) pinned to one connection for the whole pass, so one replica reaps per interval instead of all of them; losing the lease is not an error, the loser skips the pass and Postgres releases the lock if the holder's connection dies. (2) Each execution carries redispatch_count and last_redispatch_at: the gap between two re-dispatches of the same execution doubles from the reclaim interval up to FLOW_EXECUTION_REDISPATCH_BACKOFF_MAX_SECONDS (default 900, Helm flowExecution.redispatchBackoffMaxSeconds), so an hour of being unclaimed costs a handful of publishes rather than 120. A successful claim resets both, so a run that queued for an hour and then lost its owner is still recovered at once. (3) Before publishing, the worker asks JetStream whether flow tasks are already queued undelivered; when they are, executions that still need a container are left queued rather than republished into a queue nobody can drain. Executions that already have an agent session bypass the capacity and per-account filters, because an unmonitored container is worse than being one over a limit. Each pass logs exactly one line: candidates, re-dispatched, skipped_backoff, skipped_account_cap, skipped_no_capacity, failed.

Terminal notifications (flow.notifications). A flow can opt into posting a tracker comment when an execution ends, instead of asking the agent to ping the triggering issue from the prompt. on_success.comment_on_trigger_issue posts a short "PR opened: \<url>" comment when the run recorded a pr_url. The orchestrator posts it from the terminal path (including resume), and a notification failure never rewrites the execution status. The console offers the option only where it can apply: the flow opens the pull request itself (git_clone_config.create_pull_request) and the trigger is about an issue (a tracker trigger with an issue_* or comment_* event).

Failed executions always appear as console attention items of kind flow, and there is no failure comment. on_failure.alert_after_consecutive_failures (1..100, default 3) is a safety net: after that many consecutive failed runs the account's owners are emailed and pushed once per streak. The streak is the leading run of failures among the flow's terminal executions — a success resets it, an operator stop is neutral, and a SUCCEEDED run that recorded no output counts as a failure. The marker that suppresses a second alert for the same streak is an audit_log row claimed before delivery under a per-flow transaction-scoped advisory lock, so dedup holds across replicas and a delivery failure cannot become a retry storm; evaluation and delivery run off the orchestrator event loop on a short-lived session. on_failure.comment_on_trigger_issue (removed 2026-09) and on_failure.attention_item are parsed and ignored so flows stored before the removal still load. notifications is a JSONB column, so nothing is migrated. The console has no control for the threshold, but it carries a stored alert_after_consecutive_failures forward on save instead of dropping the on_failure block.

Evidence packs. Audit-style flows (backend/presets/004–007) write a human-readable evidence pack under /workspace/evidence/. Two transports move that pack to the control plane. Direct upload (FLOW_ARTIFACT_DIRECT_UPLOAD) gives hosted containers and private Docker runners an execution-scoped PUT to the encrypted artifact store (kind=evidence). Kubernetes logs then carry status markers only, including when the upload fails: there is no plaintext fallback. The legacy transport is the default. Docker copies the directory through the engine archive API and does not use the log channel, so FLOW_EVIDENCE_LOG_PLAINTEXT does not change Docker capture. Kubernetes, with the plaintext switch left at its default true, emits a size-capped base64 block (MAX_EVIDENCE_ARCHIVE_BYTES, 2 MiB) of result.json, the evidence pack, and the workspace snapshot into the pod log. That default is an exposure window: base64 is not encryption, and anyone who can read retained pod logs can read the artifacts. Set FLOW_EVIDENCE_LOG_PLAINTEXT=false to refuse that channel. Without an upload token the wrapper fails closed (markers unavailable / skipped with reason plaintext_disabled, no artifact bytes). The receipt is failed with error plaintext_disabled, which means unavailable by policy, and result capture reports a missing result rather than a decoded payload. The switch does not cover pod-spec access or ordinary agent stdout. Encrypted log transport with per-execution keys is a separate decision tracked in issue #268. GET .../result and GET .../evidence-status report the persisted receipt (kind=evidence) when bound, else the record the download serves, without decrypt; download verifies digest and tenancy and reports available / missing / expired / failed distinctly. Packs are signed at capture with the account Ed25519 key; the signature lives beside the archive so the content-addressed digest does not change. See evidence-storage.md. This is operational retention, not object-lock.

Workspace persistence (snapshot, restore, retention). An execution that fails after the agent committed used to lose the work: on Docker the named volume agent-workspace-<execution_id> survived by accident with no way to find it, and on Kubernetes the emptyDir died with the pod. Every hosted run now captures a size-capped tar.gz of /workspace on every terminal path, success or failure, next to the evidence pack. .git is kept whole (the unpushed commits are the point) and a git bundle --all of every repository is written into it first as an independent second copy of the commit graph; regenerable caches (node_modules, .venv, __pycache__, and friends) are excluded. The cap is WORKSPACE_SNAPSHOT_MAX_BYTES (default 512 MiB) and it is enforced inside the container: the tar stream is piped through head -c, so an oversized workspace never writes more than the cap to disk, never crosses the runtime boundary, and is skipped with a logged PRELOOP_WORKSPACE_SNAPSHOT_SKIPPED size_exceeds_limit reason. Docker builds the archive in a short-lived helper container mounted on the workspace volume (the agent container has already exited and cannot run tar); Kubernetes emits it through the same PRELOOP_ARTIFACT_* log channel as the evidence pack, which caps it at 2 MiB there, so larger Kubernetes workspaces are reported as skipped until snapshots move to object storage. The archive is stored on flow_execution.workspace_snapshot (BYTEA, excluded from the execution schemas) and served by GET /api/v1/flows/executions/{id}/workspace. A correlated resume (a PR comment on a PR this flow opened) loads the prior execution's snapshot, unpacks it into the new Docker workspace before the entrypoint runs, and the init script then skips the clone and does fetch + checkout + merge --ff-only of the PR branch inside the restored repository, so unpushed commits are continued rather than redone. The decision is made in the container ([ -d <repo>/.git ]), so a restore that did not land silently falls back to the normal clone; Kubernetes always falls back, because an emptyDir cannot be seeded before start. Retention is a decision rather than a leak: an hourly janitor task (cleanup_flow_workspaces) clears snapshots and removes agent-workspace-* Docker volumes older than WORKSPACE_SNAPSHOT_TTL_HOURS (default 24; 0 means delete on the next pass). Private runners keep their workspaces on the operator's own machine and never upload them.

Codebase setup (git_clone_config.setup_commands). An implementation agent that cannot run the real test suite is guessing, and "the DB-backed tests need a Postgres this container does not have" is not an agent failure. setup_commands is a list of shell strings run inside the container after the clone or workspace restore and before the agent starts, from the first repository's clone path. All output is captured to /workspace/evidence/setup.log (inside the evidence pack, so it is retrievable even when the workspace snapshot was too large to keep); the first failing command stops the block, prints PRELOOP_SETUP_FAILED exit=<n> plus the log tail, and fails the execution with the dedicated setup_failed failure category, so a broken environment never reads as a broken agent.

git_clone_config:
  enabled: true
  setup_commands:
    - pip install -r requirements/dev.txt
    - docker run -d --name pg -e POSTGRES_PASSWORD=preloop -p 5432:5432 postgres:16
    - until pg_isready -h 127.0.0.1 -U postgres; do sleep 1; done
    - echo 'DATABASE_URL=postgresql+psycopg://postgres:preloop@127.0.0.1:5432/postgres' > /workspace/.env

Setup commands run in a subshell (set -e; cd <clone path>), so export, cd, and other shell state do not reach the agent process. Persist anything the agent needs in a file (as above) rather than relying on the environment.

Services are the caveat. A docker run inside a setup command needs a runner that is allowed to touch a Docker socket: on a private runner that means the trust flags the operator opts into explicitly, and on the hosted runners it needs a per-execution service sidecar, which is the next phase of this work (a declarative services: block with a Docker network on the Docker runner and pod sidecars on Kubernetes). Until then, keep setup commands to what a plain container can do: install dependencies, build, seed fixtures, start in-process services.

Publication gate (git_clone_config.verification). A prompt that asks an implementation agent to run tests is a request, not a contract: the agent's result.json is a claim about what it did, and the publisher has no way to tell a tested change from an asserted one. The gate turns verification into something the runner owns. A flow opts in by setting verification.mode: "gate" with a trusted test profile: a versioned (v1) mapping of changed-file patterns to required checks (always inexpensive hooks required on every diff, rules matched against the diff against the PR base, and an unknown_default applied to every unmatched changed path so a mixed documentation/code diff cannot hide unknown impact). The profile ships with the flow configuration, never inside the target repository, so the agent cannot narrow required checks or edit the profile to bypass the gate within its own PR. It may propose additional checks through proposed_checks in its result artifact; only the profile decides what is required.

After the agent exits and commits, before the push and before pull-request creation, the post-execution block runs a runner-controlled verifier inside the container: it derives the changed files from the merge base, selects the required checks with the same selection implementation the runner-side contract uses (preloop.utils.verification_selection, embedded verbatim into the verifier script so the two can never drift), executes each check with PRELOOP_DISABLE_TELEMETRY=true, a per-command timeout, and the PRELOOP_VERIFY_BASE/PRELOOP_VERIFY_HEAD range contract, and records evidence: commands, exit codes, per-check logs, an environment digest, the profile version, and the exact commit SHA and tree hash it verified. The legacy verifier always runs required checks: files in the agent sandbox cannot safely certify a cached pass. Reuse requires an authenticated runner/controller cache bound to the commit, tree, complete profile and environment. Selection reasons are recorded with the evidence, so operators can read why each check was required.

The verdict is fail-closed. Publication (push and PR creation) runs only when every required check passed on the exact commit and tree being published and the tracked tree is clean; a failed check denies publication as verification_failed, a check that could not run (missing interpreter, exhausted gate budget) as verification_blocked, and anything uncertain (no verdict file, a crash, a commit that appeared after the gate) denies too. A denial exits non-zero, which fails the execution with its commits and evidence kept recoverable; nothing unverified leaves the workspace. There is no draft-publication exception in this mode. The orchestrator captures the compact evidence from the PRELOOP_VERIFICATION marker line and stores it under verification in the execution result (with source: runner), while an agent-authored verification key in result.json is demoted to verification_reported: the implemented status (status) and the verification status (verification.status) are two different, separately auditable fields. Evidence logs land in /workspace/evidence/verification/ inside the normal evidence pack. The full credential boundary for test processes (no write credentials) is a separate workstream (issue #432).

Flows saved before the gate existed resolve to mode: "off" through effective_verification_policy() (an explicit, visible "ungated, because no policy is configured" rather than a silent default), while an explicitly configured malformed policy blocks publication; it never silently downgrades to off. The flow API also validates the profile shape on save.

Transient Failure Handling

A flow run crosses two systems that fail independently and briefly: the Kubernetes API that creates the agent Job, and the model provider behind the gateway. Neither should end a run on its own. Three bounded, jittered retry layers cover them, and every terminal failure is stamped with a category so the remaining failures can be counted instead of read.

Layer 1: agent Job creation (ContainerAgentExecutor). The Job name is derived from the execution id, so a second session for the same execution collides with the first. That is exactly what the run-level retry (layer 3) and the confirmation nudge do, and on staging it turned a recoverable provider blip into Failed to start agent Job: (409) Conflict: 8 of 100 failed executions. Two changes: each additional session now asks for its own name (agent-<execution-id>-a2, …-nudge; the first attempt keeps the historic unsuffixed name so in-flight runs stay addressable), and Job creation is wrapped in a bounded retry. On a 409 AlreadyExists the existing Job is read rather than blindly recreated: if it belongs to this execution and is still live it is adopted (an idempotent create, e.g. a duplicate dispatch, must not fail the run); if it belongs to this execution and has finished it is deleted with propagationPolicy=Background, waited for, and recreated, but only while a creation attempt remains, because deleting a leftover nothing will recreate would cost its logs and fail the run anyway; if it belongs to a different execution, or carries no preloop.execution_id label to prove it is ours, the run fails immediately as runner_conflict rather than adopting or deleting someone else's agent. 429/5xx from the API server are retried. Bounds: AGENT_JOB_CREATE_MAX_ATTEMPTS (default 3, flowExecution.agentJobCreate.maxAttempts) and AGENT_JOB_CREATE_RETRY_BASE_SECONDS (default 0.5, exponential with equal jitter). Set max attempts to 1 to disable the layer: a 409 then fails the run as runner_conflict without touching the conflicting Job.

Layer 2: upstream model calls (model gateway). Inside one client request, the gateway retries an upstream failure that happened before any token was forwarded to the client: connection errors, 429, 5xx, and a stream that dies during prefetch. A partially streamed response is never retried, because the client has already seen output. Retry-After is honoured and capped so a hostile header cannot pin a worker. Bounds: MODEL_GATEWAY_UPSTREAM_RETRY_MAX_ATTEMPTS (default 3), MODEL_GATEWAY_UPSTREAM_RETRY_BASE_SECONDS (default 0.2), MODEL_GATEWAY_UPSTREAM_RETRY_AFTER_CAP_SECONDS (default 8), under environment.modelGatewayUpstreamRetry in Helm. The number of retries spent on a request is recorded on the usage row (meta_data.upstream_retries) and published to the execution timeline as retried: n on the model_gateway_call event, so a slow run that was quietly rescued is visible rather than mysterious.

Layer 3: whole-attempt retry (_run_agent_with_retries). When the agent process itself dies, the run is retried from scratch only where that is provably safe; otherwise it fails with a category. The safety boundary is the container's post-execution block (git push, PR/MR creation), which the entrypoints run only on exit 0. So an attempt is retried only when all of: the agent exited non-zero (no push/PR happened), no actions were recorded on the timeline, and the failure was classified transient by the executor's analysis of the full logs. An unknown exit code counts as unsafe. Bounds: FLOW_EXECUTION_MAX_ATTEMPTS (default 2) and FLOW_EXECUTION_RETRY_BACKOFF_SECONDS (default 15, doubling). Retries are never silent: each one writes an execution_retry_scheduled milestone, a timeline warning, and an execution_retry update; exhaustion writes execution_retries_exhausted.

Failure categories. Every terminal non-success execution stores a coarse failure_category (flow_execution.failure_category, indexed; exposed on the execution list and detail schemas) from a closed vocabulary: runner_conflict, runner_error, model_transient, model_auth, provider_billing, budget_exceeded, model_quota, model_config, no_confirmation, agent_no_progress, publication_missing, result_schema_invalid (the captured result.json does not match the flow result schema), setup_failed, verification_failed, verification_blocked, tool_error, agent_error, model_stream_idle, approval_denied, approval_timeout (a native command park was refused or expired), approval_unavailable (a hosted run was refused because an approval-required native rule could not pause it), hook_binding_invalid (a hosted Codex run's native hook credential would be refused by permission-check, so it ended before launch), resume_failed (a parked run could not be continued; also written on the parked row when its resume execution fails), gateway_bypassed (a private-runner Codex or Claude Code host CLI produced model output with no request through the gateway), timeout, cancelled, unknown. It is derived once at failure time by preloop.services.flow_failure_category.derive_failure_category from the best evidence available, in order: a category named by the raising code (e.g. AgentStartError), structural message shapes that identify the failing layer regardless of provider noise (runner conflict, runner error, timeout, cancellation, missing completion confirmation, publication-gate denial), then the agent executor's failure analysis of the full logs (which shares the upstream_errors taxonomy with the gateway), then provider-message patterns. Anything unmatched is unknown rather than being folded into agent_error: a rising unknown share is the signal to extend the module. error_message remains the human-readable detail; the category is what you group by. Existing rows are not backfilled: categories are derived from live failure context, and guessing them from historical prose would launder a heuristic into stored data.

The legacy verifier runs in the same sandbox as the agent. Its producer label, files and log markers are diagnostic data, not an authenticated attestation. Agents or repository commands can reproduce a marker. Credential-isolated publication must use a separate controller/runner-host verifier that observes command completion outside that sandbox on an immutable commit artifact. Never promote PRELOOP_VERIFICATION JSON directly to trusted publication authorization.

Controller-owned native continuations cannot use the generic execution retry endpoint: it returns 409 before creating a run, and the console omits that retry action. Open the original publishing execution's Manage PR follow-up preview instead. After the flow is enabled, an explicitly confirmed recovery can wake an existing native thread stopped specifically because the flow was disabled. Recovery requires current provider head confirmation, valid unexpired saved state, the same account/flow/source/PR/branch, remaining limits, and no running repair. A failed active attempt can be released only when it has no runtime reference, native session, tokens, tools, or cost. A runtime reference fails closed even if logs suggest initialization failed; zero counters alone cannot prove that no agent work occurred. It retains the thread's identity, expiry, usage and limits. The scheduler then reads current provider feedback and atomically reserves a repair only if work remains; recovery itself does not create an execution.

Bitbucket preview and feedback reconciliation resolve abbreviated PR hashes through the bound repository's commit API before comparing full hashes. Adoption still requires exact full-SHA confirmation and detects a head change during preflight.

Isolated publication (integration preview)

git_clone_config.publication_mode defaults to legacy for saved-flow compatibility. Legacy mode executes publication in the agent container and shares tracker write credentials with that runtime; it is not a credential isolation boundary. Operators must explicitly migrate a flow to isolated. Hosted execution has a controller-owned verification adapter. Private runner publication additionally requires the authenticated runner-host protocol. An isolated execution without controller-issued evidence fails closed before issuing write access.

Isolated flows require verification.mode: gate, a nonempty trusted profile, and verification.image set to a digest-pinned generic toolchain image. That image must provide Python 3, Git, a shell, and the dependencies needed by its configured checks. Verification does not reuse the agent's virtual environment, node_modules, writable caches, or workspace. A check may perform bounded setup using dependencies already available in the image, including local package caches and service binaries; it cannot download dependencies during verification. This is a generic toolchain contract, not a requirement to bake an application or its database into the harness image. Missing dependencies fail the check and leave the captured work recoverable.

The controller resolves the exact base commit before starting the agent. After committing recovery artifacts to durable storage it deletes the owned agent runtime and confirms its absence, including residual Kubernetes pods. A missing or failed recovery capture retains the original runtime and blocks publication. It derives changed paths from the frozen bundle, selects relevant checks from the trusted profile, and runs each check in a fresh credential-free checkout of the same head. Docker has no network, host mounts, or host namespaces. Kubernetes disables service-account automount and applies a deny-all NetworkPolicy. Because policies are additive, existing permissive policies matching the verifier's labels block startup. Hosted clusters must enforce NetworkPolicy and prevent concurrent policy or admission changes from weakening the verifier namespace's isolation. The controller observes process exit codes and confirms every verifier runtime was removed before minting writer credentials. Check diagnostics retain a bounded scrubbed output tail, per-check elapsed time, and Docker log rotation limits runtime log growth. If Kubernetes deletion cannot be confirmed, the deny-all NetworkPolicy remains with the execution's verifier label for operator recovery; removing it first would restore network access to residual pods. Log markers and result files never authorize publication.

The isolated path binds a single repository and its base/target branches from account-owned flow/project records. It never accepts a webhook clone URL as publication authority. GitHub App installation tokens are minted for exactly that repository: contents:read for the agent, then contents:write and pull_requests:write for the publisher after verification. Stored PATs and GitLab publication are rejected in this mode until a broker can enforce their scope and lifetime. The standalone metadata/provider client supports both GitHub and GitLab. Private runners require an authenticated current capability advertisement confirming protocol v1 and a locally available digest-pinned trusted helper image. Older runners cannot receive an isolated lease. Private source stays on the runner; the controller receives only bounded manifests, check outcomes, and the provider receipt.

Private verification is an ordered, nonce-bound handshake. The runner removes the agent and residual volume writers, freezes the bundle in independent storage, runs the exact controller-selected checks, and confirms verifier removal. Only then does the controller issue a repository-scoped writer to the trusted publisher helper. Agent result files and ordinary completion messages cannot advance publication. Both helper and controller revoke the writer; already-invalid token responses make repeated revocation safe.

Recovered private monitoring restores the protected policy and accepted receipt without relaunching the agent or replaying credentials. Queued isolated executions fail closed on worker recovery because their original lease cannot be safely replayed; retry explicitly after an eligible runner is available. Private frozen recovery artifacts remain local with a fixed 24-hour retention limit after verification or publication failure. A recovered hosted isolated execution currently cannot reconstruct its original trusted policy snapshot: it fails closed with an explicit diagnostic and retains its original runtime for recovery. It does not silently switch to ungated publication.

After runtime cleanup, the control-plane publisher imports a bounded, self-contained branch.bundle into a fresh bare object store. It never checks out repository code, imports agent Git configuration, or runs repository hooks, filters, credential helpers or shell commands. Each Git child has a clean environment and CPU, memory (Linux), file-size and time limits. HTTPS redirects are disabled. The publisher validates the exact verified head, checks the expected remote SHA and ancestry, then uses an atomic lease to reject concurrent remote changes. It never rewrites unexpected remote history. Provider failures mark publication failed and retain captured recovery evidence; retry reuses the existing branch and PR. Write tokens are revoked on success/failure and expire at the provider-issued deadline if revocation is unavailable.

The controller handoff is VerifiedPublication(execution_id, head_sha, bundle_sha256). It is an internal type, not an agent JSON schema. A trusted verifier adapter must construct it only after verifying the immutable artifact in an environment the agent cannot modify. Agent-written result files or log markers cannot establish this attestation. Hosted adapters construct it only after the complete trusted verification lifecycle. Deploy the control plane separately from agent workloads; never mount its process namespace, filesystem, Docker socket or signing keys into those workloads.

PR descriptions and execution provenance

Repository setup selects git_clone_config.pull_request_template when set, then the conventional GitHub/GitLab default, then the lexicographically first named Markdown template. It writes /workspace/evidence/pr-template.md for the agent to fill. No template uses Summary and Testing. Configured missing, invalid, oversized or escaping template paths fail setup. Tests that did not run remain unchecked. The implementation preset asks for problem, resulting behavior, acceptance evidence, verified commands, limitations and an issue link.

result.json retains #420's pr_title/pr_body fields and aliases. Valid agent text wins per field over configured text and commit text. Titles are one line, at most 256 UTF-8 bytes; bodies are at most 60,000 UTF-8 bytes. Invalid/missing metadata falls back with a diagnostic; no truncation silently changes meaning. All publication sources receive execution attribution. The isolated publisher owns only the preloop:executions HTML-comment region. It upserts trusted initial/repair execution links and published SHAs while preserving human edits outside that region, including metadata-only repairs. Links use PRELOOP_URL and existing authorization-protected console routes; tokens and transcripts are never provenance inputs. Legacy publication adds the current execution block on creation. When an open pull request or merge request already exists for the branch, legacy mode fetches that description, appends the current execution id and head SHA to the owned block when that pair is not already present, and updates only the body. Human prose and the title stay as they were. A malformed owned region or an oversized rewrite warns through PRELOOP_PR_METADATA_WARNING and leaves the owned provenance region unchanged; the independent failure-disclosure refresh still runs. A failed provider update is surfaced and never reported as successful publication. Isolated GitLab publication stays unsupported until a broker can enforce credential scope and lifetime.

Publication acceptance matrix (issue #431). Each cell is delivered (test name) or unsupported by design.

Mode	Provider	Create	Continuation push to an existing PR	Metadata-only retry	Failure disclosure	Human edits preserved	Provider failure surfaced
legacy	github	delivered (TestWritePrPayloadPy.test_commit_fallback_single_commit_includes_execution_link; create script calls upsert_provenance once)	delivered (test_github_continuation_appends_record_and_keeps_prose)	delivered (test_repeated_continuation_is_idempotent_and_reuses_the_pr, test_missing_metadata_warns_and_keeps_existing_prose)	delivered (test_already_pushed_commits_refresh_existing_failure_notice, test_existing_body_preserved_and_notice_idempotent)	delivered (test_github_continuation_appends_record_and_keeps_prose)	delivered (test_provider_update_failure_is_not_success; a create miss is test_no_url_anywhere_emits_no_marker)
legacy	gitlab	delivered (same create script, kind == "gitlab")	delivered (test_gitlab_continuation_appends_record)	delivered (test_repeated_continuation_is_idempotent_and_reuses_the_pr)	delivered (same failure-disclosure tests, GitLab payload)	delivered (test_gitlab_continuation_appends_record)	delivered (test_provider_update_failure_is_not_success)
isolated	github	delivered (test_provider_create_retry_metadata_update_preserves_human_edits)	delivered (same test, repair upsert)	delivered (same test: one POST, later upserts only)	out of scope (issue #599; the isolated publisher upserts provenance only)	delivered (same test)	delivered (test_provider_failure_is_observable)
isolated	gitlab	unsupported by design (flows.md: "Stored PATs and GitLab publication are rejected in this mode until a broker can enforce their scope and lifetime")	unsupported by design (same)	unsupported by design (same)	unsupported by design (same)	unsupported by design (same)	unsupported by design (same)

Continuation append keeps the first execution record and the most recent 199 repair records (PROVENANCE_RECENT_RECORDS). The 201st continuation still lands (test_append_provenance_keeps_the_first_record_and_recent_199).

The standalone metadata client still accepts a GitLab payload shape. Isolated mode does not: validate_publication_tracker rejects PAT and GitLab credentials before a lease is minted.

Preset synchronization updates uncustomized fields and marks customized saved flows as having an available update. Inspect the effective saved prompt and configuration before expecting template behavior. The publication-mode switch is deliberate and is not silently enabled by updating the prompt.

Native tool event counts

Execution log parsing counts Claude tool-use blocks and Codex tool execution items by harness call ID. Stream starts and their completed message count once; restoring an execution also restores recorded call IDs before replay. Tool arguments and results are excluded from these metric entries. Legacy text MCP markers remain supported for harnesses without structured output.

Employee event intake

Employee event intake authenticates provider events, verifies Flow/account/source scope, reserves the existing delivery-keyed execution before dispatch, and reuses Flow worker claims and recovery. Managed-agent control commands carry encrypted execution credentials; Codex/Nanobot tasks use independent bounded conversations and command-owned interruption.

Implementation feedback and forge state

Implementation PRs can use durable feedback subscriptions: PostgreSQL threads and inbox leases coordinate new execution turns, while native conversation artifacts remain isolated from workspace checkpoints. Repository events and bounded reconciliation advance CI/review gates without idle agent containers. Bitbucket review feedback requires a current changes-requested participant state and excludes automated summary comments; metadata updates skip successful reviews of the same pull request head. Feedback opt-in applies to future executions; a preview-and-adopt API binds one older publication explicitly. Live policy changes are checked again at atomic repair reservation. Missing native checkpoints fail closed unless the operator explicitly selected a source-only published-branch handoff. Unreadable repository requirements prevent readiness while fully verified feedback can still authorize bounded repairs.

Forge reconciliation keeps one forge truth for pull request state. Feedback reads split merged from closed (pr_merged, pr_closed) on GitHub, GitLab and Bitbucket, and a throttled read waits out the provider's Retry-After instead of blocking. services/pull_request_refresh.py refreshes IssueCostPullRequest.head_sha, state and merged_at for open or not-yet-observed pull requests: at most 25 records and 25 API reads per one-minute tick, including fallback trackers and Bitbucket commit lookups, each open record re-armed after five minutes. Reads use If-None-Match when the forge supplies an ETag; Bitbucket Cloud PR reads explicitly fall back to plain GETs without a validator (see forge reconciliation). A pull request webhook only marks its record due; a merge is never reverted by a later read. Webhook ingress claims each X-GitHub-Delivery, X-Gitlab-Event-UUID or Bitbucket X-Request-UUID in forge_webhook_delivery before queueing, answers a redelivery as a duplicate, and releases the claim when publication is not acknowledged. An unpublished claim can be taken over after two minutes; publication and release match the claim timestamp so a late holder cannot mutate its replacement. Copilot records follow the same bounded refresh and ingress dedupe paths.

Delivery reports

Delivery Report console reads go through account-scoped CRUD. Attention includes only pending or unconfirmed proposals, with a fingerprint that changes when the pending decisions change. Execution detail projects forge state from IssueCostPullRequest and the newest assessment at its current head SHA. Project reruns use the existing manual flow trigger, narrow the report policy's scope for that execution, and recheck the policy before running. They preserve the account's assessment budgets and tracker-write rules.

The account report policy shows recipient roles only when the Enterprise plugin advertises delivery_report_recipients. Its roster, role overrides, sponsoring lead and per-person opt-outs use the plugin recipient API and policy permissions, independently of budget editing. Budget saves retain the current recipient settings; each person retains the shared self-subscription control.

Restricted CI and read-only review publication

Restricted CI administration is an OSS human-only API, Lit console and CLI over core CRUD. It separates immutable principal/resource ownership from rotating credential lifetime, gates issuance on complete backend contracts, and offers safe expiry/revocation recovery. The OSS owner authority can be narrowed or delegated by EE through current account-local permission hooks; a fresh scalar CRUD permission read avoids cached role authority. No human role can widen the machine's explicit route ceiling. Console secrets are ephemeral and CLI secrets go to exclusive private files. The disposable release gate validates real-resource admission/ownership and callback controls; the generic operator workflow verifies persisted exact-head correlation and a separate publication receipt.

Restricted GitHub App review flows can opt into publication_mode: read_only. The principal snapshots that capability; startup reauthorizes ownership and mints only an exact-repository contents:read lease. No normal tracker token reaches the clone, no MCP or general REST authority reaches the model, and completion cannot select a writer or publication path. Local result persistence and owned signed callbacks remain available; the separate publisher holds write authority. The executor is either hosted Codex on the server pool or a Codex host_exec_profile on a named private runner pool: the lease travels only in the transient delivered checkout plan, the runner clones with host git config and credential helpers disabled, verifies the admitted SHA, revokes the lease before starting codex exec --sandbox read-only, and returns a receipt the control plane requires for success; the control plane revokes delivered leases again at completion. Legacy and isolated publication retain their existing contracts.

Isolated publication

Isolated publication verifies immutable bundles in fresh credential-free runtimes before acquiring a write lease. Controller-owned evidence reuse binds the execution, bundle, base, profile, pinned image and runtime; sandbox-written evidence cannot populate it. Failed durable repairs retain their latest workspace and native session while recovering only the prior published branch binding from authenticated thread ancestry.

Harness context and output settings

Agent harnesses receive supported context and output settings from model-row parameters and the vendored catalog. Hosted Codex adds a version-pinned local catalog for unknown custom models, preserving its generic instructions and labeling an unknown context as a CLI fallback. The model gateway controls Codex's upstream output allowance when enabled; direct provider runs retain provider defaults. Native Codex profiles stay intact. No provider capacity is inferred from an unknown model name.

Subscription models in flow routing

Flow routing admission rejects agent-bound subscription models for ordinary ephemeral execution credentials before creating executions, unless the subscription's owner opted in ("Use from my flows", ai_model.owner_flow_use_user_id) and also owns the flow (flow.created_by_user_id). The gateway applies the same rule to the execution's runtime credential (services/subscription_flow_use.py) without touching agent bindings; refreshes stay serialized on the credential row, so the onboarded agent and hosted runs share one token lineage. Matrix admission validates all cells before writing the batch. Authoritative orchestrator startup rechecks legacy queued rows before runtime launch and records incompatible subscription selections as terminal model_config failures. Persistent agents, private host profiles and persistent employee identities keep their existing admission and gateway ownership checks. See model routing.

Execution environments and hosted publication

Execution environment profiles and hosted checkpoint recovery are documented in Environments and recovery. What enforces browser network access for a flow agent is Browser access. Workspace checkpoint capture streams individual files and enforces its compressed size cap during packing. Dependency directories and caches are excluded; oversized captures leave the last complete checkpoint available. Hosted legacy GitHub App publication refreshes repository-scoped credentials on the controller immediately before push or PR creation. A signed runner capability binds the execution, account, tracker and startup repository; issuance requires an active execution. App signing keys remain on the controller. The runtime replaces stale git credentials and uses the fresh token for PR REST calls as well. The sandboxed-browser allowlist sidecar lives in environments/egress-proxy. Hosted Codex defaults to the digest-pinned universal image (ghcr.io/preloop/preloop-codex-universal), which includes the distro Perl toolchain. Private runners can still build environments/perl and select it with agent_config.image. Native host profiles use no image. Hosted Codex uses the Responses API; chat-only custom models must enable meta_data.gateway.enabled to use the gateway translation. Flow saves reject incompatible bindings with remediation, and legacy incompatible runs fail with model_config before launch. Direct providers explicitly configured for Responses remain supported. The chat-completions translation flattens Codex MCP namespaces for upstream models, then restores tool calls to the shape the client declared. Qualified, dotted and unambiguous bare aliases resolve to that same tool; flat MCP declarations retain flat calls. Hosted Codex's CLI version is pinned separately from its toolchain image. Codex uses JSON events: only the last assistant message in a completed turn can provide a plaintext completion marker. Prompt echoes, tool output and diagnostics stay escaped in the log. The completion grace timeout captures artifacts and stops the runtime before reporting terminal success.

Release backport

A flow with an enabled git_clone_config.backport block runs in a control-plane mode: the orchestrator cherry-picks the merge commit onto each target branch in a scratch repository and opens one pull request per target, with no agent container. See Release backport.

Flow delegation and execution trees

A flow execution can start another flow of the same account as a child of itself through the default-off run_flow tool, gated by both the flow's allowed_mcp_tools and its callable_flows allowlist. FlowExecution carries parent_execution_id, root_execution_id and delegation_depth, so a subtree is one query and rows that predate the columns read back as roots. GET /api/v1/flows/executions/{id}/tree returns an execution, its descendants and a rollup in the same shape the batch listing uses. run_flow(wait=true) waits in process briefly and then parks the parent on WAITING_FOR_CHILDREN, releasing the container, the runner and the runtime token exactly as a park on a human decision does; the parent resumes as a new execution that natively continues the same agent session. Depth, fan-out and subtree cost are bounded by FLOW_DELEGATION_MAX_DEPTH, FLOW_DELEGATION_MAX_CHILDREN and FLOW_DELEGATION_MAX_TREE_USD. See Flow delegation.

Agent launch payload (custom images and runners)

Linux caps one execve string (a single argv element or a single NAME=value environment entry) at MAX_ARG_STRLEN, 131072 bytes. The control plane therefore does not put an unbounded prompt or Kubernetes inner script in one string.

The rendered prompt travels as PRELOOP_AGENT_PROMPT_0..N (plus PRELOOP_AGENT_PROMPT_CHUNKS and PRELOOP_AGENT_PROMPT_BYTES), is reassembled into /tmp/preloop/prompt.txt, and is advertised as AGENT_PROMPT_FILE. AGENT_PROMPT is set only when the prompt is 64 KiB or less. The Kubernetes inner script uses the same pattern: PRELOOP_INNER_SCRIPT_0..N into /tmp/preloop/agent-script.sh, with a legacy whole-value PRELOOP_INNER_SCRIPT still honoured so old and new images interoperate.

Custom images and private runners should read AGENT_PROMPT_FILE (or reassemble the chunks) and must not require AGENT_PROMPT for large prompts. The full contract is in Agent launch payload (container environment). Private Docker launch shape is in the runner image contract.

Issue lifecycle controller

services/issue_lifecycle.py coordinates structured readiness, one authorized implementation pickup, and independent merge/deployment acceptance audits. IssueLifecycle records live scope revisions, immutable merge references, execution bindings, evidence and follow-up identities through the CRUD layer. Transaction locks serialize each tenant/issue; provider markers recover external writes after local rollback. The trigger service prepares lifecycle executions before ordinary dispatch, and the orchestrator consumes durable structured output at completion. The GitHub adapter revalidates closing-commit/merged-PR authority; manual completion alone does not start an audit. Readiness consumes approved execution-environment capabilities rather than implementing test setup. See Issue readiness and completion audits for policy, API and recovery configuration.

services/issue_triage_controller.py reuses this ledger for durable issue-revision claims across manual and automatic triage. Dedicated advisory-lock connections serialize local applies while provider-write intents commit durably. Execution credentials bind managed writes to their issue/revision; successful bounded assessment packets retain input/output revision and policy/context identity. Readiness consumes only applicable server-loaded packets as evidence, preserving its separate implementation authorization. Provider writes remain optimistic; local serialization cannot make external APIs support compare-and-swap. See Issue triage for recovery and tool boundaries.

Security maintenance controller

services/security_maintenance.py coordinates opt-in supported-release inventory, one work item per product/release/advisory/component, and fail-closed completion. Inventory is API-configured. Implementation uses the existing flow dispatcher and isolated publication receipts (head_sha). After approval, a rebuilt SBOM must be submitted for the published commit; recheck does not reuse the original inventory. Removal is derived from identities in those submitted bytes, not from model inventory omission. Initial baseline acceptance requires the controller envelope release_id and the digest of the supplied SBOM. Tests and baselines require controller-owned verification and the contracts CRA validator; caller-supplied evidence ids and agent finding_absent fields do not grant. Platform ApprovalRequest rows carry human decisions. Dispatch claims expire so a crash between claim and enqueue cannot strand a still-PENDING execution. Audit/recheck completion observes frozen Git bundles from evidence when the flow names repository URLs and the pin is an exact git SHA; HEAD.txt is not checkout proof. See Supported-release vulnerability maintenance.

Session search writes one session_search_document chunk per source row (gateway interaction, transcript, tool call, operator note, summary). Keyword indexing is gated by SESSION_SEARCH_INDEX_ENABLED. Optional vectors are a separate per-account opt-in (session_embedding_setting, summaries_only by default so an opt-in embeds the session's title and summary chunk unless the account asks for full) plus the deployment kill switch SESSION_EMBEDDING_ENABLED; a bounded worker posts batches to an OpenAI-compatible endpoint or a local model, caps spend, and records purpose-tagged usage. The shared API key is allow-listed. POST /api/v1/runtime-sessions/search reads it in keyword, semantic or hybrid mode, fusing the two candidate lists by rank; a half that cannot run is reported in a degraded block with keyword results, never as an error. GET /api/v1/runtime-sessions/{id}/similar reads the same vectors with no query at all: a session is compared by a stride sample of its own chunks, in its own model's space, and spends nothing. See Similar sessions. A question worth repeating can be saved under a name in session_saved_search and re-run from /runtime-sessions/search/saved; a saved search stores the question and never the answer, is private until its author shares it with the account, and a run says which of its filters no longer resolve. Personal saved-search writes use view_runtime_sessions; creating or changing a search to account visibility also requires share_session_searches. Optimization submissions require run_session_optimizations, while replay requires edit_ai_models. The sharing and analysis permissions are granted to owner, admin, editor and executor by the system-role initializer and an Alembic data migration for existing databases. The same ranked search is reachable as the built-in search_sessions tool (default off, own-sessions scope unless an operator grants account) and from preloop sessions search. Every content search writes one audit row through the ordinary audit path, with the mode, the filters, the result count and a stable hash of the query; the query text itself is stored only when the account opts in. Operator knobs: Session search (coverage, the history backfill, who may search) and Session embedding (the vector worker). Saved searches: Saved session searches. Search auditing: Auditing session content search. The agent-facing tool: search_sessions.

Audit collection also offers a durable delivery path behind the default-off PRELOOP_AUDIT_OUTBOX_ENABLED flag. Committed sanitized intents are relayed into audit rows transactionally; failed deliveries back off and park after a finite retry budget. Approval transitions, policy evaluation, MCP activity outcomes and model-content policy hits enqueue in their owning database transactions, bypassing bounded audit admission. Async callers await only local database work; the relay owns its sessions on a separate thread. Due lag, failed attempts and parked rows have opt-in Helm alert rules; drop alerts remain for best-effort callers. Local operators can preview and replay a bounded failed batch for one account with python -m preloop.services.audit.replay. See Audit outbox operations for activation, admission limits, metrics and recovery.

Pi and DeepSeek Harness adapters

The pi and deepseek kinds share ExtensionHarnessAgent for hosted and private Docker launches. runtime-plugins/harness-preloop supplies Pi extensions and DeepSeek Cordis plugins for MCP, fail-closed native approvals, lifecycle ingest, and authenticated active-session control. CLI enrollment owns a separate preloop.json, preserving native configuration, and registers only its own loader or marked patch nodes. DeepSeek's released headless startup is replaced with a stdin provider to keep prompts outside argv. Execution-scoped credentials can request native approvals only for their own active flow session. See the adapter guide for version pins, capability limits, and publication order.