Skip to content

Execution environments and checkpoint recovery

Editions: OSS, Cloud, Enterprise. Unless stated otherwise, everything on this page ships in OSS.

Default images are generic harness/toolchain images. They do not bundle the Preloop application, PostgreSQL, or another customer's application. A project can supply its own image and dependencies. An optional hosted environment profile supplies the application dependencies an implementation agent needs before it spends its model budget. An administrator installs a JSON registry and sets FLOW_ENVIRONMENT_PROFILES_FILE to its path. A flow selects an entry with agent_config.environment_profile. Issue descriptions cannot add images, mounts, host privileges or secrets to that registry.

Profiles require an image digest, harness (codex or opencode) and protocol version 1. The image contains Python 3.12+, the harness, and /opt/preloop-environment.json with version: 1 and the harness name. Startup rejects an incompatible image before setup. Source still comes from the requested repository checkout. The profile image does not include a stale copy of the application.

environments/preloop/Dockerfile is a Preloop-specific example and integration fixture, not a default agent image. It extends the existing Dockerfile.dev image with pinned Codex, Playwright/Chromium, a PostgreSQL Python driver, and a distro Perl toolchain (perlver, perlcritic, and prove). Build from the repository root so the hash-pinned tools/ lockfile and requirements.txt copy in. Build the dev image first, then pass its digest as DEV_IMAGE. Register the resulting image digest. environments/preloop/profile.json.example contains component, backend and full-application profiles; replace the explicit digest placeholders with operator-approved digests. The component profile only installs frontend packages. The backend profile starts disposable PostgreSQL/pgvector and NATS. Neither starts the complete application for every change. The application profile explicitly calls scripts/flow-environment-app-check.sh, which starts and health-checks the API/frontend, seeds disposable test data and cleans up its child process groups.

Each dependency has a pinned image, unique name/port, optional command and nonproduction environment variables. Docker provisions an execution-specific network with service DNS aliases. Kubernetes uses native sidecars (Kubernetes 1.29+ with sidecars enabled), localhost ports and startup probes; the kubelet terminates them when the main container ends. Services are also removed during executor cleanup. No host Docker socket is mounted. Private runners currently reject named version-1 profiles explicitly because their custom image entrypoint contract differs. Existing private custom images remain supported; they receive GIT_CLONE_CONFIG and CUSTOM_COMMANDS JSON, including setup commands. Those images must implement these entrypoint fields themselves.

A private-runner flow can select its project image directly through the flow API: agent_config: {"image": "registry.example.com/team/project-agent:release"}. The private launch log reports the effective image reference, including either custom-image alias, alongside the installed harness version. The reference is the requested tag or digest; a tag alone does not attest the pulled image digest. No named profile registry is required. docker_image is also accepted; when both keys are present the runner checks image first. Omit environment_profile for this raw image path. An explicit nonempty override takes precedence over the operator's per-harness image environment variable and the generic fallback. Docker on the private host must be able to pull that image, whose entrypoint must consume the flow environment contract. The flow form offers a Custom container image field for ephemeral execution on a private runner (an explicit runner or label, or a private account default). It loads image first and falls back to docker_image, writes a trimmed image on save, and keeps the saved value untouched while the field is unavailable (hosted, Auto, persistent, or native host execution), so an override can still be managed through the API.

Setup has its own timeout and failure marker, with output under /workspace/evidence/setup.log. Readiness runs on every attempt. Profiles may list lockfiles and cache paths; dependency setup is reused only when the profile and lockfile contents match and all declared cache paths still exist. Restoring a workspace without its reproducible dependencies forces setup to run again. Image layers remain reusable independently. Test-command groups and artifact paths describe the repository's verification contract; the publication gate chooses and verifies the relevant commands. Issue readiness requires an enabled verification gate. Every command in its always/rule/unknown-impact policy must have an environment command group with the same ID and exact shell text (multiple steps are joined with newlines). Issue acceptance command IDs must also appear in the verification policy. Capability readiness is not test-result attestation; agent-sandbox files and log markers cannot authorize isolated publication.

Universal Codex image

environments/universal/Dockerfile builds preloop-codex-universal from ghcr.io/openai/codex-universal pinned by digest, and adds the distro Perl linters and shellcheck. The Publish Universal Image workflow (.github/workflows/publish-universal-image.yml) builds linux/amd64 and linux/arm64, smokes amd64 with environments/universal/run-smoke.sh, and publishes the multi-arch index to ghcr.io/preloop/preloop-codex-universal. The container package must be public in GitHub package settings; after pushing and attesting the index, the workflow verifies anonymous digest access and fails if access is unavailable.

A pull request and the weekly schedule build and smoke only; they do not push. A version tag, a manual run on main, and a merge to main that changes the recipe publish. Tags are sha-<12 hex chars of the commit> and vX.Y.Z on a version tag. The run summary records ghcr.io/preloop/preloop-codex-universal@sha256:<digest>, and the workflow attests that digest. The hosted default is that published digest, pinned in backend/preloop/agents/images.py as ghcr.io/preloop/preloop-codex-universal@sha256:191dc7e2ef84b7a05c2b3d6403df59c9b2e51e890b0853eae1935ccc4417dd78. Helm agents.codexImage sets CODEX_IMAGE on the API and worker pods when an operator overrides it. Leave it empty to use the core default. A tag-valued CODEX_IMAGE logs one warning at startup. Private runners can still build environments/perl.

Backend and frontend tests in the image

The image built from environments/preloop/Dockerfile runs the Preloop test suites from a checkout with no network and no install step.

  • /opt/preloop-tests is a virtualenv installed from .github/requirements/app-dev.txt, the hash-pinned lock the CI backend shards install (runtime dependencies plus the dev extra, including pytest). It is on PATH. The checkout itself is not installed.
  • preloop-pytest [pytest args] runs pytest from the checkout root with backend/ on PYTHONPATH. When DATABASE_URL is unset it starts a throwaway PostgreSQL 16 cluster under /tmp (pgvector 0.8.6, built in the image because Ubuntu's 0.6.0 lacks subvector()), applies the checkout's migrations with scripts/init_db.py, and reapplies them only when backend/preloop/models/alembic changes. With DATABASE_URL set (the backend profile's sidecar) it uses that database as is. Tests, migrations and init_db.py run under an environment allowlist: provider keys are mock_key and agent API, gateway and git tokens are dropped. It works as root, uid 1000 and uid 10000 (the Docker harness user, which the image gives a passwd entry so initdb can run).
  • preloop-frontend-deps copies the image's frontend/node_modules into the checkout when frontend/package-lock.json is byte-identical to the one the image was built from, then cd frontend && npm test (or npx --no-install web-test-runner <file>) runs offline against the frontend's own pinned headless Chromium. On a different lock it exits 65; run npm --prefix frontend ci, which needs registry access.
  • A pull request that changes app-dev.txt makes preloop-pytest warn that the baked venv may lack a new dependency. Rebuild the image to pick it up.

The frontend test runner uses one browser session at a time inside flow containers, identified by both FLOW_ID and EXECUTION_ID. This avoids a host-CPU-derived default exceeding the container's resource budget. Set PRELOOP_TEST_CONCURRENCY to a positive integer to override that limit, for example PRELOOP_TEST_CONCURRENCY=2 npm test from frontend/. Invalid overrides fail configuration loading. Outside flow containers, an unset override preserves the test runner's normal default. Run scoped backend and frontend checks sequentially so they do not compete for the same container resources.

environments/preloop/python-venv-smoke.sh is the build gate (imports only). Given a checkout path it runs a database-backed backend test file and a frontend test file offline:

docker run --rm --network none --user 10000:10000 -e HOME=/tmp \
  -v "$PWD:/src:ro" <image> /opt/preloop-pip/python-venv-smoke.sh /src

backend/tests/test_preloop_backend_test_venv.py runs the same command when PRELOOP_ENVIRONMENT_IMAGE names a built image and skips otherwise. The backend venv (about 980 MB), frontend tree (about 280 MB) and second headless Chromium (about 270 MB) grow the image from 2.5 GB to 3.8 GB unpacked (1.07 GB to 1.52 GB compressed).

Browser profile

What enforces network access for this profile, and what does not, is Browser access for flow agents.

preloop-browser in environments/preloop/profile.json.example uses the same image as the component profile and adds an egress-proxy sidecar. The proxy contract (listen port, EGRESS_ALLOWED_ORIGINS, EGRESS_ALLOW_PRIVATE_CIDRS, EGRESS_DENY_PRIVATE, and GET /healthz) is the one in environments/egress-proxy/README.md. DependencyService.env already stores that service environment on the registered profile. A flow only selects the profile identifier, so the origin allowlist is fixed per profile rather than copied from agent_config.

environments/preloop/browser/enable.sh reads PRELOOP_BROWSER_PROXY (http://egress-proxy:3128 on Docker, http://127.0.0.1:3128 on Kubernetes), renders playwright-mcp.config.json, and registers a browser MCP server for PRELOOP_HARNESS (codex or claude). Codex reads ~/.codex/config.toml ([mcp_servers.browser] with command and args). The hosted Codex executor writes [mcp_servers.preloop] from backend/preloop/agents/codex.py and would replace that file; setup also leaves ~/.codex/preloop-browser-mcp.toml, which the executor appends after its own write. There is no Claude Code writer in this repository; Claude Code reads .mcp.json in the checkout, and enable.sh merges the browser entry there. The pinned package is @playwright/mcp@0.0.82, installed in the profile image next to Playwright 1.64.0-alpha-1789764292000 (the build that package bundles). The MCP command is the image binary /opt/preloop-env-tools/node_modules/.bin/playwright-mcp with --config, --proxy-server, --isolated, and --headless. Setup does not fetch the package from the npm registry. --isolated starts Chromium with an empty profile: no cookies and no operator storage state. The rendered launch args also include --proxy-bypass-list=<-loopback> so loopback is not a path around the proxy. The self-check probes the metadata address, a non-allowlisted origin, and a listener on 127.0.0.1.

The example profile does not list test_commands. A flow that gates verification on command IDs must add those IDs to the registered profile; otherwise readiness reports environment_command_missing.

Chromium is started with the two flags the proxy README requires: --proxy-server and --host-resolver-rules=MAP * ~NOTFOUND , EXCLUDE <proxy host>. Without them the browser can open connections that never pass the allowlist. --no-sandbox is added only when the probe runs as uid 0.

The proxy enforces origins. The harness MCP entry and Preloop tool permissions decide which tools the agent may call. Tool selection is not a network boundary. enable.sh then runs selfcheck.sh, which requires $PRELOOP_BROWSER_PROXY/healthz to answer ok and launches Chromium once against http://169.254.169.254/ and https://example.org/. Both probes must fail with a proxy error (egress_denied or a Chromium proxy/tunnel error). Any other outcome, including a missing proxy variable, a failed health check, or a probe that loads, exits browser_egress_not_enforced and aborts profile setup.

Routing this browser MCP server through the Preloop firewall is a follow-up. The control plane cannot reach the execution network to sit on that path today. Issue #885 would then add timeline rows for those tool calls.

Durable hosted artifacts

Enable FLOW_ARTIFACT_DIRECT_UPLOAD when the runner can reach PRELOOP_URL. The checkpoint Helm overlay enables direct uploads with a 64 MiB compressed cap and matching 80 MiB proxy limits. Merge its extraEnv entries with existing installation values. Without it, the legacy snapshot path remains in effect, including the 2 MiB Kubernetes log-channel cap. Raising WORKSPACE_SNAPSHOT_MAX_BYTES alone does not raise that log cap. FLOW_EVIDENCE_LOG_PLAINTEXT=false refuses that log channel (the snapshot is then skipped with plaintext_disabled); see evidence storage. A skipped legacy snapshot does not mean setup failed. With direct upload enabled, workspace checkpoints travel through authenticated HTTP, never the pod log channel. The service validates compressed and expanded size, archive paths and file kinds, encrypts the payload with the configured encryption key, and commits the immutable manifest and payload together in PostgreSQL. An interrupted upload cannot become the latest checkpoint. Evidence packs use the same store with kind=evidence and a separate retention window; see Evidence storage and retention. Workspace checkpoint log transport is skipped when a checkpoint PUT token is present.

Capabilities permit one artifact kind and operation for one execution, account, flow and implementation thread. They confer no general storage access. Reads check the persisted thread binding, manifest identity and payload digest. Native-session archives use their own artifact kind and capability; FLOW_NATIVE_SESSION_RETENTION_HOURS (168 by default) caps their manifest expiry independently of workspace retention, so either artifact can report its own expiry; workspace capture excludes session directories. Agent containers never receive the control plane's encryption key. Use a dedicated SECURITY__ENCRYPTION_KEY in production, protect it separately from the database, and retain it across restarts. Key rotation must preserve access to artifacts encrypted by old keys.

A five-minute loop captures source and git state. It checks file sizes, modification times and membership for changes during capture, declining a busy snapshot rather than committing inconsistent state. The last completed checkpoint survives process/pod loss; writes after that checkpoint can be lost. Controlled exits attempt a final checkpoint. Before legacy wrapper publication, a failed checkpoint blocks publication, except when the archive exceeds the storage cap: that case logs PRELOOP_CHECKPOINT skipped checkpoint_oversized, exits 0, and leaves the last completed checkpoint as the resume point. A successful capture logs PRELOOP_CHECKPOINT committed <artifact_id>; a trailing deduplicated token means the workspace was unchanged, so the existing artifact was reused and no new storage quota was used. A trusted external publisher must make this checkpoint barrier part of its handoff as well.

A checkout that has nothing to recover does not store a workspace payload. Before the archive is built, each repository under /workspace is compared with the commit that was cloned. When git status --porcelain is empty after the usual exclusion rules (no staged, unstaged, or untracked file the snapshot would have kept) and HEAD is still that commit, the client stores a metadata-only checkpoint: the manifest repositories block (branch, base_sha, head_sha) plus file_state_sha256, and no file payload. It logs PRELOOP_CHECKPOINT skipped clean_checkout. That row counts as zero bytes toward the account quota, so a full quota does not block publication of a clean review. Restore logs the checkpoint age as usual and clones the commit again, because the code host already has it. A dirty tree, an untracked file the snapshot would keep, a HEAD that is not the cloned commit, a stash, or a commit on a local branch that is not on a remote still stores a full snapshot. A metadata-only restore reclones, so those refs would otherwise be lost.

agent_config.workspace_snapshots selects the policy: when_dirty (the default), always (a full snapshot on every capture), or never (do not upload a workspace checkpoint; the run logs PRELOOP_CHECKPOINT skipped workspace_snapshots_never). Preset 002 and the other review and implementation presets set nothing and inherit when_dirty.

WORKSPACE_SNAPSHOT_TTL_HOURS defaults to 24 in open source. Hosted Preloop sets it to 168. With when_dirty, clean review runs store no workspace payload, so that window applies to implementation snapshots and to the metadata row of a clean run.

When an upload, restore or evidence transfer is refused over HTTP, the marker names the status, a short reason code and the operation, for example PRELOOP_CHECKPOINT failed HTTPError status=409 detail=artifact_execution_closed op=capture. The reason comes from the API's JSON error body. An error page from a proxy in front of the API (ingress, console nginx) reports detail=not_json, an empty body reports detail=empty, an API 5xx without a reason code (its generic "Internal server error") reports detail=server_error, and any other body reports detail=unrecognized; the URL, token and body are never printed. When the API refuses a capture or evidence upload (PUT) after verifying its capability, it also writes an audit log row with action flow_artifact_rejected (status code, reason, execution id). Three refusals that do come from the API leave no row: a 401 invalid_artifact_capability, any restore (GET) refusal, and a refusal whose audit write itself failed (that failure is logged by the API). So a PUT marker with detail=not_json and no matching row points at a proxy; read the marker's status and detail before drawing that conclusion from a missing row alone. A quota refusal (detail=artifact_quota_exceeded) also carries the byte totals, in the 422 body, the audit row and the marker (... op=capture retained=<n> quota=<n> incoming=<n>); compare them, or read GET /api/v1/account/flow-artifacts/usage at any time, as described in Evidence storage. The proxy body limit (gateway.proxy.bodySize, which sets both the ingress proxy-body-size annotation and the console nginx client_max_body_size) must stay above both WORKSPACE_SNAPSHOT_MAX_BYTES and FLOW_EVIDENCE_MAX_BYTES. The client sends each archive as the request body without further encoding, so the body on the wire is never larger than the matching cap.

Restore occurs before setup or agent startup on Docker and Kubernetes. It logs the age of the checkpoint it recovered (PRELOOP_CHECKPOINT restored age_seconds=... created_at=...), so work lost to node loss is visible rather than assumed to be zero. Source, each repository's branch, head and upstream base commit, staged/unstaged edits and required untracked files are retained. Reproducible dependencies, known credential locations, environment files, symlinks, git configuration (including submodule and worktree copies, which carry the same remote URL credential) and native session directories are excluded. This exclusion list is not a content-level secret detector; keep production credentials out of implementation workspaces. The trusted clone configuration recreates remotes. Divergent/newer remote commits are detected without checking out or overwriting the restored local branch. A restored branch that has never been pushed can continue when the remote confirms that the branch is absent. Authentication/network errors remain explicit remote_unavailable failures rather than being confused with divergence or absence. A branch identity mismatch blocks recovery. Missing, corrupt or expired checkpoints fail resume explicitly. Automatic cold branch fallback is disabled because a remote branch does not prove unpublished local work was preserved. Private executors never receive hosted artifact capabilities.

WORKSPACE_SNAPSHOT_MAX_BYTES bounds compressed uploads; FLOW_ARTIFACT_EXPANDED_MAX_BYTES bounds extraction; FLOW_ARTIFACT_ACCOUNT_QUOTA_BYTES bounds retained encrypted account payloads. Retention uses WORKSPACE_SNAPSHOT_TTL_HOURS; zero expires on the next cleanup pass. Downloads take a lease so cleanup cannot remove their payload during restore. Metadata remains with availability expired after bytes are removed. Use retention appropriate to the review window, and surface expiry instead of claiming a resumable session remains available.

Private workspaces remain in the runner's local configuration directory with restricted directory/file permissions. Operators are responsible for host disk encryption. PRELOOP_RUNNER_WORKSPACE_MAX_BYTES bounds retained workspace bytes (default 4 GiB); oldest unleased directories are removed first and a local .expired tombstone distinguishes expiry/quota loss from a missing runner. PRELOOP_RUNNER_WORKSPACE_TTL_HOURS=0 disables retention; active jobs are protected. Cleanup also runs during idle heartbeats. No private workspace is uploaded by this transport. A continuation of a persisted workspace is leased only to the runner that holds it: while that runner is offline the execution stays queued with a message naming it, and after the queue deadline it fails for operator action rather than moving to another host. If a job does reach a runner without the workspace it was told to resume, the runner refuses it (workspace_recovery_unavailable) instead of cloning cold and dropping the unpublished work; the local .expired tombstone distinguishes retention or quota loss from a wrong host.

Validation

The repository includes archive security, tenant/thread isolation, encrypted roundtrip, lease/expiry, setup timeout, cache invalidation and private runner contract tests. scripts/tests/flow_environment_integration.py runs real SQL and Chromium checks through the hosted executor, checkpoints through HTTP, kills the sandbox, and checks exact unpushed/dirty/untracked recovery in a new one. Supply a migrated disposable DATABASE_URL, image digests and, for Kubernetes, an explicit disposable --kubeconfig. It never invokes a model. Set PRELOOP_DISABLE_TELEMETRY=true for all such tests and setup scripts.

This fixture verifies the generic runtime and recovery contract. The SQL/browser probe is an optional example, not a required application stack or a claim that a project's full end-to-end suite passes. Run the repository's relevant verification commands separately; missing dependencies and unavailable checks remain explicit blocked evidence. Use immutable repository@sha256:<digest> references for both profile and service images. Raw private-runner image overrides do not require a named profile registry; named private profiles remain unsupported until the runner advertises that protocol.