Agent pod isolation¶

Editions: OSS, Cloud, Enterprise. Unless stated otherwise, everything on this page ships in OSS.

An agent pod runs code a model wrote, on a repository the model can edit, with a credential the platform minted for it. It is the least trusted thing in the deployment. This page states what such a pod can reach, what the chart's NetworkPolicy changes, and what is still open afterwards.

What runs, and where¶

The executor creates one Job per flow execution (backend/preloop/agents/container.py). Its pod carries the labels app=agent-execution, preloop.agent_type, preloop.flow_id, and preloop.execution_id. The namespace comes from AGENT_EXECUTION_NAMESPACE: the chart points it at a dedicated namespace when agentExecution.namespace.create is true, and at the release namespace otherwise. The second case is the easy one to deploy and the dangerous one to leave unguarded, because the database, NATS, the console, and the other tenants' agent pods are then neighbours.

The NetworkPolicy below is a network boundary; it does not change the kernel an agent shares with the host. A stronger boundary is a sandbox runtime: set agentExecution.runtimeClassName to a RuntimeClass the cluster already offers (for example kata-containers, gvisor, or firecracker) and every agent pod is scheduled into a VM or a user-space kernel, so a container escape reaches a guest kernel instead of the node. Pair it with agentExecution.nodeSelector and agentExecution.tolerations when only one tainted node pool runs that runtime. The same placement is applied to the hosted publication verifier Job, which also executes repository code, while Preloop's own deployments stay where they are. The settings are empty by default and render no change to the pod spec, so an existing install keeps the node's default runtime.

Nothing in the cluster dials into an agent pod. Output leaves it two ways:

the pod log stream, which the runner reads through the API server (read_namespaced_pod_log), carrying the result artifact, the evidence archive, and the workspace snapshot as base64 lines;
direct uploads made by the pod itself, when checkpointing or evidence upload is enabled (backend/preloop/services/checkpoint_runtime.py), which are outbound HTTP calls to the public Preloop URL.

The kubelet does dial in. Flows with an environment profile get native sidecar containers in the same pod, each with a TCP startupProbe (_environment_sidecars in container.py), and the kubelet runs that probe from the node's own network namespace. So the policy denies ingress from every pod and relies on the CNI admitting the local host, which the three CNIs the chart names all do:

Cilium: host-to-local-endpoint traffic is governed by --allow-localhost (default auto), not by NetworkPolicy; the Cilium variant below also names the host entity explicitly.
Calico: "Calico allows connections the host makes to the workloads running on that host. Some orchestrators like Kubernetes depend on this connectivity for health checking the workload." (Calico docs, Protect hosts.)
Antrea: node-to-local-pod traffic "will always be allowed to make sure that agents on a Node (e.g. system daemons, kubelet) can communicate with all Pods on that Node to perform liveness and readiness probes" (Antrea network policy docs; the OVS pipeline marks probe packets and bypasses the ingress tables for them).

On a CNI that does enforce node-sourced traffic, list the node addresses in agentExecution.networkPolicy.nodeCidrs and the policy re-admits them on ingress; nothing else is admitted either way. A probe that fails under the policy shows up as a sidecar that never becomes ready and a Job that ends with the sidecar's startupProbe failure in kubectl describe pod.

Without a NetworkPolicy¶

A cluster with no policy gives every pod a flat network. From an agent pod that means:

preloop-db-rw:5432, with whatever credentials it can find. If the chart was installed with its default values, those credentials are also the defaults, and the connection string used to sit in the API pod spec, which any in-cluster reader could fetch.
preloop-nats:4222, the event bus every worker reads from.
the console and the API, on every port, not just HTTP.
other agent pods, which belong to other flows and possibly other accounts.
the cloud metadata service, if the node exposes one.

None of that is needed to run an agent.

With the policy¶

helm/preloop/templates/agent-networkpolicy.yaml selects pods labelled app=agent-execution and:

denies all ingress (or admits only nodeCidrs, see above);
allows egress to the DNS pods on 53, selected by agentExecution.networkPolicy.dns.* (kube-dns in kube-system by default; Helm merges maps, so a cluster whose DNS lives elsewhere nulls the default label keys and adds its own, as the values comment shows);
allows egress to the API and gateway pods on the HTTP ports. Both 80 and 8000 are listed: a ClusterIP connection is translated to the pod IP and target port before policy is evaluated on most CNIs, so a rule naming only the service port silently drops the traffic;
allows egress to 0.0.0.0/0 minus the cluster pod and service CIDRs (agentExecution.networkPolicy.clusterCidrs; when that is empty the deprecated excludeCIDRs, and when both are empty the three RFC1918 ranges plus 169.254.169.254/32), which is what model providers, git remotes, and package registries need;
allows nothing else in-cluster. The database, NATS, the console, and other agent pods fall off the list.

agentExecution.networkPolicy.extraEgress takes verbatim rules for the endpoints a particular deployment needs (an in-cluster registry, a node-local DNS cache on a link-local address, an internal artifact store).

A second, opt-in policy (agentExecution.networkPolicy.controlPlaneIngress) says the same thing from the API and gateway side: traffic from app=agent-execution pods is accepted on the HTTP ports only, everything else is accepted as before. It needs the cluster pod CIDRs, because an ingress rule that did not re-admit node-sourced traffic would also cut off load balancer health checks and host-network ingress controllers.

Policies only do something on a CNI that enforces them. Cilium, Calico, and Antrea do. On a CNI that does not, these objects render and have no effect.

Upgrading from the previous chart¶

The previous template read excludeCIDRs and additionalEgressRules. The new keys, clusterCidrs and extraEgress, ship empty and fall back to the old ones, so a release that customised either list (a cluster on 100.64.0.0/10, say) keeps it across the upgrade without touching values. Helm lays saved values over the new chart's defaults, which is why the new defaults have to be empty: a populated clusterCidrs default would win over a saved excludeCIDRs every time. helm upgrade prints a notice while the deprecated key is the one in use.

Cilium¶

Cilium evaluates policy against identities, not addresses, and by default does not match a NetworkPolicy ipBlock against anything inside the cluster: "By default, ipBlock rules in NetworkPolicy do not match intra-cluster IPs (such as Pod or Node IPs). Setting the --policy-cidr-match-mode option (or equivalent Helm value policyCIDRMatchMode) to pods or nodes allows ipBlock rules to match intra-cluster IPs." (Cilium docs, Kubernetes NetworkPolicy.)

The consequence for the plain policy: the 0.0.0.0/0 rule never matches a node. When the deployment's public URL resolves to an address that an ingress or load balancer terminates on a node, which is the usual shape of a default install, an agent pod calling that URL is calling a node, and the plain policy drops it. Checkpoint uploads, evidence uploads, and any flow that uses the public URL for MCP or the gateway fail with a connection timeout while the database stays correctly unreachable.

agentExecution.networkPolicy.cilium.enabled=true renders a CiliumNetworkPolicy in place of the plain object, with the same allow list expressed in Cilium's terms:

ingress: fromEntities: [host], the kubelet, and nothing else (cilium.allowHostIngress=false denies all ingress instead, with an explicit ingressDeny from all);
egress to the DNS pods, selected by namespace and pod label (k8s:io.kubernetes.pod.namespace plus dns.podSelectorLabels);
egress to the API and gateway pods, selected by the chart's labels in the release namespace, on controlPlanePorts. Every label key inside these toEndpoints selectors carries the k8s: source prefix. Cilium reads a bare key there as any:, which matches the label from any source rather than the pod label alone; the prefixed form is the one Cilium documents, and the render checks fail on a bare key;
egress to toEntities: [world, host] (cilium.internetEntities). world is every address outside the cluster, so the CIDR carve-out is unnecessary: the database, NATS, the console, and other pods are cluster identities and never match. host is the node the pod runs on, which is where the public hairpin lands. Add remote-node if that address can land on a different node than the one running the pod;
egressDeny: toCIDR for cilium.egressDenyCidrs, by default the cloud metadata address, which Cilium classes as world and which a deny rule removes regardless of the allow above;
cilium.extraEgress, appended verbatim. These are CiliumNetworkPolicy egress rules; the plain extraEgress list is not translated, and label keys inside their toEndpoints should carry k8s: like the chart's own.

The namespace-wide default deny for a dedicated agent namespace stays a plain NetworkPolicy in both modes; it selects pods, not addresses, so Cilium enforces it as written. controlPlaneIngress should stay off on Cilium, since it relies on an ipBlock to re-admit node-sourced traffic to the API and gateway.

The credential an agent carries¶

create_flow_runtime_token (backend/preloop/services/flow_runtime_token.py) mints one API key per execution, named Flow Execution <execution id>:

scopes mcp:read and mcp:write;
two hour expiry;
context_data binding it to the flow, the execution, the runtime session, and the flow's allowed MCP servers and tools;
owned by the account's primary active user, because MCP tools act on that account;
revoked when the execution ends, by execution rather than by key id, so an execution handed between workers does not leave a live key behind (revoke_flow_runtime_tokens).

The key reaches the pod as PRELOOP_API_TOKEN (and inside MCP_CONFIG_JSON) only when the flow allows MCP servers or tools.

The limits worth knowing:

The key is limited to MCP scopes. A key whose scopes are all mcp:* authenticates on the MCP endpoint and on the runtime routes that check their own credentials (model gateway, agent control WebSocket, permission checks, operator note pull, browser steps, artifacts). The generic REST dependency answers 403 with detail.code api_key_scope_denied for every other /api/v1 route, and the console WebSockets refuse it (backend/preloop/api/auth/key_scopes.py). API_KEY_SCOPE_ENFORCEMENT switches this to audit (log and allow) or off; the default is enforce.
MCP capabilities are checked before dispatch. For scoped API keys, mcp:read permits discovery and resource/prompt reads; mcp:write permits tool invocation, including tools annotated as read-only. Neither implies the other. These checks also apply on middleware re-entry and approval replay. Legacy personal keys without scopes and the existing * grant retain access subject to account and tool authorization. Discovery, telemetry and enrollment-only credentials cannot authenticate MCP or runtime channels. The REST rollout setting above does not disable these capability checks.
The key dies with its execution, on most routes. Beside the revocation above, the generic REST dependency, the MCP endpoint and the model gateway refuse a flow execution key once its execution has reached a terminal status, so a missed revocation does not leave a live key there. Parked executions keep their key. Two surfaces skip this check by design and accept the key until revocation or expiry: the browser-step flush (adapters finish flushing after the run ends) and the runtime routes that also serve durable managed-agent credentials (agent control WebSocket and operator note pull).
The tool allow list is checked on every call. allowed_mcp_tools limits both the tools the MCP layer lists and the tools it will run for the key (backend/preloop/services/dynamic_fastmcp.py). allowed_mcp_servers shapes which servers are offered. Codex does not open a Preloop MCP session when both lists are empty, so an unused client cannot reconnect until the flow timeout.
Shell is a separate control. agent_config.sandbox_type: read-only launches Codex with -c 'sandbox_mode="read-only"' and -c 'approval_policy="never"', disables the shell_tool feature, and does not pass --yolo. These overrides work for both exec and exec resume, so reminders and recovery retain the same restrictions without waiting for a person. Any other value, including the preset default exec, keeps --yolo.
A silent model stream is bounded separately. agent_config.stream_idle_timeout_seconds (30..3600, default 600, capped at half the flow's timeout budget) is how long a custom-provider stream may send nothing before Codex reconnects. A run that times out on one is classified model_stream_idle, not timeout.

Within MCP, the key still acts as the account's primary user for the tools the flow allows. Giving the runtime principal its own role instead of the primary user's is the remaining backend change.

Residual risks¶

Cloud metadata service. 169.254.169.254 is link-local, so a cluster CIDR carve-out does not cover it. The chart's fallback list and the Cilium variant's egressDenyCidrs both exclude it by default; an operator who sets clusterCidrs explicitly has to keep it in the list, and the node pool should block it as well (IMDSv2 hop limit 1, GKE metadata concealment), since a NetworkPolicy cannot express a deny.
Other namespaces. The policy names the release namespace for the control plane and the internet for everything else. A workload in a third namespace with a routable ClusterIP is unreachable, but a workload reachable on a public address is not.
DNS exfiltration. Egress to kube-dns on 53 is unrestricted in content. Data can leave in query names. Restricting that needs a DNS-aware policy (CiliumNetworkPolicy toFQDNs) or an egress proxy.
Egress to the whole internet. The policy limits where an agent can go inside the cluster, not what it can post to a pastebin. Deployments that need that constraint should replace the 0.0.0.0/0 rule with an FQDN policy or route agents through a proxy (extraEgress plus the proxy env in agentExecution). A sandboxed browser uses the allowlist sidecar in environments/egress-proxy (environments/egress-proxy/README.md); that proxy, not an MCP tool list, is the network boundary.
The API is still one hop away. MCP and the model gateway are exactly what the agent is supposed to reach, so the credential above, not the network, is what limits it.
Shared namespace, shared service account. Agent Jobs do not set serviceAccountName, so they run as the namespace default account, and the token is only left unmounted in isolated publication mode (automount_service_account_token=False). A policy does not stop a pod from using a token it holds, so the default account in the agent namespace must stay unprivileged. The carve-out does help here: the API server's ClusterIP lives in the service CIDR, so the policy removes the usual path to it, and the agent manager Role (helm/preloop/templates/agent-rbac.yaml) is bound to the Preloop service account, not to the agents.
The hosted Codex command hook is enforced, within limits. Hosted Codex checks native shell commands against tool rules through the Preloop PreToolUse hook (backend/preloop/agents/codex_native_hook.py). Root writes config.toml, hooks.json, auth.json and the hook trust state, then Codex runs as the unprivileged preloop-agent user through a root-owned launcher (setpriv, no new privileges, no capabilities). The hook CLI and /opt/preloop are root-owned and not writable by that user. $CODEX_HOME is root-owned and sticky: the agent can add sessions and logs but cannot change, remove or rename the configuration files root wrote. A command that tries fails with Permission denied or Operation not permitted, and the same hook checks every later command. The workspace, the agent's home and its caches stay writable, so builds and package installs work; /root, where the base image keeps its toolchains, is traverse-only. An image without the user (published before this change) keeps the old behaviour and the run log says so in its PRELOOP_AGENT_USER line.

The execution's runtime credential stays in Codex's environment, because Codex uses it for the Preloop MCP connection and passes it to the hook. Shell commands do not inherit it (shell_environment_policy), and the node binary, every executable the Codex package ships (the codex binary and its code-mode host) and the hook CLI are execute-only (0711), so the kernel marks those processes as not dumpable and the agent user cannot read their /proc/<pid>/environ. The credentials root uses for its own steps (clone and publication tokens, the publication capability, checkpoint, evidence and session upload tokens, the MCP configuration) are removed from Codex's environment, so commands Codex runs never see them.

After the last Codex round, root takes the workspace back before its own git steps (session upload, push, pull request): it stops every agent process, makes the workspace root-owned again, restores each repository's git config from before the run, removes git hooks and the files that can redirect the configuration (commondir, config.worktree, info/attributes), and runs its git with core.hooksPath=/dev/null and core.fsmonitor=false. Hooks or configuration the agent planted do not run as root. Git remotes or upstreams the agent configured are dropped with its config changes; the publication steps set their own.

Remaining gaps:

The model key in $CODEX_HOME/auth.json is readable by the agent user, because Codex reads it as that user.
The git credential store is handed to the agent user, so its own fetches and pushes keep working as before.
Root's git still reads the agent's work tree: a .gitattributes file can name a filter or diff driver, but only the restored config could define one, so nothing the agent wrote is run.
The hook judges the command it is shown. An allowed command that starts an interpreter can do whatever that interpreter can.
Kernel or container escapes are out of scope. The pod and its credential, above, remain the outer boundary.

docs/operations/database-credentials.md: rotating the database credentials the old topology exposed.
docs/architecture/security.md: platform-wide model.