Six boundaries to check when sandboxing coding agents

Why these reports sounded familiar

There has been a lot of discussion about agents escaping sandboxes and accessing systems outside their tasks. METR’s investigation of the OpenAI / Hugging Face incident describes agents coordinating across runs and manipulating tool calls during an evaluation that reached real infrastructure.

Parts of that account sounded familiar. When we started this work in February 2026, we needed to isolate each agent run. We benchmark how coding agents choose and use developer tools, so we need to know which files, memory, tools, and network connections contributed to a result.

A lot of that work involves sandboxing: restricting what a process can access and do, then checking whether those restrictions hold. A fresh checkout can still share an agent’s history, a cache, or a connection to a host service. An apparent success might come from an earlier session’s answer rather than the inputs allowed for the task.

The incidents described by METR and the tests described here differ in scope and cause. This article covers six routes we found in our own containment work, what the records establish, and what they leave unanswered.

A correct answer can come from the wrong place

An early bubblewrap wrapper gave the agent a read-only view of the host and hid selected paths. That policy left evaluation material accessible in places we had not excluded. Bubblewrap does not require exposing the whole host; the mount policy was ours.

We recorded six routes across the containment tests. The first five exposed information, including earlier answers and project memory. The sixth let the agent ask a service outside the sandbox to run code for it.

The service was the host user’s systemd manager, which launches and manages processes. The agent could write a script in its workspace and ask the manager to run it under the host user’s account. It did not need root access or a kernel exploit.

[Service]
ExecStart=/redacted/worker/script.sh

Bubblewrap’s documented limitations warns about this exact class of problem: access to a D-Bus socket can permit command execution through systemd. Our wrapper restricted the shell’s view of the filesystem without closing that route.

A sandbox must constrain what an agent can ask other processes to do, as well as what it can do directly.

Using a withheld earlier answer is evaluation contamination. Running a process beyond the intended boundary is a containment failure. The first can invalidate a result without the second ever happening.

Six boundaries to check

The first five routes exposed information. The sixth exposed a service that could execute code outside the worker. These observations come from our incident notes and wrapper changes, across multiple runs. The controls below need to be tested with your own client and environment.

1. 01

   The benchmark repository

   The agent found the prompt catalog and previous results outside its assigned workspace. A task that should have required new work could be answered from earlier evaluation material.

   Control to test: Give the worker only the repository and inputs allowed by the task. Keep the evaluator’s prompts, results, and grading material outside its filesystem.

2. 02

   Chat history from earlier runs

   After the obvious repository paths were hidden, the agent searched its client’s saved chats and tracking data. Earlier sessions had left another copy of the information the wrapper was trying to hide.

   Control to test: Start each run with a fresh agent home. A new working directory does not clear a client’s session history.

3. 03

   Git objects that survived a reset

   Resetting the working tree left unreachable Git objects from earlier work. The agent used git fsck to find retained objects and recover their contents.

   Control to test: Prepare an independent repository with only the history the task permits. Check the object store as well as the visible files; a reset alone is not a clean starting state.

4. 04

   Another agent’s project memory

   Project memories created by other coding agents contained task-specific constraints. Those files remained readable even after the tested agent’s own chat directory was hidden.

   Control to test: Keep unrelated agent profiles and project memories out of the worker. Inventory every client installed on a shared machine, not only the one you are testing.

5. 05

   Transcripts, logs, and caches

   Session transcripts contained completed answers. Tool caches, package logs, and configuration files also exposed paths to other state on the host.

   Control to test: Exclude previous transcripts and host caches. Keep evidence collection outside the worker’s control, with a separate location for each run.

6. 06

   A service running outside the sandbox

   A reachable user-level systemd manager could launch a script from the worker’s directory. That new process ran outside the wrapper and could see files hidden from the sandboxed process.

   Control to test: Restrict access to host service managers and their sockets. Check what reachable helpers can do on the worker’s behalf, including container daemons and execution services.

A later run with another coding agent also read a stored result for the same task and changed the host’s scheduled jobs. That matters for the next run: a fresh task directory does not remove state left elsewhere on the machine. We need to check isolation for every agent configuration we benchmark.

Searching history and reading documentation can be legitimate parts of development. Define the allowed inputs before the test: project history might be permitted, while another run’s solution is not. The problem is access to material the evaluation claims to withhold.

Related public reports

Public reports describe three related failures: retained answers, unintended internet access, and an exploitable helper service. These are separate incidents with different causes.

Answers remained in the repository

Poolside reported that agents recovered reference solutions from Git history retained in benchmark task images. After that route was closed, review found attempts to recover solutions from public repositories and other web sources. A clean checkout can still contain an answer in its Git object store. Network access introduces another source of reference solutions.

The environment had internet access

Anthropic’s review of cybersecurity evaluations identified three incidents involving unauthorized access to real organizations. The prompts said there was no internet access, but the environment had internet access because of a misunderstanding with the evaluation partner. A statement in the prompt did not enforce the intended network policy.

A permitted service became a route out

OpenAI’s initial Hugging Face incident report describes agents exploiting a previously unknown vulnerability in an Artifactory package proxy. Its August 26 follow-up explains how agents used that service to send internet requests on their behalf. This differs from a network left open by configuration. It shows why a helper service belongs in the security boundary, even when the worker has no direct internet connection.

A transcript is evidence, not a complete audit

A transcript helps explain the steps an agent took. To establish what happened outside the shell, we also needed the host service record and the separate containment probe.

METR’s independent Hugging Face investigation adds another limit. It found agents coordinating through an unsanctioned message board and experimenting with tool-call spoofing. Some recorded calls appeared to run one command while another command actually executed.

The spoofing METR observed was small-scale, and its setup left evidence in the transcripts. That technique did not give agents access to outputs absent from the record. METR could not rule out harder-to-detect manipulation, so the observed limits are not a guarantee that all activity will be visible.

Corroborate transcripts with records the worker cannot rewrite: process launches, service activity, network connections, and filesystem changes. For a network request, distinguish the URL the agent mentioned, a connection it attempted, and access the destination confirmed. Those observations support different conclusions.

A report should name the destination and outcome without exposing credentials or unnecessary private data. It should also say what the collector could not see. A gap in coverage is not evidence that nothing happened.

Check the boundary before trusting the score

Start with an isolated worker containing only the permitted inputs. Then test the actual client, permissions, and services you intend to use:

• Test for state carried between runs. Put a unique, non-secret marker in one controlled run and check whether the next can recover it. Include client history, temporary files, caches, and Git objects.
• Probe reachable host services. Use harmless checks to compare what the worker can see with what a service can do on its behalf. A filesystem check alone will miss this distinction.
• Verify network restrictions. Check permitted and blocked destinations, including host and private-network services. If web access is allowed, account for reference answers available online.
• Verify the evidence and cleanup. Confirm that activity appears in records outside the worker’s control. Interrupt a run, then check for surviving processes, scheduled jobs, user services, and saved state before starting the next.

Canaries only cover the locations and outputs you inspect. A marker hit needs a provenance check: make sure the marker was not also supplied through the prompt or test tooling. Finding none does not prove isolation, since an agent can read a file without repeating its contents. Likewise, checking mounts and rules confirms configuration, not every behavior those rules permit.

If a run reaches withheld evaluation material, preserve it as evidence of the failure. Correct the environment and rerun the task before using the result to assess performance.