tbcore / docs / runbook

Run an agent experiment with tbcore

• tbcore 0.2.0
• Python 3.11.4+
• Docker 20.10+
• Checked against release-prep, 15 Oct 2026

uv sync
uv run python examples/experiment_workflow.py --store .tbcore/example

Prepare the host #

Steps 1 to 4. Once per machine, and again after upgrading tbcore.

1. 01

   Check prerequisites #

   You need uv, Python 3.11.4 or newer (the trial executor copies results out with tarfile's extraction filters, which arrived in 3.11.4; uv downloads one if none is installed), and a running Docker daemon at 20.10 or newer. The proxy starts with --pull=never, which older engines reject.

   shellCopy

   uv --version
   docker version --format '{{.Server.Version}}'

   ExpectTwo version numbers. An error mentioning docker API or dockerDesktopLinuxEngine means the daemon is not running: start Docker Desktop, or sudo systemctl start docker on Linux.

   Mark step 1 done
2. 02

   Install with uv #

   Install from a checkout so the bundled examples and scripts are on hand; uv add tbcore works too if you only need the library. uv sync creates .venv and installs tbcore in editable mode. The dev extra brings pytest, ruff, and mypy; drop --extra dev if you only run experiments.

   shellCopy

   uv sync --extra dev
   uv run python --version
   uv run python -c "import tbcore; print(tbcore.__version__)"

   ExpectA Python version of 3.11.4 or newer, then 0.2.0. Every later command in this runbook goes through uv run, which uses this environment.

   Mark step 2 done
3. 03

   Build the sandbox and proxy images #

   The build contexts ship inside the package. The proxy image is the only container that ever sees your API keys, so tbcore will not pull it from a registry: it has to exist locally.

   shellCopy

   uv run python -c "import tbcore; print(tbcore.build_default_images())"
   docker image ls tbcore/*

   Expect{'sandbox': 'tbcore/sandbox:latest', 'proxy': 'tbcore/proxy:latest'}, then both images in the listing.

   Note

   Upgrading tbcore does not rebuild images or touch running containers. Re-run this step after every upgrade, and record the new image IDs before switching workloads over. An old proxy image is refused with a "speaks protocol" error, so you can't forget for long.

   On a released version

   Skip the build and pull the published, digest-pinned images instead: images = tbcore.pull_default_images(), then pass images["sandbox"] and proxy_image=images["proxy"] to Sandbox. Unreleased checkouts and offline hosts keep building.

   Mark step 3 done
4. 04

   Prove the host works with the fixture #

   The example runs the same tiny task twice (baseline writes before, candidate writes after), grades both, exports the candidate, and replays it in a fresh store. It reuses the image you just built.

   shellCopy

   uv run python examples/experiment_workflow.py --store .tbcore/example --image tbcore/sandbox:latest

   ExpectThree run IDs, plus .tbcore/example/comparison.md, .tbcore/example/candidate-bundle.json, and a replay/ store next to them. Quality reads 0/1 for baseline and 1/1 for candidate. One trial each is a plumbing check, not evidence about anything.

   Mark step 4 done

Run an experiment #

Steps 5 to 13. The Python snippets build one script, experiment.py, top to bottom.

The running example: an agent gets a small repo with a broken parser.py and has to fix it. Baseline and candidate differ only in the system prompt. Each runs five seeds. A grader image runs the hidden tests against the returned parser.py.

1. 05

   Build your agent image #

   Trials run with no general network access, so pip install inside the container will fail. Bake the agent's dependencies into an image built on the tbcore sandbox. Pin every version in requirements.txt; a floating SDK version is a changed environment between runs.

   agent-image/DockerfileCopy

   FROM tbcore/sandbox:latest
   USER root
   COPY requirements.txt /tmp/requirements.txt
   RUN pip install --no-cache-dir -r /tmp/requirements.txt && rm /tmp/requirements.txt
   USER sandbox

   shellCopy

   docker build -t parser-agent:2026-10-15 agent-image/
   docker image inspect --format '{{.Id}}' parser-agent:2026-10-15

   ExpectA sha256:... image ID. Tag with a date or version, never latest, so a rebuild can't silently change what a rerun uses.

   Mark step 5 done
2. 06

   Open a store with a capture policy #

   The default policy keeps metadata only: IDs, hashes, timings, exit codes, token counts. You need content capture to replay inputs later or to keep the agent's output files, so most experiments turn it on. List every secret you know about in secret_values; matching text is redacted, and a captured workspace containing one fails loudly.

   Before you enable content capture

   Source files, outputs, and tool results will sit on disk under the store root and inside every exported bundle. Redaction only catches the exact values you list; it cannot find a secret it was never told about.

   python · experiment.pyCopy

   import json
   import os
   import subprocess
   from pathlib import Path
   
   import tbcore
   from tbcore import CapturePolicy, ExperimentStore
   
   API_KEY = os.environ["ANTHROPIC_API_KEY"]  # host only; never passed to the sandbox
   ROOT = Path(".tbcore/parser-exp")
   
   store = ExperimentStore(
       ROOT,
       CapturePolicy(content_enabled=True, secret_values=(API_KEY,)),
   )

   Expect.tbcore/parser-exp/experiments.sqlite3 appears. Use one store root per experiment and one writing process per root.

   Mark step 6 done
3. 07

   Freeze the inputs and the environment #

   A Git commit isn't enough when the task directory is dirty, so tbcore snapshots the files themselves, untracked ones included. .git, .venv, and .agent are skipped by default; symlinks are rejected rather than followed. Then record what the run will execute on.

   python · experiment.pyCopy

   from tbcore import EnvironmentProvenance
   
   snapshot = store.capture_workspace("tasks/fix-parser", exclude=("*.log",))
   
   IMAGE = "parser-agent:2026-10-15"
   image_id = subprocess.run(
       ["docker", "image", "inspect", "--format", "{{.Id}}", IMAGE],
       capture_output=True, text=True, check=True,
   ).stdout.strip()
   
   env_ref = store.record_environment(
       EnvironmentProvenance(
           image_id=image_id,
           registry_digest=None,
           tbcore_version=tbcore.__version__,
           agent_cli_version="parser-agent-0.3.1",
           model="claude-sonnet-5-5",
           provider="anthropic",
           resource_limits={"memory": "512m", "cpus": "1.0"},
           runtime="runc",
           policy_config={"network": "proxy-only"},
       )
   )

   Expectsnapshot.snapshot_id and env_ref are plain strings. Binary files in the task directory need ExperimentStore(..., allow_binary_workspaces=True), because binary bytes can't be redacted.

   Mark step 7 done
4. 08

   Define the trials #

   One TrialCase per task, configuration, and seed. Seeds are your statistical samples. Retries are not: a retry is another attempt at the same trial and never counts twice.

   python · experiment.pyCopy

   from tbcore import TrialCase
   
   CONFIGS = {
       "baseline": {"system_prompt": "prompts/v1.txt"},
       "candidate": {"system_prompt": "prompts/v2.txt"},
   }
   cases = [
       TrialCase(
           "fix-parser", name, snapshot.snapshot_id, config,
           seed=seed, environment_ref=env_ref, dataset_version="parser-v2",
       )
       for name, config in CONFIGS.items()
       for seed in range(1, 6)
   ]

   ExpectTen cases. If you edit a configuration later, also change the experiment ID in step 9. Under the same ID, old and new runs share a group and the comparison would count both.

   Mark step 8 done
5. 09

   Run the trials in the sandbox #

   Each attempt gets a fresh copy of the snapshot. The wrapper drops the trial's configuration in as config.json for the agent to read. With inject_headers, the API key stays in the proxy container: the agent sees ANTHROPIC_BASE_URL pointing at the proxy and a placeholder key, and the proxy reports token usage per attempt.

   This step spends money

   Every model call is billed to the key's account. proxy_budget applies per attempt and is an admission limit: calls already in flight can push past it. Ten attempts at 20 calls each is a ceiling of 200 calls, not 20.

   python · experiment.pyCopy

   from tbcore import SandboxTrialExecutor, TrialRunner
   
   executor = SandboxTrialExecutor(
       store, IMAGE, ("python", "agent.py"),
       inject_headers={"api.anthropic.com": {"x-api-key": API_KEY}},
       proxy_budget={"max_model_calls": 20, "max_total_tokens": 60_000},
   )
   
   def execute(case, workspace, attempt_id, context):
       (workspace / "config.json").write_text(json.dumps(case.configuration), encoding="utf-8")
       executor(case, workspace, attempt_id, context)
   
   runs = TrialRunner(
       store, "parser-prompt", execute,
       output_paths=("parser.py",), max_workers=2, max_attempts=2,
   ).run(cases)
   
   for run in runs:
       print(run.run_id, run.status)

   ExpectTen lines ending in completed. Anything else is visible on purpose: failed, timed_out, cancelled, and interrupted attempts stay in the store under their trial. The agent command has 60 seconds per attempt (the sandbox default); a slower agent needs its own executor with a longer Sandbox(timeout_s=...).

   Mark step 9 done
6. 10

   Grade the outputs #

   Grading reads stored artifacts, so you can re-grade without rerunning the agent. A command grader runs in its own read-only container with no network and no keys; the run's artifacts are copied into /work, and exit code 0 means pass. Keep the grading code in the grader image, outside anything the agent could write.

   python · experiment.pyCopy

   from tbcore import EvaluationSpec, Evaluator
   
   GRADER = EvaluationSpec(
       "parser-tests", "3",
       command=("python", "/grader/run_tests.py"),
       image="parser-grader:2026-10-15",
   )
   grader = Evaluator(store)
   for run in runs:
       result = grader.evaluate(run.run_id, GRADER)
       print(run.run_id, result.verdict)

   Expectpass or fail per run. error means the grader itself broke (timeout, missing image, missing artifact); that is kept separate from the agent's own status. For exact-match checks use EvaluationSpec(..., expected_text={"answer.txt": "42\n"}), which needs no image.

   Mark step 10 done
7. 11

   Price the runs #

   Prices come from a card you write and date, so an estimate always says where its numbers came from. Rates are USD per million tokens. Copy them from the provider's price list on the day you run, not from this page.

   python · experiment.pyCopy

   from tbcore import CostLedger, PriceCard, TokenRates
   
   CARD = PriceCard(
       version="anthropic-2026-10-15",
       source="Anthropic price list, read 2026-10-15",
       effective_date="2026-10-15",
       currency="USD",
       rates={"claude-sonnet-5-5": TokenRates(input="2", output="10", cache_read="0.10", cache_write="2.50")},
   )
   ledger = CostLedger(store)
   for run in runs:
       ledger.price_run(run.run_id, CARD)
       totals = ledger.run_totals(run.run_id)
       print(run.run_id, totals.known_subtotal, totals.unknown_calls, totals.total)

   Expectunknown_calls of 0 and a numeric total. If total is None, at least one call had missing usage or a model ID that isn't in rates. That's deliberate: tbcore will not turn an unknown into zero. New prices later? Call price_run(..., recalculate=True) with a new card; the original estimate stays.

   Mark step 11 done
8. 12

   Compare baseline and candidate #

   The comparison takes one attempt per trial, pairs trials by task and seed, and refuses to mix dataset or evaluator versions.

   python · experiment.pyCopy

   from tbcore import compare_experiments
   
   report = compare_experiments(
       store, "parser-prompt", "baseline", "candidate",
       "parser-v2", "parser-tests", "3",
   )
   (ROOT / "comparison.md").write_text(report.to_markdown(), encoding="utf-8")
   (ROOT / "comparison.json").write_text(report.to_json(), encoding="utf-8")

   Read the denominators before the rates. Quality, latency, tokens, and cost each have their own, and anything excluded is counted. The success interval is a Wilson 95% interval; the paired difference is an effect estimate, not a significance test. With five seeds a side the interval will be wide. That's the honest answer, not a bug.

   Expectcomparison.md showing 5 trials per side. Fewer means some trials failed execution or grading; the report says which.

   Mark step 12 done
9. 13

   Export, close, and check for leftovers #

   A bundle carries one run with everything needed to replay it elsewhere: spec, events, artifacts, workspace blobs, and environment provenance. Close the store before you copy or back it up.

   python · experiment.pyCopy

   best = runs[-1]
   bundle = store.export_run(best.run_id)
   (ROOT / f"{best.run_id}.bundle.json").write_text(json.dumps(bundle, indent=2), encoding="utf-8")
   store.close()

   shellCopy

   docker ps -a --filter name=tbcore- --format '{{.Names}}  {{.Status}}'
   docker network ls --filter label=owner=tbcore

   ExpectBoth listings empty. Leftovers mean a process died mid-run; see R5. Treat the bundle like a backup of the store: it holds whatever content you captured.

   Mark step 13 done

Verify the result #

Tick these before you quote a number from the comparison to anyone.

• docker image ls shows the exact agent and grader tags recorded in step 7, and the image IDs still match.
• Every run is completed, or you know why it isn't and the report counts it as excluded.
• The comparison's denominators equal the number of seeds you queued per side.
• unknown_calls is 0 for every run whose cost you plan to compare.
• No tbcore- containers or networks are left behind.
• You opened the exported bundle and checked it for anything sensitive before sharing it.

Recover #

Short procedures for the things that go wrong between runs. Each one is safe to follow from the top.

uv run python experiment.py

mkdir -p ~/tbcore-backups
cp -a .tbcore/parser-exp ~/tbcore-backups/parser-exp-$(date +%Y%m%d-%H%M)New-Item -ItemType Directory -Force "$HOME\tbcore-backups" | Out-Null
Copy-Item -Recurse .tbcore\parser-exp "$HOME\tbcore-backups\parser-exp-$(Get-Date -Format yyyyMMdd-HHmm)"

docker ps -a --filter name=tbcore- --format '{{.Names}}  {{.Status}}  {{.CreatedAt}}'
docker network ls --filter label=owner=tbcore

docker rm -f -v tbcore-sess-<id>
docker network rm tbcore-net-<id>

import json
from pathlib import Path

from tbcore import CapturePolicy, ExperimentStore

# the file written in step 13, copied over from the original host
bundle = json.loads(Path("RUN_ID.bundle.json").read_text(encoding="utf-8"))
replay = ExperimentStore(".tbcore/parser-replay", CapturePolicy(content_enabled=True))
replay.import_bundle(bundle)

Troubleshoot #

Search this table for the first line of the error.

What tbcore promises, and what it doesn't #

Background for when you have time. These are the places people over-read the results.

Reference #

Files tbcore writes

Docker names

Model config file

Optional. Copy tbcore.example.toml, then load it by path with load_model_config("tbcore.toml") and pass Client(name, config=config). tbcore never finds the file on its own. It sets endpoints, the name of the key variable (api_key_env), and prices; a literal api_key is an error. A config entry wins over a built-in of the same name, and a name in neither raises instead of being sent as a raw model id. allow_hosts and inject_headers are not read from it.

Environment variables