Design

Evaluator is the measuring instrument of the pipeline. It is a leaf block with no downstream consumer, and it produces no training data: its whole job is to say how good a model is on a chosen benchmark, and to leave behind enough evidence to explain the number. That framing drives the decisions below, because a measurement is only useful when the conditions it was taken under are known and repeatable.

Repositories

Evaluator runs one pinned external repository. scripts/update_repos.sh checks it out at the commit recorded in meta_info.repos in config.yaml, and the checkout is read-only: local edits are overwritten on the next update, so changes belong upstream.

Tracer pins its own Harbor checkout separately, so the two blocks can sit on different commits without disturbing each other.

1. The Benchmark Comes from a Registry

Unlike Tracer, Evaluator stages no tasks locally. It names a benchmark and lets Harbor resolve it:

runtime_info:
  input:
    task_source:
      provider: harbor_registry
      dataset_name: swebench-verified
      version: "1.0"
      registry_path: repos/harbor/registry.json

The reason is comparability. A benchmark that is copied and edited locally stops being the benchmark everyone else reports against, so switching one here is two config fields and nothing else. The cost of that choice is real: the task set does not sit in the block where you can inspect it, and it moves when the Harbor pin moves. Treat the pin as part of the result, and confirm that a new commit still carries your (dataset_name, version) before bumping it.

2. Hardening is a Network Decision

Set task_source.no_hack: true and Evaluator remaps SWE-bench Verified onto a local hardened registry, then gives the agent an egress allowlist limited to the LiteLLM host.

The threat model is specific: an agent with open network access can look up the upstream fix instead of deriving it, and the resulting score measures retrieval rather than engineering. Egress is restricted on the agent side only. The verifier keeps public access, because it legitimately needs to resolve dependencies to run the tests.

3. Every Job Records Its Own Conditions

Each run gets its own LiteLLM proxy, generated from the block config into that job's directory with Harbor's trajectory logger attached. Alongside it, Evaluator copies the resolved config.yaml into the job directory.

Both come from the same idea: a score is meaningless without the conditions that produced it. The snapshot keeps a job interpretable after the block config has moved on, and the per-job proxy means two evaluations running at once cannot quietly share state or model settings. The proxy exposing both OpenAI- and Anthropic-compatible paths is what lets different agent scaffolds share one evaluation shell; see Supported Scaffolds.

4. Pass or Fail is Decided by the Verifier

Harbor runs one trial per benchmark task and writes the agent side and the verifier side separately:

artifacts/jobs/<job_name>/<task_id>/
├── agent/
│   └── litellm-trajectory.jsonl
├── verifier/
│   └── report.json
└── result.json

The separation matters. The agent's own account of what it did carries no authority over the outcome; the verifier's test run does. When a task fails, both sides are on disk, which is what makes it possible to tell an agent that gave up from an environment that never built.

Concurrency, task cap, retries and timeout multiplier all come from runtime_info.input.harbor_job, so a smoke run and a full benchmark run take the same code path and differ only in numbers.

5. Analysis is Layered on Top, Never Underneath

After Harbor finishes, scripts/analyze_job.sh attributes failures, tags tasks and builds the reports the dashboard reads, into analysis/ inside the job directory.

It is deliberately non-fatal and deliberately separate. A completed evaluation is a fact; an analysis of it is an interpretation, and can be rerun, improved, or run against old jobs at any time. If analysis fails, or its gold dataset cannot be generated, the raw job and its score are untouched.

For the shortest runnable path, see Getting Started. For the exact files written by each stage, see Output Format.