Overview

Evaluator runs registered benchmarks through Harbor against a configurable upstream LLM, then stores per-task results and trajectories.

It is the evaluation stage of the LegoFlow pipeline, and it has no downstream consumer. Where tracer generates training data and trainer produces checkpoints, Evaluator measures a model on a chosen benchmark. It uses a runtime shell similar to Tracer: a pinned Harbor checkout behind a per-job LiteLLM proxy. The difference is that Evaluator loads tasks from Harbor's registry.json, not from a local Curator export.

Evaluator provides:

• Registry-driven benchmark selection across SWE-bench Verified, Multilingual, Pro, Terminal-Bench, and other Harbor benchmarks.
• A per-job LiteLLM proxy that exposes OpenAI- and Anthropic-compatible endpoints and records trajectories.
• Containerized rollouts through Harbor, with configurable concurrency, retries, and timeouts.
• Pluggable agent scaffolds, including Claude Code, OpenHands SDK, and OpenCode.
• Remote API or local-checkpoint evaluation with the same block workflow.
• Post-eval job analysis for attribution, scoring, and failure breakdowns.
• A dashboard for browsing jobs, reports, and trajectories.

Where to go next

• Getting Started: install the dependencies and run your first benchmark.
• Design: see how Evaluator resolves benchmarks, starts the proxy, runs Harbor, and analyzes jobs.
• Usage: configure benchmarks, proxying, local models, and job analysis.
• Output Format: find scores, trajectories, verifier reports, analysis, and archives.
• Dashboard: browse jobs, reports, and trajectories.