Getting Started

This page gets Evaluator from a benchmark config to completed scores, trajectories, and analysis reports. The path is: fill config.yaml, run setup, check the block, launch the benchmark, then open the dashboard.

You can read it as a concrete walkthrough: follow the example config snippets and commands in order, and you will see how an Evaluator run is put together.

Evaluator wraps pinned Harbor and LiteLLM environments. It does not stage Curator tasks; it resolves benchmark tasks from Harbor's registry.

Prerequisites

Before starting, make sure these are available:

• Claude Code with the Evaluator plugin loaded.
• Docker, used by Harbor to run benchmark task containers.
• uv, Python, and Python 3.13 for the Harbor and LiteLLM environments.
• An OpenAI- or Anthropic-compatible model endpoint.
• A benchmark registered in Harbor's registry.json.
• Optional vLLM serving on a GPU node, only when using a local checkpoint.

If the Evaluator slash commands are not visible in Claude Code, see FAQ.

Setup

Most Evaluator issues come from the benchmark name, model endpoint, agent runtime, or proxy port. Once config.yaml is right, the skills take over.

For a first run, leave meta_info and runtime_info.output alone. Fill the runtime_info.input pieces below. For every field, see the Configuration Guide.

Benchmark config

Evaluator is registry-driven. Pick the benchmark and version first:

runtime_info:  input:    task_source:      provider: harbor_registry      dataset_name: swebench-verified          # e.g. swebench-verified, swebench_multilingual, swebenchpro      version: "1.0"      registry_path: repos/harbor/registry.json      no_hack: true                            # hardened swebench-verified mode

For a fast first run, either choose a small benchmark variant when available or set harbor_job.n_tasks below.

Receive Trainer output

If you are evaluating a checkpoint produced by Trainer, serve it with vLLM on a GPU node and make Evaluator call that OpenAI-compatible endpoint:

runtime_info:  input:    local_model_serving:      model_path: ../trainer/artifacts/model/<run> # Trainer checkpoint directory to serve      model_name: Qwen3.5-35B-A3B                 # name exposed by the vLLM server    llm_api:      api_key: dummy-key                          # must match the vLLM server key      api_base_url: http://<GPU_NODE_IP>:8000/v1  # vLLM OpenAI-compatible endpoint      model: openai/Qwen3.5-35B-A3B               # openai/<model_name> exposed by vLLM

This handoff is not declared in meta_info.dependencies today because Evaluator can benchmark many model sources: a Trainer checkpoint, a remote API, or another local model. The important pieces are model_path, which chooses the checkpoint to serve, and llm_api, which tells Evaluator how to call it. Keep local_model_serving.model_name aligned with llm_api.model.

LLM API config

Evaluator starts a per-job LiteLLM proxy in front of your model. For a remote API, point llm_api at the shared endpoint:

runtime_info:  input:    llm_api:      api_key: <YOUR_API_KEY>      api_base_url: https://your-openai-compatible-endpoint/v1      model: openai/Qwen3.5-35B-A3B             # openai/<served-model-name>      protocols: [openai_compatible, anthropic_compatible]      served_via: per_job_litellm_proxy      input_cost_per_token: 0.0      output_cost_per_token: 0.0    litellm_proxy:      port: 4101      master_key: dummy-key-cf

For a standalone local checkpoint, use the same shape and point model_path at that model directory:

runtime_info:  input:    local_model_serving:      model_path: artifacts/models/Qwen3.5-35B-A3B      model_name: Qwen3.5-35B-A3B    llm_api:      api_key: dummy-key                         # must match the vLLM server key      api_base_url: http://<GPU_NODE_IP>:8000/v1      model: openai/Qwen3.5-35B-A3B

See Local Model Evaluation for the vLLM serving steps.

Harbor job config

Keep a first run small:

runtime_info:  input:    harbor_job:      jobs_dir: artifacts/jobs      n_concurrent: 4                           # keep small for the first run      n_tasks: 10                               # null = full benchmark      max_retries: 2      timeout_multiplier: 1

Agent config

Choose one scaffold and keep its runtime image or host path aligned:

runtime_info:  input:    agent:      name: custom-openhands-sdk                # custom-openhands-sdk | custom-claude-code | custom-opencode      version: 1.33.0      runtime_image: docker.io/jierun/c-oh-sdk-1.33.0:v0.1      runtime_host_path: artifacts/runtime/openhands-sdk      max_turns: 200      temperature: 1.0

Job analysis config

Post-eval analysis creates the artifacts used by the dashboard. Leave it on for normal runs:

runtime_info:  input:    job_analysis:      enabled: true      tag_llm:        base_url: ""                            # empty = reuse llm_api        model: ""        api_key: ""

Now run setup:

/evaluator:setup

This prepares the pinned Harbor checkout, builds the Harbor and LiteLLM environments, and checks the local runtime paths. Re-running it is fine.

Check

Before launching the benchmark, run:

/evaluator:check

This is read-only. It checks config, Harbor repo state, environments, benchmark registry, LLM endpoint, proxy port, Docker, and agent runtime. For exact pass conditions, see Validation Checks.

Fix required failures before continuing. If you are using a local checkpoint, also make sure the vLLM endpoint is already serving before the check.

Run Evaluator

Launch the benchmark:

/evaluator:run

Evaluator starts the per-job LiteLLM proxy, builds the Harbor command from config.yaml, runs the selected benchmark, captures trajectories and verifier results, then runs post-eval analysis.

Behind the command, the pipeline is:

STEP 0  Resolve benchmark      Harbor registry -> task set
STEP 1  Start model proxy      LiteLLM per-job proxy -> trajectory logger
STEP 2  Run Harbor trials      task containers -> rewards and trajectories
STEP 3  Analyze job            failure reports -> dashboard artifacts

The main outputs are:

artifacts/jobs/<job>/result.json
artifacts/jobs/<job>/<task>/agent/litellm-trajectory.jsonl
artifacts/jobs/<job>/<task>/verifier/report.json
artifacts/jobs/<job>/analysis/

For benchmark selection and proxy details, see Select Benchmark and LiteLLM Proxy. For the full artifact layout, see Output Format.

Final outputs

Evaluator is the terminal block in the current LegoFlow pipeline, so there is no next block to configure. Its outputs are the final benchmark artifacts:

runtime_info:  output:    eval_results_dir:      path: artifacts/jobs                         # benchmark jobs, trajectories, verifier reports, and analysis      results_summary_format: artifacts/jobs/<job>/result.json      trajectory_format: artifacts/jobs/<job>/<task>/agent/litellm-trajectory.jsonl

Use these paths for reporting, dashboard review, or any custom analysis you add after evaluation.

Dashboard visualization

After an Evaluator run starts or finishes, open the dashboard:

/evaluator:dashboard

The dashboard helps browse jobs, benchmark scores, trajectories, verifier reports, and post-eval analysis. It reads artifacts/jobs/ directly, so keep the job directory and run log around when debugging.

For the internal Evaluator mechanism, see Design.