Configuration Guide
Evaluator config controls benchmark selection, model endpoint, agent scaffold,
job scale, and optional post-run analysis. Most runs only need a few
runtime_info.input fields. Leave meta_info, repository pins, environment
paths, and output declarations alone unless you are changing the block contract
itself.
Common fields
| Area | Edit when | Main fields |
|---|---|---|
| Model endpoint | You evaluate a remote API or a locally served checkpoint. | llm_api.*, local_model_serving.* |
| Benchmark source | You change the benchmark or Harbor registry entry. | task_source.dataset_name, version, registry_path, no_hack |
| Evaluation scale | You run a smoke benchmark or a full benchmark. | harbor_job.n_tasks, n_concurrent, max_retries, timeout_multiplier |
| Agent scaffold | You switch the coding agent used for evaluation. | agent.name, version, runtime_image, runtime_host_path |
| Remote machine | You want evaluation to run on another benchmark server. | meta_info.resources.ip, user, pwd, directory |
| Post-run analysis | You want rubric and tag-level analysis after evaluation. | job_analysis.enabled, tag_llm.* |
An example reference
Start by evaluating a locally served checkpoint on SWE-bench Verified with a small task cap. This checks the model endpoint, Harbor runtime, agent scaffold, and analysis path before a full benchmark.
runtime_info: input: llm_api: api_key: <YOUR_API_KEY> api_base_url: http://127.0.0.1:8000/v1 model: openai/Qwen3.5-35B-A3B protocols: [openai_compatible, anthropic_compatible] served_via: per_job_litellm_proxy local_model_serving: model_path: artifacts/models/Qwen3.5-35B-A3B model_name: Qwen3.5-35B-A3B task_source: provider: harbor_registry dataset_name: swebench-verified version: "1.0" registry_path: repos/harbor/registry.json no_hack: true harbor_job: jobs_dir: artifacts/jobs n_concurrent: 2 n_tasks: 10 max_retries: 1 timeout_multiplier: 1 agent: name: custom-openhands-sdk version: 1.33.0 runtime_image: docker.io/jierun/c-oh-sdk-1.33.0:v0.1 runtime_host_path: artifacts/runtime/openhands-sdk max_turns: 200 temperature: 1.0Use /evaluator:check after editing. It checks benchmark resolution, model
endpoint reachability, Docker/runtime availability, local checkpoint settings,
and job analysis configuration before /evaluator:run starts.
Configuration breakdown
Remote execution
Evaluator runs locally by default. Configure meta_info.resources when the
benchmark job should run on a remote Docker server or GPU node instead.
meta_info: resources: ip: 203.0.113.30 # local = current machine; otherwise SSH target user: ubuntu # SSH user for the remote server pwd: null # prefer SSH keys or your private secret channel directory: /home/ubuntu/LegoFlow/blocks/evaluator public_storage: ./artifactsUse a remote machine when the model server, Docker runtime, or benchmark environment is easier to keep close to the evaluation node. The directory should point at the Evaluator block checkout on that machine.
Model endpoint
Evaluator can point to a remote API or a local checkpoint served through vLLM.
In both cases, llm_api.model is the model name the agent will call.
- Remote API:
runtime_info: input: llm_api: api_key: <YOUR_API_KEY> api_base_url: https://your-openai-compatible-endpoint/v1 model: openai/Qwen3.5-35B-A3B protocols: [openai_compatible, anthropic_compatible] served_via: per_job_litellm_proxy- Local checkpoint:
runtime_info: input: llm_api: api_key: <YOUR_API_KEY> api_base_url: http://127.0.0.1:8000/v1 model: openai/Qwen3.5-35B-A3B local_model_serving: model_path: artifacts/models/Qwen3.5-35B-A3B model_name: Qwen3.5-35B-A3BKeep local_model_serving.model_name aligned with the model name served by
vLLM and referenced from llm_api.model.
Benchmark source
Evaluator normally reads registered benchmarks from Harbor's registry.
runtime_info: input: task_source: provider: harbor_registry dataset_name: swebench-verified version: "1.0" registry_path: repos/harbor/registry.json no_hack: trueUse no_hack: true for the hardened SWE-bench Verified path. Keep the registry
path at repos/harbor/registry.json unless you are intentionally using another
Harbor registry.
Smoke evaluation
Use a small task cap and modest concurrency when checking a new checkpoint, endpoint, or runtime image.
runtime_info: input: harbor_job: n_concurrent: 2 n_tasks: 10 max_retries: 1 timeout_multiplier: 1Production evaluation
For a full benchmark, set n_tasks: null and choose concurrency based on your
endpoint capacity.
runtime_info: input: harbor_job: n_concurrent: 24 n_tasks: null max_retries: 2 timeout_multiplier: 1Agent version presets
The agent fields move as a set. Pick one row and keep agent.name,
agent.version, agent.runtime_image, and agent.runtime_host_path aligned.
| Scaffold | agent.name | agent.version | agent.runtime_image | agent.runtime_host_path |
|---|---|---|---|---|
| Claude Code | custom-claude-code | 2.1.118 | docker.io/jierun/c-cc-2.1.118:v0.1 | artifacts/runtime/claude-code |
| OpenCode | custom-opencode | 1.18.7 | docker.io/jierun/c-oc-1.18.7:v0.1 | artifacts/runtime/opencode |
| OpenHands SDK | custom-openhands-sdk | 1.33.0 | docker.io/jierun/c-oh-sdk-1.33.0:v0.1 | artifacts/runtime/openhands-sdk |
Handoff with neighboring blocks
Evaluator usually receives a trained checkpoint from Trainer. It does not
currently declare a formal dependency in meta_info, so the practical handoff
is the checkpoint path you put into local_model_serving.model_path.
Receive Trainer checkpoint
After Trainer finishes, point Evaluator at the trained checkpoint directory and serve it through the local model path.
runtime_info: input: llm_api: api_base_url: http://127.0.0.1:8000/v1 model: openai/Qwen3.5-35B-A3B local_model_serving: model_path: ../trainer/artifacts/<training-output-dir> model_name: Qwen3.5-35B-A3BField reference
| Field | Required? | Default | When to change |
|---|---|---|---|
llm_api.api_key | Yes | human | Put the real key in env/private config. Never commit it. |
llm_api.api_base_url | Yes | human | Point Evaluator at a remote endpoint or local vLLM server. |
llm_api.model | Yes | openai/Qwen3.5-35B-A3B | Change when evaluating another model/checkpoint. |
local_model_serving.model_path | For local eval | artifacts/models/Qwen3.5-35B-A3B | Point to the checkpoint served by vLLM. |
local_model_serving.model_name | For local eval | Qwen3.5-35B-A3B | Keep aligned with the served model name. |
litellm_proxy.port | Yes | 4101 | Change only if the local port is occupied. |
task_source.dataset_name | Yes | swebench-verified | Choose another Harbor benchmark. |
task_source.version | Yes | "1.0" | Match the Harbor registry entry. |
task_source.registry_path | Yes | repos/harbor/registry.json | Change only for a custom Harbor registry. |
task_source.no_hack | No | true | Toggle hardened SWE-bench Verified behavior. |
harbor_job.n_concurrent | Yes | 24 | Tune for endpoint capacity and machine resources. |
harbor_job.n_tasks | No | null | Use a number for smoke runs; null runs the full benchmark. |
agent.name | Yes | custom-openhands-sdk | Switch the evaluation scaffold. Change the full agent row together. |
agent.version | Yes | 1.33.0 | Must match the selected scaffold and runtime image tag. |
job_analysis.enabled | Yes | true | Disable only when you want a faster run without analysis. |
job_analysis.tag_llm.* | No | empty | Fill only when tagging should use a separate model endpoint. |
meta_info.resources.ip | No | local | Set a host/IP when Evaluator should run on a remote server. |
meta_info.resources.user | For remote | null | SSH user for the remote server. |
meta_info.resources.directory | For remote | . | Evaluator block working directory on the remote server. |
meta_info.resources.public_storage | No | ./artifacts | Optional public artifact location for dashboards or sharing. |
Complete config.yaml
Use this as the complete reference after you understand the few fields above.
blocks/evaluator/config.yaml
meta_info: name: evaluator label: "Evaluator" description: Runs registered benchmark evaluations through Harbor against a configurable upstream LLM. Mirrors tracer's runtime shell, but loads tasks from Harbor's registry.json instead of a local curator export. parent: legoflow blocks: {} dependencies: from: {} to: {} resources: ip: local user: null pwd: null directory: . description: Local execution; set host-specific values in your private run profile public_storage: ./artifacts repositories: harbor: url: https://github.com/LegoX/harbor.git branch: main commit: 73fd120e731217b38f31667932ea26a28cdd4fff path: repos/harbor readonly: true environment: harbor_uv: artifacts/env/harbor-uv litellm_uv: artifacts/env/litellm-venv litellm: python_version: "3.13" litellm_version: "1.83.14"runtime_info: input: # MODE A (remote API) example: /evaluator:setup reference; MODE B (local vLLM checkpoint) active; # How to fill: docs/content/docs/reference/io.mdx -> LLM endpoint llm_api: api_key: human api_base_url: human model: "openai/Qwen3.5-35B-A3B" description: Example local checkpoint served via vLLM; see scripts/serve_local_model.sh. protocols: [openai_compatible, anthropic_compatible] served_via: per_job_litellm_proxy input_cost_per_token: 0.0 output_cost_per_token: 0.0 local_model_serving: # read by scripts/serve_local_model.sh; ignored for remote APIs model_path: artifacts/models/Qwen3.5-35B-A3B model_name: Qwen3.5-35B-A3B # must match basename of llm_api.model litellm_proxy: config_template: scripts/serve_llm/litellm_config.example.yaml port: 4101 master_key: dummy-key-cf # functional dummy task_source: provider: harbor_registry # benchmark catalog: CLAUDE.md "Benchmark Selection" dataset_name: swebench-verified version: "1.0" registry_path: repos/harbor/registry.json # When true: generate/use local swebench-verified-nohack (agent egress allowlist to # LiteLLM only; verifier stays public). Currently supports swebench-verified only. no_hack: true harbor_job: jobs_dir: artifacts/jobs n_concurrent: 24 n_tasks: null # null = full benchmark; int = smoke cap max_retries: 2 timeout_multiplier: 1 job_analysis: enabled: true # post-eval analysis via scripts/analyze_job.sh tag_llm: # "" = fall back to llm_api; see /evaluator:check reference base_url: "" model: "" api_key: "" # reference to /evaluator:setup agent: name: custom-openhands-sdk version: 1.33.0 runtime_image: docker.io/jierun/c-oh-sdk-1.33.0:v0.1 runtime_host_path: artifacts/runtime/openhands-sdk max_turns: 200 temperature: 1.0 env_extra: HARBOR_EXCLUDE_TASKS: "" # space-separated task IDs to skip (prior timeouts/OOMs) LITELLM_STICKY_ROUTING_ALIASES: "" # workaround for Harbor serve_litellm.sh under set -u output: eval_results_dir: description: Harbor job directories containing per-task evaluation results and LiteLLM trajectory logs path: artifacts/jobs job_layout: "artifacts/jobs/<job>/<task>/{agent,verifier}/" trajectory_format: "artifacts/jobs/<job>/<task>/agent/litellm-trajectory.jsonl" results_summary_format: "artifacts/jobs/<job>/result.json"