Local Model Evaluation

The current eval profile benchmarks a local checkpoint (an SFT output from the pipeline). Serve it with vLLM and keep llm_api pointed at that endpoint. You do not need to change start.sh, because the block only talks to an OpenAI/Anthropic-compatible HTTP endpoint.

Topology

The eval node is CPU-only, so vLLM must run on a GPU node:

eval node (CPU)                                  GPU node
  agent container → eval LiteLLM :4101  ─────────▶ vLLM :8000/v1  (local ckpt)
                    (start.sh, trajectory_logger)   api_base_url

0. One-time: create the vLLM env (GPU node)

The eval node's /evaluator:setup does not build this. It only builds the CPU-side Harbor uv env and LiteLLM venv. For a standard bf16/fp16 checkpoint, a plain pip install is enough:

conda create -y -n vllm_0.18.1 python=3.12
conda activate vllm_0.18.1
pip install vllm==0.18.1

Only FP8 models with custom kernels (e.g. GLM-5.1-FP8) need the heavy source build in repos/harbor/scripts/serve_llm/install_vllm_32b717_cu128.sh. serve_local_model.sh activates $VLLM_CONDA_ENV (default vllm_0.18.1) and errors with this exact recipe if vllm isn't found.

1. Serve the checkpoint (GPU node)

bash scripts/serve_local_model.sh   # runs in foreground; use tmux

The checkpoint path and served name are read from config.yaml → runtime_info.input.local_model_serving (model_path / model_name). Treat that config as the source of truth, not a hardcoded script default. Everything else is env-overridable (MODEL_PATH, MODEL_NAME, VLLM_PORT, TENSOR_PARALLEL_SIZE, API_KEY, VLLM_CONDA_ENV, ...). It serves on :8000 by default. When ready, it prints the exact llm_api block to paste.

local_model_serving:  model_path: artifacts/models/Qwen3.5-35B-A3B  model_name: Qwen3.5-35B-A3B

Use a private profile or the MODEL_PATH environment override when the checkpoint lives outside the repository; do not commit machine-specific absolute paths.

2. Point llm_api at it (eval node)

Confirm that the active config.yaml → runtime_info.input.llm_api matches the local-vLLM recipe:

llm_api:  api_key: dummy-key                       # must match vLLM --api-key  api_base_url: "http://<GPU_NODE_IP>:8000/v1"  model: "openai/Qwen3.5-35B-A3B"          # openai/<served-model-name>  protocols: [openai_compatible, anthropic_compatible]  served_via: per_job_litellm_proxy  input_cost_per_token: 0.0  output_cost_per_token: 0.0

The served name must match the basename of llm_api.model (openai/<model_name>).

3. Validate and run

bash scripts/dryrun.sh
bash scripts/probe_llm_completion.sh
bash scripts/start.sh

dryrun.sh validates static configuration and environments; the separate completion probe verifies the live vLLM endpoint. start.sh repeats both gates before launching Harbor.