LiteLLM Proxy
Evaluator never points the agent directly at your upstream model API. Each job starts its own LiteLLM proxy, which normalizes the endpoint, exposes both OpenAI- and Anthropic-compatible protocols, and attaches the trajectory logger.
Why a proxy
- Different agent scaffolds expect OpenAI or Anthropic message formats. The proxy serves both from one upstream model.
- The proxy logger writes each request and response to
litellm-trajectory.jsonl, so a run can be replayed and analyzed later. - The agent only talks to the proxy, so the same setup works with a remote API or a local vLLM checkpoint.
- Each job gets a fresh per-job config, so concurrent or sequential jobs do not share state.
Configuration
The upstream model is declared in config.yaml under runtime_info.input.llm_api. The block ships two interchangeable recipes, with only one uncommented at a time:
# MODE A — remote API (commented example)llm_api: api_key: "<your-api-key>" api_base_url: "https://api.example.com/v1" model: "openai/<served-model-name>" protocols: [openai_compatible, anthropic_compatible] served_via: per_job_litellm_proxy input_cost_per_token: 0.0 output_cost_per_token: 0.0# MODE B — local vLLM checkpoint (active)llm_api: api_key: dummy-key # must match vLLM --api-key api_base_url: "http://<GPU_NODE_IP>:8000/v1" model: "openai/Qwen3.5-35B-A3B" # openai/<vLLM --served-model-name> protocols: [openai_compatible, anthropic_compatible] served_via: per_job_litellm_proxy input_cost_per_token: 0.0 output_cost_per_token: 0.0The proxy itself is configured under runtime_info.input.litellm_proxy:
litellm_proxy: config_template: scripts/serve_llm/litellm_config.example.yaml port: 4101 master_key: dummy-key-cfKeep credentials private
For a remote API, put the real key only in a private run profile or secret-injection step. For a local vLLM checkpoint, an example dummy-key is acceptable, but api_key must still match vLLM's --api-key.
Lifecycle
scripts/start.sh handles the proxy automatically:
- Renders a per-job config from the template into
artifacts/litellm/<job>/litellm_config_eval.yaml. - Starts the proxy on
litellm_proxy.port, passing the config to Harbor's LiteLLM serve script viaLITELLM_CONFIG. - Runs the Harbor job against the proxy.
One proxy only
The model-serving layer for a local checkpoint is vLLM only. start.sh already runs the per-job LiteLLM proxy with the trajectory logger, sticky routing, and Anthropic-format support. Do not start a second LiteLLM next to vLLM because it would collide on the port and bypass that logging. When a job's inference is finished but the proxy is still up, stop the process started for this job, and only that one.
set -u workaround
Harbor's serve_litellm.sh dereferences LITELLM_STICKY_ROUTING_ALIASES
without a default under set -u. The block defaults it to an empty string via
runtime_info.input.env_extra.LITELLM_STICKY_ROUTING_ALIASES in config.yaml,
so leave that key in place.