Getting Started

This page gets Trainer from LF data to a trained checkpoint. The path is: fill config.yaml, run setup, check the block, launch training, then open the dashboard.

You can read it as a concrete walkthrough: follow the example config snippets and commands in order, and you will see how a Trainer run is put together.

Prerequisites

Before starting, make sure these are available:

• Claude Code with the Trainer plugin loaded.
• A GPU node for training; the checked-in example targets one 8-GPU node.
• uv, CUDA, and visible GPUs on the training host.
• Training data: a Tracer LF file, a Hugging Face LF dataset/file, or a raw Tracer Harbor job.
• A base model, either a Hugging Face model ID or a local model directory.
• Optional wandb and Hugging Face credentials, only if your run needs them.

Validated environment

Trainer is the one block with a real hardware floor, so it is worth checking before you configure anything. What is reported here is what has actually been run end to end, not what the code theoretically supports:

Smaller models and smaller GPU counts will run, but the shipped defaults are tuned for the row above; expect to adjust batch size, gradient accumulation and the DeepSpeed config together when you move away from it. Multi-node training is not wired up.

The numbers above come from the reported end-to-end run, which also hit two failures worth knowing about: host memory exhaustion from use_unsloth_gc at 128k context, and NCCL defaulting to eth0 on a single multi-GPU host.

If the Trainer slash commands are not visible in Claude Code, see FAQ.

Setup

Most Trainer issues come from pointing the block at the wrong data source, model, output directory, or GPU count. Once config.yaml is right, the skills take over.

For a first run, leave meta_info and runtime_info.output alone. Fill the runtime_info.input pieces below. For every field, see the Configuration Guide.

Receive Tracer output

For the normal Tracer -> Trainer handoff, point Trainer at Tracer's converted LF file:

runtime_info:  input:    source:      type: local_lf      lf_path: ../tracer/artifacts/sft_data/<job>/lf.json # Tracer's converted SFT file    conversion:      data_name: my_tracer_dataset      max_instances: 0

The local_lf mode is the simplest handoff when Tracer has already produced artifacts/sft_data/<job>/lf.json. If you want Trainer to receive raw Harbor jobs and run conversion itself, use the dependency-backed harbor_job mode below. In that mode, meta_info.dependencies.from records that source.job_dir is supplied by Tracer:

meta_info:  dependencies:    from:      source.job_dir:        from: tracer.output.raw_trajectories_dir        when: {source.type: harbor_job}

Use environment variables for private Hub access instead of committing tokens.

If you are not continuing from Tracer, Trainer can also consume a ready-made LF file from Hugging Face:

runtime_info:  input:    source:      type: hf_lf      hf_hub_url: Lego-X/samples_for_llama_factory_sft      hf_file_name: jierun_glm52_openswe_mirror_score4_proto_oh_sdk_512_for_qwen3_5.json      hf_split: train    conversion:      data_name: glm52_openswe_mirror_score4_proto_oh_sdk_512_qwen3_5 # dataset key used by LLaMA-Factory      max_instances: 0                         # 0 = all samples

Conversion config

If Tracer has not produced lf.json yet, Trainer can convert a raw Harbor job. Keep max_instances small for a smoke run:

runtime_info:  input:    source:      type: harbor_job      scaffold: openhands-sdk                  # must match the Tracer rollout scaffold      job_dir: ../tracer/artifacts/jobs/<job>    conversion:      max_instances: 100                       # small cap for a smoke run; 0 = all samples      exclude_repos_file: scripts/excluded_repos.txt # filters eval benchmark repos      data_name: my_converted_dataset

The harbor_job path uses the same conversion stack documented in Tracer. If you already have lf.json, prefer local_lf.

Model and Training Config

Set the base model, chat template, and a unique output directory. Relative output directories are stored under artifacts/model/.

runtime_info:  input:    model:      model_name_or_path: Qwen/Qwen3.5-35B-A3B-Base      trust_remote_code: true    training:      finetuning_type: full      template: qwen3_5      cutoff_len: 131072      rope_scaling: null      output_dir: qwen3_5_35b_a3b_base_glm52_openswe_mirror_score4_proto_oh_sdk_512_gbs64pbs1acc8_lr5e-5_epo3      per_device_train_batch_size: 1      gradient_accumulation_steps: 8      learning_rate: 5.0e-5      num_train_epochs: 3.0      lr_scheduler_type: cosine      warmup_ratio: 0.1      deepspeed: scripts/deepspeed/ds_z3_config.json      overwrite_output_dir: false

Use the template family that matches the model (qwen3, qwen3_nothink, qwen3_5, or qwen3_5_nothink). If cutoff_len is larger than the model's native context window, choose a model-supported RoPE scaling method such as yarn; otherwise keep rope_scaling: null.

The effective global batch size is per_device_train_batch_size x gradient_accumulation_steps x n_gpus_per_node. In the example, that is 1 x 8 x 8 = 64.

Checkpoint and Performance Config

These defaults make the example practical for a large full-SFT run:

runtime_info:  input:    training:      save_strategy: steps                     # steps | epoch | no      save_steps: 200      save_only_model: true                    # smaller checkpoints; cannot resume optimizer state      bf16: true      flash_attn: fa2      max_samples: 1000000

save_only_model: true saves disk but prevents optimizer-state resume. Use a new output_dir for each real run; existing output directories are rejected unless overwrite is explicitly enabled.

Infrastructure and Credentials

Keep the GPU count aligned with the node. Credentials can come from environment variables instead of the tracked config file.

runtime_info:  input:    infrastructure:      n_gpus_per_node: 8    experiment:      wandb_mode: offline                      # online | offline | disabled    credentials:      wandb_api_key: ""                        # prefer $WANDB_API_KEY      hf_token: ""                             # prefer $HF_TOKEN for private datasets/models

Now run setup:

/trainer:setup

This initializes the pinned repos, builds the uv training environment, installs LLaMA-Factory and swe_data_process, and checks the key imports. Re-running it is fine.

The training environment is a uv venv under meta_info.environment.sft_uv (default artifacts/env/lf). It is generated locally and not version-controlled.

Check

Before using GPUs, run:

/trainer:check

This is read-only. It checks config, repo pins, the training environment, source data, converter availability, model path, output directory safety, GPU visibility, and wandb/HF settings. For exact pass conditions, see Validation Checks.

Fix required failures before launching training.

Run training

Launch the full training pipeline:

/trainer:run

Trainer obtains LF data, registers it in LLaMA-Factory's dataset_info.json, generates a training YAML, runs LLaMA-Factory, and updates runtime_info.output with the checkpoint and metrics.

Behind the command, the pipeline is:

STEP 0  Obtain LF data         local LF / Hub LF / raw Harbor conversion
STEP 1  Register dataset       selected source -> dataset_info.json
STEP 2  Train                  generated YAML -> LLaMA-Factory torchrun

Long training runs should live inside tmux on the GPU node. The launch uses NPROC_PER_NODE from infrastructure.n_gpus_per_node and DeepSpeed ZeRO-3 by default, so keep that GPU count aligned with the machine.

The main outputs are:

artifacts/data/lf_data/<data_name>.json
artifacts/data/lf_data/dataset_info.json
artifacts/training_config/<run>.yaml
artifacts/model/<run>/
artifacts/logs/<run>_<timestamp>.log

For the example profile above, a historical 8-GPU run trained 512 samples for 3 epochs: 24 optimizer steps, about 2h 7m inside LLaMA-Factory, mean training loss 0.3424, and roughly 131 GiB under artifacts/model/<run>/. Treat these numbers as cost intuition, not a benchmark; GPU type, cache state, and storage throughput can change the runtime.

The generated LLaMA-Factory YAML is saved under artifacts/training_config/; you do not need to maintain a separate checked-in training YAML. For the full artifact layout and checkpoint handoff, see Output Format.

Output for Evaluator

Trainer writes the model under artifacts/model/<run>/ and records the latest checkpoint in runtime_info.output.checkpoint_path:

runtime_info:  output:    checkpoint_path:      description: Path to the latest trained model checkpoint      value: artifacts/model/<run>

Trainer does not currently declare a meta_info.dependencies.to edge because Evaluator can benchmark any remote API or local checkpoint, not only Trainer's latest output. When you do want to evaluate this checkpoint, continue with Evaluator Getting Started; Evaluator will show how to serve the checkpoint and point its config at the served model.

Dashboard visualization

After a run starts, open the dashboard:

/trainer:dashboard

The dashboard reads training logs and model artifacts directly. Use it to watch loss curves, compare runs, and inspect final metrics. During long training, also keep the terminal or tmux log open so failures are easy to diagnose.

For the internal training mechanism, see Design.

Cleanup

Use scripts/clean.sh only when you intentionally want to remove temporary run outputs under artifacts/. Its dry-run mode is the safest way to check what would be deleted before cleaning a training workspace.