Fine-tuning starts before training.

Most fine-tuning requests start one decision too late. Someone has a folder of documents, a GPU, and a plan to run LoRA. The unanswered question is whether those facts should be trained into the model at all.

I built the LLM fine-tuning skill to make coding agents stop at that fork. It gives Claude Code, OpenCode, Codex, or any markdown-capable agent a practical playbook for deciding what to train, preparing the data, proving the run fits, measuring the result, and serving the artifact with vLLM.

The first question is where the knowledge should live

A request such as “train this model on our documentation” can describe several different goals. Exact product codes and changing API names belong in retrieval. Output format, domain style, task framing, and tool behavior are better candidates for weights. Most domain assistants need both.

The skill therefore starts with three choices:

The opinionated part is simple: a low-rank adapter is a poor database. If the success metric is exact recall, the agent should say that before it spends hours encoding unstable facts into weights.

A method ladder instead of a default recipe

Once training is justified, the skill climbs only as high as the goal requires:

1. Prompt, RAG, or long context. The baseline and often the complete solution for factual work.
2. LoRA SFT. The default for teaching behavior, output shape, workflow, and style with a small detachable adapter.
3. Continued pretraining. Useful when the domain's language and distribution are genuinely unusual, not as an automatic prelude to SFT.
4. More adapter capacity. Higher rank, more exposure, or trainable embeddings when the earlier stage has clearly plateaued.
5. Full fine-tuning or distillation. The expensive end of the ladder, used only when lighter stages cannot reach the target.

Each stage has an exit. If continued pretraining beats the later SFT checkpoint on the metric that matters, the earlier checkpoint ships. More stages are not automatically more progress.

Why this lives as an agent skill

Fine-tuning work spreads across the repository, the dataset, the machine, and the serving target. A static checklist cannot inspect those pieces. A coding agent can.

The main SKILL.md holds the decision process. Six references are loaded only when needed: data preparation, teacher–student generation, evaluation, training operations, serving, and troubleshooting. An evaluation question does not spend context on vLLM startup flags; a serving failure does not reload the entire data guide.

The content is model- and domain-agnostic. Unsloth with a PEFT/TRL-style loop is the reference path for single-GPU LoRA and QLoRA because that is where the field notes came from. A different trainer does not invalidate the decisions.

The mistakes I wanted the agent to remember

The useful parts are the failures that look reasonable until they waste a day.

• Loading is not training. A checkpoint can fit during inference and fail on the first backward pass. The skill requires one forward and backward pass at the real sequence length and batch size on the actual GPU.
• A wrong response mask can train on nothing. SFT labels must be checked against the live tokenizer template. A nearly correct assistant anchor can mask every label while the job still appears to run.
• Four-bit weights are packed. Casting a packed uint8 buffer to bf16 does not dequantize it. It silently produces plausible garbage. Any merge or weight surgery must use the quantizer's dequantization path and verify tensor shapes.
• Quantization format is architectural. Runtime 4-bit is dependable for dense models but may leave mixture-of-experts weights untouched. Block-FP8 checkpoints can load and still fail in training kernels.
• A long run needs a failure plan. The process must survive the shell, checkpoint often, cap host memory, and prove that resume works before hour six.

The skill keeps a run log as part of the work: exact checkpoint revision, resolved dependency versions, GPU and peak memory, adapter design, data split, decoding settings, every evaluation number, and every non-obvious failure.

An evaluation that can say no

Fine-tuning is easy to flatter. Teacher-forced accuracy gives the model the correct prefix and asks for the next token. A user does not. The skill separates five measurements:

The gaps are more useful than the headline score. High teacher-forced performance with weak generation means the knowledge is present but not surfacing. That calls for output-behavior work, decoding constraints, or retrieval—not another round of continued pretraining.

The field notes also show why precision belongs in the report. In observed runs, teacher-forced accuracy was roughly double free-generation accuracy, and bf16 recalled 50–60% more than the same model at 4-bit serving precision. Those figures are calibration, not universal benchmarks. The rule is universal: report the generative number produced by the artifact and endpoint that users will receive.

Training decisions reach all the way into vLLM

A small adapter that touches attention and MLP projections can remain detachable. vLLM can keep one base model on the GPU and select among adapters per request. Training embeddings or the output head changes that artifact: those matrices are saved in full, often turning megabytes into gigabytes and forcing a merge.

That is why the skill asks about serving before training. It helps choose between a detachable adapter, a merged model, and a merged-plus-quantized model. It also carries the verification through startup:

1. Wait for the application to finish loading and compiling.
2. Check the health endpoint.
3. Confirm the expected model is listed.
4. Run a real generation through the same completion or chat path used by the evaluation.

A process existing is not the same as a model serving. A health check alone is not the same as the correct artifact generating a useful answer.

Using the skill

The installer symlinks the same skill into Claude Code, Codex, and OpenCode, so a pull updates every harness in place:

git clone https://github.com/abhiram1809/unsloth-finetuning-vllm-skill.git
cd unsloth-finetuning-vllm-skill
./install.sh

It can also install for one harness or into the current project. Claude Code users can install it as a plugin. Any agent without a skill loader can read the markdown directly.

The skill triggers on planning, debugging, and evaluation questions—not only at the start of a run. A request such as “the loss went down but the model is not better” should lead the agent to inspect the label mask, compare base and fine-tuned generations, and locate the gap before recommending more training.