Machine Learning16 min read

Fine-tuning gpt-oss-120b with LoRA: A Practical Guide

Fine-tuning a 120-billion-parameter model is an infrastructure project, not a larger version of a notebook tutorial. LoRA reduces the number of trainable parameters, but it does not make the base model disappear: you still need memory for its weights, activations, attention, and the training runtime.

This guide focuses on the decisions that determine whether the project succeeds: when adaptation is justified, how to construct training examples, how to estimate hardware needs, what to evaluate, and how to deploy adapters safely.

First, clarify the terminology

LoRA and Text-to-LoRA are not interchangeable. Standard LoRA learns small low-rank adapter matrices from training examples through gradient descent. Text-to-LoRA refers to a separate family of methods that generates or predicts adapters from a natural-language task description. The PEFT workflow below is standard supervised LoRA fine-tuning, which is the mature and widely supported option.

If your implementation creates a LoraConfig and trains on examples, you are doing LoRA—not automatically Text-to-LoRA.

Do you need fine-tuning at all?

Start with the cheapest intervention. Fine-tuning is useful when a behavior must be consistent across many requests; retrieval is better when knowledge changes; tools are better when answers require live data or deterministic actions.

  • Use a better prompt when the task is clear but instructions are incomplete.
  • Use retrieval-augmented generation when the model needs private, cited, or frequently changing knowledge.
  • Use tool calling when the task depends on databases, APIs, calculations, or side effects.
  • Use LoRA when you have repeated examples of a stable behavior the base model does not perform reliably.
  • Use a smaller model first when latency, cost, or deployment simplicity matters more than maximum capability.

What LoRA changes

For a selected linear layer with frozen weight W, LoRA learns two much smaller matrices A and B. During inference, the layer behaves like W plus a scaled low-rank update BA. The rank r controls adapter capacity: a larger rank can express more change, but increases memory, storage, and overfitting risk.

Base layer:       y = Wx
LoRA layer:       y = Wx + (alpha / r)BAx
Trainable values: A and B
Frozen values:    W

LoRA saves optimizer and gradient memory because only adapter parameters are trained. It does not imply a 10,000× end-to-end memory reduction, and the final requirement depends heavily on precision, sequence length, batch size, target modules, checkpointing, and the training implementation.

Know the base model

gpt-oss-120b is a mixture-of-experts reasoning model with roughly 117B total parameters and about 5.1B active parameters per token. Its MXFP4 representation can run for inference on a single 80 GB accelerator, but training adapters usually needs substantially more memory than inference. Do not turn an inference memory claim into a training hardware estimate.

1. Define behavior and a measurable baseline

Write the evaluation set before the training set. Collect representative prompts, difficult edge cases, safety cases, and examples the current model already handles well. Record the base model's output so you can measure improvement and regression rather than relying on a few impressive demos.

  • Classification: macro F1, per-class recall, and confusion matrix.
  • Extraction: exact match plus field-level precision, recall, and F1.
  • Generation: task-specific checks, rubric-based review, and factuality or citation accuracy.
  • Code: executable tests and pass rate, not similarity to a reference answer.
  • Operations: latency, peak memory, throughput, and cost per successful request.

2. Build training data that matches inference

Quality and coverage matter more than raw row count. Match the exact chat template, system instruction, tool schema, and output format that production will use. Remove duplicates and leakage across splits. Keep the validation and test sets independent from training—including paraphrases of the same source example.

{"messages": [
  {"role": "system", "content": "Extract the requested fields as JSON."},
  {"role": "user", "content": "<representative production input>"},
  {"role": "assistant", "content": "{"field": "verified target"}"}
]}
  • Include ordinary cases, rare cases, ambiguous inputs, and valid refusals.
  • Keep target answers concise; the model learns unnecessary verbosity too.
  • Use real reviewed examples where policy permits, with secrets and personal data removed.
  • Check label consistency by having two reviewers independently score a sample.
  • Avoid synthetic-data loops unless humans verify both correctness and diversity.

3. Measure memory before launching a full run

Run a small smoke test with the real model, sequence length, precision, and target modules. Capture peak allocated memory for one forward and backward pass, then add operational headroom. Sequence length can dominate activation memory, so halving it is often more effective than tuning minor settings.

  • BF16 LoRA keeps high-precision base weights and commonly requires multiple large accelerators for this model.
  • Quantized LoRA reduces base-weight memory but requires a runtime that explicitly supports the model's quantization and backward path.
  • Gradient checkpointing saves activation memory by recomputing during backward, trading speed for capacity.
  • FSDP or DeepSpeed ZeRO can shard model states, but adds communication and configuration complexity.
  • CPU or NVMe offload may make a run fit while making it too slow to be economical.

4. Configure LoRA deliberately

Do not copy target-module names from a different architecture. Inspect the loaded model, identify repeated linear projections, and verify that PEFT reports a plausible number of trainable parameters. Start with attention projections; expand to expert or MLP projections only if evaluation shows insufficient capacity.

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()

# Verify these module names against the exact model and library version.
# A successful constructor is not proof that the intended layers were adapted.

Rank 8 or 16 is a sensible experiment, not a universal optimum. Compare a small matrix of rank, learning rate, and target modules while keeping data and evaluation fixed. Stop when validation quality plateaus or regressions increase.

5. Train with safeguards

  • Mask padding and, when appropriate, user/system tokens so loss is applied only to the desired assistant response.
  • Log training and validation loss, gradient norm, learning rate, tokens per second, and peak memory.
  • Save adapter checkpoints and the exact base-model revision, tokenizer, chat template, code revision, and configuration.
  • Use deterministic seeds where supported, but expect distributed kernels to retain some nondeterminism.
  • Stop early based on task evaluation, not training loss alone.

6. Evaluate the adapter as a change, not in isolation

Compare base and adapted models on the same held-out prompts with the same decoding settings. Segment results by input type so a high average does not hide a critical failure. Review regressions on general capability, safety behavior, output formatting, and tasks that should remain unchanged.

Release gate
[ ] Target metric improves by the agreed minimum
[ ] Critical slices and safety cases pass
[ ] No unacceptable regression from the base model
[ ] Output schema validation passes
[ ] Latency and memory stay within budget
[ ] Human reviewers approve a blinded sample

7. Deploy adapters without losing reproducibility

Keep the adapter separate when you need rapid rollback or several specializations on one base model. Merge it only when the serving stack benefits and you have verified that the merged artifact matches unmerged outputs closely enough. Version the base and adapter as a pair; an adapter is not a self-contained model.

  • Warm the adapter before traffic and test switching behavior under concurrency.
  • Canary a small traffic share and monitor task success, refusals, latency, and malformed outputs.
  • Retain the base-model path as a rollback option.
  • Log the model, adapter, prompt, and schema versions needed to reproduce an incident.

Practical conclusion

LoRA makes adaptation smaller; it does not make a 120B model small. The reliable path is to prove that fine-tuning beats prompting or retrieval, establish evaluation first, run a measured memory smoke test, train on reviewed production-shaped examples, and ship the adapter behind a reversible rollout.

References

  • OpenAI, Introducing gpt-oss: https://openai.com/index/introducing-gpt-oss/
  • OpenAI, gpt-oss model card: https://openai.com/index/gpt-oss-model-card/
  • Hugging Face PEFT documentation: https://huggingface.co/docs/peft/