QLoRA Economics: What 4-Bit Fine-tuning Actually Buys You

"Fine-tune an LLM" sounds like it means one thing, but full fine-tuning and QLoRA fine-tuning are different enough in their resource profile that they're really different techniques wearing the same name. Full fine-tuning updates every parameter in the model, which means storing gradients and optimizer state for the whole thing — for anything above a few billion parameters, that's enterprise-GPU territory. QLoRA gets usable fine-tuning results on a single consumer GPU or a free-tier Kaggle instance. The numbers behind that gap are worth being precise about, because "parameter-efficient" is a vague phrase until you've measured what it actually buys.

I've run this approach across four fairly different fine-tuning projects — price prediction on Llama and Mistral, a Korean math solver on NAVER's HyperCLOVA X 1.5B, and a Nepali recipe generator on Qwen2-1.5B — and the economics hold up consistently across all of them.

What QLoRA actually changes

Two techniques stack together:

4-bit quantisation (via BitsAndBytes, NF4 format with double quantisation) loads the frozen base model's weights at 4 bits per parameter instead of 16 or 32. The base model becomes read-only and dramatically smaller in memory. Double quantisation squeezes a bit more by also quantising the quantisation constants themselves — a small additional saving that adds up at scale.

LoRA (Low-Rank Adaptation) freezes the base model entirely and injects small trainable low-rank matrices alongside the existing weight matrices. Instead of learning a full-rank update ΔW (the same shape as the original weight matrix W), LoRA learns ΔW ≈ BA, where B and A are much smaller matrices with an inner rank r that's typically 8–64. Only A and B get gradients; the frozen 4-bit base contributes nothing to the optimizer state.

from peft import LoraConfig, get_peft_model
 
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)

Targeting just the attention projection matrices (q_proj, v_proj) rather than every linear layer is itself a lever — more target modules means more trainable parameters and better adaptation capacity, at the cost of memory and training time. r=16 with alpha=32 (a 2:1 alpha-to-rank ratio) was the practical sweet spot across these projects: enough capacity to shift the model's behaviour meaningfully, without the memory footprint creeping back toward full fine-tuning territory.

The numbers, and what they actually mean

On the price-prediction work fine-tuning Llama and Mistral, QLoRA delivered:

• 90% reduction in trainable parameters — because the vast majority of the model is frozen; only the injected LoRA matrices receive gradients.
• 75% lower GPU memory usage versus full fine-tuning — from the combination of 4-bit base weights and a much smaller optimizer state (no momentum/variance buffers for parameters that aren't training).
• Training time cut from 15h to 5h — less compute per step (smaller gradient computation graph) and, practically, fewer OOM-driven batch-size compromises.

The trainable-parameter number is the one people tend to over-read. A 90% reduction in trainable parameters is not a 90% reduction in capability — the frozen 4-bit base still does almost all the representational work; LoRA is steering it, not rebuilding it. That's precisely why QLoRA is well-suited to tasks that are adaptations of what the base model can already mostly do (structured extraction, domain-specific formatting, style transfer) and less well-suited to teaching a model something genuinely absent from its pretraining.

Where the constraint actually bites: completion-only training

The Nepali recipe generator (Qwen2-1.5B) surfaced a different lesson than raw memory savings. Training on full prompt+response pairs wastes gradient signal on tokens the model doesn't need to learn — the prompt is given, only the completion is what the model should get better at producing. DataCollatorForCompletionOnlyLM masks the prompt tokens out of the loss, so gradients only flow from the assistant's response:

from trl import DataCollatorForCompletionOnlyLM
 
collator = DataCollatorForCompletionOnlyLM(
    response_template="### Response:",
    tokenizer=tokenizer,
)

Combined with 4-bit quantisation, this kept the Qwen2-1.5B fine-tune tractable on limited hardware while making every training step count toward the actual objective — generating the response, not memorising the prompt template.

Fitting a fine-tune on a single Kaggle P100

The HyperCLOVA X 1.5B Korean math solver was constrained differently again: one Kaggle P100 GPU, 16GB of VRAM, no negotiating room. That's where QLoRA stops being "more efficient" and becomes "the only thing that fits at all." TRL's SFTTrainer handled the supervised fine-tuning loop over a synthetic instruction-following dataset generated specifically for the task (structured Korean math question/answer pairs, since no off-the-shelf dataset covered it), with BitsAndBytes 4-bit loading keeping the base model's memory footprint inside the P100's budget throughout training.

The honest trade-off

QLoRA isn't free efficiency — it's a real trade of adaptation capacity for resource footprint. A LoRA adapter with rank 16 on two projection matrices has meaningfully less capacity to change model behaviour than full fine-tuning does, and for some tasks that ceiling matters. What I've found across four fine-tunes on four different base models is that for adaptation-shaped problems — steering a capable base model toward a specific format, domain, or language — that ceiling is rarely the binding constraint. GPU memory and training time are. That's the trade QLoRA is actually offering, and on consumer or free-tier hardware, it's usually the only trade on the table.