Quantized weights, full precision: The math behind 75% cheaper fine-tuning

We on the Crusoe post-training team have built proprietary technology that performs LoRA training directly on 4-bit quantized base models. It introduces no train-inference accuracy loss and reduces GPU memory requirements by up to 75%. This post walks through the forward and backward math that makes this possible.

Why base weights are stored in bf16 for LoRA fine-tuning

Crusoe Serverless Fine-Tuning uses LoRA fine-tuning to empower developers, researchers, and product managers to customize powerful frontier models with proprietary knowledge, style, and reasoning capabilities. LoRA enables fine-tuning with minimal quality loss at a fraction of the cost of full-parameter fine-tuning.

State-of-the-art frontier models released in 2026, such as the DeepSeek-V4, Kimi-K3, and GLM-5 series, are heavily quantized from 16-bit to 4-bit weights to enable significantly faster, cheaper serving. NVIDIA and AMD have developed proprietary compute to handle quantized operations at extreme speed. At Crusoe we leverage state-of-the-art techniques such as MemoryAlloy, proprietary kernels, and scheduling optimizations to serve 4-bit models efficiently.

In both full-parameter and LoRA fine-tuning, model weights are usually stored in bf16 precision in GPU memory, even during quantization-aware training for a model that will ship in 4-bit. This means that to fine-tune a model like Kimi-K2.6, which has 1T parameters, at least 2 TB of GPU memory is needed to hold the base model weights alone. That is four NVIDIA HGX™ H100 nodes with eight 80 GB GPUs each or two NVIDIA HGX™ H200 nodes with eight 141 GB GPUs each. This is the implementation available in popular training frameworks such as transformers, AutoModel, and Megatron, often for ease of use. That’s because many kernels, especially backward/backprop ones, do not exist in open source for lower precisions.

In full-parameter training, bf16 storage is required because weights are actively being updated during training. If the master copy of the weights were in 4-bit, the gap between possible 4-bit values would be too coarse to enable weight updates on most optimizer steps. However, in LoRA training, base weights are not updated. This raises a question. Can base weights be kept in 4-bit while still guaranteeing accuracy and precision fidelity between training and inference?

To answer this question, we look at the exact operations that are performed in LoRA training, and the quantizations they are performed in.

LoRA operations and quantization

When evaluating the order and quantization of operations, we need to consider what happens in both inference and training. In inference, there are only forward operations. In training, there are both forward and backward. Furthermore, for LoRA, we also need to consider whether we are doing unmerged or merged LoRA. Which one we employ during inference dictates how we should do training. In merged LoRA, LoRA weights are fused back into the base weights. In unmerged LoRA, the LoRA weights stay separate from the base weights, and their output is added to the base output at each adapted layer. Here we consider unmerged LoRA.

Terminology

• X: input activations
• Y: output activations, broken down into Y^base and Y^LoRA for unmerged LoRA
• W: base weights
• A,B: down and up LoRA matrices

Forward pass in inference

With unmerged LoRA, the order of operations is:

Now let's fold in quantization and consider these LoRA operations on the mxfp4 experts of DeepSeek-V4-Flash on NVIDIA Blackwell GPUs, which supports mxfp4-native ops. The base weights are in mxfp4. LoRA weights are in bf16, which is still cheap because they are ~1% the size of the base weights. The quantizations turn out to be:

Forward pass in training

During the forward pass in training, the exact same operations and quantizations can be performed. The only potential difference is that if we are using NVIDIA Hopper GPUs, which don’t support mxfp4-native ops, the first operation must be performed in bf16 (or fp8):

There is a question of whether this train-inference precision mismatch can damage accuracy. In our internal training evaluations, we observed no accuracy degradation from this mismatch. In fact, the method used to coerce bf16 weights onto 4-bit grids in full-parameter training, quantization-aware training, relies on this same "quantization passthrough" assumption.

Note, however, that there is a clever trick we can employ here. If the base weights were already released in 4-bit, then even if we perform compute on them in bf16, they can be stored at rest in 4-bit and dequantized in flight to bf16 inside the kernel, with zero precision loss because they were originally as coarse as 4-bit already:

This suggests an exciting idea. If we also don't need bf16-resident base weights in the backward pass and optimizer step, then we could store them in 4-bit with no accuracy degradation while still being able to train on Hopper!

Backward pass and optimizer step in training

Reasoning now in the backward direction, we have the output gradient dL/dY_bf16 and want dL/dA_bf16, dL/dB_bf16, and dL/dX_bf16 (no dL/dW_bf16 because the base weights are frozen in LoRA). The equations are:

Again, W can be dequantized on the fly from 4-bit storage to bf16 for the compute kernel with no accuracy loss. Moreover, in the optimizer step, W is not updated, unlike in full-parameter fine-tuning. That means that W never needs to be stored in higher precision.

4-bit base weight storage with on-the-fly dequantization

Taking this discovery, we wrote dequant-on-the-fly kernels, implemented them in the Crusoe training stack, and benchmarked accuracy and training throughput. We confirmed that this approach led to no accuracy degradation, while enabling faster training by freeing up GPU memory from base weight storage and unlocking higher batching and longer context lengths.

Ultimately, this discovery enables training of large state-of-the-art mixture-of-experts (MoE) models with up to 75% less hardware footprint than current open-source LoRA implementations in frameworks like transformers, AutoModel, and Megatron.

Get started

Serverless Fine-Tuning is generally available today in Crusoe Intelligence Foundry. You pick a base model from a curated library that includes Kimi, GLM, DeepSeek, Llama, Qwen, Gemma, GPT-OSS, and Nemotron, bring your own data, and run a LoRA fine-tuning job without provisioning a single GPU. Billing is per training token, and early stopping ends the job the moment your model stops improving. When training finishes, you can deploy the adapter directly on Crusoe or download the weights in safetensors format and serve them anywhere.

Sign up for Crusoe Intelligence Foundry to get started, or clone the Crusoe for Developers Hub repository and start fine-tuning on your own data today. The fine-tuning example notebook walks through LoRA fine-tuning of Qwen3 8B, then deploys the result and compares it against a Llama 3.3 70B baseline. To learn more about the platform, visit the Serverless Fine-Tuning product page.