Commit 90bc7d4
committed
Update base for Update on "Improve QAT nvfp4 numerics"
**Summary:** Similar to #2986, this commit improves the prepare vs convert SQNR of NVFP4 QAT from 12 to inf. This is achieved by refactoring NVFP4 QAT to mimic the PTQ numerics exactly, using a new linear class to incorporate both the quantization and mm logic.
**Unit tests:**
```
python test/quantization/test_qat.py -k test_qat_nvfp4
python test/quantization/test_qat.py -k test_quantize_api_nvfp4
```
**End-to-end tests:**
Fine-tuning Llama3.2-3B with and without this PR in axolotl:
- fine-tune for 1 epoch on yahma/alpaca-cleaned
- batch size 512, learning rate 2e-5, no gradient accumulation
Wikitext:
- With this PR, QAT nvfp4 quantized model achieved 15% lower perplexity than the quantized baseline
- Without this PR, QAT nvfp4 quantized model was about the same as the quantized baseline
```
==> Llama3.2-3B_baseline_bs512/eval_float.log <==
| | |none | 0|word_perplexity|↓ |9.418|± | N/A|
==> Llama3.2-3B_baseline_bs512/eval_quantized.log <==
| | |none | 0|word_perplexity|↓ |10.3681|± | N/A|
# QAT with this PR (quantized)
==> unsloth_model_lora_qat_int4_output/lm_eval_quantized.log <==
| | |none | 0|word_perplexity|↓ |10.2281|± | N/A|
```
[ghstack-poisoned]1 parent bf2208c commit 90bc7d4
File tree
0 file changed
+0
-0
lines changed0 file changed
+0
-0
lines changed
0 commit comments