Does nvfp4 inference be supported？

[ZhuYe0729]

Hi,

Thanks for your great job! I can use the model_quant.py to ‘realquant’ or 'pseudoquant' large models successfully. However, when I used the transformers library to inference(code as the readme's example), I got some ValueErrors, such as "ValueError: Only 'mxfp4' is supported for forward_dtype for now.". I was very confused since it seemed that nvfp4 kernels is already supported? So if I remove the raise valueerror logic in transformers library and add changes to fp_quant.py as below, it can work well. So whether NVFP4-type inference is truly supported or not,or the issue actually lies in the Transformers library?

def adapt_fp_quant_config(config: FPQuantConfig):
    if config.forward_dtype == "mxfp4":
        forward_dtype = FPQuantDtype.MXFP4
    else:

        # raise ValueError(f"Unsupported forward dtype: {config.forward_dtype}")
        print(f"Warning: Unsupported forward dtype: {config.forward_dtype}, using NVFP4 instead.")
        forward_dtype = FPQuantDtype.NVFP4

    if config.backward_dtype == "bf16":
        backward_dtype = FPQuantDtype.BF16
    else:
        raise ValueError(f"Unsupported backward dtype: {config.backward_dtype}")

    return FPQuantLinearConfig(
        forward_dtype=forward_dtype,
        forward_method=config.forward_method,
        backward_dtype=backward_dtype,
        store_master_weights=config.store_master_weights,
        hadamard_group_size=config.hadamard_group_size,
        pseudoquantization=config.pseudoquantization,
        modules_to_not_convert=config.modules_to_not_convert,
    )

[BlackSamorez]

Please install transformers from the latest main commit.
It hasn't made it into the official release yet.

[ZhuYe0729]

Thanks for your help！However I got new error：
TypeError: Can't instantiate abstract class FPQuantHfQuantizer without an implementation for abstract method '_process_model_after_weight_loading'
It seems that in transformers4.56.1，transformers/quantizers/quantizer_fp_quant.py has:

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        return model

however in the latest version of transformers，it is missing.

[BlackSamorez]

Yes. Looks like somebody on transformers team rewrote the whole dispatch pipeline for quantized models and everything is broken now. I'm working on fixing it.

[BlackSamorez]

I have a draft of a fix.

[ZhuYe0729]

Thank you for your reply!
I have another question. I used qutlass to inference the real quantized model(llama3.1-8b) on RTX 5090, I found prefill phase was well accelerated but decoding phase was much slower than the original model. Is this situation reasonable? Below are my results.

To the best of my knowledge, W4A4 quantization will accelerate both the prefill and decoding stages: during the prefill stage, Tensor Cores can be leveraged to boost speed, while during the decoding stage, when the batch size is small, the reduced memory bandwidth requirements from quantization lead to faster decoding.

So I’m really confused—why does the decoding phase become significantly slower? I also tried a quick speed test by directly using model.generate to measure generation speed, and indeed found that the quantized model generates tokens more slowly.

[BlackSamorez]

model.generate is pretty terrible without torch.compile and fullgraph=True.
Also, for speedups it's better if you used the vLLM integration we provide: vllm-project/vllm#24440

However, the speedups are indeed smaller for smaller batch sizes on 5090. We haven't fully tuned the kernels for it.

[ZhuYe0729]

Thank you! My problem has been well solved.