[BUG] 3bit quant and/or inference regression vs AutoGPTQ

[sidhantls]

Describe the bug
Quantizing a model to 3 bits using this repo leads to completely deteriorated performance. On MMLU, it gets 22%. However, when I quantize it using https://github.com/AutoGPTQ/AutoGPTQ, (which is where this repo was forked from?), I get 57%. This was using Llama-3.1-8B-Instruct.

Using this repo, for 4bit I get 66% on MMLU, which is in line with what AutoGPTQ gets for 4 bits

Anyone else noticed that 3bit doesn't work here but works in AutoGPTQ ?

Software Info

Operation System/Version + Python Version
python 3.10

To Reproduce
Quantize model to 3 bits:

    calibration_dataset = load_dataset(
        "allenai/c4",
        data_files="en/c4-train.00001-of-01024.json.gz",
        split="train"
    ).select(range(1024//2))["text"]

    # calibration_dataset = [" ".join(item.split()[:20]) for item in calibration_dataset] # speedup

    quantize_config = QuantizeConfig(
        bits=3,
        group_size=64,
    )

    model = GPTQModel.load(args.model_name, quantize_config)
    tokenizer = AutoTokenizer.from_pretrained(args.model_name, cache_dir=args.cache_dir)

    # increase `batch_size` to match gpu/vram specs to speed up quantization
    model.quantize(calibration_dataset, batch_size=args.batch_size)

Expected behavior

Performance not to break on 3bits and to allign with AutoGPTQ library

[Qubitium]

I am aware of the regression. We recently did a arc test for all bits and 3bits had unexpected lower score than 2bits.

We will backtrack and find out which commit(s) broke 3bit quant or inference.

[Qubitium]

@sidhantls We have currently bissected the commits and the 3bit ppl/accuracy regression happened between v1.6.0 and v1.8.0 (v1.6.0 had correct 3bit quality). We should have a fix isolated soon.

[Qubitium]

@sidhantls Please check main and see if 3 bits quality has returned to normal. It is now passing our 3bit quality ci tests.

[sidhantls]

@sidhantls Please check main and see if 3 bits quality has returned to normal. It is now passing our 3bit quality ci tests.

Thank you! Will do in the next few days

[sidhantls]

@Qubitium Hey, I just checked, it still does not seem to work - Does the CI test you used included evaluation on downstream data?

For Llama-3.2-3B, 3bits I get much higher performance on AutoGPTQ than here. For eg, MMLU is 50% using AutoGPTQ but this repo it is 24%

[Qubitium]

@sidhantls Can you send us the command and/or script code that yo uused to generate mmlu score? We need to replicate both the quant and scoring code for bug fix alignment. Thanks!

On a side note, gptqmodel calibration data handling is very different from autogptq and would inherently create two different quants. By default, we do not concat calibration data together, by default, and even if calibration_dataset_concat_size (concat mode) is used to mimic autogptq, we do token concatenation (not strings as as autogptq). Obviously, we feel our method produces better quants, based on our hundreds of experiements, given same high quality/realstic calibration data. There are also cases where concat dataset works better for native datasets that is not aligned with native model output. But overall, we recommend non-concat mode for the best quality at the cost of slower quants which can be offset with dataset batching.

With that said, the difference is too large to not be a bug. Once we isolate it, we can finally fix it for good. Thanks for your reports!

You can think of GPTQModel as a fork of autogptq in the begining but now vastly different in almost every possible way. With pending 2.0 release, you will be hard to recognize a single contious block code that is shared with autogptq without a microscrope. =)

[Qubitium]

Also please post your complete mmlu score if possible for the two models as mmlu (sum) is sub-divided into many categories. Currently our ci tests uses faster arc-challenge for downstream validation for many tests but I need to check if bits tests is using ppl or more rigorous lm-eval.

[Qubitium]

@sidhantls Ran out of time today but for tonight we got the following result: Benchmarked using Torch kernel, all 4bits, group_size=128, C4. We will do more testing tomorrow. Please send us your benchmarking script so we can align our test env.