Skip to content

--quantize is doing something surprising #788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
malfet opened this issue May 13, 2024 · 0 comments · Fixed by #791
Closed

--quantize is doing something surprising #788

malfet opened this issue May 13, 2024 · 0 comments · Fixed by #791
Assignees
@malfet

Comments

@malfet
Copy link
Contributor

malfet commented May 13, 2024

It's either surprisingly fast or it's not quantizing all the layers, based on almost identical timing it took to quantize stories110M and llama2:

% python3 torchchat.py generate stories110M --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 0.85 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 0.10 seconds

% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 31.72 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 0.20 seconds
@malfet malfet self-assigned this May 13, 2024
malfet added a commit that referenced this issue May 13, 2024
@malfet
Fix LinearInt8 recursive quantization
e9e1b24 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
@malfet malfet mentioned this issue May 13, 2024
Merged
malfet added a commit that referenced this issue May 14, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
1298983 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
malfet added a commit that referenced this issue Jul 17, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
ffe1131 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
malfet added a commit that referenced this issue Jul 17, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
3306def 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
malfet added a commit that referenced this issue Jul 17, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
99d95a1 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
malfet added a commit that referenced this issue Jul 17, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
2345ee5 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
malfet added a commit that referenced this issue Jul 17, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
cc09cec 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
malfet added a commit that referenced this issue Jul 17, 2024
@malfet
Fix LinearInt8 recursive quantization (#791)
7088618 
Test plan:
```
% python3 torchchat.py generate llama2 --dtype float16 --quantize '{"linear:int8": {"groupsize": 0}}' --prompt "Once upon a time," --device mps
Using device=mps 
Loading model...
Time to load model: 29.03 seconds
Quantizing the model with: {'linear:int8': {'groupsize': 0}}
Time to quantize model: 14.37 seconds
```

Fixes #788
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@malfet malfet

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix LinearInt8 recursive quantization pytorch/torchchat
1 participant
@malfet