Model execution runs excruciatingly slow/doesn't run at all

[JakeMalis]

🐛 Describe the bug

When running any inferences, the model doesn't load anything except maybe the first line. I've tried using Ollama and everything runs instantaneously, but I haven't gotten anything out of running any model on torchchat. Any idea why this is happening?

(.venv) (base) jakemalis@Jakes-MacBook-Pro torchchat % python3 torchchat.py generate llama3.1 --prompt "What's your favorite color?"              
Downloading meta-llama/Meta-Llama-3.1-8B-Instruct from HuggingFace...
original/params.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 199/199 [00:00<00:00, 1.91MB/s]
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 855/855 [00:00<00:00, 1.96MB/s]
.gitattributes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 1.65MB/s]
generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 184/184 [00:00<00:00, 245kB/s]
README.md: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44.0k/44.0k [00:00<00:00, 3.38MB/s]
USE_POLICY.md: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.69k/4.69k [00:00<00:00, 15.7MB/s]
LICENSE: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.63k/7.63k [00:00<00:00, 49.4MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 296/296 [00:00<00:00, 3.53MB/s]
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 55.4k/55.4k [00:00<00:00, 3.53MB/s]
tokenizer.model: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.18M/2.18M [00:00<00:00, 3.05MB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.09M/9.09M [00:02<00:00, 3.63MB/s]
consolidated.00.pth: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.1G/16.1G [08:07<00:00, 32.9MB/s]
Fetching 12 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [08:07<00:00, 40.66s/it]
Converting meta-llama/Meta-Llama-3.1-8B-Instruct to torchchat format...███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.09M/9.09M [00:02<00:00, 3.64MB/s]NumExpr defaulting to 10 threads.
PyTorch version 2.7.0.dev20250124 available.
Warning: PTEModel (ExecuTorch) not available with exception: No module named 'executorch'
known configs: ['llava-1.5', '13B', '70B', 'CodeLlama-7b-Python-hf', 'Meta-Llama-3.1-70B-Tune', 'Granite-3B-Code', '34B', 'Meta-Llama-3.1-8B', 'stories42M', 'Llama-Guard-3-1B', '30B', 'Meta-Llama-3.1-8B-Tune', 'stories110M', 'Granite-3.1-8B-Instruct', 'Llama-3.2-11B-Vision', 'Meta-Llama-3.2-3B', 'Meta-Llama-3.1-70B', 'Meta-Llama-3.2-1B', 'Granite-3.0-2B-Instruct', 'Granite-3.0-8B-Instruct', '7B', 'stories15M', 'Llama-Guard-3-1B-INT4', 'Mistral-7B', 'Granite-8B-Code', 'Meta-Llama-3-70B', 'Granite-3.1-2B-Instruct', 'Meta-Llama-3-8B']
Model config {'block_size': 131072, 'vocab_size': 128256, 'n_layers': 32, 'n_heads': 32, 'dim': 4096, 'hidden_dim': 14336, 'n_local_heads': 8, 'head_dim': 128, 'rope_base': 500000.0, 'norm_eps': 1e-05, 'multiple_of': 1024, 'ffn_dim_multiplier': 1.3, 'use_tiktoken': True, 'use_hf_tokenizer': False, 'tokenizer_prepend_bos': True, 'max_seq_length': 8192, 'rope_scaling': {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192}, 'n_stages': 1, 'stage_idx': 0, 'attention_bias': False, 'feed_forward_bias': False, 'tie_word_embeddings': False, 'embedding_multiplier': None, 'attention_multiplier': None, 'residual_multiplier': None, 'logits_scaling': None}
Moving checkpoint to /Users/jakemalis/.torchchat/model-cache/downloads/meta-llama/Meta-Llama-3.1-8B-Instruct/model.pth.
Done.
Moving model to /Users/jakemalis/.torchchat/model-cache/meta-llama/Meta-Llama-3.1-8B-Instruct.
Unable to import torchao experimental quant_api with error:  [Errno 2] No such file or directory: '/Users/jakemalis/Downloads/torchchat/torchao-build/src/ao/torchao/experimental/quant_api.py'
Using device=mps 
Loading model...
Time to load model: 23.71 seconds
-----------------------------------------------------------
What's your favorite color? - A simple

Versions

CMakeLists.txt		CONTRIBUTING.md		README.md		collect_env.py		docs			runner			tokenizer		torchchat.py
CODE_OF_CONDUCT.md	LICENSE			assets			dist_run.py		install			tests			torchchat
(.venv) (base) jakemalis@Jakes-MacBook-Pro torchchat % python collect_env.py
Collecting environment information...
PyTorch version: 2.7.0.dev20250124
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.3 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.6)
CMake version: version 3.31.4
Libc version: N/A

Python version: 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:54:21) [Clang 16.0.6 ] (64-bit runtime)
Python platform: macOS-15.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==2.2.2
[pip3] torch==2.7.0.dev20250124
[pip3] torchao==0.8.0+git2f97b095
[pip3] torchtune==0.6.0.dev20250124+cpu
[pip3] torchvision==0.22.0.dev20250124
[conda] numpy                     2.2.1                    pypi_0    pypi
[conda] numpy-base                2.1.3           py312he047099_0  
[conda] pytorch                   2.6.0.dev20241112        py3.12_0    pytorch-nightly
[conda] torchaudio                2.5.0.dev20241118       py312_cpu    pytorch-nightly
[conda] torchvision               0.20.0.dev20241118       py312_cpu    pytorch-nightly

[manuelcandales]

You are running the 8B model with no quantization. I think this is happening because of high Memory Pressure. How much RAM does your M1 Pro have?
The same thing happens to me on an M1 Pro with 16GB of RAM, even when nothing else is open at the same time. And it also happens to me on an M1 Pro with 32GB of RAM, if running a few other programs at the same time (Chrome, VC Code, etc).
You can confirm this, by opening the Activity Monitor app on your Mac, and watch it while you run torchchat on your terminal.

I haven't gotten anything out of running any model on torchchat

Try running it with 4-bit quantization:

python3 torchchat.py generate llama3.1 --dtype float16 --quantize '{"linear:int4": {"groupsize": 256}}' --prompt "Once upon a time," --num-samples 3

Also, have you tried running the 1B model? Memory Pressure should not be a problem at all, if you run the 1B model with 4-bit quantization.

Thank you!

[manuelcandales]

cc: @malfet @swolchok this user seems to be running into the same issue I was talking about.

[swolchok]

How much RAM does your M1 Pro have?
Have you tried running the 1B model?

if the same model works with another runtime, then the presumption should be that the problem is on our end.

[swolchok]

oh, I see -- ollama uses quantization by default. I ran llama3.2 and llama3.1 and sure enough it was llama.cpp's Q4_K_M. Then yeah ollama's default settings don't seem to be an apples-to-apples comparison with torchchat's default settings as they currently stand.

[mikekgfb]

#1465 adds the ability to save and load snapshots, so we can have prequantized models as well, and users don't have to start up and pay for quantization every time. That being said, redistributing quantized snapshots is something that Meta employees can do without detailed review, but third parties would not be blocked from doing what TheBloke does for quantizing models for llama.cpp which I think is what ollama may be using in this setting?

[JakeMalis]

You are running the 8B model with no quantization. I think this is happening because of high Memory Pressure. How much RAM does your M1 Pro have? The same thing happens to me on an M1 Pro with 16GB of RAM, even when nothing else is open at the same time. And it also happens to me on an M1 Pro with 32GB of RAM, if running a few other programs at the same time (Chrome, VC Code, etc). You can confirm this, by opening the Activity Monitor app on your Mac, and watch it while you run torchchat on your terminal.

I haven't gotten anything out of running any model on torchchat

Try running it with 4-bit quantization:

python3 torchchat.py generate llama3.1 --dtype float16 --quantize '{"linear:int4": {"groupsize": 256}}' --prompt "Once upon a time," --num-samples 3

Also, have you tried running the 1B model? Memory Pressure should not be a problem at all, if you run the 1B model with 4-bit quantization.

Thank you!

My Mac has 16GB or RAM. Here's running llama3.1 8b:

(.venv) (base) jakemalis@Jakes-MacBook-Pro torchchat % python3 torchchat.py generate llama3.1 --dtype float16 --quantize '{"linear:int4": {"groupsize": 256}}' --prompt "Once upon a time," --num-samples 3
NumExpr defaulting to 10 threads.
PyTorch version 2.7.0.dev20250124 available.
Warning: PTEModel (ExecuTorch) not available with exception: No module named 'executorch'
Unable to import torchao experimental quant_api with error:  [Errno 2] No such file or directory: '/Users/jakemalis/Downloads/torchchat/torchao-build/src/ao/torchao/experimental/quant_api.py'
Using device=mps 
Loading model...
Time to load model: 75.42 seconds
Quantizing the model with: {'linear:int4': {'groupsize': 256}}
linear: model.layers.0.attention.wq, in=4096, out=4096
linear: model.layers.0.attention.wk, in=4096, out=1024
linear: model.layers.0.attention.wv, in=4096, out=1024
linear: model.layers.0.attention.wo, in=4096, out=4096
linear: model.layers.0.feed_forward.w1, in=4096, out=14336
linear: model.layers.0.feed_forward.w2, in=14336, out=4096
linear: model.layers.0.feed_forward.w3, in=4096, out=14336
linear: model.layers.1.attention.wq, in=4096, out=4096
linear: model.layers.1.attention.wk, in=4096, out=1024
linear: model.layers.1.attention.wv, in=4096, out=1024
linear: model.layers.1.attention.wo, in=4096, out=4096
linear: model.layers.1.feed_forward.w1, in=4096, out=14336
linear: model.layers.1.feed_forward.w2, in=14336, out=4096
linear: model.layers.1.feed_forward.w3, in=4096, out=14336
linear: model.layers.2.attention.wq, in=4096, out=4096
linear: model.layers.2.attention.wk, in=4096, out=1024
linear: model.layers.2.attention.wv, in=4096, out=1024
linear: model.layers.2.attention.wo, in=4096, out=4096
linear: model.layers.2.feed_forward.w1, in=4096, out=14336
linear: model.layers.2.feed_forward.w2, in=14336, out=4096
linear: model.layers.2.feed_forward.w3, in=4096, out=14336
linear: model.layers.3.attention.wq, in=4096, out=4096
linear: model.layers.3.attention.wk, in=4096, out=1024
linear: model.layers.3.attention.wv, in=4096, out=1024
linear: model.layers.3.attention.wo, in=4096, out=4096
linear: model.layers.3.feed_forward.w1, in=4096, out=14336
linear: model.layers.3.feed_forward.w2, in=14336, out=4096
linear: model.layers.3.feed_forward.w3, in=4096, out=14336
linear: model.layers.4.attention.wq, in=4096, out=4096
linear: model.layers.4.attention.wk, in=4096, out=1024
linear: model.layers.4.attention.wv, in=4096, out=1024
linear: model.layers.4.attention.wo, in=4096, out=4096
linear: model.layers.4.feed_forward.w1, in=4096, out=14336
linear: model.layers.4.feed_forward.w2, in=14336, out=4096
linear: model.layers.4.feed_forward.w3, in=4096, out=14336
linear: model.layers.5.attention.wq, in=4096, out=4096
linear: model.layers.5.attention.wk, in=4096, out=1024
linear: model.layers.5.attention.wv, in=4096, out=1024
linear: model.layers.5.attention.wo, in=4096, out=4096
linear: model.layers.5.feed_forward.w1, in=4096, out=14336
linear: model.layers.5.feed_forward.w2, in=14336, out=4096
linear: model.layers.5.feed_forward.w3, in=4096, out=14336
linear: model.layers.6.attention.wq, in=4096, out=4096
linear: model.layers.6.attention.wk, in=4096, out=1024
linear: model.layers.6.attention.wv, in=4096, out=1024
linear: model.layers.6.attention.wo, in=4096, out=4096
linear: model.layers.6.feed_forward.w1, in=4096, out=14336
linear: model.layers.6.feed_forward.w2, in=14336, out=4096
linear: model.layers.6.feed_forward.w3, in=4096, out=14336
linear: model.layers.7.attention.wq, in=4096, out=4096
linear: model.layers.7.attention.wk, in=4096, out=1024
linear: model.layers.7.attention.wv, in=4096, out=1024
linear: model.layers.7.attention.wo, in=4096, out=4096
linear: model.layers.7.feed_forward.w1, in=4096, out=14336
linear: model.layers.7.feed_forward.w2, in=14336, out=4096
linear: model.layers.7.feed_forward.w3, in=4096, out=14336
linear: model.layers.8.attention.wq, in=4096, out=4096
linear: model.layers.8.attention.wk, in=4096, out=1024
linear: model.layers.8.attention.wv, in=4096, out=1024
linear: model.layers.8.attention.wo, in=4096, out=4096
linear: model.layers.8.feed_forward.w1, in=4096, out=14336
linear: model.layers.8.feed_forward.w2, in=14336, out=4096
linear: model.layers.8.feed_forward.w3, in=4096, out=14336
linear: model.layers.9.attention.wq, in=4096, out=4096
linear: model.layers.9.attention.wk, in=4096, out=1024
linear: model.layers.9.attention.wv, in=4096, out=1024
linear: model.layers.9.attention.wo, in=4096, out=4096
linear: model.layers.9.feed_forward.w1, in=4096, out=14336
linear: model.layers.9.feed_forward.w2, in=14336, out=4096
linear: model.layers.9.feed_forward.w3, in=4096, out=14336
linear: model.layers.10.attention.wq, in=4096, out=4096
linear: model.layers.10.attention.wk, in=4096, out=1024
linear: model.layers.10.attention.wv, in=4096, out=1024
linear: model.layers.10.attention.wo, in=4096, out=4096
linear: model.layers.10.feed_forward.w1, in=4096, out=14336
linear: model.layers.10.feed_forward.w2, in=14336, out=4096
linear: model.layers.10.feed_forward.w3, in=4096, out=14336
linear: model.layers.11.attention.wq, in=4096, out=4096
linear: model.layers.11.attention.wk, in=4096, out=1024
linear: model.layers.11.attention.wv, in=4096, out=1024
linear: model.layers.11.attention.wo, in=4096, out=4096
linear: model.layers.11.feed_forward.w1, in=4096, out=14336
linear: model.layers.11.feed_forward.w2, in=14336, out=4096
linear: model.layers.11.feed_forward.w3, in=4096, out=14336
linear: model.layers.12.attention.wq, in=4096, out=4096
linear: model.layers.12.attention.wk, in=4096, out=1024
linear: model.layers.12.attention.wv, in=4096, out=1024
linear: model.layers.12.attention.wo, in=4096, out=4096
linear: model.layers.12.feed_forward.w1, in=4096, out=14336
linear: model.layers.12.feed_forward.w2, in=14336, out=4096
linear: model.layers.12.feed_forward.w3, in=4096, out=14336
linear: model.layers.13.attention.wq, in=4096, out=4096
linear: model.layers.13.attention.wk, in=4096, out=1024
linear: model.layers.13.attention.wv, in=4096, out=1024
linear: model.layers.13.attention.wo, in=4096, out=4096
linear: model.layers.13.feed_forward.w1, in=4096, out=14336
linear: model.layers.13.feed_forward.w2, in=14336, out=4096
linear: model.layers.13.feed_forward.w3, in=4096, out=14336
linear: model.layers.14.attention.wq, in=4096, out=4096
linear: model.layers.14.attention.wk, in=4096, out=1024
linear: model.layers.14.attention.wv, in=4096, out=1024
linear: model.layers.14.attention.wo, in=4096, out=4096
linear: model.layers.14.feed_forward.w1, in=4096, out=14336
linear: model.layers.14.feed_forward.w2, in=14336, out=4096
linear: model.layers.14.feed_forward.w3, in=4096, out=14336
linear: model.layers.15.attention.wq, in=4096, out=4096
linear: model.layers.15.attention.wk, in=4096, out=1024
linear: model.layers.15.attention.wv, in=4096, out=1024
linear: model.layers.15.attention.wo, in=4096, out=4096
linear: model.layers.15.feed_forward.w1, in=4096, out=14336
linear: model.layers.15.feed_forward.w2, in=14336, out=4096
linear: model.layers.15.feed_forward.w3, in=4096, out=14336
linear: model.layers.16.attention.wq, in=4096, out=4096
linear: model.layers.16.attention.wk, in=4096, out=1024
linear: model.layers.16.attention.wv, in=4096, out=1024
linear: model.layers.16.attention.wo, in=4096, out=4096
linear: model.layers.16.feed_forward.w1, in=4096, out=14336
Time to quantize model: 34.20 seconds
Traceback (most recent call last):
  File "/Users/jakemalis/Downloads/torchchat/torchchat.py", line 96, in <module>
    generate_main(args)
  File "/Users/jakemalis/Downloads/torchchat/torchchat/generate.py", line 1651, in main
    run_generator(args)
  File "/Users/jakemalis/Downloads/torchchat/torchchat/generate.py", line 1619, in run_generator
    gen = Generator(
          ^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/torchchat/generate.py", line 381, in __init__
    self.model = _initialize_model(self.builder_args, self.quantize, self.tokenizer)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/torchchat/cli/builder.py", line 736, in _initialize_model
    quantize_model(
  File "/Users/jakemalis/Downloads/torchchat/torchchat/utils/quantize.py", line 144, in quantize_model
    model = quant_handler.quantize(model)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/.venv/lib/python3.12/site-packages/torchao/quantization/GPTQ.py", line 816, in quantize
    state_dict = self._create_quantized_state_dict(model)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/.venv/lib/python3.12/site-packages/torchao/quantization/GPTQ.py", line 775, in _create_quantized_state_dict
    (w_int4x8, scales_and_zeros) = groupwise_affine_quantize_tensor(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/.venv/lib/python3.12/site-packages/torchao/quantization/utils.py", line 462, in groupwise_affine_quantize_tensor
    w_int4x8 = groupwise_affine_quantize_tensor_from_qparams(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jakemalis/Downloads/torchchat/.venv/lib/python3.12/site-packages/torchao/quantization/utils.py", line 407, in groupwise_affine_quantize_tensor_from_qparams
    int_data = (int_data[::, ::2] << 4 | int_data[::, 1::2]).to(torch.uint8)
                ~~~~~~~~~~~~~~~~~~^^~~
RuntimeError: MPS backend out of memory (MPS allocated: 18.11 GB, other allocations: 384.00 KB, max allowed: 18.13 GB). Tried to allocate 112.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

Here's running the 1b model:

(.venv) (base) jakemalis@Jakes-MacBook-Pro torchchat % python3 torchchat.py generate llama3.2-1b --dtype float16 --quantize '{"linear:int4": {"groupsize": 256}}' --prompt "Once upon a time," --num-samples 3
NumExpr defaulting to 10 threads.
PyTorch version 2.7.0.dev20250124 available.
Warning: PTEModel (ExecuTorch) not available with exception: No module named 'executorch'
Unable to import torchao experimental quant_api with error:  [Errno 2] No such file or directory: '/Users/jakemalis/Downloads/torchchat/torchao-build/src/ao/torchao/experimental/quant_api.py'
Using device=mps 
Loading model...
Time to load model: 1.27 seconds
Quantizing the model with: {'linear:int4': {'groupsize': 256}}
linear: model.layers.0.attention.wq, in=2048, out=2048
linear: model.layers.0.attention.wk, in=2048, out=512
linear: model.layers.0.attention.wv, in=2048, out=512
linear: model.layers.0.attention.wo, in=2048, out=2048
linear: model.layers.0.feed_forward.w1, in=2048, out=8192
linear: model.layers.0.feed_forward.w2, in=8192, out=2048
linear: model.layers.0.feed_forward.w3, in=2048, out=8192
linear: model.layers.1.attention.wq, in=2048, out=2048
linear: model.layers.1.attention.wk, in=2048, out=512
linear: model.layers.1.attention.wv, in=2048, out=512
linear: model.layers.1.attention.wo, in=2048, out=2048
linear: model.layers.1.feed_forward.w1, in=2048, out=8192
linear: model.layers.1.feed_forward.w2, in=8192, out=2048
linear: model.layers.1.feed_forward.w3, in=2048, out=8192
linear: model.layers.2.attention.wq, in=2048, out=2048
linear: model.layers.2.attention.wk, in=2048, out=512
linear: model.layers.2.attention.wv, in=2048, out=512
linear: model.layers.2.attention.wo, in=2048, out=2048
linear: model.layers.2.feed_forward.w1, in=2048, out=8192
linear: model.layers.2.feed_forward.w2, in=8192, out=2048
linear: model.layers.2.feed_forward.w3, in=2048, out=8192
linear: model.layers.3.attention.wq, in=2048, out=2048
linear: model.layers.3.attention.wk, in=2048, out=512
linear: model.layers.3.attention.wv, in=2048, out=512
linear: model.layers.3.attention.wo, in=2048, out=2048
linear: model.layers.3.feed_forward.w1, in=2048, out=8192
linear: model.layers.3.feed_forward.w2, in=8192, out=2048
linear: model.layers.3.feed_forward.w3, in=2048, out=8192
linear: model.layers.4.attention.wq, in=2048, out=2048
linear: model.layers.4.attention.wk, in=2048, out=512
linear: model.layers.4.attention.wv, in=2048, out=512
linear: model.layers.4.attention.wo, in=2048, out=2048
linear: model.layers.4.feed_forward.w1, in=2048, out=8192
linear: model.layers.4.feed_forward.w2, in=8192, out=2048
linear: model.layers.4.feed_forward.w3, in=2048, out=8192
linear: model.layers.5.attention.wq, in=2048, out=2048
linear: model.layers.5.attention.wk, in=2048, out=512
linear: model.layers.5.attention.wv, in=2048, out=512
linear: model.layers.5.attention.wo, in=2048, out=2048
linear: model.layers.5.feed_forward.w1, in=2048, out=8192
linear: model.layers.5.feed_forward.w2, in=8192, out=2048
linear: model.layers.5.feed_forward.w3, in=2048, out=8192
linear: model.layers.6.attention.wq, in=2048, out=2048
linear: model.layers.6.attention.wk, in=2048, out=512
linear: model.layers.6.attention.wv, in=2048, out=512
linear: model.layers.6.attention.wo, in=2048, out=2048
linear: model.layers.6.feed_forward.w1, in=2048, out=8192
linear: model.layers.6.feed_forward.w2, in=8192, out=2048
linear: model.layers.6.feed_forward.w3, in=2048, out=8192
linear: model.layers.7.attention.wq, in=2048, out=2048
linear: model.layers.7.attention.wk, in=2048, out=512
linear: model.layers.7.attention.wv, in=2048, out=512
linear: model.layers.7.attention.wo, in=2048, out=2048
linear: model.layers.7.feed_forward.w1, in=2048, out=8192
linear: model.layers.7.feed_forward.w2, in=8192, out=2048
linear: model.layers.7.feed_forward.w3, in=2048, out=8192
linear: model.layers.8.attention.wq, in=2048, out=2048
linear: model.layers.8.attention.wk, in=2048, out=512
linear: model.layers.8.attention.wv, in=2048, out=512
linear: model.layers.8.attention.wo, in=2048, out=2048
linear: model.layers.8.feed_forward.w1, in=2048, out=8192
linear: model.layers.8.feed_forward.w2, in=8192, out=2048
linear: model.layers.8.feed_forward.w3, in=2048, out=8192
linear: model.layers.9.attention.wq, in=2048, out=2048
linear: model.layers.9.attention.wk, in=2048, out=512
linear: model.layers.9.attention.wv, in=2048, out=512
linear: model.layers.9.attention.wo, in=2048, out=2048
linear: model.layers.9.feed_forward.w1, in=2048, out=8192
linear: model.layers.9.feed_forward.w2, in=8192, out=2048
linear: model.layers.9.feed_forward.w3, in=2048, out=8192
linear: model.layers.10.attention.wq, in=2048, out=2048
linear: model.layers.10.attention.wk, in=2048, out=512
linear: model.layers.10.attention.wv, in=2048, out=512
linear: model.layers.10.attention.wo, in=2048, out=2048
linear: model.layers.10.feed_forward.w1, in=2048, out=8192
linear: model.layers.10.feed_forward.w2, in=8192, out=2048
linear: model.layers.10.feed_forward.w3, in=2048, out=8192
linear: model.layers.11.attention.wq, in=2048, out=2048
linear: model.layers.11.attention.wk, in=2048, out=512
linear: model.layers.11.attention.wv, in=2048, out=512
linear: model.layers.11.attention.wo, in=2048, out=2048
linear: model.layers.11.feed_forward.w1, in=2048, out=8192
linear: model.layers.11.feed_forward.w2, in=8192, out=2048
linear: model.layers.11.feed_forward.w3, in=2048, out=8192
linear: model.layers.12.attention.wq, in=2048, out=2048
linear: model.layers.12.attention.wk, in=2048, out=512
linear: model.layers.12.attention.wv, in=2048, out=512
linear: model.layers.12.attention.wo, in=2048, out=2048
linear: model.layers.12.feed_forward.w1, in=2048, out=8192
linear: model.layers.12.feed_forward.w2, in=8192, out=2048
linear: model.layers.12.feed_forward.w3, in=2048, out=8192
linear: model.layers.13.attention.wq, in=2048, out=2048
linear: model.layers.13.attention.wk, in=2048, out=512
linear: model.layers.13.attention.wv, in=2048, out=512
linear: model.layers.13.attention.wo, in=2048, out=2048
linear: model.layers.13.feed_forward.w1, in=2048, out=8192
linear: model.layers.13.feed_forward.w2, in=8192, out=2048
linear: model.layers.13.feed_forward.w3, in=2048, out=8192
linear: model.layers.14.attention.wq, in=2048, out=2048
linear: model.layers.14.attention.wk, in=2048, out=512
linear: model.layers.14.attention.wv, in=2048, out=512
linear: model.layers.14.attention.wo, in=2048, out=2048
linear: model.layers.14.feed_forward.w1, in=2048, out=8192
linear: model.layers.14.feed_forward.w2, in=8192, out=2048
linear: model.layers.14.feed_forward.w3, in=2048, out=8192
linear: model.layers.15.attention.wq, in=2048, out=2048
linear: model.layers.15.attention.wk, in=2048, out=512
linear: model.layers.15.attention.wv, in=2048, out=512
linear: model.layers.15.attention.wo, in=2048, out=2048
linear: model.layers.15.feed_forward.w1, in=2048, out=8192
linear: model.layers.15.feed_forward.w2, in=8192, out=2048
linear: model.layers.15.feed_forward.w3, in=2048, out=8192
linear: model.output, in=2048, out=128256
Time to quantize model: 0.48 seconds
-----------------------------------------------------------
Once upon a time, in a small, private chapel, a beautiful, ornate chandelier hung from the ceiling, casting a warm, golden light down on the rows of pews. The air was thick with the scent of fresh flowers, and the sound of soft, whispery music drifted through the open window.

As she sat in the front pew, a young girl named Sarah felt a sense of peace settle over her. She had been there before, many times before, and the familiarity of the familiar rituals and images allowed her to feel a sense of calm wash over her. She had always felt safe, and she knew that she would be taken care of.

This was her 10th wedding anniversary, and her husband, John, had planned a special celebration. It was going to be a quiet, intimate affair, just the two of them. They had always been so dedicated to their marriage, and Sarah loved the way they looked at each other, and how they did everything together.

But for Sarah,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
Generated 199 tokens                 
Time for inference 1: 6.9387 sec total                 
Time to first token: 1.8589 sec with parallel prefill.                

      Total throughput: 28.8239 tokens/sec, 0.0347 s/token                 
First token throughput: 0.5379 tokens/sec, 1.8589 s/token                 
 Next token throughput: 39.1750 tokens/sec, 0.0255 s/token                     

Bandwidth achieved: 33.51 GB/s
*** This first iteration will include cold start effects for dynamic import, hardware caches. ***

========================================

Once upon a time, there was a man named Jack, who was a bit of an oddball. He was a bit of a loner, and he preferred to keep to himself, but deep down, he was a romantic at heart. He longed for connection and companionship, but he was afraid of getting hurt.

Jack's life was like a rollercoaster. One day he would leave the safety of his home and head out into the world, only to find himself alone and unfulfilled. The next day, he would return home, exhausted and dejected, only to find himself stuck in his routine and stuck in a loveless relationship. He felt like he was stuck in a cage, and he couldn't get out.

Jack's family and friends were not very understanding of his escapades. They thought he was crazy, or worse, they thought he didn't care about them. They didn't know why Jack did what he did. But deep down, Jack knew that he was unhappy,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
Generated 199 tokens                 
Time for inference 2: 4.4095 sec total                 
Time to first token: 0.0179 sec with parallel prefill.                

      Total throughput: 45.3563 tokens/sec, 0.0220 s/token                 
First token throughput: 55.8047 tokens/sec, 0.0179 s/token                 
 Next token throughput: 45.3137 tokens/sec, 0.0221 s/token                     

Bandwidth achieved: 52.73 GB/s

========================================

Once upon a time, there were two mice, Max and Midge. Midge was a bit of a rebel, while Max was a bit of a square.

Max was a bit of a skeptic, and he liked to do things by the book. He always wanted to know what was going to happen before the event, and he rarely questioned authority.

Midge, on the other hand, was a free spirit. She didn't always want to do things by the book, and she liked to take risks and try new things. She believed in following her own intuition, and she never hesitated to ignore the rules or conventions.

One day, they stumbled upon a map that looked like it might have been drawn by a child. It showed a path from one of the old buildings to the other, a path that Midge had always known existed. But it was a path that Max had never seen before.

"This can't be right," Max said, his eyes narrowing as he gazed at the map.

"It
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
Generated 199 tokens                 
Time for inference 3: 4.2582 sec total                 
Time to first token: 0.0179 sec with parallel prefill.                

      Total throughput: 46.9683 tokens/sec, 0.0213 s/token                 
First token throughput: 55.9042 tokens/sec, 0.0179 s/token                 
 Next token throughput: 46.9306 tokens/sec, 0.0213 s/token                     

Bandwidth achieved: 54.61 GB/s

========================================


Warning: Excluding compile in calculations                 
      Average tokens/sec (total): 40.38                 
Average tokens/sec (first token): 37.42                 
Average tokens/sec (next tokens): 43.81

When I ran the chat interface version of llama3.2-1b, it was still significantly slower than running llama3.1-8b on Ollama. Are you sure that the issue only has to do with the way Ollama uses quantization? Is there a chance they are using MLX? Does torch chat support/use MLX?

[JakeMalis]

Also, why am I able to download the models from Ollama without logging into Huggingface, but I need to log in to Huggingface and request access before using models with Torchchat?

[mikekgfb]

This worked fine on my Mac with 36 GB. Ostensibly, this should work on smaller models as well, but may need more sophistication during quantization? It seems we're running out of memory at that point? I think we've seen this previously for limited GPU memory as well -- if you load the entire model into GPU and then start to quantize, the process may not be successfull, because the model might not fit, or just barely fit.

The way to solve this might be to hold all weights on CPU, and transfer each weight only when it is ready to be quantized -- that way memory won't get overloaded by unquantized weights (although CPU and MPS share system memory on Mac? That being said, I think you might be able to page out CPU memory but not MPS?).

Also, making sure that any unquantized tensors are de-allocated. As an experiment you might try setting w= None before the line in torchao - I don't know whether it will be used anywhere else - and it may have to be done at the call site to wipe out all references to the unquantized weights

python3 torchchat.py generate llama3.1 --dtype float16 --quantize '{"linear:int4": {"groupsize": 256}}' --prompt "Once upon a time," --num-samples 3

Here's what I get qith the 4b quantization setting from above

Time to quantize model: 7.10 seconds
-----------------------------------------------------------

[...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Generated 199 tokens
Time for inference 3: 10.1121 sec total
Time to first token: 0.3303 sec with parallel prefill.

      Total throughput: 19.7782 tokens/sec, 0.0506 s/token
First token throughput: 3.0278 tokens/sec, 0.3303 s/token
 Next token throughput: 20.3437 tokens/sec, 0.0492 s/token

Bandwidth achieved: 97.32 GB/s

========================================


Warning: Excluding compile in calculations
      Average tokens/sec (total): 18.20
Average tokens/sec (first token): 2.05
Average tokens/sec (next tokens): 19.99

And here is without quantization:

Time for inference 3: 26.4162 sec total
Time to first token: 0.1896 sec with parallel prefill.

      Total throughput: 7.5711 tokens/sec, 0.1321 s/token
First token throughput: 5.2743 tokens/sec, 0.1896 s/token
 Next token throughput: 7.5877 tokens/sec, 0.1318 s/token

Bandwidth achieved: 121.60 GB/s

========================================


Warning: Excluding compile in calculations
      Average tokens/sec (total): 7.46
Average tokens/sec (first token): 3.89
Average tokens/sec (next tokens): 7.55

One way to sidestep the issue with quantization would be to distribute quantized models, like ollama does. That being said, I do not know what arrangements they have with Meta because the llama community license limits redistribution and requires users to obtain weights through the HF repo. As such, I don't think I can share the quantized models.

But if you have access to a Mac with larger memory, and tweaking quantization won't allow you to overcome the out of memory condition, you might use #1465 to quantize on a Mac system with more memory, and then load it on your laptop.

As an alternative (bandaid), you might also be able to quantize on CPU, and then tweak the loading procedure to load to load it into MPS even though it was quantized on CPU - may require a minor addition if it doesn't happen automatically... like adding model.to(device="mps") to the loader?

[Jack-Khuu]

Thanks for trying things out @JakeMalis

I've tried using Ollama and everything runs instantaneously, but I haven't gotten anything out of running any model on torchchat. Any idea why this is happening?

As @manuelcandales and @swolchok eluded to, the first cause is that torchchat uses the unquantizated model by default, while Ollama typically uses a 4-bit quantized variant of the model. This results in memory pressure. That said, we are considering changing torchchat to use create quantized variants by default.

When I ran the chat interface version of llama3.2-1b, it was still significantly slower

I believe this might be a bug on our side that has popped up a few times in the past (that doesn't appear to affect CUDA). We'll look into it

Is there a chance they are using MLX? Does torch chat support/use MLX?

torchchat currently doesn't support MLX. Last I checked Ollama was looking at MLX, but I don't recall if it's landed

why am I able to download the models from Ollama without logging into Huggingface, but I need to log in to Huggingface and request access before using models with Torchchat?

Ollama can runs on generated GGUFs, which don't necessarily require extra authentication from HF. If someone use Ollama with a private GGUF that does require additional auth, you would need to login to HF

torchchat is mostly using the official Meta HF repo (or IBM, etc.) which does require additional authentication.

[JakeMalis]

Thanks everyone for the information, I was able to get it working again using @mikekgfb's solution of python3 torchchat.py generate llama3.2-1b --dtype float16 --quantize '{"linear:int4": {"groupsize": 256}}' --prompt "Once upon a time," --num-samples 3 worked to get the model running. I don't have a specific use case, I was just play testing torch chat.

@Jack-Khuu I feel like Ollama may have to be using MLX because I was able to run Llama3.3 70B whereas I'm having issues running Llama 3.1:8b with torchchat. Thank you for the information!

Also, does torch chat support DeepSeek since it's on Huggingface?

[Jack-Khuu]

Also, does torch chat support DeepSeek since it's on Huggingface?

Not yet, but it's on my mind