Mixtral-8x7B-v0.1 TP 8 GPUS EDIT: TypeError: PaddedGatherOp.forward() takes 6 positional arguments but 7 were given

[orellavie1212]

fixed Error:
KeyError: 'model.layers.13.block_sparse_moe.experts.4.w2.weight'
fixed with 'pt' as mentioned in #2020

the new error is
TypeError: PaddedGatherOp.forward() takes 6 positional arguments but 7 were given

model:
https://huggingface.co/mistralai/Mixtral-8x7B-v0.1

python: 3.10

gpus:
g5.48xlarge
8 x 24G A10 on AWS

logs:

023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: llm = LLM(model=model_name, tensor_parallel_size=num_gpus, dtype=dtype, trust_remote_code=trust_remote_code,
-- | --
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 93, in __init__
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: self.llm_engine = LLMEngine.from_engine_args(engine_args)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 246, in from_engine_args
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: engine = cls(*engine_configs,
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 107, in __init__
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: self._init_workers_ray(placement_group)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 194, in _init_workers_ray
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: self._run_workers(
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 750, in _run_workers
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: self._run_workers_in_batch(workers, method, *args, **kwargs))
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 727, in _run_workers_in_batch
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: all_outputs = ray.get(all_outputs)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: return fn(*args, **kwargs)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2524, in get
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: raise value.as_instanceof_cause()
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: ray.exceptions.RayTaskError(KeyError): #033[36mray::RayWorkerVllm.execute_method()#033[39m (pid=5603, ip=169.254.181.2, actor_id=9f26cd20cd05e16c7c08648d01000000, repr=<vllm.engine.ray_utils.RayWorkerVllm object at 0x7f84a1d667a0>)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: return executor(*args, **kwargs)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 72, in load_model
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: self.model_runner.load_model()
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 36, in load_model
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: self.model = get_model(self.model_config)
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader.py", line 124, in get_model
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: model.load_weights(model_config.model, model_config.download_dir,
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 531, in load_weights
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: param = params_dict[name]
  | 2023-12-11T17:25:22.531+02:00 | [WARN ] PyProcess - W-1511-model-stderr: KeyError: 'model.layers.13.block_sparse_moe.experts.4.w2.weight'

now the problem is with the logs

llm = LLM(model=model_name, tensor_parallel_size=num_gpus, dtype=dtype, trust_remote_code=trust_remote_code,
--
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 93, in __init__
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self.llm_engine = LLMEngine.from_engine_args(engine_args)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 246, in from_engine_args
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: engine = cls(*engine_configs,
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 112, in __init__
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self._init_cache()
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 208, in _init_cache
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: num_blocks = self._run_workers(
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 750, in _run_workers
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self._run_workers_in_batch(workers, method, *args, **kwargs))
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 727, in _run_workers_in_batch
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: all_outputs = ray.get(all_outputs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return fn(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2524, in get
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: raise value.as_instanceof_cause()
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: ray.exceptions.RayTaskError(TypeError): #033[36mray::RayWorkerVllm.execute_method()#033[39m (pid=4660, ip=169.254.181.2, actor_id=f424ccef141bef01e9887cf001000000, repr=<vllm.engine.ray_utils.RayWorkerVllm object at 0x7ed62ad827a0>)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return executor(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 88, in profile_num_available_blocks
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self.model_runner.profile_run()
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 321, in profile_run
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self.execute_model(seqs, kv_caches)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 279, in execute_model
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: hidden_states = self.model(
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return self._call_impl(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return forward_call(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 488, in forward
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: hidden_states = layer(
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return self._call_impl(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return forward_call(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 439, in forward
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: r = self.block_sparse_moe(self.ffn_norm(h))
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return self._call_impl(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return forward_call(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 353, in forward
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: x = ops.padded_gather(x, indices, bin_ids, bins, padded_bins,
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 539, in apply
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return super().apply(*args, **kwargs) # type: ignore[misc]
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/stk/backend/autocast.py", line 28, in decorate_fwd
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return fwd(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: TypeError: PaddedGatherOp.forward() takes 6 positional arguments but 7 were given

requirements.txt:

langdetect
fastapi
uvicorn[standard]
ninja # For faster builds.
psutil
ray==2.6.3
numpy
huggingface-hub>=0.16.4
wrapt-timeout-decorator
pydantic < 2 # Required for OpenAI server.
scipy
pandas
pyarrow
safetensors
sentencepiece
einops
torch==2.1.0
torchvision==0.16.0
deepspeed>=0.12.3
transformers==4.36.0
accelerate>=0.24.1
peft>=0.6.2
bitsandbytes>=0.41.2.post2
auto_gptq>=0.5.1
datasets==2.15.0
megablocks #mixtral
stanford-stk #mixtral
git+https://github.com/vllm-project/vllm.git@b5f882cc98e2c9c6dde7357dbac2ec0c2c57d8cd

Note:
with TP 4 I can't even run it (ofc OOM problems, the gpu memory for 4 gpus is not enough I guess)

[void-b583x2-NULL]

[#2020 ] Would this help?

[orellavie1212]

[#2020 ] Would this help?

it fixed the problem mentioned here
raised another one

`
llm = LLM(model=model_name, tensor_parallel_size=num_gpus, dtype=dtype, trust_remote_code=trust_remote_code,

  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 93, in init
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self.llm_engine = LLMEngine.from_engine_args(engine_args)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 246, in from_engine_args
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: engine = cls(*engine_configs,
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 112, in init
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self._init_cache()
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 208, in _init_cache
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: num_blocks = self._run_workers(
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 750, in _run_workers
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self._run_workers_in_batch(workers, method, *args, **kwargs))
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 727, in _run_workers_in_batch
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: all_outputs = ray.get(all_outputs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return fn(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2524, in get
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: raise value.as_instanceof_cause()
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: ray.exceptions.RayTaskError(TypeError): #33[36mray::RayWorkerVllm.execute_method()#33[39m (pid=4660, ip=169.254.181.2, actor_id=f424ccef141bef01e9887cf001000000, repr=<vllm.engine.ray_utils.RayWorkerVllm object at 0x7ed62ad827a0>)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return executor(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 88, in profile_num_available_blocks
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self.model_runner.profile_run()
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 321, in profile_run
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: self.execute_model(seqs, kv_caches)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 279, in execute_model
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: hidden_states = self.model(
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return self._call_impl(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return forward_call(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 488, in forward
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: hidden_states = layer(
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return self._call_impl(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return forward_call(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 439, in forward
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: r = self.block_sparse_moe(self.ffn_norm(h))
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return self._call_impl(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return forward_call(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return func(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/mixtral.py", line 353, in forward
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: x = ops.padded_gather(x, indices, bin_ids, bins, padded_bins,
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 539, in apply
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return super().apply(*args, **kwargs) # type: ignore[misc]
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: File "/usr/local/lib/python3.10/dist-packages/stk/backend/autocast.py", line 28, in decorate_fwd
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: return fwd(*args, **kwargs)
  | 2023-12-11T18:13:26.782+02:00 | [WARN ] PyProcess - W-575-model-stderr: TypeError: PaddedGatherOp.forward() takes 6 positional arguments but 7 were given

`

[tgale96]

Hi, can you try re-installing megablocks? We've updated our PyPi packages just now and I expect that should fix your issue.

[tgale96]

More on other related issue: #2017 (comment)

[orellavie1212]

Hi, can you try re-installing megablocks? We've updated our PyPi packages just now and I expect that should fix your issue.

I think you are right, the new logs show now everything fixed, besides a cache problem (tried with 0.2 on tp=8 x 24g a10)

033[2m#033[36m(RayWorkerVllm pid=4645)#033[0m container-0:4645:5341 [0] NCCL INFO Channel 00/02 : 0 1 2 3 4 5 6 7
--
  | 2023-12-11T20:10:23.756+02:00 | [INFO ] PyProcess - W-562-model-stdout: #033[2m#033[36m(RayWorkerVllm pid=4645)#033[0m container-0:4645:5341 [0] NCCL INFO Channel 01/02 : 0 1 2 3 4 5 6 7
  | 2023-12-11T20:10:23.756+02:00 | [INFO ] PyProcess - W-562-model-stdout: INFO 12-11 18:10:23 llm_engine.py:222] # GPU blocks: 0, # CPU blocks: 16384
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: Traceback (most recent call last):
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/tmp/.djl.ai/python/0.24.0-SNAPSHOT/djl_python_engine.py", line 117, in run_server
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: outputs = self.service.invoke_handler(function_name, inputs)
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/tmp/.djl.ai/python/0.24.0-SNAPSHOT/djl_python/service_loader.py", line 29, in invoke_handler
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: return getattr(self.module, function_name)(inputs)
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/opt/ml/model/model.py", line 155, in handle
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: model, sampling_params = get_model(inputs.get_properties())
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/opt/ml/model/model.py", line 60, in get_model
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: llm = LLM(model=model_name, tensor_parallel_size=num_gpus, dtype=dtype, trust_remote_code=trust_remote_code,
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 93, in __init__
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: self.llm_engine = LLMEngine.from_engine_args(engine_args)
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 246, in from_engine_args
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: engine = cls(*engine_configs,
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 112, in __init__
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: self._init_cache()
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 226, in _init_cache
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: raise ValueError("No available memory for the cache blocks. "
  | 2023-12-11T20:10:23.756+02:00 | [WARN ] PyProcess - W-562-model-stderr: ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine.

[tgale96]

Excellent! Thanks for verifying :)

[WoosukKwon]

Closed as it is fixed by pip install megablocks. Thanks @tgale96 for the helpful comments and quick updates!

[tgale96]

Happy to help :)