Add Mixture of Experts: Mixtral 8x 7B release

[casper-hansen]

Mistral AI released their new model called Mixtral which is an MoE architecture based on MegaBlocks. It includes 8 experts with the size being 7 billion parameters each.

Here is the model configuration:

• dim: 4096
• n_layers: 32
• head_dim: 128
• hidden_dim: 14336
• n_heads: 32
• n_kv_heads: 8
• norm_eps: 1e-05
• vocab_size: 32000
• moe:
  • num_experts_per_tok: 2
  • num_experts: 8

Weights: https://twitter.com/MistralAI/status/1733150512395038967
Paper: https://arxiv.org/pdf/2211.15841.pdf
Code: https://github.com/stanford-futuredata/megablocks

CC: @WoosukKwon @zhuohan123 for visibility.

[WoosukKwon]

It's coming soon! Please stay tuned.

[tjtanaa]

mistralai/megablocks-public@8897a04

Mistral AI has forked the megablocks. In that fork there is a branch called 'mixtral'

@casper-hansen @WoosukKwon

[tonysy]

A quick example of inference code: https://github.com/open-compass/mixtralkit
Evaluation results will be updated soon

[tjtanaa]

@WoosukKwon @casper-hansen

Our repo (branch mixtral8x7b) has an experimental support for Mixtral which is a naive implementation of MoE that is based on modeling_moe_mistral.py from DiscoResearch/mixtral-7b-8expert .

[casper-hansen]

@WoosukKwon @casper-hansen

Our repo (branch mixtral8x7b) has an experimental support for Mixtral which is a naive implementation of MoE that is based on modeling_moe_mistral.py from DiscoResearch/mixtral-7b-8expert .

This is a great first start. I think the key to implementing mixtral in vLLM will be the parallelization of the experts forward.

[tjtanaa]

@WoosukKwon @casper-hansen
Our repo (branch mixtral8x7b) has an experimental support for Mixtral which is a naive implementation of MoE that is based on modeling_moe_mistral.py from DiscoResearch/mixtral-7b-8expert .

This is a great first start. I think the key to implementing mixtral in vLLM will be the parallelization of the experts forward.

@casper-hansen @WoosukKwon Performance reference of the experimental implementation of Mixtral in vLLM.
Performance on 2 x Instinct MI210 Accelerators Throughput: 1.54 requests/s, 748.09 tokens/s
Performance on 2 x A100 Throughput: 3.15 requests/s, 1527.27 tokens/s

[nivibilla]

There is another implementation here
https://github.com/EmbeddedLLM/vllm-rocm/tree/mixtral8x7b

[0-hero]

@tjtanaa couldn't get the vllm-roc mixtral branch working. Had problems building the package. So ported your mixtral code to a latest fork of vllm here. Able to use this with DiscoResearch

[tjtanaa]

@tjtanaa couldn't get the vllm-roc mixtral branch working. Had problems building the package. So ported your mixtral code to a latest fork of vllm here. Able to use this with DiscoResearch

@0-hero We are able to run the branch.
May I know what issue are you encountering?

[WoosukKwon]

closed by #2011