Mixtral Support When?

[cubesstar]

Unsurprising the new Mixtral-8x7B and more specifically Mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf does not work. As experienced from other users they get the error create_tensor: tensor 'blk.0.ffn_gate.weight' not found. I understand that it just came out and will take some time for it to get up and working I'm just trying to put it on the radar as I haven't seen anyone talk about it here. If support for it gets added in the next update I'd be happy :D

[LostRuins]

Is this supported upstream in llama.cpp yet? If so, it'll be in the next release once I merge it

[cubesstar]

I don't know much about llama.cpp however from what I have seen no. Tho there is some experimental stuff going on.

[aleksusklim]

A fork claims it supports Mixtral: https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/discussions/8
(https://github.com/Nexesenex/kobold.cpp/releases/tag/1.52_mix)

I didn't have a chance to test it yet.

[Dirky14]

It's supported on the mixtral branch of llamacpp. Tested it with Mixtral Instruct Q4_M from TheBloke, it works fine.

[SrVill]

A fork claims it supports Mixtral: https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/discussions/8 (https://github.com/Nexesenex/kobold.cpp/releases/tag/1.52_mix)

I didn't have a chance to test it yet.

It is better to wait for the release from LostRuins. That fork is questionable
https://www.virustotal.com/gui/file/96f44726176da3a00bd3b07895f8b40dbc860d3589fe616ee97fd24836f0d50c

[rosemash]

@SrVill

It is better to wait for the release from LostRuins. That fork is questionable https://www.virustotal.com/gui/file/96f44726176da3a00bd3b07895f8b40dbc860d3589fe616ee97fd24836f0d50c

I personally wouldn't trust the compiled release on a random fork either, but a few heuristic positives on virustotal isn't a reliable indicator to know whether an executable is dangerous. This will probably show up for a lot of unknown executables from github. It wouldn't be too hard for anyone who wants to use this fork to review the code and compile it (it's only 2 commits ahead, 1 of which is a merge from the upstream mixtral branch of llama.cpp)

[aleksusklim]

(For information)

I've tested that fork KoboldCPP_Frankenstein_Experimental_1.52_Mixtral with mixtral-V8x7b-v0.1.Q5_K_M.gguf

It worked really good!

But then, after about 800 tokens of rollplay, it suddenly got completely off-rails, printing absolute nonsense like:

nevertheless which means therefore ultimately speaking thus meaning consequently thereby resulting finally henceforth accordingly wherefore eventually subsequently afterwards following suit aftermath etcetera ad infinitum et cetera blahblahblahwhateveretceteraadinfinitumandsoonerorlatereventuallyweallgetoldanddieanywayright?

Restarting does not help. Lowering temperature does not help either. Tried 32k and 8k contexts.

Also I got main thread is not in main loop error occasionally. And also, looks like BLAS Batching does not work with positive batch sizes. (For me it stuck on the first [BLAS] 128/X)

I don't know what's going on, but given the superior model quality on short stories – this must be a bug somewhere (maybe on my side if nobody else is seeing this).
I'll wait for official support of course.

[rosemash]

But then, after about 800 tokens of rollplay, it suddenly got completely off-rails, printing absolute nonsense like:

nevertheless which means therefore ultimately speaking thus meaning consequently thereby resulting finally henceforth accordingly wherefore eventually subsequently afterwards following suit aftermath etcetera ad infinitum et cetera blahblahblahwhateveretceteraadinfinitumandsoonerorlatereventuallyweallgetoldanddieanywayright?

Anecdotally this output looks to me like what happens when RoPE is misconfigured.

[Enferlain]

Thought rope gets auto set for gguf. Had similar output when I tried going above 4k context

[LostRuins]

The PR to track is here: ggml-org#4406

[aleksusklim]

RoPE is misconfigured.

Hmm: ggml-org#4406 (comment)

Mixtral should be 1000000

I've tried manually setting RoPE Base to 1000000.0 or to 10000.0 with context length of 32000, 32768, ~8300 – but nothing seemed to be resolving the issue.

[rosemash]

The PR to track is here: ggerganov#4406

It's merged

[LostRuins]

v1.52 is out, mixtral support is added, please try it.

Note: Mixtral currently does prompt processing very slowly. You may want to try with --noblas or --blasbatchsize -1

[Deathcow]

Note: Mixtral currently does prompt processing very slowly. You may want to try with --noblas or --blasbatchsize -1

Maybe I'm dumb, but disabling batch processing doesn't make it go any faster, they are both slow and if someone put a gun to my head, I'd say batches of 512 are still a little bit faster than no batch at all. To me it seems no batch just looks faster because it's updating more often in the cli.

But yeah, it's real painful for context sizes >4000

[aleksusklim]

I downloaded mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf and tried again on the same place where mixtral-8x7b-v0.1.Q5_K_M.gguf failed.

It worked great!!
No more of those ultimately resulting instead only positive reinforcement occurring throughout entirety duration journey undertaken henceforth forthwith ad infinitum forevermore amen etcetera et cetera blah blah blah yadda yadda yadda yawn…..zzzZZZzzzzzzzz………. (actual output!)

In both Frankenstein and in official koboldcpp-1.52, with the same exact settings.
Then I assume something is wrong in the "base" model file. I can't believe that it should behave like that!

Moreover, my story does not include special [INST] tags, so the base model ought to behave even better than instruct one. And it does until it breaks.

P.S. Blas batching is working in 1.52 like normal.