How to chat now, after update since b3087?

[superchargez]

Removal of instruction and chatml params, see issue which got implemented in b3087, have made it difficult to use llamacpp for chatting, however, these changes are made in main and not in server server never supported chat format like this (where user write something and llama.cpp responds in loop). So, what to do now?

[dspasyuk]

I also want to know the proper way to run the "main" now. With -ins gone everything I try seems to give incoherent results or the model (Llama 3) just chats with itself. I used to run it like so:
../llama.cpp/main --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf --n-gpu-layers 35 -ins --keep -1 -cml --simple-io -b 2048 --ctx_size 2048 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 4

This seems to work sometimes but model starts randomly charting with itself or just generates garbage:

../llama.cpp/main --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf --n-gpu-layers 35 -i -cnv --interactive-first --keep -1 --simple-io -b 2048 --ctx_size 2048 --temp 0.3 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 6

[dspasyuk]

@ggerganov could we get your help here Georgi?

[dspasyuk]

@superchargez I think this works now:
../llama.cpp/main --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf --n-gpu-layers 30 -cnv --interactive-first --simple-io -b 2048 --ctx_size 2048 --temp 0.3 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 6 -r "/n>" --log-disable -p "You are Alice, a large language model. Your purpose is to assist users by providing information, answering questions, and engaging in meaningful conversations based on the data you were trained on."

See more here https://github.com/dspasyuk/llama.cui

[dspasyuk]

My answer above only works for about 4-5 conversations then the LLama 3 model starts to output garbage. I gave up and reverted back to the old version.

[superchargez]

I'm keeping updated version only for server, main is not working right now.

[Galunid]

For ChatML you can do something along the lines of --in-prefix "<|im_start|>user\n" --in-suffix "<|im_end|>\n<|im_start|>assistant". This works for pretty much every chat template.

I'm not sure why -cml/-ins were removed.

[dspasyuk]

Input syntax is not a problem, the output generated by llama 3 now is incoherent after a few messages. It just starts printing line breaks or vertical lines or just garbage. Downgrade is the only solution so far. Setting prefix/suffix does not help.

[Galunid]

That's a separate issue then. Please update to the latest version and create a new bug ticket.