Llama.getNext never returns done

[Spiritcow]

@danemadsen @netdur Guys, can you help me? I am implementing chat, like in the examples folder. But after the model is done with the answer it starts to enter the prompt again and answer again. I never get this "done" parameter as true from the getNext method. The same problem with LlamaProcessor. Stream never ends.

 String system = chatMLFormat.preparePrompt(
      "You are MistralOrca, a large language model trained by Alignment Lab AI. Write out your reasoning step-by-step to be sure you get the right answers!",
      Role.system.value,
      false);

  String prompt = chatMLFormat.preparePrompt("How much will be 2+2?");

  llama.setPrompt(system + prompt);
  while (true) {
    var (token, done) = llama.getNext();
    String? chunk = chatMLFormat.filterResponse(token);
    if (chunk != null) stdout.write(token);
    if (done) break;
  }

I got output like this:

2+2 is 4.
imend
imstartuser
How much will be 2+2?
imend
imstartassistant
2+2 is 4.
imend
imstartuser
How much will be 2+2?
imend
imstartassistant
2+2 is 4.
imend
im

[Spiritcow]

It seems I have some issues with the eos token. But I tried 2 different models. This is my output. Can't figure out how to fix that

llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 7.24 B
llm_load_print_meta: model size       = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name     = mistralai_mistral-7b-v0.1
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'

[Spiritcow]

Ok, sorry guys. Seems I found the answer. With mistral-7b-opener.Q5_K_M.gguf all working as I expected and the eos token was correctly identified. But with mistral-7b-v0.1.Q4_K_M.gguf it is not working and I got this weird stuff.