Roadmap (short-term)

[ggerganov]

These will be the priorities for the next few days:

• Reduce inference memory usage via ggml scratch buffers, no hardcoded memory buffer sizes and support infinite interactive mode

  I know how to fix this and this is important since the GH issues are being flooded with complaints about seg faults and crashes

• Finalize SIMD accelerated quantization and merge ggml back in the parent repo:

  • AVX
    • quantize_row_q4_0()
    • quantize_row_q4_1()
    • dequantize_row_q4_0()
    • dequantize_row_q4_1()
  • ARM_NEON
    • quantize_row_q4_0()
    • quantize_row_q4_1()
    • dequantize_row_q4_0()
    • dequantize_row_q4_1()

  I suspect this could improve performance for prompt batch processing

• Deprecate ggml_vec_mad_xxx() routines and simplify ggml_forward_mul_mat_xxx()

  This should lead to some significant code reduction in ggml.c

• Separate the perplexity computation from main.cpp into standalone example program called perplexity

• Move main.cpp into a standalone example program and move utils.h/utils.cpp into ./examples to be shared by all examples

• Add llama_state to allow parallel text generation sessions with a single model

  I will do this in a similar way it is done in whisper.cpp

• Extend llama_state to support loading individual model tensors. Needed for LoRA personalities support

• Add 2-bit integer quantization

---

When the above things are ready we will have a good foundation to start porting more models and create more example applications to demonstrate the usage of ggml.

New roadmap: #784

[niansa]

Are these sorted top-priority first?

[Green-Sky]

more or less i think

[jasontitus]

Can we update the README with usage of the new perplexity tool since the 'main --perplexity' way stopped working?

[Green-Sky]

it's now it's own program
(note: make it link to examples/perplexity/README.md and fill in the doc there)

[wildcat7534]

What's append, is the projet become re activate later ? u_u