llama : add support for llama2.c models

[ggerganov]

The new llama2.c project provides means for training "baby" llama models stored in a custom binary format, with 15M and 44M models already available and more potentially coming out soon.

We should provide a simple conversion tool from llama2.c bin format to ggml format so we can run inference of the models in llama.cpp

Great task for people looking to get involved in the project

[jagtesh]

I can take a stab at it. Been meaning to dive deeper into the GGML format.

Since convert.py only does GGML conversion, and quantize is called explicitly for quantization. In theory - only convert.py will need to be modified.

Would an existing model (HF/PyTorch) serve as a good starting point?

[slaren]

Trying to add this to convert.py may be overkill, and a lot harder than it needs to be. Writing a standalone script would probably be a lot easier.

The easiest to understand description of the file format is probably in the training example here: https://github.com/ggerganov/llama.cpp/blob/41c674161fb2459bdf7806d1eebead15bc5d046e/examples/train-text-from-scratch/train-text-from-scratch.cpp#L2609

[Mistobaan]

@ggerganov why not use the safetensor format? seems way more practical than custom binary ggml formats

[klosax]

@Mistobaan See this note in the spec of the upcoming gguf fileformat gguf.md#why-not-other-formats and PR ggml-org/ggml#302

[byte-6174]

began a super WIP (not completely functional) attempt at this here
will update as I go along piecing together corresponding variables between the two.

[jagtesh]

@byte-6174 nice - I took a similar approach. Currently finding myself going deep in the rabbit hole of converting llama2.c tensors that use calloc based memory assignment (with no metadata afaik) to ggml_tensor.

Also I don’t quite understand yet why does the vocab need to be saved in the model, when it is also available in an external file? In any case, I believe llama2.c is using the exact same format for vocab.

I’ll end this update with a few words in the voice of Yoda, “Long journey, it is. Learn, I must.”

[byte-6174]

Here.
also, find a mapping that was required to figure out how to match the variables 🙂

This reads the llama2.c model file and saves all weights to ggml compatible tensor format.
Let me know how it works...

[byte-6174]

re. mapping can someone with more exp with llama.cpp tensors point to how these RoPE tensors should be mapped? Indicated with ? in mapping md file above ☝️

[slaren]

Not 100%, but I believe these are lookup tables for the RoPE, and are not necessary for llama.cpp.

[byte-6174]

Right, I looked at llama2.c code and its surely for RoPE, good to know it's not needed for llama.cpp. I can remove it.
I'm now wanting to run this model now, any pointers? digging now..

[ggerganov]

You can run the most basic inference using: ./main -m converted-model.bin

[byte-6174]

Got it, it runs pretty well!.
Perhaps now I can quatize it...
Output seem to contain non-english words...humm..

gives 359 tok/sec, vs. llama2.cs ~100 tok/sec, (perhaps not a fair comparison though).

main: build = 909 (5a87675)
main: seed  = 1690733963
llama.cpp: loading model from abcd1.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 288
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 6
llama_model_load_internal: n_head_kv  = 6
llama_model_load_internal: n_layer    = 6
llama_model_load_internal: n_rot      = 48
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 1.0e-06
llama_model_load_internal: n_ff       = 768
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 0 (all F32)
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.02 MB
llama_model_load_internal: mem required  =  395.13 MB (+    3.38 MB per state)
llama_new_context_with_model: kv self size  =    3.38 MB

system_info: n_threads = 8 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 200, n_keep = 0


 One day, Lily met a Shoggoth anyonefather whilem enjo but gl than the acc sn ahead nap very off ru sc things People table fasterer beet wants hiding anywheresklär customer al Contact round weeks sad sick someone somethingore tagBeil as attached where mine recommend ourselvesanalipirst leako each each that reck deal laterchy pair Then dar all blocking Whenrai accomplished yourself backrain this Hamb dr slehetableView short b looked alsoch goal d who down down downnight check out bird calows helper home walliling patch agree je either of whole cl heavily visited visit p up up up up up off headD vCODEBack Withoutotoss timeys pat On new warm compared corner kitcheniously front раз happenionungsplaceĕiveness belong slower run running toten replace a away, MangWe these mention topakesFound herself down if courageished facts rocksepar hear oneessU as b A meanshed turn I Or laugh save without out was destroyunt doA bur byണ University shop' the chance alone alone alone alone someone particular
llama_print_timings:        load time =   115.33 ms
llama_print_timings:      sample time =   143.12 ms /   200 runs   (    0.72 ms per token,  1397.48 tokens per second)
llama_print_timings: prompt eval time =    10.21 ms /    12 tokens (    0.85 ms per token,  1175.20 tokens per second)
llama_print_timings:        eval time =   409.92 ms /   199 runs   (    2.06 ms per token,   485.46 tokens per second)
llama_print_timings:       total time =   581.63 ms