Finetuning doesn't seem to use GPU

[tom-adsfund]

I was using a T4 on Colab, I compiled with CuBLAS, and tried with a 7B model and a 1.3B model.

In both situations, it appeared that perhaps only the LoRA data was actually put on the GPU. (The memory used was much less than 1.3B and 7B.) And whatever happened, there was no speed up.

[Tachyon5]

If you try offloading layers to gpu via --gpu-layers it will fail with the following:

GGML_ASSERT: ggml-cuda.cu:8148: false

./llama.cpp-master/finetune --model-base mistral-7b-openorca.Q4_0.gguf --lora-out lora-open-llama-3b-v2-q8_0-ITERATION.bin --train-data "mistral_metadata_training_data_orca2_equallysampled_col_description.txt" --save-every 41 --threads 12 --adam-iter 369 --batch 4 --ctx 640 --lora-r 128 --lora-alpha 128 --no-checkpointing --n-gpu-layers 33 main: seed: 1708130945 main: model base = 'mistral-7b-openorca.Q4_0.gguf' ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA A100-SXM4-40GB, compute capability 8.0, VMM: yes llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from mistral-7b-openorca.Q4_0.gguf (version GGUF V2) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = open-orca_mistral-7b-openorca llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 11: general.file_type u32 = 2 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32002] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32002] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32002] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 32000 llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 19: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_0: 225 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 261/32002 ). llm_load_print_meta: format = GGUF V2 llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32002 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 3.83 GiB (4.54 BPW) llm_load_print_meta: general.name = open-orca_mistral-7b-openorca llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 32000 '<dummy32000>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.22 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU buffer size = 70.32 MiB llm_load_tensors: CUDA0 buffer size = 3847.56 MiB .................................................................................................. llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA0 KV buffer size = 64.00 MiB llama_new_context_with_model: KV self size = 64.00 MiB, K (f16): 32.00 MiB, V (f16): 32.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 10.01 MiB llama_new_context_with_model: CUDA0 compute buffer size = 73.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 8.00 MiB llama_new_context_with_model: graph splits (measure): 3 main: init model print_params: n_vocab : 32002 print_params: n_ctx : 640 print_params: n_embd : 4096 print_params: n_ff : 14336 print_params: n_head : 32 print_params: n_head_kv : 8 print_params: n_layer : 32 print_params: norm_rms_eps : 0.000010 print_params: rope_freq_base : 10000.000000 print_params: rope_freq_scale : 1.000000 print_lora_params: n_rank_attention_norm : 1 print_lora_params: n_rank_wq : 128 print_lora_params: n_rank_wk : 128 print_lora_params: n_rank_wv : 128 print_lora_params: n_rank_wo : 128 print_lora_params: n_rank_ffn_norm : 1 print_lora_params: n_rank_ffn_gate : 128 print_lora_params: n_rank_ffn_down : 128 print_lora_params: n_rank_ffn_up : 128 print_lora_params: n_rank_tok_embeddings : 128 print_lora_params: n_rank_norm : 1 print_lora_params: n_rank_output : 128 main: total train_iterations 0 main: seen train_samples 0 main: seen train_tokens 0 main: completed train_epochs 0 main: lora_size = 2760882976 bytes (2633.0 MB) main: opt_size = 4140621808 bytes (3948.8 MB) main: opt iter 0 main: input_size = 327710752 bytes (312.5 MB) main: compute_size = 62204783168 bytes (59323.1 MB) main: evaluation order = RIGHT_TO_LEFT main: tokenize training data from mistral_metadata_training_data_orca2_equallysampled_col_description.txt main: sample-start: main: include-sample-start: false tokenize_file: total number of samples: 316488 main: number of training tokens: 317128 main: number of unique tokens: 3442 main: train data seems to have changed. restarting shuffled epoch. main: begin training main: work_size = 1536880 bytes (1.5 MB) train_opt_callback: iter= 0 sample=1/316488 sched=0.000000 loss=0.000000 |-> ggml_cuda_op_bin_bcast: unsupported types: dst: f32, src0: q4_0, src1: f32 GGML_ASSERT: ggml-cuda.cu:8148: false Aborted

`~$ nvidia-smi
Sat Feb 17 00:51:18 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... Off | 00000000:00:04.0 Off | 0 |
| N/A 31C P0 49W / 400W | 0MiB / 40960MiB | 27% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
`

Seems ggml-cuda has some type checks that break. I'll. look more into it.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.