Welcome to LLaMA-Factory Discussions!

[hiyouga]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.

👋 欢迎！

讨论区是一个与广泛社区成员建立联系的平台。在这里您可以：

• 提出问题
• 分享想法
• 与其他人互动

[lh-kyf]

[hiyouga]

请单独创建新主题讨论，不要在这里回复

[SS12dev]

Hello, I was trying to fine-tune QWEN2.5-0.5b-Instruct model on my custom dataset ( it has around 700 entries with simple prompts (instruction), huge queries (input) and respective response (outputs)). When i keep the token limit to 32768 (QWEN2.5-0.5b model's token limit) the models is not able to load when i start fine-tuning. It gives the following error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.49 GiB. GPU 0 has a total capacity of 15.57 GiB of which 1.05 GiB is free. Process 328169 has 102.00 MiB memory in use. Including non-PyTorch memory, this process has 14.42 GiB memory in use. Of the allocated memory 14.08 GiB is allocated by PyTorch, and 221.22 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I have also tried to enable expandable_segments, that did not help. Can you suggest something?

[SS12dev]

Can you tell if the tokenized tensors are being stored in VRAM completely during the fine tuning process? I thought they are stored in system storage (RAM or SSD) and retrieved as required. I am thinking this might be the issue with the memory getting full. Can someone look into this?