Skip to content

Not unload a model from VRAM anymore. #434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
berkut1 opened this issue Jun 29, 2023 · 2 comments
Closed

Not unload a model from VRAM anymore. #434

berkut1 opened this issue Jun 29, 2023 · 2 comments

Comments

@berkut1
Copy link

berkut1 commented Jun 29, 2023

The last version 0.1.66, not unload model from VRAM.
I thought the problem was with oobabooga/text-generation-webui oobabooga/text-generation-webui#2920, but after digging into the code from projects, I think the problem is with this library.
I think there

def __del__(self):
should be added a new llama_free_model function.

Sorry if I'm wrong, I don't work with python.

@abetlen
Copy link
Owner

abetlen commented Jun 29, 2023

@berkut1 thank you for caching that, will have publish a new version shortly

@iactix
Copy link

iactix commented Jun 30, 2023

It still only "deletes" the model though, VRAM still ends up a lot higher than before loading the model.
#223

antoine-lizee pushed a commit to antoine-lizee/llama-cpp-python that referenced this issue Oct 30, 2023
* File load progress reporting

* Move llama_progress_handler into llama_context_params

* Renames

* Use seekg to find file size instead

* More correct load progress

* Call progress callback more frequently

* Fix typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants