Skip to content

extract spin lock to functions to be able to estimate the effect #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bogdad
Copy link
Owner

@bogdad bogdad commented Apr 3, 2023

its hard to estimate the time spent in spinlocking, this is a try to make it easier to see in sampling profiler.

instruments run with the large prompt - dan.
./build/bin/main -m ./models/65B/ggml-model-q4_0.bin --color -f ./prompts/dan.txt -n 64 -t 8
Screenshot 2023-04-03 at 20 36 38

or 7b model
./build/bin/main -m ./models/7B/ggml-model-q4_0.bin --color -f ./prompts/dan.txt -n 64 -t 8
Screenshot 2023-04-03 at 20 48 25

compared to the thread pool
Screenshot 2023-04-03 at 21 00 50

i think that shows that spinning amounts to 20% on the large prompts, large model, or 40% small model large prompts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant