Skip to content

musa: workaround for Guilty Lockup in cleaning src0 in #10032 #10042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 28, 2024

Conversation

yeahdongcn
Copy link
Collaborator

We’re encountering an MTGPU Guilty Lockup issue during the model warm-up stage after merging #10032. This PR reverts this change for MUSA only.

I've raised an internal issue and will remove this workaround once it has been resolved.

@yeahdongcn yeahdongcn marked this pull request as ready for review October 25, 2024 08:16
@yeahdongcn
Copy link
Collaborator Author

Hi @JohannesGaessler

Could you please review this PR? I know the code looks ugly, but it works for now.

Copy link
Collaborator

@JohannesGaessler JohannesGaessler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you're aware that this results in broken K cache quantization.

@yeahdongcn
Copy link
Collaborator Author

I assume you're aware that this results in broken K cache quantization.

Yes. After reviewing all contexts, this approach appears to be the only viable solution to avoid a crash.

Thanks for approving this!

@JohannesGaessler JohannesGaessler merged commit 524afee into ggml-org:master Oct 28, 2024
53 checks passed
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 15, 2024
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 18, 2024
yeahdongcn added a commit to makllama/llama.cpp that referenced this pull request Feb 12, 2025
yeahdongcn added a commit to makllama/llama.cpp that referenced this pull request Feb 13, 2025
yeahdongcn added a commit to makllama/llama.cpp that referenced this pull request Feb 13, 2025
ngxson pushed a commit that referenced this pull request Feb 13, 2025
* musa: Update MUSA SDK version to rc3.1.1

Signed-off-by: Xiaodong Ye <[email protected]>

* musa: Remove workaround in PR #10042

Signed-off-by: Xiaodong Ye <[email protected]>

---------

Signed-off-by: Xiaodong Ye <[email protected]>
orca-zhang pushed a commit to orca-zhang/llama.cpp that referenced this pull request Feb 26, 2025
* musa: Update MUSA SDK version to rc3.1.1

Signed-off-by: Xiaodong Ye <[email protected]>

* musa: Remove workaround in PR ggml-org#10042

Signed-off-by: Xiaodong Ye <[email protected]>

---------

Signed-off-by: Xiaodong Ye <[email protected]>
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Feb 26, 2025
* musa: Update MUSA SDK version to rc3.1.1

Signed-off-by: Xiaodong Ye <[email protected]>

* musa: Remove workaround in PR ggml-org#10042

Signed-off-by: Xiaodong Ye <[email protected]>

---------

Signed-off-by: Xiaodong Ye <[email protected]>
mglambda pushed a commit to mglambda/llama.cpp that referenced this pull request Mar 8, 2025
* musa: Update MUSA SDK version to rc3.1.1

Signed-off-by: Xiaodong Ye <[email protected]>

* musa: Remove workaround in PR ggml-org#10042

Signed-off-by: Xiaodong Ye <[email protected]>

---------

Signed-off-by: Xiaodong Ye <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants