Skip to content

Reduce memory usage and allocate enough memory for largest context #473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 24, 2023
Merged

Reduce memory usage and allocate enough memory for largest context #473

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9330ff0
Reduce memory usage and allocate enough memory for large contexts
ggerganov Mar 24, 2023
ea60d21
Simpler scratch buffer usage
ggerganov Mar 24, 2023
3634c31
Reenable BLAS for quantized mul_mat
ggerganov Mar 24, 2023
0b4e849
Fix number of layers in 30B and 65B
ggerganov Mar 24, 2023
d0f7519
Fix KV cache size for F32
ggerganov Mar 24, 2023
6feb572
Merge branch 'master' into mem-fix
ggerganov Mar 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions ggml.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5846,7 +5846,8 @@ static bool ggml_compute_forward_mul_mat_use_blas(
const struct ggml_tensor * src0,
const struct ggml_tensor * src1,
struct ggml_tensor * dst) {
UNUSED(src0);
const int ne00 = src0->ne[0];
const int ne01 = src0->ne[1];

const int ne10 = src1->ne[0];

Expand All @@ -5856,7 +5857,14 @@ static bool ggml_compute_forward_mul_mat_use_blas(
// TODO: find the optimal values for these
if (ggml_is_contiguous(src0) &&
ggml_is_contiguous(src1) && ((ne0 >= 32 && ne1 >= 32 && ne10 >= 32))) {
//printf("BLAS: %d %d %d\n", ne0, ne1, ne10);

//// disable BLAS for Q4_0 and Q4_1
//// looks like there is no benefit and we only waste a lot of memory
//if (src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_1) {
// return false;
//}

//printf("BLAS: %d %d %d %d %d\n", ne0, ne1, ne10, ne00, ne01);
return true;
}

Expand Down
Loading