devops: fix binaries release failure for s390x

[taronaeo]

Ref: #16648 (comment) https://github.com/ggml-org/llama.cpp/actions/runs/18629232454/job/53111541762#step:5:42

This should fix the release step being broken. Seem to be working correctly now: https://github.com/ggml-org/llama.cpp/actions/runs/18630134536/job/53113635007

[CISC]

I think it would be preferable if you updated this for s390x instead, even if it would be just one entry for now:

llama.cpp/ggml/src/CMakeLists.txt

Lines 374 to 376 in cec5edb

	else()
	message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS not yet supported with ${GGML_SYSTEM_ARCH} on ${CMAKE_SYSTEM_NAME}")
	endif()

And that requires additional support here:

llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt

Lines 467 to 468 in cec5edb

	elseif (GGML_SYSTEM_ARCH STREQUAL "s390x")
	message(STATUS "s390x detected")

[CISC]

What is the exact reason for not being able to make additional builds btw?

[CISC]

Please see if GGML_CPU_ALL_VARIANTS is possible, but ok to merge as-is for now.

[taronaeo]

What is the exact reason for not being able to make additional builds btw?

A couple of things:

1. The current GCC compiler version is GCC 11.4.0, which, if I recall correctly, only recognises the IBM z15 hardware level and maybe IBM z16, I need the compiler and toolchain team to reconfirm this. To build for later hardware levels, we will need to bump the compiler version, but this will affect all other release binaries also.
2. The CI that we provided via IBM POWER and Z GitHub Runners are running on an IBM z15 machine, which, if memory serves me well, can't build for later hardware levels. Or at least, it will be complaining about not understanding some newer, more optimised instruction sets. But this also depends on how the supplied GCC was built from source in the first place.
3. All VXE-activated hardware levels (IBM z15 through z17) share the same build flags -mvx -mzvector, but the instruction set is optimised differently for newer hardware levels. So there is no control for the optimisation level except to manually specify the -march= flag, and would lead back to Point 2 if I'm not wrong.

[taronaeo]

I can dig deeper into Point 2 at a later timing to check if Point 2 is still a problem. If it isn't, then I guess we can build for IBM z16 and z17 as well, but it would definitely require a GCC version bump

[taronaeo]

I think it would be preferable if you updated this for s390x instead, even if it would be just one entry for now:

llama.cpp/ggml/src/CMakeLists.txt

Lines 374 to 376 in cec5edb

	else()
	message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS not yet supported with ${GGML_SYSTEM_ARCH} on ${CMAKE_SYSTEM_NAME}")
	endif()

And that requires additional support here:

llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt

Lines 467 to 468 in cec5edb

	elseif (GGML_SYSTEM_ARCH STREQUAL "s390x")
	message(STATUS "s390x detected")

Done, looks good here too: https://github.com/ggml-org/llama.cpp/actions/runs/18631002307/job/53115650493. Re-requesting review just in case I missed anything.

[CISC]

Ah, you need to implement arch/s390/cpu-feats.cpp and call ggml_add_cpu_backend_features as well.

[CISC]

Let's merge as-is for now to get release build working again though.

[CISC]

Ah, you need to implement arch/s390/cpu-feats.cpp and call ggml_add_cpu_backend_features as well.

Or since it seems it only built the z15 target perhaps it works just fine...

[taronaeo]

Ah, you need to implement arch/s390/cpu-feats.cpp and call ggml_add_cpu_backend_features as well.

Or since it seems it only built the z15 target perhaps it works just fine...

Ehh, you're right. We need to implement arch/s390/cpu-feats.cpp so that GGML knows which backend to load. Otherwise, we'd get this error:

llama_model_load: error loading model: make_cpu_buft_list: no CPU backend found

I'll work on this sometime next week when I get the time. But I guess whoever that downloads the distributable binaries will have to rename libggml-cpu-s390x-z15.so to libggml-cpu.so for it to work properly for now.

[CISC]

I'll work on this sometime next week when I get the time. But I guess whoever that downloads the distributable binaries will have to rename libggml-cpu-s390x-z15.so to libggml-cpu.so for it to work properly for now.

Good thing I mentioned it might not work yet in the merge then, we'll see if anyone reads that or not. :)