inference on iPhone

[tstanek390]

Hello,
thanks for sharing these very promising LLMs. I'd like to run it on iOS, but as I'm a noob at coding and stuff, I don't really know how to start. I'm an healtcare professional and would love to give it a try. Any hints/tips ?
Thanks a lot and all the best to the new year!
Teodor

[huyiming2018]

Maybe you can refer to this example from lamma.cpp community: ggml-org/llama.cpp#4508

[tstanek390]

Thx, but I guess I would need GGUF version of the model to run it with LLama.cpp..?

[YangYang-DLUT]

You can convert the model to GGUF follow the instruction of https://github.com/ggerganov/llama.cpp/tree/master/examples/llava. The detailed deployment instruction is in process and will be released soon.

[YangYang-DLUT]

We have updated README.md with the deployment instruction on Android devices with customized llama.cpp. Have a try!

The inference plan on iPhone is on schedule but not proposed yet. Any ideas?

[jasonw247]

Thanks for adding this! Are you planning on submitting a PR to llama.cpp, or keeping the forked repo?

[YangYang-DLUT]

We have submitted a PR to llama.cpp. We want to get the support of open-source communities, since that there are a lot space for optimization of the deployment on mobile devices.

[er-muyue]

Hi, we are closing this issue due to the inactivity. Hope your question has been resolved. If you have any further concerns, please feel free to re-open it or open a new issue. Thanks!