This repository was archived by the owner on Jun 24, 2024. It is now read-only.
This repository was archived by the owner on Jun 24, 2024. It is now read-only.
Use RMSNorm for normalization #80
Description
Upstream has switched to using RMSNorm for normalization, which is more accurate to the original implementation: ggml-org/llama.cpp#173
We didn't do this at first because there seemed to be some issues, but those seem to have been resolved.