android's gki_defconfig failing to boot

[nickdesaulniers]

https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/4944574335

https://github.com/clangbuiltlinux/continuous-integration2/actions/workflows/android14-5.15-clang-android.yml

android14-5.15 has been failing to boot with LLVM_VERSION=android

[nickdesaulniers]

https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/4977177272/jobs/8907239682#step:5:342

[nickdesaulniers]

I can reproduce this with mainline llvm and android's gki_defconfig on 5.15.y (and 6.1.y is failing in CI with the same issue). QEMU hangs after:

[    0.636462][    T1] kvm [1]: IPA Size Limit: 48 bits

can't reproduce on android-mainline but can on mainline, but lto is also disabled there. lto is also disabled on 6.1, so maybe not LTO specific.

[nickdesaulniers]

did stable regress?

v6.1.25 bad
v6.1.24 bad
v6.1.20 bad...

and I can repro on mainline. That said, on mainline it's a panic+splat, not hang. Maybe this HVC_DCC stuff is a red herring.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1303657 mentions

Kernel hangs at boot waiting for a console via the JTAG interface that will
obviously never appear - disable it.

---

Maybe android enabled CONFIG_HVC_DRIVER or CONFIG_HVC_DCC around may 9-10? https://android-review.googlesource.com/c/kernel/common/+/1451420 makes it look like no.

Ah, mainline is failing with a splat+panic for that config; different issue.

---

maybe something regressed on the llvm side between may 9 and may 10? but clang-16 also fails...maybe a backport?

9c07aa75b961 bad
f09b0e35b6d8f19385f37b213b029a7ca9cb6368 bad

"clang": "Debian clang version 17.0.0 (++20230507093322+728b8a139804-1~exp1~20230507093335.652)", good
"clang": "Debian clang version 17.0.0 (++20230507093322+728b8a139804-1~exp1~20230507093335.652)", bad
clang revision: 728b8a139804 good
clang revision: 728b8a139804 bad

so clang version is not the variable.

---

The qemu version is 7.2.0 in both good and bad builds.

---

"git_sha": "dac17bb8f7c135f7915d9820a52272626bc3f48a", good
"git_sha": "de71877f29cc1868c4fe57845d1006f534d42ff1", bad

Confirmed that's a bad range; bisecting now.

[nickdesaulniers]

444b34b83a072c5d9d441b2cf52e88c70a8f6a66 is the first bad commit
commit 444b34b83a072c5d9d441b2cf52e88c70a8f6a66
Author: Quentin Perret <qperret@google.com>
Date:   Fri Feb 3 14:18:57 2023 +0000

    ANDROID: KVM: arm64: Allow tweaking HFGWTR_EL2 from modules
    
    pKVM modules may need to enable fine grain traps. To allow this
    use-case, expose a function allowing to manipulate HFGWTR_EL2 on the
    current CPU. This configuration will be persistent across power
    management cycles and such as it is propagated all the way to the
    init params per-cpu struct.
    
    Bug: 245034629
    Change-Id: If9c29849df50847d3a796345927bd51c37c358bc
    Signed-off-by: Quentin Perret <qperret@google.com>

 arch/arm64/include/asm/kvm_asm.h         |  1 +
 arch/arm64/include/asm/kvm_pkvm_module.h |  1 +
 arch/arm64/kernel/asm-offsets.c          |  1 +
 arch/arm64/kvm/hyp/nvhe/hyp-init.S       |  3 +++
 arch/arm64/kvm/hyp/nvhe/modules.c        | 11 +++++++++++
 5 files changed, 17 insertions(+)

https://android-review.googlesource.com/q/If9c29849df50847d3a796345927bd51c37c358bc
cc @qperret

[qperret]

Thanks for the report -- also just got one from the Android side. There is a temporary workaround ( https://android-review.googlesource.com/c/kernel/common/+/2592054) that has been pushed while I work on a proper fix.

…

On Tue, May 16, 2023 at 11:28 PM Nick Desaulniers ***@***.***> wrote: 444b34b is the first bad commit commit 444b34b Author: Quentin Perret ***@***.***> Date: Fri Feb 3 14:18:57 2023 +0000 ANDROID: KVM: arm64: Allow tweaking HFGWTR_EL2 from modules pKVM modules may need to enable fine grain traps. To allow this use-case, expose a function allowing to manipulate HFGWTR_EL2 on the current CPU. This configuration will be persistent across power management cycles and such as it is propagated all the way to the init params per-cpu struct. Bug: 245034629 Change-Id: If9c29849df50847d3a796345927bd51c37c358bc Signed-off-by: Quentin Perret ***@***.***> arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/include/asm/kvm_pkvm_module.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kvm/hyp/nvhe/hyp-init.S | 3 +++ arch/arm64/kvm/hyp/nvhe/modules.c | 11 +++++++++++ 5 files changed, 17 insertions(+) https://android-review.googlesource.com/q/If9c29849df50847d3a796345927bd51c37c358bc cc @qperret <https://github.com/qperret> — Reply to this email directly, view it on GitHub <#1848 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVAXUUBKPS227W6NSWLXI3XGPWOVANCNFSM6AAAAAAYD77OEU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[qperret]

Update: the workaround has been merged, and the "proper" fix is up for review at https://android-review.googlesource.com/c/kernel/common/+/2594087 On Wed, May 17, 2023 at 8:52 AM Quentin Perret ***@***.***> wrote:

…

Thanks for the report -- also just got one from the Android side. There is a temporary workaround ( https://android-review.googlesource.com/c/kernel/common/+/2592054) that has been pushed while I work on a proper fix. On Tue, May 16, 2023 at 11:28 PM Nick Desaulniers ***@***.***> wrote: > 444b34b is the first bad commit > commit 444b34b > Author: Quentin Perret ***@***.***> > Date: Fri Feb 3 14:18:57 2023 +0000 > > ANDROID: KVM: arm64: Allow tweaking HFGWTR_EL2 from modules > > pKVM modules may need to enable fine grain traps. To allow this > use-case, expose a function allowing to manipulate HFGWTR_EL2 on the > current CPU. This configuration will be persistent across power > management cycles and such as it is propagated all the way to the > init params per-cpu struct. > > Bug: 245034629 > Change-Id: If9c29849df50847d3a796345927bd51c37c358bc > Signed-off-by: Quentin Perret ***@***.***> > > arch/arm64/include/asm/kvm_asm.h | 1 + > arch/arm64/include/asm/kvm_pkvm_module.h | 1 + > arch/arm64/kernel/asm-offsets.c | 1 + > arch/arm64/kvm/hyp/nvhe/hyp-init.S | 3 +++ > arch/arm64/kvm/hyp/nvhe/modules.c | 11 +++++++++++ > 5 files changed, 17 insertions(+) > > > https://android-review.googlesource.com/q/If9c29849df50847d3a796345927bd51c37c358bc > cc @qperret <https://github.com/qperret> > > — > Reply to this email directly, view it on GitHub > < #1848 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AEVAXUUBKPS227W6NSWLXI3XGPWOVANCNFSM6AAAAAAYD77OEU > > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#1848 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVAXUQQOAA6OKSHCUTH2QDXGRYS7ANCNFSM6AAAAAAYD77OEU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>