Linux/ARM: Bus error with -O3 flag while running the CoreCLR unit-test

[leemgs]

Recently, We enabled CoreCLR with -O3 flag in release build. We could successfully run simple C# console applications after applying for PR dotnet/coreclr#5598.

• Run ./MS Official/ Hello World/HelloWorld.exe
  

However, We have got the "Bus error" from 3,000+ unit-tests when we have tried to run the CoreCLR UnitTest for stabilization as a next step.

• Bus error message from coreclr's unit-test

arm@ubuntu14.04#> ./my-unit-test.sh 
               . . .  Upper Omission . . . 
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_NEW
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_STATIC_BASE
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_STATIC_BASE
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_STATIC_BASE
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_NEWARR_1
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_STATIC_BASE
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_NEW
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_NEW
               ReadyToRun: Method entrypoint cannot be encoded
               ReadyToRun: Method entrypoint cannot be encoded
               ReadyToRun: Method entrypoint cannot be encoded
               ReadyToRun: Method entrypoint cannot be encoded
               ReadyToRun: JIT helper not supported: CORINFO_HELP_READYTORUN_NEW
               Native image mainv2.ni.exe generated successfully.
               ./mainv2.sh: line 68: 11353 Bus error               
               $CORE_ROOT/crossgen -readytorun -platform_assemblies_paths $CORE_ROOT:`pwd` -out test.ni.dll test.dll
               /unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay/corerun mainv2.exe
               Unhandled Exception: ./mainv2.sh: line 75: 11356 Bus error               
               $_DebuggerFullPath "$CORE_ROOT/corerun" mainv2.exe $CLRTestExecutionArguments
               Expected: 100
               Actual: 135
               END EXECUTION - FAILED
               FAILED   - reflection/regression/dev10bugs/Dev10_629953/Dev10_629953.sh
               BEGIN EXECUTION
               /unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay/corerun Dev10_629953.exe               
               Negative case - Enum type is Int32, value type is byte
               ./Dev10_629953.sh: line 66: 11342 Bus error               
               $_DebuggerFullPath "$CORE_ROOT/corerun" Dev10_629953.exe $CLRTestExecutionArguments
               Expected: 100
               Actual: 135
               END EXECUTION - FAILED
         - reflection/regression/dev10bugs/Dev10_630880/Dev10_630880.sh

              . . .  MIDDLE OMISSION . . . .

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : /unit-test/bin.coreclr.release.20160613.O3/Product/Linux.arm.Release
# Tests Discovered : 9873
# Passed           : 5624
# Failed           : 3888
# Skipped          : 361
=======================
210 minutes and 43 seconds taken to run CoreCLR tests.

real    212m2.303s
user    259m55.705s
sys 25m7.060s

• Checking the number of the "Bus error" related issues

dxl@target:/unit-test/Windows_NT.x64.Release.20160519$ cat ./coreclrtests-release-20160613-o3.xml  | grep "Bus error"  | wc -l
3805

• Kernel messages
  . . . . . Upper Omission . . . .
  [14041.099385] Alignment trap: not handling instruction ed940a00 at []
  [14041.099455] Unhandled fault: alignment exception (0x011) at 0xb570405d
  [14041.101480] pgd = ebbec000
  [14041.103260] [b570405d] *pgd=bbc77835
  [14041.105123] Alignment trap: not handling instruction ed940a00 at []
  [14041.108652] Unhandled fault: alignment exception (0x011) at 0xb570406f
  [14041.110451] pgd = ebbec000
  [14041.113633] [b570406f] *pgd=bbc77835
  [14042.734539] Alignment trap: not handling instruction ed940a00 at []
  [14042.734611] Unhandled fault: alignment exception (0x011) at 0xb4f0805d
  [14042.746142] pgd = c642c000
  [14042.747780] [b4f0805d] *pgd=6bb5e835, *pte=946127df, *ppte=94612e7f
  [14042.783738] Alignment trap: not handling instruction ed940b00 at []
  [14042.785409] Unhandled fault: alignment exception (0x011) at 0xb2507193
  [14042.787300] pgd = ebbec000
  [14042.788940] [b2507193] *pgd=ba4ad835
  [14042.790739] Alignment trap: not handling instruction ed940b00 at []
  [14042.792643] Unhandled fault: alignment exception (0x011) at 0xb25071a1
  [14042.794286] pgd = ebbec000
  . . . . Below Omission . . . .
• System call tracing:

              . .  . Upper Omission . . . .
mmap2(NULL, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0581000
mprotect(0xb0581000, 65536, PROT_READ|PROT_WRITE) = 0
mmap2(NULL, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0571000
mprotect(0xb0571000, 65536, PROT_READ|PROT_WRITE) = 0
mmap2(NULL, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0561000
mprotect(0xb0561000, 65536, PROT_READ|PROT_WRITE) = 0
mmap2(NULL, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0551000
mprotect(0xb0551000, 65536, PROT_READ|PROT_WRITE) = 0
munmap(0xb0591000, 65536)               = 0
munmap(0xb0581000, 65536)               = 0
munmap(0xb0571000, 65536)               = 0
munmap(0xb0561000, 65536)               = 0
munmap(0xb0551000, 65536)               = 0
cacheflush(0xb223361c, 0xb2233b44, 0, 0x23e0, 0x528) = 0
cacheflush(0xb2233b84, 0xb2233bf0, 0, 0x23e0, 0x6c) = 0
cacheflush(0xb2233c10, 0xb2233c58, 0, 0x23e0, 0x48) = 0
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRALN, si_addr=0xb20378cd} ---
+++ killed by SIGBUS +++
Bus error

As an one example, below is test result of AboveStackLimit.exe test case.

root@u1404:/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit# pwd
/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit

• with -O1 + release build ( ./Regressions/common/AboveStackLimit/AboveStackLimit.exe )

root@u1404:/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit# ./AboveStackLimit.sh  -coreroot=/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay-o1
BEGIN EXECUTION
/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay-o1/corerun AboveStackLimit.exe
Beginning test case Jitting a large method at 6/18/16 5:31:23 AM
Random seed: 20010415; set environment variable CORECLR_SEED to this value to repro

Beginning scenario: Jitting ~8099 arguments
Method JITTed and ran
Done

Ending test case at 6/18/16 5:31:24 AM
Expected: 100
Actual: 100
END EXECUTION - PASSED


root@u1404:/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit# /unit-test/Windows_NT.x64.Release.20160511/Tests/coreoverlay-o1/corerun  ./AboveStackLimit.exe
Beginning test case Jitting a large method at 6/18/16 5:49:02 AM
Random seed: 20010415; set environment variable CORECLR_SEED to this value to repro

Beginning scenario: Jitting ~8099 arguments
Method JITTed and ran
Done

Ending test case at 6/18/16 5:49:03 AM

AboveStackLimit.tar.gz

• with -O3 + release build ( ./Regressions/common/AboveStackLimit/AboveStackLimit.exe )

root@u1404:/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit# ./AboveStackLimit.sh  -coreroot=/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay-o3
BEGIN EXECUTION
/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay-o3/corerun AboveStackLimit.exe
./AboveStackLimit.sh: line 65: 15083 Bus error               $_DebuggerFullPath "$CORE_ROOT/corerun" AboveStackLimit.exe $CLRTestExecutionArguments $Host_Args
Expected: 100
Actual: 135
END EXECUTION - FAILED
root@u1404:/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit#

root@u1404:/unit-test/Windows_NT.x64.Release.20160519/Regressions/common/AboveStackLimit# /unit-test/Windows_NT.x64.Release.20160511/Tests/coreoverlay-o3/corerun  ./AboveStackLimit.exe
Bus error

root@u1404:/unit-test# demsg | tail 

[ 3849.057055] Alignment trap: not handling instruction ed940b00 at [<b21db226>]
[ 3849.059380] Unhandled fault: alignment exception (0x011) at 0xb1f378e2
[ 3849.061762] pgd = e308c000
[ 3849.064056] [b1f378e2] *pgd=914bc835
[ 3904.059584] Alignment trap: not handling instruction ed940a00 at [<b21db176>]
[ 3904.059653] Unhandled fault: alignment exception (0x011) at 0xb1f378c1
[ 3904.061931] pgd = e308c000
[ 3904.064180] [b1f378c1] *pgd=6d791835, *pte=65adb7df, *ppte=65adbe7f
[ 3904.066624] Alignment trap: not handling instruction ed940b00 at [<b21db226>]
[ 3904.068875] Unhandled fault: alignment exception (0x011) at 0xb1f378e2
[ 3904.071088] pgd = e308c000
[ 3904.073303] [b1f378e2] *pgd=6d791835, *pte=65adb7df, *ppte=65adbe7f
[ 3952.350271] Alignment trap: not handling instruction ed940a00 at [<b227b176>]
[ 3952.350339] Unhandled fault: alignment exception (0x011) at 0xb20378c1
[ 3952.352759] pgd = ed554000
[ 3952.354909] [b20378c1] *pgd=914cb835

• Debugging information with gdb (for -O3)

(gdb)
From        To          Syms Read   Shared Object Library
0xb6fdf840  0xb6ff21a0  Yes         /lib/ld-linux-armhf.so.3
0xb6fc99a0  0xb6fca24c  Yes         /lib/arm-linux-gnueabihf/libdl.so.2
0xb6fb2090  0xb6fbba54  Yes         /lib/arm-linux-gnueabihf/libpthread.so.0
0xb6f4aae0  0xb6f8e264  Yes (*)     /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
0xb6e9bc68  0xb6ec5c48  Yes         /lib/arm-linux-gnueabihf/libm.so.6
0xb6e84198  0xb6e8ef74  Yes (*)     /lib/arm-linux-gnueabihf/libgcc_s.so.1
0xb6daecc0  0xb6e519c0  Yes         /lib/arm-linux-gnueabihf/libc.so.6
0xb6969b90  0xb6c8c5a8  Yes         /unit-test/Windows_NT.x64.Release.20160511/Tests/coreoverlay-o3/libcoreclr.so
0xb6934840  0xb6936ed0  Yes         /lib/arm-linux-gnueabihf/librt.so.1
0xb6927f28  0xb69298e0  Yes (*)     /lib/arm-linux-gnueabihf/libuuid.so.1
0xb68f1dc0  0xb68f7934  Yes (*)     /usr/lib/arm-linux-gnueabihf/libunwind.so.8
0xb68b76d0  0xb68c09bc  Yes (*)     /usr/lib/arm-linux-gnueabihf/libunwind-arm.so.8
0xb68978b8  0xb68a5be8  Yes (*)     /lib/arm-linux-gnueabihf/liblzma.so.5
0xb228ced8  0xb23a4d3c  Yes         /unit-test/Windows_NT.x64.Release.20160511/Tests/coreoverlay-o3/libclrjit.so
0xb2272480  0xb2276afc  Yes         /unit-test/Windows_NT.x64.Release.20160511/Tests/coreoverlay-o3/System.Globalization.Native.so
0xb1f0f158  0xb1f81554  Yes (*)     /usr/lib/arm-linux-gnueabihf/libicuuc.so.52
0xb1dfb980  0xb1ea890c  Yes (*)     /usr/lib/arm-linux-gnueabihf/libicui18n.so.52
0xb070c430  0xb070c524  Yes (*)     /usr/lib/arm-linux-gnueabihf/libicudata.so.52
(*): Shared library is missing debugging information.
(gdb) info threads
  Id   Target Id         Frame
  5    Thread 0xb2bfe450 (LWP 15800) "corerun" __libc_do_syscall () at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:43
  4    Thread 0xb57e5450 (LWP 15799) "corerun" __libc_do_syscall () at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:43
  3    Thread 0xb5fe5450 (LWP 15798) "corerun" 0xb6fb9e1e in open () at ../sysdeps/unix/syscall-template.S:81
  2    Thread 0xb6855450 (LWP 15797) "corerun" 0xb6e22fc0 in poll () at ../sysdeps/unix/syscall-template.S:81
* 1    Thread 0xb6ff6000 (LWP 15790) "corerun" 0xb22db22a in getI8LittleEndian (ptr=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.hpp:766
(gdb)

(gdb)
(gdb) bt
#0  0xb22db22a in getI8LittleEndian (ptr=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.hpp:766
dotnet/coreclr#1  getR8LittleEndian (ptr=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.hpp:778
dotnet/coreclr#2  Compiler::impImportBlockCode (this=0xb4f25010, block=0xb4f275b4) at /work/dotnet/coreclr.git/src/jit/importer.cpp:9254
dotnet/coreclr#3  0xb22e8660 in operator() (this=<optimized out>, pParam=<optimized out>) at /work/dotnet/coreclr.git/src/jit/importer.cpp:14660
dotnet/coreclr#4  Compiler::impImportBlock (this=0xb4f25010, block=0xb4f275b4) at /work/dotnet/coreclr.git/src/jit/importer.cpp:14670
dotnet/coreclr#5  0xb22e9e1c in Compiler::impImport (this=0xb4f25010, method=<optimized out>) at /work/dotnet/coreclr.git/src/jit/importer.cpp:15741
dotnet/coreclr#6  0xb22ada8c in Compiler::fgImport (this=0xb4f25010) at /work/dotnet/coreclr.git/src/jit/flowgraph.cpp:6601
dotnet/coreclr#7  0xb229e220 in Compiler::compCompile (this=0xb4f25010, methodCodePtr=0xbefdec94, methodCodeSize=0xbefded70, compileFlags=0xbefdec98)
    at /work/dotnet/coreclr.git/src/jit/compiler.cpp:3488
dotnet/coreclr#8  0xb229eaae in Compiler::compCompileHelper (this=0xb4f25010, classPtr=<optimized out>, compHnd=<optimized out>, methodInfo=<optimized out>,
    methodCodePtr=0x1, methodCodeSize=0xb4f26534, compileFlags=0xbefdec98, instVerInfo=(unknown: 3035779088))
    at /work/dotnet/coreclr.git/src/jit/compiler.cpp:4912
dotnet/coreclr#9  0xb229e752 in operator() (this=<optimized out>, __JITpParam=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.cpp:4305
dotnet/coreclr#10 Compiler::compCompile (this=0xb4f25010, methodHnd=<optimized out>, classPtr=0xb602c010, compHnd=0xbefdeda8, methodInfo=0xb4f265c4,
    methodCodePtr=<optimized out>, methodCodeSize=<optimized out>, compileFlags=0xbefdec98) at /work/dotnet/coreclr.git/src/jit/compiler.cpp:4327
dotnet/coreclr#11 0xb229f16a in operator() (this=<optimized out>, __JITpParam=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.cpp:5523
dotnet/coreclr#12 operator() (this=<optimized out>, __JITpParam=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.cpp:5552
dotnet/coreclr#13 jitNativeCode (methodHnd=0xb06ed2f0, classPtr=0xb602c010, compHnd=0xbefdeda8, methodInfo=0xbefdee10, methodCodePtr=<optimized out>,
    methodCodeSize=<optimized out>, compileFlags=<optimized out>, inlineInfoPtr=0x0) at /work/dotnet/coreclr.git/src/jit/compiler.cpp:5554
dotnet/coreclr#14 0xb22a1dfe in CILJit::compileMethod (this=<optimized out>, compHnd=0xbefdeda8, methodInfo=0xbefdee10, flags=<optimized out>, entryAddress=0x0,
    nativeSizeOfCode=0xbefded70) at /work/dotnet/coreclr.git/src/jit/ee_il_dll.cpp:295
dotnet/coreclr#15 0xb69b1c38 in invokeCompileMethodHelper (jitMgr=0x3ff28, comp=<optimized out>, info=0xbefdee10, flags=1048592, flags2=0, nativeEntry=0xbefded74,
    nativeSizeOfCode=0xbefded70) at /work/dotnet/coreclr.git/src/vm/jitinterface.cpp:11834
dotnet/coreclr#16 0xb69b1cc8 in invokeCompileMethod (jitMgr=0x3ff28, comp=0xbefdeda8, info=0xbefdee10, flags=1048592, flags2=<optimized out>,
    nativeEntry=<optimized out>, nativeSizeOfCode=<optimized out>) at /work/dotnet/coreclr.git/src/vm/jitinterface.cpp:11889
dotnet/coreclr#17 0xb69b1d94 in operator() (this=<optimized out>, pParam=<optimized out>) at /work/dotnet/coreclr.git/src/vm/jitinterface.cpp:11951
dotnet/coreclr#18 CallCompileMethodWithSEHWrapper (jitMgr=0x0, comp=0x1, info=0xb4f26534, flags=1048592, flags2=0, nativeEntry=<optimized out>,
    nativeSizeOfCode=0xbefded70, ftn=<optimized out>) at /work/dotnet/coreclr.git/src/vm/jitinterface.cpp:11994
dotnet/coreclr#19 0xb69b247c in UnsafeJitFunction (ftn=0xb06ed2f0, ILHeader=<optimized out>, flags=<optimized out>, flags2=0, pSizeOfCode=0x0)
    at /work/dotnet/coreclr.git/src/vm/jitinterface.cpp:12635
dotnet/coreclr#20 0xb6b52538 in MethodDesc::MakeJitWorker (this=0xb06ed2f0, ILHeader=<optimized out>, flags=0, flags2=0)
    at /work/dotnet/coreclr.git/src/vm/prestub.cpp:462
dotnet/coreclr#21 0xb6b53280 in MethodDesc::DoPrestub (this=0xb06ed2f0, pDispatchingMT=0x0) at /work/dotnet/coreclr.git/src/vm/prestub.cpp:1454
dotnet/coreclr#22 0xb6b52c2a in PreStubWorker (pTransitionBlock=<optimized out>, pMD=0xb06ed2f0) at /work/dotnet/coreclr.git/src/vm/prestub.cpp:1041
dotnet/coreclr#23 0xb6aa3b72 in ThePreStub () from /unit-test/Windows_NT.x64.Release.20160511/Tests/coreoverlay-o3/libcoreclr.so
dotnet/coreclr#24 0xb05db7b0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info registers
r0             0x0      0
r1             0x1      1
r2             0xb4f26534       3035784500
r3             0x28     40
r4             0xb207d3ef       2986857455
r5             0xb4f25010       3035779088
r6             0xb4f25010       3035779088
r7             0xbefde998       3204311448
r8             0x7      7
r9             0x23     35
r10            0x4      4
r11            0x0      0
r12            0x1528   5416
sp             0xbefde520       0xbefde520
lr             0xb4f26534       -1259182796
pc             0xb22db22a       0xb22db22a <Compiler::impImportBlockCode(BasicBlock*)+7182>
cpsr           0x800e0030       -2146566096


(gdb) info frame
Stack level 0, frame at 0xbefde9b0:
 pc = 0xb22db22a in getI8LittleEndian (/work/dotnet/coreclr.git/src/jit/compiler.hpp:766); saved pc = 0xb22e8660
 inlined into frame 1
 source language c++.
 Arglist at unknown address.
 Locals at unknown address, Previous frame's sp in sp
(gdb) bt

Dump of assembler code for function Compiler::impImportBlockCode(BasicBlock*):
   0xb22d961c <+0>:     stmdb   sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}
   0xb22d9620 <+4>:     add     r7, sp, dotnet/coreclr#12
   0xb22d9622 <+6>:     sub.w   sp, sp, dotnet/runtime#4305   ; 0x468
   0xb22d9626 <+10>:    sub     sp, dotnet/coreclr#4
   0xb22d9628 <+12>:    mov     r10, r0
   0xb22d962a <+14>:    ldr.w   r0, [pc, dotnet/runtime#5135] ; 0xb22da2b4 <Compiler::impImportBlockCode(BasicBlock*)+3224>
   0xb22d962e <+18>:    mov     r8, r1
   0xb22d9630 <+20>:    ldr.w   r1, [pc, dotnet/runtime#5132] ; 0xb22da2b8 <Compiler::impImportBlockCode(BasicBlock*)+3228>
   0xb22d9634 <+24>:    add     r0, pc
   0xb22d9636 <+26>:    str.w   r10, [sp, dotnet/runtime#3950] ; 0xc4
   0xb22d963a <+30>:    ldr     r0, [r1, r0]
   0xb22d963c <+32>:    ldr     r0, [r0, #0]
   0xb22d963e <+34>:    str.w   r0, [r7, #-20]
   0xb22d9642 <+38>:    ldr.w   r0, [r10, dotnet/runtime#4556]        ; 0x6b0
   0xb22d9646 <+42>:    cbz     r0, 0xb22d9658 <Compiler::impImportBlockCode(BasicBlock*)+60>
   0xb22d9648 <+44>:    ldr.w   r0, [r0, dotnet/runtime#4075]  ; 0x228
   0xb22d964c <+48>:    mov.w   r9, #4294967295
   0xb22d9650 <+52>:    ldr     r0, [r0, dotnet/runtime#3880]   ; 0x3c
   0xb22d9652 <+54>:    str.w   r0, [r10, dotnet/runtime#4646]        ; 0x7ac
   0xb22d9656 <+58>:    b.n     0xb22d971a <Compiler::impImportBlockCode(BasicBlock*)+254>
   0xb22d9658 <+60>:    mov.w   r0, #2147483648 ; 0x80000000
   0xb22d965c <+64>:    movw    r1, dotnet/coreclr#5412       ; 0x1524
   0xb22d9660 <+68>:    str.w   r0, [r10, dotnet/runtime#4646]        ; 0x7ac
   0xb22d9664 <+72>:    movw    r0, dotnet/coreclr#5308       ; 0x14bc
   0xb22d9668 <+76>:    ldr.w   r0, [r10, r0]
   0xb22d966c <+80>:    mov     r5, r10
   0xb22d966e <+82>:    ldr.w   r10, [r5, r1]
   0xb22d9672 <+86>:    ldr     r4, [r0, dotnet/runtime#3893]   ; 0x54
   0xb22d9674 <+88>:    cmp.w   r10, dotnet/coreclr#255       ; 0xff
   0xb22d9678 <+92>:    bhi.n   0xb22d968a <Compiler::impImportBlockCode(BasicBlock*)+110>
   0xb22d967a <+94>:    movw    r0, dotnet/coreclr#5300       ; 0x14b4
   0xb22d967e <+98>:    ldrb    r0, [r5, r0]
   0xb22d9680 <+100>:   tst.w   r0, dotnet/coreclr#1
   0xb22d9684 <+104>:   it      ne
   0xb22d9686 <+106>:   strne.w r4, [r5, dotnet/runtime#4646] ; 0x7ac
   0xb22d968a <+110>:   cbnz    r4, 0xb22d969c <Compiler::impImportBlockCode(BasicBlock*)+128>
   0xb22d968c <+112>:   movs    r0, #0
   0xb22d968e <+114>:   cmp.w   r10, dotnet/coreclr#255       ; 0xff
   0xb22d9692 <+118>:   it      hi
   0xb22d9694 <+120>:   movhi.w r0, #2147483648 ; 0x80000000
   0xb22d9698 <+124>:   str.w   r0, [r5, dotnet/runtime#4646] ; 0x7ac
   0xb22d969c <+128>:   movw    r0, dotnet/coreclr#5296       ; 0x14b0
   0xb22d96a0 <+132>:   ldr     r6, [r5, r0]
---Type <return> to continue, or q <return> to quit---
   0xb22d96a2 <+134>:   cmp     r6, #0
   0xb22d96a4 <+136>:   beq.n   0xb22d96dc <Compiler::impImportBlockCode(BasicBlock*)+192>
   0xb22d96a6 <+138>:   mul.w   r0, r6, r4
   0xb22d96aa <+142>:   mov.w   r1, dotnet/coreclr#5216       ; 0x1460
   0xb22d96ae <+146>:   ldr     r1, [r5, r1]
   0xb22d96b0 <+148>:   blx     0xb228cac4
   0xb22d96b4 <+152>:   cmp     r0, r6
   0xb22d96b6 <+154>:   mov     r12, r5
   0xb22d96b8 <+156>:   it      cs
   0xb22d96ba <+158>:   subcs   r0, r6, dotnet/coreclr#1
   0xb22d96bc <+160>:   subs    r1, r0, dotnet/coreclr#1
   0xb22d96be <+162>:   movw    r0, dotnet/runtime#5961       ; 0x14ac
   0xb22d96c2 <+166>:   adds    r2, r5, r0
   0xb22d96c4 <+168>:   cmp.w   r1, #4294967295
   0xb22d96c8 <+172>:   beq.n   0xb22d96e4 <Compiler::impImportBlockCode(BasicBlock*)+200>
   0xb22d96ca <+174>:   ldr     r0, [r2, #0]
   0xb22d96cc <+176>:   ldr.w   r3, [r0, r1, lsl dotnet/coreclr#2]
   0xb22d96d0 <+180>:   subs    r1, dotnet/coreclr#1
   0xb22d96d2 <+182>:   cmp     r3, r4
   0xb22d96d4 <+184>:   bcs.n   0xb22d96c4 <Compiler::impImportBlockCode(BasicBlock*)+168>
   0xb22d96d6 <+186>:   add.w   r9, r1, dotnet/coreclr#2
   0xb22d96da <+190>:   b.n     0xb22d96ea <Compiler::impImportBlockCode(BasicBlock*)+206>
   0xb22d96dc <+192>:   mov.w   r9, #4294967295
   0xb22d96e0 <+196>:   mov     r10, r5
   0xb22d96e2 <+198>:   b.n     0xb22d971a <Compiler::impImportBlockCode(BasicBlock*)+254>
   0xb22d96e4 <+200>:   ldr     r0, [r2, #0]
   0xb22d96e6 <+202>:   mov.w   r9, #0
   0xb22d96ea <+206>:   ldr.w   r1, [r0, r9, lsl dotnet/coreclr#2]
   0xb22d96ee <+210>:   cmp     r1, r4
   0xb22d96f0 <+212>:   bcs.n   0xb22d9700 <Compiler::impImportBlockCode(BasicBlock*)+228>
   0xb22d96f2 <+214>:   add.w   r9, r9, dotnet/coreclr#1
   0xb22d96f6 <+218>:   cmp     r6, r9
   0xb22d96f8 <+220>:   bne.n   0xb22d96ea <Compiler::impImportBlockCode(BasicBlock*)+206>
   0xb22d96fa <+222>:   mov     r9, r6
   0xb22d96fc <+224>:   mov     r10, r12
   0xb22d96fe <+226>:   b.n     0xb22d971a <Compiler::impImportBlockCode(BasicBlock*)+254>
   0xb22d9700 <+228>:   bne.n   0xb22d9718 <Compiler::impImportBlockCode(BasicBlock*)+252>
   0xb22d9702 <+230>:   cmp.w   r10, dotnet/coreclr#255       ; 0xff
   0xb22d9706 <+234>:   add.w   r9, r9, dotnet/coreclr#1
   0xb22d970a <+238>:   mov     r10, r12
   0xb22d970c <+240>:   it      hi
   0xb22d970e <+242>:   orrhi.w r4, r4, #2147483648     ; 0x80000000
   0xb22d9712 <+246>:   str.w   r4, [r10, dotnet/runtime#4646]        ; 0x7ac
   0xb22d9716 <+250>:   b.n     0xb22d971a <Compiler::impImportBlockCode(BasicBlock*)+254>
---Type <return> to continue, or q <return> to quit---

• Related Source code

> #0  0xb22db22a in getI8LittleEndian (ptr=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.hpp:766
> dotnet/coreclr#1  getR8LittleEndian (ptr=<optimized out>) at /work/dotnet/coreclr.git/src/jit/compiler.hpp:778
> dotnet/coreclr#2  Compiler::impImportBlockCode (this=0xb4f25010, block=0xb4f275b4) at /work/dotnet/coreclr.git/src/jit/importer.cpp:9254
> 

[ getI8LittleEndian function in ./coreclr.git/src/jit/compiler.hpp]

 751  
 752 inline
 753   signed __int8     getI1LittleEndian(const BYTE * ptr)
 754 { return * (UNALIGNED signed __int8 *)ptr; }
 755  
 756 inline
 757   signed __int16    getI2LittleEndian(const BYTE * ptr)
 758 { return * (UNALIGNED signed __int16 *)ptr; }
 759  
 760 inline
 761   signed __int32    getI4LittleEndian(const BYTE * ptr)
 762 { return *(UNALIGNED signed __int32*)ptr; }
 763  
 764 inline
 765   signed __int64    getI8LittleEndian(const BYTE * ptr)                                   
 766 { return *(UNALIGNED signed __int64*)ptr; }
 767  
 768 inline
 769 float               getR4LittleEndian(const BYTE * ptr)
 770 {
 771     __int32 val = getI4LittleEndian(ptr);
 772     return *(float *)&val;
 773 }
 774  

[leemgs]

We have still got the same issue even though we try to build coreCLR with "-mno-unaligned-access" compiler flag. Anyone who has similar experience?

• reference:
  http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20130819/086848.html
  http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0203g/Chdbiafd.html
  https://developer.arm.com/docs/dui0774/latest/compiler-command-line-options/-munaligned-access-mno-unaligned-access
• how to do unit-test (e.g., baseservices/exceptions/unittests)

cd ./coreclr.git/Tests/
time ./runtest.sh  \
--testRootDir=/unit-test/${WINDOWS_NT_DIR}  \
--testNativeBinDir=/unit-test/${BIN_CORECLR}/obj/Linux.arm.${BUILD_TYPE}  \
--coreClrBinDir=/unit-test/${BIN_CORECLR}/Product/Linux.arm.${BUILD_TYPE}  \
--mscorlibDir=/unit-test/${BIN_CORECLR}/Product/Linux.arm.${BUILD_TYPE}  \
--coreFxBinDir="/unit-test/${BIN_COREFX}/Linux.AnyCPU.${BUILD_TYPE};/unit-test/${BIN_COREFX}/Unix.AnyCPU.${BUILD_TYPE};/unit-test/${BIN_COREFX}/AnyOS.AnyCPU.${BUILD_TYPE}"  \
--coreFxNativeBinDir=/unit-test/${BIN_COREFX}/Linux.arm.${BUILD_TYPE} \
--testDir=baseservices/exceptions/unittests \

• with O1 + release build

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : /unit-test/bin.coreclr.release.20160613.O1/Product/Linux.arm.Release
# Tests Discovered : 19
# Passed           : 19
# Failed           : 0
# Skipped          : 0
=======================
0 minutes and 19 seconds taken to run CoreCLR tests.

real    0m37.967s
user    0m32.825s
sys 0m4.735s

• with O3 + release build

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : /unit-test/bin.coreclr.release.20160706-1440.O3.no-unaligned-access/Product/Linux.arm.Release
# Tests Discovered : 19
# Passed           : 8
# Failed           : 11
# Skipped          : 0
=======================
0 minutes and 19 seconds taken to run CoreCLR tests.

real    0m52.666s
user    0m17.825s
sys 0m5.280s
root@target:/unit-test/coreclr.git/tests# 

• O3 + release build : bus error ( )

baseservices    compilerservices    dynamicobjectproperties TestAPIs    TestAPIs
baseservices    exceptions  regressions v4  test640474
baseservices    exceptions  sharedexceptions    emptystacktrace OOMException01
baseservices    exceptions  simple  ArrayInit   ArrayInit
baseservices    exceptions  simple  finally finally
baseservices    exceptions  simple  HardwareEh  HardwareEh
baseservices    exceptions  StackTracePreserve  StackTracePreserveTests StackTracePreserveTests
baseservices    exceptions  unittests   Baadbaad    Baadbaad
baseservices    exceptions  unittests   BaseClass   BaseClass
baseservices    exceptions  unittests   EHPatternTests  EHPatternTests
baseservices    exceptions  unittests   Pending Pending
baseservices    exceptions  unittests   Recurse Recurse
baseservices    exceptions  unittests   RecursiveRethrow    RecursiveRethrow
baseservices    exceptions  unittests   RecursiveRethrow    RecursiveThrowNew
baseservices    exceptions  unittests   RethrowAndFinally   RethrowAndFinally

• Monitoring the alignment cases in kernel space

root@chrome~# cat /proc/cpu/alignment 
User:       246719
System:     0 (  (null))
Skipped:    8042
Half:       0
Word:       0
DWord:      195386
Multi:      32548
User faults:    2 (fixup)
root@target:~# 

[leemgs]

Below is some of the test cases to avoid "Bus error" message. And I have got the same error message when I built coreCLR with -O3 both debug build and release build.

Case 1: ( with O3)

dxl@target:/unit-test$ cd /unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ ./corerun /unit-test/example/hello.exe
Bus error

Case 2: ( with O3)

dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ 
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ su 
Password: 
root@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay# ./corerun /unit-test/example/hello.exe
Bus error

Case 3: ( with O3)

dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ 
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ 
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ su -
Password: 
root@target:~# cd /unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay
root@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay# ./corerun /unit-test/example/hello.exe
Hello World

The Case 4 was reported by @myungjoo.

Case 4: with user account (dxl) , with none lang setting

dxl@target:/$  cd /unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ 
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ set | grep LC
LC_ADDRESS=ko_KR.UTF-8
LC_IDENTIFICATION=ko_KR.UTF-8
LC_MEASUREMENT=ko_KR.UTF-8
LC_MONETARY=ko_KR.UTF-8
LC_NAME=ko_KR.UTF-8
LC_NUMERIC=ko_KR.UTF-8
LC_PAPER=ko_KR.UTF-8
LC_TELEPHONE=ko_KR.UTF-8
LC_TIME=ko_KR.UTF-8
MAILCHECK=60
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ cat no-locale.sh
export LANG=""
export LC_ADDRESS=""
export LC_IDENTIFICATION=""
export LC_MEASUREMENT=""
export LC_MONETARY=""
export LC_NAME=""
export LC_NUMERIC=""
export LC_PAPER=""
export LC_TELEPHONE=""
export LC_TIME=""

dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ ./no-locale.sh
dxl@target:/unit-test/Windows_NT.x64.Release.20160519/Tests/coreoverlay$ ./corerun  ./hello.exe 
Hello World

\CC: @lemmaa, @chunseoklee, @seanshpark, @mylibero, @parjong, @idkiller, @ILyoan, @watere , @sjsinju, @hongsekim, @jyoungyun, @papaslavik , @hqueue , @sam-baravy

\CC: @jkotas , @benpye

[leemgs]

Adding: @tilmannOSG
Could you give me some hints to resolve this issue if you already have similar experience?

[jkotas]

JIT helper not supported: CORINFO_HELP_READYTORUN_NEW

Not sure whether it is related to the crashes you are seeing ... but these messages are not expected when this test is running. Looks like there is a missing case in codegenlegacy.cpp - you can try the following fix:

            helperNum = compiler->eeGetHelperNum(methHnd);

+ #ifdef FEATURE_READYTORUN_COMPILER
+             if (call->gtCall.gtEntryPoint.addr != NULL)
+             {
+                accessType = call->gtCall.gtEntryPoint.accessType;
+                addr = call->gtCall.gtEntryPoint.addr;
+            }
+            else
+ #endif
            if (callType == CT_HELPER)

[leemgs]

Looks like there is a missing case in codegenlegacy.cpp - you can try the following fix:

@jkotas , Thank you so much. I will check the effect of the above patch.

[leemgs]

Here is evaluation result. Unfortunately, we could not resolve "Bus error" with the above patch.

• Test folders: --testDir=readytorun
• Test result:

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : /unit-test/bin.coreclr.release.o3.20160617-jkotas/Product/Linux.arm.Release
# Tests Discovered : 2
# Passed           : 0
# Failed           : 2
# Skipped          : 0
=======================
0 minutes and 1 seconds taken to run CoreCLR tests.

• Kernel messages:

root@arm# demsg | tail

[ 3849.057055] Alignment trap: not handling instruction ed940b00 at [<b21db226>]
[ 3849.059380] Unhandled fault: alignment exception (0x011) at 0xb1f378e2
[ 3849.061762] pgd = e308c000
[ 3849.064056] [b1f378e2] *pgd=914bc835
[ 3904.059584] Alignment trap: not handling instruction ed940a00 at [<b21db176>]
[ 3904.059653] Unhandled fault: alignment exception (0x011) at 0xb1f378c1
[ 3904.061931] pgd = e308c000
[ 3904.064180] [b1f378c1] *pgd=6d791835, *pte=65adb7df, *ppte=65adbe7f
[ 3904.066624] Alignment trap: not handling instruction ed940b00 at [<b21db226>]
[ 3904.068875] Unhandled fault: alignment exception (0x011) at 0xb1f378e2
[ 3904.071088] pgd = e308c000
[ 3904.073303] [b1f378e2] *pgd=6d791835, *pte=65adb7df, *ppte=65adbe7f
[ 3952.350271] Alignment trap: not handling instruction ed940a00 at [<b227b176>]
[ 3952.350339] Unhandled fault: alignment exception (0x011) at 0xb20378c1
[ 3952.352759] pgd = ed554000
[ 3952.354909] [b20378c1] *pgd=914cb835

[benpye]

I recall hitting this before, I thought I contributed a fix in some PR but perhaps it didn't get merged. If you check https://github.com/dotnet/coreclr/blob/master/src/pal/inc/pal.h#L204 you can see that the UNALIGNED keyword will only be defined on a certain set of compilers. ARM requires aligned access I believe for 2 or 4 byte reads (on a 2 or 4 byte boundry respectively). I believe on ARM you'll have to do something similar to the big endian helpers, or you could memcpy into the desired width type I believe (I think this is what I was doing). See http://stackoverflow.com/a/29856789

[myungjoo]

Here we go: @hqueue

[leemgs]

This comment is to update my experiences about 3,800 failures in case of the unit test with O3. First of all, I am sorry for late comment because of the long vacation. Recently, I have got a strange phenomenon about this issue between real target device and our own linux/arm emulator. We cannot execute some unit tests on real target device (e.g., raspberry Pi2 board , ARM Chromebook ) However, we can always execute some unit tests tests on linux/arm emulator.

• linux/arm emulator - https://github.com/dotnet/coreclr/issues/3805

For example,

1. on linux/arm emulator (It's okay.)

arm@emul#> ./corerun  /nfs/common/CompEx/CompEx.exe 
Beginning test case CompareExchange(Int64&,Int64,Int64) at 16. 7. 7. 오전 5:52:51
Random seed: 20010415; set environment variable CORECLR_SEED to this value to repro
Beginning scenario: CompareExchange(0,1,0)
BEFORE: T(0) NI(0) NV(1) I(0)
AFTER: T(1) NI(0) NV(1) I(0)
BEFORE: T(1) NI(0) NV(2) I(1)
AFTER: T(2) NI(1) NV(2) I(1)
Ending test case at 16. 7. 7. 오전 5:52:52
PASS

1. on real arm target device (unit tests were always failed. )

root@target:/unit-test/runtime-arm-u1404-release-mode-20160707.2# ./corerun  /unit-test/common/CompEx/CompEx.exe 
Bus error

Below is kernel log messages in case of the real target device. 
[91909.525677] Alignment trap: not handling instruction ed940b00 at [<b21db3d2>]
[91909.525747] Unhandled fault: alignment exception (0x011) at 0xb1f7e3e3
[91909.527989] pgd = ed1e4000
[91909.530218] [b1f7e3e3] *pgd=58d53835, *pte=76bd77df, *ppte=76bd7e7f
[91909.532890] Alignment trap: not handling instruction ed940b00 at [<b21db3d2>]
[91909.535146] Unhandled fault: alignment exception (0x011) at 0xb1f7e3f5
[91909.537402] pgd = ed1e4000
[91909.539628] [b1f7e3f5] *pgd=58d53835, *pte=76bd77df, *ppte=76bd7e7f

If someone already have a similar experience like below case, Could you share me that? I have been discussing the arm core specific backgrounds with Linaro/ARM engineers. Now, I am going to concentrate on the appropriate solution among useful some approaches in order to fix 3800 failures of coreclr's unit test with O3 optimization levels.

1. build coreclr with __unaligned
2. build coreclr with __packed
3. try memcpy to handle 4byte access of ARM core (But, read 8byte access in case of 64bit)
4. apply kernel-level ARM errata (from ARM company)
5. use no-unaligned flag with latest llvm
6. build coreclr with llvm's "#pragma pack(1) ... #pragma options align=reset "
7. understanding ARM core structure (infocenter.arm.com - How does the ARM Compiler support unaligned accesses?)
8. and so on

[jkotas]

These failures may be also caused by a Linux-arm specific bug in FEATURE_64BIT_ALIGNMENT - just something to add to your list of potential culprits.