SIGILL on old CPU

[antekone]

I'm trying to use Cargo on a computer with an old CPU:

(1:664)$ cat /proc/cpuinfo
processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 6
model       : 10
model name  : AMD Athlon(tm) XP 3200+
stepping    : 0
(...)
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow

This CPU is compatible with i686 architecture. When I'm trying to use Cargo (i.e. by using "cargo new --bin"), OR when trying to compile Cargo myself, I'm getting a crash:

(1:663)$ LC_ALL="C" make
"/usr/bin/rustc" -v
rustc 0.12.0-dev
target/snapshot/bin/cargo build --target i686-unknown-linux-gnu  
Makefile:77: recipe for target 'cargo-i686-unknown-linux-gnu' failed
make: *** [cargo-i686-unknown-linux-gnu] Illegal instruction (core dumped)

Here is some debug info, extracted from the coredump generated in the crash above:

Core was generated by `target/snapshot/bin/cargo build --target i686-unknown-linux-gnu'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0xb759ed44 in stack_overflow::imp::signal_handler::term::h95cc93235a9133cb2Oa ()
(gdb) disassemble 
Dump of assembler code for function _ZN14stack_overflow3imp14signal_handler4term20h95cc93235a9133cb2OaE:
   0xb759ed00 <+0>: cmp    %gs:0x30,%esp
   0xb759ed07 <+7>: ja     0xb759ed19 <_ZN14stack_overflow3imp14signal_handler4term20h95cc93235a9133cb2OaE+25>
   0xb759ed09 <+9>: push   $0x0
   0xb759ed0e <+14>:    push   $0x1c
   0xb759ed13 <+19>:    call   0xb705a80c <__morestack>
   0xb759ed18 <+24>:    ret    
   0xb759ed19 <+25>:    push   %ebx
   0xb759ed1a <+26>:    push   %esi
   0xb759ed1b <+27>:    sub    $0x14,%esp
   0xb759ed1e <+30>:    mov    %ecx,%esi
   0xb759ed20 <+32>:    call   0xb759ed25 <_ZN14stack_overflow3imp14signal_handler4term20h95cc93235a9133cb2OaE+37>
   0xb759ed25 <+37>:    pop    %ebx
   0xb759ed26 <+38>:    add    $0x2220e3,%ebx
   0xb759ed2c <+44>:    mov    %esi,(%esp)
   0xb759ed2f <+47>:    movl   $0x0,0x4(%esp)
   0xb759ed37 <+55>:    call   0xb7001240 <signal@plt>
   0xb759ed3c <+60>:    mov    %esi,(%esp)
   0xb759ed3f <+63>:    call   0xb7001990 <raise@plt>
=> 0xb759ed44 <+68>:    ud2    
End of assembler dump.
(gdb) bt
#0  0xb759ed44 in stack_overflow::imp::signal_handler::term::h95cc93235a9133cb2Oa ()
#1  0xb759ecd5 in stack_overflow::imp::signal_handler::hed7998b12c57a9dcJOa ()
#2  <signal handler called>
#3  0xb735e02d in lh_insert ()
#4  0xb72f7f44 in int_err_set_item ()
#5  0xb72f8afe in ERR_load_ERR_strings ()
#6  0xb7350c68 in ERR_load_crypto_strings ()
#7  0xb73067a8 in SSL_load_error_strings ()
#8  0xb728f27a in init_ssl () at /home/rustbuild/src/rust-buildbot/slave/cargo-linux-32/build/.cargo/git/checkouts/git2-rs-cf258e7bcbaee34d/no-build-cmd/libgit2/build/libgit2/src/global.c:72
#9  init_once () at /home/rustbuild/src/rust-buildbot/slave/cargo-linux-32/build/.cargo/git/checkouts/git2-rs-cf258e7bcbaee34d/no-build-cmd/libgit2/build/libgit2/src/global.c:241
#10 init_once () at /home/rustbuild/src/rust-buildbot/slave/cargo-linux-32/build/.cargo/git/checkouts/git2-rs-cf258e7bcbaee34d/no-build-cmd/libgit2/build/libgit2/src/global.c:229
#11 0xb6f4673e in pthread_once () from /usr/lib/libpthread.so.0
#12 0xb728f447 in git_threads_init () at /home/rustbuild/src/rust-buildbot/slave/cargo-linux-32/build/.cargo/git/checkouts/git2-rs-cf258e7bcbaee34d/no-build-cmd/libgit2/build/libgit2/src/global.c:248
#13 0xb728f0a3 in init::closure.10156 ()
#14 0xb759d646 in one::Once::doit::h9e2f899a6ed191f1F7b ()
#15 0xb728c867 in repo::Repository::open::hdfc674d4138e412cIsf ()
#16 0xb71e4424 in sources::git::utils::GitRemote::db_at::h9596b2c826953124N9o ()
#17 0xb7064ce6 in sources::git::source::GitSource$LT$$x27a$C$$x20$x27b$GT$.Source::update::h4ea97016ef1ca3f2RSp ()
#18 0xb70b103a in core::registry::PackageRegistry$LT$$x27a$GT$::load::ha6cd4266d11b13edvug()
#19 0xb70bd349 in core::registry::PackageRegistry$LT$$x27a$GT$.Registry::query::he5fb17744c47eb94WBg ()
#20 0xb71da751 in core::resolver::activate::closure.27091 ()
#21 0xb71d965a in core::resolver::activate::h5098706157662105859 ()
#22 0xb70e1e32 in ops::resolve::resolve_with_previous::h64749ffa8746717dAao ()
#23 0xb70ded3f in ops::resolve::resolve_pkg::hae746acc4e161a6e88n ()
#24 0xb70d5dfa in ops::cargo_compile::compile_pkg::h5d89a256d8ef97aekgh ()
#25 0xb70d3a1f in ops::cargo_compile::compile::h47c8064cbc132422mbh ()
#26 0xb701b67e in call_main_without_stdin::h12692426207735314450 ()
#27 0xb700a592 in execute::hb805ce1b2bf062d7fda ()
#28 0xb70053f6 in call_main_without_stdin::h1306324012682129432 ()
#29 0xb7003502 in main::h0f7df40cd3e74b916ca ()
#30 0xb7494125 in start::closure.2821 ()
#31 0xb75a2554 in unwind::try::try_fn::hdb813388f395c605x0c ()
#32 0xb75a3543 in rust_try_inner ()
#33 0xb75a350c in rust_try ()
#34 0xb75a0baf in unwind::try::hcaa3d901d7810b5bmYc ()
#35 0xb75a0a69 in task::Task::run::h5d76bfe6ab305402u4b ()
#36 0xb7493f79 in start::h854aa1e34251dd912ma ()
#37 0xb7493ded in lang_start::hcae264f3f21854belma ()
#38 0xb70036a7 in main ()
(gdb) 

As a sidenote, the same executable file (cargo) on a different CPU (i7-4470k) works perfectly OK!

[alexcrichton]

This looks like it's actually the stack overflow handler getting triggered a little too early, not entirely sure why...

[antekone]

When I try to use bundled cargo to compile the source code, I'm getting SIGFPE instead:

Program received signal SIGFPE, Arithmetic exception.
0x8043066a in lh_insert ()
(gdb) bt
#0  0x8043066a in lh_insert ()
#1  0x80479714 in int_err_set_item ()
#2  0x8047a2ce in ERR_load_ERR_strings ()
#3  0x80427ae8 in ERR_load_crypto_strings ()
#4  0x80404ae8 in SSL_load_error_strings ()
#5  0x8034e59d in init_ssl () at /home/rustbuild/src/rust-buildbot/slave/cargo-linux-32/build/.cargo/git/checkouts/git2-rs-cf258e7bcbaee34d/master/libgit2-sys/libgit2/src/global.c:72
#6  init_once () at /home/rustbuild/src/rust-buildbot/slave/cargo-linux-32/build/.cargo/git/checkouts/git2-rs-cf258e7bcbaee34d/master/libgit2-sys/libgit2/src/global.c:241
(...)

Rest of the stacktrace is the same as above.

[alexcrichton]

Can you run a disassembly of lh_insert to see what instruction it's faulting on inside that function as well as get a backtrace for all other threads in the program?

[antekone]

Of course, here it is:

(2:658)$ gdb -q --args cargo build
Reading symbols from cargo...done.
(gdb) r
Starting program: /home/antek/bin/cargo/cargo-nightly-i686-unknown-linux-gnu/bin/cargo build
[...]

Program received signal SIGFPE, Arithmetic exception.
0x8043066a in lh_insert ()
(gdb) disassemble
Dump of assembler code for function lh_insert:
0x80430630 <+0>: push %ebp
0x80430631 <+1>: xor %edx,%edx
0x80430633 <+3>: push %edi
0x80430634 <+4>: push %esi
0x80430635 <+5>: push %ebx
0x80430636 <+6>: sub $0x2c,%esp
0x80430639 <+9>: mov 0x40(%esp),%ebx
0x8043063d <+13>: mov 0x24(%ebx),%eax
0x80430640 <+16>: mov 0xc(%ebx),%ecx
0x80430643 <+19>: movl $0x0,0x5c(%ebx)
0x8043064a <+26>: shl $0x8,%eax
0x8043064d <+29>: div %ecx
0x8043064f <+31>: cmp %eax,0x1c(%ebx)
0x80430652 <+34>: jbe 0x804306e0 <lh_insert+176>
0x80430658 <+40>: mov 0x44(%esp),%esi
0x8043065c <+44>: mov %esi,(%esp)
0x8043065f <+47>: call *0x8(%ebx)
0x80430662 <+50>: xor %edx,%edx
0x80430664 <+52>: addl $0x1,0x38(%ebx)
0x80430668 <+56>: mov %eax,%ebp
=> 0x8043066a <+58>: divl 0x18(%ebx)
(...)
(gdb) p/x $ebx
$2 = 0x8080d3c8
(gdb) p/x $eax
$3 = 0x1000069
(gdb) p/x $edx
$4 = 0x0
(gdb) x/4bx $ebx+0x18
0x8080d3e0:     0x00    0x00    0x00    0x00

I'm not 100% sure I remember exactly how DIV works, but it seems that it's a Null Pointer Dereference error during trying to dereference a pointer stored in [ebx + 18h] -- which seems to be 'nullptr'. If that's not the case, then sorry for confusion ;).

Threading info -- seems it's just one thread:

(gdb) info threads
Id Target Id Frame
* 1 Thread 0xb7d40700 (LWP 23213) "cargo" 0x8043066a in lh_insert () 

[alexcrichton]

I think that's a divide-by-zero exception (causing the SIGFPE). I feel like I've seen this in the past, but I forget how we fixed it... The internet tells me that this is generally due to concurrent initialization of openssl (bad), but there's only one thread here, so I'm... not entirely sure what's going on!

[antekone]

While I still don't know for 100%, but I guess this is most likely due to the binaries built to something newer than i686, despite the i686 tag names in filenames. I have this kind of problems all over the place.

1. I'm using ArchLinux, which is i686 compatible, and some packages are built with SSE2 support,
2. Ruby developers are using SSE2 even when building for i486 targets,
3. Now I've checked that rustc executable, taken from website's rustup.sh script, despite being named as i686 compatible, contains SSE2 instructions (namely, librustc_llvm's sys::MemoryFence() contains mfence instruction), so this rustc crashes on my old CPU with SIGILL.

I won't dig this issue anymore, because my problem is most likely very isolated (not many people are using CPUs without SSE2 I guess). I'll simply continue using rust on other CPUs.

[codyps]

@antekone that sounds like a legitimate issue. Opening a bug report on for rust itself ( https://github.com/rust-lang/rust ) and referencing this one would be useful

[alexcrichton]

@antekone does this still reproduce?

[antekone]

Just checked with rustup.sh:

(2:737)$ curl -s https://static.rust-lang.org/rustup.sh | sudo sh
[sudo] password for antek: 
rustup: CFG_CURL             := /usr/bin/curl (7.38.0)
rustup: CFG_TAR              := /usr/bin/tar (1.28)
rustup: CFG_FILE             := /usr/bin/file (5.20)
rustup: CFG_SHA256SUM        := /usr/bin/sha256sum (256sum)
rustup: CFG_SHASUM           := /usr/bin/core_perl/shasum (5.88)
rustup: 
rustup: processing sh args
rustup: 
rustup: CFG_PREFIX           :=  
rustup: CFG_DATE             :=  
rustup: 
rustup: validating sh args
rustup: 
rustup: host triple: i686-unknown-linux-gnu
rustup: Downloading https://static.rust-lang.org/dist/rust-nightly-i686-unknown-linux-gnu.tar.gz to     /tmp/tmp.iaTKx8pNHc/rust-nightly-i686-unknown-linux-gnu.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  116M  100  116M    0     0   429k      0  0:04:36  0:04:36 --:--:--  511k
rustup: Downloading https://static.rust-lang.org/dist/rust-nightly-i686-unknown-linux-gnu.tar.gz.sha256
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   109  100   109    0     0    161      0 --:--:-- --:--:-- --:--:--   161
rustup: Verifying hash
rustup: Extracting /tmp/tmp.iaTKx8pNHc/rust-nightly-i686-unknown-linux-gnu.tar.gz
install: looking for install programs
install: 
install: found mkdir
install: found printf
install: found cut
install: found grep
install: found uname
install: found tr
install: found sed
install: found chmod
install: 
install: processing /tmp/tmp.iaTKx8pNHc/rust-nightly-i686-unknown-linux-gnu/install.sh args
install: 
install: CFG_DESTDIR          :=  
install: CFG_PREFIX           := /usr/local 
install: CFG_LIBDIR           := /usr/local/lib 
install: CFG_MANDIR           := /usr/local/share/man 
install: 
install: validating /tmp/tmp.iaTKx8pNHc/rust-nightly-i686-unknown-linux-gnu/install.sh args
install: 
install: verifying platform can run binaries
/tmp/tmp.iaTKx8pNHc/rust-nightly-i686-unknown-linux-gnu/install.sh: linia 381:  9544 Błędna instrukcja     (zrzut pamięci) "${CFG_SRC_DIR}/bin/${TEMPLATE_VERIFY_BIN}" --version 2> /dev/null > /dev/null
install: error: can't execute binaries on this platform
rustup: error: failed to install Rust

"Błędna instrukcja (zrzut pamięci)" means "Invalid instruction (core dumped)".

Later, I've checked by directly downloading rust package like this:

$ wget https://static.rust-lang.org/dist/rust-nightly-i686-unknown-linux-gnu.tar.gz
[...]
$ tar zxf rust-nightly-i686-unknown-linux-gnu.tar.gz
$ cd rust-nightly-i686-unknown-linux-gnu
$ LD_LIBRARY_PATH=lib bin/rustc
Błędna instrukcja (zrzut pamięci)

(again, SIGILL plus core dumped). This was done today between 8:00 - 10:30 CET.

(2:755)$ LD_LIBRARY_PATH=lib gdb --args bin/rustc
Reading symbols from bin/rustc...(no debugging symbols found)...done.
(gdb) r
Starting program: /home/antek/dev/rust/rust-nightly-i686-unknown-linux-gnu/bin/rustc 
[...]

Program received signal SIGILL, Illegal instruction.
0xb5a50b80 in llvm::sys::MemoryFence() () from lib/librustc_llvm-4e7c5e5c.so
(gdb) x/16i $eip
=> 0xb5a50b80 <_ZN4llvm3sys11MemoryFenceEv>:    mfence 
   0xb5a50b83 <_ZN4llvm3sys11MemoryFenceEv+3>:  ret    
(gdb) bt
#0  0xb5a50b80 in llvm::sys::MemoryFence() () from lib/librustc_llvm-4e7c5e5c.so
#1  0xb5a0b80d in llvm::PassRegistry::getPassRegistry() () from lib/librustc_llvm-4e7c5e5c.so
#2  0xb5a0594c in llvm::PassNameParser::PassNameParser() () from lib/librustc_llvm-4e7c5e5c.so
#3  0xb59ecc0c in llvm::cl::list<llvm::PassInfo const*, bool, llvm::PassNameParser>::list<char [13], llvm::cl::desc, llvm::cl::OptionHidden>(char const (&) [13], llvm::cl::desc const&, llvm::cl::OptionHidden const&) () from lib/librustc_llvm-4e7c5e5c.so
#4  0xb59f6656 in _GLOBAL__sub_I_LegacyPassManager.cpp () from lib/librustc_llvm-4e7c5e5c.so
#5  0xb7feac40 in call_init.part () from /lib/ld-linux.so.2
#6  0xb7fead50 in _dl_init_internal () from /lib/ld-linux.so.2
#7  0xb7fdcbaf in _dl_start_user () from /lib/ld-linux.so.2

Also, for latest cargo itself, the error and stack trace is completely the same as in my previous post - SIGFPE inside lh_insert during libgit2 initialization in init_ssl.

[alexcrichton]

Ok, thanks for the update!

[MagaTailor]

It's safe to close this one as well.

[steveklabnik]

@petevine oh? does it not reproduce?

[MagaTailor]

I was thinking more along the lines of WONTFIX with appropriate build instructions.