OCaml SIGSEGV in invert_pointer_at() (OCaml Garbage Collector / Compaction)

[vicuna]

Original bug ID: 7431
Reporter: alexmarkley
Assigned to: @mshinwell
Status: closed (set by @mshinwell on 2016-12-22T07:16:05Z)
Resolution: not a bug
Priority: normal
Severity: crash
Platform: x86_64
OS: Linux
OS Version: Fedora 25
Version: 4.04.0
Category: runtime system and C interface

Bug description

When running the Unison File Synchronizer (a project written in OCaml: https://www.cis.upenn.edu/~bcpierce/unison/index.html ) against a large replica (1TB), I am encountering a showstopping segfault every single time.

• I have tried multiple versions of Unison, including stable versions which were working fine for me in the past and newer beta versions.

• I initially tried the official Fedora builds of OCaml (4.02.3-3) and when I was having no success with those, I removed them from my system and I built/installed OCaml 4.04.0 myself.

• I finally got a really good backtrace (included in this report) running OCaml 4.04 and Unison git master. As you can see, the segmentation fault occurred within the OCaml heap compaction portion of the garbage collection routine.

• It is worth noting that I never had this problem in earlier releases of Fedora, even with the same or earlier versions of OCaml and Unison. (I'm not sure what this implies, except that perhaps the bug is actually being triggered by a lower-level system component, like GCC or a system library.)

Related bug reports:

https://bugzilla.redhat.com/show_bug.cgi?id=1401759

bcpierce00/unison#48

Steps to reproduce

NOTE: These steps may only successfully reproduce the issue if the client is running Fedora 25 on x86_64, and if both OCaml and Unison were built on that machine.

1. Create a large, complicated dataset on the server for Unison to synchronize. Ideally this will be over 1TB in size and require over 2 hours to transfer.

2. Perform a synchronization between the client and the server, requiring the majority of the data to be transferred from the server to the client. (This mimics initial synchronization of a new hub/spoke node.)

3. Observe the client fails to synchronize the entire dataset. Client is terminated with SIGSEGV.

Additional information

===SNIP===
/home/alex/Temp/galculator-2.1.3/intltool-extract.in has already been transferred
/home/alex/Temp/galculator-2.1.3/intltool-merge.in has already been transferred
/home/alex/Temp/galculator-2.1.3/intltool-update.in has already been transferred
33% 100:25 ETA
Program received signal SIGSEGV, Segmentation fault.
0x00000000004ec76b in invert_pointer_at (p=p@entry=0x7fffd38c7b28) at compact.c:90
90 compact.c: No such file or directory.
(gdb) thread apply all bt full

Thread 1 (process 19298):
#0 0x00000000004ec76b in invert_pointer_at (p=p@entry=0x7fffd38c7b28) at compact.c:90
val = 140736742586384
hp = 0x7461705f77617264
q = 140736742586416
#1 0x00000000004ec90c in do_compaction () at compact.c:228
q =
i =
sz = 6
t =
infixes =
p = 0x7fffd38c7b10
ch = 0x7fffbf2fc000 "\363\273M"
chend = 0x7ffff09f1000 ""
#2 0x00000000004ecdea in caml_compact_heap () at compact.c:426
target_wsz =
live =
#3 0x00000000004ed24a in caml_compact_heap_maybe () at compact.c:547
fw =
fp = 170.748871
#4 0x00000000004daf4a in caml_major_collection_slice (howmuch=howmuch@entry=-1) at major_gc.c:785
p = 0.0043600637275738388
dp =
filt_p = 0.0043600637275738388
spend =
computed_work = 1522479
i =
#5 0x00000000004dbedf in caml_gc_dispatch () at minor_gc.c:463
trigger =
#6 0x00000000004dbf77 in caml_check_urgent_gc (extra_root=) at minor_gc.c:482
caml__frame = 0x0
caml__roots_extra_root = {next = 0x0, ntables = 1, nitems = 1, tables = {0x7fffffffd758, 0x7fffffffd870, 0x4dc96a <caml_alloc_shr+170>, 0x22, 0x7fff9d02f6b0}}
#7 0x00000000004dcfe5 in caml_alloc_string (len=65497) at alloc.c:103
result =
offset_index =
wosize = 8188
#8 0x000000000047205c in camlBytearray__sub_1422 () at /root/unison-git/src/bytearray.ml:63
No locals.
#9 0x0000000000447812 in camlTransfer__receiveRec_1568 () at /root/unison-git/src/transfer.ml:295
No locals.
#10 0x0000000000427cef in camlCopy__decompr_2936 () at /root/unison-git/src/transfer.ml:304
No locals.
#11 0x0000000000426bca in camlCopy__fun_3367 () at /root/unison-git/src/copy.ml:401
No locals.
#12 0x000000000046cc11 in camlUtil__convertUnixErrorsToExn_1955 () at /root/unison-git/src/ubase/util.ml:170
No locals.
#13 0x000000000043f46a in camlRemote__processStream_2291 () at /root/unison-git/src/remote.ml:664
No locals.
#14 0x000000000043fe26 in camlRemote__fun_4468 () at /root/unison-git/src/remote.ml:732
No locals.
#15 0x0000000000464e4d in camlLwt__apply_1225 () at /root/unison-git/src/lwt/lwt.ml:75
No locals.
#16 0x000000000046510e in camlLwt__fun_1451 () at /root/unison-git/src/lwt/lwt.ml:94
No locals.
#17 0x000000000048d101 in camlList__iter_1252 () at list.ml:77
No locals.
#18 0x0000000000464b2e in camlLwt__restart_1211 () at /root/unison-git/src/lwt/lwt.ml:31
No locals.
#19 0x000000000046182e in camlLwt_unix_impl__fun_2430 () at /root/unison-git/src/lwt/generic/lwt_unix_impl.ml:153
No locals.
#20 0x000000000048d101 in camlList__iter_1252 () at list.ml:77
No locals.
#21 0x0000000000461671 in camlLwt_unix_impl__run_1579 () at /root/unison-git/src/lwt/generic/lwt_unix_impl.ml:148
No locals.
#22 0x000000000040e80a in camlUitext__doTransport_1863 () at /root/unison-git/src/uitext.ml:490
No locals.
#23 0x000000000040f84e in camlUitext__doit_1922 () at /root/unison-git/src/uitext.ml:556
No locals.
#24 0x0000000000410034 in camlUitext__synchronizeOnce_1968 () at /root/unison-git/src/uitext.ml:718
No locals.
#25 0x000000000041094a in camlUitext__loop_2237 () at /root/unison-git/src/uitext.ml:788
No locals.
#26 0x0000000000410b4d in camlUitext__synchronizeUntilDone_2242 () at /root/unison-git/src/uitext.ml:810
No locals.
#27 0x0000000000410df7 in camlUitext__start_2249 () at /root/unison-git/src/uitext.ml:870
No locals.
#28 0x00000000004085fa in camlMain__Body_1550 () at /root/unison-git/src/main.ml:241
No locals.
#29 0x0000000000407a93 in camlLinktext__entry () at /root/unison-git/src/linktext.ml:19
No locals.
#30 0x0000000000404369 in caml_program ()
No symbol table info available.
#31 0x00000000004ef12e in caml_start_program ()
No symbol table info available.
#32 0x00000000004ef475 in caml_main (argv=0x7fffffffdca8) at startup.c:145
exe_name =
proc_self_exe = "/usr/local/bin/unison", '\000' <repeats 234 times>
res =
tos = 0 '\000'
#33 0x0000000000403c5c in main (argc=, argv=) at main.c:37
No locals.
(gdb)
===SNIP===

File attachments

• debug_runtime.txt
• vagrant_run_unoptimized.txt
• valgrind_run.txt

[vicuna]

Comment author: @gasche

Wow, thanks for the reproduction work. Have you contacted the Unison maintainers and Richard Jones, as the OCaml package for Fedora and pointed them at this issue report? They may have additional insights (I'm no GC expert but my impression is that a segfault in compaction can easily be caused by any other source of memory corruption in the process, not necessarily the GC code itself, it is rather that this GC phase touches a lot of memory).

[vicuna]

Comment author: alexmarkley

gasche, thanks for responding! Actually you're the first person to respond directly to any of my bug reports.

Since my troubleshooting has lead me deeper and deeper, I am thinking of adjusting my Redhat Bugzilla issue, so that it is a report against the ocaml package instead of a report against the unison package. After all, the ocaml package is probably used by quite a bit more people, so I would imagine it will get more exposure that way.

Also, I am quite aware how memory-intensive this kind of operation must be. In fact, since I'm running this on a brand-new laptop, I have been worried that this may actually be pointing to a hardware problem. However, so far, tools like memtest86+ have not reported any issues!

My other sneaking suspicion is that the compiler is producing broken code again. Fedora 25 ships with GCC 6.2.1, which is another major update to the compiler, and I'm not confident the dust has fully settled from all of this:

https://fedoraproject.org/wiki/Changes/GCC6

http://www.phoronix.com/scan.php?page=news_item&px=Fedora-GCC6-Rebuild-Results

I am currently in the process of building OCaml 4.04.0 with all optimizations disabled. Keeping my fingers crossed that this will "magically" fix the problem.

As always, any insight is greatly appreciated.

[vicuna]

Comment author: alexmarkley

I rebuilt OCaml 4.04.0 with all GCC optimizations disabled. (My methodology for doing so can be discussed in more depth if that is of interest.)

I was able to reproduce the segfault again, and interestingly it landed in a different spot:

[alex@obsidian ~]$ gdb /usr/local/bin/unison
GNU gdb (GDB) Fedora 7.12-29.fc25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/unison...done.
(gdb) run -batch -prefer ssh://alex@elbmin//home/alex alexHome
Starting program: /usr/local/bin/unison -batch -prefer ssh://alex@elbmin//home/alex alexHome
Unison 2.50.0 (ocaml 4.04.0): Contacting server...
Detaching after fork from child process 9140.
alex@elbmin's password:
Connected [//elbmin.malexmedia.net//home/alex -> //obsidian.malexmedia.net//home/alex]
Looking for changes
Waiting for changes from server
Reconciling changes
error tmprequest
Error in digesting /home/alex/tmprequest:
/home/alex/tmprequest: Permission denied
<---- new dir .bitcoin
local : absent
elbmin.ma... : new dir modified on 2016-12-06 at 7:53:14 size 110501785182 rwx------
<---- new dir .config
local : absent
elbmin.ma... : new dir modified on 2016-12-05 at 23:09:03 size 3326278 rwx------
<---- new dir .dbus-keyrings
local : absent
elbmin.ma... : new dir modified on 2015-11-04 at 20:21:26 size 71 rwx------
<---- new dir .dvdcss
local : absent
elbmin.ma... : new dir modified on 2014-05-05 at 21:01:28 size 9335 rwxr-x---
<---- new dir .gimp-2.8
===SNIP===
Shortcut: copied /home/alex/Documents/Temp/loresque/resources/crate3d-02.swiv140.png from local file /home/alex/.unison.Documents.7f78c97b4bc1aea68cc67a71dd2bde7f.unison.tmp/Temp/loresque_win32/resources/crate3d-02.swiv140.png
Shortcut: copied /home/alex/Documents/Temp/loresque/resources/crate3d-02.swiv142.png from local file /home/alex/.unison.Documents.7f78c97b4bc1aea68cc67a71dd2bde7f.unison.tmp/Temp/loresque_win32/resources/crate3d-02.swiv142.png
Shortcut: copied /home/alex/Documents/Temp/loresque/resources/crate3d-02.swiv144.png from local file /home/alex/.unison.Documents.7f78c97b4bc1aea68cc67a71dd2bde7f.unison.tmp/Temp/loresque_win32/resources/crate3d-02.swiv144.png
Shortcut: copied /home/alex/Documents/Temp/loresque/resources/crate3d-02.swiv146.png from local file /home/alex/.unison.Documents.7f78c97b4bc1aea68cc67a71dd2bde7f.unison.tmp/Temp/loresque_win32/resources/crate3d-02.swiv146.png
Shortcut: copied /home/alex/Documents/Temp/loresque/resources/crate3d-02.swiv148.png from local file /home/alex/.unison.Documents.7f78c97b4bc1aea68cc67a71dd2bde7f.unison.tmp/Temp/loresque_win32/resources/crate3d-02.swiv148.png
66% 65:19 ETA
Program received signal SIGSEGV, Segmentation fault.
0x000000000048d0f6 in camlList__iter_1252 () at list.ml:75
75 list.ml: No such file or directory.
(gdb) thread apply all bt full

Thread 1 (process 9136):
#0 0x000000000048d0f6 in camlList__iter_1252 () at list.ml:75
No locals.
#1 0x0000000000464b2e in camlLwt__restart_1211 () at /root/unison-git/src/lwt/lwt.ml:31
No locals.
#2 0x00000000004646db in camlLwt_util__leave_region_1278 () at /root/unison-git/src/lwt/lwt.ml:83
No locals.
#3 0x00000000004647a8 in camlLwt_util__fun_1442 () at /root/unison-git/src/lwt/lwt_util.ml:75
No locals.
#4 0x0000000000464e4d in camlLwt__apply_1225 () at /root/unison-git/src/lwt/lwt.ml:75
No locals.
#5 0x000000000046510e in camlLwt__fun_1451 () at /root/unison-git/src/lwt/lwt.ml:94
No locals.
#6 0x000000000048d101 in camlList__iter_1252 () at list.ml:77
No locals.
#7 0x0000000000464b2e in camlLwt__restart_1211 () at /root/unison-git/src/lwt/lwt.ml:31
No locals.
#8 0x0000000000460f92 in camlLwt_unix_impl__restart_threads_1278 () at /root/unison-git/src/lwt/lwt.ml:83
No locals.
#9 0x0000000000461661 in camlLwt_unix_impl__run_1579 () at /root/unison-git/src/lwt/generic/lwt_unix_impl.ml:147
No locals.
#10 0x000000000040e80a in camlUitext__doTransport_1863 () at /root/unison-git/src/uitext.ml:490
No locals.
#11 0x000000000040f84e in camlUitext__doit_1922 () at /root/unison-git/src/uitext.ml:556
No locals.
#12 0x0000000000410034 in camlUitext__synchronizeOnce_1968 () at /root/unison-git/src/uitext.ml:718
No locals.
#13 0x000000000041094a in camlUitext__loop_2237 () at /root/unison-git/src/uitext.ml:788
No locals.
#14 0x0000000000410b4d in camlUitext__synchronizeUntilDone_2242 () at /root/unison-git/src/uitext.ml:810
No locals.
#15 0x0000000000410df7 in camlUitext__start_2249 () at /root/unison-git/src/uitext.ml:870
No locals.
#16 0x00000000004085fa in camlMain__Body_1550 () at /root/unison-git/src/main.ml:241
No locals.
#17 0x0000000000407a93 in camlLinktext__entry () at /root/unison-git/src/linktext.ml:19
No locals.
#18 0x0000000000404369 in caml_program ()
No symbol table info available.
#19 0x00000000004ef12e in caml_start_program ()
No symbol table info available.
#20 0x00000000004ef475 in caml_main (argv=0x7fffffffdc98) at startup.c:145
exe_name =
proc_self_exe = "/usr/local/bin/unison", '\000' <repeats 234 times>
res =
tos = 0 '\000'
#21 0x0000000000403c5c in main (argc=, argv=) at main.c:37
No locals.
(gdb)

[vicuna]

Comment author: @mshinwell

Can you confirm that when you disabled GCC optimisation, for building OCaml, that you did not remove the "-fno-strict-aliasing -fwrapv" compiler flags? These are needed to ensure correct compilation (and the former may be even more important for GCC 6).

In the OCaml source tree without GCC optimisation enabled please enter the testsuite/ directory and run "make all". Do any of the tests fail?

[vicuna]

Comment author: alexmarkley

@shinwell, thanks for your attention to this issue!

My methodology for removing GCC optimization was to remove -O flags only. The following files and variables were modified:

• configure (bytecccompopts)
• asmrun/Makefile (CFLAGS)
• otherlibs/Makefile (OPTCOMPFLAGS)
• otherlibs/dynlink/Makefile (OPTCOMPFLAGS)
• otherlibs/systhreads/Makefile (OPTCOMPFLAGS)
• otherlibs/systhreads/Makefile.nt (OPTCOMPFLAGS)
• stdlib/Makefile.shared (OPTCOMPFLAGS)

I ran the test suite as you suggested. Both with GCC optimizations and without, the results were exactly the same:

Summary:
635 tests passed
12 tests skipped
0 tests failed
0 unexpected errors
647 tests considered

List of skipped tests:
tests/lib-dynlink-csharp
tests/unwind
tests/asmcomp/static_float_array_flambda_opaque
tests/asmcomp/is_static_flambda
tests/asmcomp/unrolling_flambda
tests/lib-bigarray-2
tests/asmcomp/static_float_array_flambda
tests/asmcomp/unrolling_flambda2
tests/manual-intf-c

[vicuna]

Comment author: alexmarkley

Also of note for this issue, one of the Unison developers suggested that the issue may be a stack overflow: bcpierce00/unison#48 (comment)

However, I carefully tested for and eliminated that possibility using a sequence of tests described here: bcpierce00/unison#48 (comment)

I am still open to any suggestion for troubleshooting steps and/or feedback on my troubleshooting so far.

Steps I am currently pursuing:

• See if I can get Unison running via a non-native compilation option. (I'm very unfamiliar with OCaml, so this will require some research on my part.)

• Repurpose some computer hardware and attempt to reproduce this issue on another computer. (I have run every hardware diagnostic I can think of, so I am fairly confident there are no hardware issues with my new laptop, but reproducing this issue on separate hardware would be one way to be sure.)

[vicuna]

Comment author: @gasche

I don't have experience building unison but their makefiles appear to have a NATIVE variable that you could set to 'false' to get a bytecode executable (DEBUGGING=true might also help diagnose a problem, although it is mostly useful to generate OCaml stack traces using OCAMLRUNPARAM=b, I'm not sure it makes a difference when you have a core dump).

[vicuna]

Comment author: @mshinwell

I'm making a few investigations into this problem, but in the meantime, I think it would be useful to try to rule out the C compiler change.

Can you try to build an old version of GCC and use that? I believe 4.4.7 is OK for OCaml compilation, but I'm not sure if it will build on your system. However try to stick to a 4.x version if you can.

This should be fairly easy. You need dev packages for GMP and MPFR which there should be Fedora packages for. Then download the source and make a build directory which must be outside of the extracted source tree. From the build directory I think you do:

../gcc-4.4.7/configure --prefix=/path/to/install
make bootstrap
make install

Then just add /path/to/install to your path and rebuild OCaml and Unison. Does it still fail?

[vicuna]

Comment author: @mshinwell

I suspect this is not the source of the problem, but it's quite hard to think about so just in case:

In src/bytearray_stubs.c line 32, in the Unison tree, there is this code:

char *src = String_val(s) + Int_val(i);

I think this is wrong. [i] is the offset into a string, which might be large; in particular, it might be so large that the result of the "(int)" cast inside "Int_val" might be a negative number (since on your platform "int" is probably only 32 bits wide, and I imagine your platform is 64 bit). It should be "Long_val" instead.

[vicuna]

Comment author: alexmarkley

@gasche, thanks for the find in the Unison Makefile. I'll give that a try if/when I move on to attempting non-native OCaml execution.

@shinwell, regarding the Int_val vs. Long_val, that's a great catch. I tried re-building Unison with Long_val on line 32 of src/bytearray_stubs.c and unfortunately it did not fix the problem. (Regardless, if you think Long_val is the more appropriate way to go, I will probably open a pull request against the Unison project and see if they accept it.)

Regarding building an older compiler, GCC 4.4.7 would not compile for me. However, GCC 5.4.0 DID compile okay, so I got that installed in an isolated corner of my system and I am in the process of attempting to reproduce the bug on Fedora 25 on GCC 5.

In addition to all of this, I did manage to build another system with enough disk space to test this issue. It's another x86_64 machine, but for my first test, it was running Ubuntu 16.04.

Interestingly, I was able to successfully run the initial Unison synchronization (the entire 1TB+ replica) to the Ubuntu machine with Ocaml 4.04.0 and Unison git master. I was able to do this with GCC 5.4 (included with Ubuntu 16.04) AND GCC 6.2 (from the toolchain test PPA: https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=xenial ).

So I think I was able to confirm that nothing about the data set itself (nor the size of the data set) was causing the Unison codebase to segfault.

My next step is to more thoroughly examine the GCC bug option by attempting to eliminate the issue on my laptop running Fedora 25 by using GCC 5.4.

In parallel, I am going to continue to attempt to reproduce the issue on the new desktop by installing Fedora 25 and using GCC 6.2, which would reduce or eliminate the likelihood of a hardware bug.

[vicuna]

Comment author: @mshinwell

I do think Long_val is correct in that circumstance, yes.

[vicuna]

Comment author: alexmarkley

Another update here:

I was able to reproduce the bug on completely separate hardware. A desktop machine running Fedora 25 segfaulted while synchronizing the replica this afternoon.

Also, I did build GCC 5 on my laptop and used that to build OCaml 4.04.0 (and then used that to build Unison git master). Unfortunately, this did not make the problem go away.

What does this mean? Well, while I'm more sure than ever now that I'm dealing with a software bug (as opposed to a hardware bug), I'm now starting to think that the issue is deeper than I previously thought.

ldd says that the native unison binary requires the following shared libraries:

linux-vdso.so.1 (0x00007ffd31d90000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f6501a73000)
libm.so.6 => /lib64/libm.so.6 (0x00007f650176a000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6501566000)
libc.so.6 => /lib64/libc.so.6 (0x00007f65011a0000)
/lib64/ld-linux-x86-64.so.2 (0x000055d772c81000)

Presumably the issue could be:

• A bug in one or more of these libraries?
• A compiler bug causing one or more of these libraries to misbehave, introducing memory corruption.
• A kernel bug.
• Intel is out to get me.
• Faeries.

As always, feedback is welcome.

[vicuna]

Comment author: @mshinwell

You said you'd run this successfully on earlier versions of Fedora. Can you tell us the most recent version of Fedora on which this works? (e.g. 24 maybe?)

[vicuna]

Comment author: @chambart

If it is a GC or binding bug, it might be possible to narrow the search space by using the debug runtime.
Set the environment variable OCAMLPARAM=runtime-variant=d,_ before compiling

[vicuna]

Comment author: @gasche

(That is, compiling the compiler distribution with the --with-debug-runtime configure option.)