firedrake-check ==> SIGSEGV #4647
-
|
Hello, Firedrake developers. I have been struggling to get firedrake working on an Ubuntu 22.04 machine. For context, I do not have root on this machine, and had to build petsc with many "--download-*" options. petsc makes and checks without errors now. However, after the pip installation of firedrake (which is also uneventful), 'firedrake-check' results in the following error:
I tried editing firedrake check to see whether any one of those tests is responsible for the segfault, but even when I comment out all but one of them the result is the same, irrespective of the test. I tried running under a debugger, but unfortunately the subprocess command is opaque, and I can't see what is provoking the segfault. I attach the petsc configure.log file. Happy to try anything that you might suggest. Thanks, |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 10 replies
-
|
Can you try running:
The gdb command should show us where the segfault is happening. Note that |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the response. I only got to (1):
|
Beta Was this translation helpful? Give feedback.
-
|
Interesting. Can you try setting the |
Beta Was this translation helpful? Give feedback.
-
|
Sure:
So it appears to be something about the PETSC build that is responsible for the error, despite the fact that that build passed "make check". Another fact that I should mention: I had no problems building Firedrake on an Ubuntu 24.04 machine: the recipe on the "Installing Firedrake" page worked the first time. I do have root on that machine, and I was able to install all the dependencies, so I did not have to ask PETSC to download anything. Unfortunately, that's my laptop, and it's not suitable for the kinds of runs that I need. Out of curiosity, does the standard installation recipe work on Ubuntu 22.04 machines when all the dependencies are installed? I could ask our admins to do the honors if that is the case. |
Beta Was this translation helpful? Give feedback.
-
You could try running this inside gdb to get a traceback: Also I don't know if it needs to be
Users have definitely managed to run on Ubuntu 22.04. I think that everything should just work but I cannot guarantee that. However it looks like the issue is to do with the MPI configuration of the new machine, so doing that may not end up fixing the bug. Seeing a proper traceback will tell us more. |
Beta Was this translation helpful? Give feedback.
-
|
OK, good idea. Here's what happens:
|
Beta Was this translation helpful? Give feedback.
-
|
It is segfaulting inside an MPI instruction. Do you reproduce the error with |
Beta Was this translation helpful? Give feedback.
-
|
No, 'python -c "from mpi4py import MPI" ' returns without errors. |
Beta Was this translation helpful? Give feedback.
-
|
I wonder if the system MPI is simply buggy and in Firedrake we are hitting it harder than PETSc is and causing the break. You could try passing |
Beta Was this translation helpful? Give feedback.
-
|
Well, we have a new error then, which may in fact represent progress. Configured PETSC with:
(The --download-scalapack and --download-cmake were required in consequence of the --download-openmpi). PETSC make all and make check are uneventful. Then
...much installation noise ensues, eventually followed by
Check time:
So the segfault is gone, replaced by a
|
Beta Was this translation helpful? Give feedback.
-
|
Nice. I have a guess as to what is causing this:
To fix I think you just have to set |
Beta Was this translation helpful? Give feedback.
-
|
More progress! Now the serial smoke tests pass, but all the parallel ones fail. With apologies for the extremely verbose output...:
|
Beta Was this translation helpful? Give feedback.
-
|
Pretty sure I know the fix here too. Run |
Beta Was this translation helpful? Give feedback.
-
|
That worked! Thanks! I'll write a suggested answer anon. However, the thing I still need to understand is, how do I make sure that my own code invokes firedrake with the correct mpiexec path? Is there an environment variable that I need to set? |
Beta Was this translation helpful? Give feedback.
-
|
Excellent! I'd recommend adding I will at some point write a guide to using a non-system MPI so this is easier for people in future. |
Beta Was this translation helpful? Give feedback.
-
|
Here is the bottom line (thanks, @connorjward): The problem is installing Firedrake on a Ubuntu machine where it is not possible to install all the required package dependencies described in the Install page. The following recipe works on a Ubuntu 22.04 machine. First, download PETSC as described in the Firedrake Install page. Now configure PETSC, with
This pretty much asks PETSC to bypass essentially all the system-installed packages, and instead download and build those dependencies. Note that openmpi is among the downloaded dependencies. After the lengthy configuration, 'make' and 'make check' as instructed by the build scripts, and, assuming the 'make check' passed, 'cd ..' to go back to the parent directory. Now perform the step from the Install page
This sets a few environment variables, but two of them are no longer valid. So type
Also, you need to tell the loader where to find the libraries, and firedrake where to find binaries, so that you don't link to the system libraries instead of the ones that PETSC just built, or execute the wrong executables. So
(It might be worthwhile setting the $PETSC_DIR, $PETSC_ARCH, $LD_LIBRARY_PATH, and $PATH environment variables in your shell start-up scripts). Now you can follow the remaining firedrake Install recipe, creating the venv, doing the pip install, and the firedrake-check step. |
Beta Was this translation helpful? Give feedback.
Here is the bottom line (thanks, @connorjward):
The problem is installing Firedrake on a Ubuntu machine where it is not possible to install all the required package dependencies described in the Install page. The following recipe works on a Ubuntu 22.04 machine.
First, download PETSC as described in the Firedrake Install page. Now configure PETSC, with