Enhancements for Euler-Lagrange bubble/particle model

[wilfonba]

Description

This PR adds a few enhancements to the EL model in MFC

• Moving EL particles/bubbles with halo exchange for multiple MPI ranks using both adaptive and non-adaptive time stepping. The current implementation is naive in that it is not necessarily memory-efficient, but for small numbers of particles relative to grid cells, this should not be a significant issue. Future PRs can improve the memory footprint.
• EL particles/bubbles can now interact with immersed boundaries
• EL particles/bubbles can now interact with solid domain boundaries
• I/O for EL particles/bubbles is improved to mitigate the need for a reduction to the root process (ie, I/O is actually parallel)
• The particles/bubbles are now included in the silo files for visualization with interactive tools
• The user can now select which particle/bubble quantities to write to the .csv output or .silo output to reduce file sizes

Type of change

• New feature (non-breaking change which adds functionality)

Scope

• This PR comprises a set of related changes with a common goal (to improve the EL model in MFC)

How Has This Been Tested?

I'll need to add a lot here...

Checklist

• I have added comments for the new code
• I added Doxygen docstrings to the new code
• I have made corresponding changes to the documentation (docs/)
• I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• I have added example cases in examples/ that demonstrate my new feature performing as expected.
  They run to completion and demonstrate "interesting physics"
• I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• Ran a Rocprof Systems profile using ./mfc.sh run XXXX --gpu -t simulation --rsys --hip-trace, and have attached the output file and plain text results to this PR.
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

[codecov]

Codecov Report

❌ Patch coverage is 10.47694% with 1145 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.23%. Comparing base (fe87fc5) to head (9b70f9d).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_bubbles_EL.fpp	10.00%	368 Missing and 10 partials ⚠️
src/simulation/m_mpi_proxy.fpp	0.33%	298 Missing ⚠️
src/post_process/m_data_output.fpp	0.00%	296 Missing and 1 partial ⚠️
src/simulation/m_bubbles_EL_kernels.fpp	2.12%	138 Missing ⚠️
src/simulation/m_bubbles.fpp	15.38%	9 Missing and 2 partials ⚠️
src/common/m_mpi_common.fpp	65.21%	7 Missing and 1 partial ⚠️
src/simulation/m_ibm.fpp	50.00%	5 Missing ⚠️
src/simulation/m_time_steppers.fpp	0.00%	1 Missing and 2 partials ⚠️
src/post_process/m_start_up.f90	33.33%	2 Missing ⚠️
src/simulation/m_rhs.fpp	60.00%	2 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #991      +/-   ##
==========================================
- Coverage   40.92%   39.23%   -1.69%     
==========================================
  Files          70       70              
  Lines       20299    21282     +983     
  Branches     2521     2682     +161     
==========================================
+ Hits         8307     8350      +43     
- Misses      10454    11399     +945     
+ Partials     1538     1533       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.