diff --git a/docs/building-apps/building-static-apps.rst b/docs/building-apps/building-static-apps.rst index 7536087fb76..9ded85c1330 100644 --- a/docs/building-apps/building-static-apps.rst +++ b/docs/building-apps/building-static-apps.rst @@ -1,3 +1,5 @@ +.. _label-building-fully-static-apps: + Building fully-static MPI applications ====================================== diff --git a/docs/building-apps/extracting-wrapper-flags.rst b/docs/building-apps/extracting-wrapper-flags.rst index 2c04fbb52b8..538e2a8a7b3 100644 --- a/docs/building-apps/extracting-wrapper-flags.rst +++ b/docs/building-apps/extracting-wrapper-flags.rst @@ -23,7 +23,7 @@ that will show what commands would have been invoked. shell$ mpicc --showme .. note:: If you pass ``--showme`` *and additional command line - parameters* to the wrapper compiler, sure to *also* pass in a + parameters* to the wrapper compiler, be sure to *also* pass in a filename. Otherwise, the ``--showme`` functionality will not display output as expected. diff --git a/docs/building-apps/quickstart.rst b/docs/building-apps/quickstart.rst index c4bd5dfd6c2..9d1637e2499 100644 --- a/docs/building-apps/quickstart.rst +++ b/docs/building-apps/quickstart.rst @@ -17,16 +17,20 @@ probably work in many environments. Open MPI provides "wrapper" compilers that should be used for compiling MPI and OpenSHMEM applications: -+---------+--------------------------+ -| C | ``mpicc``, ``oshcc`` | -+---------+--------------------------+ -| C++ | ``mpiCC``, ``oshCC`` (or | -| | ``mpic++`` if your | -| | filesystem is | -| | case-insensitive) | -+---------+--------------------------+ -| Fortran | ``mpifort``, ``oshfort`` | -+---------+--------------------------+ ++---------+-----------------------------------+ +| C | :ref:`mpicc(1) `, | +| | :ref:`oshcc(1) ` | ++---------+-----------------------------------+ +| C++ | :ref:`mpic++(1) `, | +| | :ref:`oshc++(1) ` | +| | (and :ref:`mpiCC(1) ` | +| | and :ref:`oshCC(1) ` | +| | if you filesystem is | +| | case-sensitive) | ++---------+-----------------------------------+ +| Fortran | :ref:`mpifort(1) `, | +| | :ref:`oshfort(1) ` | ++---------+-----------------------------------+ .. caution:: The legacy names ``mpif77`` and ``mpif90`` still exist, and are simply symbolic links to the ``mpifort`` wrapper diff --git a/docs/contributing.rst b/docs/contributing.rst index eb6f2f6126c..fc1422eccf0 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -25,6 +25,12 @@ There are many ways to contribute. Here are a few: #. Submit bug fixes to the main code base. * Awesome! Open a new GitHub pull request with the patch. + + * Please submit bug fixes / new features on the ``main`` branch + first, and then port them to the relevant release branch(es) + after they have been accepted on ``main``. :ref:`See this + section for more information `. + * Ensure the PR description clearly describes the problem and solution. If there is an existing GitHub issue open describing this bug, please include it in the description so we can track @@ -50,6 +56,18 @@ There are many ways to contribute. Here are a few: * Be sure to see the :ref:`Open source contributions ` section, below. +#. Submit fixes and/or entirely new content to this documentation. + + * Docs are great! We always need help with documentation. + * These docs are authored using ReStructured Text markup with the + Sphinx rendering tool under the ``docs/`` directory in the + repository. This means you can submit a pull request with your + docs updates, just like you would for any Open MPI code contribution. + * :ref:`See this section ` and + :ref:`also this section ` for + information about how to install Sphinx locally and some + high-level logistical instructions on how to write these docs. + #. Provide testing resources: #. For Github Pull Request Continuous Integration (CI) diff --git a/docs/developers/git-clone.rst b/docs/developers/git-clone.rst deleted file mode 100644 index ccb79abd9ba..00000000000 --- a/docs/developers/git-clone.rst +++ /dev/null @@ -1,20 +0,0 @@ -Obtaining a Git clone -===================== - -Open MPI's Git repositories are `hosted at GitHub -`_. - -#. First, you will need a Git client. We recommend getting the latest - version available. If you do not have the command ``git`` in your - path, you will likely need to download and install Git. -#. `ompi `_ is the main Open MPI - repository where most active development is done. Git clone this - repository. Note that the use of the ``--recursive`` CLI option is - necessary because Open MPI uses Git submodules:: - - shell$ git clone --recursive https://github.com/open-mpi/ompi.git - -Note that Git is natively capable of using many forms of web -proxies. If your network setup requires the user of a web proxy, -`consult the Git documentation for more details -`_. diff --git a/docs/developers/git-github.rst b/docs/developers/git-github.rst new file mode 100644 index 00000000000..f9d5c75671a --- /dev/null +++ b/docs/developers/git-github.rst @@ -0,0 +1,196 @@ +GitHub, Git, and related topics +=============================== + +GitHub +------ + +Open MPI's Git repositories are `hosted at GitHub +`_. + +#. First, you will need a Git client. We recommend getting the latest + version available. If you do not have the command ``git`` in your + path, you will likely need to download and install Git. +#. `ompi `_ is the main Open MPI + repository where most active development is done. Git clone this + repository. Note that the use of the ``--recursive`` CLI option is + necessary because Open MPI uses Git submodules:: + + shell$ git clone --recursive https://github.com/open-mpi/ompi.git + +Note that Git is natively capable of using many forms of web +proxies. If your network setup requires the user of a web proxy, +`consult the Git documentation for more details +`_. + +Git commits: open source / contributor's declaration +---------------------------------------------------- + +In order to remain open source, all new commits to the Open MPI +repository must include a ``Signed-off-by:`` line, indicating the +submitter's agreement to the :ref:`Open MPI Contributor's Declaration +`. + +.. tip:: You can use the ``-s`` option to ``git commit`` to + automatically add the ``Signed-off-by:`` line to your commit + message. + +.. _git-github-branch-scheme-label: + +Git branch scheme +----------------- + +Generally, Open MPI has two types of branches in its Git repository: + +#. ``main``: + + * All active development occurs on the ``main`` branch (new features, + bug fixes, etc.). + +#. Release branches of the form ``vMAJOR.MINOR.x`` (e.g., ``v4.0.x``, + ``v4.1.x``, ``v5.0.x``). + + * The ``.x`` suffix indicates that this branch is used to create + all releases in the Open MPI vMAJOR.MINOR series. + * Periodically, the Open MPI community will make a new release + branch, typically from ``main``. + * A Git tag of the form ``vMAJOR.MINOR.RELEASE`` is used to + indicate the specific commit on a release branch from where + official Open MPI release tarball was created (e.g., ``v4.1.0``, + ``v4.1.1``, ``v4.1.2``, etc.). + +Once a bug is fixed or a new feature is implemented on ``main``, it is +cherry-picked over to the relevant release branch(es). + +.. attention:: It may seem odd to some, but the Open MPI community + development model does *not* PR bug fixes or new + features directly to release branches. Instead, + initial bug-fix / feature PRs are generally first made + to ``main``. + + This helps us ensure that future releases (with + ``main`` as a Git ancestor) will contain the bug fix / + feature. + +For example: + +.. code:: sh + + shell$ git checkout main + shell$ git pull --rebase + shell$ git checkout pr/bug-fix + + # ... make changes / fix a bug / etc. ... + + shell$ git add ... + shell$ git commit -s ... + shell$ git push myfork + +At this point, you go create a PR from your fork's ``pr/bug-fix`` +branch to the Open MPI community GitHub repo ``main`` branch. Work +with the community to get the PR completed and merged. Then you can +open a new PR to cherry pick the Git commits from that bug fix to each +of the relevant release branches. + +Depending on how far the release branch has diverged from ``main``, +there may be some porting effort involved in the cherry-pick. + +For example, if your bug fix on ``main`` is comprised of a single Git +commit hash ``123abc``: + +.. code:: sh + + # Fetch all upstream git activity, including the merge of the "main" PR. + shell$ get fetch --all + + # Check out the target release branch, and advance to the most recent commit. + shell$ git checkout v5.0.x + shell$ git pull --rebase + + # Make a branch for your bug fix + shell$ git checkout -b pr/v5.0.x/bug-fix + # Cherry pick the commit from the "main" branch + shell$ git cherry-pick -x 123abc + # Push to your fork + shell$ git push myfork + +The Open MPI development community *requires* adding the following +line to the commit message of cherry-picked commits on release +branches: + +.. code:: text + + (cherry picked from commit [git_hash_of_original_commit]) + +.. note:: Note the use of the ``-x`` option to ``git cherry-pick``. + This option automatically adds the ``(cherry picked from + ...)`` line to your commit message. + +.. admonition:: Rationale + :class: tip + + Git does not actually store any meta data about Git cherry-picks in + the commit. Having a standardized text line containing the source + Git commit hash in the commit messages helps the Open MPI + development community track where commits came from on release + branches, and therefore allows us to check whether all relevant + commits have been ported to a given release branch. + +Once your commits are ready and pushed up to your fork, make a PR to +the target release branch. + +.. warning:: A GitHub PR CI job checks all commits on release branches + for the ``(cherry picked from...)`` line. It will also + ensure that the Git hash cited in that line actually + exists on the ``main`` branch. + + This check ensures that commits are not made to release + branches before their corresponding ``main`` PR was + merged. + +All this being said, sometimes there is a need for a non-cherry-picked +commit on a release branch. E.g., sometimes a release branch has +diverged so much that the bug no longer exists on ``main``. It would +therefore not make sense |mdash| or even be impossible |mdash| to +commit the bug fix in question to ``main``. + +In such cases, make a regular PR to the target branch (with commits +that do *not* include ``(cherry picked from ...)`` lines). In the PR +description, add a line with the following token: + +.. code:: text + + bot:notacherrypick + +This tells the GitHub CI job that this PR contains commits that are +not cherry-picked from ``main``. + +.. warning:: ``bot:notacherrypick`` should only be used when + absolutely necessary. It is not a license to avoid + the process of PR'ing to ``main`` first. + +CI (testing) +------------ + +The Open MPI community generally runs two flavors of testing: + +#. A bunch of tests on each PR (Continuous Integration / CI). These + tests are a mixture of GitHub Actions and other CI systems (e.g., + Jenkins). Examples include (but are not limited to): + + * Check each Git commit for bozo email addresses + * Check that each Git commit contains a ``Signed-off-by`` line + * Check that commits on release branches contain a cherry-pick + notice + * Build and publish the docs + * Build Open MPI in a variety of environments and run sanity tests + with that installation + +#. Daily testing via the MPI Testing Tool (MTT). + + * These are generally tests that take much longer to run than on a + per-PR basis. `A "nightly snapshot" tarball + `_ is created for ``main`` and + each relevant release branch. + * MTT tests are run with this snapshot tarball so that all + organizations are testing with the same snapshots. + * `Results are available here `_. diff --git a/docs/developers/index.rst b/docs/developers/index.rst index 83de74048b9..0bc31dbd3e8 100644 --- a/docs/developers/index.rst +++ b/docs/developers/index.rst @@ -12,7 +12,7 @@ probably don't need to read this section. :maxdepth: 1 prerequisites - git-clone + git-github compiler-pickyness autogen building-open-mpi diff --git a/docs/developers/prerequisites.rst b/docs/developers/prerequisites.rst index b5638887409..bcca4b1b9ef 100644 --- a/docs/developers/prerequisites.rst +++ b/docs/developers/prerequisites.rst @@ -47,6 +47,14 @@ build them manually, see the :ref:`how to build and install GNU Autotools section ` for much more detail. +Perl +---- + +Open MPI still uses Perl for a few of its build scripts (most notably, +``autogen.pl``). + +Generally speaking, any recent-ish release of Perl 5 should be +sufficient to correctly execute Open MPI's Perl scripts. Flex ---- @@ -80,12 +88,12 @@ MacPorts on MacOS), see `the Flex Github repository `_. -Sphinx ------- +Sphinx (and therefore Python) +----------------------------- -`Sphinx `_ is used to generate both the -HTML version of the documentation (that you are reading right now) and -the nroff man pages. +`Sphinx `_ is a Python-based tool used to +generate both the HTML version of the documentation (that you are +reading right now) and the nroff man pages. Official Open MPI distribution tarballs contain pre-built HTML documentation and man pages. This means that -- similar to the GNU diff --git a/docs/developers/source-code.rst b/docs/developers/source-code.rst index 2e55c2ad74f..134dc95d14c 100644 --- a/docs/developers/source-code.rst +++ b/docs/developers/source-code.rst @@ -79,18 +79,23 @@ C / C++ * In Open MPI, we **always** ``#define`` a logical macro to be either 0 or 1 -- we never ``#undef`` it. * The reason for this is defensive programming: if you are only - checking if a preprocessor macro is defined (via ``#ifdef`` or - ``"#if`` defined(FOO)"), you will get no warning when compiling if + checking if a preprocessor macro is defined (via ``#ifdef FOO`` or + ``#if defined(FOO)``), you will get no warning when compiling if you accidentally misspell the macro name. However, if you use the logic test ``#if FOO`` with an undefined macro (e.g., because you misspelled it), you'll get a compiler warning or error. - Misspelled macro names can be tremendously difficult to find when - they are buried in thousands of lines of code, so we will take all - the help from the preprocessor/compiler that we can get! + + .. admonition:: Rationale + :class: tip + + Misspelled macro names can be tremendously difficult to find + when they are buried in thousands of lines of code; we will + take all the help from the preprocessor/compiler that we can + get! .. code-block:: c - /* Gnu Way - you will get no warning from the compiler if you + /* GNU Way - you will get no warning from the compiler if you misspell "FOO"; the test will simply be false */ #ifdef FOO ... @@ -155,18 +160,6 @@ and try to use a similar style. # This is good if test "$foo" = "bar"; then -* Do not use the ``-a`` or ``-o`` operators for ``test`` |mdash| this - has caused portability problems with ``test(1)`` on BSD systems. - Instead, use the ``&&`` or ``||`` shell operators. - - .. code-block:: sh - - # This is bad - if test "$foo" = "bar" -a "$baz" = "yow"; then - - # This is good - if test "$foo" = "bar" && test "$baz" = "yow"; then - m4 ^^^ diff --git a/docs/developers/sphinx.rst b/docs/developers/sphinx.rst index a57c3f35f3c..13eded4af5c 100644 --- a/docs/developers/sphinx.rst +++ b/docs/developers/sphinx.rst @@ -1,11 +1,34 @@ .. _developers-installing-sphinx-label: +Installing and running Sphinx (building the Open MPI docs) +========================================================== + +As with all content in the Developer's Guide, this section is only +relevant for developers who work in the Open MPI code base itself. +End users who install a binary Open MPI package or build from an +official Open MPI distribution tarball do not need to have Sphinx +installed. + +Installing Python +----------------- + +The `Sphinx tool `_ is written in Python, +and therefore needs to have Python available. As of late 2022, Sphinx +requires Python >= v3.7. + +This documentation does not contain detailed instructions for +installing a Python version sufficient for using Sphinx. Consult your +local OS documentation for how to obtain Python >= v3.7, or search the +internet for further information. + Installing Sphinx -================= +----------------- -The Sphinx documentation recommends installing Sphinx (and its -required Python dependencies) via ``pip``, which typically requires -connectivity to the general internet. +`The Sphinx documentation +`_ +recommends installing Sphinx (and its required Python dependencies) +via ``pip``, which typically requires connectivity to the general +internet. .. note:: If you are running on MacOS, you may be tempted to use Homebrew or MacPorts to install Sphinx. The Sphinx documentation @@ -16,7 +39,7 @@ There are three general ways to install Sphinx; you only need one of them. Install Sphinx in a Python virtual environment ----------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The preferred method of installing Sphinx for Open MPI documentation development is to install Sphinx in a Python virtual environment. @@ -51,7 +74,7 @@ environment. Sphinx in your ``PATH`` *before* running ``configure``. Install Sphinx globally ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ If Python virtual environments are not desirable on your system, you can install Sphinx globally on your system (you may need to run with @@ -75,7 +98,7 @@ your ``PATH``, then you need to add it to your ``PATH``. Install Sphinx locally ----------------------- +^^^^^^^^^^^^^^^^^^^^^^ If you cannot or do not want to install Sphinx globally on your system, the following will install Sphinx somewhere under your @@ -102,3 +125,116 @@ installed and add it to your ``PATH``. .. important:: You will need to ensure that Sphinx is in your ``PATH`` *before* running ``configure``. + +Running Sphinx +-------------- + +Open MPI's build environment is setup to invoke Sphinx automatically; +you should not need to invoke Sphinx manually. + +.. important:: You will need to ensure that Sphinx is in your ``PATH`` + *before* running ``configure``. + +As long as ``configure`` found Sphinx, ``make`` will invoke Sphinx to +build the documentation. You can also run ``make`` directly in the +``docs/`` directory to build *just* the docs and skip building the +rest of the Open MPI software. This can be a huge time-saver when +iteratively writing, rendering, and viewing/proofing documentation. + +.. note:: The fully-built HTML and man page docs are included in + official Open MPI distribution tarballs. Meaning: if you + download an Open MPI tarball from + https://www.open-mpi.org/software/ompi (version v5.0.0 or + later), the pre-built HTML and man page files are included + in the tarball. + + Sphinx is a requirement for *developers* who want to build + the Open MPI docs. End users do *not* need to have Sphinx + available to build Open MPI or have its docs installed from + an official distribution tarball. + +Sphinx execution time +^^^^^^^^^^^^^^^^^^^^^ + +The first time you invoke Sphinx on a clean tree, it takes a little +time to render all the docs. + +However, Sphinx is stateful: subsequent runs can be significantly +faster because Sphinx will only re-render HTML files that have +changes. This is an enormous time saver for Open MPI (e.g., if you +are iterating over writing the docs and running ``make`` to see how +they rendered in HTML). + +.. caution:: Sphinx is only *somewhat* smart in its partial + re-rendering. If you change a title in an RST file, for + example, Sphinx will (by default) only re-render *that* + page. The Tables of Contents / left hand navigation on + other pages may not be updated. + + You can always force a full re-render via: + + .. code:: sh + + shell$ cd docs + shell$ rm -rf _build + shell$ make + +Checking Sphinx HTML links +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``linkcheck`` ``make`` target will invoke Sphinx's functionality +to check all the external links in the documentation:: + +.. code:: sh + + shell$ cd docs + shell$ make linkcheck + +.. important:: You will need to be on a computer that has good access + to the internet when running this command. + +Viewing docs locally +^^^^^^^^^^^^^^^^^^^^ + +Once you have built the docs in your local Git clone, you can view +them locally in the build tree: + +#. Open ``docs/_build/html/index.html`` in a browser to view the HTML + docs. For example, on MacOS, the following command opens the build + tree docs in the default web browser: + + .. code:: sh + + shell$ open docs/_build/html/index.html + +#. Use the ``man`` command to view the Nroff files in + ``docs/_build/man`` (you may need to use an absolute or relative + filename to prevent ``man`` from using its search paths). For + example: + + .. code:: sh + + shell$ cd docs/_build/man + shell$ man ./MPI_Send.3 + +Alternatively, you can view these files in their installed locations +after running ``make install``: + +#. The HTML docs are installed (by default) to + ``$prefix/share/doc/openmpi/html``. You can use a web browser to + open the ``index.html`` in that directory to view the docs locally. + For example, on MacOS, the following command opens the installed + docs in the default web browser: + + .. code:: sh + + shell$ open $prefix/share/doc/openmpi/html/index.html + +#. The man pages are installed (by default) to ``$preix/share/man``. + If your man page search path includes this location, you can invoke + commands similar to the following to see the same content that you + see in these HTML pages: + + .. code:: sh + + shell$ man MPI_Send diff --git a/docs/getting-help.rst b/docs/getting-help.rst index eadf5e26d39..fd0b94f2514 100644 --- a/docs/getting-help.rst +++ b/docs/getting-help.rst @@ -52,14 +52,14 @@ places. If you have: than this, please try compressing it and/or posting it on the web somewhere for people to download. A `Github Gist `_ or a `Pastebin - `_ might be an easy choice for posting + `_ might be a good choice for posting large text files. .. important:: Please **use a descriptive "subject" line in your email!** Some Open MPI question-answering people decide whether to read a mail based on its subject line (e.g., to see if it's a - question that they can answer). So please please please use a - good subject line that succinctly describes your problem. + question that they can answer). So please *please* **please** + use a good subject line that succinctly describes your problem. #. **A bug report:** you should probably post it to `Open MPI's Github issue tracker `_. Follow diff --git a/docs/installing-open-mpi/configure-cli-options/installation.rst b/docs/installing-open-mpi/configure-cli-options/installation.rst index bf425d56301..7529bad1e6e 100644 --- a/docs/installing-open-mpi/configure-cli-options/installation.rst +++ b/docs/installing-open-mpi/configure-cli-options/installation.rst @@ -1,3 +1,5 @@ +.. _label-building-installation-cli-options: + Installation options ^^^^^^^^^^^^^^^^^^^^ @@ -21,6 +23,11 @@ be used with ``configure``: libraries and disabling shared libraries are two independent options. + .. tip:: + + :ref:`See this section ` for + advice to packagers about this CLI option. + * ``--enable-static``: Build MPI and OpenSHMEM as static libraries, and statically link in all components. Note that this option does *not* imply @@ -30,6 +37,11 @@ be used with ``configure``: Be sure to read the description of ``--without-memory-manager``, below; it may have some effect on ``--enable-static``. + .. tip:: + + :ref:`See this section ` for + advice to packagers about this CLI option. + * ``--disable-wrapper-runpath`` / ``--disable-wrapper-rpath``: By default, the wrapper compilers (e.g., ``mpicc``) will explicitly add "runpath" and "rpath" linker flags when linking user executables on @@ -246,6 +258,11 @@ be used with ``configure``: shell$ ./configure --enable-mca-dso=btl-tcp --enable-mca-static=btl-tcp + .. tip:: + + :ref:`See this section ` for + advice to packagers about this CLI option. + * ``--enable-mca-no-build=LIST``: Comma-separated list of ``-`` pairs that will not be built. For example, ``--enable-mca-no-build=threads-qthreads,pml-monitoring`` will diff --git a/docs/installing-open-mpi/index.rst b/docs/installing-open-mpi/index.rst index e7249d6daad..9d6440cc5e2 100644 --- a/docs/installing-open-mpi/index.rst +++ b/docs/installing-open-mpi/index.rst @@ -16,3 +16,4 @@ Building and installing Open MPI configure-output-summary make-targets installation-location + packagers diff --git a/docs/installing-open-mpi/make-targets.rst b/docs/installing-open-mpi/make-targets.rst index 44716be0a87..efcf801a6c3 100644 --- a/docs/installing-open-mpi/make-targets.rst +++ b/docs/installing-open-mpi/make-targets.rst @@ -11,10 +11,3 @@ Automake, such as: Once Open MPI has been built and installed, it is safe to run ``make clean`` and/or remove the entire build tree. - -Generally speaking, the only thing that users need to do to use Open -MPI is ensure that ``PREFIX/bin`` is in their ``PATH`` and -``PREFIX/lib`` is in their ``LD_LIBRARY_PATH``. Users may need to -ensure to set the ``PATH`` and ``LD_LIBRARY_PATH`` in their shell -setup files (e.g., ``.bashrc``, ``.cshrc``) so that non-interactive -``ssh``-based logins will be able to find the Open MPI executables. diff --git a/docs/installing-open-mpi/packagers.rst b/docs/installing-open-mpi/packagers.rst new file mode 100644 index 00000000000..6a658ab0e7c --- /dev/null +++ b/docs/installing-open-mpi/packagers.rst @@ -0,0 +1,111 @@ +Advice for packagers +==================== + +.. _label-install-packagers-do-not-use-internal: + +Do not use Open MPI's internal dependent libraries +-------------------------------------------------- + +The Open MPI community **strongly** suggests that binary Open MPI +packages should *not* include Hwloc, Libevent, PMIx, or PRRTE. +:ref:`Although several of these libraries are required by Open MPI +` (and are therefore bundled +in the Open MPI source code distribution for end-user convenience), +binary Open MPI packages should limit themselves solely to Open MPI +artifacts. Specifically: ensure to configure and build Open MPI +against external installations of these required packages. + +Packagers may therefore wish to configure Open MPI with something like +the following: + +.. code-block:: sh + + ./configure --with-libevent=external --with-hwloc=external \ + --with-pmix=external --with-prrte=external ... + +The ``external`` keywords will force Open MPI's ``configure`` to +ignore all the bundled libraries and only look for external versions +of these support libraries. This also has the benefit of causing +``configure`` to fail if it cannot find the required support libraries +outside of the Open MPI source tree |mdash| a good sanity check to +ensure that your package is correctly relying on the +independently-built and installed versions. + +:ref:`See this section +` for more +information about the required support library ``--with-FOO`` command +line options. + +.. _label-install-packagers-dso-or-not: + +Components ("plugins"): DSO or no? +---------------------------------- + +Open MPI contains a large number of components (sometimes called +"plugins") to effect different types of functionality in MPI. For +example, some components effect Open MPI's networking functionality: +they may link against specialized libraries to provide +highly-optimized network access. + +Open MPI |ompi_ver| has two ``configure``-time defaults regarding the +treatment of components that may be of interest to packagers: + +#. Open MPI's libraries default to building as shared libraries + (vs. static libraries). For example, on Linux, Open MPI will + default to building ``libmpi.so`` (vs. ``libmpi.a``). + + .. note:: See the descriptions of ``--disable-shared`` and + ``--enable-static`` :ref:`in this section + ` for more + details about how to change this default. + + Also be sure to :ref:`see this warning about building + static apps `. + +#. Open MPI will default to including its components in its libraries + (as opposed to being compiled as dynamic shared objects, or DSOs). + For example, ``libmpi.so`` on Linux systems will contain the UCX + PML component, instead of the UCX PML being compiled into + ``mca_pml_ucx.so`` and dynamically opened at run time via + ``dlopen(3)``. + + .. note:: See the descriptions of ``--enable-mca-dso`` and + ``--enable-mca-static`` :ref:`in this section + ` for more + details about how to change this defaults. + +A side effect of these two defaults is that all the components +included in the Open MPI libraries will bring their dependencies with +them. For example (on Linux), if the XYZ PML component in the MPI +layer requires ``libXYZ.so``, then these defaults mean that +``libmpi.so`` will depend on ``libXYZ.so``. This dependency will +likely be telegraphed into the Open MPI binary package that includes +``libmpi.so``. + +Conversely, if the XYZ PML component was built as a DSO, then |mdash| +assuming no other parts of Open MPI require ``libXYZ.so`` |mdash| +``libmpi.so`` would *not* be dependent on ``libXYZ.so``. Instead, the +``mca_pml_xyz.so`` DSO would have the dependency upon ``libXYZ.so``. + +Packagers can use these facts to potentially create multiple binary +Open MPI packages, each with different dependencies by, for example, +using ``--enable-mca-dso`` to selectively build some components as +DSOs and leave the others included in their respective Open MPI +libraries. + +.. code:: sh + + # Build all the "accelerator" components as DSOs (all other + # components will default to being built in their respective + # libraries) + shell$ ./configure --enable-mca-dso=accelerator ... + +This allows packaging ``$libdir`` as part of the "main" Open MPI +binary package, but then packaging +``$libdir/openmpi/mca_accelerator_*.so`` as sub-packages. These +sub-packages may inherit dependencies on the CUDA and/or ROCM +packages, for example. User can always install the "main" Open MPI +binary package, and can install the additional "accelerator" Open MPI +binary sub-package if they actually have accelerator hardware +installed (which will cause the installation of additional +dependencies). diff --git a/docs/installing-open-mpi/required-support-libraries.rst b/docs/installing-open-mpi/required-support-libraries.rst index e3aaa2b2776..c8ca359e8e9 100644 --- a/docs/installing-open-mpi/required-support-libraries.rst +++ b/docs/installing-open-mpi/required-support-libraries.rst @@ -39,9 +39,10 @@ Open MPI requires the following support libraries with the minimum listed versio | tree, or compile/link against an external PRRTE installation. | :ref:`See this section for details about how to specify each method `. -Since these support libraries are fundamental to Open MPI's operation, -they are directly incorporated into Open MPI's configure, build, and -installation process. More on this below. +Since these support libraries are fundamental to Open MPI's operation +and not universally available in all environments, they are directly +incorporated into Open MPI's configure, build, and installation +process. More on this below. Library dependencies -------------------- @@ -245,26 +246,9 @@ versions of the required libraries. If you are an Open MPI packager, we **strongly** suggest that your Open MPI package should not include Hwloc, Libevent, PMIx, or PRRTE. -Instead, it should depend on independently-built versions of these -packages. +Instead, it should depend on external, independently-built versions of +these packages. -You may wish to configure Open MPI with something like the -following: - -.. code-block:: sh - - ./configure --with-libevent=external --with-hwloc=external \ - --with-pmix=external --with-prrte=external ... - -The ``external`` keywords will force ``configure`` to ignore all the -bundled libraries and only look for external versions of these support -libraries. This also has the benefit of causing ``configure`` to fail -if it cannot find the required support libraries outside of the Open -MPI source tree |mdash| a good sanity check to ensure that your -package is correctly relying on the independently-built and installed -versions. - -:ref:`See this section -` for more -information about the required support library ``--with-FOO`` command -line options. +See the :ref:`Advice for packagers +` section for more +details. diff --git a/docs/man-openmpi/index.rst b/docs/man-openmpi/index.rst index 2291fea87b8..3d18ee3e880 100644 --- a/docs/man-openmpi/index.rst +++ b/docs/man-openmpi/index.rst @@ -1,6 +1,19 @@ Open MPI manual pages ===================== +All of the content in this section is also installed as individual +manual ("man") pages in an Open MPI installation. If your man page +search path includes the location where Open MPI installed its man +pages (which defaults to ``$prefix/share/man``), you can invoke +commands similar to the following to see the same content that you see +in these HTML pages: + +.. code:: sh + + shell$ man MPI_Send + +Open MPI man pages, by section: + .. toctree:: :maxdepth: 1 diff --git a/docs/man-openmpi/man1/mpirun.1.rst b/docs/man-openmpi/man1/mpirun.1.rst index 3c44cff90ac..87a8541f118 100644 --- a/docs/man-openmpi/man1/mpirun.1.rst +++ b/docs/man-openmpi/man1/mpirun.1.rst @@ -9,8 +9,10 @@ mpirun / mpiexec mpirun, mpiexec |mdash| Execute serial and parallel jobs in Open MPI. -.. note:: ``mpirun`` and ``mpiexec`` are all synonyms for each other. - Using either of the names will produce the same behavior. +.. note:: ``mpirun`` and ``mpiexec`` are synonyms for each other. + Indeed, they are symbolic links to the same executable. + Using either of the names will produce the exact same + behavior. SYNOPSIS -------- diff --git a/docs/man-openshmem/index.rst b/docs/man-openshmem/index.rst index eb118aa0ca4..1dd92431628 100644 --- a/docs/man-openshmem/index.rst +++ b/docs/man-openshmem/index.rst @@ -1,6 +1,19 @@ OpenSHMEM manual pages ====================== +All of the content in this section is also installed as individual +manual ("man") pages in an OpenSHMEM installation. If your man page +search path includes the location where OpenSHMEM installed its man +pages (which defaults to ``$prefix/share/man``), you can invoke +commands similar to the following to see the same content that you see +in these HTML pages: + +.. code:: sh + + shell$ man shmem_init + +OpenSHMEM man pages, by section: + .. toctree:: :maxdepth: 1 diff --git a/docs/man-openshmem/man1/index.rst b/docs/man-openshmem/man1/index.rst index 37c171f8d55..7e7d6fd1086 100644 --- a/docs/man-openshmem/man1/index.rst +++ b/docs/man-openshmem/man1/index.rst @@ -6,3 +6,4 @@ Commands (section 1) oshmem-wrapper-compiler.1.rst oshmem_info.1.rst + oshrun.1.rst diff --git a/docs/man-openshmem/man1/oshrun.1.rst b/docs/man-openshmem/man1/oshrun.1.rst new file mode 100644 index 00000000000..b7cbcf88ddb --- /dev/null +++ b/docs/man-openshmem/man1/oshrun.1.rst @@ -0,0 +1,16 @@ +.. _man1-oshrun: + + +oshrun +====== + +.. include_body + +oshrun |mdash| Execute serial and OpenSHMEM parallel jobs. + +The ``oshrun(1)`` command is functionality identical to the +:ref:`mpirun(1) ` command. See the :ref:`mpirun(1) +` man page for details. + +.. seealso:: + :ref:`mpirun(1) ` diff --git a/docs/running-apps/index.rst b/docs/running-apps/index.rst index e9343a3cf4d..52352b6012c 100644 --- a/docs/running-apps/index.rst +++ b/docs/running-apps/index.rst @@ -19,6 +19,7 @@ but they can generally be broken down into two categories: quickstart pmix-and-prrte + prerequisites tuning localhost diff --git a/docs/running-apps/prerequisites.rst b/docs/running-apps/prerequisites.rst new file mode 100644 index 00000000000..84e6eea175d --- /dev/null +++ b/docs/running-apps/prerequisites.rst @@ -0,0 +1,29 @@ +Prerequisites +============= + +In general, successful execution of Open MPI jobs requires the ability +to find Open MPI's executables and shared libraries on all nodes at +run time. If these can be via in system-default search paths (e.g., + +* If Open MPI is installed in ``/usr/bin`` and ``/usr/lib``), that is + usually sufficient. +* If Open MPI is installed in a location that is not searched by + default, users may need to add ``$prefix/bin`` to their ``PATH`` and + ``$libdir`` (which defaults to ``$prefix/lib``) to their + ``LD_LIBRARY_PATH``. + + .. caution:: In scheduled environments, ensuring Open MPI's + executables and libraries can be found on the node that + executes :ref:`mpirun(1) ` may be + sufficient. + + In non-scheduled environments, users may need to set + the ``PATH`` and ``LD_LIBRARY_PATH`` environment + variables in their shell setup files (e.g., + ``$HOME/.bashrc``) so that non-interactive + ``ssh``-based logins will be able to find the Open MPI + executables and libraries. + +If users are unable to add the relevant directories to ``PATH`` and +``LD_LIBRARY_PATH``,the :ref:`mpirun(1) ` ``--prefix`` +option can be used diff --git a/docs/running-apps/quickstart.rst b/docs/running-apps/quickstart.rst index 6f414fb7ca4..5ad4bfd894a 100644 --- a/docs/running-apps/quickstart.rst +++ b/docs/running-apps/quickstart.rst @@ -14,8 +14,9 @@ probably work in many environments. Please consult the other sections in this chapter for more details, if necessary. -Open MPI supports both ``mpirun`` and ``mpiexec`` (they are exactly -equivalent) to launch MPI applications. For example: +Open MPI supports both :ref:`mpirun(1) ` and +:ref:`mpiexec(1) ` (they are exactly equivalent) to +launch MPI applications. For example: .. code-block:: sh @@ -28,17 +29,17 @@ equivalent) to launch MPI applications. For example: are all equivalent. For simplicity, the rest of this documentation will simply refer to ``mpirun``. -.. error:: TODO Link to the mpirun(1) page here. - -Note that the ``mpirun`` command supports a *large* number of options. -Be sure to see the ``mpirun`` man page for much more information. +Note that the :ref:`mpirun(1) ` command supports a +*large* number of options. Be sure to see the :ref:`mpirun(1) +` man page for much more information. Launching on a single host -------------------------- It is common to develop MPI applications on a single laptop or -workstation. In such cases, use ``mpirun`` and specify how many MPI -processes you want to launch via the ``-n`` option: +workstation. In such cases, use :ref:`mpirun(1) ` and +specify how many MPI processes you want to launch via the ``-n`` +option: .. code-block:: sh @@ -48,9 +49,9 @@ processes you want to launch via the ``-n`` option: ... Hello world, I am 5 of 6 (running on my-laptop) -If you do not specify the ``-n`` option, ``mpirun`` will default to -launching as many MPI processes as there are processor cores (not -hyperthreads) on the machine. +If you do not specify the ``-n`` option, :ref:`mpirun(1) +` will default to launching as many MPI processes as +there are processor cores (not hyperthreads) on the machine. Launching in a non-scheduled environments (via ``ssh``) ------------------------------------------------------- @@ -64,8 +65,8 @@ applications: #. Open MPI's libraries must be findable (e.g., in your ``LD_LIBRARY_PATH``). -``mpirun`` accepts a ``--hostfile`` parameter to specify a hostfile -containing one hostname per line: +:ref:`mpirun(1) ` accepts a ``--hostfile`` parameter to +specify a hostfile containing one hostname per line: .. code-block:: sh @@ -119,25 +120,25 @@ processes and the hosts to use) to Open MPI. There are two ways to launch in a scheduled environment. Nominally, they both achieve the same thing: they launch MPI processes. Them main user-observable difference between the two methods is that -``mpirun`` has many more features than scheduler direct launchers. - -Using Open MPI's ``mpirun`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. note:: Technically, Open MPI's ``mpirun`` is a thin layer around - the PRRTE ``prun``. Hence, most of the functionality - described here is really about ``prun``. For simplicity, - however, this docmentation will describe everything in terms - of ``mpirun``. - -.. error:: TODO Link to mpirun(1) here. - -When using the full-featured ``mpirun`` in a scheduled environment, -there is no need to specify a hostfile or number of MPI processes to -launch. ``mpirun`` will receive this information directly from the -scheduler. Hence, if you want to launch an MPI job that completely -"fills" your scheduled allocation (i.e., one MPI process for each slot -in the scheduled allocation), you can simply: +:ref:`mpirun(1) ` has many more features than scheduler +direct launchers. + +Using Open MPI's :ref:`mpirun(1) ` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: Technically, Open MPI's :ref:`mpirun(1) ` is a + thin layer around the PRRTE ``prun``. Hence, most of the + functionality described here is really about ``prun``. For + simplicity, however, this docmentation will describe + everything in terms of ``mpirun``. + +When using the full-featured :ref:`mpirun(1) ` in a +scheduled environment, there is no need to specify a hostfile or +number of MPI processes to launch. :ref:`mpirun(1) ` +will receive this information directly from the scheduler. Hence, if +you want to launch an MPI job that completely "fills" your scheduled +allocation (i.e., one MPI process for each slot in the scheduled +allocation), you can simply: .. code-block:: sh @@ -178,19 +179,18 @@ The above example shows that simply invoking ``mpirun mpi-hello-world`` |mdash| with no other CLI options |mdash| obtains the number of processes to run and hosts to use from the scheduler. -.. error:: TODO Link to mpirun(1) here. - -``mpirun`` has many more features not described in this Quick Start -section. For example, while uncommon in scheduled environments, you -can use ``-n`` and/or ``--hostfile`` to launch in subsets of the -overall scheduler allocation. See the mpirun man page for more -details. +:ref:`mpirun(1) ` has many more features not described in +this Quick Start section. For example, while uncommon in scheduled +environments, you can use ``-n`` and/or ``--hostfile`` to launch in +subsets of the overall scheduler allocation. See the :ref:`mpirun(1) +` man page for more details. -Using the scheduler to "direct launch" (without ``mpirun``) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Using the scheduler to "direct launch" (without :ref:`mpirun(1) `) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some schedulers (such as Slurm) have the ability to "direct launch" -MPI processes without using Open MPI's ``mpirun``. For example: +MPI processes without using Open MPI's :ref:`mpirun(1) `. +For example: .. code-block:: sh @@ -204,4 +204,4 @@ MPI processes without using Open MPI's ``mpirun``. For example: Similar to the prior example, this example launches 40 copies of ``mpi-hello-world``, but it does so via the Slurm ``srun`` command -without using ``mpirun``. +without using :ref:`mpirun(1) `. diff --git a/docs/running-apps/tuning.rst b/docs/running-apps/tuning.rst index 88d2a782f90..e90fb61ba84 100644 --- a/docs/running-apps/tuning.rst +++ b/docs/running-apps/tuning.rst @@ -172,6 +172,15 @@ in priority order: #. Tuning MCA parameter files #. Configuration files +.. danger:: Due to how the PMIx and PRRTE projects both evolved to + become independent projects from Open MPI (:ref:`see this + section for more detail + `), they both have + their own MCA system for setting MCA parameters. + + Hence, all the information about MCA parameters below + *also* applies to PMIx and PRRTE. + Command line parameters ^^^^^^^^^^^^^^^^^^^^^^^ @@ -194,6 +203,19 @@ format used on the command line is ``--mca ``. shell$ mpirun --mca param "value with multiple words" ... +.. warning:: Setting Open MPI MCA parameters via the command line + entails using the ``--mca`` CLI option. When setting + PMIx- and PRRTE-specific MCA parameters via configuration + files, use a different CLI option: + + +----------+----------------+ + | Open MPI | ``--mca`` | + +----------+----------------+ + | PMIx | ``--pmixmca`` | + +----------+----------------+ + | PRRTE | ``--prtemca`` | + +----------+----------------+ + Environment variables ^^^^^^^^^^^^^^^^^^^^^ @@ -215,10 +237,23 @@ shells): shell$ export OMPI_MCA_param="value with multiple words" +.. warning:: Setting Open MPI MCA parameters via environment variables + entails prefixing the parameter name with ``OMPI_MCA_``. + When setting PMIx- and PRRTE-specific MCA parameters via + environment variables, use a different prefix: + + +----------+----------------+ + | Open MPI | ``OMPI_MCA_`` | + +----------+----------------+ + | PMIx | ``PMIX_MCA_`` | + +----------+----------------+ + | PRRTE | ``PRRTE_MCA_`` | + +----------+----------------+ + Tuning MCA parameter files ^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. error:: TODO This entire entry needs to be checked for correctness. +.. error:: TODO This entire section needs to be checked for correctness. Simple text files can be used to set MCA parameter values for a specific application. @@ -333,6 +368,25 @@ precedence. customization, which is especially relevant in heterogeneous environments. +.. error:: TODO This table needs to be checked for correctness. + +.. warning:: Setting Open MPI MCA parameters via configuration files + entails editing (by default) the ``mca-params.conf`` or + ``openmpi-mca-params.conf`` files. When setting PMIx- + and PRRTE-specific MCA parameters via configuration + files, set them (by default) in different files: + + +----------+------------------------------------------+ + | Open MPI | ``$HOME/.openmpi/mca-params.conf`` or | + | | ``$prefix/etc/openmpi-mca-params.conf`` | + +----------+------------------------------------------+ + | PMIx | ``$HOME/.pmix/mca-params.conf`` or | + | | ``$prefix/etc/openpmix-mca-params.conf`` | + +----------+------------------------------------------+ + | PRRTE | ``$HOME/.prrte/mca-params.conf`` or | + | | ``$prefix/etc/prte-mca-params.conf`` | + +----------+------------------------------------------+ + ///////////////////////////////////////////////////////////////////////// .. _label-running-selecting-framework-components: @@ -388,6 +442,14 @@ Open MPI has a *large* number of MCA parameters available. Users can use the :ref:`ompi_info(1) ` command to see *all* available MCA parameters. +.. note:: Similarly, you can use the ``pmix_info(1)`` and + ``prte_info(1)`` commands to see all the MCA parameters + available for the PMIx and PRRTE projects, respectively. + + The documentation for these commands are not included in the + Open MPI docs, but they are both quite similar to + :ref:`ompi_info(1) `. + The vast majority of these MCA parameters, however, are not useful to most users. Indeed, there only are a handful of MCA parameters that are commonly used by end users. :ref:`As described in the diff --git a/docs/validate.rst b/docs/validate.rst index a9a43703368..ddfc9b45919 100644 --- a/docs/validate.rst +++ b/docs/validate.rst @@ -4,17 +4,18 @@ Validating your installation Checking your Open MPI configuration ------------------------------------ -The ``ompi_info`` command can be used to check the status of your Open -MPI installation (located in ``PREFIX/bin/ompi_info``). Running it with -no arguments provides a summary of information about your Open MPI -installation. +The :ref:`ompi_info(1) ` command can be used to check +the status of your Open MPI installation (located in +``$prefix/bin/ompi_info``). Running it with no arguments provides a +summary of information about your Open MPI installation. -Note that the ``ompi_info`` command is extremely helpful in determining -which components are installed as well as listing all the run-time -settable parameters that are available in each component (as well as -their default values). +Note that the :ref:`ompi_info(1) ` command is +extremely helpful in determining which components are installed as +well as listing all the run-time settable parameters that are +available in each component (as well as their default values). -The following ``ompi_info`` options may be helpful: +The following :ref:`ompi_info(1) ` options may be +helpful: * ``--all``: Show a *lot* of information about your Open MPI installation. @@ -36,11 +37,8 @@ The following ``ompi_info`` options may be helpful: for a given component. See "The Modular Component Architecture (MCA)" section, below, for a fuller explanation. -.. error:: TODO move the ref below to a better / more-prominent place - to explain how to set MCA params. - -Changing the values of these parameters is explained in the -:ref:`Internal frameworks section `. +Changing the values of these MCA parameters is explained in the +:ref:`Tuning section `. Testing your Open MPI installation @@ -49,22 +47,25 @@ Testing your Open MPI installation When verifying a new Open MPI installation, we recommend running the following tests in order (the tests build upon each other): -#. Use ``mpirun`` to launch a non-MPI program (e.g., ``hostname`` or - ``uptime``) across multiple nodes. -#. Use ``mpirun`` to launch a trivial MPI program that does no MPI - communication (e.g., the ``hello_c`` program in the ``examples/`` - directory in the Open MPI distribution). -#. Use ``mpirun`` to launch a trivial MPI program that sends and - receives a few MPI messages (e.g., the ``ring_c`` program in the - ``examples/`` directory in the Open MPI distribution). -#. Use ``oshrun`` to launch a non-OpenSHMEM program across multiple - nodes. -#. Use ``oshrun`` to launch a trivial MPI program that does no OpenSHMEM - communication (e.g., ``hello_shmem.c`` program in the ``examples/`` - directory in the Open MPI distribution.) -#. Use ``oshrun`` to launch a trivial OpenSHMEM program that puts and - gets a few messages (e.g., the ``ring_shmem.c`` in the ``examples/`` - directory in the Open MPI distribution.) +#. Use :ref:`mpirun(1) ` to launch a non-MPI program + (e.g., ``hostname`` or ``uptime``) across multiple nodes. +#. Use :ref:`mpirun(1) ` to launch a trivial MPI program + that does no MPI communication (e.g., the ``hello_c`` program in + the ``examples/`` directory in the Open MPI distribution). +#. Use :ref:`mpirun(1) ` to launch a trivial MPI program + that sends and receives a few MPI messages (e.g., the ``ring_c`` + program in the ``examples/`` directory in the Open MPI + distribution). +#. Use :ref:`oshrun(1) ` to launch a non-OpenSHMEM + program across multiple nodes. +#. Use :ref:`oshrun(1) ` to launch a trivial MPI program + that does no OpenSHMEM communication (e.g., ``hello_shmem.c`` + program in the ``examples/`` directory in the Open MPI + distribution.) +#. Use :ref:`oshrun ` to launch a trivial OpenSHMEM + program that puts and gets a few messages (e.g., the + ``ring_shmem.c`` in the ``examples/`` directory in the Open MPI + distribution.) If you can run all of these tests successfully, that is a good indication that Open MPI built and installed properly.