From 441579cdc5cf7683e526490578b55f5d30740cd8 Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Thu, 31 Oct 2024 09:50:42 -0700 Subject: [PATCH 1/7] init push --- doc/launch_settings.rst | 328 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 328 insertions(+) create mode 100644 doc/launch_settings.rst diff --git a/doc/launch_settings.rst b/doc/launch_settings.rst new file mode 100644 index 000000000..9800f25ee --- /dev/null +++ b/doc/launch_settings.rst @@ -0,0 +1,328 @@ +************** +LaunchSettings +************** +======== +Overview +======== +The ``LaunchSettings`` class manages launcher configuration settings to enable the injection of +launcher-specific behavior into jobs. SmartSim ``LaunchSettings`` supports several launchers and allows users to +configure launch arguments and environment variables. Additionally, the object provides methods to access and modify +launch arguments, environment variables and retrieve information associated with the ``LaunchSettings`` object. + +**Dragon** is the fastest and most versatile distributed runtime available for HPC workflows. It can function as a +launcher within a Slurm allocation, providing rapid, interactive, and customizable execution of complex workflows +on large HPC systems. As a scheduler-agnostic solution, Dragon allows the same SmartSim script to run seamlessly +on both Slurm and PBS systems, with future support for additional schedulers. + +=================== +Supported Launchers +=================== +The ``LaunchSettings`` class supports multiple launchers, each customized for different environments. Among these, +**Dragon** stands out as the fastest and most versatile option, ideal for large-scale HPC jobs. The following launchers +are categorized based on their specific use cases and the systems they are designed for. + +HPE Cray Specific Launchers +=========================== +SLURM and PALS launchers are specifically designed for HPE Cray systems, providing efficient management and +execution of parallel applications. + +.. list-table:: HPE Cray Specific Launchers + :header-rows: 1 + + * - **Run Command** + - **Description** + * - ``aprun`` + - Application Level Placement Scheduler for HPE Cray systems. + * - ``palsrun`` + - Parallel Application Launch Service for distributed computing. + + +IMB Specific Launchers +====================== +The LSF launcher is tailored for use with the IBM Spectrum Load Sharing Facility (LSF), which is commonly used in +enterprise environments to distribute and manage workloads. + +.. list-table:: IMB Specific Launchers + :header-rows: 1 + + * - **Run Command** + - **Description** + * - ``jsrun`` + - Load Sharing Facility for launching parallel jobs. + +Standard MPI Launchers +====================== +These launchers are used for running MPI (Message Passing Interface) applications, providing flexibility +and control over parallel job execution. + +.. list-table:: MPI Launchers + :header-rows: 1 + + * - **Run Command** + - **Description** + * - ``mpiexec`` + - Standard MPI launcher for parallel jobs. + * - ``mpirun`` + - Another MPI launcher, often used with Open MPI. + * - ``orterun`` + - Open MPI's runtime environment launcher. + +General HPC Launchers +===================== +The SLURM and Dragon launchers are suitable for general high-performance computing (HPC) environments, offering robust solutions +for job scheduling and resource management. + +.. list-table:: General HPC Launchers + :header-rows: 1 + + * - **Run Command** + - **Description** + * - ``srun`` + - Open-source job scheduler and resource manager for Linux clusters. + * - ``dragonrun`` + - High-performance computing launcher for large-scale jobs. + +Other +===== + +.. list-table:: General HPC Launchers + :header-rows: 1 + + * - **Run Command** + - **Description** + * - ``local`` + - Runs jobs on the local machine. + +The ``LaunchSettings`` class ensures that users can efficiently manage and execute their HPC jobs across +various environments. + +========== +Initialize +========== +The ``LaunchSettings`` class provides a way to configure and manage the execution environment for your +applications. This section outlines the steps to configure the essential parameters that determine how +your application launches. + +**Step 1: Import LaunchSettings** + +After installing Smartsim, ``LaunchSettings`` may be imported in Python code like: + +.. code-block:: python + + from smartsim import LaunchSettings + +**Step 2: Set the Launcher Type** + +Set the launcher type using either a string or a ``LauncherType`` enum. This step is crucial as it +determines the specific launcher configuration that will be applied. The following table lists the +supported strings and their corresponding enums: + +.. list-table:: Supported Launcher Strings and Enums + :header-rows: 1 + + * - **Launcher** + - **String** + - **Enum** + * - Dragon + - ``"dragon"`` + - ``LauncherType.Dragon`` + * - SLURM + - ``"slurm"`` + - ``LauncherType.Slurm`` + * - PALS + - ``"pals"`` + - ``LauncherType.Pals`` + * - ALPS + - ``"alps"`` + - ``LauncherType.Alps`` + * - Local + - ``"local"`` + - ``LauncherType.Local`` + * - Mpiexec + - ``"mpiexec"`` + - ``LauncherType.Mpiexec`` + * - Mpirun + - ``"mpirun"`` + - ``LauncherType.Mpirun`` + * - Orterun + - ``"orterun"`` + - ``LauncherType.Orterun`` + * - LSF + - ``"lsf"`` + - ``LauncherType.Lsf`` + + +**Step 3: Provide Launch Arguments and Environment Variables** +Optionally, you can provide ``launch_args`` and ``env_vars`` to customize the job execution environment: + +* ``launch_args``: A dictionary where keys are argument names (strings) and values are argument values (strings). + These arguments are specific to the launcher being used. Example: + + .. code-block:: python + + launch_args = {"--time": "01:00:00", "--nodes": "2"} + +* ``env_vars``: A dictionary where keys are environment variable names (strings) and values are environment + variable values (strings). These variables set the environment for the job execution. Example: + + .. code-block:: python + + env_vars = {"MY_VAR": "my_value", "ANOTHER_VAR": "another_value"} + +Here's how you can initialize ``LaunchSettings`` with these parameters: + +**Example using a launcher String:** +Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSettings``, use a +launcher string such as `"slurm"`. For example: + +.. code-block:: python + + launch_settings = LaunchSettings( + launcher="slurm", + launch_args={"--time": "01:00:00"}, + env_vars={"MY_VAR": "my_value"} + ) + +**Example using an Enum:** +Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSettings``, use a +``LauncherType`` enum such as ``LauncherType.Slurm``. For example: + +.. code-block:: python + + launch_settings = LaunchSettings( + launcher=LauncherType.Slurm, + launch_args={"--time": "01:00:00"}, + env_vars={"MY_VAR": "my_value"} + ) + +========= +Configure +========= +After initializing a ``LaunchSettings`` object, you might want to go back and configure ``launch_args`` or +``env_vars`` for several reasons. Customizing these settings allows you to fine-tune the execution environment +to meet the specific needs of different jobs, optimize performance, and achieve better resource utilization and +faster execution times. Additionally, modifying these settings can aid in debugging by enabling additional +logging, setting debug flags, or changing the execution environment. Experimentation with different configurations +can also help you understand how changes affect performance or behavior, which is particularly useful in research +and development settings. Finally, adjusting these settings allows for a more personalized and efficient workflow, +accommodating different user preferences and requirements. + +Launch Arguments +================ +Configure `launch_args` in two ways: + +1. Use ``LaunchSettings.launch_args.set``. +2. Use custom methods specific to each launcher. + +These functions allow you to customize `launch_args` after initializing the ``LaunchSettings`` object. + +**Option 1: Use LaunchSettings.launch_args.set** + +To set additional launch arguments after initializing the ``LaunchSettings`` object, +use the `set` method on `launch_args` as shown below: + +.. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + +**Option 2: Use custom methods specific to each launcher** + +The ``LaunchSettings`` class provides custom methods to set launch arguments for different supported launchers. +These methods tailor the job execution environment to meet specific requirements of each launcher. +Below are examples of how to use these custom methods for various launchers. + +.. tabs:: + + .. tab:: Dragon + + **Set Launch Arguments for Dragon:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: SLURM + + **Set Launch Arguments for SLURM:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: PALS + + **Set Launch Arguments for PALS:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: ALPS + + **Set Launch Arguments for ALPS:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: Local + + **Set Launch Arguments for Local:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: Mpiexec + + **Set Launch Arguments for Mpiexec:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: Mpirun + + **Set Launch Arguments for Mpirun:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: Orterun + + **Set Launch Arguments for Orterun:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + + .. tab:: LSF + + **Set Launch Arguments for LSF:** + + .. code-block:: python + + launch_settings.launch_args.set("--nodes", "2") + +For detailed information on these methods, refer to the API reference page. + +Environment Variables +===================== +To update the ``env_vars`` after initializing the ``LaunchSettings`` object, pass in a dictionary where +each key and value are strings to ``LaunchSettings.update_env``. This function updates the existing dictionary +of environment settings without overwriting it. For example: + +.. code-block:: python + + launch_settings.update_env({"MY_VAR": "new_value", "ANOTHER_VAR": "another_value"}) + +======== +Examples +======== + +Local +===== + +HPC +=== \ No newline at end of file From 98e45907006b338638c70f5081a875ed679a1fe8 Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Thu, 31 Oct 2024 15:51:06 -0700 Subject: [PATCH 2/7] updates to ls --- doc/launch_settings.rst | 169 ++++++++++++++++++++++------------------ 1 file changed, 94 insertions(+), 75 deletions(-) diff --git a/doc/launch_settings.rst b/doc/launch_settings.rst index 9800f25ee..dfb8abb9a 100644 --- a/doc/launch_settings.rst +++ b/doc/launch_settings.rst @@ -4,15 +4,15 @@ LaunchSettings ======== Overview ======== -The ``LaunchSettings`` class manages launcher configuration settings to enable the injection of -launcher-specific behavior into jobs. SmartSim ``LaunchSettings`` supports several launchers and allows users to -configure launch arguments and environment variables. Additionally, the object provides methods to access and modify -launch arguments, environment variables and retrieve information associated with the ``LaunchSettings`` object. +The ``LaunchSettings`` class manages launcher configuration settings to enable the injection of launcher-specific behavior into jobs. +SmartSim’s ``LaunchSettings`` supports multiple launchers, allowing users to set both launch arguments and environment variables. +Additionally, this class provides methods to access and configure these settings, as well as retrieve information associated with +the ``LaunchSettings`` object. **Dragon** is the fastest and most versatile distributed runtime available for HPC workflows. It can function as a launcher within a Slurm allocation, providing rapid, interactive, and customizable execution of complex workflows -on large HPC systems. As a scheduler-agnostic solution, Dragon allows the same SmartSim script to run seamlessly -on both Slurm and PBS systems, with future support for additional schedulers. +on large HPC systems. As a scheduler-agnostic solution, Dragon allows the same SmartSim script to run on both Slurm and +PBS systems, with future support for additional schedulers. =================== Supported Launchers @@ -21,10 +21,11 @@ The ``LaunchSettings`` class supports multiple launchers, each customized for di **Dragon** stands out as the fastest and most versatile option, ideal for large-scale HPC jobs. The following launchers are categorized based on their specific use cases and the systems they are designed for. -HPE Cray Specific Launchers -=========================== -SLURM and PALS launchers are specifically designed for HPE Cray systems, providing efficient management and -execution of parallel applications. +HPE Cray +======== +SmartSim provides support for two HPE Cray system launchers: **ALPS** and **PALS**. ALPS is designed for legacy Cray platforms +to address both future Cray platform needs and the limitations of older software components. PALS acts as a launcher for third-party +workload managers (WLMs) that do not have their own launchers. .. list-table:: HPE Cray Specific Launchers :header-rows: 1 @@ -32,15 +33,15 @@ execution of parallel applications. * - **Run Command** - **Description** * - ``aprun`` - - Application Level Placement Scheduler for HPE Cray systems. - * - ``palsrun`` - - Parallel Application Launch Service for distributed computing. + - Application Level Placement Scheduler (ALPS). + * - ``mpiexec`` + - Parallel Application Launch Service (PALS). -IMB Specific Launchers -====================== -The LSF launcher is tailored for use with the IBM Spectrum Load Sharing Facility (LSF), which is commonly used in -enterprise environments to distribute and manage workloads. +IBM +=== +Smartsim provides support for an IBM launcher: **LSF**. IBM's LSF is a platform for managing workloads and scheduling jobs +in distributed high-performance computing (HPC) environments. .. list-table:: IMB Specific Launchers :header-rows: 1 @@ -48,12 +49,12 @@ enterprise environments to distribute and manage workloads. * - **Run Command** - **Description** * - ``jsrun`` - - Load Sharing Facility for launching parallel jobs. + - Load Sharing Facility (LSF). -Standard MPI Launchers -====================== -These launchers are used for running MPI (Message Passing Interface) applications, providing flexibility -and control over parallel job execution. +Standard MPI +============ +SmartSim supports three commands to launch Message Passing Interface (MPI) applications: ``mpiexec``, ``mpirun`` and ``orterun``. +Each execute serial and parallel jobs in Open MPI. .. list-table:: MPI Launchers :header-rows: 1 @@ -61,16 +62,16 @@ and control over parallel job execution. * - **Run Command** - **Description** * - ``mpiexec`` - - Standard MPI launcher for parallel jobs. + - Used command to start MPI applications. * - ``mpirun`` - - Another MPI launcher, often used with Open MPI. + - Alternative command for launching MPI programs. * - ``orterun`` - - Open MPI's runtime environment launcher. + - Backend launcher for Open MPI's runtime environment. -General HPC Launchers -===================== -The SLURM and Dragon launchers are suitable for general high-performance computing (HPC) environments, offering robust solutions -for job scheduling and resource management. +General +======= +SmartSim provides support for two general launchers: **Dragon** and **SLURM**. Dragon is the most rapid and adaptable distributed +runtime for HPC workflows. SLURM is a open-source job scheduler and resource manager for Linux clusters. .. list-table:: General HPC Launchers :header-rows: 1 @@ -78,14 +79,15 @@ for job scheduling and resource management. * - **Run Command** - **Description** * - ``srun`` - - Open-source job scheduler and resource manager for Linux clusters. + - Simple Linux Utility for Resource Management (SLURM). * - ``dragonrun`` - - High-performance computing launcher for large-scale jobs. + - High-performance computing launcher for large-scale jobs (Dragon). -Other +Local ===== +SmartSim provides support to launch applications on the local machine. -.. list-table:: General HPC Launchers +.. list-table:: Local Launcher :header-rows: 1 * - **Run Command** @@ -99,9 +101,9 @@ various environments. ========== Initialize ========== -The ``LaunchSettings`` class provides a way to configure and manage the execution environment for your -applications. This section outlines the steps to configure the essential parameters that determine how -your application launches. +The ``LaunchSettings`` class allows you to set your application’s execution +environment. This section details the steps to set up the input parameters for +launching your application. **Step 1: Import LaunchSettings** @@ -114,8 +116,8 @@ After installing Smartsim, ``LaunchSettings`` may be imported in Python code lik **Step 2: Set the Launcher Type** Set the launcher type using either a string or a ``LauncherType`` enum. This step is crucial as it -determines the specific launcher configuration that will be applied. The following table lists the -supported strings and their corresponding enums: +determines the applied launcher. The following table lists the supported launcher strings and +their corresponding ``LauncherType`` enums: .. list-table:: Supported Launcher Strings and Enums :header-rows: 1 @@ -151,9 +153,15 @@ supported strings and their corresponding enums: - ``"lsf"`` - ``LauncherType.Lsf`` +To use an enum, import ``LauncherType`` into Python code like: + +.. code-block:: python + + from smartsim import LauncherType **Step 3: Provide Launch Arguments and Environment Variables** -Optionally, you can provide ``launch_args`` and ``env_vars`` to customize the job execution environment: + +Optionally, you can provide ``launch_args`` and ``env_vars`` to customize the application's execution environment: * ``launch_args``: A dictionary where keys are argument names (strings) and values are argument values (strings). These arguments are specific to the launcher being used. Example: @@ -163,17 +171,17 @@ Optionally, you can provide ``launch_args`` and ``env_vars`` to customize the jo launch_args = {"--time": "01:00:00", "--nodes": "2"} * ``env_vars``: A dictionary where keys are environment variable names (strings) and values are environment - variable values (strings). These variables set the environment for the job execution. Example: + variable values (strings). These variables set the environment for the application execution. Example: .. code-block:: python env_vars = {"MY_VAR": "my_value", "ANOTHER_VAR": "another_value"} -Here's how you can initialize ``LaunchSettings`` with these parameters: +Here's how you can initialize ``LaunchSettings`` with input parameters: **Example using a launcher String:** -Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSettings``, use a -launcher string such as `"slurm"`. For example: +Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSettings``, set the input +varaible ``launcher`` to a launcher string such as `"slurm"`. For example: .. code-block:: python @@ -184,8 +192,8 @@ launcher string such as `"slurm"`. For example: ) **Example using an Enum:** -Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSettings``, use a -``LauncherType`` enum such as ``LauncherType.Slurm``. For example: +Once you have imported ``LaunchSettings`` and ``LauncherType`` using ``from smartsim import LaunchSettings, LauncherType``, +set the input variable ``launcher`` to a ``LauncherType`` enum such as ``LauncherType.Slurm``. For example: .. code-block:: python @@ -199,22 +207,17 @@ Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSet Configure ========= After initializing a ``LaunchSettings`` object, you might want to go back and configure ``launch_args`` or -``env_vars`` for several reasons. Customizing these settings allows you to fine-tune the execution environment -to meet the specific needs of different jobs, optimize performance, and achieve better resource utilization and -faster execution times. Additionally, modifying these settings can aid in debugging by enabling additional -logging, setting debug flags, or changing the execution environment. Experimentation with different configurations -can also help you understand how changes affect performance or behavior, which is particularly useful in research -and development settings. Finally, adjusting these settings allows for a more personalized and efficient workflow, -accommodating different user preferences and requirements. +``env_vars``. Configuring these settings allows you to fine-tune the execution environment +to meet the specific needs of different jobs. Launch Arguments ================ -Configure `launch_args` in two ways: +There are two methods to configure ``launch_args``: 1. Use ``LaunchSettings.launch_args.set``. -2. Use custom methods specific to each launcher. +2. Use custom methods specific to each HPC launcher. -These functions allow you to customize `launch_args` after initializing the ``LaunchSettings`` object. +These functions allow you to customize launch arguments after initializing the ``LaunchSettings`` object. **Option 1: Use LaunchSettings.launch_args.set** @@ -225,11 +228,10 @@ use the `set` method on `launch_args` as shown below: launch_settings.launch_args.set("--nodes", "2") -**Option 2: Use custom methods specific to each launcher** +**Option 2: Use custom HPC launcher methods** -The ``LaunchSettings`` class provides custom methods to set launch arguments for different supported launchers. -These methods tailor the job execution environment to meet specific requirements of each launcher. -Below are examples of how to use these custom methods for various launchers. +The ``LaunchSettings`` class provides custom methods to set launch arguments for different launchers. +Below are examples of how to use the custom methods for the supported launchers. .. tabs:: @@ -239,7 +241,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set the nodes for a Dragon job + launch_settings.launch_args.set_nodes(128) .. tab:: SLURM @@ -247,7 +250,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set the nodes for a SLURM job + launch_settings.launch_args.set_nodes(128) .. tab:: PALS @@ -255,7 +259,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set tasks per node for a PALS job + launch_settings.launch_args.set_tasks_per_node(8) .. tab:: ALPS @@ -263,15 +268,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") - - .. tab:: Local - - **Set Launch Arguments for Local:** - - .. code-block:: python - - launch_settings.launch_args.set("--nodes", "2") + # Set tasks per node for a ALPS job + launch_settings.launch_args.set_tasks_per_node(8) .. tab:: Mpiexec @@ -279,7 +277,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set tasks per node for a Mpiexec job + launch_settings.launch_args.set_tasks_per_node(8) .. tab:: Mpirun @@ -287,7 +286,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set tasks per node for a Mpirun job + launch_settings.launch_args.set_tasks_per_node(8) .. tab:: Orterun @@ -295,7 +295,8 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set tasks per node for a Orterun job + launch_settings.launch_args.set_tasks_per_node(8) .. tab:: LSF @@ -303,12 +304,30 @@ Below are examples of how to use these custom methods for various launchers. .. code-block:: python - launch_settings.launch_args.set("--nodes", "2") + # Set tasks for a LSF job + launch_settings.launch_args.set_tasks(8) For detailed information on these methods, refer to the API reference page. Environment Variables ===================== +There are two methods to configure ``env_vars``: + +1. Use ``LaunchSettings.env_vars`` to overwrite the existing executable arguments. +2. Use ``LaunchSettings.update_env`` to add to the existing environment variables. + +**Option 1: Use LaunchSettings.env_vars** + +To overwrite the ``env_vars`` after initializing the ``LaunchSettings`` object, set a dictionary where +each key and value are strings to ``LaunchSettings.env_vars``. This method overwrites the existing dictionary +of environment settings. For example: + +.. code-block:: python + + launch_settings.env_vars = {"MY_VAR": "new_value", "ANOTHER_VAR": "another_value"} + +**Option 1: Use LaunchSettings.update_env** + To update the ``env_vars`` after initializing the ``LaunchSettings`` object, pass in a dictionary where each key and value are strings to ``LaunchSettings.update_env``. This function updates the existing dictionary of environment settings without overwriting it. For example: From ded7d087cdd74fb6fef121e7fba3eaac06674353 Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Fri, 1 Nov 2024 14:05:05 -0700 Subject: [PATCH 3/7] upgrade sphinx --- doc/launch_settings.rst | 32 +++++++++++--------------------- setup.py | 6 +++--- 2 files changed, 14 insertions(+), 24 deletions(-) diff --git a/doc/launch_settings.rst b/doc/launch_settings.rst index dfb8abb9a..32192291f 100644 --- a/doc/launch_settings.rst +++ b/doc/launch_settings.rst @@ -9,10 +9,9 @@ SmartSim’s ``LaunchSettings`` supports multiple launchers, allowing users to s Additionally, this class provides methods to access and configure these settings, as well as retrieve information associated with the ``LaunchSettings`` object. -**Dragon** is the fastest and most versatile distributed runtime available for HPC workflows. It can function as a -launcher within a Slurm allocation, providing rapid, interactive, and customizable execution of complex workflows -on large HPC systems. As a scheduler-agnostic solution, Dragon allows the same SmartSim script to run on both Slurm and -PBS systems, with future support for additional schedulers. +**Dragon** is a launcher which that enables fast, interactive, and customizable execution of complex +workflows on HPC systems. It works within a Slurm allocation, making it easy to run large workflows +efficiently on HPC machines. Additional launcher support is coming soon. =================== Supported Launchers @@ -80,7 +79,7 @@ runtime for HPC workflows. SLURM is a open-source job scheduler and resource man - **Description** * - ``srun`` - Simple Linux Utility for Resource Management (SLURM). - * - ``dragonrun`` + * - ``n/a`` - High-performance computing launcher for large-scale jobs (Dragon). Local @@ -191,7 +190,7 @@ varaible ``launcher`` to a launcher string such as `"slurm"`. For example: env_vars={"MY_VAR": "my_value"} ) -**Example using an Enum:** +**Example using a LauncherType Enum:** Once you have imported ``LaunchSettings`` and ``LauncherType`` using ``from smartsim import LaunchSettings, LauncherType``, set the input variable ``launcher`` to a ``LauncherType`` enum such as ``LauncherType.Slurm``. For example: @@ -203,9 +202,9 @@ set the input variable ``launcher`` to a ``LauncherType`` enum such as ``Launche env_vars={"MY_VAR": "my_value"} ) -========= -Configure -========= +====== +Modify +====== After initializing a ``LaunchSettings`` object, you might want to go back and configure ``launch_args`` or ``env_vars``. Configuring these settings allows you to fine-tune the execution environment to meet the specific needs of different jobs. @@ -222,12 +221,13 @@ These functions allow you to customize launch arguments after initializing the ` **Option 1: Use LaunchSettings.launch_args.set** To set additional launch arguments after initializing the ``LaunchSettings`` object, -use the `set` method on `launch_args` as shown below: +use the ``set`` method on ``launch_args`` as shown below: .. code-block:: python launch_settings.launch_args.set("--nodes", "2") +TODO: dragon example, syntanx is truly launcher specific **Option 2: Use custom HPC launcher methods** The ``LaunchSettings`` class provides custom methods to set launch arguments for different launchers. @@ -334,14 +334,4 @@ of environment settings without overwriting it. For example: .. code-block:: python - launch_settings.update_env({"MY_VAR": "new_value", "ANOTHER_VAR": "another_value"}) - -======== -Examples -======== - -Local -===== - -HPC -=== \ No newline at end of file + launch_settings.update_env({"MY_VAR": "new_value", "ANOTHER_VAR": "another_value"}) \ No newline at end of file diff --git a/setup.py b/setup.py index cd5ace55d..ccf0e504c 100644 --- a/setup.py +++ b/setup.py @@ -143,12 +143,12 @@ class BuildError(Exception): "typing_extensions>=4.1.0", ], "docs": [ - "Sphinx==6.2.1", + "Sphinx==8.1.3", "breathe==4.35.0", "sphinx-fortran==1.1.1", - "sphinx-book-theme==1.0.1", + "sphinx-book-theme", "sphinx-copybutton==0.5.2", - "sphinx-tabs==3.4.4", + "sphinx-tabs", "nbsphinx==0.9.3", "docutils==0.18.1", "torch==2.0.1", From 4e96e5aaddfa62ad27c5672d22405a167c01cc40 Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Fri, 1 Nov 2024 14:12:34 -0700 Subject: [PATCH 4/7] unpinn docutils --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index ccf0e504c..1e483ee28 100644 --- a/setup.py +++ b/setup.py @@ -150,7 +150,7 @@ class BuildError(Exception): "sphinx-copybutton==0.5.2", "sphinx-tabs", "nbsphinx==0.9.3", - "docutils==0.18.1", + "docutils", "torch==2.0.1", "tensorflow>=2.14,<3.0", "ipython", From 6c310ff3cb8559c9d3545f55b94f32796d89536a Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Fri, 1 Nov 2024 14:36:06 -0700 Subject: [PATCH 5/7] attempting to fix readthedocs --- .readthedocs.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 88f270ba7..997877703 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -7,6 +7,8 @@ build: tools: python: "3.10" jobs: + post_create_environment: + GRPC_VERBOSITY: "NONE" post_checkout: # Cancel building pull requests when there aren't changed in the docs directory or YAML file. # You can add any other files or directories that you'd like here as well, From cea3d6d73df558350efd0cb525d80ef6e3e32fa9 Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Sun, 3 Nov 2024 21:40:52 -0800 Subject: [PATCH 6/7] docs --- doc/launch_settings.rst | 61 ++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 31 deletions(-) diff --git a/doc/launch_settings.rst b/doc/launch_settings.rst index 32192291f..133ca62ff 100644 --- a/doc/launch_settings.rst +++ b/doc/launch_settings.rst @@ -4,21 +4,20 @@ LaunchSettings ======== Overview ======== -The ``LaunchSettings`` class manages launcher configuration settings to enable the injection of launcher-specific behavior into jobs. -SmartSim’s ``LaunchSettings`` supports multiple launchers, allowing users to set both launch arguments and environment variables. -Additionally, this class provides methods to access and configure these settings, as well as retrieve information associated with -the ``LaunchSettings`` object. +The ``LaunchSettings`` class manages launcher configuration settings to support launcher-specific behavior in jobs. +SmartSim supports multiple launchers and allows users to set launch arguments and environment variables. +Additionally, the ``LaunchSettings`` class provides methods to access and configure launch arguments and +environment variables. -**Dragon** is a launcher which that enables fast, interactive, and customizable execution of complex -workflows on HPC systems. It works within a Slurm allocation, making it easy to run large workflows -efficiently on HPC machines. Additional launcher support is coming soon. +**Dragon** is a launcher which that enables fast execution of complex workflows on HPC systems. It works within +a Slurm allocation, making it easy to run large workflows efficiently on HPC machines. =================== Supported Launchers =================== -The ``LaunchSettings`` class supports multiple launchers, each customized for different environments. Among these, -**Dragon** stands out as the fastest and most versatile option, ideal for large-scale HPC jobs. The following launchers -are categorized based on their specific use cases and the systems they are designed for. +The ``LaunchSettings`` class supports multiple launchers, each customized for different environments. Among the launchers, +**Dragon** stands out as fast and ideal for large-scale HPC jobs. The following launchers are categorized +based on the systems or programs they are designed for. HPE Cray ======== @@ -39,7 +38,7 @@ workload managers (WLMs) that do not have their own launchers. IBM === -Smartsim provides support for an IBM launcher: **LSF**. IBM's LSF is a platform for managing workloads and scheduling jobs +SmartSim provides support for an IBM launcher: **LSF**. IBM's LSF is a platform for managing workloads and scheduling jobs in distributed high-performance computing (HPC) environments. .. list-table:: IMB Specific Launchers @@ -100,13 +99,12 @@ various environments. ========== Initialize ========== -The ``LaunchSettings`` class allows you to set your application’s execution -environment. This section details the steps to set up the input parameters for -launching your application. +The ``LaunchSettings`` class allows you to customize your application’s execution +environment. This section details the steps to set up a ``LaunchSettings`` instance. **Step 1: Import LaunchSettings** -After installing Smartsim, ``LaunchSettings`` may be imported in Python code like: +After installing SmartSim, ``LaunchSettings`` may be imported in Python code like: .. code-block:: python @@ -114,8 +112,8 @@ After installing Smartsim, ``LaunchSettings`` may be imported in Python code lik **Step 2: Set the Launcher Type** -Set the launcher type using either a string or a ``LauncherType`` enum. This step is crucial as it -determines the applied launcher. The following table lists the supported launcher strings and +Set the launcher type using either a string or a ``LauncherType`` enum. This step is important as it +determines the job's launcher. The following table lists the supported launcher strings and their corresponding ``LauncherType`` enums: .. list-table:: Supported Launcher Strings and Enums @@ -160,7 +158,7 @@ To use an enum, import ``LauncherType`` into Python code like: **Step 3: Provide Launch Arguments and Environment Variables** -Optionally, you can provide ``launch_args`` and ``env_vars`` to customize the application's execution environment: +Optionally, you can provide ``launch_args`` and ``env_vars``: * ``launch_args``: A dictionary where keys are argument names (strings) and values are argument values (strings). These arguments are specific to the launcher being used. Example: @@ -180,7 +178,7 @@ Here's how you can initialize ``LaunchSettings`` with input parameters: **Example using a launcher String:** Once you have imported ``LaunchSettings`` using ``from smartsim import LaunchSettings``, set the input -varaible ``launcher`` to a launcher string such as `"slurm"`. For example: +variable ``launcher`` to a launcher string such as `"slurm"`. For example: .. code-block:: python @@ -205,13 +203,13 @@ set the input variable ``launcher`` to a ``LauncherType`` enum such as ``Launche ====== Modify ====== -After initializing a ``LaunchSettings`` object, you might want to go back and configure ``launch_args`` or -``env_vars``. Configuring these settings allows you to fine-tune the execution environment -to meet the specific needs of different jobs. +After initializing a ``LaunchSettings`` object, you might want to go back and modify the ``launch_args`` or +``env_vars``. Configuring these settings allows you to change the execution environment to meet the +requirements of different jobs. Launch Arguments ================ -There are two methods to configure ``launch_args``: +There are two methods to modify ``launch_args``: 1. Use ``LaunchSettings.launch_args.set``. 2. Use custom methods specific to each HPC launcher. @@ -220,14 +218,15 @@ These functions allow you to customize launch arguments after initializing the ` **Option 1: Use LaunchSettings.launch_args.set** -To set additional launch arguments after initializing the ``LaunchSettings`` object, -use the ``set`` method on ``launch_args`` as shown below: +To add or modify launch arguments after initializing the ``LaunchSettings`` object, +use the ``set`` method on ``launch_args`` as demonstrated below: .. code-block:: python launch_settings.launch_args.set("--nodes", "2") TODO: dragon example, syntanx is truly launcher specific + **Option 2: Use custom HPC launcher methods** The ``LaunchSettings`` class provides custom methods to set launch arguments for different launchers. @@ -313,14 +312,14 @@ Environment Variables ===================== There are two methods to configure ``env_vars``: -1. Use ``LaunchSettings.env_vars`` to overwrite the existing executable arguments. -2. Use ``LaunchSettings.update_env`` to add to the existing environment variables. +1. Use ``LaunchSettings.env_vars`` to overwrite the environment variables. +2. Use ``LaunchSettings.update_env`` to add environment variables. **Option 1: Use LaunchSettings.env_vars** To overwrite the ``env_vars`` after initializing the ``LaunchSettings`` object, set a dictionary where -each key and value are strings to ``LaunchSettings.env_vars``. This method overwrites the existing dictionary -of environment settings. For example: +each key and value are strings to ``LaunchSettings.env_vars``. This method overwrites the existing +environment settings. For example: .. code-block:: python @@ -329,8 +328,8 @@ of environment settings. For example: **Option 1: Use LaunchSettings.update_env** To update the ``env_vars`` after initializing the ``LaunchSettings`` object, pass in a dictionary where -each key and value are strings to ``LaunchSettings.update_env``. This function updates the existing dictionary -of environment settings without overwriting it. For example: +each key and value are strings to ``LaunchSettings.update_env``. This function updates the existing +environment settings without overwriting. For example: .. code-block:: python From 25da2d6a0ae49a56f157d44c200426ff9e6db07c Mon Sep 17 00:00:00 2001 From: Amanda Richardson Date: Sun, 3 Nov 2024 23:12:06 -0800 Subject: [PATCH 7/7] remove fail on warning --- .readthedocs.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 997877703..6eb70f2ff 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -38,4 +38,4 @@ build: # Build documentation in the "docs/" directory with Sphinx sphinx: configuration: doc/conf.py - fail_on_warning: true + fail_on_warning: false