Merge branch 'main' into feat/beginners-tutorial

Author: SJaffa

docs/source/alternative_install.rst

@@ -0,0 +1,71 @@
+:orphan:
+
+.. _alternative_install:
+
+.. jinja:: first_ctx
+
+    #################################
+    Alternative installation methods
+    #################################
+
+    ******************
+    {{ app_name }} CLI
+    ******************
+
+    The {{ app_name }} CLI can be installed on Linux, macOS, and Windows through a terminal
+    or shell prompt:
+
+    .. tab-set::
+
+        .. tab-item:: Linux/macOS
+
+            Open a terminal, paste the command shown below and press enter.
+
+            .. code-block:: bash
+
+                (touch tmp.sh && curl -fsSL https://raw.githubusercontent.com/hpcflow/install-scripts/main/src/install-{{ app_package_name }}.sh > tmp.sh && bash tmp.sh --prerelease --path --univlink) ; rm tmp.sh
+
+        .. tab-item:: Windows
+
+            Open a Powershell terminal, paste the command shown below and press enter.
+
+            .. code-block:: powershell
+
+                & $([scriptblock]::Create((New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/hpcflow/install-scripts/main/src/install-{{ app_package_name }}.ps1'))) -PreRelease -UnivLink
+
+    .. admonition:: What does this script do?
+        :class: note dropdown
+        
+        The above command downloads a script from the {{ app_name }} GitHub repository and runs it. The script does the following:
+
+        #. It downloads the latest prerelease version of {{ app_name }} zip archived in a single folder.
+        #. The archive is extracted and the folder placed in an accessible location. The location depends on the operating system. In Linux it is ``/.local/share/{{ app_package_name }}``. In macOS it is ``~/Library/Application Support/{{ app_package_name }}``. In Windows it is ``Username\AppData\Local\{{ app_package_name }}``.
+        #. A symbolic link (Linux/macOS) or an alias pointing to the file is created. This allows {{ app_name }} to be run by entering a simple command.
+        #. A command is added to ``.bashrc``/``.zshrc`` (linux/macOS) or the Powershell profile (Windows) that allows {{ app_name }} to be run from any folder.
+
+        If the script detects that the version of {{ app_name }} it is trying to install is already there, it will stop 
+        running and exit.
+
+
+
+    .. hint::
+      
+      If you are installing {{ app_name }} on an HPC resource, check that you can connect
+      to the internet first. You might need to load a proxy module, for example.
+
+    
+    ********************************
+    Download CLI binaries (advanced)
+    ********************************
+
+    Binaries are available in two formats, corresponding to the two different formats that
+    PyInstaller `can generate <https://pyinstaller.org/en/stable/usage.html#what-to-generate>`_:
+
+    * A single executable file containing everything.
+    * A folder containing an executable and supporting files.
+
+    Click below to download the {{ app_name }} binary for your platform:
+
+    .. raw:: html
+
+        {{ download_links_table_html }}

docs/source/conf.py

@@ -229,6 +229,9 @@ def prepare_task_schema_action_info(app: BaseApp):
 with open("../../pyproject.toml") as fp:
     dist_name = tomlkit.load(fp)["tool"]["poetry"]["name"]
     supported_python_versions = tomlkit.load(fp)["tool"]["poetry"]["dependencies"]["python"]
+    pyproject_config = tomlkit.load(fp)
+    dist_name = pyproject_config["tool"]["poetry"]["name"]
+    supported_python = pyproject_config["tool"]["poetry"]["dependencies"]["python"]
 
 extensions = [
     "sphinx.ext.autodoc",
@@ -276,7 +279,7 @@ def prepare_task_schema_action_info(app: BaseApp):
         "download_links_table_html": generate_download_links_table(),
         "github_user": github_user,
         "github_repo": github_repo,
-        "supported_python": supported_python_versions,
+        "supported_python": supported_python,
     }
 }
 

docs/source/installation.rst

@@ -22,90 +22,69 @@
 
     The CLI and the Python package can be used simultaneously.
 
-    ******************
-    {{ app_name }} CLI
-    ******************
-
-    The {{ app_name }} CLI can be installed on Linux, macOS, and Windows through a terminal
-    or shell prompt:
-
-    .. tab-set::
-
-        .. tab-item:: Linux/macOS
-
-            Open a terminal, paste the command shown below and press enter.
-
-            .. code-block:: bash
-
-                (touch tmp.sh && curl -fsSL https://raw.githubusercontent.com/hpcflow/install-scripts/main/src/install-{{ app_package_name }}.sh > tmp.sh && bash tmp.sh --prerelease --path --univlink) ; rm tmp.sh
-
-        .. tab-item:: Windows
-
-            Open a Powershell terminal, paste the command shown below and press enter.
-
-            .. code-block:: powershell
-
-                & $([scriptblock]::Create((New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/hpcflow/install-scripts/main/src/install-{{ app_package_name }}.ps1'))) -PreRelease -UnivLink
-
-    .. admonition:: What does this script do?
-        :class: note dropdown
-        
-        The above command downloads a script from the {{ app_name }} GitHub repository and runs it. The script does the following:
-
-        #. It downloads the latest prerelease version of {{ app_name }} zip archived in a single folder.
-        #. The archive is extracted and the folder placed in an accessible location. The location depends on the operating system. In Linux it is ``/.local/share/{{ app_package_name }}``. In macOS it is ``~/Library/Application Support/{{ app_package_name }}``. In Windows it is ``Username\AppData\Local\{{ app_package_name }}``.
-        #. A symbolic link (Linux/macOS) or an alias pointing to the file is created. This allows {{ app_name }} to be run by entering a simple command.
-        #. A command is added to ``.bashrc``/``.zshrc`` (linux/macOS) or the Powershell profile (Windows) that allows {{ app_name }} to be run from any folder.
-
-        If the script detects that the version of {{ app_name }} it is trying to install is already there, it will stop 
-        running and exit.
-
-
-
-    .. hint::
-      
-      If you are installing {{ app_name }} on an HPC resource, check that you can connect
-      to the internet first. You might need to load a proxy module, for example.
-
-    .. _def_python_versions:
-
-    *****************************
-    {{ app_name }} Python package
-    *****************************
-
-    {{ app_name }} is compatible with Python versions {{ supported_python }}.
-
     Using pip
     ==========================
 
-    Use pip to install the Python package from PyPI::
-
-      pip install {{ dist_name }}=="{{ app_version }}"
+    The recommended way to install {{ app_name }} is to
+    use pip to install the Python package from PyPI::
 
-    Using conda
-    ===========
+      pip install {{ dist_name }}
 
-    Coming soon!
-
-    ********************************
-    Download CLI binaries (advanced)
-    ********************************
-
-    Binaries are available in two formats, corresponding to the two different formats that
-    PyInstaller `can generate <https://pyinstaller.org/en/stable/usage.html#what-to-generate>`_:
-
-    * A single executable file containing everything.
-    * A folder containing an executable and supporting files.
-
-    Click below to download the {{ app_name }} binary for your platform:
-
-    .. raw:: html
-
-        {{ download_links_table_html }}
+    This installs the python package, which also gives the CLI version of {{ app_name }}.
 
     *************
     Release notes
     *************
 
     Release notes for this version ({{app_version}}) are `available on GitHub <https://github.com/{{ github_user }}/{{ github_repo }}/releases/tag/v{{ app_version }}>`_.
     Use the version switcher in the top-right corner of the page to download/install other versions.
+
+    Alternative installation methods
+    ================================
+    Although *not currently recommended*,
+    advanced users may wish to use one of the :ref:`alternative installation methods <alternative_install>`.
+
+
+    #############
+    Configuration
+    #############
+
+    {{ app_name }} uses a config file to control details of how it executes workflows.
+    A :ref:`default config file <default_config>` will be created the first time you submit a workflow.
+    This will work without modification on a personal machine,
+    however if you are using {{ app_name }} on HPC you will likely need to make some
+    modifications to describe the job scheduler, and settings for multiple cores,
+    and to point to your {{ app_name }} environments file.
+
+    `Some examples <https://github.com/hpcflow/matflow-configs>`_ are given
+    for the University of Manchester's CSF.
+
+    The path to your config file can be found using ``{{ app_module }} manage get-config-path``,
+    or to open the config file directly, use ``{{ app_module }} open config``.
+
+    #############
+    Environments
+    #############
+
+    {{ app_name }} has the concept of environments, similar to python virtual environments.
+    These are required so that tasks can run using the specific software they require.
+    Your {{ app_name }} environments must be defined in your environments (YAML) file before {{ app_name }}
+    can run workflows, and this environment file must be pointed to in the config file
+    via the ``environment_sources`` key.
+    Once this has been done, your environment file can be be opened using ``{{ app_module }} open env-source``.
+
+    Below is an example environments file that defines an environment for running Pyton scripts.
+    Domain-specific tools can be added to the environments file as required, each with their own 
+    setup instructions for loading that tool on your machine.
+
+    .. code-block:: yaml
+
+      - name: python_env
+        executables:
+          - label: python_script
+            instances:
+        - command: python "<<script_path>>" <<args>>
+          num_cores:
+            start: 1
+            stop: 32
+          parallel_mode: null

docs/source/reference/config_file.rst

@@ -3,6 +3,8 @@ Configuration file
 
 Configurable settings that can be set within the configuration file are listed here.
 
+.. _default_config:
+
 Default config file
 ~~~~~~~~~~~~~~~~~~~
 

docs/source/reference/glossary.rst

@@ -1,7 +1,91 @@
-Glossary
-========
+.. jinja:: first_ctx
 
-workflow
---------
+  Glossary
+  ========
 
-A pipeline that processes data in some way.
+  API
+  ---
+
+  Application programming interface. {{ app_name }}'s API allows us to design and execute workflows from within a Python interpreter or Jupyter notebook.
+
+  .. _def_command_files:
+
+  Command files
+  -------------
+  If you want to refer to any files that are used as inputs or output,
+  they should be listed under ``command_files`` in the workflow file
+
+  .. code-block:: console
+
+      command_files:
+        - label: new_inp_file
+          name:
+            name: friction_conductance.inp
+
+
+  CLI
+  ---
+
+  Command line interface. The CLI is typically how we interact with {{ app_name }} on HPC systems.
+
+  cluster
+  -------
+
+  See HPC
+
+
+  Environment/virtual environment
+  -------------------------------
+
+  An environment is an isolated set of installed software. 
+  Using environments allows you to have multiple copies of the same software installed in different environments so you can run different versions, or to run two pieces of software with competing dependencies on the same machine. 
+  Using and sharing environments helps make your work reproducible because someone can use the same environment on a different machine and be sure they have the same versions of everything.
+
+
+  HPC
+  ---
+
+  High-performance computer/computing
+
+
+  jobscript
+  ---------
+
+  A job submission script that is used to queue a job on a batch scheduler system, such as SLURM or SGE.
+  Jobscripts are generated by {{ app_name }} during workflow submission.
+
+  .. _def_task:
+
+  Tasks
+  -------------
+  These are actual usages of a :ref:`task schema <def_task_schema>`, run with defined inputs.
+
+  .. _def_task_schema:
+
+  Task schema
+  -------------
+  This is a template for a task you want to run,
+  with definitions of the input and outputs that are expected.
+
+  {{ app_name }} has many :ref:`built-in task schemas <task_schemas>`, but you may want to write your own.
+
+  .. _def_workflow:
+
+  Workflow
+  --------
+
+  A pipeline that processes data in some way.
+  A workflow is a list of tasks that run one after the other.
+
+
+  .. _def_workflow_template:
+
+  Workflow template
+  ------------------
+
+  A workflow template parameterises a workflow,
+  providing the required input values for the task schemas of the workflow.
+  However, it doesn't actually run the :ref:`workflow <def_workflow>`.
+  A workflow template is usually just the list of tasks,
+  but can optionally include {{ app_name }} environment,
+  the :ref:`task schemas <def_task_schema>`, and the :ref:`command files <def_command_files>`.

docs/source/reference/index.rst

@@ -0,0 +1,12 @@
+Reference
+=========
+
+.. toctree::
+   :maxdepth: 2
+
+   Python API <api>
+   Command-line interface <cli>
+   Template components <template_components>
+   Demo workflows <workflows>
+   Configuration file <config_file>
+   Glossary <glossary>

docs/source/reference/template_components/task_schemas.rst

@@ -1,3 +1,5 @@
+.. _task_schemas:
+
 Task schemas
 ============