diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b601b89 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +*DS_Store \ No newline at end of file diff --git a/doc/cpp_bids-hcppipeline.slurm b/doc/cpp_bids-hcppipeline.slurm new file mode 100644 index 0000000..4b279e4 --- /dev/null +++ b/doc/cpp_bids-hcppipeline.slurm @@ -0,0 +1,101 @@ +#!/bin/bash + +#SBATCH --job-name=fMRIprep +#SBATCH --time=9:00:00 # hh:mm:ss + +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=9 +#SBATCH --mem-per-cpu=10000 # megabytes +#SBATCH --partition=batch,debug + +#SBATCH --mail-user=marco.barilari@uclouvain.be +#SBATCH --mail-type=ALL +#SBATCH --output=/home/ucl/irsp/marcobar/jobs_report/fmriprep_job-%j.txt + +#SBATCH --comment=project-name + +#export OMP_NUM_THREADS=9 +#export MKL_NUM_THREADS=9 + +## CPP frmiprep script for CECI cluster +# +# writtent by CPP people +# +# Submission command for Lemaitre3 +# +# sbatch cpp_fmriprep.slurm +# +# examples: +# - 1 subject 1 task +# sbatch cpp_fmriprep.slurm sub-01 visMotLocalizer +# +# - 1 subject all task +# sbatch cpp_fmriprep.slurm sub-01 '' +# +# - all subjects 1 task +# sbatch cpp_fmriprep.slurm '' visMotLocalizer +# +# - multiple subjects +# sbatch cpp_fmriprep.slurm 'sub-01 sub-02' visMotLocalizer +# +# - multiple tasks +# sbatch cpp_fmriprep.slurm sub-01 'visMotLocalizer audMotLocalizer' +# +# - submit all the subjects (1 per job) all at once +# read subj list to submit each to a job for all the tasks +# !!! to run from within `raw` folder +# ls -d sub* | xargs -n1 -I{} sbatch path/to/cpp_fmriprep.slurm {} '' + +# create jobs_report folder in case they don't exist +mkdir -p $HOME/jobs_report/ + +# fail whenever something is fishy +# -e exit immediately +# -x to get verbose logfiles +# -u to fail when using undefined variables +# https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html +set -e -x -u -o pipefail + +module --force purge + +subjID=$1 +TaskName=$2 + +# "latest" or procide specific version number +FMRIPREP_VERSION="23.1.4" + +# set username to locate scratch folder +ceci_username="marcobar" + +# set fmriprep arguments +nb_dummy_scans=0 + +path_to_singularity_image="$HOME/tools/containers/images/bids/bids-fmriprep--${FMRIPREP_VERSION}.sing" +scratch_dir="/scratch/users/m/a/${ceci_username}" +bids_dir="$HOME/fmriprep_trial/raw" +output_dir="$HOME/fmriprep_trial/derivatives/fmriprep" +freesurfer_license_folder="$HOME/tools/freesurfer_license" + +# make the scratch folder, here there is no limit space and fmriprep can store stuff in case of crash and do not start from zero again +mkdir -p "${scratch_dir}"/work-fmriprep + +# create output folder in case it does not exists +mkdir -p "${output_dir}" + +singularity run --cleanenv \ + -B "${scratch_dir}":/scratch_dir \ + -B "${bids_dir}":/bids_dir \ + -B "${output_dir}":/output \ + -B "${freesurfer_license_folder}":/freesurfer_license \ + "${path_to_singularity_image}" \ + /bids_dir \ + /output \ + participant --participant-label "${subjID}" \ + --task "${TaskName}" \ + --work-dir /scratch_dir/work-fmriprep \ + --fs-license-file /freesurfer_license/license.txt \ + --output-spaces MNI152NLin2009cAsym T1w \ + --dummy-scans ${nb_dummy_scans} \ + --notrack \ + --skip_bids_validation \ + --stop-on-first-crash diff --git a/doc/install_datalad.md b/doc/install_datalad.md index f67e5c5..25fe8d9 100644 --- a/doc/install_datalad.md +++ b/doc/install_datalad.md @@ -10,24 +10,25 @@ To contribute see [here](https://cpp-lln-lab.github.io/CPP_HPC/contributing/) pip3 install --user --upgrade pip ``` -- Follow the - [datalad handbook](http://handbook.datalad.org/en/latest/intro/installation.html#norootinstall) +- Following the guide from [datalad handbook](http://handbook.datalad.org/en/latest/intro/installation.html#norootinstall) installation for HPC (no root access) or copy-paster the code below. ```bash -# get anaconda and install it +# get anaconda wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh + +# install it bash Miniconda3-latest-Linux-x86_64.sh -conda install -c conda-forge datalad ``` +Check that conda is installed with `conda --version` and try again. + If this outputs that `conda` is command not found, try: ```bash -export PATH="/home/ucl/irsp/YOUR-USER-NAME/miniconda3/bin:$PATH" +source .bashrc ``` - -Check that conda is installed with `conda --version` and try again. +Install datalad ```bash conda install -c conda-forge datalad @@ -44,7 +45,13 @@ cd YOUR-CLONED-REPO datalad get . ``` -- If you need to create an ssh key follow this on +it won't probably work since ssh keys to github/gin/etc has to be set up. + +Set up the einviroment as you did on your personal computer and remebere that if your password is not working is is possible that github/gin/etc are expecting the token code you have to create for each machine. + +Some usefule links and notes (but not very tidy): + +If you need to create an ssh key follow this on [github website](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) or just type: diff --git a/doc/run_bids-hcppipelines.md b/doc/run_bids-hcppipelines.md new file mode 100644 index 0000000..49e599c --- /dev/null +++ b/doc/run_bids-hcppipelines.md @@ -0,0 +1,164 @@ +# Run fmriprep on the cluster + +Written by CPP lab people + +To contribute see [here](https://cpp-lln-lab.github.io/CPP_HPC/contributing/) + +## General tips + +- The more resources required, the faster it can be but the more waiting time + +- To try things, set `--time=00:05:00` and `--partition=debug` so it starts + right away and you can check if it at least starts without problems (eg the + singularity images is running, data are bids compatible or data folders are + loaded proprerly). See below in the section [Submit a fmriprep job via sbatch command](#submit-a-fmriprep-job-via-sbatch-command-without-a-script-mainly-for-debug-purposes) + +## Prepare to run fmriprep on the cluster + +- have your data on the cluster +- install datalad on your user (see [here](https://github.com/cpp-lln-lab/CPP_HPC/install_datalad)) +- get the fmriprep singularity image as follow: + +here the example is with `fmriprp version 21.0.1` but check for newer version, list of fmriprep version available [here](https://hub.docker.com/r/nipreps/fmriprep/tags/) + +```bash +datalad install https://github.com/ReproNim/containers.git + +cd containers + +datalad get images/bids/bids-fmriprep--21.0.1.sing +``` + +In case you have installe the repo a while a ago and you want to use a new version of fmriprep., update the `containers` repo via: + +```bash +# go to the repo folder +cd path/to/containers + +datald update --merge +`````` + +Depending on the cluster “unlock” is needed or not. No need for `lemaitre3`. + +```bash +datalad unlock containers/images/bids/bids-fmriprep--21.0.1.sing +``` + + - get your `freesurfer` license (user specific) for free [here](https://surfer.nmr.mgh.harvard.edu/registration.html) and move it to the cluster + +## Submit a fmriprep job via a `slurm` script + +- pros: + - easy to run for multiple subject +- cons: + - the `slurm` script can be hard to edit from within the cluster in case of error or a change of mind with fmriprep + options. You can edit via `vim` or locally and then + uploading a newversion. + +Content of the `cpp_fmriprep.slurm` file (download and edit from [here](cpp_fmriprep.slurm)) + +!!! Warning + + 1. Read the fmriprep documentation to know what you are doing and how the arguments of the run call effects the results + 2. All the paths and email are set afte Marco's users for demosntration. + 3. Edit the scripts with the info you need to make it run for your user from top to buttom of the script, do not over look the first "commented" chunk cause it is not a real commented section (check the email and job report path, data paths and the `username` etc.). + +```bash +{% include "cpp_fmriprep.slurm" %} +``` + +On the cluster prompt, submit the jobs as: + +```bash +# Submission command for Lemaitre3 + +# USAGE on cluster: + +sbatch cpp_fmriprep.slurm + +# examples: +# - 1 subject 1 task + +sbatch cpp_fmriprep.slurm sub-01 visMotLocalizer + +# - 1 subject all task +sbatch cpp_fmriprep.slurm sub-01 '' + +# - all subjects 1 task +sbatch cpp_fmriprep.slurm '' visMotLocalizer + +# - multiple subjects +sbatch cpp_fmriprep.slurm 'sub-01 sub-02' visMotLocalizer + +# - multiple tasks +sbatch cpp_fmriprep.slurm sub-01 'visMotLocalizer audMotLocalizer' + +# submit all the subjects (1 per job) all at once +# read subj list to submit each to a job for all the tasks +# !!! to run from within `raw` folder +ls -d sub* | xargs -n1 -I{} sbatch path/to/cpp_fmriprep.slurm {} '' +``` + +## Submit a fmriprep job via sbatch command without a script (mainly for DEBUG purposes) + +- pros: + - fast to edit and debug +- cons: + - if copy pasted in the terminal looses the lines structure so hard to edit + (use vscode ;) ) + - at the moment it only submit one subject per job + +```bash +# slurm job set up +sbatch --job-name=fmriprep_trial \ + --comment=cpp_cluster_hackaton \ + --time=42:00:00 \ + --ntasks=1 \ + --cpus-per-task=9 \ + --mem-per-cpu=10000 \ + --partition=batch,debug \ + --mail-user=marco.barilari@uclouvain.be \ + --mail-type=ALL \ + --output=fmriprep-slurm_{}-job-%j.txt + --wrap \ + “singularity run --cleanenv \ + -B /scratch/users/m/a/marcobar:/scratch \ + -B ~/sing_temp:/sing_temp \ + ~/sing_temp/containers/images/bids/bids-fmriprep--21.0.1.sing \ + /sing_temp/raw \ + /sing_temp/fmriprep \ + participant \ + --participant-label sub-01 \ + --work-dir /scratch/work-fmriprep \ + --fs-license-file /sing_temp/license.txt \ + --output-spaces MNI152NLin2009cAsym T1w \ + --notrack \ + --stop-on-first-crash” +``` + +## TIPS + +### check your job + +see [here](https://github.com/cpp-lln-lab/CPP_HPC/cluster_code_snippets/#check-your-running-jobs) + +### crashes + +- if fmriprep stops (eg timeout, error), rerunning the subject(s) might crash + due to the fact that freesurfer is not happy that parcellation started already + +### bids fiter file + +Add a `bids_filter_file.json` config file to help you define what fmriprep +should consider as `bold` as `T1w`. + +The one below corresponds to the fMRIprep default (also available inside this +repo). + +See this part of the FAQ for more info [here](https://fmriprep.org/en/21.0.2/faq.html#how-do-I-select-only-certain-files-to-be-input-to-fMRIPrep) + +```json +{% include "bids_filter_file.json" %} +``` + +you will need to add the argument `--bids-filter-file path/to/bids_filter_file.json` when running fmriprep. diff --git a/mkdocs.yml b/mkdocs.yml index fcaeb98..94749b3 100755 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - install datalad on the cluster: install_datalad.md - run fmriprep: run_fmriprep.md - run mriqc: run_mriqc.md + - run bids-HCPpipelines: run_bids-hcpipelines.md - Contributing: contributing.md - Archive: - fmriprep cluster workshop (2023):