--- title: "Working with Slurm" author: "George G Vega Yon" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Working with Slurm} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` # Introduction Nowadays, high-performance-computing (HPC) clusters are commonly available tools for either in or out of cloud settings. [Slurm Work Manager](https://slurm.schedmd.com/) (formerly *Simple Linux Utility for Resource Manager*) is a program written in C that is used to efficiently manage resources in HPC clusters. The slurmR R package provides tools for using R in HPC settings that work with Slurm. It provides wrappers and functions that allow the user to seamlessly integrate their analysis pipeline with HPC clusters, putting emphasis on providing the user with a family of functions similar to those that the parallel R package provides. # Definitions First, some important discussion points within the context of Slurm+R that users in general will find useful. Most of the points have to do with options available for Slurm, and in particular, with the `sbatch` command with is used to submit batch jobs to Slurm. Users who have used Slurm in the past may wish to skip this and continue reading the following section. * **Node** A single computer in the HPC: A lot of times jobs will be submitted to a single node. The simplest way of using R+Slurm is submitting a single job and requesting multiple CPUs to use, for example, `parallel::parLapply` or `parallel::mclapply`. Usually users do not need to request an specific number of nodes to be used as Slurm will allocate the resources as needed. A common mistake of R users is to specify the number of nodes and expect that their script will be parallelized. This won't happen unless the user explicitly writes a parallel computing script. The relevant flag for `sbatch` is `--nodes`. * **Partition** A group of nodes in HPC. Generally large nodes may have multiple partitions, meaning that nodes may be grouped in various ways. For example, nodes belonging to a single group of users may be in a single partition, nodes dedicated to work with large data may be in another partition. Usually, partitions are associated with account privileges, so users may need to specify which account are they using when telling Slurm what partition they plan to use. The relevant flag for `sbatch` is `--partition`. * **Account** Accounts may be associated with partitions. Accounts can have privileges to use a partition or set of nodes. Often, users need to specify the account when submitting jobs to a particular partition. The relevant flag for `sbatch` is `--account`. * **Task** A step within a job. A particular job can have multiple tasks. tasks may span multiple nodes, so if the user wants to submit a multicore job, this option may not be the right one. The relevant flag for `sbatch` is `--ntasks` * **CPU** generally this refers to core or thread (which may be different in systems supporting multithreaded cores). Users may want to specify how many CPUs they want to use for a task. And this is the relevant option when using things like OpenMP or functions that allow creating cluster objects in R (e.g. `makePSOCKcluster`, `makeForkCluster`). The relevant option in `sbatch` is `--cpus-per-task`. More information regarding CPUs in Slurm can be found [here](https://slurm.schedmd.com/cpu_management.html). Information regarding how Slurm counts CPUs/cores/threads can be found [here](https://slurm.schedmd.com/faq.html#cpu_count). * **Job Array** Slurm supports job arrays. A job array is in simple terms a job that is repeated multiple times by Slurm, this is, replicates a single job as requested per the user. In the case of R, when using this option, a single R script is spanned in multiple jobs, so the user can take advantage of this and parallelize jobs across multiple nodes. Besides from the fact that jobs within a Job Array may be spanned across multiple nodes, each job in that array has a unique ID that is available to the user via environment variables, in particular `SLURM_ARRAY_TASK_ID`. Within R, and hence the Rscript submitted to Slurm, users can access this environment variable with `Sys.getenv("SLURM_ARRAY_TASK_ID")`. Some of the functionalities of `slurmR` rely on Job Arrays. More information on Job Arrays can be found [here](https://slurm.schedmd.com/job_array.html). The relevant option for this in `sbatch` is `--array`. More information about Slurm can be found their official website [here](https://slurm.schedmd.com/). A tutorial about how to use Slurm with R can be found [here](https://uscbiostats.github.io/slurmr-workshop/). # Submitting jobs via sbatch In general, users will submit jobs to Slurm using the `sbatch` command line function. The `sbatch` function's main argument is the name (path) to a bash script that holds the instructions (and sometimes options) associated to the program. Here is an example of an bash file to be submitted to Slurm ```bash #!/bin/bash #SBATCH --time=01:00:00 #SBATCH --job-name="A long job" #SBATCH --mem=5GB #SBATCH --output=long-job.out cd /path/where/to/start/the/job # This may vary per HPC system. At USC's hpc system # we use: source /usr/usc/R/default/setup.sh module load R Rscript --vanilla long-job-rscript.R ``` This example bash file, which we name "long-job-rscript.slurm", has the following components: * `#!/bin/bash` The interpreter directive that is common to bash scripts.[^shvsbash] * The `#SBATCH` lines specify options for scheduling the job. In order, these options are: Set a maximum time of 1 hour, name the job `A long job`, allocate 5GB of memory to the job, write all the output (including `Rscript`'s) to `long-job.out`. * The `cd` line changes the directory to some other place where the Rscript needs to be executed. * The `module` line loads R. There are various ways to do this, but it is a common requirement for the user to specify that it will be using R. * Finally, `Rscript` executes the R script named `long-job-rscript.R`. This batch script can be submitted to Slurm using the `sbatch` command line tool: ```bash $ sbatch long-job-rscript.slurm ``` This is what happens under-the-hood in `slurmR` overall. [^shvsbash]: For more on this see [this thread on StackExchange](https://askubuntu.com/questions/141928/what-is-the-difference-between-bin-sh-and-bin-bash).