Autodetect max number of cores and pass as an argument in Nextflow

Question

I am creating a pipeline in Nextflow. One step is creating a pangenome with Roary. Roary takes threads as an argument and if no number of threads is supplied as an argument it defaults to one.

Is there a way in Nextflow to pass the maximum number of available threads as an argument? At the moment the best I can come up with is defining an arbitrarily large number of threads as an argument which will cap out and run with the max number of threads. I would like to know, is there a 'cleaner' or more canonical way of doing this?

At the moment my process looks like this:


process roary {

    publishDir "${params.outdir}/roary", mode: 'copy'

    cpus 200

    input:
    file gff from gff.collect()

    output:
    file("*") into roary
    file("pan_genome_reference.fa") into pan_genome
    file("pan_genome_sequences/*") into alignment_files
    file("gene_presence_absence.Rtab") into gene_presence
    set file("*accessory*"),
        file("*.Rtab"),
        file("_*"),
        file("*.txt"),
        file("*csv") into roary_out


    script:
    """
    roary -p ${task.cpus} -e -n -v -z $gff
    """
}

Thanks in advance. Also, apologies as I can see why this might not strictly be bioinformatics, but it's where Nextflow has the most traction.

Workflow management has a firm foothold in bioinformatics and is in my opinion directly on topic on this site. — Daniel Standage, May 29 '19 at 13:01

score 5 · Accepted Answer · edited May 29 '19 at 13:57

5

I have not read the full Nextflow manual in depth, but I think there is no Nextflow command to detect the maximum number of threads you can use.

However, you can try to automatize this task by using the nproc command.

nproc gives you the number of processing units available. An idea would be to include something such nproc - 1 as the max number of threads for Roary.

edited May 29 '19 at 13:57

terdon

10,071
5
22
48

answered May 29 '19 at 12:20

plat

1,032
5
15

1

Thanks, as far I could see there are ways to dynamically allocate resources (https://www.nextflow.io/docs/latest/process.html#dynamic-computing-resources) but not detect threads, so this is a nice workaround. – TW93 May 29 '19 at 14:46

score 3 · Answer 2 · answered Apr 08 '20 at 00:54

Here's an example I wrote recently:

#!/usr/bin/env nextflow

params.requestedCPU = 80
params.requestedMem = 50

maxcpus = Runtime.runtime.availableProcessors()
requestedCpus = Math.round((params.requestedCPU * maxcpus) / 100)

maxmem = Runtime.runtime.totalMemory() / 10241024
requestedMem = (Math.round((params.requestedMem * maxmem) / 100))

process foo {
    echo true
    cpus = requestedCpus
    memory = requestedMem.GB

    script:
        """
        echo $requestedCpus
    echo $requestedMem
        """
}

You can pass the number of CPUs or Memory directly on the command line in percentage, e.g. nextflow run main.nf --requestedCPU 100 to have all CPU usages. Hope this helps.

Autodetect max number of cores and pass as an argument in Nextflow

2 Answers2