Skip to content

Commit 4069f66

Browse files
authored
Merge branch 'master' into 6570-slurm-job-state-query
2 parents 4dcbed2 + 7c40ee5 commit 4069f66

File tree

26 files changed

+192
-127
lines changed

26 files changed

+192
-127
lines changed

docs/reference/cli.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ Available options:
3838

3939
`-q, -quiet`
4040
: Do not print the Nextflow banner and execution progress to the console.
41-
: Does not affect messages printed by the pipeline or error messages.
41+
: Does not affect error messages.
4242

4343
`-remote-debug`
4444
: Enable JVM interactive remote debugging (experimental).

docs/reference/process.md

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -290,9 +290,6 @@ The following options are available for all process outputs:
290290

291291
### accelerator
292292

293-
:::{versionadded} 19.09.0-edge
294-
:::
295-
296293
The `accelerator` directive allows you to request hardware accelerators (e.g. GPUs) for the task execution. For example:
297294

298295
```nextflow
@@ -313,17 +310,23 @@ This directive is only used by certain executors. Refer to the {ref}`executor-pa
313310
:::
314311

315312
:::{note}
316-
Additional options may be required to fully enable the use of accelerators. When using containers with GPUs, you must pass the GPU drivers through to the container. For Docker, this requires the option `--gpus all` in the docker run command. For Apptainer/Singularity, this requires the option `--nv`. The specific implementation details depend on the accelerator and container type being used.
313+
Additional options may be required to fully enable the use of accelerators. When using containers with GPUs, you must pass the GPU drivers through to the container. For Docker, this requires the option `--gpus all` in the `docker run` command. For Apptainer/Singularity, this requires the option `--nv`. The specific implementation details depend on the accelerator and container type being used.
317314
:::
318315

319-
:::{note}
320-
The accelerator `type` option depends on the target execution platform. Refer to the platform-specific documentation for details on the available accelerators:
316+
The following options are available:
321317

322-
- [Google Cloud](https://cloud.google.com/compute/docs/gpus/)
323-
- [Kubernetes](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-gpus)
318+
`request: Integer`
319+
: The number of requested accelerators.
320+
: Specifying this directive with a number (e.g., `accelerator 4`) is equivalent to the `request` option (e.g., `accelerator request: 4`).
324321

325-
The accelerator `type` option is not supported for AWS Batch. You can control the accelerator type indirectly through the allowed instance types in your Compute Environment. See the [AWS Batch FAQs](https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_) for more information.
326-
:::
322+
`type: String`
323+
: The accelerator type.
324+
: The meaning of this option depends on the target execution platform. See the platform-specific documentation for more information about the available accelerators:
325+
326+
- [Google Cloud](https://cloud.google.com/compute/docs/gpus/)
327+
- [Kubernetes](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-gpus)
328+
329+
: This option is not supported for AWS Batch. You can control the accelerator type indirectly through the allowed instance types in your Compute Environment. See the [AWS Batch FAQs](https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_) for more information.
327330

328331
(process-afterscript)=
329332

docs/reports.md

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -156,27 +156,27 @@ nextflow run <pipeline> -with-trace
156156

157157
It will create a file named `trace.txt` in the current directory. The content looks like the above example:
158158

159-
| task_id | hash | native_id | name | status | exit | submit | duration | walltime | %cpu | rss | vmem | rchar | wchar |
160-
| ------- | --------- | --------- | -------------- | --------- | ---- | ----------------------- | -------- | -------- | ------ | -------- | -------- | -------- | -------- |
161-
| 19 | 45/ab752a | 2032 | blast (1) | COMPLETED | 0 | 2014-10-23 16:33:16.288 | 1m | 5s | 0.0% | 29.8 MB | 354 MB | 33.3 MB | 0 |
162-
| 20 | 72/db873d | 2033 | blast (2) | COMPLETED | 0 | 2014-10-23 16:34:17.211 | 30s | 10s | 35.7% | 152.8 MB | 428.1 MB | 192.7 MB | 1 MB |
163-
| 21 | 53/d13188 | 2034 | blast (3) | COMPLETED | 0 | 2014-10-23 16:34:17.518 | 29s | 20s | 4.5% | 289.5 MB | 381.6 MB | 33.3 MB | 0 |
164-
| 22 | 26/f65116 | 2035 | blast (4) | COMPLETED | 0 | 2014-10-23 16:34:18.459 | 30s | 9s | 6.0% | 122.8 MB | 353.4 MB | 33.3 MB | 0 |
165-
| 23 | 88/bc00e4 | 2036 | blast (5) | COMPLETED | 0 | 2014-10-23 16:34:18.507 | 30s | 19s | 5.0% | 195 MB | 395.8 MB | 65.3 MB | 121 KB |
166-
| 24 | 74/2556e9 | 2037 | blast (6) | COMPLETED | 0 | 2014-10-23 16:34:18.553 | 30s | 12s | 43.6% | 140.7 MB | 432.2 MB | 192.7 MB | 182.7 MB |
167-
| 28 | b4/0f9613 | 2041 | exonerate (1) | COMPLETED | 0 | 2014-10-23 16:38:19.657 | 1m 30s | 1m 11s | 94.3% | 611.6 MB | 693.8 MB | 961.2 GB | 6.1 GB |
168-
| 32 | af/7f2f57 | 2044 | exonerate (4) | COMPLETED | 0 | 2014-10-23 16:46:50.902 | 1m 1s | 38s | 36.6% | 115.8 MB | 167.8 MB | 364 GB | 5.1 GB |
169-
| 33 | 37/ab1fcc | 2045 | exonerate (5) | COMPLETED | 0 | 2014-10-23 16:47:51.625 | 30s | 12s | 59.6% | 696 MB | 734.6 MB | 354.3 GB | 420.4 MB |
170-
| 31 | d7/eabe51 | 2042 | exonerate (3) | COMPLETED | 0 | 2014-10-23 16:45:50.846 | 3m 1s | 2m 6s | 130.1% | 703.3 MB | 760.9 MB | 1.1 TB | 28.6 GB |
171-
| 36 | c4/d6cc15 | 2048 | exonerate (6) | COMPLETED | 0 | 2014-10-23 16:48:48.718 | 3m 1s | 2m 43s | 116.6% | 682.1 MB | 743.6 MB | 868.5 GB | 42 GB |
172-
| 30 | 4f/1ad1f0 | 2043 | exonerate (2) | COMPLETED | 0 | 2014-10-23 16:45:50.961 | 10m 2s | 9m 16s | 95.5% | 706.2 MB | 764 MB | 1.6 TB | 172.4 GB |
173-
| 52 | 72/41d0c6 | 2055 | similarity (1) | COMPLETED | 0 | 2014-10-23 17:13:23.543 | 30s | 352ms | 0.0% | 35.6 MB | 58.3 MB | 199.3 MB | 7.9 MB |
174-
| 57 | 9b/111b5e | 2058 | similarity (6) | COMPLETED | 0 | 2014-10-23 17:13:23.655 | 30s | 488ms | 0.0% | 108.2 MB | 158 MB | 317.1 MB | 9.8 MB |
175-
| 53 | 3e/bca30f | 2061 | similarity (2) | COMPLETED | 0 | 2014-10-23 17:13:23.770 | 30s | 238ms | 0.0% | 6.7 MB | 29.6 MB | 190 MB | 91.2 MB |
176-
| 54 | 8b/d45b47 | 2062 | similarity (3) | COMPLETED | 0 | 2014-10-23 17:13:23.808 | 30s | 442ms | 0.0% | 108.1 MB | 158 MB | 832 MB | 565.6 MB |
177-
| 55 | 51/ac19c6 | 2064 | similarity (4) | COMPLETED | 0 | 2014-10-23 17:13:23.873 | 30s | 6s | 0.0% | 112.7 MB | 162.8 MB | 4.9 GB | 3.9 GB |
178-
| 56 | c3/ec5f4a | 2066 | similarity (5) | COMPLETED | 0 | 2014-10-23 17:13:23.948 | 30s | 616ms | 0.0% | 10.4 MB | 34.6 MB | 238 MB | 8.4 MB |
179-
| 98 | de/d6c0a6 | 2099 | matrix (1) | COMPLETED | 0 | 2014-10-23 17:14:27.139 | 30s | 1s | 0.0% | 4.8 MB | 42 MB | 240.6 MB | 79 KB |
159+
| task_id | hash | native_id | name | status | exit | submit | duration | realtime | %cpu | peak_rss | peak_vmem | rchar | wchar |
160+
| ------- | --------- | --------- | -------------- | --------- | ---- | ----------------------- | -------- | -------- | ------ | --------- | --------- | -------- | -------- |
161+
| 19 | 45/ab752a | 2032 | blast (1) | COMPLETED | 0 | 2014-10-23 16:33:16.288 | 1m | 5s | 0.0% | 29.8 MB | 354 MB | 33.3 MB | 0 |
162+
| 20 | 72/db873d | 2033 | blast (2) | COMPLETED | 0 | 2014-10-23 16:34:17.211 | 30s | 10s | 35.7% | 152.8 MB | 428.1 MB | 192.7 MB | 1 MB |
163+
| 21 | 53/d13188 | 2034 | blast (3) | COMPLETED | 0 | 2014-10-23 16:34:17.518 | 29s | 20s | 4.5% | 289.5 MB | 381.6 MB | 33.3 MB | 0 |
164+
| 22 | 26/f65116 | 2035 | blast (4) | COMPLETED | 0 | 2014-10-23 16:34:18.459 | 30s | 9s | 6.0% | 122.8 MB | 353.4 MB | 33.3 MB | 0 |
165+
| 23 | 88/bc00e4 | 2036 | blast (5) | COMPLETED | 0 | 2014-10-23 16:34:18.507 | 30s | 19s | 5.0% | 195 MB | 395.8 MB | 65.3 MB | 121 KB |
166+
| 24 | 74/2556e9 | 2037 | blast (6) | COMPLETED | 0 | 2014-10-23 16:34:18.553 | 30s | 12s | 43.6% | 140.7 MB | 432.2 MB | 192.7 MB | 182.7 MB |
167+
| 28 | b4/0f9613 | 2041 | exonerate (1) | COMPLETED | 0 | 2014-10-23 16:38:19.657 | 1m 30s | 1m 11s | 94.3% | 611.6 MB | 693.8 MB | 961.2 GB | 6.1 GB |
168+
| 32 | af/7f2f57 | 2044 | exonerate (4) | COMPLETED | 0 | 2014-10-23 16:46:50.902 | 1m 1s | 38s | 36.6% | 115.8 MB | 167.8 MB | 364 GB | 5.1 GB |
169+
| 33 | 37/ab1fcc | 2045 | exonerate (5) | COMPLETED | 0 | 2014-10-23 16:47:51.625 | 30s | 12s | 59.6% | 696 MB | 734.6 MB | 354.3 GB | 420.4 MB |
170+
| 31 | d7/eabe51 | 2042 | exonerate (3) | COMPLETED | 0 | 2014-10-23 16:45:50.846 | 3m 1s | 2m 6s | 130.1% | 703.3 MB | 760.9 MB | 1.1 TB | 28.6 GB |
171+
| 36 | c4/d6cc15 | 2048 | exonerate (6) | COMPLETED | 0 | 2014-10-23 16:48:48.718 | 3m 1s | 2m 43s | 116.6% | 682.1 MB | 743.6 MB | 868.5 GB | 42 GB |
172+
| 30 | 4f/1ad1f0 | 2043 | exonerate (2) | COMPLETED | 0 | 2014-10-23 16:45:50.961 | 10m 2s | 9m 16s | 95.5% | 706.2 MB | 764 MB | 1.6 TB | 172.4 GB |
173+
| 52 | 72/41d0c6 | 2055 | similarity (1) | COMPLETED | 0 | 2014-10-23 17:13:23.543 | 30s | 352ms | 0.0% | 35.6 MB | 58.3 MB | 199.3 MB | 7.9 MB |
174+
| 57 | 9b/111b5e | 2058 | similarity (6) | COMPLETED | 0 | 2014-10-23 17:13:23.655 | 30s | 488ms | 0.0% | 108.2 MB | 158 MB | 317.1 MB | 9.8 MB |
175+
| 53 | 3e/bca30f | 2061 | similarity (2) | COMPLETED | 0 | 2014-10-23 17:13:23.770 | 30s | 238ms | 0.0% | 6.7 MB | 29.6 MB | 190 MB | 91.2 MB |
176+
| 54 | 8b/d45b47 | 2062 | similarity (3) | COMPLETED | 0 | 2014-10-23 17:13:23.808 | 30s | 442ms | 0.0% | 108.1 MB | 158 MB | 832 MB | 565.6 MB |
177+
| 55 | 51/ac19c6 | 2064 | similarity (4) | COMPLETED | 0 | 2014-10-23 17:13:23.873 | 30s | 6s | 0.0% | 112.7 MB | 162.8 MB | 4.9 GB | 3.9 GB |
178+
| 56 | c3/ec5f4a | 2066 | similarity (5) | COMPLETED | 0 | 2014-10-23 17:13:23.948 | 30s | 616ms | 0.0% | 10.4 MB | 34.6 MB | 238 MB | 8.4 MB |
179+
| 98 | de/d6c0a6 | 2099 | matrix (1) | COMPLETED | 0 | 2014-10-23 17:14:27.139 | 30s | 1s | 0.0% | 4.8 MB | 42 MB | 240.6 MB | 79 KB |
180180

181181
(trace-fields)=
182182

docs/snippets/ifempty-1.nf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Channel .of(1, 2, 3) .ifEmpty('Hello') .view()
1+
channel.of(1, 2, 3).ifEmpty('Hello').view()

docs/snippets/ifempty-2.nf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Channel .empty() .ifEmpty('Hello') .view()
1+
channel.empty().ifEmpty('Hello').view()

docs/strict-syntax.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -534,7 +534,7 @@ To enable these checks, set **Nextflow > Error reporting mode** to **paranoid**
534534

535535
### Using params outside the entry workflow
536536

537-
While params can be used anywhere in the pipeline code, they are only intended to be used in the entry workflow.
537+
While params can be used anywhere in the pipeline code, they are only intended to be used in the entry workflow and the `output` block.
538538

539539
As a best practice, processes and workflows should receive params as explicit inputs:
540540

docs/tutorials/static-types.md

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -352,13 +352,13 @@ While static types can be adopted progressively with existing code, some coding
352352

353353
This section provides best practices for writing Nextflow code that works well with static types.
354354

355-
<h3>Use the strict syntax</h3>
355+
### Use the strict syntax
356356

357357
The {ref}`strict syntax <strict-syntax-page>` remains optional in Nextflow 25.10. However, it is required to use type annotations.
358358

359359
Before you migrate to static types, ensure your code adheres to the strict syntax using `nextflow lint` or the language server.
360360

361-
<h3>Avoid deprecated patterns</h3>
361+
### Avoid deprecated patterns
362362

363363
When preparing for the strict syntax, try to address {ref}`deprecation warnings <strict-syntax-deprecated>` as much as possible. For example:
364364

@@ -373,7 +373,7 @@ The above example shows how to avoid three deprecated patterns:
373373
2. Using the deprecated `channel.from` factory (use `channel.of` or `channel.fromList` instead)
374374
3. Using the implicit `it` closure parameter (declare the parameter explicitly instead)
375375

376-
<h3>Avoid <code>set</code> and <code>tap</code> operators</h3>
376+
### Avoid `set` and `tap` operators
377377

378378
Nextflow provides three ways to assign a channel: a standard assignment, the `set` operator, and the `tap` operator:
379379

@@ -385,7 +385,28 @@ channel.of(10, 20, 30).tap { ch } // tap
385385

386386
However, the type checker cannot validate code that uses `set` or `tap`. Use standard assignments in your dataflow logic to enable full type checking.
387387

388-
<h3>Avoid <code>each</code> input qualifier</h3>
388+
### Avoid `|` and `&` operators
389+
390+
The {ref}`special operators <workflow-special-operators>` `|` and `&` provide shorthands for writing dataflow logic:
391+
392+
```nextflow
393+
channel.of('Hello', 'Hola', 'Ciao')
394+
| greet
395+
| map { v -> v.toUpperCase() }
396+
| view
397+
```
398+
399+
However, the type checker cannot validate code that uses these special operators. Use standard assignments and method calls instead:
400+
401+
```nextflow
402+
ch_input = channel.of('Hello', 'Hola', 'Ciao')
403+
ch_greet = greet(ch_input)
404+
ch_greet
405+
.map { v -> v.toUpperCase() }
406+
.view()
407+
```
408+
409+
### Avoid `each` input qualifier
389410

390411
The {ref}`each <process-input-each>` input qualifier is not supported by typed processes. Use the {ref}`operator-combine` operator to create a single tuple channel instead.
391412

@@ -436,7 +457,7 @@ workflow {
436457
The `each` qualifier is discouraged in modern Nextflow code. While it provides a convenient shorthand for combining multiple inputs, it couples the process definition with external workflow logic. Since the introduction of DSL2, Nextflow aims to treat processes as standalone modules that are decoupled from workflow logic.
437458
:::
438459

439-
<h3>Aviod functions with ambiguous return types</h3>
460+
### Avoid functions with ambiguous return types
440461

441462
Some {ref}`standard library <stdlib-types>` functions and {ref}`operators <operator-page>` have ambiguous return types. While you can still use them with static types, the type checker will not be able to fully validate your code.
442463

docs/workflow.md

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ workflow {
2828

2929
Parameters can be declared in a Nextflow script with the `params` block or with *legacy* parameter declarations.
3030

31-
### Params block
31+
### Typed parameters
3232

3333
:::{versionadded} 25.10.0
3434
:::
@@ -66,7 +66,7 @@ workflow {
6666
```
6767

6868
:::{note}
69-
As a best practice, parameters should only be used directly in the entry workflow and passed to workflows and processes as explicit inputs.
69+
As a best practice, parameters should only be referenced in the entry workflow or `output` block. Parameters can be passed to workflows and processes as explicit inputs.
7070
:::
7171

7272
The default value can be overridden by the command line, params file, or config file. Parameters from multiple sources are resolved in the order described in {ref}`cli-params`. Parameters specified on the command line are converted to the appropriate type based on the corresponding type annotation.
@@ -354,10 +354,16 @@ The same process can be called in different workflows without using an alias, li
354354
The fully qualified process name can be used as a {ref}`process selector <config-process-selectors>` in a Nextflow configuration file, and it takes priority over the simple process name.
355355
:::
356356

357+
(workflow-special-operators)=
358+
357359
## Special operators
358360

359361
The following operators have a special meaning when used in a workflow with process and workflow calls.
360362

363+
:::{note}
364+
As a best practice, avoid these operators when {ref}`type checking <preparing-static-types>` is enabled. Using these operators will prevent the type checker from validating your code.
365+
:::
366+
361367
### Pipe `|`
362368

363369
The `|` *pipe* operator can be used to chain processes, operators, and workflows:
@@ -375,7 +381,7 @@ process greet {
375381
}
376382
377383
workflow {
378-
channel.of('Hello','Hola','Ciao')
384+
channel.of('Hello', 'Hola', 'Ciao')
379385
| greet
380386
| map { v -> v.toUpperCase() }
381387
| view
@@ -388,9 +394,11 @@ The same code can also be written as:
388394

389395
```nextflow
390396
workflow {
391-
ch1 = channel.of('Hello','Hola','Ciao')
392-
ch2 = greet( ch1 )
393-
ch2.map { v -> v.toUpperCase() }.view()
397+
ch_input = channel.of('Hello', 'Hola', 'Ciao')
398+
ch_greet = greet(ch_input)
399+
ch_greet
400+
.map { v -> v.toUpperCase() }
401+
.view()
394402
}
395403
```
396404

modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -323,6 +323,8 @@ class CmdRun extends CmdBase implements HubOptions {
323323

324324
printBanner()
325325

326+
Plugins.init()
327+
326328
// -- resolve main script
327329
final scriptFile = getScriptFile(pipeline)
328330

@@ -359,7 +361,6 @@ class CmdRun extends CmdBase implements HubOptions {
359361
Map configParams = builder.getConfigParams()
360362

361363
// -- Load plugins (may register secret providers)
362-
Plugins.init()
363364
Plugins.load(config)
364365

365366
// -- Initialize real secrets system

modules/nextflow/src/main/groovy/nextflow/config/ConfigValidator.groovy

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,16 +106,13 @@ class ConfigValidator {
106106
names.clear()
107107

108108
if( value instanceof Map ) {
109-
log.debug "validate config block ${names}"
110109
if( isSelector(key) )
111110
names.removeLast()
112-
log.debug " is map option ${isMapOption(names)}"
113111
if( isMapOption(names) )
114112
continue
115113
validate(value, names)
116114
}
117115
else {
118-
log.debug "validate config option ${names}"
119116
validateOption(names)
120117
}
121118
}

0 commit comments

Comments
 (0)