Skip to content

Commit 43d69fc

Browse files
bentshermanbentsherman
committed
Allow hint values to be any raw data type
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
1 parent 03ff31c commit 43d69fc

17 files changed

Lines changed: 111 additions & 146 deletions

File tree

‎adr/20260323-hints-process-directive.md‎

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ The `hints` directive accepts a map of key-value pairs:
6363
process runDragen {
6464
cpus 4
6565
memory '16 GB'
66-
hints consumableResources: 'my-dragen-license=1,other-license=2'
66+
hints consumableResources: ['my-dragen-license': 1, 'other-license': 2]
6767
6868
script:
6969
"""
@@ -77,30 +77,30 @@ process runDragen {
7777
process {
7878
withName: 'runDragen' {
7979
hints = [
80-
consumableResources: 'my-dragen-license=1,other-license=2'
80+
consumableResources: ['my-dragen-license': 1, 'other-license': 2]
8181
]
8282
}
8383
}
8484
```
8585

86-
Both keys and values are arbitrary strings. Executors are responsible for defining which hints they recognize, as well as the expected structure for a given hint value. This approach keeps the `hints` directive simple (`Map<String,String>`) while allowing executors to structure hint values however they want (as long as it's a string).
86+
Keys are strings. Values may be any raw data type: strings, numbers, booleans, lists, or maps. Executors are responsible for defining which hints they recognize and what value type each hint expects.
8787

88-
In the above example, the `consumableResources` hint is given as a comma-separated string of `<name>=<count>` entries. The AWS Batch executor would parse this string into a map and supply it to each job request using `ConsumableResourceProperties`.
88+
In the above example, the `consumableResources` hint is given as a map of resource name to quantity. The AWS Batch executor supplies it to each job request using `ConsumableResourceProperties`.
8989

9090
### Namespacing
9191

9292
Keys can use dot-separated scopes to namespace settings as needed:
9393

9494
```groovy
95-
hints consumableResources: 'my-dragen-license=1'
95+
hints consumableResources: ['my-dragen-license': 1]
9696
hints 'scheduling.priority': 10
9797
hints 'scheduling.provisioningModel': 'spot'
9898
```
9999

100100
Keys can be routed to specific executors by prefixing with the executor name and a slash (`/`):
101101

102102
```groovy
103-
hints 'awsbatch/consumableResources': 'my-dragen-license'
103+
hints 'awsbatch/consumableResources': ['my-dragen-license': 1]
104104
hints 'seqera/scheduling.provisioningModel': 'spot'
105105
hints 'k8s/nodeSelector': 'gpu=true'
106106
```
@@ -126,7 +126,7 @@ process {
126126
127127
// specific hint replaces generic hint
128128
withLabel: 'dragen' {
129-
hints = [consumableResources: 'my-dragen-license=1']
129+
hints = [consumableResources: ['my-dragen-license': 1]]
130130
}
131131
}
132132
```
@@ -137,12 +137,12 @@ Within a process definition, the `hints` directive uses *accumulation semantics*
137137
process runDragen {
138138
// multiple separate hints
139139
hints provisioningModel: 'spot'
140-
hints consumableResources: 'my-dragen-license=1,other-license=2'
140+
hints consumableResources: ['my-dragen-license': 1, 'other-license': 2]
141141
142142
// equivalent to...
143143
hints (
144144
provisioningModel: 'spot',
145-
consumableResources: 'my-dragen-license=1,other-license=2'
145+
consumableResources: ['my-dragen-license': 1, 'other-license': 2]
146146
)
147147
148148
// ...
@@ -158,7 +158,7 @@ process {
158158
hints = [provisioningModel: 'spot']
159159
160160
withLabel: 'dragen' {
161-
hints = [provisioningModel: 'spot', consumableResources: 'my-dragen-license=1']
161+
hints = [provisioningModel: 'spot', consumableResources: ['my-dragen-license': 1]]
162162
}
163163
}
164164
```
@@ -169,11 +169,11 @@ While this approach may lead to duplication, it gives users and developers more
169169

170170
The following hints should be supported initially:
171171

172-
| Hint name | Executors | Use case |
173-
|--|--|--|
174-
| `consumableResources` | AWS Batch | License-aware scheduling ([#5917](https://github.com/nextflow-io/nextflow/issues/5917)) |
175-
| `scheduling.priority` | AWS Batch | Job scheduling priority ([#6998](https://github.com/nextflow-io/nextflow/issues/6998)) |
176-
| `scheduling.provisioningModel` | Google Batch | Spot VM scheduling ([#3530](https://github.com/nextflow-io/nextflow/issues/3530)) |
172+
| Hint name | Value type | Executors | Use case |
173+
|--|--|--|--|
174+
| `consumableResources` | `Map<String, Integer>` | AWS Batch | License-aware scheduling ([#5917](https://github.com/nextflow-io/nextflow/issues/5917)) |
175+
| `scheduling.priority` | `Integer` | AWS Batch | Job scheduling priority ([#6998](https://github.com/nextflow-io/nextflow/issues/6998)) |
176+
| `scheduling.provisioningModel` | `String` | Google Batch | Spot VM scheduling ([#3530](https://github.com/nextflow-io/nextflow/issues/3530)) |
177177

178178
## Links
179179

‎docs/executor.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ The following {ref}`hints <process-hints>` are supported:
3838
- `consumableResources`: Specify [AWS Batch consumable resources](https://docs.aws.amazon.com/batch/latest/userguide/resource-aware-scheduling.html) as a list of name-value pairs. For example:
3939

4040
```nextflow
41-
hints consumableResources: 'my-license-a=1,my-license-b=2'
41+
hints consumableResources: ['my-license-a': 1, 'my-license-b': 2]
4242
```
4343

4444
See {ref}`aws-batch` for more information.
@@ -463,7 +463,7 @@ The following {ref}`hints <process-hints>` are supported:
463463
- `machineRequirement.maxSpotAttempts`
464464
- `machineRequirement.provisioning`
465465

466-
Each hint overrides the corresponding field of the `seqera.executor.machineRequirement` config scope on a per-process basis. Values must be strings; keys may be used as-is or with the `seqera/` prefix to restrict them to this executor.
466+
Each hint overrides the corresponding field of the `seqera.executor.machineRequirement` config scope on a per-process basis. Keys may be used as-is or with the `seqera/` prefix to restrict them to this executor.
467467

468468
For example, to override the provisioning mode for a single process:
469469

‎docs/reference/process.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -844,13 +844,13 @@ The above example produces:
844844

845845
### hints
846846

847-
The `hints` directive specifies executor-specific hints as key-value pairs. Each executor uses the hints it recognizes and ignores the rest. Hint values must be strings.
847+
The `hints` directive specifies executor-specific hints as key-value pairs. Each executor uses the hints it recognizes and ignores the rest. Hint values can be any raw value -- numbers, strings, booleans, lists, and maps.
848848

849849
Unprefixed keys are available to **every** executor -- any executor that recognizes the key consumes it. Prefixing a key with an executor name (e.g. `awsbatch/...`) restricts the hint to that executor only. For example:
850850

851851
```nextflow
852852
process hello {
853-
hints consumableResources: 'my-license=1'
853+
hints consumableResources: ['my-license': 1]
854854
855855
script:
856856
"""
@@ -862,7 +862,7 @@ process hello {
862862
To restrict a hint to a single executor, prefix the key with the executor name:
863863

864864
```nextflow
865-
hints 'awsbatch/consumableResources': 'my-license=1'
865+
hints 'awsbatch/consumableResources': ['my-license': 1]
866866
```
867867

868868
When the same hint is provided both unprefixed and with a matching executor prefix, the prefixed form takes precedence for that executor.

‎modules/nextflow/src/main/groovy/nextflow/processor/HintDefs.groovy‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ import groovy.transform.CompileStatic
2424
* The core is intentionally agnostic about which hint keys are supported:
2525
* each executor validates the keys it recognizes (prefixed with its own
2626
* namespace, e.g. {@code awsbatch/...}, {@code seqera/...}). This class
27-
* only enforces that the map conforms to {@code Map<String,String>}.
27+
* only enforces that the map conforms to {@code Map<String,Object>} with
28+
* raw data type values.
2829
*
2930
* @author Paolo Di Tommaso <paolo.ditommaso@gmail.com>
3031
*/
@@ -40,17 +41,18 @@ class HintDefs {
4041
* <li>keys must be non-empty</li>
4142
* <li>keys may contain at most one {@code /} separating the optional
4243
* executor namespace from the hint name</li>
43-
* <li>values must be {@code String} or {@code null}</li>
44+
* <li>values must be a raw data type (String, Number, Boolean, List,
45+
* Map) or {@code null}</li>
4446
* </ul>
4547
*
4648
* @param hints the hint map to validate (may be {@code null})
4749
* @throws IllegalArgumentException if the map is malformed
4850
*/
49-
static void validateHints(Map<String, ?> hints) {
51+
static void validateHints(Map<String, Object> hints) {
5052
if( !hints )
5153
return
5254

53-
for( Map.Entry<String, ?> entry : hints.entrySet() ) {
55+
for( final entry : hints.entrySet() ) {
5456
final key = entry.key
5557
final value = entry.value
5658

@@ -60,9 +62,18 @@ class HintDefs {
6062
if( key.count('/') > 1 )
6163
throw new IllegalArgumentException("Invalid hint key '${key}': expected 'name' or 'executor/name'")
6264

63-
if( value != null && !(value instanceof CharSequence) )
64-
throw new IllegalArgumentException("Invalid hint value for key '${key}': expected String, got ${value.getClass().getName()}")
65+
if( !isValidHintValue(value) )
66+
throw new IllegalArgumentException("Invalid hint value for key '${key}': expected String, Number, Boolean, List, or Map, got ${value.getClass().getName()}")
6567
}
6668
}
6769

70+
private static boolean isValidHintValue(Object value) {
71+
return value == null
72+
|| value instanceof CharSequence
73+
|| value instanceof Number
74+
|| value instanceof Boolean
75+
|| value instanceof List
76+
|| value instanceof Map
77+
}
78+
6879
}

‎modules/nextflow/src/main/groovy/nextflow/processor/TaskConfig.groovy‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -529,8 +529,8 @@ class TaskConfig extends LazyMap implements Cloneable {
529529
return CmdLineOptionMap.emptyOption()
530530
}
531531

532-
Map<String, String> getHints() {
533-
return get('hints') as Map<String, String> ?: Collections.<String,String>emptyMap()
532+
Map<String, Object> getHints() {
533+
return get('hints') as Map<String, Object> ?: Collections.<String,Object>emptyMap()
534534
}
535535

536536
Map<String, String> getResourceLabels() {

‎modules/nextflow/src/main/groovy/nextflow/script/ProcessConfig.groovy‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,8 +173,8 @@ class ProcessConfig implements Map<String,Object>, Cloneable {
173173
HashMode.of(configProperties.cache) ?: HashMode.DEFAULT()
174174
}
175175

176-
Map<String,String> getHints() {
177-
(configProperties.get('hints') ?: Collections.emptyMap()) as Map<String, String>
176+
Map<String,Object> getHints() {
177+
(configProperties.get('hints') ?: Collections.emptyMap()) as Map<String, Object>
178178
}
179179

180180
Map<String,Object> getResourceLabels() {

‎modules/nextflow/src/main/groovy/nextflow/script/dsl/ProcessBuilder.groovy‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ class ProcessBuilder {
234234
*
235235
* @param map
236236
*/
237-
void hints(Map<String, String> map) {
237+
void hints(Map<String, Object> map) {
238238
if( !map ) return
239239
HintDefs.validateHints(map)
240240

‎modules/nextflow/src/test/groovy/nextflow/processor/HintDefsTest.groovy‎

Lines changed: 10 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -28,41 +28,30 @@ class HintDefsTest extends Specification {
2828
def 'should accept valid hints'() {
2929
when:
3030
HintDefs.validateHints([
31-
consumableResources: 'my-license=1',
32-
'awsbatch/consumableResources': 'a=1,b=2',
31+
consumableResources: ['my-license': 1],
32+
'awsbatch/consumableResources': ['a': 1, 'b': 2],
33+
'seqera/machineRequirement.diskEncrypted': true,
34+
'seqera/machineRequirement.machineTypes': ['m5', 'm6i'],
35+
'seqera/machineRequirement.priority': 10,
3336
'seqera/machineRequirement.provisioning': 'spot',
3437
])
3538
then:
3639
noExceptionThrown()
3740
}
3841

3942
def 'should accept null and empty maps'() {
40-
expect:
41-
HintDefs.validateHints(null) == null
42-
HintDefs.validateHints([:]) == null
43-
}
44-
45-
def 'should accept null hint value'() {
4643
when:
47-
HintDefs.validateHints([consumableResources: null])
44+
HintDefs.validateHints(null)
45+
HintDefs.validateHints([:])
4846
then:
4947
noExceptionThrown()
5048
}
5149

52-
def 'should reject non-string value'() {
53-
when:
54-
HintDefs.validateHints([consumableResources: 42])
55-
then:
56-
def e = thrown(IllegalArgumentException)
57-
e.message.contains("Invalid hint value")
58-
e.message.contains("consumableResources")
59-
}
60-
61-
def 'should reject list value'() {
50+
def 'should accept null hint value'() {
6251
when:
63-
HintDefs.validateHints([consumableResources: ['a', 'b']])
52+
HintDefs.validateHints([consumableResources: null])
6453
then:
65-
thrown(IllegalArgumentException)
54+
noExceptionThrown()
6655
}
6756

6857
def 'should reject closure value'() {

‎modules/nextflow/src/test/groovy/nextflow/processor/TaskConfigTest.groovy‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -629,15 +629,15 @@ class TaskConfigTest extends Specification {
629629
when:
630630
def process = new ProcessConfig(script)
631631
def dsl = new ProcessBuilder(process)
632-
dsl.hints( 'seqera/machineRequirement.arch': 'arm64', consumableResources: 'my-license=1' )
632+
dsl.hints( 'seqera/machineRequirement.arch': 'arm64', consumableResources: ['my-license': 1] )
633633

634634
then:
635-
process.get('hints') == ['seqera/machineRequirement.arch': 'arm64', consumableResources: 'my-license=1']
635+
process.get('hints') == ['seqera/machineRequirement.arch': 'arm64', consumableResources: ['my-license': 1]]
636636

637637
when:
638638
def config = process.createTaskConfig()
639639
then:
640-
config.getHints() == ['seqera/machineRequirement.arch': 'arm64', consumableResources: 'my-license=1']
640+
config.getHints() == ['seqera/machineRequirement.arch': 'arm64', consumableResources: ['my-license': 1]]
641641
}
642642

643643
def 'should return empty map when no hints set'() {
@@ -654,15 +654,15 @@ class TaskConfigTest extends Specification {
654654
when: 'set hints in process definition'
655655
def process = new ProcessConfig(script)
656656
def dsl = new ProcessBuilder(process)
657-
dsl.hints( 'seqera/machineRequirement.arch': 'arm64', consumableResources: 'my-license' )
657+
dsl.hints( 'seqera/machineRequirement.arch': 'arm64', consumableResources: ['my-license': 1] )
658658
then:
659-
process.getHints() == ['seqera/machineRequirement.arch': 'arm64', consumableResources: 'my-license']
659+
process.getHints() == ['seqera/machineRequirement.arch': 'arm64', consumableResources: ['my-license': 1]]
660660

661661
when: 'config override replaces the entire map'
662662
def config = process.createTaskConfig()
663-
config.put('hints', ['scheduling.priority': '5'])
663+
config.put('hints', ['scheduling.priority': 5])
664664
then:
665-
config.getHints() == ['scheduling.priority': '5']
665+
config.getHints() == ['scheduling.priority': 5]
666666
}
667667

668668
def 'should report error on negative cpus' () {

‎modules/nextflow/src/test/groovy/nextflow/script/dsl/ProcessBuilderTest.groovy‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -194,19 +194,25 @@ class ProcessBuilderTest extends Specification {
194194
config.getHints() == ['seqera/machineRequirement.arch': 'x86_64', 'seqera/machineRequirement.provisioning': 'spot', 'seqera/machineRequirement.maxSpotAttempts': '3']
195195
}
196196

197-
def 'should reject non-string hint values' () {
197+
def 'should reject closure hint values' () {
198198
given:
199199
def builder = createBuilder()
200200

201201
when:
202202
builder.hints 'seqera/machineRequirement.provisioning': { 'spot' }
203203
then:
204204
thrown(IllegalArgumentException)
205+
}
206+
207+
def 'should accept number and boolean hint values' () {
208+
given:
209+
def builder = createBuilder()
205210

206211
when:
207212
builder.hints 'seqera/machineRequirement.maxSpotAttempts': 3
213+
builder.hints 'seqera/machineRequirement.diskEncrypted': true
208214
then:
209-
thrown(IllegalArgumentException)
215+
noExceptionThrown()
210216
}
211217

212218
def 'should check a valid label' () {

0 commit comments

Comments
 (0)
⚡