Record types [preview build 2]

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

Author: bentsherman

docs/process-typed.md

@@ -94,27 +94,28 @@ Inputs with type `Record` can declare the name and type of each record field:
 ```nextflow
 process fastqc {
     input:
-    (id: String, fastq: Path): Record
+    sample: Record {
+        id: String
+        fastq: Path
+    }
 
     script:
     """
-    echo 'id: ${id}`
-    echo 'fastq: ${fastq}'
+    echo 'id: ${sample.id}`
+    echo 'fastq: ${sample.fastq}'
     """
 }
 ```
 
-This pattern is called *record destructuring*. Each record field is staged into the task the same way as an individual input.
+In this example, the record is staged into the task as `sample`, and `sample.fastq` is staged as an input file since the `fastq` field is declared with type `Path`.
 
 When the process is invoked, the incoming record should contain the specified fields, or else the run will fail. If the record has additional fields not declared by the process input, they are ignored.
 
 :::{tip}
 Record inputs are a useful way to select a subset of fields from a larger record. This way, the process only stages what it needs, allowing you to keep related data together in your workflow logic.
 :::
 
-### Record type inputs
-
-Record inputs can also be declared using a custom record type:
+You can achieve the same behavior using an external record type:
 
 ```nextflow
 process fastqc {
@@ -134,9 +135,7 @@ record Sample {
 }
 ```
 
-In this example, the record is staged into the task as `sample`, and `sample.fastq` is staged as an input file since the `fastq` field is declared with type `Path`.
-
-When the process is invoked, the incoming record should contain the fields specified by the record type, or else the run will fail. If the record has additional fields not declared by the record type, they are ignored.
+This approach is useful when the record type can be re-used elsewhere in the pipeline.
 
 ### Tuple inputs
 
@@ -311,10 +310,13 @@ The `record()` standard library function can be used to create a record:
 ```nextflow
 process fastqc {
     input:
-    (id: String, fastq: Path): Record
+    sample: Record {
+        id: String
+        fastq: Path
+    }
 
     output:
-    record(id: id, fastqc: file('fastqc_logs'))
+    record(id: sample.id, fastqc: file('fastqc_logs'))
 
     script:
     // ...

docs/tutorials/records.md

@@ -74,11 +74,15 @@ Is converted to:
 ```nextflow
 process ALIGN {
     input:
-    (meta: Map, reads: Path, index: Path): Record
+    sample: Record {
+        meta: Map
+        reads: Path
+        index: Path
+    }
 
     output:
     record(
-        meta: meta,
+        meta: sample.meta,
         bam: file('*.bam'),
         bai: file('*.bai')
     )
@@ -188,56 +192,67 @@ The `FASTQ` process is defined with the following inputs and outputs:
 
 ```nextflow
 process FASTQC {
-    // ...
+    tag id
+    conda 'bioconda::fastqc=0.12.1'
 
     input:
     (id, fastq_1, fastq_2): Tuple<String, Path, Path>
 
     output:
     tuple(id, file("fastqc_${id}_logs"))
 
-    // ...
+    script:
+    """
+    fastqc.sh "${id}" "${fastq_1} ${fastq_2}"
+    """
 }
 ```
 
 To migrate this process, rewrite the inputs and outputs as follows:
 
 ```nextflow
 process FASTQC {
-    // ...
+    tag sample.id
+    conda 'bioconda::fastqc=0.12.1'
 
     input:
-    (id: String, fastq_1: Path, fastq_2: Path): Record
+    sample: Record {
+        id: String
+        fastq_1: Path
+        fastq_2: Path
+    }
 
     output:
     record(
-        id: id,
-        fastqc: file("fastqc_${id}_logs")
+        id: sample.id,
+        fastqc: file("fastqc_${sample.id}_logs")
     )
 
-    // ...
+    script:
+    """
+    fastqc.sh "${sample.id}" "${sample.fastq_1} ${sample.fastq_2}"
+    """
 }
 ```
 
 In the above:
 
 - The tuple input is converted to a record input using the type `Record`. The field types are specified alongside the field names.
 
+- Since the record input cannot be destructured like a tuple, you must define a name for the record itself (e.g., `sample`), and you must update all references to tuple inputs (e.g., replace `id` with `sample.id`).
+
 - The tuple output is converted to a record by using the `record()` function and specifying a name for each record field.
 
 - Whereas tuple elements must be specified in a particular order, record fields can be specified in any order. The records supplied by the calling workflow must have the same field names and types as the process definition.
 
-:::{note}
-Other process sections, such as the directives and the `script:` block, are not shown because they do not require changes. As long as the inputs and outputs declare and reference the same variable names and file patterns, the other process sections will behave the same as before.
-:::
-
 <h4>QUANT</h4>
 
 The `QUANT` process is defined with the following inputs and outputs:
 
 ```nextflow
 process QUANT {
-    // ...
+    tag id
+    conda 'bioconda::salmon=1.10.3'
 
     input:
     (id, fastq_1, fastq_2): Tuple<String, Path, Path>
@@ -246,27 +261,50 @@ process QUANT {
     output:
     tuple(id, file("quant_${id}"))
 
-    // ...
+    script:
+    """
+    salmon quant \
+        --threads ${task.cpus} \
+        --libType=U \
+        -i ${index} \
+        -1 ${fastq_1} \
+        -2 ${fastq_2} \
+        -o quant_${id}
+    """
 }
 ```
 
 To migrate this process, rewrite the inputs and outputs as follows:
 
 ```nextflow
 process QUANT {
-    // ...
+    tag sample.id
+    conda 'bioconda::salmon=1.10.3'
 
     input:
-    (id: String, fastq_1: Path, fastq_2: Path): Record
+    sample: Record {
+        id: String
+        fastq_1: Path
+        fastq_2: Path
+    }
     index: Path
 
     output:
     record(
-        id: id,
-        quant: file("quant_${id}")
+        id: sample.id,
+        quant: file("quant_${sample.id}")
     )
 
-    // ...
+    script:
+    """
+    salmon quant \
+        --threads ${task.cpus} \
+        --libType=U \
+        -i ${index} \
+        -1 ${sample.fastq_1} \
+        -2 ${sample.fastq_2} \
+        -o quant_${sample.id}
+    """
 }
 ```
 

modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy

@@ -1768,7 +1768,7 @@ class TaskProcessor {
             final param = declaredInputs.getParams()[i]
             final value = values[i]
             if( param instanceof ProcessTupleInput )
-                assignTaskStructuredInput(task, param, value, i)
+                assignTaskTupleInput(task, param, value, i)
             else
                 assignTaskInput(task, param, value, i)
         }
@@ -1825,29 +1825,20 @@ class TaskProcessor {
     }
 
     @CompileStatic
-    private void assignTaskStructuredInput(TaskRun task, ProcessTupleInput param, Object value, int index) {
+    private void assignTaskTupleInput(TaskRun task, ProcessTupleInput param, Object value, int index) {
         if( value == null && !param.optional ) {
             throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} cannot be null -- append `?` to the type annotation to mark it as nullable")
         }
-        if( value instanceof List ) {
-            final tupleParams = param.getComponents()
-            final tupleValues = value as List
-            if( tupleParams.size() != tupleValues.size() ) {
-                throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a tuple with ${tupleParams.size()} elements but received ${tupleValues.size()} -- offending value: $tupleValues")
-            }
-            for( int i = 0; i < tupleParams.size(); i++ ) {
-                assignTaskInput(task, tupleParams[i], tupleValues[i], index)
-            }
+        if( value !instanceof List ) {
+            throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a tuple but received: ${value} [${value.class.simpleName}]")
         }
-        else if( value instanceof Map ) {
-            final record = value as Map
-            for( final fieldParam : param.getComponents() ) {
-                final fieldName = fieldParam.getName()
-                assignTaskInput(task, fieldParam, record[fieldName], index)
-            }
+        final tupleParams = param.getComponents()
+        final tupleValues = value as List
+        if( tupleParams.size() != tupleValues.size() ) {
+            throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a tuple with ${tupleParams.size()} elements but received ${tupleValues.size()} -- offending value: $tupleValues")
         }
-        else {
-            throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a record or tuple but received: ${value} [${value.class.simpleName}]")
+        for( int i = 0; i < tupleParams.size(); i++ ) {
+            assignTaskInput(task, tupleParams[i], tupleValues[i], index)
         }
     }
 

modules/nf-lang/src/main/antlr/ScriptParser.g4

@@ -240,10 +240,21 @@ processInputs
 
 processInput
     :   identifier (COLON type)?
-    |   LPAREN nls nameTypePair (nls COMMA nls nameTypePair)* nls rparen (COLON type)?
+    |   processRecordInput
+    |   processTupleInput
     |   statement
     ;
 
+processRecordInput
+    :   identifier (COLON type)? nls LBRACE
+        nls recordBody?
+        nls RBRACE
+    ;
+
+processTupleInput
+    :   LPAREN identifier (COMMA identifier)* rparen (COLON type)?
+    ;
+
 processStage
     :   STAGE COLON nls statement (sep statement)*
     ;

modules/nf-lang/src/main/java/nextflow/script/ast/TupleParameter.java

@@ -15,8 +15,6 @@
  */
 package nextflow.script.ast;
 
-import java.util.List;
-
 import org.codehaus.groovy.ast.ClassNode;
 import org.codehaus.groovy.ast.Parameter;
 
@@ -33,9 +31,4 @@ public TupleParameter(ClassNode type, Parameter[] components) {
         super(type, "");
         this.components = components;
     }
-
-    public boolean isRecord() {
-        return "Record".equals(getType().getUnresolvedName())
-            || "Record".equals(getType().getNameWithoutPackage());
-    }
 }

modules/nf-lang/src/main/java/nextflow/script/control/ProcessToGroovyVisitorV2.java

@@ -24,6 +24,7 @@
 import nextflow.script.ast.AssignmentExpression;
 import nextflow.script.ast.ProcessNodeV2;
 import nextflow.script.ast.RecordNode;
+import nextflow.script.ast.ScriptNode;
 import nextflow.script.ast.TupleParameter;
 import org.codehaus.groovy.ast.ClassHelper;
 import org.codehaus.groovy.ast.ClassNode;
@@ -56,10 +57,13 @@ public class ProcessToGroovyVisitorV2 {
 
     private SourceUnit sourceUnit;
 
+    private ScriptNode moduleNode;
+
     private ScriptToGroovyHelper sgh;
 
     public ProcessToGroovyVisitorV2(SourceUnit sourceUnit) {
         this.sourceUnit = sourceUnit;
+        this.moduleNode = (ScriptNode) sourceUnit.getAST();
         this.sgh = new ScriptToGroovyHelper(sourceUnit);
     }
 
@@ -149,6 +153,8 @@ private void visitProcessInputType(Variable param, Expression target, BlockState
             stagers.addStatement(stager);
         }
         else if( isRecordType(cn) ) {
+            if( cn.getNameWithoutPackage().startsWith("__Record") )
+                moduleNode.addClass(cn);
             for( var fn : cn.getFields() )
                 visitProcessInputType(fn, propX(target, fn.getName()), stagers);
         }

modules/nf-lang/src/main/java/nextflow/script/control/ScriptResolveVisitor.java

@@ -149,11 +149,10 @@ private void resolveTypedOutputs(Statement block) {
     @Override
     public void visitProcessV2(ProcessNodeV2 node) {
         for( var input : node.inputs ) {
-            resolver.resolveOrFail(input.getType(), input);
-            if( input instanceof TupleParameter tp && tp.isRecord() ) {
-                for( var component : tp.components )
-                    resolver.resolveOrFail(component.getType(), component);
-            }
+            var type = input.getType();
+            resolver.resolveOrFail(type, input);
+            if( type.getNameWithoutPackage().startsWith("__Record") )
+                visitRecord((RecordNode) type.redirect());
         }
         resolver.visit(node.directives);
         resolver.visit(node.stagers);