Implement Record Types (generalization of tuple types)

[jmchilton]

Extend collections in a direction compatible with CWL support some day and that can improve Galaxy until then.

Support Hierarchical Inputs to Tools and Workflows

Comment from @jxtx - "I was thinking we should introduce a new "grouping" element for record, which is like a repeat but with only one repeat. So just a grouping of parameters. In the workflow editor, you could connect two records with the same structure, but also connect the individual elements. So connectors become hierarchical.

[ ] Paired Record
    [ ] first : data[ext=fastq]
    [ ] second : data[ext=fastq]

One could make connections from and of the three [ ]."

Additional Reading:

My CWL WIP branch has the concept of non-file datasets - that capture things like simple, non-file types as well as Galaxy expression tools and WIP on using this to pass values around in a workflow. That whole concept probably needs to be tied into collections though and other datasets though - and right now it is only being used with simple things. common-workflow-lab@89e5b7b

[jxtx]

I'm not sure I understand where the distinction is between a non-file dataset and another type of parameter.

[jmchilton]

@jxtx The CWL branch implements a new dataset type - expression.json that has a JSON datatype associated it. An expression.json output for instance can be connected to an integer input to a tool. I use this to represent for instance tools that output numbers, strings, floats, etc... from CWL tools - though all the plumbing and stuff appears before the main CWL commit in that branch and can easily be applied directly to Galaxy tools.

I updated the description of this issue with your e-mail comment about hierarchal inputs - I think it is an excellent idea.

Update: I should say that I am reconsidering expression.json being implemented as a dataset and not a new kind of history / collection content. These values should perhaps just be JSON blobs dumped in the database that can be attached to datasets and collections. This is a lot more work to implement - but might be more clean ultimately.

[jmchilton]

WIP on "record" type collections @ common-workflow-lab@2841da4.

[jmchilton]

Initial implementation merged as part of #19377 - follow up issue @ #20540.