External Tool Test Follow Up Work

[jmchilton]

xref #5628 (external tool testing is now possible - i.e. running tool tests on existing Galaxy instances running 18.05 or newer)

• galaxy-lib and Planemo integration for a pip installable scripting, pretty reports, etc... (WIP @ Implement external Galaxy engine. planemo#781, Bring in Galaxy updates to run externally-driven tool tests. galaxy-lib#91).
• ephemeris integration - Project: Run tests on tools updates ephemeris#76
• Implement tool test uploads - right now it depends on an admin key because the API endpoint exposes paths. It'd be cool if the upload tool for instance could just take test data URIs - (e.g. gxtestdata://<tool_id>?filename=<filename>. Then we could just use one user key to run the tests and skip the admin key.
• A problematic piece of the test-data code is the piece that expands tool test params into fully qualified params for consumption by the tool API (i.e. this stuff). This really should happen against the tool's JSON description on the client - this would allow running sort of externally defined tests as well with a similar syntax and behavior pattern. Planemo does allow external tool tests in JSON or YAML files next to tools - but the inputs are not treated the same way - the job definition must described the nesting of parameters and such (this is probably better anyway - so maybe it isn't worth the effort to export this code).

[martenson]

xref galaxyproject/galaxy-lib#137

[jmchilton]

We made enough progress on this that I'm removing it from the roadmap. Though clearly getting these actually run on main should still be a roadmap-level issue - but this issue doesn't describe that.