Set up automatic execution of training material WFs...

[nekrut]

...against main using Jenkins and Planemo. This tutorial is a good start.

[hexylena]

We have some stuff for this already! https://github.com/usegalaxy-eu/workflow-testing

We're going to expand it to include automatic testing of all workflows from the training materials, but had not had the time to do that due to other more time sensitive issues.

[martenson]

@erasche yeah, we were hoping to piggyback :)

[hexylena]

@martenson awesomesauce!

[luke-c-sargent]

current status:

• forked the linked repo from @erasche (thanks!)
• added a Jenkinsfile and registered a Jenkins job named automated workflow testing in the test_jobs tab of the job list
• adapted the .travis.yml's approach of getting a planemo test script, setting the env, and providing a workflow file
• running against test.galaxyproject.org

some issues:

• Jenkins job is failing as it is unable to see the credentials i created for that purpose; there are probably some scoping issues i am not getting.
• not really sure whats happening with the actual test; as best i can tell, passing the workflow file to the script anchors it in the workflow's directory, making planemo aware of the .yml job and test files (which i have yet to write for my trivial example)

any clarification, suggestions or well-meaning admonishments welcome

[bgruening]

Hi @luke-c-sargent!

How does your jenkins script look like? We are running this here:

pip install https://github.com/galaxyproject/planemo/archive/master.zip
pip install galaxy-parsec
export PLANEMO_OPTIONS="--verbose"
bash fetch-urls.sh
bash run_galaxy_workflow_tests.sh $WORKFLOW

WORKFLOW is the path to the folder of every workflow in this case.

[hexylena]

If you use our run_galaxy_workflow_tests.sh script there are a couple of fun bonuses:

• We write out some HTML files that are available to users
  • the tool test output is made visible (but this is currently quite bare in planemo)
  • And we generate an IFrame with a link to the history that was created, so you can see the actual tools that were executed, which is sometimes useful.

[luke-c-sargent]

using run_galaxy_workflow_tests.sh on example1, i get "execution_problem": "'file_name'" -- not an especially telling error.

cmd: ./run_galaxy_workflow_tests.sh example1/wf3-shed-tools.ga

this creates a history, e.g. example1/wf3-shed-tools.ga 1540243522, and uploads the precursor datasets Genes.bed & GSE37268_mof3.out.hpeak.txt, then throws the above error without executing the workflow. i feel like i must be missing something obvious.

thanks!

(running on Mac OS -- taking Jenkins out of the equation for proof-of-concept)

[hexylena]

Have you tried running just the planemo command? I.e. ignoring all of the fancy things from our script?

planemo test \
	--galaxy_url "https://usegalaxy.eu" \
	--galaxy_user_key "$GALAXY_USER_KEY" \
    --no_shed_install \
	--engine external_galaxy \
    example1/wf3-shed-tools.ga

[luke-c-sargent]

same deal :(
Test case [<planemo.runnable.TestCase object at 0x1053009d0>] resulted in run response [Run failed with message ['file_name']]

from

planemo -v test \
        --history_name "a_test_history" \
        --galaxy_url "https://test.galaxyproject.org" \
        --galaxy_user_key "<REDACTED>" --no_shed_install \
        --engine external_galaxy \
        example1/wf3-shed-tools.ga;

which does indeed create a history on test with the input data and uploads the requisite workflow. this also happens when testing against main.

also tested example1 and a custom workflow against usegalaxy.eu (as user sargentl if you happen to feel like looking in to it) -- first execution stalled, so i ctrl+c'd it, added a -v flag and tried again and it went through to completion (the 'stall' could have just been impatience on my part). further executions worked as expected.

i wonder if it's a 18.09 issue -- what version does usegalaxy.eu run?

tl;dr: usegalaxy.org & test.galaxyproject.org give this file_name error, usegalaxy.eu works with example workflows and my own custom workflow

[hexylena]

We run 18.05 still, you can always check at https://usegalaxy.eu/api/version

Sounds like potentially an 18.09 specific issue.

[luke-c-sargent]

to delve deeper i ran two local versions of galaxy (18.05 and 18.09) and both threw:

Test case [<planemo.runnable.TestCase object at 0x10549c9d0>] resulted in run response [Run failed with message ['file_name']]

error, so that seems to rule out it being a mere 18.09 issue. perhaps it's some sort of race condition? ping takes 60ms on usegalaxy.org and 170ms on usegalaxy.eu -- but that seems like a stretch. i will do some local server log digging to see what's what and report back.

[martenson]

I am able reproduce the error @luke-c-sargent reported and will debug it next week hopefully with help of @jmchilton since this smells planemo-related.