GLS Compatibility

[lordkev]

Opening this issue per #209 (comment).

As mentioned, the main issue I had was with staging. For example, here a "dummy" file is passed:

eager/main.nf

Line 1553 in 916403a

def poststats = file('dummy_postfilterflagstat.stats')

Nextflow tries to stage this file but it doesn't exist, causing the job to fail. There's another one here that I ran into:

eager/main.nf

Line 1193 in 916403a

tuple samplename, libraryid, lane, seqtype, organism, strandedness, udg, file(r1), file(r2) from ch_lanemerge_for_bwa

When the data is single-ended and there is no r2, Nextflow tries to stage a weird malformed gs:// URL ("gs://gen-datawork9") that seems to be a combination of the bucket I'm using "gen-data" and the last portion of the workdir path "work9".

I'm not sure of the best way to solve the problem. Is there a way to pass a variable length tuple without throwing warnings? I suppose an alternative might be to pass a dummy file in baseDir that actually exists (but doesn't actually end up getting used)?

Happy to test as needed.

[jfy133]

Hey @lordkev could you post an example command and TSV input file? For the dummy poststats file maybe I've configured this wrong.

For the seccond issue, I wonder if it is my way of passing R1/R2s (when in paired exported to hte channel in an array (or collection), when single end it's just a single object), I wonder if the array script mangles the URL? Like you said, might be safer to use a dummy file there as well.

[lordkev]

Sure, here's an example file. (Renamed to txt extension for upload)

EX1.txt

Here's an example command:

../nextflow run ../eager/main.nf -profile gls --input EX1.tsv --fasta 'gs://genomics-public-data/references/hs37d5/hs37d5.fa.gz' --paired_end --mapper bwamem --outdir 'gs://gen-data/EX1/eagerwork1/out' -w 'gs://gen-data/EX1/eagerwork1' --tracedir ./EX1-eagertrace1 --skip_trim -resume

[lordkev]

Here's the error message for the poststats staging:

####################################################
## nf-core/eager execution completed unsuccessfully! ##

####################################################
The exit status of the task that caused the workflow execution to fail was: null.
The full error message was:

Error executing process > 'endorSpy (null)'

Caused by:
  Can't stage file file:///home/kevin/dummy_postfilterflagstat.stats -- file does not exist

[lordkev]

As for the r2 bug, the error message was:

CommandException: No URLs matched

Then by looking in the command.run file I was able to find the offending command in nxf_stage():

    gsutil -m -q cp gs://gen-datawork9 /EX1/work9/81/e833c71b758f227327e69d1fecd83f/work9

[jfy133]

Ok, so I guess it's best to physically create an empty dummy file for the postfilterflagstat thing.

r2, I'll need to think more about.

[jfy133]

I've tried the 'real' dummy file the postfilterflagstat thing in 03fd055

Can you try it and see if it works?

Note that --input is now slightly different because it also now accepts direct paths and wildcardglobs for FASTQ files, so keep an eye for an error and let us know!

[jfy133]

@lordkev just FYI, we 've been encountering similar(ish) issues with AWS (#473) so hopefully some of the issues will be resolved there. Also as a headsup, if you've not tried it yet, tower.nf now allows launching nextflow piplines directly to GLS for you. Maybe you can find it useful if you do furhter test.

[jfy133]

Had a successful AWS run the other day with -r fix-aws, after having very similar errors as reported above. Running few last checks on AWS and then will request test.

[jfy133]

@lordkev as the issue on AWS was the same as what you reported it seems (dummy file not being loaded correctly - replaced it from file() to path(), and fixed an array issue), we will close this for now. Please feel free to re-open if you continue to have problems.