Cannot reuse a outDir on aws

[alneberg]

Describe the bug
When using a outDir on s3 which already exists, I get the error ERROR ~ Unexpected error [UnsupportedOperationException]

To Reproduce
Steps to reproduce the behavior:

1. Create a results directory on s3
2. Run sarek main.nf or any other workflow specifying that directory as the --outDir

Expected behavior
It should use that directory I guess?

System (please complete the following information):

• Hardware: Both from laptop and cluster
• Executor: awsbatch

Nextflow (please complete the following information):

• Version: 18.10.1 build 5003

Sarek (please complete the following information):

• version: Current latest dev.

[maxulysse]

Mentionned that to Paolo on gitter, and:

it should work, open an issue with an example and error stack trace

[KochTobi]

can confirm that this issue exists. I have the same problem.
The problem also exists for Nextflow version 0.32.0 build 4897

[KochTobi]

Maybe sth to do with the publishDirMode = 'copy'? I don't quite understand what it does but the nextflow documentation warns about this mode.

Files are copied into the specified directory in an asynchronous manner, thus they may not be immediately available in the published directory at the end of the process execution. For this reason files published by a process must not be accessed by other downstream processes.

[alneberg]

This error is also triggered when starting variant calling and using the same outDir as was used for preprocessing. This is a bit more annoying than if one is only rerunning the preprocessing since one would not get the preprocessing in the same outdir as the variantcalling...

[alneberg]

Even more annoying is that this completely blocks annotation from being run on AWS since it has to find the VariantCalling within the outDir:

Sarek/annotate.nf

Line 49 in 35e7f70

directoryMap = SarekUtils.defineDirectoryMap(params.outDir)

Sarek/annotate.nf

Line 68 in 35e7f70

Channel.fromPath("${directoryMap.haplotypecaller}/*.vcf.gz")

However, I think I'm starting to understand this error. Reading the log carefully:

java.lang.UnsupportedOperationException: null
        at com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:554)
        at java.nio.file.Files.move(Files.java:1395)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:46)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
        at nextflow.extension.FilesEx.renameTo(FilesEx.groovy:608)
        at nextflow.extension.FilesEx$renameTo.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
        at nextflow.extension.FilesEx.rollFile0(FilesEx.groovy:1359)
        at nextflow.extension.FilesEx.rollFile0(FilesEx.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:102)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:196)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:224)
        at nextflow.extension.FilesEx.rollFile(FilesEx.groovy:1333)
        at nextflow.extension.FilesEx.rollFile(FilesEx.groovy)
        at nextflow.trace.TraceFileObserver.onFlowStart(TraceFileObserver.groovy:201)
        at nextflow.Session$_start_closure17.doCall(Session.groovy:496)
        at nextflow.Session$_start_closure17.call(Session.groovy)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2296)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2281)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2322)
        at nextflow.Session.start(Session.groovy:496)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:157)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:240)
        at nextflow.cli.Launcher.run(Launcher.groovy:428)
        at nextflow.cli.Launcher.main(Launcher.groovy:586)

I narrowed it down to

com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:554)
nextflow.extension.FilesEx.rollFile(FilesEx.groovy)
nextflow.trace.TraceFileObserver.onFlowStart(TraceFileObserver.groovy:201)

and indeed if one adds the following to the conf:

trace {
  enabled = false
}

it's possible to launch the pipeline. However, I suspect the other reports will cause the pipeline to fail when the final job is done.

[maxulysse]

fixed by #682