Failure during foreign files download cause execution to stop abruptly

[msmootsgi]

My pipeline has been crashing when nextflow is unable to read a file from S3 in a timely manner to "normalize" the file. I get the following exception:

Caused by:
  Read timed out

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
        at sun.security.ssl.InputRecord.read(InputRecord.java:532)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200)
        at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:108)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:147)
        at java.nio.file.Files.copy(Files.java:2908)
        at java.nio.file.Files.copy(Files.java:3027)
        at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:46)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
        at nextflow.processor.TaskProcessor.normalizeInputToFile(TaskProcessor.groovy:1541)

... much more omitted ...

It would be ideal if a problem like this would crash a process instead of the whole pipeline.

[pditommaso]

I gave a try to this and it turns out that's more tricky that it can seem. The main problem is that currently the resume mechanism assumes the all input files are correctly staged. In this case, since the download is failing, the task unique hash cannot be determined. As a consequence the resume will fail as well.

[msmootsgi]

I'm not sure whether resume functionality is used when a process is retried, but I was imagining that the download would fail, the exception would be caught, and the process to be retried according to the error/retry settings for the process.

[pditommaso]

In principle I agree that it makes sense. Not so trivial to implement at this stage.

[msmootsgi]

I certainly understand. This isn't a huge issue and something I believe I can ameliorate with AWS settings. Feel free to close this issue if that helps.

[pditommaso]

related to #265

[pditommaso]

OK, I'm fixing this issue. I'm wondering if when a remote file download fails NF should report a process error and eventually retry the task execution or the file download should be silently retried (up to a fixed number of times) by the downloader process.

[msmootsgi]

Awesome! I'd probably vote for "loud retry" by the downloader process. I guess it could be argued either way, but I see this as less of a process error than a setup or staging error. By "loud" I mean emitting warning messages to STDOUT in addition to .nextflow.log.

[pditommaso]

So, showing a waning message and retrying without reporting a process error, I agree!

[pditommaso]

What about a NotSuchFileException, in this case I would consider an hard failure and stop the execution even if a retry strategy is set. Or may have sense to return a process error and eventually retry it ?

[msmootsgi]

I could go either way on this. Making a NoSuchFileException a hard failure seems like a minor optimization (to avoid X more NoSuchFileExceptions when retrying), but I'm also having a hard time imagining a situation where a file would suddenly appear between retries, so I don't think it we'd miss anything.

I guess once a download fails, then the process will error for a missing file and then everything switches to normal error process handling, however that is defined, right?

[pditommaso]

I could go either way on this. Making a NoSuchFileException a hard failure seems like a minor optimization (to avoid X more NoSuchFileExceptions when retrying), but I'm also having a hard time imagining a situation where a file would suddenly appear between retries, so I don't think it we'd miss anything.

Yes, but I think it could be even raised due to latencies. Maybe it's better to report a recoverable error so that user has chance to retry the process if needed.

I guess once a download fails, then the process will error for a missing file and then everything switches to normal error process handling, however that is defined, right?

Almost, the remote files are downloaded before the process execution starts, but then yes, the error management is the same.