[BUG] Error executing process but process finishes successfully

[rpensch]

Hi,

I have been running the pipeline on tumor/normal WGS data to do both mapping and now variant calling. Recently I haven been running into many of the same issues at different steps, with different callers and different samples. What I see happening is that I get an email that the pipeline has crashed on a particular step, usually with the information "Command exit status: -, Command output: (empty)". But then I check the working directory indicated in the email and the particular process that is supposedly affected has continued to run for some time after this error has been raised and finishes with an exit code 0. I have also checked the slurm stats of such jobs and according to those the jobs are completed successfully. I am providing an example error message and the contents of the command files below. I'd appreciate any help with what could be causing these problems.

Error message 17/5/2022, 15:00:

nf-core/sarek execution completed unsuccessfully!

The exit status of the task that caused the workflow execution to fail was: null.

The full error message was:

Error executing process > 'StrelkaBP (T-CCB070010_vs_N-CCB070010)'

Caused by:
  Process `StrelkaBP (T-CCB070010_vs_N-CCB070010)` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

  configureStrelkaSomaticWorkflow.py         --tumor T-CCB070010.recal.reheader.bam         --normal N-CCB070010.recal.reheader.bam         --referenceFasta cf4.b6.14.fa         --indelCandidates Manta_T-CCB070010_vs_N-CCB070010.candidateSmallIndels.vcf.gz                  --runDir Strelka
  
  python Strelka/runWorkflow.py -m local -j 4
  
  mv Strelka/results/variants/somatic.indels.vcf.gz         StrelkaBP_T-CCB070010_vs_N-CCB070010_somatic_indels.vcf.gz
  mv Strelka/results/variants/somatic.indels.vcf.gz.tbi         StrelkaBP_T-CCB070010_vs_N-CCB070010_somatic_indels.vcf.gz.tbi
  mv Strelka/results/variants/somatic.snvs.vcf.gz         StrelkaBP_T-CCB070010_vs_N-CCB070010_somatic_snvs.vcf.gz
  mv Strelka/results/variants/somatic.snvs.vcf.gz.tbi         StrelkaBP_T-CCB070010_vs_N-CCB070010_somatic_snvs.vcf.gz.tbi

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  [2022-05-17T12:36:09.193251Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2086_chrUn_S895H1059_0000_to_chromId_2090_chrUn_S899H1063_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.193546Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2086_chrUn_S895H1059_0000_to_chromId_2090_chrUn_S899H1063_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.194066Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2091_chrUn_S89H248_0000_to_chromId_2095_chrUn_S903H1067_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.194368Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2091_chrUn_S89H248_0000_to_chromId_2095_chrUn_S903H1067_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.194913Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2096_chrUn_S904H1068_0000_to_chromId_2101_chrUn_S909H1073_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.233351Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2096_chrUn_S904H1068_0000_to_chromId_2101_chrUn_S909H1073_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.281002Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2102_chrUn_S90H249_0000_to_chromId_2106_chrUn_S914H1079_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.347781Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2102_chrUn_S90H249_0000_to_chromId_2106_chrUn_S914H1079_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.424089Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2107_chrUn_S916H1081_0000_to_chromId_2111_chrUn_S91H250_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.424373Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2107_chrUn_S916H1081_0000_to_chromId_2111_chrUn_S91H250_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.424825Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2112_chrUn_S920H1085_0000_to_chromId_2115_chrUn_S923H1089_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.425054Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2112_chrUn_S920H1085_0000_to_chromId_2115_chrUn_S923H1089_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.425539Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2116_chrUn_S924H1090_0000_to_chromId_2121_chrUn_S929H1095_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.425777Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2116_chrUn_S924H1090_0000_to_chromId_2121_chrUn_S929H1095_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.426237Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2122_chrUn_S92H251_0000_to_chromId_2126_chrUn_S933H1099_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.426468Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2122_chrUn_S92H251_0000_to_chromId_2126_chrUn_S933H1099_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.427047Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2127_chrUn_S934H1100_0000_to_chromId_2132_chrUn_S939H1105_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.427286Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2127_chrUn_S934H1100_0000_to_chromId_2132_chrUn_S939H1105_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.427769Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2133_chrUn_S93H252_0000_to_chromId_2137_chrUn_S943H1109_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.428000Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2133_chrUn_S93H252_0000_to_chromId_2137_chrUn_S943H1109_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.428465Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2138_chrUn_S944H1110_0000_to_chromId_2143_chrUn_S949H1115_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.428696Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2138_chrUn_S944H1110_0000_to_chromId_2143_chrUn_S949H1115_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.429150Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2144_chrUn_S94H253_0000_to_chromId_2148_chrUn_S953H1119_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.429379Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2144_chrUn_S94H253_0000_to_chromId_2148_chrUn_S953H1119_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.429957Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2149_chrUn_S954H1120_0000_to_chromId_2153_chrUn_S95H254_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.430194Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2149_chrUn_S954H1120_0000_to_chromId_2153_chrUn_S95H254_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.430714Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2154_chrUn_S960H1126_0000_to_chromId_2159_chrUn_S965H1131_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.430949Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2154_chrUn_S960H1126_0000_to_chromId_2159_chrUn_S965H1131_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.431403Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2160_chrUn_S966H1132_0000_to_chromId_2164_chrUn_S96H255_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.431634Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2160_chrUn_S966H1132_0000_to_chromId_2164_chrUn_S96H255_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.432202Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2165_chrUn_S970H1136_0000_to_chromId_2170_chrUn_S975H1141_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.432502Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2165_chrUn_S970H1136_0000_to_chromId_2170_chrUn_S975H1141_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.433032Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2171_chrUn_S976H1142_0000_to_chromId_2175_chrUn_S97H256_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.433313Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2171_chrUn_S976H1142_0000_to_chromId_2175_chrUn_S97H256_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.433847Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2176_chrUn_S980H1146_0000_to_chromId_2181_chrUn_S985H1151_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.434145Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2176_chrUn_S980H1146_0000_to_chromId_2181_chrUn_S985H1151_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.434677Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2182_chrUn_S986H1152_0000_to_chromId_2186_chrUn_S98H257_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.434972Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2182_chrUn_S986H1152_0000_to_chromId_2186_chrUn_S98H257_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.435501Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2187_chrUn_S990H1157_0000_to_chromId_2192_chrUn_S995H1162_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.435809Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2187_chrUn_S990H1157_0000_to_chromId_2192_chrUn_S995H1162_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.436314Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+callGenomeSegment_chromId_2193_chrUn_S996H1163_0000_to_chromId_2197_chrUn_S99H258_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.437239Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+compressSegmentOutput_chromId_2193_chrUn_S996H1163_0000_to_chromId_2197_chrUn_S99H258_0000' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.437523Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+completedAllGenomeSegments' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.443677Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+SNV_concat_vcf' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.483453Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+SNV_index_vcf' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.485174Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+Indel_concat_vcf' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.485406Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+Indel_index_vcf' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.488626Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+mergeRunStats' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.489194Z] [s118.uppmax.uu.se] [60986_1] [WorkflowRunner] Adding command task 'CallGenome+removeTmpDir' to sub-workflow 'CallGenome'
  [2022-05-17T12:36:09.489458Z] [s118.uppmax.uu.se] [60986_1] [TaskRunner:CallGenome] Finished task specification for sub-workflow

Work dir:
  /crex/proj/uppstore2017228/KLT.05.GRUS/TK-2771_2/run/work/de/8376ebf4fc403489a62e5eaf0b2b90

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

.exitcode

0

.command.out

Successfully created workflow run script.
To execute the workflow, run the following script and set appropriate options:

/scratch/6216633/nxf.n1PHON6g9F/Strelka/runWorkflow.py

.command.log
command.log.txt

.command.err
command.err.txt

[pontus]

What version of nextflow is this running with? I don't recall enough keywords to find the issue and thus which version should be safe, but it does sound a lot like an issue nextflow used to have where it could sometimes decide jobs in a slurm queue were gone (incorrectly).

What happens if you -resume after this has happened?

[rpensch]

Hi and sorry for the late reply. The nextflow version is 21.10.3. and when I resumed the pipeline, the process that had failed is recognized as a cached process. However, then it fails for the same reason but on another process very quickly, now for this example after around 20 minutes.