Failed to pull singularity image

[jgarces02]

Hi there,

Maybe I'm misconsidering something but I have some problems when running the last version (2.7.2) of sarek (Nextflow version 21.10.6).

The command,

nextflow run nf-core/sarek -r 2.7.2 -params-file myparams.yaml -profile singularity -c mynextflow.config

myparams.yaml,

## input/output options
input          : "/home/jgarces/projects/GEMRNA.WES/nxf_tests_3/samples.tsv"

## main options
target_bed     : "/home/jgarces/genomes/TWIST/Twist_Exome_RefSeq_targets_hg38.bed"

## preprocessing options
save_bam_mapped: false

## reference genome options
igenomes_base  : /home/jgarces/genomes/iGenomes/

## variant calling and annotation
tools          : ASCAT,Control-FREEC,Manta,MSIsensor,Mutect2, Strelka,snpEff,VEP

mynexflow.config,

process {
        executor  = 'slurm'
        queue     = 'medium'
        clusterOptions = "--exclude=nodo11"
        time = 21.h
}

executor.queueSize = 6 //array limit

and the error log:

nf-core/sarek execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: null.

The full error message was:

Error executing process > 'VEP (377_SP - Strelka - Strelka_377_SP_variants.vcf.gz)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name nfcore-sarekvep-2.7.2.GRCh38.img.pulling.1656111108337 docker://nfcore/sarekvep:2.7.2.GRCh38 > /dev/null
  status : 143
  message:
    Activating Modules:
      1) Java/13.0.2     2) Nextflow/21.10.6
    
     [34mINFO:    [0m Starting build...
    Getting image source signatures
    Copying blob sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b
    
     0 B / 25.84 MiB [-------------------------------------------------------------]
     1.00 MiB / 25.84 MiB [==>-----------------------------------------------------]
     1.50 MiB / 25.84 MiB [===>----------------------------------------------------]
     2.00 MiB / 25.84 MiB [====>---------------------------------------------------]
     4.19 MiB / 25.84 MiB [=========>----------------------------------------------]
     (...)
     25.84 MiB / 25.84 MiB [====================================================] 3s
    Copying blob sha256:9c388eb6d33c40662539172f0d9a357287bd1cd171692ca5c08db2886bc527c3
    
     0 B / 76.50 MiB [-------------------------------------------------------------]
     1.01 MiB / 76.50 MiB [>-------------------------------------------------------]
     2.54 MiB / 76.50 MiB [=>------------------------------------------------------]
     (...)
     76.50 MiB / 76.50 MiB [====================================================] 8s
    Copying blob sha256:96cf53b3a9dd496f4c91ab86eeadca2c8a31210c2e5c2a82badbb0dcf8c8f76b
    
     0 B / 48.03 MiB [-------------------------------------------------------------]
     1.00 MiB / 48.03 MiB [=>------------------------------------------------------]
     (...)
     48.03 MiB / 48.03 MiB [====================================================] 5s
    Copying blob sha256:3fbba84366ddeb8b6dece9fb1c547db6afa6e727c4bfe599a5ca2cfdff4c964e
    
     0 B / 761.52 KiB [------------------------------------------------------------]
     761.52 KiB / 761.52 KiB [=====================================================]
     761.52 KiB / 761.52 KiB [==================================================] 0s
    Copying blob sha256:4d6e4d839cd155e0308bf249833dcccb7cfae00466cc98516b015f623cc4f086
    
     0 B / 262 B [-----------------------------------------------------------------]
     262 B / 262 B [============================================================] 0s
    Copying blob sha256:d1d7141290df2a590d413f1cf436ac7089297f9f5df25673162b6a1773049c8c
    
     0 B / 349.36 MiB [------------------------------------------------------------]
     (...)
     349.36 MiB / 349.36 MiB [=================================================] 38s
    Copying blob sha256:bf8443906c36e2794392a902931ad474b8c490fd7a69a237312b67930befcec6
    
     0 B / 13.79 GiB [-------------------------------------------------------------]
     1.46 MiB / 13.79 GiB [>-------------------------------------------------------]
    (...)
     3.56 GiB / 13.79 GiB [==============>-----------------------------------------]
     (...)
     6.28 GiB / 13.79 GiB [=========================>------------------------------]
     (...)
     7.86 GiB / 13.79 GiB [===============================>------------------------]

Any idea about what I'm missing, please? Thanks in advance.

[jgarces02]

I've tried to remove the ~/.singularity folder as is suggested in this other pipeline but with no success.

[FriederikeHanssen]

hm the container was only pushed friday, did you run it since then?

I can see that it exists: https://hub.docker.com/r/nfcore/sarekvep/tags

Does docker pull nfcore/sarekvep:2.7.2.GRCh38 work for you?

[jgarces02]

I have got to work with singularity because my cluster configuration (docker is not installed), but singularity pull nfcore/sarekvep:2.7.2.GRCh38 doesn't work 😢

FATAL:   While getting image info: get did not succeed: 400
        invalid container ref: nfcore/sarekvep

[FriederikeHanssen]

This works for me 👇 which singularity version/nextflow version are you using?

singularity pull docker://nfcore/sarekvep:2.7.2.GRCh38
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 68ced04f60ab skipped: already exists
Copying blob 9c388eb6d33c skipped: already exists
Copying blob 96cf53b3a9dd skipped: already exists
Copying blob 3fbba84366dd skipped: already exists
Copying blob 4d6e4d839cd1 done

[jgarces02]

You're right, it seems to work now... I don't know what was happening. My Nextflow and singularity versions are 21.10.6 and 3.2.1-1.1.el7, respectively.

I'm running again a subset of samples to see if the problem persists. I'll let you know! Thanks!

[FriederikeHanssen]

Great! Let me know how it goes

[jgarces02]

I'm having another error, this time it's different and can be caused by singularity's version. I'm going to update it and I'll tell you ASAP, thanks for your patience.

Error executing process > 'PileupSummariesForMutect2 (350_SP-chr20_31051509-31107036)'

Caused by:
  Process `PileupSummariesForMutect2 (350_SP-chr20_31051509-31107036)` terminated with an error exit status (255)

Command executed:

  gatk --java-options "-Xmx7g"         GetPileupSummaries         -I 350_SP.recal.bam         -V gnomAD.r2.1.1.GRCh38.PASS.AC.AF.only.vcf.gz         -L chr20_31051509-31107036.bed         -O chr20_31051509-31107036_350_SP_pileupsummaries.table

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   mcontainer creation failed: failed to add  as session directory: path . is not an absolute path

[jgarces02]

Nope, it's not because the singularity's version. Any idea about what's happening?

[maxulysse]

Created the #612 for this separate issue