Singularity container execution of pangolin crashes with recent nextflow versions

[Krannich479]

Hi,
I noticed that poreCov crashes when running pangolin via singularity and a reasonably modern nextflow version. It seems that recent updates in the path mounting defaults of nextflow broke the default behavior of poreCov test profiles when using Singularity. More precise, when using the command

nextflow run poreCov.nf -profile test_fastq,slurm,singularity --update

with nextflow 23.10.1 and poreCov 1.9.4 I end up with

[f3/91e3e5] NOTE: Process `determine_lineage_wf:pangolin (1)` terminated with an error exit status (1) -- Execution is retried (1)
ERROR ~ Error executing process > 'determine_lineage_wf:pangolin (1)'

Caused by:
  Process `determine_lineage_wf:pangolin (1)` terminated with an error exit status (1)

Command executed:

  pangolin --skip-scorpio -t 4 SARSCoV2.consensus.fasta

  mv lineage_report.csv lineage_report_SARSCoV2.csv

  find . -name "*.csv" -size  0 -print -delete

Command exit status:
  1

Command output:
  ****
  Pangolin skipping scorpio steps.
  ****
  ****
  Pangolin running in usher mode.
  ****
  Maximum ambiguity allowed is 0.3.
  ****
  Query file:ameSARSCoV2.consensus.fasta
  ****
  Data files found:
  usher_pb:     /opt/conda/lib/python3.7/site-packages/pangolin_data/data/lineageTree.pb
  ****

Command error:
  Traceback (most recent call last):
    File "/opt/conda/lib/python3.7/site-packages/snakemake/__init__.py", line 642, in snakemake
      latency_wait=latency_wait,
    File "/opt/conda/lib/python3.7/site-packages/snakemake/workflow.py", line 242, in __init__
      self.sourcecache = SourceCache()
    File "/opt/conda/lib/python3.7/site-packages/snakemake/sourcecache.py", line 358, in __init__
      os.makedirs(self.cache, exist_ok=True)
    File "/opt/conda/lib/python3.7/os.py", line 213, in makedirs
      makedirs(head, exist_ok=exist_ok)
    File "/opt/conda/lib/python3.7/os.py", line 213, in makedirs
      makedirs(head, exist_ok=exist_ok)
    File "/opt/conda/lib/python3.7/os.py", line 213, in makedirs
      makedirs(head, exist_ok=exist_ok)
    [Previous line repeated 1 more time]
    File "/opt/conda/lib/python3.7/os.py", line 223, in makedirs
      mkdir(name, mode)
  OSError: [Errno 30] Read-only file system: '/home/<myHPCusername>'

Additional Info:

The same command works fine and tests finish if I use nextflow 21.04.0. @MarieLataretu found these update notes that seem on point for this issue:
https://github.com/nextflow-io/nextflow/blob/d8f8bf56cfe79dc05db7c149dafa0174717ae7f4/docs/container.md?plain=1#L580
https://github.com/nextflow-io/nextflow/blob/d8f8bf56cfe79dc05db7c149dafa0174717ae7f4/docs/container.md?plain=1#L584

[replikation]

ah "OSError: [Errno 30] Read-only file system: '/home/'"

is a singularity error @hoelzer can you tune in you had this OS errors before

[replikation]

maybe the answer with the --writable helps for singularity:

https://stackoverflow.com/questions/65599721/unable-to-build-writable-singularity-container

[MarieLataretu]

I'm wondering why this error occurs (only?) with the pangolin container. Apparently, it needs to have access to home?

@Krannich479, did you also try setting NXF_SINGULARITY_HOME_MOUNT to true with a recent Nexflow version?

[replikation]

based on the error pangolin wants to write something in its own container home directory. and singularitys home is not writable. as far as i understand (?) but this is something that the user controls with its own singularity system. it could be that however, we need to add as a flag to the config or so

[hoelzer]

Uff... yes that is annoying and someone else here had this problem recently - for me it still worked bc my nf version is 22.x.x or so.

I am not sure what is happening inside the pangolin container but yeah, the problem is that pangolin wants to write something in /home which is fine with a encapsulated Docker container but fails with Singularity. Because here the process actually tries to write in the HPC system path.

[Krannich479]

@Krannich479, did you also try setting NXF_SINGULARITY_HOME_MOUNT to true with a recent Nexflow version?

Tried this, didn't work for me :/

[replikation]

adding singularity.autoMounts=false to the config could work maybe?

so changing this. (could be the reason why the variable is not working if the config overwrites it)

poreCov/nextflow.config

Line 146 in f86681e

autoMounts = true

autoMounts = false

[hoelzer]

Hey, as a followup here, I was running again into this error on our HPC w/ nf version 24. Then I tried version 23 and same error. But it works with v 22 o_O

[matthuska]

For me, it works if I set the environment variable NXF_SINGULARITY_HOME_MOUNT=true as mentioned here (thanks @MarieLataretu)

I'm using Nextflow version 24.04.3 build 5916.

You can do it like this:

$ export NXF_SINGULARITY_HOME_MOUNT=true
$ nextflow run ...

or as a one liner:

$ NXF_SINGULARITY_HOME_MOUNT=true nextflow run ...

or even throw it in your ~/.bashrc if you want it set all the time.

[hoelzer]

A lifesaver... I just ran into this problem again and the above fix solved it (again).

Maye we can add that to FAQ on the README.md of the Git?

[DataSpott]

@hoelzer I added the information to the troubleshooting-section of the Git-ReadMe now, together with a link to matthuska's response so other users can quickly check for the fix.

[hoelzer]

@hoelzer I added the information to the troubleshooting-section of the Git-ReadMe now, together with a link to matthuska's response so other users can quickly check for the fix.

perfecto! thanks! I would then close the issue...