[BUG] MarkDuplicatesSpark fails

[jacorvar]

Check Documentation

I have checked the following places for your error:

• [ x ] nf-core website: troubleshooting
• [ x ] nf-core/sarek pipeline documentation

Description of the bug

I'm experiencing issues when enabling the use of spark (--use_gatk_spark).
I edited the files /etc/security/limits.conf and /etc/sysctl.conf on computing nodes as suggested in https://nf-co.re/sarek/usage#spark-related-issues, but not /etc/sysconfig/docker since I'm using singularity (should I edit a different file?).

Steps to reproduce

Steps to reproduce the behavior (I added the SINGULARITYENV_* variables according to #295 (comment)):

1. Command line:

export NXF_OPTS='-Xms1g -Xmx4g'
export SINGULARITYENV_SPARK_LOCAL_IP=127.0.0.1
export SINGULARITYENV_SPARK_PUBLIC_DNS=127.0.0.1
nextflow run nf-core/sarek -r 2.7.1 --cpus $SLURM_CPUS_PER_TASK --max_cpus 64 --single_cpu_mem '8 GB' --max_memory '500 GB' --input ../data/sample_data.tsv -profile singularity --tools FreeBayes,HaplotypeCaller,VEP --use_gatk_spark --outdir ../results --step mapping -resume

2. See error:

  12:36:04.453 INFO  MarkDuplicatesSpark - ------------------------------------------------------------
  12:36:04.454 INFO  MarkDuplicatesSpark - The Genome Analysis Toolkit (GATK) v4.1.7.0
  12:36:04.454 INFO  MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/
  12:36:04.458 INFO  MarkDuplicatesSpark - Initializing engine
  12:36:04.458 INFO  MarkDuplicatesSpark - Done initializing engine
  12:36:04.691 INFO  MarkDuplicatesSpark - Shutting down engine
  [May 23, 2022 12:36:04 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapse
d time: 0.01 minutes.
  Runtime.totalMemory()=4557111296
  Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.spark.SparkConf$.<init>(SparkConf.scala:716)
        at org.apache.spark.SparkConf$.<clinit>(SparkConf.scala)
        at org.apache.spark.SparkConf.set(SparkConf.scala:95)
        at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:77)
        at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:76)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:76)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:71)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:58)
        at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.setupSparkConf(SparkContextFactory.java:173)
        at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.createSparkContext(SparkContextFactory.java:183)
        at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.getSparkContext(SparkContextFactory.java:117)
        at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:28)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206)
        at org.broadinstitute.hellbender.Main.main(Main.java:292)
  Caused by: java.net.UnknownHostException: nodo01: nodo01: No address associated with hostname
        at java.net.InetAddress.getLocalHost(InetAddress.java:1506)
        at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:946)
        at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:939)
        at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:939)
        at org.apache.spark.util.Utils$$anonfun$localCanonicalHostName$1.apply(Utils.scala:996)
        at org.apache.spark.util.Utils$$anonfun$localCanonicalHostName$1.apply(Utils.scala:996)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.util.Utils$.localCanonicalHostName(Utils.scala:996)
        at org.apache.spark.internal.config.package$.<init>(package.scala:302)
        at org.apache.spark.internal.config.package$.<clinit>(package.scala)
        ... 23 more
  Caused by: java.net.UnknownHostException: nodo01: No address associated with hostname
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1501)
        ... 32 more

Log files

log.txt
nextflow.log.txt

Have you provided the following extra information/files:

• [ x ] The command used to run the pipeline
• [ x ] The .nextflow.log file

System

• Hardware: HPC
• Executor: slurm
• OS: CentOS
• Version 7.9

Nextflow Installation

• Version: 21.04.1

Container engine

• Engine: Singularity
• version: 3.7
• Image tag: nfcore/sarek:2.7.1

Should I increase the java memory options or is it a bug?

[jacorvar]

I had to add the hostname and host IP to /etc/hosts. Now it works.