Singularity v3.8.5 - Failed to create user namespace: user namespace disabled

[sjfleck]

Version of Singularity:

3.8.5
singularity shell image.sif

Expected behavior

Entering singularity environment

Actual behavior

INFO: Converting SIF file to temporary sandbox...
FATAL: while extracting image.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled
: exit status 1

Steps to reproduce this behavior

module use -a /projects/academic/vaalbert/modulefiles/Allluas
module load anaconda3
eval "$(/projects/academic/vaalbert/modulefiles/anaconda3/bin/conda shell.bash hook)"
conda activate singularity-3.8.5
cd full/path/to/dir/containing/image
singularity shell image.sif
NOTE: This image was given to me by a collaborator and I'm running it on my university's cluster. I do not (and will never) have root privileges. Is it possible for me to run Singularity on our cluster using the SLURM workload manager?

What OS/distro are you running

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How did you install Singularity

conda create -n singularity-3.8.5 -c conda-forge singularity=3.8.5

[sjfleck]

I'd like to add that I'm very new to using images and running containers. While I've been installing and running other software for years, both locally and on my university's cluster. I've only recently tried to learn how to use Docker and Singularity containers.

The majority of my experience is with conda environments and installing from source. My university uses the LUA system for loading and unloading modules in our front end and within .sh scripts that we submit to the cluster using the SLURM workload manager.

I've only tried docker locally, since it seems to require root privileges. The container that was shared with me by my collaborator is compatible with SLURM job management systems as long as Singularity 3 is available. When given the option between running locally (max 8 CPUs & 16 GB memory) or on our cluster (max 40 CPUs & 754 GB memory), I prefer to run programs on the cluster (if it's possible).

[ikaneshiro]

Hi @sjfleck, in order to help us understand the potential problem could you please run your command with singularity -d ... to enable debug output and include it in a comment.

Based off of the initial output, it looks like this is an unprivileged singularity installation (non-setuid) on your cluster, but unprivileged user namespaces are not enabled which is a requirement for this type of installation to function.

I am not a CentOS user personally, but you should be able to check if unprivileged user namespaces are enabled on that system by checking that cat /proc/sys/user/max_user_namespaces returns a value greater than 0. If it is 0, you will need to talk to your system administrator about enabling this kernel feature, or installing Singularity with setuid privileges

[sjfleck]

I ran cat /proc/sys/user/max_user_namespaces and it returned 0. It looks like I'll have to chat with our Center for Computational Research.

Here is the same command from my original comment with the debug output enabled:

singularity.d.shell.image.sif.txt

[sjfleck]

I'm going to close this issue. It turns out that on July 20, 2021 (4 months ago), my university's Center for Computational Research added singularity-v3.8.4-1.el7 as a default program (no need to load modules or anything). I didn't expect them to add this capability, but I'm lucky to start this job when I did. Thanks!