Julia: user depot not accessible with Pkg.add

[lexming]

The Julia easyconfigs and JuliaPackages follow the usual approach of prepending installation paths to the environment. In this case, new modules prepend their installation path to JULIA_DEPOT_PATH.

However, this breaks the capability of end-users to install new packages with Pkg.add() in Julia. This tool only uses the top path in DEPOT_PATH as destination of new installations. This seems to be by design as described in the Pkg documentation

The depot path is controlled by the Julia DEPOT_PATH global variable which is populated at startup based on the value of the JULIA_DEPOT_PATH environment variable. The first entry is the “user depot” and should be writable by and owned by the current user. The user depot is where: registries are cloned, new package versions are installed, named environments are created and updated, package repos are cloned, newly compiled package image files are saved, log files are written, development packages are checked out by default, and global configuration data is saved. Later entries in the depot path are treated as read-only and are appropriate for registries, packages, etc. installed and managed by system administrators.

So we would have to append paths to JULIA_DEPOT_PATH to not break Pkg. The issue is that this approach allows user installs to overwrite what is loaded with modules from EasyBuild (not desirable).

An alternative approach might be to use JULIA_LOAD_PATH in our easyconfigs to control locations with available installations and only set JULIA_DEPOT_PATH to the user's depot.

[lexming]

So I tested several configurations for this issue and there is no setup that allows users to easily install their own Julia packages on top of our installations.

With easybuilders/easybuild-easyblocks#2935 we get back the user depot ~/.julia as the top entry in JULIA_DEPOT_PATH. This is an improvement over the current situation, but users will not be able to add their own packages yet.

Regardless of the contents of JULIA_DEPOT_PATH, the active environment is defined by its associated Project.toml. In the case of our JuliaPackage modules, they have their own environment defined in $EBROOTXXX/environments/v1.x/Project.toml. For instance this is the environment after loading IJulia:

$ julia -E 'Base.DEPOT_PATH'
["~/.julia",
"~/easybuild/install/skylake/software/Julia/1.8.5-linux-x86_64/share/julia",
"~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5"]

$julia -E 'Base.load_path()'
["~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/environments/v1.8/Project.toml",
"~/easybuild/install/skylake/software/Julia/1.8.5-linux-x86_64/share/julia/stdlib/v1.8"]

$ julia -E 'Base.active_project()'
"~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/environments/v1.8/Project.toml"

Hence, executing Pkg.add will indeed install the new packages under ~/.julia/packages (top dir in JULIA_DEPOT_PATH) but it will also try to modify the Project.toml in the installation directory of IJulia and fail (as it is read-only).

Therefore, users wanting to add their own packages on top of our modules have to create their own environments. This is similar in approach to what is done with virtual environments in Python, where users create a venv on top of our modules and install their stuff there. The problem is that Julia does not provide any easy method to use the environment defined in the Project.toml from our modules into their own environments. This is as simple as copying the TOML file over a new directory, but it has to be manually done.

$ base_project=$(julia -E 'Base.active_project()')
$ cp -r "$(dirname ${base_project:1:-1})" myNewEnv
$ chmod -R +w myNewEnv
$ julia -E 'using Pkg; Pkg.activate("myNewEnv"); Pkg.status()'
  Activating project at `/vscmnt/brussel_pixiu_data/_data_brussel/vo/000/bvo00005/vsc10122/tests/julia/myNewEnv`
Status `/vscmnt/brussel_pixiu_data/_data_brussel/vo/000/bvo00005/vsc10122/tests/julia/myNewEnv/Project.toml`
[...]
  [7073ff75] IJulia v1.24.0 `~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/packages/IJulia`
[...]

Once the custom user environment is active, installations of additional packages will take place inside the environment folder while those packages provided by the modules will be also available and taken into account.

[lexming]

Regarding the use of JULIA_LOAD_PATH instead of JULIA_DEPOT_PATH. It does not make any difference.

Whenever we append new installations of JULIA_DEPOT_PATH, those paths get automatically prepended to JULIA_LOAD_PATH. So packages in the last JuliaPackage loaded already have preference over the rest based on LOAD_PATH. See previous comment

$ julia -E 'Base.DEPOT_PATH'
["~/.julia",
"~/easybuild/install/skylake/software/Julia/1.8.5-linux-x86_64/share/julia",
"~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5"]

$julia -E 'Base.load_path()'
["~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/environments/v1.8/Project.toml",
"~/easybuild/install/skylake/software/Julia/1.8.5-linux-x86_64/share/julia/stdlib/v1.8"]

$ julia -E 'Base.active_project()'
"~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/environments/v1.8/Project.toml"

If we would manually prepend new installations to JULIA_LOAD_PATH instead of appending them to JULIA_DEPOT_PATH, the result is the same with the exception that the new loaded path is missing from JULIA_DEPOT_PATH.

$ julia -E 'Base.DEPOT_PATH'
["~/.julia",
"~/easybuild/install/skylake/software/Julia/1.8.5-linux-x86_64/share/julia"]

$julia -E 'Base.load_path()'
["~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/environments/v1.8/Project.toml",
"~/easybuild/install/skylake/software/Julia/1.8.5-linux-x86_64/share/julia/stdlib/v1.8"]

$ julia -E 'Base.active_project()'
"~/easybuild/install/skylake/software/IJulia/1.24.0-Julia-1.8.5/environments/v1.8/Project.toml"

However, effectively, the environment works in the same way in both cases. The active project switches to that of the last JuliaPackage loaded. So users still need to create their own environments to add their own packages on top of our modules. As explained in previous comment.

[lexming]

Fixed by easybuilders/easybuild-easyblocks#2935 and #17976.

See explanation in this issue for a solution to allow end-users to add their packages on top of the installations from EasyBuild: #17455 (comment)