Skip to content

avoid repeating to build SIF image for Singularity: use local SIF image if an env variable LOCAL_SIF_PATH is set #30

Open
szcf-weiya wants to merge 6 commits intodynverse:masterfrom
szcf-weiya:master
Open

avoid repeating to build SIF image for Singularity: use local SIF image if an env variable LOCAL_SIF_PATH is set #30
szcf-weiya wants to merge 6 commits intodynverse:masterfrom
szcf-weiya:master

Conversation

@szcf-weiya
Copy link
Copy Markdown

@szcf-weiya szcf-weiya commented Dec 27, 2023

When I run the program in the cluster with Singularity, I found that it takes a long time to build a SIF image. And for another run, it again takes a long time to build image, it seems that the cached image in .singularity folder does not help. Here is the running time (330s)

> library(dyno)     
> x = matrix(rpois(9900, 1), nrow = 99)                                                                                                                                               
>     rownames(x) = 1:nrow(x)                                                                                                                                                         
>     colnames(x) = 1:ncol(x)                                                                                                                                                         
>     dataset = wrap_expression(counts = x, expression = log(x+1))     
> system.time({infer_trajectory(dataset, "comp1", debug=T)})                                                                                                                          
Running singularity exec 'docker://dynverse/ti_comp1:v0.9.9.01' echo hi
Running /usr/bin/singularity exec --containall -B \ 
  '/tmp/RtmpzcjE6q/file32c7f55a74b8e9/:/copy_mount,/tmp/RtmpzcjE6q/file32c7f541ac7236/tmp:/tmp2' \ 
  'docker://dynverse/ti_comp1:v0.9.9.01' cp /code/definition.yml /copy_mount/
Executing 'comp1' on '20231227_181629__data_wrapper__jSj89c7coa'        
With parameters: list(dimred = "pca", ndim = 2L, component = 1L) 
...
...
                                                                                                              
Running /usr/bin/singularity exec --containall --pwd /ti/workspace -B \
  '/tmp/RtmpzcjE6q/file32c7f542b877db/ti:/ti,/tmp/RtmpzcjE6q/file32c7f54c14b061/tmp:/tmp2' \
  'docker://dynverse/ti_comp1:v0.9.9.01' bash
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:bd5da474b8baade003eb744e862574f758b269c20cd2bc0d3c6860e1b3b4e08a
Copying blob sha256:844c33c7e6ea19e3f6847e0667befdfb3ef02d6fc735b22c2d070261b6263b97
...
...

INFO:    Using cached SIF image      
   user  system elapsed 
309.030  12.012 330.661 

To avoid repeating to build the SIF image, I introduce an environment variable, LOCAL_SIF_PATH. If the variable is set, then pull the image and save as .sif locally via

singularity pull xxxx.sif docker://

To align with the existing code, if the env variable is unset, then just run as before.

After saving to a local SIF image, we can speed it up significantly --- only around 2 seconds for the same dataset.

devtools::install_github("szcf-weiya/babelwhale")
Sys.setenv(LOCAL_SIF_PATH=getwd())

image

user  system elapsed
  0.865   0.663   1.952

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant