/config/ folder gets emptied after config file edit

[kajocina]

Hello,

I know I reported this to Bjoern already in email, but thought others might find the same issue hapenning to them. I am using docker 1.8.1 on Ubuntu 14.04LTS.
After running the docker-galaxy-stable container with the command:

sudo docker run -i -t -p 8080:80 -v /home/XXX/Software/galaxy_storage/:/export/ --name galaxy bgruening/galaxy-stable

I get galaxy nice and running. I make an account for myself, go to home/XXX/Software/galaxy_storage/galaxy-central/config and do

sudo nano galaxy.ini.sample where I change the line

admin_users = None

into
admin_users = kajocina@gmail.com

I save the file under a new name galaxy.ini and stop the container. Subsequently, I start the container again using the same command as shown above. What happens is that now I get 403 forbidden in my localhost:8080 access and strangely enough, the config folder gets totally emptied.

Anyone ever encountered such behavior ?

EDIT 1: I also noticed that in case I change this config file myself and get 403 Forbidden, the logs in the galaxy container stop on:

==> /home/galaxy/slurmd.log <==
[2015-09-08T11:46:36.002] Node configuration differs from hardware: CPUs=4:4(hw) Boards=1:1(hw) SocketsPerBoard=4:1(hw) CoresPerSocket=1:4(hw) ThreadsPerCore=1:1(hw)
[2015-09-08T11:46:36.002] Gathering cpu frequency information for 4 cpus
[2015-09-08T11:46:36.003] slurmd version 2.6.5 started
[2015-09-08T11:46:36.003] slurmd started on Tue, 08 Sep 2015 11:46:35 +0000
[2015-09-08T11:46:36.003] CPUs=4 Boards=1 Sockets=4 Cores=1 Threads=1 Memory=32115 TmpDisk=210818 Uptime=17625

and never go any further, while with "healthy" galaxy run, without changing configs I get

==> /home/galaxy/uwsgi.log <==

after this and a whole huge list of lines starting with DEBUG.... etc.

Best,
Piotr

[bgruening]

Hi Piotr,

is it related to this one here: #64
Are you using restart? If not can you provide us with a step by step guide how to reproduce this?

Thanks,
Bjoern

[kajocina]

Hi Bjoern,

I am trying to recreate it and so far can't, I might be missing something that I did before.

Could you just tell me what is the purpose of galaxy.ini.docker_sample vs. galaxy.ini.sample ? Why there are two such files in /export/ config folder ?

[bgruening]

galaxy.ini.docker_sample contains Docker specific modifications for your job handlers and the uwsgi session.

[kajocina]

So right now I tried going step-by-step to at least reproduce the problem, but I am ending up with what I described in the EDIT1 part of my original post, i.e.:

I started docker image (the local export folder empty):
docker run -i -t -p 8080:80 -v /home/XXX/Software/galaxy_storage/:/export/ bgruening/galaxy-stable /bin/bash

Typed "startup" and hit enter. Got galaxy going on localhost:8080 where I went to create a new user.
I then went to /home/XXX/Software/galaxy_storage/galaxy-central/config and made a change in galaxy.ini.sample in line
#admin_users = None
into
admin_users = kajocina@gmail.com
After this, saved the file and renamed it to galaxy.ini.

Now I did docker stop . The container stopped.
And ran the container like the first time: docker run -i -t -p 8080:80 -v /home/XXX/Software/galaxy_storage/:/export/ bgruening/galaxy-stable /bin/bash

However this time Galaxy doesn't load, but got stuck on this part:

==> /home/galaxy/slurmd.log <==
[2015-09-08T14:30:49.694] Node configuration differs from hardware: CPUs=4:4(hw) Boards=1:1(hw) SocketsPerBoard=4:1(hw) CoresPerSocket=1:4(hw) ThreadsPerCore=1:1(hw)
[2015-09-08T14:30:49.695] Gathering cpu frequency information for 4 cpus
[2015-09-08T14:30:49.696] slurmd version 2.6.5 started
[2015-09-08T14:30:49.696] slurmd started on Tue, 08 Sep 2015 14:30:49 +0000
[2015-09-08T14:30:49.696] CPUs=4 Boards=1 Sockets=4 Cores=1 Threads=1 Memory=32115 TmpDisk=210818 Uptime=27479

Why would this happen ? Am I doing something totally different than you guys ?

[kajocina]

I managed to reproduce a moment when /config/ on my local export gets emptied.

1. Create an empty /home/user/galaxy_storage/ folder
2. Start the container docker run -i -t -p 8080:80 -v /home/user/Software/galaxy_storage/:/export/ bgruening/galaxy-stable /bin/bash
3. Run 'startup', Galaxy gets up and running, /export/ on local drive gets populated
4. While in container, ctrl+c to stop Galaxy, /export/ on local drive still full, config in its place
5. Run 'startup', Python script throws an error about empty directory:

Traceback (most recent call last):
File "/usr/local/bin/export_user_files.py", line 63, in
change_path('/galaxy-central/config/')
File "/usr/local/bin/export_user_files.py", line 32, in change_path
shutil.rmtree( src )
File "/usr/lib/python2.7/shutil.py", line 256, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
os.rmdir(path)
OSError: [Errno 20] Not a directory: '/galaxy-central/config/'

Finally: The local /export/ config folder is now empty, Galaxy is up and running.

It seems to me the problem indeed lies in export_user_files.py, but so far I can't pinpoint why exactly.

[bgruening]

@kajocina thanks for the detailes analysis I will have a look at it.
I never tested this scenario of running the startup script twice in one session.

[kajocina]

I don't know if it helps, but looking at shutil documentation:

"Changed in version 2.6: Explicitly check for path being a symbolic link and raise OSError in that case."

Isn't 'src' after first run a symbolic link ? When there is a function call:

change_path('/galaxy-central/config/') then I would assume that the script calls:
shutil.rmtree( '/galaxy-central/config/' ) which should then return:

OSError: [Errno 20] Not a directory: '/galaxy-central/config/'

Which it does in my case.

[kajocina]

My initial problem with Galaxy not starting up after changing admin in galaxy.ini.sample and changing file name to galaxy.ini was resolved by simply using galaxy.ini.docker_sample. This is the file where the admin should be changed and subsequently renamed to galaxy.ini.

Now all works fine.

[bgruening]

Thanks @kajocina!

[bgruening]

@kajocina could you please try to reproduce your config issue with the latest :dev release. I added a safeguard to this particular issue. Image will be ready in one hour.

[kajocina]

I used the image bgruening/galaxy-stable:dev.

Ran it with docker run -i -t -p 8081:80 -v /home/user/Software/galaxy_storage2/:/export/ bgruening/galaxy-stable:dev /bin/bash

Used startup script.
Galaxy loaded. Did ctrl+c in the container to stop Galaxy, re-ran startup script.

Got this:

root@d0e1e22f4c85:/galaxy-central# startup
umount: /var/lib/docker: must be superuser to umount
Traceback (most recent call last):
File "/usr/local/bin/export_user_files.py", line 64, in
change_path('/galaxy-central/config/')
File "/usr/local/bin/export_user_files.py", line 33, in change_path
shutil.rmtree( src )
File "/usr/lib/python2.7/shutil.py", line 256, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
os.rmdir(path)
OSError: [Errno 20] Not a directory: '/galaxy-central/config/'

So it is the same as previously.

[bgruening]

Applied the fix to :dev. Container is building right now.