Collect useful scripts from ansible roles?

[mvdbeek]

What do you think about putting a couple more scripts in here for better reference?
Ping @jmchilton @bgruening @afgane

I'm thinking of

• https://github.com/galaxyproject/ansible-galaxy-extras/blob/master/templates/create_galaxy_user.py.j2
• https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/manage_bootstrap_user.py
• https://github.com/ARTbio/ansible-artimed/blob/master/extra-files/get_tool_yml_from_gi.py
• https://github.com/ARTbio/ansible-artimed/blob/master/extra-files/generate_tool_list_from_ga_workflow_files.py

The first two can probably be combined.

[jmchilton]

These two...

• https://github.com/ARTbio/ansible-artimed/blob/master/extra-files/get_tool_yml_from_gi.py
• https://github.com/ARTbio/ansible-artimed/blob/master/extra-files/generate_tool_list_from_ga_workflow_files.py

Look perfect. The second one is actually a TODO item on this work (galaxyproject/planemo#496) in planemo so that is particularly exciting to me personally.

The other two require more than bioblend and galaxy-lib - I don't necessarily think they belong on PyPI in that form? I wish they in galaxy 'core' scripts instead. That said if we added them today we would still want to target older versions of Galaxy so that probably wouldn't be enough huh? So I'd say - feel free to add them if you think they should be there - we may just have to exclude them from doc generation, testing, and linting I think because of import errors?

[mvdbeek]

I wish they in galaxy 'core' scripts instead

Yes, that makes sense to me.

Look perfect. The second one is actually a TODO item on this work (galaxyproject/planemo#496) in planemo so that is particularly exciting to me personally.

The generate_tool_list_from_ga_workflow_files.py has been written by @drosofff, so I'll leave the PR up to him. The other script could use a rework, since we don't need the admin api key anymore to get the toolshed repository entries.

[afgane]

Since scripts are piling on, there's also this one for installing reference data: https://github.com/gvlproject/gvl.ansible.filesystem/blob/master/files/scripts/install_genome_indices.py

(just adding it here FYI as I won't be able to add it until well after GCC though)

[drosofff]

@afgane looks nice !
Same for me, I have still some work to do on https://github.com/ARTbio/ansible-artimed/blob/gks/extra-files/generate_tool_list_from_ga_workflow_files.py and most probably will not be able to PR here until after GCC

[bgruening]

@mvdbeek good idea. We should move our scripts to this project. I guess a smal guideline what qualifies a script to be included here, vs. into Galaxy main would be nice.

Here is a small script to populate data libraries: https://github.com/bgruening/docker-galaxyp/blob/master/setup_data_libraries.py
And in the Docker container we have scripts to install biojs components and tours, this should probably also end up here.

[martenson]

guideline what qualifies a script to be included here vs. into Galaxy codebase

Yes, please. This needs to be clarified.

As I see it this library should target to be eventually used in all Galaxy, BioBlend (would create cyclic dependency as @jmchilton pointed below), Planemo, and any other project that wants to inter/face the Galaxy plugin system. It needs to stay concise and minimal, as we already have problems with too many projects that are invisible/unclear for any outsider.

What I am resolving into is, that basically this library should remain under-the-hood deep-dev-only thing. User facing project is Galaxy - there shouldn't be more. Planemo and BioBlend are dev-facing.

@jmchilton ?

[jmchilton]

@martenson This is all a moving target right. There are several factors that prevent us from doing a perfect job at these things:

• Galaxy has many dependencies and isn't decomposed reasonably at all - it isn't even pip installable at this time. galaxy-lib sucks and it is the best we have done yet.
• Many projects such as ansible-galaxy-tools and bioblend probably want to continue to target non-dev releases of Galaxy - this means not depending on having access to a GALAXY_ROOT.

Within those restrictions though... a couple guidelines I would use, when I think about this and efforts such as breaking up Galaxy into pieces (galaxyproject/galaxy#1396 and galaxy-lib).

• Dependencies should be directed acyclic graphs (DAGs). So despite the comment by @martenson - bioblend should not depend on this! Galaxy and planemo can depend on this and they in turn can depend on bioblend - but we shouldn't be creating cycles. The principle applies at the project level, at the level of decomposing Galaxy into sub projects, and at the level of individual modules within any of these projects.
• Stuff in PyPI should have the correct dependencies. Since this project can't import galaxy.model I didn't think scripts that used it should be a part of this project - they would be broken by design. In Break project into namespaces galaxy#1396 I made the case that as galaxy-lib materializes the next target should be "just enough" models to be dangerous. I should be possible to leverage the Galaxy data model without requiring web server components, a workflow runner, and IE backend.
• Once something that leverages the Galaxy models directly and can be used to read a "Galaxy configuration" in some fashion is in PyPI - one can imagine taking most of the database management stuff out of $GALAXY_ROOT/scripts as well as stuff like bootstrapping users and making a project out of it. This is very dependent on Break project into namespaces galaxy#1396 however.
• I think as a principle - in general, it is better to develop smaller components that don't have deep dependencies outside of Galaxy core. This is the natural growth of large projects and keeps the design decoupled as well as the administration. If apt-get install galaxy or pip install galaxy should have access to this scripts - we can now pretty easily create a dependency.
• There have been efforts to keep bioblend fairly un-opinionated - it tries to be a fairly thin layer of the Galaxy API (e.g. Adding methods to convert workflow formats bioblend#143) - this is why I put these scripts that I considered opinionated here instead of in bioblend. I didn't initially agree @nsoranzo on this - but I think now that I'm more comfortable with creating and maintaining projects - it seems like a decent design decision. Likewise, this is why gxformat2 is its own project (https://github.com/jmchilton/gxformat2).
• I think this project is the realm for things that manage Galaxy but don't require Galaxy - management that leverages the API. For the time being, things that require Galaxy and Galaxy dependencies - should still be placed in $GALAXY_ROOT/scripts.
• In general, projects should prefer the Galaxy API to the models and file system or database access to Galaxy, the models to filesystem or database access to Galaxy.

I guess all the same principles of isolation and decoupling and preferring interfaces to implementation, etc... apply at the level of the libraries just as much as at the level of class design.

[jmchilton]

@martenson You changed your comment on me and made me look silly. I don't agree entirely and I don't disagree entirely either. I don't like your usage of the word "user" - I would use the word "deployer" - "user" of Galaxy to me in a bench scientist using the GUI.

• If a "user" is someone deploying Galaxy - they should get everything they need with

[apt-get|brew|pip] install galaxy

That should have all the management scripts they need - yes IMO.

• If a "user" is a bioinformatician deploying it on a personal machine - docker-galaxy-stable should be the only thing they need to get going.
  That should have all the management scripts they need bundled - yes IMO.
• If a "user" is a savy bench scientist that wants to use it on the cloud - the only thing they should need to get going is cloudlaunch.
  All the management they should need should be doable from a UI within CloudMan or Galaxy.

[martenson]

I am sorry for changing my comment while you were writing yours, I will do a better job in the future to post when ready.

[mvdbeek]

I guess we can close this, the last two scripts are now in ephemeris and the first two are merged in ansible-galaxy-extras and should eventually make its way to galaxy/scripts.

And I like @jmchilton's description:

this project is the realm for things that manage Galaxy but don't require Galaxy - management that leverages the API.

We may want to put that right into the README.rst.