The Roadmap

[martenson]

Main Galaxy Roadmap

as described in issues.rst

The next major review of this roadmap will be done by the Galaxy Project after the 18.01 release.
The discussion under this issue is moderated to keep the roadmap clear, but please feel free to comment on any part of it.

Priorities

Main/usegalaxy.org

• Scaling Jetstream @natefoo @afgane [In progress]
  • Merge Swarm Manager work in to SlurmScale
  • Deploy
• Mull conda environments and disable env copying - mostly done but still need to upgrade Conda @natefoo @jmchilton [Done on test, not main]
• Fix data issues - Create a data manager to retrieve liftOver files from UCSC #1904 @natefoo @jennaj [New Roadmap]
• New landing page @jxtx @nekrut @dannon [New Roadmap]
• Two-tiered jobs ready to run query (Two tiered jobs-ready-to-run-query #2270) @natefoo [Killed]
• Stampede allocation and preparation for version 2 @jxtx @natefoo @afgane [No progress on their end]
• Bridges optimization (improve dynamic core/runtime calculation) @natefoo [In progress]

UI rewrite

• Eliminate makos [New Roadmap]
• Routing issues @guerler @dannon [New Roadmap]
• Older grids [New Roadmap]

Dataset collections

• More collection operations (Implement Collection Operation to Relabelling Identifiers #3603 , Fix collection operations not having a history. #3786) @jmchilton
• Improve tool collection input UI @guerler
• Deeply nested collections Discuss Better Support for Deeply Nested Dataset Structures #4013 | Sample Sheet Import of Datasets and Collections #4733 @jmchilton
• Robust handling of cluster failures @jmchilton
• Rerunning a failed dataset collection element does substitute the failed element Rerunning a failed dataset collection element should substitute the failed element #2235
• Rerunning entire collection @jmchilton
• Cleanup representation in history panel @jmchilton Simplified User-Facing Dataset Collection Model #1810 *
• ~Generalization of collections i.e. paired lists to generic tuple lists ~ Implement records - heterogenous collections (Issue Implement Record Types (generalization of tuple types) #3834) @jmchilton
• Pagination / Search (Enhancement: dataset search should work within a dataset collection view #4115)

Interactive environments

[Move to new Roadmap]

• GIE hardening GIE Hardening #4651
• Allow running more than one GIE per session
• Bring RStudio to on Main
• Working IEs out of the box + uwsgi switch
• RStudio
• Test/alerts for live GIEs
• Jupyter cannot be uploaded from other places (ipynb files cannot be uploaded into Galaxy #1922)
• Render notebooks in history (serve user content from separate domain [Planning] Allow serving user generated content from a separate domain #1932)
• Improving provenance: IEs as tools (IE Provevance #1923)
• IE roadmap Roadmap: Galaxy Interactive Environments #372
• Developing tests

Workflows and Job Running

• Common workflow language (CWL) - at least for GA4GH challenge Phase 2. [New Roadmap, Future]
  • WIP Implement a subset of the Common Workflow Language. common-workflow-lab/galaxy#47
  • Open Issues https://github.com/common-workflow-language/galaxy/issues
• Scaling collection performance (Scaling Collection Performance - A meta issue #3939) @jmchilton
• uWSGI mules as job handlers @natefoo
• User feedback on scheduling @natefoo [New Roadmap]
• Status updates (stdout, stderr, output) while jobs are running e.g. by displaying output file sizes @natefoo [New Roadmap]
• Clarify the path for users to request missing workflow tool @martenson
• Galaxy workflow format version 2 [New Roadmap]
• Publish workflows to Github @jmchilton @martenson [Killed]
• Decision points (issue will be written by @blankenberg) (Allow Decision Points / Control Statements in Workflows #4798) [New Roadmap, Future]
  • Assertions (Assertion checking in workflows #3954)
• Subworkflow Enhancements [New Roadmap, Future]

Data Libraries

• uploading structured data (mainly for training-material folders/datasets) Hierarchical upload API optimized for folders & collections. #5220

Tool Shed / tool administration

• Improve admin Galaxy interface to TS @martenson @davebx
  • Allow mass update
  • Switch default installation interface to API-enabled.
• Data Table duplication bugs (Data fixes to-do summarized usegalaxy-playbook#55) [New Roadmap]

Galaxy Hub and Training Materials

• Navigation improvements @dannon [Killed]
• Incorporate admin training materials @natefoo [WIP]
• Dockerize/isolate admin activities @natefoo [Killed]

Metrics collection

• Allow deployers to send metrics to assist grant (GRT improvements #3952) @martenson @erasche

Uploading large numbers of datasets

• support chunked uploads Chunked file uploads #573 [New Roadmap]
• upload directly to collections [New Roadmap]

Cloud

• Galaxy-on-the-Cloud merges with the GVL → 17.09 release
• CloudLaunch
  • Move out of beta
  • Write/update docs [WIP]
• CloudBridge support for Azure & GCE
• CloudMan
  • Core framework; integration w/ CloudBridge
  • Ability to deploy K8S [New Roadmap]
  • Launchable in containerized mode via CloudLaunch [Cloud Board]
  • UI [Cloud Board]
• Containerized Galaxy deployment [Cloud Board]
• ObjectStore
  • User-Based ObjectStore (User-based ObjectStore #4840) [New Roadmap]
  • Integration with CloudBridge (Cloud ObjectStore #4487)

OAuth

• Authentication and Authorization
  • OIDC rp; a PSA-based implementation (User authentication using OpenID Connect protocol  #4474) [New Roadmap]
  • Integration with AWS authentication

---

---

Guided tours

• easier to share
• Create initial set of tours for 101, rna-seq, etc**. @nekrut [WIP]

Histories

[New Roadmap]

• Visualizations and workflow invocations in history panel
• Graph view - better view of the histories (Tree view of histories #1929)
• Improve access to visualizations/GIEs/charts/display applications from history History Panel Style Access to Workflow Invocations #5142
• Meta-history: showing many more histories in a multi-history view
• Features (archiving, org, folders, etc) (Allow archiving histories and workflows #1734)
• Bundles

main / performance

• Main performance stats @dannon
  • Build reports
  • Analyze/fix

Tools

• Make it easier to keep Main’s toolset in top shape
  • Make Test/Main use the Ansible tools role to manage installed tools ** @natefoo
  • Extend Use the API to install repositories instead of loading the toolshed in an iframe. #1392 to install repositories discovered from another galaxy instance’s /api/tools. @davebx
• Find high quality WF on other Galaxies and mirror the toolset and import the WF as well** @nekrut
• Install tools needed for a workflow (Installing tools+versions needed for a workflow #559) ** @martenson @afgane
• Provide admins with framework and interface to test installed tools @martenson @jmchilton
• Known Tool issues @jennaj
• Known Data issues @jennaj
• Data Updates Genome Indexes, Known DM Issues and Ephemeris for Main?

Main Maintenance/Operations

• Provide UI for Tool panel ordering/etc. @martenson @davebx @jmchilton [New Roadmap Future]
  • Separate the presentation layer of the tool panel to an extra file?
• Labels on tool sections (derive labels dynamically)
• Automate Installation AND maintenance (updates, deprecations, labels) @afgane @martenson

New / enhanced tools suites

• Metagenomics *** @blankenberg
• RNA-seq **
• Long reads (PB and ONT) -- assembly, polishing, hybrid, variant calling, iso-seq *** @jxtx @nekrut @jgoecks @blankenberg
• Viral population analysis tools** @nekrut @jgoecks @blankenberg
• Methylation analyses - this is available mostly
• R tool integration * @blankenberg
• R tool integration wiki/github instruction page with Galaxy + Best practices to handle bioconductor tool integration
• Chromatin Conformation @msauria

Workflows and Job Running

• Job Piping/streaming from tool to tool

TS/dependencies

• Remove support for uploading repos with more than one tool * @martenson
• Automatically increment tool versions and don't expose changeset revisions.*
• Offer version management to the tool author.

Workflow hosting service

• Architecture and UX study @martenson @jmchilton

---

• This set of priorities was developed and extensively discussed during 2016 Galaxy Team meeting (March 14-15, Williamsport, PA).
• Post 16.04 Review occurred on 5/24/2016 via hangouts.
• Post 16.07 Review occurred on 9/xx/2016 via hangouts.
• Post 16.10 Review occurred on 12/9/2016 via hangouts.
• Post 17.01 Review occurred on 4/20/2017 after SAB meeting.
• Post 17.05 Review occurred on 6/xx/2017 at GCC.
• Post 17.09 Review occurred on 11/9/2017 on zoom. And again on 12/7.

[jmchilton]

@nekrut Wanted me to summarize n Slack conversation that ran away with itself on this issue. If I mis-characterize anyone - feel free to just edit my comment on Github.

• @jxtx "We scale or we die."
• Broad agreement that mulled and Conda are still good and should continue to be developed and improved. Small disagreements perhaps in how good and how rocky things have been - and whether things have been oversold.
• Broad agreement that CWL should be implemented within Galaxy - and broad concern about whether it can really replace Galaxy tools. People think it should be implemented regardless and that the spec needs to continue or re-focus on how to build great interfaces from these abstract tool and workflow descriptions.
  • Concern with CWL is that it cannot be used to make "great" GUIs.
  • @jxtx has the concern with CWL that templating broadly speaking is better than input binding (on this point @jmchilton disagrees).
• Agreement that @jmchilton should prepare a cwltool + conda + mulled demo for Biological Data Science poster in October. @jmchilton should also present this work to GA4GH working group on containers.
• With respect to Dockstore - the push in Galaxy should be to consume things from Dockstore less than using it to store Galaxy tools.
• Some discussion on "what is Galaxy" and "what is an idealized tool" - probably too abstract to be actionable.
• There is concern from @jxtx that Galaxy shouldn't do workflow scheduling - we do too much already and we need to find places to collaborate. @jmchilton ❤️s collaboration but worries the process of adapting to something like Mesos/Pegasus/Condor/toil would mean losing features and doing things sub-optimally. @jmchilton in short - "enhancing the core things we do is easier than replacing the core of what we do".
• @jmchilton believe scaling to 10,000 datasets in a collection across large workflows is very do-able if we focus on it - going higher is possible too if we ignore the GUI concerns.
• Some in the community have the perception that Galaxy can't scale - this worries everyone and @jmchilton made the case that it can and has scaled. No real discussion of the public relations problem - focus is more on continuing to build a technically great product that actually can scale.
• We need to work on dealing with cluster and infrastructure failures if we aren't going to pursue another backend, we should find collaborators to help implement these things if it will help save time for our limited devs and admins.
• @jxtx in short - GA4GH is building failure detection into the spec and learning from our mistakes in this regard.
• Predicting resource usage is hard, people want to work on it though.

[jxtx]

There is concern from @jxtx that Galaxy shouldn't do workflow scheduling

I don't want to make that strong a comment. I think we should evaluate whether we can benefit from adopting (and potentially improving) a scheduling / planning library that is shared with others. The usual tradeoffs apply, but I think we should evaluate options.