Dataverse Store

[4tikhonov]

Hi all,

In the new SSHOC DataverseEU project we're thinking about building application layer around Dataverse as we need viewers and external tools to be plugged in easily, at different levels of maturity. The biggest problem is efficient apps maintenance as some tools can die and go out of the support, or have security holes or bugs.

I have a vision that it can be kind of Dataverse store available in the dashboard where users can select software they want to install or remove from the running system.

Indeed it's going to be a mess if everything will be integrated in the core system, it can break Dataverse. I have an idea to build kind of universal Docker image for web applications (Apache/Ngix with CORS available) that will download source code of selected applications from different GitHub repositories, build, pass all tests automatically with Selenium and will be finally available for users. For Javascript frameworks it's feasible to do right now.

Dataverse already has very rich set of APIs so I think with Docker integration we can make tools and VMs secure enough to operate in the common workspace. I'm only deeply concerned about exposing sensitive data so we can even offer encryption tools to encode/decode restricted content coming to browser.

Please let me know if you think it can be something useful for you as well.

[RightInTwo]

Hi Slava, I think this would be very useful!

a) I see it as a prerequisite that Dataverse + Docker is a stable and supported setup for building something as close to the core project as a "Dataverse Store". Is that the right way of thinking?

b) What kind of software would be relevant for this? Do you think it makes sense to feature a list of candidates here? Like...

• TwoRavens
• Whole Tale
• DMPRoadmap
• a connector to a custom CRIS
• a workflow management software
• ...

[pdurbin]

I just spotted a slide about a (future) Dataverse App Store website over at https://twitter.com/4tykhonov/status/1116229640232873984

@4tikhonov I really appreciate you laying out your vision for more modularity in Dataverse! Great stuff! 🎉 Please let us know how we can help!

@RightInTwo one kind of software we talked about in our design meeting yesterday was the idea of "preview" tools at the file level (#3758). For example, what if instead of a large icon for a tabular file (screenshot below) a preview appeared? What would the look like? Would it be interactive? #3758 is the place for a deeper dive on preview at the file level but this is just an example of what a future app store for Dataverse could have in it. Please keep the ideas coming! 💡 💡 💡

[RightInTwo]

@4tikhonov wrote:

I have an idea to build kind of universal Docker image for web applications (Apache/Ngix with CORS available) that will download source code of selected applications from different GitHub repositories, build, pass all tests automatically

Since there are plenty of free and open container managers with GUIs available, why not choose one of them and provide templates (incl. automatic integration with each other by setting the correct ports or exchanging secrets) to set up and maintain an environment of research data software around Dataverse?

[RightInTwo]

The idea "Monitoring dashboard for shared resources" #5760 would fit very well with this.

[pdurbin]

I'm cross-posting a new spreadsheet I just mentioned at #5028 (comment)

It's called "External Tools" and here's a link: https://docs.google.com/spreadsheets/d/1OwIxpgpWVPDPSFwDsnPfk8ivNRXUIiaAlnbFccCnfsQ/edit?usp=sharing

Here's a screenshot:

This is my attempt to understand what external tools exist or are being talked about. It feels related enough to this concept of an "app store" for Dataverse to mention it here in this issue.

Comments are open on that spreadsheet. Feedback is welcome here or there or anywhere. 😄

[pdurbin]

A few things.

In addition to that Google spreadsheet above where I tried to start classifying various external tools, I also refactored the guides themselves so that external tools are now in a TSV file: http://guides.dataverse.org/en/4.17/_downloads/dataverse-external-tools.tsv

I did this so that the tools could appear on multiple pages in the guides but a side effect of this is that @4tikhonov or others could treat this TSV as data. The columns are:

• name
• type (explore or configure)
• scope (dataset or file)
• description

Here's how I explain the new "dataverse-external-tools.tsv" file at http://guides.dataverse.org/en/4.17/api/external-tools.html#adding-your-tool-to-the-inventory-of-external-tools

The other thing is that I keep dreaming that https://demo.dataverse.org (or some other server) could be powered by Kubernetes and all these tools could be showcased together. I wrote about this in a couple places:

• Community documentation on setting up a local OSF instance CenterForOpenScience/osf.io#8748 (comment)
• TwoRavens is not working with dataverse, even after running the TwoRavens instance #5859 (comment)

Finally, over at #6256 (comment) I saw @4tikhonov just posted the slide about a Dataverse App Store. Great stuff!

[4tikhonov]

Hi @pdurbin, I have the same dream. If you will follow this
instruction then you'll get Dataverse with all integrated tools in Kubernetes, we did the integration of external tools in the separate Docker container.

And more interesting integrations are coming from us, just wait for the official announcement.

[pdurbin]

@4tikhonov great!

What's the "definition of done" for this issue? When do we say, "Now there is a Dataverse App Store"?

I should just wait for the announcement? 😄