Feature Request: Allow creating OAI sets with harvested datasets

[JacekChudzik]

Based on conversation in google group

https://groups.google.com/g/dataverse-community/c/CSyFnl--MqM/m/HC3Wt-UnAQAJ?utm_medium=email&utm_source=footer

I would like to kindly request for allowing creating OAI sets with harvested datasets.

At this moment there is a statemant "IS_HARVESTED:false" that was insert into the Solr search query that's used to define an OAI set.

This could be default behavior but there should be an option for admin to change it.

This can ba helpful to share datasets and index them in different databases, like - in our case - OpenAIRE.

[landreev]

Copy-and-pasting my message from the initial exchange in the Google group:

... looked at the code and no, it won't allow you to create a set containing non-local datasets. (There's a hard-coded "IS_HARVESTED:false" that we insert into the Solr search query that's used to define an OAI set).
... I don't see a problem with making this possible.
I would keep the current behavior as the default (simply because I'm assuming this is what most Dataverse instances want; as I can't remember anyone else asking about or requesting this). But we should be able to add this as an optional feature.

Also, just to clarify, making this a feature is not going to be as trivial as simply changing the code that searches for datasets to include in an OAI set. We'll need to also export the oai_dc and oai_ddi metadata records used in harvesting, which we are not doing for harvested datasets now. But, should still be very doable.

[jggautier]

What's described in #9524 seems related, and maybe there's tension between what's proposed here or what would need to be done and the changes proposed in #9524.

[JacekChudzik]

If I can clarify antything or help in anyway please let me know. At this point this issue blocks us from integrating with OpenAIRE.

[jggautier]

Hey @JacekChudzik. I can't say how or when what you've described might be resolved.

But one of my priorities this year, described in #11916, is also related to OpenAIRE and harvesting, so I'd like to make sure I understand your situation. And maybe that could help others who might be able to help resolve what you've described in this GitHub issue.

In https://groups.google.com/g/dataverse-community/c/CSyFnl--MqM/m/HC3Wt-UnAQAJ you wrote that you have a bunch of Dataverse installations which are harvested to one main instance, and you'd like to tell OpenAIRE to harvest the harvested metadata that's in that one main instance.

Would it be possible instead to tell OpenAIRE to harvest from each of the bunch of Dataverse installations you mentioned, instead of telling OpenAIRE to harvest from only the one main instance? If not, why not?

[JacekChudzik]

Hey @jggautier,
the answer to your question is a mixture of political and historical decisions, but in the end it comes down to the fact that our repository model looks like this: we have local Dataverse instances (like: uj.rodbuk.pl, agh.rodbuk.pl), each representing one University or science organization. All of them are aggregated to the main one: rodbuk.pl. The aggregator is our final product in a sense, and is also submitted for certification, etc. All the individual instances work together, there is a working group, etc. That's why we wanted to be in OpenAire as one, as a whole group, our entire community.

[jggautier]

Ah, yes I remember rodbuk.pl. Thanks for the context!

[JacekChudzik]

Hey, I would like to know is there a chance for this feature and if so, what is the time window for its implementation?

We would like to know, because for us it is the problem of integration with OPENAire.

[pdurbin]

@JacekChudzik there are no plans for the core team to work on this.

We would welcome a pull request, of course. If anyone out there wants to work on this, please read the comment above by Leonid.

[JacekChudzik]

@pdurbin
Thank you for your reply. If our group finds the resources to make a pull request for this, what is the estimated timeframe for such a new feature to be released into production?

[pdurbin]

@JacekChudzik hi! I've been chatting with @landreev about this (internal Slack link). We're both supportive of the feature in general. We're happy to do code review and such, even if we're not so sure we'd want it in production for Harvard's instance of Dataverse. I never what to guarantee that we'll merge a pull request but please reach out when you want us to take a look at a branch, even for a work in progress. Thanks!

Update: If look at https://github.com/IQSS/dataverse/tags you'll see our current release cadence is roughly every three months. In most cases, PRs from contributors make it in. Also, we're happy to chat before you write any code if you like!

[JacekChudzik]

Thanks @pdurbin, I was asking about time frame because connecting with OpenAIRE is important for our curators. I don't know if we will find resources for code contribution, if so - I will let you know. At this time our Dataverse instances probably will have to send data to OpenAIRE separately. From our perspective this is lot of work/decision about organization and future development to be made. Keep in mind, that we started before 6.2 so there was no option for one Dataverse instance with separate DOI per collection. This is the moment we have to adjust :) I think if not this feature, if we decide to go without it, than we will try to contribute to upgrade harvester options.