harmonize formats for metadata schema and dataset creation

[pameyer]

User stories:

• As a repository administrator / curator / metadata person, I would like to deal with hierarchical metadata schema in a way that is less awkward than a spreadsheet.
• As a curator / metadata person, I would like to be able to configure which metadata fields should be included in particular export formats without having to do additional development.
• As a data depositor, I would like there to be a clearer relationship between the definition of the metadata schema for my dataset and the API calls to create that dataset.
• As an external (aka - developing stuff that interoperates with dataverse) developer / data depositor, I would like to be able to verify that my attempt to create or edit a dataset via API will match what Dataverse is expecting prior to making an API call and failing.
• As a metadata person / dataverse developer, I would like to have the metadata crosswalk handled programmatically (rather than requiring a person to keep TSV files in synchronization with google spreadsheets).
• As a researcher, I would like to reduce the amount of development time required for Dataverse to provide metadata to newly developed/discovered systems, and to adapt to updated versions of existing metadata schemas.

---

A first step towards addressing this could be replacing the TSV files used for metadatablocks / DatasetFieldTypes with a file format supporting hierarchical structures (JSON, YAML, XML), and updating the APIs in Dataverse reading these (with no additional information provided). This format should be something that either can be, or can be easily transformed into a form to, validate dataset creation/edit API input files.

• For a second step, define fields in the metadata schema format to indicate a metadata element should be exported (or potentially a more detailed indication about which types of exporters the element should be sent to), and indicate which other metadata schema this element corresponds to.
• For a third step, places where metadata is currently exported (HTML meta tags, schema.org, DataCite/EZID DOI registration / Handle registration?, OAI-PMH feeds, others?) would be modified to use the provided information (aka - remove hard-coding).
• For a fourth step, new (or updated) dataset creation/edit/export APIs should be made compatible with the schema in the first step.

[pdurbin]

I keep thinking that if we tweak our tsv files a bit they'll look nice on GitHub but isaacs/github#848 has convinced me that it won't be easy to make them "beautiful and searchable":

[pdurbin]

#3168 is related in the sense that perhaps we should document our crazy TSV format before we switch to JSON or YAML or XML or whatever.

[jggautier]

Just adding that #5960 is related to some of the points in this issue.

[pdurbin]

Related:

• Re-evaluate metadata blocks, metadata display, help, and metadata sources Re-evaluate metadata blocks, metadata display, help, and metadata sources #6030
• Crazy idea: make metadata blocks more flexible Crazy idea: make metadata blocks more flexible #6700

I have a question about metadata customization. We have created a new metadata block for our Dataverse installation at the State Archives, but while we can rely on it appearing in the JSON metadata output for now, we would like to see it included in the DDI output (and it would follow the standard by translating into DDI 2.5 approved fields).

pameyer wrote:

• As a curator / metadata person, I would like to be able to configure which metadata fields should be included in particular export formats without having to do additional development.

I believe this means that customizing metadata output is not yet possible but it is under consideration. Is that correct?

[pdurbin]

@BPeuch fields in custom metadata blocks can be exported in Dataverse's native JSON format but are not automatically exported in any other format (such as DDI). If you would like to see some new fields appear in DDI output you should open an issue about this. 😄

Okay, I will. Thanks @pdurbin. So good to have you back aboard!

[qqmyers]

Just reading metadata related issues - fwiw: the OAI-ORE format automatically includes all the metadata in added metadata blocks, and associates them with whatever community vocab URL was specified in the block.