Variable metadata for restricted files in Croissant export

[LauraHuisintveld]

What steps does it take to reproduce the issue?

Find a dataset that has a file with variable information available on file level. I created this one on demo.dataverse.org: https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/ELQXIX

Make an export of the metadata in Croissant format via the 'Metadata export' button.

The export will contain the variable information of the restricted file.

• When does this issue occur?
  When creating a metadata export in Croissant format

• Which page(s) does it occurs on?
  Dataset page, metadata tab.

• What happens?
  Information from restricted files is visible in the export.

• To whom does it occur (all users, curators, superusers)?
  all users who use this function.

• What did you expect to happen?
  The variable information should not be in the metadata export.
  Just like it is not present in the DDI exports.

Which version of Dataverse are you using?
6.5 (and also on 6.7.1 Harvard demo environment)

Any related open or closed issues to this bug report?

• Variable metadata for restricted files visible in search results #11751
• No Summary Stats and Variable Names/Labels for Restricted Files #7619
• restricted summary stats #7619 #7642

[pdurbin]

@LauraHuisintveld thanks for opening this issue and for the discussion.

This reminds me strongly of how it was once possible to download variable metadata from DDI exports (#7619), which we fixed in #7642.

This emphasizes to how external exporters absolutely need to be trusted by the installations that are installing them. I created a debug exporter at https://github.com/gdcc/exporter-debug for example, but it shouldn't be used in production because it also has access to potentially restricted file metadata.

An easy fix would be to add the logic to not expose restricted file metadata directly in the Croissant exporter at https://github.com/gdcc/exporter-croissant and I assume that's what we'll do. However, we might want to also consider a change to the export framework to prevent future exporter authors from having to put special logic in their exporters. It's similar, in my mind, to how we're making exporters like Croissant sanitize HTML to prevent possible vulnerabilities ( gdcc/exporter-croissant#8 ). Again, maybe the framework should offer more support in this regard. Maybe the exporter authors should have less to worry about.

[pdurbin]

I created an issue and PR in the Croissant repo:

• variable name, variable description, and variable type are exposed for restricted files gdcc/exporter-croissant#19
• no variable name, desc, type exposure for restricted files gdcc/exporter-croissant#20

I'm putting this in "ready for review".

[pdurbin]

We discussed this at standup today. The PR above only fixes the reported problem: restricted files. It does not address embargoed files or files with a retention period. These are more complicated and will probably require a timer to be added that triggers a reexport. I think they should be handled in a separate issue and pull request.

[qqmyers]

FWIW: The DDI export is refreshed when embargoes end due to

dataverse/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java

Lines 164 to 204 in de2b48f

	// The DDI export is limited for restricted and actively embargoed files (no
	// data/file description sections).and when an embargo ends, we need to refresh
	// this export.
	boolean clearCachedExport = false;
	if (formatName.equals(DDIExporter.PROVIDER_NAME) && (exportInputStream != null)) {
	// We want ddi and there was a cached version
	LocalDate exportLocalDate = null;
	Date lastExportDate = dataset.getLastExportTime();
	// if lastExportDate == null, assume it's not set because were exporting for the
	// first time now (e.g. during publish) and therefore no changes are needed
	if (lastExportDate != null) {
	exportLocalDate = lastExportDate.toInstant().atZone(ZoneId.systemDefault()).toLocalDate();
	logger.fine("Last export date: " + exportLocalDate.toString());
	// Track which embargoes we've already checked
	Set<Long> embargoIds = new HashSet<Long>();
	// Check for all files in the latest released version
	for (FileMetadata fm : dataset.getLatestVersionForCopy().getFileMetadatas()) {
	// ToDo? This loop is necessary because we have not stored the date when the
	// next embargo in this datasetversion will end. If we knew that (another
	// dataset/datasetversion column), we could make
	// one check that nextembargoEnd exists and is after the last export and before
	// now versus scanning through files until we potentially find such an embargo.
	Embargo e = fm.getDataFile().getEmbargo();
	if (e != null) {
	logger.fine("Datafile: " + fm.getDataFile().getId());
	logger.fine("Embargo end date: " + e.getFormattedDateAvailable());
	}
	if (e != null && !embargoIds.contains(e.getId()) && e.getDateAvailable().isAfter(exportLocalDate)
	&& e.getDateAvailable().isBefore(LocalDate.now())) {
	logger.fine("Request that the ddi export be cleared.");
	// The file has been embargoed and the embargo ended after the last export and
	// before the current date, so we need to remove the cached DDI export and make
	// it refresh
	clearCachedExport = true;
	break;
	} else if (e != null) {
	logger.fine("adding embargo to checked list: " + e.getId());
	embargoIds.add(e.getId());
	}
	}
	}

- perhaps that can be extended.