IQSS/7068 Reserve File Pids by qqmyers · Pull Request #7334 · IQSS/dataverse

qqmyers · 2020-10-16T14:53:14Z

What this PR does / why we need it: Reserves file PIDs at creation time, if enabled, analogous to how datasets are now handled.

Which issue(s) this PR closes:

Closes #7068

Special notes for your reviewer:

Suggestions on how to test this: Should be able to see file PIDs/DOIs after file creation in the datacite console (e.g. doi.test.datacite.org). Deleting a file, or the dataset, prior to publication should remove the draft PID. Nominally this code calls whatever PID provider you use, so it should work with other providers capable of reserving PIDs. I'm not sure it is necessary to test them though - if the PID provider works, and this code works with DOIs, it should also work for those others.

Does this PR introduce a user interface change? If mockups are available, please link/include them here: no

Is there a release notes update needed for this change?:

Additional documentation:

…le_PIDs

(analogous to dataset pid register if needed) Conflicts: src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateDatasetVersionCommand.java src/main/java/propertyFiles/Bundle.properties

coveralls · 2020-10-16T15:06:16Z

coverage: 20.796% (-0.008%) from 20.804%
when pulling 0fd4ee1 on QualitativeDataRepository:IQSS/7068-Reserve_file_PIDs
into 53f9b45 on IQSS:develop.

djbrooke · 2020-10-19T15:13:33Z

Adding this to code review, because I think it is desirable and non-controversial functionality for those who have turned on File PIDs.

I'd be interested in thoughts from the Harvard Dataverse team as we've seen some issues with long running jobs around PID registration, and I wonder if this would potentially make it worse since there's now two steps (reservation and flipping to public) instead of one.

pdurbin

Overall, this looks fine. I'm only choosing "request changes" because I think we need a release note. I'm also suggesting a couple code changes but I don't see either one as showstoppers.

pdurbin · 2020-10-30T16:08:56Z

+        String currentGlobalIdProtocol = ctxt.settings().getValueForKey(SettingsServiceBean.Key.Protocol, "");
+        String currentGlobalAuthority = ctxt.settings().getValueForKey(SettingsServiceBean.Key.Authority, "");
+        String dataFilePIDFormat = ctxt.settings().getValueForKey(SettingsServiceBean.Key.DataFilePIDFormat, "DEPENDENT");
+        boolean shouldRegister = ctxt.systemConfig().isFilePIDsEnabled()                                  // We use file PIDs


I thinking about how we iterate though all the files below. Couldn't we short circuit that iteration and simply return early if file level PIDs are not enabled?

pulled the test of shouldRegister out of the loop.

pdurbin · 2020-10-30T16:12:47Z

+        logger.fine("IsFilePIDsEnabled: " + ctxt.systemConfig().isFilePIDsEnabled());
+        logger.fine("RegWhenPub: " +  !idServiceBean.registerWhenPublished());
+        logger.fine("OK provider: " + ((currentGlobalIdProtocol.equals(protocol) && currentGlobalAuthority.equals(authority)) // the dataset PID is a protocol/authority Dataverse can create new PIDs in
+                        || dataFilePIDFormat.equals("INDEPENDENT")));  


I know it's more code but rather than having hardcoded strings like "INDEPENDENT" my preference is to reference the place where the string comes from like this: SystemConfig.DataFilePIDFormat.INDEPENDENT.toString()

That way if you do a "find usages" in your editor, it easily comes up. And if we ever change the string (which is doubtful), it's easier to find.

This might just be a me think. Please feel free to get opinions from others.

why have constants if you don't use them - changed "INDEPENDENT" and "DEPENDENT" everywhere

pdurbin

Looking good as of d76a738 so I'm moving this to QA. I hear @djbrooke will add a release note.

pdurbin · 2020-10-30T17:51:15Z

Sorry, sorry, I just noticed merge conflicts. @qqmyers can you please fix?

IQSS/7068-Reserve_file_PIDs Conflicts: src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java

djbrooke · 2020-10-30T17:57:35Z

Thanks @qqmyers and @pdurbin - adding the release note now.

…itativeDataRepository/dataverse into IQSS/7068-Reserve_file_PIDs

…orted

djbrooke · 2020-11-03T16:24:01Z

Thanks @qqmyers and @pdurbin for talking with me about this. I made release note additions and API doc edits because I thought we recommended reservation for file PIDs similar to how we did reservations for Dataset PIDs when we moved to the new pre-publish verification workflow in 5.0 (https://github.com/IQSS/dataverse/blob/develop/doc/release-notes/5.0-release-notes.md#dataverse-installations-using-datacite-upgrade-action-recommended). I backed these changes out as we don't need to reserve PIDs for existing files and the API does not currently support this. This does not result in a failure during publishing, it just means that those File PIDs will be registered at that time.

pdurbin · 2020-11-03T16:31:00Z

@djbrooke the release note looks good.

@kcondon I'd suggest testing with many files to see how it performs.

…le_PIDs

Conflicts: src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractDatasetCommand.java src/main/java/propertyFiles/Bundle.properties

IQSS/7068-Reserve_file_PIDs

qqmyers · 2024-03-27T20:44:43Z

-     * @param datafileId
-     * @param storageLocation
-     * @return  {@code true} iff the global identifier is unique.
-     */


Not related to the PR - this just doesn't fit the method below which doesn't involve PIDs

…le_PIDs

landreev · 2024-07-19T18:09:28Z

Looks good overall.
Could you please sync the branch and resolve the conflict(s).

What is the back story, was there a specific reason why it got stuck and sat for 4 years? Was it just because of the concerns about file uploads becoming slower?

IQSS/7068-Reserve_file_PIDs

qqmyers · 2024-07-19T18:24:23Z

As far as I know, performance was the only potential issue - could still be an issue (although QDR uses it), but easier to manage overall since file PIDs can be enabled/disabled as desired now.

landreev

(re-)approving, in order to move the PR along into "ready for QA".

…le_PIDs

qqmyers added 4 commits October 2, 2020 16:21

file pid reservation

b1a88b4

Merge remote-tracking branch 'IQSS/develop' into IQSS/7068-Reserve_fi…

94bda80

…le_PIDs

add file pid reservation step to publish

7f90269

(analogous to dataset pid register if needed) Conflicts: src/main/java/edu/harvard/iq/dataverse/engine/command/impl/UpdateDatasetVersionCommand.java src/main/java/propertyFiles/Bundle.properties

comment change

77bd653

djbrooke assigned qqmyers Oct 19, 2020

djbrooke unassigned qqmyers Oct 26, 2020

pdurbin self-assigned this Oct 30, 2020

pdurbin requested changes Oct 30, 2020

View reviewed changes

pdurbin assigned qqmyers and unassigned pdurbin Oct 30, 2020

check if file PIDs used once, use constants - per comments

d76a738

pdurbin approved these changes Oct 30, 2020

View reviewed changes

Merge remote-tracking branch 'IQSS/develop' into

e0b975e

IQSS/7068-Reserve_file_PIDs Conflicts: src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java

djbrooke added 2 commits October 30, 2020 13:59

adding release note

98a7a83

Merge branch 'IQSS/7068-Reserve_file_PIDs' of https://github.com/Qual…

2ec3bac

…itativeDataRepository/dataverse into IQSS/7068-Reserve_file_PIDs

djbrooke assigned djbrooke and unassigned qqmyers Oct 30, 2020

djbrooke added 3 commits October 30, 2020 15:22

release notes, API doc update

8a41176

reflecting datasets and files for the PID endpoint

4d9b497

removing release note about pre-reg for file PIDs as this is not supp…

979d402

…orted

djbrooke removed their assignment Nov 3, 2020

qqmyers added 2 commits November 3, 2020 15:05

Merge remote-tracking branch 'IQSS/develop' into IQSS/7068-Reserve_fi…

1e567dc

…le_PIDs

file pid pre-reservation

cdace4f

Conflicts: src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractDatasetCommand.java src/main/java/propertyFiles/Bundle.properties

qqmyers mentioned this pull request Nov 3, 2020

Iqss/7383 file pid reservation api call #7384

Closed

cmbz added the Size: 3 A percentage of a sprint. 2.1 hours. label Mar 26, 2024

Merge remote-tracking branch 'IQSS/develop' into

682858d

IQSS/7068-Reserve_file_PIDs

qqmyers commented Mar 27, 2024

View reviewed changes

qqmyers added 3 commits March 27, 2024 16:46

release note update

e3b01d5

cleanup reformatting

c5b490c

further cleanup

3c194db

qqmyers removed their assignment Mar 27, 2024

qqmyers added 3 commits March 27, 2024 17:12

Merge remote-tracking branch 'IQSS/develop' into IQSS/7068-Reserve_fi…

afefed4

…le_PIDs

Merge remote-tracking branch 'IQSS/develop' into IQSS/7068-Reserve_fi…

9a4ddd3

…le_PIDs

set createTime earlier

da87bd1

qqmyers mentioned this pull request Jun 5, 2024

Reconcile the PID of a Dataset (If Multiple PID Providers Are Enabled) #10567

Merged

landreev self-requested a review July 19, 2024 18:05

landreev self-assigned this Jul 19, 2024

Merge remote-tracking branch 'IQSS/develop' into

404ccff

IQSS/7068-Reserve_file_PIDs

landreev approved these changes Jul 19, 2024

View reviewed changes

landreev removed their assignment Jul 19, 2024

stevenwinship assigned stevenwinship and unassigned stevenwinship Aug 14, 2024

Merge remote-tracking branch 'IQSS/develop' into IQSS/7068-Reserve_fi…

0fd4ee1

…le_PIDs

stevenwinship self-assigned this Aug 14, 2024

stevenwinship merged commit cf174b2 into IQSS:develop Aug 15, 2024

stevenwinship removed their assignment Aug 15, 2024

pdurbin added this to the 6.4 milestone Aug 20, 2024

qqmyers mentioned this pull request Aug 26, 2024

With file PIDs able to be pre-registered, an api to do that for existing files would be useful #7383

Closed

adswa mentioned this pull request Oct 28, 2024

NativeAPI: Replace endpoint returns "Failed to add file to dataset" when file PIDs are enabled #10975

Closed

luddaniel mentioned this pull request Mar 11, 2025

Reserve file-level DOIs? #7068

Closed

qqmyers mentioned this pull request Jun 3, 2025

Publication fails with file PIDs enabled for some files #11546

Closed

vkush mentioned this pull request Feb 23, 2026

Handle.net PIDs - exception adding files to dataset, file PID registration failed #12174

Open

Conversation

qqmyers commented Oct 16, 2020

Uh oh!

coveralls commented Oct 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

djbrooke commented Oct 19, 2020

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

pdurbin Oct 30, 2020

Choose a reason for hiding this comment

Uh oh!

qqmyers Oct 30, 2020

Choose a reason for hiding this comment

Uh oh!

pdurbin Oct 30, 2020

Choose a reason for hiding this comment

Uh oh!

qqmyers Oct 30, 2020

Choose a reason for hiding this comment

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

pdurbin commented Oct 30, 2020

Uh oh!

djbrooke commented Oct 30, 2020

Uh oh!

djbrooke commented Nov 3, 2020

Uh oh!

pdurbin commented Nov 3, 2020

Uh oh!

qqmyers Mar 27, 2024

Choose a reason for hiding this comment

Uh oh!

landreev commented Jul 19, 2024

Uh oh!

qqmyers commented Jul 19, 2024

Uh oh!

landreev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

coveralls commented Oct 16, 2020 •

edited

Loading