Merge pull request #9568 from IQSS/9148-license-via-api

kcondon · web-flow · commit f02ad47c2d8b · 2023-05-08T15:50:34.000-04:00
9148 Ensure that license/terms are set via API (create/update/publish)
diff --git a/doc/release-notes/9148-license-via-api.md b/doc/release-notes/9148-license-via-api.md
@@ -0,0 +1,28 @@
+# License management via API
+
+See https://github.com/IQSS/dataverse/issues/9148. 
+
+When publishing a dataset via API, it now requires the dataset to either have a standard license configured, or have valid Custom Terms of Use (if allowed by the instance). Attempting to publish a dataset without such **will fail with an error message**. This introduces a backward incompatibility, and if you have scripts that automatically create, update and publish datasets, this last step may start failing. Because, unfortunately, there were some problems with the datasets APIs that made it difficult to manage licenses, so an API user was likely to end up with a dataset missing either of the above. In this release we have addressed it by making the following fixes:
+
+We fixed the incompatibility between the format in which license information was *exported* in json, and the format the create and update APIs were expecting it for *import* (https://github.com/IQSS/dataverse/issues/9155). This means that the following json format can now be imported:
+```
+"license": {
+   "name": "CC0 1.0",
+   "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+}
+```
+However, for the sake of backward compatibility the old format
+```
+"license" : "CC0 1.0"
+```
+will be accepted as well.
+
+We have added the default license (CC0) to the model json file that we provide and recommend to use as the model in the Native API Guide (https://github.com/IQSS/dataverse/issues/9364).
+
+And we have corrected the misleading language in the same guide where we used to recommend to users that they select, edit and re-import only the `.metadataBlocks` fragment of the json metadata representing the latest version. There are in fact other useful pieces of information that need to be preserved in the update (such as the `"license"` section above). So the recommended way of creating base json for updates via the API is to select *everything but* the `"files"` section, with (for example) the following `jq` command:
+
+```
+jq '.data | del(.files)'
+```
+
+Please see the [Update Metadata For a Dataset](https://guides.dataverse.org/en/latest/api/native-api.html#update-metadata-for-a-dataset) section of our Native Api guide for more information. 
diff --git a/doc/sphinx-guides/source/_static/api/dataset-update-metadata.json b/doc/sphinx-guides/source/_static/api/dataset-update-metadata.json
@@ -1,4 +1,8 @@
 {
+  "license": {
+    "name": "CC0 1.0",
+    "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+  },
   "metadataBlocks": {
     "citation": {
       "displayName": "Citation Metadata",
diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst
@@ -1049,23 +1049,26 @@ The fully expanded example above (without environment variables) looks like this
 
   curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT https://demo.dataverse.org/api/datasets/:persistentId/versions/:draft?persistentId=doi:10.5072/FK2/BCCP9Z --upload-file dataset-update-metadata.json
 
-Note that in the example JSON file above, there is a single JSON object with ``metadataBlocks`` as a key. When you download a representation of your dataset in JSON format, the ``metadataBlocks`` object you need is nested inside another object called ``datasetVersion``. To extract just the ``metadataBlocks`` key when downloading a JSON representation, you can use a tool such as ``jq`` like this:
+Note that in the example JSON file above, there are only two JSON objects with the ``license`` and ``metadataBlocks`` keys respectively. When you download a representation of your latest dataset version in JSON format, these objects will be nested inside another object called ``data`` in the API response. Note that there may be more objects in there, in addition to the ``license`` and ``metadataBlocks`` that you may need to preserve and re-import as well. Basically, you need everything in there except for the ``files``. This can be achived by downloading the metadata and selecting the sections you need with a JSON tool such as ``jq``, like this:
 
 .. code-block:: bash
 
   export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   export SERVER_URL=https://demo.dataverse.org
   export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/BCCP9Z
 
-  curl -H "X-Dataverse-key: $API_TOKEN" $SERVER_URL/api/datasets/:persistentId/versions/:latest?persistentId=$PERSISTENT_IDENTIFIER | jq '.data | {metadataBlocks: .metadataBlocks}' > dataset-update-metadata.json
-
+  curl -H "X-Dataverse-key: $API_TOKEN" $SERVER_URL/api/datasets/:persistentId/versions/:latest?persistentId=$PERSISTENT_IDENTIFIER | jq '.data | del(.files)' > dataset-update-metadata.json
+  
 The fully expanded example above (without environment variables) looks like this:
 
 .. code-block:: bash
 
   curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" https://demo.dataverse.org/api/datasets/:persistentId/versions/:latest?persistentId=doi:10.5072/FK2/BCCP9Z | jq '.data | {metadataBlocks: .metadataBlocks}' > dataset-update-metadata.json
 
-Now that the resulting JSON file only contains the ``metadataBlocks`` key, you can edit the JSON such as with ``vi`` in the example below::
+
+Now you can edit the JSON produced by the command above with a text editor of your choice. For example, with ``vi`` in the example below.
+
+Note that you don't need to edit the top-level fields such as ``versionNumber``, ``minorVersonNumber``, ``versionState`` or any of the time stamps - these will be automatically updated as needed by the API::
 
     vi dataset-update-metadata.json
 
diff --git a/scripts/api/data/dataset-create-new.json b/scripts/api/data/dataset-create-new.json
@@ -4,6 +4,10 @@
   "persistentUrl": "http://dx.doi.org/10.5072/FK2/9",
   "protocol": "chadham-house-rule",
   "datasetVersion": {
+    "license": {
+      "name": "CC0 1.0",
+      "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+    },
     "metadataBlocks": {
       "citation": {
         "displayName": "Citation Metadata",
@@ -121,4 +125,4 @@
       }
     }
   }
-}
+}
diff --git a/scripts/api/data/dataset-finch1_fr.json b/scripts/api/data/dataset-finch1_fr.json
@@ -1,6 +1,10 @@
 {
   "metadataLanguage": "fr",
   "datasetVersion": {
+    "license": {
+      "name": "CC0 1.0",
+      "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+    },
     "metadataBlocks": {
       "citation": {
         "fields": [
diff --git a/scripts/search/tests/data/dataset-finch1-nolicense.json b/scripts/search/tests/data/dataset-finch1-nolicense.json
@@ -0,0 +1,77 @@
+{
+  "datasetVersion": {
+    "metadataBlocks": {
+      "citation": {
+        "fields": [
+          {
+            "value": "Darwin's Finches",
+            "typeClass": "primitive",
+            "multiple": false,
+            "typeName": "title"
+          },
+          {
+            "value": [
+              {
+                "authorName": {
+                  "value": "Finch, Fiona",
+                  "typeClass": "primitive",
+                  "multiple": false,
+                  "typeName": "authorName"
+                },
+                "authorAffiliation": {
+                  "value": "Birds Inc.",
+                  "typeClass": "primitive",
+                  "multiple": false,
+                  "typeName": "authorAffiliation"
+                }
+              }
+            ],
+            "typeClass": "compound",
+            "multiple": true,
+            "typeName": "author"
+          },
+          {
+            "value": [ 
+                { "datasetContactEmail" : {
+                    "typeClass": "primitive",
+                    "multiple": false,
+                    "typeName": "datasetContactEmail",
+                    "value" : "finch@mailinator.com"
+                },
+                "datasetContactName" : {
+                    "typeClass": "primitive",
+                    "multiple": false,
+                    "typeName": "datasetContactName",
+                    "value": "Finch, Fiona"
+                }
+            }],
+            "typeClass": "compound",
+            "multiple": true,
+            "typeName": "datasetContact"
+          },
+          {
+            "value": [ {
+               "dsDescriptionValue":{
+                "value":   "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds.",
+                "multiple":false,
+               "typeClass": "primitive",
+               "typeName": "dsDescriptionValue"
+            }}],
+            "typeClass": "compound",
+            "multiple": true,
+            "typeName": "dsDescription"
+          },
+          {
+            "value": [
+              "Medicine, Health and Life Sciences"
+            ],
+            "typeClass": "controlledVocabulary",
+            "multiple": true,
+            "typeName": "subject"
+          }
+        ],
+        "displayName": "Citation Metadata"
+      }
+    }
+  }
+}
diff --git a/scripts/search/tests/data/dataset-finch1.json b/scripts/search/tests/data/dataset-finch1.json
@@ -1,5 +1,9 @@
 {
   "datasetVersion": {
+    "license": {
+      "name": "CC0 1.0",
+      "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+    },
     "metadataBlocks": {
       "citation": {
         "fields": [
diff --git a/scripts/search/tests/data/dataset-finch2.json b/scripts/search/tests/data/dataset-finch2.json
@@ -1,5 +1,9 @@
 {
   "datasetVersion": {
+    "license": {
+      "name": "CC0 1.0",
+      "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+    },
     "metadataBlocks": {
       "citation": {
         "fields": [
diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java
@@ -548,6 +548,10 @@ public static License getLicense(DatasetVersion dsv) {
 
     public static String getLicenseName(DatasetVersion dsv) {
         License license = DatasetUtil.getLicense(dsv);
+        return getLocalizedLicenseName(license);
+    }
+    
+    public static String getLocalizedLicenseName(License license) {
         return license != null ? getLocalizedLicenseDetails(license,"NAME")
                 : BundleUtil.getStringFromBundle("license.custom");
     }
diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/PublishDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/PublishDatasetCommand.java
@@ -23,6 +23,7 @@
 import static java.util.stream.Collectors.joining;
 import static edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetResult.Status;
 import static edu.harvard.iq.dataverse.dataset.DatasetUtil.validateDatasetMetadataExternally;
+import edu.harvard.iq.dataverse.util.StringUtil;
 
 
 /**
@@ -204,6 +205,12 @@ private void verifyCommandArguments(CommandContext ctxt) throws IllegalCommandEx
             throw new IllegalCommandException("Only authenticated users can release a Dataset. Please authenticate and try again.", this);
         }
         
+        if (getDataset().getLatestVersion().getTermsOfUseAndAccess() == null
+                || (getDataset().getLatestVersion().getTermsOfUseAndAccess().getLicense() == null 
+                && StringUtil.isEmpty(getDataset().getLatestVersion().getTermsOfUseAndAccess().getTermsOfUse()))) {
+            throw new IllegalCommandException("Dataset must have a valid license or Custom Terms Of Use configured before it can be published.", this);
+        }
+        
         if ( (getDataset().isLockedFor(DatasetLock.Reason.Workflow)&&!ctxt.permissions().isMatchingWorkflowLock(getDataset(),request.getUser().getIdentifier(),request.getWFInvocationId())) 
                 || getDataset().isLockedFor(DatasetLock.Reason.Ingest) 
                 || getDataset().isLockedFor(DatasetLock.Reason.finalizePublication)
diff --git a/src/main/java/edu/harvard/iq/dataverse/license/LicenseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/license/LicenseServiceBean.java
@@ -3,6 +3,7 @@
 import edu.harvard.iq.dataverse.actionlogging.ActionLogRecord;
 import edu.harvard.iq.dataverse.actionlogging.ActionLogServiceBean;
 import edu.harvard.iq.dataverse.api.AbstractApiBean.WrappedResponse;
+import static edu.harvard.iq.dataverse.dataset.DatasetUtil.getLocalizedLicenseName;
 
 import javax.ejb.EJB;
 import javax.ejb.Stateless;
@@ -64,6 +65,31 @@ public License getByNameOrUri(String nameOrUri) {
             return null;
         }
     }
+    
+    public License getByPotentiallyLocalizedName(String name) {
+        // First, try the name against the name column in the License table, 
+        // verbatim: 
+        License license = getByNameOrUri(name); 
+        if (license != null) {
+            return license; 
+        }
+        
+        // Then, if still here, go through the list, see if any of the names
+        // match this string as a translated name:
+        List<License> allActiveLicenses = listAllActive();
+        if (allActiveLicenses == null) {
+            return null; 
+        }
+        for (License activeLicense : allActiveLicenses) {
+            // This is DatasetUtil.getLicenseName(), it will return the 
+            // localized/translated name, if available.
+            if (name.equals(getLocalizedLicenseName(activeLicense))) {
+                return activeLicense;
+            }
+        }
+        
+        return null; 
+    }
 
     public int setDefault(Long id) throws WrappedResponse{
         License candidate = getById(id);
diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java
@@ -369,7 +369,28 @@ public DatasetVersion parseDatasetVersion(JsonObject obj, DatasetVersion dsv) th
             dsv.setUNF(obj.getString("UNF", null));
             // Terms of Use related fields
             TermsOfUseAndAccess terms = new TermsOfUseAndAccess();
-            License license = parseLicense(obj.getString("license", null));
+
+            License license = null; 
+            
+            try {
+                // This method will attempt to parse the license in the format 
+                // in which it appears in our json exports, as a compound
+                // field, for ex.:
+                // "license": {
+                //    "name": "CC0 1.0",
+                //    "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+                // }
+                license = parseLicense(obj.getJsonObject("license"));
+            } catch (ClassCastException cce) {
+                logger.fine("class cast exception parsing the license section (will try parsing as a string)");
+                // attempt to parse as string: 
+                // i.e. this is for backward compatibility, after the bug in #9155
+                // was fixed, with the old style of encoding the license info 
+                // in input json, for ex.: 
+                // "license" : "CC0 1.0"
+                license = parseLicense(obj.getString("license", null));
+            }
+            
             if (license == null) {
                 terms.setLicense(license);
                 terms.setTermsOfUse(obj.getString("termsOfUse", null));
@@ -427,6 +448,48 @@ private edu.harvard.iq.dataverse.license.License parseLicense(String licenseName
         if (license == null) throw new JsonParseException("Invalid license: " + licenseNameOrUri);
         return license;
     }
+    
+    private edu.harvard.iq.dataverse.license.License parseLicense(JsonObject licenseObj) throws JsonParseException {
+        if (licenseObj == null){
+            boolean safeDefaultIfKeyNotFound = true;
+            if (settingsService.isTrueForKey(SettingsServiceBean.Key.AllowCustomTermsOfUse, safeDefaultIfKeyNotFound)){
+                return null;
+            } else {
+                return licenseService.getDefault();
+            }
+        }
+        
+        String licenseName = licenseObj.getString("name", null);
+        String licenseUri = licenseObj.getString("uri", null);
+        
+        License license = null; 
+        
+        // If uri is provided, we'll try that first. This is an easier lookup
+        // method; the uri is always the same. The name may have been customized
+        // (translated) on this instance, so we may be dealing with such translated
+        // name, if this is exported json that we are processing. Meaning, unlike 
+        // the uri, we cannot simply check it against the name in the License
+        // database table. 
+        if (licenseUri != null) {
+            license = licenseService.getByNameOrUri(licenseUri);
+        }
+        
+        if (license != null) {
+            return license;
+        }
+        
+        if (licenseName == null) {
+            String exMsg = "Invalid or unsupported license section submitted" 
+                    + (licenseUri != null ? ": " + licenseUri : ".");
+            throw new JsonParseException("Invalid or unsupported license section submitted."); 
+        }
+        
+        license = licenseService.getByPotentiallyLocalizedName(licenseName);
+        if (license == null) {
+            throw new JsonParseException("Invalid or unsupported license: " + licenseName);
+        }
+        return license;
+    }
 
     public List<DatasetField> parseMetadataBlocks(JsonObject json) throws JsonParseException {
         Set<String> keys = json.keySet();
diff --git a/src/main/java/propertyFiles/License.properties b/src/main/java/propertyFiles/License.properties
@@ -1,4 +1,4 @@
 license.cc0_1.0.description=Creative Commons CC0 1.0 Universal Public Domain Dedication. 
 license.cc_by_4.0.description=Creative Commons Attribution 4.0 International License.
 license.cc0_1.0.name=CC0 1.0
-license.cc_by_4.0.name=CC-BY 4.0
+license.cc_by_4.0.name=CC BY 4.0
diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java
@@ -2380,7 +2380,8 @@ public void testSemanticMetadataAPIs() {
         String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse);
 
         // Create a dataset using native api
-        Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken);
+        // (this test requires that we create a dataset without the license set; note the extra boolean arg.:)
+        Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken, true);
         createDatasetResponse.prettyPrint();
         Integer datasetId = UtilIT.getDatasetIdFromResponse(createDatasetResponse);
 
diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java
@@ -376,9 +376,13 @@ static Response showDataverseContents(String alias, String apiToken) {
                 .header(API_TOKEN_HTTP_HEADER, apiToken)
                 .when().get("/api/dataverses/" + alias + "/contents");
     }
-
+    
     static Response createRandomDatasetViaNativeApi(String dataverseAlias, String apiToken) {
-        String jsonIn = getDatasetJson();
+        return createRandomDatasetViaNativeApi(dataverseAlias, apiToken, false);
+    }
+
+    static Response createRandomDatasetViaNativeApi(String dataverseAlias, String apiToken, boolean withNoLicense) {
+        String jsonIn = getDatasetJson(withNoLicense);
         Response createDatasetResponse = given()
                 .header(API_TOKEN_HTTP_HEADER, apiToken)
                 .body(jsonIn)
@@ -388,7 +392,16 @@ static Response createRandomDatasetViaNativeApi(String dataverseAlias, String ap
     }
 
     private static String getDatasetJson() {
-        File datasetVersionJson = new File("scripts/search/tests/data/dataset-finch1.json");
+        return getDatasetJson(false); 
+    }
+     
+    private static String getDatasetJson(boolean nolicense) {
+        File datasetVersionJson; 
+        if (nolicense) {
+            datasetVersionJson = new File("scripts/search/tests/data/dataset-finch1-nolicense.json");
+        } else {
+            datasetVersionJson = new File("scripts/search/tests/data/dataset-finch1.json");
+        }
         try {
             String datasetVersionAsJson = new String(Files.readAllBytes(Paths.get(datasetVersionJson.getAbsolutePath())));
             return datasetVersionAsJson;
diff --git a/src/test/java/edu/harvard/iq/dataverse/export/ddi/dataset-hdl.json b/src/test/java/edu/harvard/iq/dataverse/export/ddi/dataset-hdl.json
diff --git a/src/test/resources/json/complete-dataset-with-files.json b/src/test/resources/json/complete-dataset-with-files.json
diff --git a/src/test/resources/json/dataset-finch1.json b/src/test/resources/json/dataset-finch1.json

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,8 @@`
`1`	`1`	`{`
	`2`	`+ "license": {`
	`3`	`+ "name": "CC0 1.0",`
	`4`	`+ "uri": "http://creativecommons.org/publicdomain/zero/1.0"`
	`5`	`+ },`
`2`	`6`	`"metadataBlocks": {`
`3`	`7`	`"citation": {`
`4`	`8`	`"displayName": "Citation Metadata",`
Original file line number	Diff line number	Diff line change
`@@ -4,6 +4,10 @@`
`4`	`4`	`"persistentUrl": "http://dx.doi.org/10.5072/FK2/9",`
`5`	`5`	`"protocol": "chadham-house-rule",`
`6`	`6`	`"datasetVersion": {`
	`7`	`+ "license": {`
	`8`	`+ "name": "CC0 1.0",`
	`9`	`+ "uri": "http://creativecommons.org/publicdomain/zero/1.0"`
	`10`	`+ },`
`7`	`11`	`"metadataBlocks": {`
`8`	`12`	`"citation": {`
`9`	`13`	`"displayName": "Citation Metadata",`
`@@ -121,4 +125,4 @@`
`121`	`125`	`}`
`122`	`126`	`}`
`123`	`127`	`}`
`124`		`-}`
	`128`	`+}`