Merge branch 'develop' into 11323-includeDeaccesioned-compare-ds-versions

Author: sekmiller

.github/workflows/deploy_beta_testing.yml

@@ -68,7 +68,7 @@ jobs:
           overwrite: true
 
       - name: Execute payara war deployment remotely
-        uses: appleboy/ssh-action@v1.2.1
+        uses: appleboy/ssh-action@v1.2.2
         env:
           INPUT_WAR_FILE: ${{ env.war_file }}
         with:

doc/release-notes/11004-docker-block-solr.md

@@ -0,0 +1 @@
+The tutorial on running Dataverse in Docker has been updated to include [how to load a metadata block](https://dataverse-guide--11204.org.readthedocs.build/en/11204/container/running/demo.html#additional-metadata-blocks) and then update Solr to know about the new fields. See also #11004 and #11204

doc/release-notes/11191-assign-role-api-error-message.md

@@ -0,0 +1,6 @@
+### Modify Assign Role API error message
+
+Change error message from specifying 'User' and 'Dataverse' since it is used for groups and Datasets as well
+Generic message: datasets.api.grant.role.assignee.has.role.error=Role has already been granted.
+
+Note: Re-Translation needed.

doc/release-notes/11243-editmetadata-api-extension.md

@@ -0,0 +1,5 @@
+### Edit Dataset Metadata API extension
+
+- This endpoint now allows removing fields (by sending empty values), as long as they are not required by the dataset.
+- New ``sourceInternalVersionNumber`` optional query parameter, which prevents inconsistencies by managing updates that
+  may occur from other users while a dataset is being edited.

doc/sphinx-guides/source/_static/api/dataset-edit-metadata-delete-fields-sample.json

@@ -0,0 +1,50 @@
+{
+  "fields": [
+    {
+      "typeName": "alternativeTitle",
+      "multiple": true,
+      "typeClass": "primitive",
+      "value": []
+    },
+    {
+      "typeName": "distributor",
+      "multiple": true,
+      "typeClass": "compound",
+      "value": [
+        {
+          "distributorName": {
+            "typeName": "distributorName",
+            "multiple": false,
+            "typeClass": "primitive",
+            "value": ""
+          },
+          "distributorAffiliation": {
+            "typeName": "distributorAffiliation",
+            "multiple": false,
+            "typeClass": "primitive",
+            "value": ""
+          }
+        }
+      ]
+    },
+    {
+      "fields": [
+        {
+          "typeName": "author",
+          "value": [
+            {
+              "authorName": {
+                "typeName": "authorName",
+                "value": "Belicheck, Bill"
+              },
+              "authorAffiliation": {
+                "typeName": "authorIdentifierScheme",
+                "value": ""
+              }
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}

doc/sphinx-guides/source/_static/api/harvesting-client.json

@@ -7,5 +7,6 @@
   "metadataFormat": "oai_dc",
   "customHeaders": "x-oai-api-key: xxxyyyzzz",
   "set": "user-lmops",
+  "schedule": "Weekly, Sat 5 AM",
   "allowHarvestingMissingCVV":true
 }

doc/sphinx-guides/source/api/native-api.rst

@@ -2135,6 +2135,30 @@ The fully expanded example above (without environment variables) looks like this
 
 For these edits your JSON file need only include those dataset fields which you would like to edit. A sample JSON file may be downloaded here: :download:`dataset-edit-metadata-sample.json <../_static/api/dataset-edit-metadata-sample.json>` 
 
+This endpoint also allows removing fields, as long as they are not required by the dataset. To remove a field, send an empty value (``""``) for individual fields. For multiple fields, send an empty array (``[]``). A sample JSON file for removing fields may be downloaded here: :download:`dataset-edit-metadata-delete-fields-sample.json <../_static/api/dataset-edit-metadata-delete-fields-sample.json>`
+
+If another user updates the dataset version metadata before you send the update request, data inconsistencies may occur. To prevent this, you can use the optional ``sourceInternalVersionNumber`` query parameter. This parameter must include the internal version number corresponding to the dataset version being updated. Note that internal version numbers increase sequentially with each version update.
+
+If this parameter is provided, the update will proceed only if the internal version number remains unchanged. Otherwise, the request will fail with an error.
+
+Example using ``sourceInternalVersionNumber``:
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/BCCP9Z
+  export SOURCE_INTERNAL_VERSION_NUMBER=5
+
+  curl -H "X-Dataverse-key: $API_TOKEN" -X PUT "$SERVER_URL/api/datasets/:persistentId/editMetadata?persistentId=$PERSISTENT_IDENTIFIER&replace=true&sourceInternalVersionNumber=$SOURCE_INTERNAL_VERSION_NUMBER" --upload-file dataset-update-metadata.json
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/datasets/:persistentId/editMetadata/?persistentId=doi:10.5072/FK2/BCCP9Z&replace=true&sourceInternalVersionNumber=5" --upload-file dataset-update-metadata.json
+
+
 Delete Dataset Metadata
 ~~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/container/running/demo.rst

@@ -213,6 +213,54 @@ If you want to specify fewer previewers, you can edit the ``compose.yml`` file.
 
 ``INCLUDE_PREVIEWERS=text,html,pdf,csv``
 
+
+.. _additional-metadata-blocks:
+
+Additional Metadata Blocks
+++++++++++++++++++++++++++
+
+Metadata fields such as "Title" are part of a metadata block such as "Citation". See :ref:`metadata-references` in the User Guide for the metadata blocks that ship with Dataverse.
+
+At a high level, we will be loading a metadata block and then adjusting our Solr config to know about it.
+
+Care should be taken when adding additional metadata blocks. There is no way to `preview <https://github.com/IQSS/dataverse/issues/2551>`_ or `delete <https://github.com/IQSS/dataverse/issues/9628>`_ a metadata block so please use a throwaway environment.
+
+:ref:`metadata-references` lists some experimental metadata blocks. In the example below, we'll use the CodeMeta block.
+
+First, download a metadata block or create one by following :doc:`/admin/metadatacustomization` in the Admin Guide.
+
+Load the metadata block like this:
+
+``curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file codemeta.tsv``
+
+Next, reconfigure Solr to know about the new metadata block.
+
+You can back up your existing Solr schema like this:
+
+``cp docker-dev-volumes/solr/data/data/collection1/conf/schema.xml docker-dev-volumes/solr/data/data/collection1/conf/schema.xml.orig``
+
+You can see the existing fields Solr knows about like this:
+
+``curl http://localhost:8983/solr/collection1/schema/fields``
+
+Update your Solr schema with the following command:
+
+``curl http://localhost:8080/api/admin/index/solr/schema | docker run -i --rm -v ./docker-dev-volumes/solr/data:/var/solr gdcc/configbaker:unstable update-fields.sh /var/solr/data/collection1/conf/schema.xml``
+
+Then, reload Solr:
+
+``curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"``
+
+You can get a diff of your old and new Solr schema like this:
+
+``diff docker-dev-volumes/solr/data/data/collection1/conf/schema.xml.orig docker-dev-volumes/solr/data/data/collection1/conf/schema.xml``
+
+You should be able to see the new fields from the metadata block you added in the following output:
+
+``curl http://localhost:8983/solr/collection1/schema/fields``
+
+At this point you can proceed with testing the metadata block in the Dataverse UI. First you'll need to enable it for a collection (see :ref:`general-information` in the User Guide section about collection). Afterwards, create a new dataset, save it, and then edit the metadata for that dataset. Your metadata block should appear.
+
 Next Steps
 ----------
 

doc/sphinx-guides/source/developers/making-releases.rst

@@ -331,15 +331,19 @@ Deploy Final Release on Demo
 
 Above you already did the hard work of deploying a release candidate to https://demo.dataverse.org. It should be relatively straightforward to undeploy the release candidate and deploy the final release.
 
+.. _update-schemaspy:
+
 Update SchemaSpy
 ----------------
 
-We maintain SchemaSpy at URLs like https://guides.dataverse.org/en/6.3/schemaspy/index.html
+We maintain SchemaSpy at URLs like https://guides.dataverse.org/en/latest/schemaspy/index.html and (for example) https://guides.dataverse.org/en/6.6/schemaspy/index.html
 
 Get the attention of the core team and ask someone to update it for the new release.
 
 Consider updating `the thread <https://groups.google.com/g/dataverse-community/c/f95DQU-wlVM/m/cvUp3E9OBgAJ>`_ on the mailing list once the update is in place.
 
+See also :ref:`schemaspy`.
+
 Alert Translators About the New Release
 ---------------------------------------
 

doc/sphinx-guides/source/developers/testing.rst

@@ -477,25 +477,6 @@ reduced anyway.
 
 You will obviously have to utilize caching functionality of your CI service or do proper Docker layering.
 
-The Phoenix Server
-~~~~~~~~~~~~~~~~~~
-
-How the Phoenix Tests Work
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A server at http://phoenix.dataverse.org has been set up to test the latest code from the develop branch. Testing is done using chained builds of Jenkins jobs:
-
-- A war file is built from the latest code in develop: https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-build-develop/
-- The resulting war file is depoyed to the Phoenix server: https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-deploy-develop/
-- REST Assured Tests are run across the wire from the Jenkins server to the Phoenix server:  https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-apitest-develop/
-
-How to Run the Phoenix Tests
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-- Take a quick look at http://phoenix.dataverse.org to make sure the server is up and running Dataverse. If it's down, fix it.
-- Log into Jenkins and click "Build Now" at https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-build-develop/
-- Wait for all three chained Jenkins jobs to complete and note if they passed or failed. If you see a failure, open a GitHub issue or at least get the attention of some developers.
-
 Accessibility Testing
 ---------------------