Merge branch 'develop' into 10404-fix-NPE

Author: luddaniel

doc/release-notes/10554-avoid-solr-join-guest.md

@@ -0,0 +1,5 @@
+Two experimental features flag called "add-publicobject-solr-field" and "avoid-expensive-solr-join" have been added to change how Solr documents are indexed for public objects and how Solr queries are constructed to accommodate access to restricted content (drafts, etc.). It is hoped that it will help with performance, especially on large instances and under load.
+
+Before the search feature flag ("avoid-expensive...") can be turned on, the indexing flag must be enabled, and a full reindex performed. Otherwise publicly available objects are NOT going to be shown in search results.
+
+For details see https://dataverse-guide--10555.org.readthedocs.build/en/10555/installation/config.html#feature-flags and #10555.

doc/release-notes/10568-Fix File Reingest.md

@@ -0,0 +1 @@
+A bug that prevented the Ingest option in the File page Edit File menu from working has been fixed

doc/sphinx-guides/source/admin/harvestclients.rst

@@ -47,3 +47,8 @@ What if a Run Fails?
 Each harvesting client run logs a separate file per run to the app server's default logging directory (``/usr/local/payara6/glassfish/domains/domain1/logs/`` unless you've changed it). Look for filenames in the format  ``harvest_TARGET_YYYY_MM_DD_timestamp.log`` to get a better idea of what's going wrong.
 
 Note that you'll want to run a minimum of Dataverse Software 4.6, optimally 4.18 or beyond, for the best OAI-PMH interoperability.
+
+Harvesting Non-OAI-PMH
+~~~~~~~~~~~~~~~~~~~~~~
+
+`DOI2PMH <https://github.com/IQSS/doi2pmh-server>`__ is a community-driven project intended to allow OAI-PMH harvesting from non-OAI-PMH sources.  

doc/sphinx-guides/source/api/apps.rst

@@ -133,6 +133,13 @@ https://github.com/libis/rdm-integration
 PHP
 ---
 
+DOI2PMH
+~~~~~~~
+
+The DOI2PMH server allow Dataverse instances to harvest DOI through OAI-PMH from otherwise unharvestable sources.
+
+https://github.com/IQSS/doi2pmh-server
+
 OJS
 ~~~
 

doc/sphinx-guides/source/api/native-api.rst

@@ -1179,7 +1179,7 @@ See also :ref:`batch-exports-through-the-api` and the note below:
   export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB
   export METADATA_FORMAT=ddi
 
-  curl "$SERVER_URL/api/datasets/export?exporter=$METADATA_FORMAT&persistentId=PERSISTENT_IDENTIFIER"
+  curl "$SERVER_URL/api/datasets/export?exporter=$METADATA_FORMAT&persistentId=$PERSISTENT_IDENTIFIER"
 
 The fully expanded example above (without environment variables) looks like this:
 

doc/sphinx-guides/source/developers/deployment.rst

@@ -91,17 +91,11 @@ Download `ec2-create-instance.sh`_ and put it somewhere reasonable. For the purp
 
 .. _ec2-create-instance.sh: https://raw.githubusercontent.com/GlobalDataverseCommunityConsortium/dataverse-ansible/master/ec2/ec2-create-instance.sh
 
-To run it with default values you just need the script, but you may also want a current copy of the ansible `group vars <https://raw.githubusercontent.com/GlobalDataverseCommunityConsortium/dataverse-ansible/master/defaults/main.yml>`_ file.
+To run the script, you can make it executable (``chmod 755 ec2-create-instance.sh``) or run it with bash, like this with ``-h`` as an argument to print the help:
 
-ec2-create-instance accepts a number of command-line switches, including:
+``bash ~/Downloads/ec2-create-instance.sh -h``
 
-* -r: GitHub Repository URL (defaults to https://github.com/IQSS/dataverse.git)
-* -b: branch to build (defaults to develop)
-* -p: pemfile directory (defaults to $HOME)
-* -g: Ansible GroupVars file (if you wish to override role defaults)
-* -h: help (displays usage for each available option)
-
-``bash ~/Downloads/ec2-create-instance.sh -b develop -r https://github.com/scholarsportal/dataverse.git -g main.yml``
+If you run the script without any arguments, it should spin up the latest version of Dataverse.
 
 You will need to wait for 15 minutes or so until the deployment is finished, longer if you've enabled sample data and/or the API test suite. Eventually, the output should tell you how to access the Dataverse installation in a web browser or via SSH. It will also provide instructions on how to delete the instance when you are finished with it. Please be aware that AWS charges per minute for a running instance. You may also delete your instance from https://console.aws.amazon.com/console/home?region=us-east-1 .
 

doc/sphinx-guides/source/developers/performance.rst

@@ -118,6 +118,10 @@ Solr
 
 While in the past Solr performance hasn't been much of a concern, in recent years we've noticed performance problems when Harvard Dataverse is under load. Improvements were made in `PR #10050 <https://github.com/IQSS/dataverse/pull/10050>`_, for example.
 
+We are tracking performance problems in `#10469 <https://github.com/IQSS/dataverse/issues/10469>`_.
+
+In a meeting with a Solr expert on 2024-05-10 we were advised to avoid joins as much as possible. (It was acknowledged that many Solr users make use of joins because they have to, like we do, to keep some documents private.) Toward that end we have added two feature flags called ``avoid-expensive-solr-join`` and ``add-publicobject-solr-field`` as explained under :ref:`feature-flags`. It was confirmed experimentally that performing the join on all the public objects (published collections, datasets and files), i.e., the bulk of the content in the search index, was indeed very expensive, especially on a large instance the size of the IQSS prod. archive, especially under indexing load. We confirmed that it was in fact unnecessary and were able to replace it with a boolean field directly in the indexed documents, which is achieved by the two feature flags above. However, as of writing this, this mechanism should still be considered experimental.
+
 Datasets with Large Numbers of Files or Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/installation/config.rst

@@ -3268,6 +3268,12 @@ please find all known feature flags below. Any of these flags can be activated u
     * - api-session-auth
       - Enables API authentication via session cookie (JSESSIONID). **Caution: Enabling this feature flag exposes the installation to CSRF risks!** We expect this feature flag to be temporary (only used by frontend developers, see `#9063 <https://github.com/IQSS/dataverse/issues/9063>`_) and for the feature to be removed in the future.
       - ``Off``
+    * - avoid-expensive-solr-join
+      - Changes the way Solr queries are constructed for public content (published Collections, Datasets and Files). It removes a very expensive Solr join on all such documents, improving overall performance, especially for large instances under heavy load. Before this feature flag is enabled, the corresponding indexing feature (see next feature flag) must be turned on and a full reindex performed (otherwise public objects are not going to be shown in search results). See :doc:`/admin/solr-search-index`. 
+      - ``Off``
+    * - add-publicobject-solr-field
+      - Adds an extra boolean field `PublicObject_b:true` for public content (published Collections, Datasets and Files). Once reindexed with these fields, we can rely on it to remove a very expensive Solr join on all such documents in Solr queries, significantly improving overall performance (by enabling the feature flag above, `avoid-expensive-solr-join`). These two flags are separate so that an instance can reindex their holdings before enabling the optimization in searches, thus avoiding having their public objects temporarily disappear from search results while the reindexing is in progress. 
+      - ``Off``
 
 **Note:** Feature flags can be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
 ``DATAVERSE_FEATURE_XXX`` (e.g. ``DATAVERSE_FEATURE_API_SESSION_AUTH=1``). These environment variables can be set in your shell before starting Payara. If you are using :doc:`Docker for development </container/dev-usage>`, you can set them in the `docker compose <https://docs.docker.com/compose/environment-variables/set-environment-variables/>`_ file.

src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java

@@ -19,8 +19,6 @@
 import edu.harvard.iq.dataverse.export.ExportService;
 import edu.harvard.iq.dataverse.globus.GlobusServiceBean;
 import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean;
-import edu.harvard.iq.dataverse.pidproviders.PidProvider;
-import edu.harvard.iq.dataverse.pidproviders.PidUtil;
 import edu.harvard.iq.dataverse.search.IndexServiceBean;
 import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
 import edu.harvard.iq.dataverse.util.BundleUtil;
@@ -41,11 +39,10 @@
 import jakarta.ejb.TransactionAttributeType;
 import jakarta.inject.Named;
 import jakarta.persistence.EntityManager;
-import jakarta.persistence.LockModeType;
 import jakarta.persistence.NoResultException;
+import jakarta.persistence.NonUniqueResultException;
 import jakarta.persistence.PersistenceContext;
 import jakarta.persistence.Query;
-import jakarta.persistence.StoredProcedureQuery;
 import jakarta.persistence.TypedQuery;
 import org.apache.commons.lang3.StringUtils;
 
@@ -115,28 +112,32 @@ public Dataset find(Object pk) {
      * @return a dataset with pre-fetched file objects
      */
     public Dataset findDeep(Object pk) {
-        return (Dataset) em.createNamedQuery("Dataset.findById")
-            .setParameter("id", pk)
-            // Optimization hints: retrieve all data in one query; this prevents point queries when iterating over the files
-            .setHint("eclipselink.left-join-fetch", "o.files.ingestRequest")
-            .setHint("eclipselink.left-join-fetch", "o.files.thumbnailForDataset")
-            .setHint("eclipselink.left-join-fetch", "o.files.dataTables")
-            .setHint("eclipselink.left-join-fetch", "o.files.auxiliaryFiles")
-            .setHint("eclipselink.left-join-fetch", "o.files.ingestReports")
-            .setHint("eclipselink.left-join-fetch", "o.files.dataFileTags")
-            .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas")
-            .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.fileCategories")
-            .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.varGroups")
-            //.setHint("eclipselink.left-join-fetch", "o.files.guestbookResponses
-            .setHint("eclipselink.left-join-fetch", "o.files.embargo")
-            .setHint("eclipselink.left-join-fetch", "o.files.retention")
-            .setHint("eclipselink.left-join-fetch", "o.files.fileAccessRequests")
-            .setHint("eclipselink.left-join-fetch", "o.files.owner")
-            .setHint("eclipselink.left-join-fetch", "o.files.releaseUser")
-            .setHint("eclipselink.left-join-fetch", "o.files.creator")
-            .setHint("eclipselink.left-join-fetch", "o.files.alternativePersistentIndentifiers")
-            .setHint("eclipselink.left-join-fetch", "o.files.roleAssignments")
-            .getSingleResult();
+        try {
+            return (Dataset) em.createNamedQuery("Dataset.findById")
+                    .setParameter("id", pk)
+                    // Optimization hints: retrieve all data in one query; this prevents point queries when iterating over the files
+                    .setHint("eclipselink.left-join-fetch", "o.files.ingestRequest")
+                    .setHint("eclipselink.left-join-fetch", "o.files.thumbnailForDataset")
+                    .setHint("eclipselink.left-join-fetch", "o.files.dataTables")
+                    .setHint("eclipselink.left-join-fetch", "o.files.auxiliaryFiles")
+                    .setHint("eclipselink.left-join-fetch", "o.files.ingestReports")
+                    .setHint("eclipselink.left-join-fetch", "o.files.dataFileTags")
+                    .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas")
+                    .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.fileCategories")
+                    .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.varGroups")
+                    //.setHint("eclipselink.left-join-fetch", "o.files.guestbookResponses
+                    .setHint("eclipselink.left-join-fetch", "o.files.embargo")
+                    .setHint("eclipselink.left-join-fetch", "o.files.retention")
+                    .setHint("eclipselink.left-join-fetch", "o.files.fileAccessRequests")
+                    .setHint("eclipselink.left-join-fetch", "o.files.owner")
+                    .setHint("eclipselink.left-join-fetch", "o.files.releaseUser")
+                    .setHint("eclipselink.left-join-fetch", "o.files.creator")
+                    .setHint("eclipselink.left-join-fetch", "o.files.alternativePersistentIndentifiers")
+                    .setHint("eclipselink.left-join-fetch", "o.files.roleAssignments")
+                    .getSingleResult();
+        } catch (NoResultException | NonUniqueResultException ex) {
+            return null;
+        }
     }
 
     public List<Dataset> findByOwnerId(Long ownerId) {

src/main/java/edu/harvard/iq/dataverse/FilePage.java

@@ -522,10 +522,9 @@ public String ingestFile() throws CommandException{
             return null;
         }
 
-        DataFile dataFile = fileMetadata.getDataFile();
-        editDataset = dataFile.getOwner();
+        editDataset = file.getOwner();
 
-        if (dataFile.isTabularData()) {
+        if (file.isTabularData()) {
             JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.ingest.alreadyIngestedWarning"));
             return null;
         }
@@ -537,25 +536,25 @@ public String ingestFile() throws CommandException{
             return null;
         }
 
-        if (!FileUtil.canIngestAsTabular(dataFile)) {
+        if (!FileUtil.canIngestAsTabular(file)) {
             JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.ingest.cantIngestFileWarning"));
             return null;
 
         }
 
-        dataFile.SetIngestScheduled();
+        file.SetIngestScheduled();
 
-        if (dataFile.getIngestRequest() == null) {
-            dataFile.setIngestRequest(new IngestRequest(dataFile));
+        if (file.getIngestRequest() == null) {
+            file.setIngestRequest(new IngestRequest(file));
         }
 
-        dataFile.getIngestRequest().setForceTypeCheck(true);
+        file.getIngestRequest().setForceTypeCheck(true);
 
         // update the datafile, to save the newIngest request in the database:
         datafileService.save(file);
 
         // queue the data ingest job for asynchronous execution: 
-        String status = ingestService.startIngestJobs(editDataset.getId(), new ArrayList<>(Arrays.asList(dataFile)), (AuthenticatedUser) session.getUser());
+        String status = ingestService.startIngestJobs(editDataset.getId(), new ArrayList<>(Arrays.asList(file)), (AuthenticatedUser) session.getUser());
 
         if (!StringUtil.isEmpty(status)) {
             // This most likely indicates some sort of a problem (for example, 
@@ -565,9 +564,9 @@ public String ingestFile() throws CommandException{
             // successfully gone through the process of trying to schedule the 
             // ingest job...
 
-            logger.warning("Ingest Status for file: " + dataFile.getId() + " : " + status);
+            logger.warning("Ingest Status for file: " + file.getId() + " : " + status);
         }
-        logger.fine("File: " + dataFile.getId() + " ingest queued");
+        logger.fine("File: " + file.getId() + " ingest queued");
 
         init();
         JsfHelper.addInfoMessage(BundleUtil.getStringFromBundle("file.ingest.ingestQueued"));