10217 source name harvesting client

doc/release-notes/11217-source-name-harvesting-client.md

@@ -0,0 +1,13 @@
+### Metadata Source Facet Can Now Differentiate Between Harvested Sources
+
+The behavior of the feature flag `index-harvested-metadata-source` and the "Metadata Source" facet, which were added and updated, respectively, in [Dataverse 6.3](https://github.com/IQSS/dataverse/releases/tag/v6.3) (through pull requests #10464 and #10651), have been updated. A new field called "Source Name" has been added to harvesting clients.
+
+Before Dataverse 6.3, all harvested content (datasets and files) appeared together under "Harvested" under the "Metadata Source" facet. This is still the behavior of Dataverse out of the box. Since Dataverse 6.3, enabling the `index-harvested-metadata-source` feature flag (and reindexing) resulted in harvested content appearing under the nickname for whatever harvesting client was used to bring in the content. This meant that instead of having all harvested content lumped together under "Harvested", content would appear under "client1", "client2", etc.
+
+Now, as this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to group harvested content from various harvesting clients under the same name if you wish (by reusing the same source name).
+
+Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://dataverse-guide--11217.org.readthedocs.build/en/11217/installation/config.html#feature-flags). See also #10217 and #11217.
+
+## Upgrade instructions
+
+If you have enabled the `dataverse.feature.index-harvested-metadata-source` feature flag and given some of your harvesting clients a source name, you should reindex to have those source names appear under the "Metadata Source" facet.

doc/sphinx-guides/source/api/native-api.rst

@@ -5697,6 +5697,8 @@ Shows a Harvesting Client with a defined nickname::
     }
 
 
+.. _create-a-harvesting-client:
+
 Create a Harvesting Client
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -5715,6 +5717,7 @@ You must supply a JSON file that describes the configuration, similarly to the o
 
 The following optional fields are supported:
 
+- sourceName: When :ref:`feature-flags` `index-harvested-metadata-source` is enabled, it can overload the nickname in the Metadata source facet. It can be used to map many harvesting client under the same name.
 - archiveDescription: What the name suggests. If not supplied, will default to "This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data."
 - set: The OAI set on the remote server. If not supplied, will default to none, i.e., "harvest everything".
 - style: Defaults to "default" - a generic OAI archive. (Make sure to use "dataverse" when configuring harvesting from another Dataverse installation).

doc/sphinx-guides/source/installation/config.rst

@@ -3493,6 +3493,9 @@ please find all known feature flags below. Any of these flags can be activated u
     * - globus-use-experimental-async-framework
       - Activates a new experimental implementation of Globus polling of ongoing remote data transfers that does not rely on the instance staying up continuously for the duration of the transfers and saves the state information about Globus upload requests in the database. Added in v6.4. Affects :ref:`:GlobusPollingInterval`. Note that the JVM option :ref:`dataverse.files.globus-monitoring-server` described above must also be enabled on one (and only one, in a multi-node installation) Dataverse instance. 
       - ``Off``
+    * - index-harvested-metadata-source
+      - Index the nickname or the source name (See `sourceName` optional field in :ref:`create-a-harvesting-client`) of the harvesting client as the "Metadata Source" of harvested datasets and files; if enabled, the Metadata Source facet will show separate entries for the content harvested from different sources, instead of the current, default behavior where there is one "Harvested" facet for all such content.
+      - ``Off``
 
 **Note:** Feature flags can be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
 ``DATAVERSE_FEATURE_XXX`` (e.g. ``DATAVERSE_FEATURE_API_SESSION_AUTH=1``). These environment variables can be set in your shell before starting Payara. If you are using :doc:`Docker for development </container/dev-usage>`, you can set them in the `docker compose <https://docs.docker.com/compose/environment-variables/set-environment-variables/>`_ file.

docker-compose-dev.yml

@@ -17,6 +17,7 @@ services:
       SKIP_DEPLOY: "${SKIP_DEPLOY}"
       DATAVERSE_JSF_REFRESH_PERIOD: "1"
       DATAVERSE_FEATURE_API_BEARER_AUTH: "1"
+      DATAVERSE_FEATURE_INDEX_HARVESTED_METADATA_SOURCE: "1"
       DATAVERSE_FEATURE_API_BEARER_AUTH_PROVIDE_MISSING_CLAIMS: "1"
       DATAVERSE_MAIL_SYSTEM_EMAIL: "dataverse@localhost"
       DATAVERSE_MAIL_MTA_HOST: "smtp"

src/main/java/edu/harvard/iq/dataverse/HarvestingClientsPage.java

@@ -78,7 +78,7 @@ public class HarvestingClientsPage implements java.io.Serializable {
     private Long dataverseId = null;
     private HarvestingClient selectedClient;
     private boolean setListTruncated = false;
-    
+
     //private static final String solrDocIdentifierDataset = "dataset_";
 
     public enum PageMode {
@@ -242,6 +242,7 @@ public void editClient(HarvestingClient harvestingClient) {
         setSelectedClient(harvestingClient);
 
         this.newNickname = harvestingClient.getName();
+        this.sourceName = harvestingClient.getSourceName();
         this.newHarvestingUrl = harvestingClient.getHarvestingUrl();
         this.customHeader = harvestingClient.getCustomHttpHeaders();
         this.initialSettingsValidated = false;
@@ -323,10 +324,9 @@ public void deleteClient() {
     }
 
     public void createClient(ActionEvent ae) {
-
-        HarvestingClient newHarvestingClient = new HarvestingClient(); // will be set as type OAI by default
-
-        newHarvestingClient.setName(newNickname);
+
+        // will be set as type OAI by default
+        HarvestingClient newHarvestingClient = fillHarvestingClient(new HarvestingClient());
 
         if (getSelectedDestinationDataverse() == null) {
             JsfHelper.JH.addMessage(FacesMessage.SEVERITY_ERROR,BundleUtil.getStringFromBundle("harvest.create.error"));
@@ -338,35 +338,6 @@ public void createClient(ActionEvent ae) {
         }
         getSelectedDestinationDataverse().getHarvestingClientConfigs().add(newHarvestingClient);
 
-        newHarvestingClient.setHarvestingUrl(newHarvestingUrl);
-        newHarvestingClient.setCustomHttpHeaders(customHeader);
-        if (!StringUtils.isEmpty(newOaiSet)) {
-            newHarvestingClient.setHarvestingSet(newOaiSet);
-        }
-        newHarvestingClient.setMetadataPrefix(newMetadataFormat);
-        newHarvestingClient.setHarvestStyle(newHarvestingStyle);
-
-        if (isNewHarvestingScheduled()) {
-            newHarvestingClient.setScheduled(true);
-
-            if (isNewHarvestingScheduledWeekly()) {
-                newHarvestingClient.setSchedulePeriod(HarvestingClient.SCHEDULE_PERIOD_WEEKLY);
-                if (getWeekDayNumber() == null) {
-                    // create a "week day is required..." error message, etc. 
-                    // but we may be better off not even giving them an opportunity 
-                    // to leave the field blank - ?
-                }
-                newHarvestingClient.setScheduleDayOfWeek(getWeekDayNumber());
-            } else {
-                newHarvestingClient.setSchedulePeriod(HarvestingClient.SCHEDULE_PERIOD_DAILY);
-            }
-
-            if (getHourOfDay() == null) {
-                // see the comment above, about the day of week. same here.
-            }
-            newHarvestingClient.setScheduleHourOfDay(getHourOfDay());
-        }
-
         // make default archive url (used to generate links pointing back to the 
         // archival sources, when harvested datasets are displayed in search results),
         // from the harvesting url:
@@ -412,51 +383,9 @@ public void createClient(ActionEvent ae) {
     // this saves an existing client that the user has edited: 
 
     public void saveClient(ActionEvent ae) {
-
-        HarvestingClient harvestingClient = getSelectedClient(); 
-
-        if (harvestingClient == null) {
-            // TODO: 
-            // tell the user somehow that the client cannot be saved, and advise
-            // them to save the settings they have entered. 
-            // as of now - we will show an error message, but only after the 
-            // edit form has been closed.        
-        }
-
-        // nickname is not editable for existing clients:
-        //harvestingClient.setName(newNickname);
-        harvestingClient.setHarvestingUrl(newHarvestingUrl);
-        harvestingClient.setCustomHttpHeaders(customHeader);
-        harvestingClient.setHarvestingSet(newOaiSet);
-        harvestingClient.setMetadataPrefix(newMetadataFormat);
-        harvestingClient.setHarvestStyle(newHarvestingStyle);
-
-        if (isNewHarvestingScheduled()) {
-            harvestingClient.setScheduled(true);
-
-            if (isNewHarvestingScheduledWeekly()) {
-                harvestingClient.setSchedulePeriod(HarvestingClient.SCHEDULE_PERIOD_WEEKLY);
-                if (getWeekDayNumber() == null) {
-                    // create a "week day is required..." error message, etc. 
-                    // but we may be better off not even giving them an opportunity 
-                    // to leave the field blank - ?
-                }
-                harvestingClient.setScheduleDayOfWeek(getWeekDayNumber());
-            } else {
-                harvestingClient.setSchedulePeriod(HarvestingClient.SCHEDULE_PERIOD_DAILY);
-            }
-
-            if (getHourOfDay() == null) {
-                // see the comment above, about the day of week. same here.
-            }
-            harvestingClient.setScheduleHourOfDay(getHourOfDay());
-        } else {
-            harvestingClient.setScheduled(false);
-        }
-
-        // will try to save it now:
-
         try {
+            HarvestingClient harvestingClient = fillHarvestingClient(getSelectedClient());
+
             harvestingClient = engineService.submit( new UpdateHarvestingClientCommand(dvRequestService.getDataverseRequest(), harvestingClient));
 
             configuredHarvestingClients = harvestingClientService.getAllHarvestingClients();
@@ -477,9 +406,50 @@ public void saveClient(ActionEvent ae) {
         }
         setPageMode(PageMode.VIEW);
 
-        
+
     }
-
+
+    /**
+     * Based on a new harvestingClient instance or an existing one, it will update basics fields with new UI fields values
+     * @param harvestingClient new or existing harvestingClient to update
+     * @return harvestingClient with updated values
+     */
+    private HarvestingClient fillHarvestingClient(HarvestingClient harvestingClient) {
+        // update nickname if it's a new object otherwise is not editable for existing clients
+        if(harvestingClient.getId() == null) {
+            harvestingClient.setName(newNickname);
+        }
+        harvestingClient.setSourceName(sourceName);
+        harvestingClient.setHarvestingUrl(newHarvestingUrl);
+        harvestingClient.setCustomHttpHeaders(customHeader);
+        if (!StringUtils.isEmpty(newOaiSet)) {
+            harvestingClient.setHarvestingSet(newOaiSet);
+        }
+        harvestingClient.setMetadataPrefix(newMetadataFormat);
+        harvestingClient.setHarvestStyle(newHarvestingStyle);
+
+        harvestingClient.setScheduled(isNewHarvestingScheduled());
+        if (isNewHarvestingScheduled()) {
+            if (isNewHarvestingScheduledWeekly()) {
+                harvestingClient.setSchedulePeriod(HarvestingClient.SCHEDULE_PERIOD_WEEKLY);
+                if (getWeekDayNumber() == null) {
+                    // create a "week day is required..." error message, etc.
+                    // but we may be better off not even giving them an opportunity
+                    // to leave the field blank - ?
+                }
+                harvestingClient.setScheduleDayOfWeek(getWeekDayNumber());
+            } else {
+                harvestingClient.setSchedulePeriod(HarvestingClient.SCHEDULE_PERIOD_DAILY);
+            }
+
+            if (getHourOfDay() == null) {
+                // see the comment above, about the day of week. same here.
+            }
+            harvestingClient.setScheduleHourOfDay(getHourOfDay());
+        }
+        return harvestingClient;
+    }
+
     public void validateMetadataFormat(FacesContext context, UIComponent toValidate, Object rawValue) {
         String value = (String) rawValue;
         UIInput input = (UIInput) toValidate;
@@ -717,6 +687,7 @@ public void backToStepThree() {
     UIInput selectedDataverseMenu;
 
     private String newNickname = "";
+    private String sourceName = "";
     private String newHarvestingUrl = "";
     private String customHeader = null; 
     private boolean initialSettingsValidated = false;
@@ -741,6 +712,7 @@ public void backToStepThree() {
     public void initNewClient(ActionEvent ae) {
         //this.selectedClient = new HarvestingClient();
         this.newNickname = "";
+        this.sourceName = "";
         this.newHarvestingUrl = "";
         this.customHeader = null; 
         this.initialSettingsValidated = false;
@@ -842,6 +814,14 @@ public int getHarvestingScheduleRadio() {
     public void setHarvestingScheduleRadio(int harvestingScheduleRadio) {
         this.harvestingScheduleRadio = harvestingScheduleRadio;
     }
+
+    public String getSourceName() {
+        return sourceName;
+    }
+
+    public void setSourceName(String sourceName) {
+        this.sourceName = sourceName;
+    }
 
     public boolean isNewHarvestingScheduled() {
         return this.harvestingScheduleRadio != harvestingScheduleRadioNone;

src/main/java/edu/harvard/iq/dataverse/api/HarvestingClients.java

@@ -278,7 +278,10 @@ public Response modifyHarvestingClient(@Context ContainerRequestContext crc, Str
             // Go through the supported editable fields and update the client accordingly: 
             // TODO: We may want to reevaluate whether we really want/need *all*
             // of these fields to be editable.
-
+
+            if (newHarvestingClient.getSourceName() != null) {
+                harvestingClient.setSourceName(newHarvestingClient.getSourceName());
+            }
             if (newHarvestingClient.getHarvestingUrl() != null) {
                 harvestingClient.setHarvestingUrl(newHarvestingClient.getHarvestingUrl());
             }

src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClient.java

@@ -29,13 +29,11 @@
 import jakarta.persistence.NamedQueries;
 import jakarta.persistence.NamedQuery;
 import jakarta.persistence.OneToMany;
-import jakarta.persistence.OneToOne;
 import jakarta.persistence.OrderBy;
 import jakarta.persistence.Table;
-import jakarta.persistence.Temporal;
-import jakarta.persistence.TemporalType;
 import jakarta.validation.constraints.Pattern;
 import jakarta.validation.constraints.Size;
+import org.apache.commons.lang3.StringUtils;
 import org.hibernate.validator.constraints.NotBlank;
 
 /**
@@ -192,6 +190,20 @@ public void setHarvestingUrl(String harvestingUrl) {
             this.harvestingUrl = harvestingUrl.trim();
         }
     }
+
+    private String sourceName;
+
+    public String getSourceName() {
+        return sourceName;
+    }
+
+    public void setSourceName(String sourceName) {
+        this.sourceName = sourceName;
+    }
+
+    public String getMetadataSource() {
+        return StringUtils.isNotBlank(this.sourceName) ? this.sourceName : this.name;
+    }
 
     private String archiveUrl; 
 
@@ -476,5 +488,4 @@ public boolean equals(Object object) {
     public String toString() {
         return "edu.harvard.iq.dataverse.harvest.client.HarvestingClient[ id=" + id + " ]";
     }
-
 }

src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java

@@ -1005,7 +1005,7 @@ public SolrInputDocuments toSolrDocs(IndexableDataset indexableDataset, Set<Long
                 // New - as of 6.3 - option of indexing the actual origin of 
                 // harvested objects as the metadata source:
                 solrInputDocument.addField(SearchFields.METADATA_SOURCE,
-                                        dataset.getHarvestedFrom() != null ? dataset.getHarvestedFrom().getName() : HARVESTED);
+                                        dataset.getHarvestedFrom() != null ? dataset.getHarvestedFrom().getMetadataSource() : HARVESTED);
             } else {
                 solrInputDocument.addField(SearchFields.METADATA_SOURCE, HARVESTED);
             }
@@ -1577,7 +1577,7 @@ public SolrInputDocuments toSolrDocs(IndexableDataset indexableDataset, Set<Long
                                 // New - as of 6.3 - option of indexing the actual origin of 
                                 // harvested objects as the metadata source:
                                 datafileSolrInputDocument.addField(SearchFields.METADATA_SOURCE,
-                                        dataset.getHarvestedFrom() != null ? dataset.getHarvestedFrom().getName() : HARVESTED);
+                                        dataset.getHarvestedFrom() != null ? dataset.getHarvestedFrom().getMetadataSource() : HARVESTED);
                             } else {
                                 datafileSolrInputDocument.addField(SearchFields.METADATA_SOURCE, HARVESTED);
                             }

src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java

@@ -1058,6 +1058,7 @@ public String parseHarvestingClient(JsonObject obj, HarvestingClient harvestingC
         String dataverseAlias = obj.getString("dataverseAlias",null);
 
         harvestingClient.setName(obj.getString("nickName",null));
+        harvestingClient.setSourceName(obj.getString("sourceName",null));
         harvestingClient.setHarvestStyle(obj.getString("style", "default"));
         harvestingClient.setHarvestingUrl(obj.getString("harvestUrl",null));
         harvestingClient.setArchiveUrl(obj.getString("archiveUrl",null));

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

@@ -1051,6 +1051,7 @@ public static JsonObjectBuilder json(HarvestingClient harvestingClient) {
         }
 
         return jsonObjectBuilder().add("nickName", harvestingClient.getName()).
+                add("sourceName", harvestingClient.getSourceName()).
                 add("dataverseAlias", harvestingClient.getDataverse().getAlias()).
                 add("type", harvestingClient.getHarvestType()).
                 add("style", harvestingClient.getHarvestStyle()).

src/main/java/propertyFiles/Bundle.properties

@@ -579,6 +579,8 @@ harvestclients.newClientDialog.nickname.helptext=Consists of letters, digits, un
 harvestclients.newClientDialog.nickname.required=Client nickname cannot be empty!
 harvestclients.newClientDialog.nickname.invalid=Client nickname can contain only letters, digits, underscores (_) and dashes (-); and must be at most 30 characters.
 harvestclients.newClientDialog.nickname.alreadyused=This nickname is already used.
+harvestclients.newClientDialog.sourcename=Source name
+harvestclients.newClientDialog.sourcename.helptext=When feature flag index-harvested-metadata-source is enabled, it can overload the nickname in the Metadata source facet. It can be used to map many harvesting client under the same name.
 harvestclients.newClientDialog.customHeader=Custom HTTP Header
 harvestclients.newClientDialog.customHeader.helptext=(Optional) Custom HTTP header to add to requests, if required by this OAI server.
 harvestclients.newClientDialog.customHeader.watermark=Enter an http header, as in header-name: header-value

src/main/resources/db/migration/V6.5.0.6.sql

@@ -0,0 +1,2 @@
+-- Add this text will help to customized name in metadata source facet
+ALTER TABLE harvestingclient ADD COLUMN IF NOT EXISTS sourcename TEXT;