Merge branch 'indexingperf2' into indexingperf3

Author: qqmyers

doc/release-notes/11085-suppress-breadcrumbs

@@ -0,0 +1 @@
+This release fixes a bug which was allowing the viewing of host collections' names when using anonymized preview URLS. 

doc/release-notes/11606-hide-spa-oidc-providers-from-jsf-login-screen.md

@@ -0,0 +1,7 @@
+This release fixes a bug where the value of the dataverse.auth.oidc.enabled setting, available when Provisioning an authentication provider via JVM options (see ref: https://guides.dataverse.org/en/latest/installation/oidc.html#provision-via-jvm-options) was not being not being propagated to the current Dataverse user interface (where enabled=false providers are not displayed for login/registration) or represented in the GET api/admin/authenticationProviders API call.
+
+A new JVM setting ('dataverse.auth.oidc.hidden-jsf') was added to hide an enabled OIDC Provider from the JSF UI. 
+
+For Dataverse instances deploying both the current JSF UI and the new SPA UI, this fix allows the OIDC Keycloak provider configured for the SPA to be hidden in the JSF UI (useful in cases where it would duplicate other configured providers).
+
+Note: The API to create a new Auth Provider can only be used to create a provider for both JSF and SPA. Use JVM / MicroProfile config setting to create SPA only providers.

doc/release-notes/11740-api-file-download-with-bearer-token.md

@@ -0,0 +1,7 @@
+## Bug / Not Bug in Dataverse. Bug is in SPA Frontend
+
+Cleaned up Access APIs to localize getting user from session for JSF backward compatibility
+
+This bug requires a front end fix to send the Bearer Token in the API call.
+
+See: #11740

doc/release-notes/11787-signed-url-improvement.md

@@ -0,0 +1 @@
+In prior versions of Dataverse, configuring a proxy to forward to Dataverse over an http connection could result in failure of signed Urls (e.g. for external tools). This version of Dataverse supports having a proxy send an X-Forwarded-Proto header set to https to avoid this issue. 

doc/release-notes/359-enhance-publishing-message-acknowledgement.md

@@ -0,0 +1,21 @@
+## Publishing Enhancement ##
+
+Before a Dataset can be published the user must acknowledge acceptance of the disclaimer if it is required.
+
+The setting "PublishDatasetDisclaimerText", when set, will prevent a draft dataset from being published without the user acknowledging the disclaimer.
+The approved disclaimer text is `"By publishing this dataset, I fully accept all legal responsibility for ensuring that the deposited content is: anonymized, free of copyright violations, and contains data that is computationally reusable. I understand and agree that any violation of these conditions may result in the immediate removal of the dataset by the repository without prior notice."`
+
+To enable/disable the acknowledgement requirement an Admin can set/delete the setting using the following APIs:
+
+`curl -X PUT -d "By publishing this dataset, I fully accept all legal responsibility for ensuring that the deposited content is: anonymized, free of copyright violations, and contains data that is computationally reusable. I understand and agree that any violation of these conditions may result in the immediate removal of the dataset by the repository without prior notice." http://localhost:8080/api/admin/settings/:PublishDatasetDisclaimerText` 
+
+`curl -X DELETE http://localhost:8080/api/admin/settings/:PublishDatasetDisclaimerText`
+
+The UI will prevent the user from publishing a Dataset unless the disclaimer is acknowledged.
+
+The APIs will continue to publish without the acknowledgement for now. An Info API getter was added for non-superusers to get the disclaimer text.
+
+`curl -X GET http://localhost:8080/api/info/settings/:PublishDatasetDisclaimerText`
+
+See:
+- [#359](https://github.com/IQSS/dataverse.harvard.edu/issues/359)  

doc/sphinx-guides/source/api/external-tools.rst

@@ -174,6 +174,10 @@ The signed URL mechanism is more secure than exposing API tokens and therefore r
 - For tools invoked via a GET call, Dataverse will include a callback query parameter with a Base64 encoded value. The decoded value is a signed URL that can be called to retrieve a JSON response containing all of the queryParameters and allowedApiCalls specified in the manfiest.
 - For tools invoked via POST, Dataverse will send a JSON body including the requested queryParameters and allowedApiCalls. Dataverse expects the response to the POST to indicate a redirect which Dataverse will use to open the tool.
 
+.. note::
+
+   **For Dataverse site administrators:** When Dataverse is behind a proxy, signed URLs may not work correctly due to protocol mismatches (HTTP vs HTTPS). Please refer to the :ref:`signed-urls-forwarded-proto-header` section to ensure signed URLs work properly in proxy environments.
+
 API Token
 ^^^^^^^^^
 

doc/sphinx-guides/source/api/native-api.rst

@@ -6366,11 +6366,14 @@ The fully expanded example above (without environment variables) looks like this
 
   curl "https://demo.dataverse.org/api/info/server"
 
+.. _show-custom-popup-for-publishing-datasets:
+
 Show Custom Popup Text for Publishing Datasets
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 For now, only the value for the :ref:`:DatasetPublishPopupCustomText` setting from the Configuration section of the Installation Guide is exposed:
 
+.. note:: See :ref:`show-disclaimer-for-publishing-datasets` if you want the user to acknowledge before publishing.
 .. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
 
 .. code-block:: bash
@@ -6385,6 +6388,28 @@ The fully expanded example above (without environment variables) looks like this
 
   curl "https://demo.dataverse.org/api/info/settings/:DatasetPublishPopupCustomText"
 
+.. _show-disclaimer-for-publishing-datasets:
+
+Show Disclaimer for Publishing Datasets
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The setting "PublishDatasetDisclaimerText", when set, will prevent a draft dataset from being published through the UI without the user acknowledging the disclaimer.
+
+.. note:: See :ref:`show-custom-popup-for-publishing-datasets` if the user acknowledgment is not required but you want the message to be displayed in the UI.
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+
+  curl "$SERVER_URL/api/info/settings/:PublishDatasetDisclaimerText"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl "https://demo.dataverse.org/api/info/settings/:PublishDatasetDisclaimerText"
+
 .. _api-get-app-tou:
 
 Get Application Terms of Use (General Terms of Use)
@@ -7654,6 +7679,8 @@ Add new authentication provider. The POST data is in JSON format, similar to the
 
   POST http://$SERVER/api/admin/authenticationProviders
 
+.. note:: This endpoint will create providers for both JSF and SPA. Use :ref:`jvm-options` / *MicroProfile Config* if you need to create SPA only providers.
+
 Show Authentication Provider
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/installation/config.rst

@@ -94,6 +94,16 @@ First of all, confirm that access is denied! If you are in fact able to access t
 
 Still feel like activating this option in your configuration? - Have fun and be safe!
 
+.. _signed-urls-forwarded-proto-header:
+
+Using X-Forwarded-Proto for Signed URLs
++++++++++++++++++++++++++++++++++++++++
+
+If you use a proxy such as Apache or Nginx, or have a firewall such as Anubis, and they are configured to forward traffic to Dataverse over HTTP
+(i.e. your proxy receives user calls over HTTPS but forwards locally to Dataverse over HTTP), signed URLs, used by external tools and 
+upload apps (such as DVWebloader), are likely to fail unless you configure your proxy to send an X-Forwarded-Proto HTTP Header.
+This allows Dataverse to recognize that the communication from the user was over HTTPS and that validation of signed URLs should assume 
+they started with https:// (rather than http:// as received from the proxy).
 
 .. _PrivacyConsiderations:
 
@@ -5279,6 +5289,15 @@ This post-publish workflow is useful for actions such as sending notifications a
 
 See :ref:`Workflow Admin section <workflow_admin>` for more details and context.
 
+.. _:PublishDatasetDisclaimerText:
+
+:PublishDatasetDisclaimerText
++++++++++++++++++++++++++++++
+
+The text displayed to the user that must be acknowledged prior to publishing a Dataset. When not set the acknowledgment is not required nor displayed.
+
+``curl -X PUT -d "By publishing this dataset, I fully accept all legal responsibility for ensuring that the deposited content is: anonymized, free of copyright violations, and contains data that is computationally reusable. I understand and agree that any violation of these conditions may result in the immediate removal of the dataset by the repository without prior notice." http://localhost:8080/api/admin/settings/:PublishDatasetDisclaimerText``
+
 .. _:BagItHandlerEnabled:
 
 :BagItHandlerEnabled

doc/sphinx-guides/source/installation/oidc.rst

@@ -151,6 +151,10 @@ The following options are available:
     - Enable or disable provisioning the provider via MicroProfile.
     - N
     - ``false``
+  * - ``dataverse.auth.oidc.hidden-jsf``
+    - Show or Hide the provider from the JSF UI via MicroProfile.
+    - N
+    - ``false``
   * - ``dataverse.auth.oidc.client-id``
     - The client-id of the application to identify it at your provider.
     - Y
@@ -187,4 +191,4 @@ The following options are available:
     - Tune the maximum age, in seconds, of all OIDC providers' verifier cache entries. Default is 5 minutes, equivalent to lifetime
       of many OIDC access tokens.
     - N
-    - 300
+    - 300

docker-compose-dev.yml

@@ -22,6 +22,7 @@ services:
       DATAVERSE_MAIL_SYSTEM_EMAIL: "dataverse@localhost"
       DATAVERSE_MAIL_MTA_HOST: "smtp"
       DATAVERSE_AUTH_OIDC_ENABLED: "1"
+      DATAVERSE_AUTH_OIDC_HIDDEN_JSF: "1"
       DATAVERSE_AUTH_OIDC_CLIENT_ID: test
       DATAVERSE_AUTH_OIDC_CLIENT_SECRET: 94XHrfNRwXsjqTqApRrwWmhDLDHpIYV8
       DATAVERSE_AUTH_OIDC_AUTH_SERVER_URL: http://keycloak.mydomain.com:8090/realms/test