Merge pull request #9482 from QualitativeDataRepository/QDR/checksumalg

IQSS/9431-support checksum alg in direct uploads

Author: kcondon

doc/release-notes/9431-checksum-alg-in-direct-uploads.md

@@ -0,0 +1,4 @@
+Direct upload via the Dataverse UI will now support any algorithm configured via the :FileFixityChecksumAlgorithm setting.
+External apps using the direct upload API can now query Dataverse to discover which algorithm should be used.
+
+Sites that have been using an algorithm other than MD5 and direct upload and/or dvwebloader may want to use the /api/admin/updateHashValues call (see https://guides.dataverse.org/en/latest/installation/config.html?highlight=updatehashvalues#filefixitychecksumalgorithm) to replace any MD5 hashes on existing files.

doc/sphinx-guides/source/api/native-api.rst

@@ -3007,6 +3007,22 @@ The response is a JSON object described in the :doc:`/api/external-tools` sectio
 
   curl -H "X-Dataverse-key: $API_TOKEN" -H "Accept:application/json" "$SERVER_URL/api/files/$FILE_ID/metadata/$FILEMETADATA_ID/toolparams/$TOOL_ID
 
+.. _get-fixity-algorithm:
+
+Get Fixity Algorithm
+~~~~~~~~~~~~~~~~~~~~~~
+
+This API call can be used to discover the configured fixity/checksum algorithm being used by a Dataverse installation (as configured by - :ref:`:FileFixityChecksumAlgorithm`).
+Currently, the possible values are MD5, SHA-1, SHA-256, and SHA-512.
+This algorithm will be used when the Dataverse software manages a file upload and should be used by external clients uploading files to a Dataverse instance. (Existing files may or may not have checksums with this algorithm.) 
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+
+  curl "$SERVER_URL/api/files/fixityAlgorithm
+
+
 Users Token Management
 ----------------------
 
@@ -4395,6 +4411,26 @@ It will report the specific files that have failed the validation. For example::
   
 These are only available to super users.
 
+.. _UpdateChecksums:
+
+Update Checksums To Use New Algorithm
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fixity algorithm used on existing files can be changed by a superuser using this API call. An optional query parameter (num) can be used to limit the number of updates attempted (i.e. to do processing in batches).
+The API call will only update the algorithm and checksum for a file if the existing checksum can be validated against the file.
+Statistics concerning the updates are returned in the response to the API call with details in the log.
+The primary use for this API call is to update existing files after the algorithm used when uploading new files is changes - see - :ref:`:FileFixityChecksumAlgorithm`.
+Allowed values are MD5, SHA-1, SHA-256, and SHA-512
+
+.. code-block:: bash
+
+  export ALG=SHA-256
+  export BATCHSIZE=1
+
+  curl http://localhost:8080/api/admin/updateHashValues/$ALG
+  curl http://localhost:8080/api/admin/updateHashValues/$ALG?num=$BATCHSIZE
+
+
 .. _dataset-validation-api:
 
 Dataset Validation

doc/sphinx-guides/source/installation/config.rst

@@ -3112,6 +3112,8 @@ This curl command...
 
 See also :doc:`oauth2`.
 
+.. _:FileFixityChecksumAlgorithm:
+
 :FileFixityChecksumAlgorithm
 ++++++++++++++++++++++++++++
 
@@ -3121,12 +3123,9 @@ The default checksum algorithm used is MD5 and should be sufficient for establis
 
 ``curl -X PUT -d 'SHA-512' http://localhost:8080/api/admin/settings/:FileFixityChecksumAlgorithm``
 
-The fixity algorithm used on existing files can be changed by a superuser using the API. An optional query parameter (num) can be used to limit the number of updates attempted.
-The API call will only update the algorithm and checksum for a file if the existing checksum can be validated against the file.
-Statistics concerning the updates are returned in the response to the API call with details in the log.
+To update the algorithm used for existing files, see :ref:`UpdateChecksums`
 
-``curl http://localhost:8080/api/admin/updateHashValues/{alg}``
-``curl http://localhost:8080/api/admin/updateHashValues/{alg}?num=1``
+The fixity checksum algorithm in use can be discovered via API. See :ref:`get-fixity-algorithm` in the API Guide.
 
 .. _:PVMinLength:
 

src/main/java/edu/harvard/iq/dataverse/api/Files.java

@@ -812,4 +812,10 @@ public Response getExternalToolFMParams(@Context ContainerRequestContext crc, @P
         eth = new ExternalToolHandler(externalTool, target.getDataFile(), apiToken, target, locale);
         return ok(eth.createPostBody(eth.getParams(JsonUtil.getJsonObject(externalTool.getToolParameters()))));
     }
+    
+    @GET
+    @Path("fixityAlgorithm")
+    public Response getFixityAlgorithm() {
+        return ok(systemConfig.getFileFixityChecksumAlgorithm().toString());
+    }
 }

src/main/webapp/editFilesFragment.xhtml

@@ -11,9 +11,20 @@
     xmlns:o="http://omnifaces.org/ui"
     xmlns:iqbs="http://xmlns.jcp.org/jsf/composite/iqbs">
 
-    <script src="#{resource['js/fileupload.js']}?version=#{systemConfig.getVersion()}"></script>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.2/components/core.js"></script>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.2/components/md5.js"></script>
+
+    <ui:param name="useDirectUpload" value="#{systemConfig.directUploadEnabled(dataset)}"/>
+    <ui:param name="checksumAlgName" value="#{systemConfig.getFileFixityChecksumAlgorithm().toString()}"/>
+    
+    <h:outputScript name='js/fileupload.js?version=#{systemConfig.getVersion()}' />
+    <ui:fragment rendered = '#{useDirectUpload}'>
+    <h:outputScript name='js/crypto-js/4.0.0/core.js'  />
+    <h:outputScript name='js/crypto-js/4.0.0/x64-core.js' rendered='#{(checksumAlgName eq "SHA-512")}'/>
+    <h:outputScript name='js/crypto-js/4.0.0/md5.js' rendered='#{true or checksumAlgName eq "MD5"}'/>
+    <h:outputScript name='js/crypto-js/4.0.0/sha1.js'  rendered='#{checksumAlgName eq "SHA-1"}'/>
+    <h:outputScript name='js/crypto-js/4.0.0/sha256.js'  rendered='#{checksumAlgName eq "SHA-256"}'/>
+    <h:outputScript name='js/crypto-js/4.0.0/sha512.js' rendered='#{checksumAlgName eq "SHA-512"}'/>
+    </ui:fragment>
+
     <!-- Static Tab Layout -->
     <div data-widget="content" class="ui-tabs ui-widget ui-widget-content ui-corner-all ui-hidden-container ui-tabs-top" id="datasetForm:tabView">
         <ul role="tablist" class="ui-tabs-nav ui-helper-reset ui-helper-clearfix ui-widget-header ui-corner-all">
@@ -109,7 +120,7 @@
 
                         $(document).ready(function () {
                             uploadWidgetDropMsg();
-                            setupDirectUpload(#{systemConfig.directUploadEnabled(EditDatafilesPage.dataset)});
+                             #{useDirectUpload ? 'setupDirectUpload(true);':''}
                         });
                         //]]>
                     </script>
@@ -584,7 +595,7 @@
         <p class="text-warning"><span class="glyphicon glyphicon-warning-sign"/> #{EditDatafilesPage.warningMessageForFileTypeDifferentPopUp}</p>
         <div class="button-block">
             <p:commandButton styleClass="btn btn-default" value="#{bundle['file.delete']}" onclick="PF('fileTypeDifferentPopup').hide()" oncomplete="uploadWidgetDropMsg();
-                            setupDirectUpload(#{systemConfig.directUploadEnabled(EditDatafilesPage.dataset)});"
+																	 #{useDirectUpload ? 'setupDirectUpload(true);': ''}"
                              action="#{EditDatafilesPage.deleteFiles()}"
                              update=":#{p:resolveClientId('datasetForm:filesTable', view)},:messagePanel,:#{p:resolveClientId('datasetForm:fileUpload', view)},uploadMessage"/>
             <button class="btn btn-default" onclick="PF('fileTypeDifferentPopup').hide();" type="button">

src/main/webapp/provenance-popups-fragment.xhtml

@@ -7,7 +7,10 @@
                 xmlns:o="http://omnifaces.org/ui"
                 xmlns:jsf="http://xmlns.jcp.org/jsf"
                 xmlns:iqbs="http://xmlns.jcp.org/jsf/composite/iqbs">
-    <script src="#{resource['js/fileupload.js']}"></script>
+
+
+    <h:outputScript name='js/fileupload.js?version=#{systemConfig.getVersion()}' />
+
     <p:dialog id="editProvenancePopup" styleClass="smallPopUp" header="#{bundle['file.editProvenanceDialog']}" widgetVar="editProvenancePopup" modal="true" >
         <p:fragment id="assignMessages">
             <div class="popupMessagePanel">