Merge pull request #3967 from IQSS/3776-public-only

3776 public only

Author: kcondon

doc/sphinx-guides/source/api/native-api.rst

@@ -212,7 +212,7 @@ Add a file to an existing Dataset. Description and tags are optional::
 
 A more detailed "add" example using curl::
 
-    curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' -F 'jsonData={"description":"My description.","categories":["Data"]}' "https://example.dataverse.edu/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"
+    curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' -F 'jsonData={"description":"My description.","categories":["Data"], "restrict":"true"}' "https://example.dataverse.edu/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"
 
 Example python code to add a file. This may be run by changing these parameters in the sample code:
 
@@ -303,6 +303,16 @@ Files
 
 .. note:: Please note that files can be added via the native API but the operation is performed on the parent object, which is a dataset. Please see the "Datasets" endpoint above for more information.
 
+Restrict or unrestrict an existing file where ``id`` is the database id of the file to restrict::
+    
+    PUT http://$SERVER/api/files/{id}/restrict
+
+Note that some Dataverse installations do not allow the ability to restrict files.
+
+A more detailed "restrict" example using curl::
+
+    curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -d true http://$SERVER/api/files/{id}/restrict
+
 Replace an existing file where ``id`` is the database id of the file to replace. Note that metadata such as description and tags are not carried over from the file being replaced::
 
     POST http://$SERVER/api/files/{id}/replace?key=$apiKey

doc/sphinx-guides/source/installation/config.rst

@@ -222,6 +222,10 @@ Then run the create command:
 
 ``./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.files.storage-driver-id=swift"``
 
+You also have the option to set a custom container name separator. It is initialized to ``_``, but you can change it by running the create command:
+
+``./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.files.swift-folder-path-separator=-"``
+
 .. _Branding Your Installation:
 
 Branding Your Installation
@@ -891,6 +895,15 @@ Set the name of the cloud environment you've integrated with your Dataverse inst
 
 ``curl -X PUT -d 'Massachusetts Open Cloud (MOC)' http://localhost:8080/api/admin/settings/:CloudEnvironmentName``
 
+:PublicInstall
++++++++++++++++++++++
+
+Setting an installation to public will remove the ability to restrict data files or datasets. This functionality of Dataverse will be disabled from your installation.
+
+This is useful for specific cases where an installation's files are stored in public access. Because files stored this way do not obey Dataverse's file restrictions, users would still be able to access the files even when they're restricted. In these cases it's best to use :PublicInstall to disable the feature altogether.
+
+``curl -X PUT -d true http://localhost:8080/api/admin/settings/:PublicInstall``
+
 :DataCaptureModuleUrl
 +++++++++++++++++++++
 

doc/sphinx-guides/source/user/dataset-management.rst

@@ -49,11 +49,13 @@ File Handling + Uploading
 
 To upload new files to a dataset, click the "Edit" button at the top of the dataset page and from the dropdown list select "Files (Upload)" or click the "Upload Files" button above the files table in the Files tab. From either option you will be brought to the Upload Files page for that dataset.
 
-Once you have uploaded files, you will be able to edit file metadata, restrict access to files, and/or add tags. Click "Save Changes" to complete the upload. If you uploaded a file by mistake, you can delete it before saving by clicking the checkbox to select the file, and then clicking the "Delete" button above the Files Table.
+Once you have uploaded files, you will be able to edit file metadata, restrict access to files [#f1]_ , and/or add tags. Click "Save Changes" to complete the upload. If you uploaded a file by mistake, you can delete it before saving by clicking the checkbox to select the file, and then clicking the "Delete" button above the Files Table.
 
 File upload limit size varies based on Dataverse installation. The file upload size limit can be found in the text above where files are uploaded in the application. If you have further questions, contact support for that installation by clicking on the Support link at the top of the application.
 
-The file types listed below are supported by additional functionality, which can include downloading in different formats, subsets, file-level metadata preservation, file-level data citation; and exploration through data visualization and analysis. 
+The file types listed in the following sections are supported by additional functionality, which can include downloading in different formats, subsets, file-level metadata preservation, file-level data citation; and exploration through data visualization and analysis. 
+
+.. [#f1] Some Dataverse installations do not allow this feature.
 
 Tabular Data Files
 ------------------
@@ -156,7 +158,7 @@ Go to the dataset you would like to edit, where you will see the listing of file
 
 You will not have to leave the dataset page to complete these action, except for editing file metadata, which will bring you to the Edit Files page. There you will have to click the "Save Changes" button to apply your edits and return to the dataset page.
 
-If you restrict files, you will also prompted with a popup asking you to fill out the Terms of Access for the files. If Terms of Access already exist, you will be asked to confirm them.
+If you restrict files, you will also prompted with a popup asking you to fill out the Terms of Access for the files. If Terms of Access already exist, you will be asked to confirm them. Note that some Dataverse installations do not allow for file restrictions.
 
 File Tags
 ---------
@@ -202,6 +204,8 @@ Restricted Files + Terms of Access
 
 If you restrict any files in your dataset, you will be prompted by a pop-up to enter Terms of Access for the data. This can also be edited in the Terms tab or selecting Terms in the "Edit" dropdown button in the dataset. You may also allow users to request access for your restricted files by enabling "Request Access". To add more information about the Terms of Access, click on "Additional Information \[+]".
 
+**Note:** Some Dataverse installations do not allow for file restriction.
+
 Guestbook
 ---------
 
@@ -224,7 +228,7 @@ The panel below that is "Roles", where you can find all the roles set up in your
 File-Level
 ----------
 
-If you have restricted specific files the file-level permissions is where you will need to go to grant users/groups access to
+If you have restricted specific files, the file-level permissions is where you will need to go to grant users/groups access to
 specific restricted files. Dataset file permissions are located under Permissions in the Edit button on a dataset page. 
 The file permissions page has two sections: Users/Groups and Files.
 

src/main/java/Bundle.properties

@@ -1211,6 +1211,7 @@ dataset.message.deleteFailure=This dataset draft could not be deleted.
 dataset.message.deaccessionFailure=This dataset could not be deaccessioned.
 dataset.message.createFailure=The dataset could not be created.
 dataset.message.termsFailure=The dataset terms could not be updated.
+dataset.message.publicInstall=File Access - Files are stored on a publicly accessible storage server.
 dataset.metadata.publicationDate=Publication Date
 dataset.metadata.publicationDate.tip=The publication date of a dataset.
 dataset.metadata.persistentId=Dataset Persistent ID

src/main/java/edu/harvard/iq/dataverse/DataFile.java

@@ -534,6 +534,7 @@ public boolean isRestricted() {
         return restricted;
     }
 
+    
     public void setRestricted(boolean restricted) {
         this.restricted = restricted;
     }

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -5,7 +5,9 @@
 import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinUserServiceBean;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser;
+import edu.harvard.iq.dataverse.dataaccess.DataFileIO;
 import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter;
+import edu.harvard.iq.dataverse.dataaccess.SwiftAccessIO;
 import edu.harvard.iq.dataverse.dataset.DatasetThumbnail;
 import edu.harvard.iq.dataverse.datavariable.VariableServiceBean;
 import edu.harvard.iq.dataverse.engine.command.Command;
@@ -76,6 +78,7 @@
 import java.util.logging.Level;
 import edu.harvard.iq.dataverse.datasetutility.TwoRavensHelper;
 import edu.harvard.iq.dataverse.datasetutility.WorldMapPermissionHelper;
+import edu.harvard.iq.dataverse.engine.command.impl.RestrictFileCommand;
 
 import javax.faces.event.AjaxBehaviorEvent;
 
@@ -389,21 +392,41 @@ public String getDataverseSiteUrl() {
     public void setDataverseSiteUrl(String dataverseSiteUrl) {
         this.dataverseSiteUrl = dataverseSiteUrl;
     }
-    //TODO: 
-    //Consolidate this & FilePage in static function in the SwiftAccessIO
 
-    public String getSwiftContainerName(){
-        String swiftContainerName = null;
-        String swiftFolderPathSeparator = "-";
-        if (persistentId != null) {
-            dataset = datasetService.findByGlobalId(persistentId); 
-            String authorityNoSlashes = dataset.getAuthority().replace(dataset.getDoiSeparator(), swiftFolderPathSeparator);
-            swiftContainerName = dataset.getProtocol() + swiftFolderPathSeparator + authorityNoSlashes.replace(".", swiftFolderPathSeparator)
-                + swiftFolderPathSeparator + dataset.getIdentifier();
-            logger.fine("Swift container name: " + swiftContainerName);
+    public DataFile getInitialDataFile(){
+        Long datasetVersion = workingVersion.getId();
+        if (datasetVersion != null) {
+                int unlimited = 0;
+                int maxResults = unlimited;
+            List<FileMetadata> metadatas = datafileService.findFileMetadataByDatasetVersionId(datasetVersion, maxResults, fileSortField, fileSortOrder);        
+                logger.fine("metadatas " + metadatas);
+            if (metadatas != null && metadatas.size() > 0) {
+                return metadatas.get(0).getDataFile();
+            }
         }
+        return null;
+    }
+    
+    public String getSwiftContainerName(){
+
+        String swiftContainerName;
+        try {
+            DataFileIO dataFileIO = getInitialDataFile().getDataFileIO();
+            try {
+                SwiftAccessIO swiftIO = (SwiftAccessIO)dataFileIO;
+                swiftIO.open();
+                swiftContainerName = swiftIO.getSwiftContainerName();
+                logger.info("Swift container name: " + swiftContainerName);
+                return swiftContainerName;
 
-        return swiftContainerName;
+            } catch (Exception e) {
+                e.printStackTrace();
+            }
+        }
+        catch (Exception e){
+            e.printStackTrace();
+        }
+        return "";
     }
 
     public void setSwiftContainerName(String name){
@@ -415,20 +438,14 @@ public void setSwiftContainerName(String name){
     //SF 
     public Boolean isSwiftStorage(){
         Boolean swiftBool = false;
-        //dataset = datasetService.findByGlobalId(persistentId);
-        Long datasetVersion = workingVersion.getId();
-        if (datasetVersion != null) {
-                int unlimited = 0;
-                int maxResults = unlimited;
-            List<FileMetadata> metadatas = datafileService.findFileMetadataByDatasetVersionId(datasetVersion, maxResults, fileSortField, fileSortOrder);        
-                logger.fine("metadatas " + metadatas);
-            if (metadatas != null && metadatas.size() > 0) {
-                if ("swift".equals(System.getProperty("dataverse.files.storage-driver-id")) 
-                    && metadatas.get(0).getDataFile().getStorageIdentifier().startsWith("swift://")) {
-                    swiftBool = true;
-                }
+        //containers without datafiles will not be stored in swift storage, so no compute
+        if (getInitialDataFile() != null){
+            if ("swift".equals(System.getProperty("dataverse.files.storage-driver-id")) 
+                && getInitialDataFile().getStorageIdentifier().startsWith("swift://")) {
+                swiftBool = true;
             }
         }
+        
         return swiftBool;
     }
 
@@ -1309,6 +1326,10 @@ private String init(boolean initFull) {
                 workingVersion = dataset.getCreateVersion();
                 updateDatasetFieldInputLevels();
             }
+            
+            if (settingsService.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)){
+                JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("dataset.message.publicInstall"));
+            }
 
             resetVersionUI();
 
@@ -2107,7 +2128,7 @@ public String testSelectedFilesForRestrict(){
     }
 
 
-    public String restrictSelectedFiles(boolean restricted){
+    public String restrictSelectedFiles(boolean restricted) throws CommandException{
 
         RequestContext requestContext = RequestContext.getCurrentInstance();
         if (selectedFiles.isEmpty()) {
@@ -2147,8 +2168,8 @@ public String restrictSelectedFiles(boolean restricted){
         return  returnToDraftVersion();
     }
 
-    public void restrictFiles(boolean restricted) {
-
+    public void restrictFiles(boolean restricted) throws CommandException {
+   
         //if (previouslyRestrictedFiles == null) {
         // we don't need to buther with this "previously restricted" business 
         // when in Create mode... because all the files are new, so none could 
@@ -2165,7 +2186,8 @@ public void restrictFiles(boolean restricted) {
                     previouslyRestrictedFiles.add(fmd);
                 }
             }
-
+            
+            Command cmd;
             String fileNames = null;
             for (FileMetadata fmw : workingVersion.getFileMetadatas()) {
                 for (FileMetadata fmd : this.getSelectedFiles()) {
@@ -2182,13 +2204,17 @@ public void restrictFiles(boolean restricted) {
                         }
                     }
                     if (fmd.getDataFile().equals(fmw.getDataFile())) {
-                        fmw.setRestricted(restricted);
-                        if (workingVersion.isDraft() && !fmw.getDataFile().isReleased()) {
-                            // We do not really need to check that the working version is 
-                            // a draft here - it must be a draft, if we've gotten this
-                            // far. But just in case. -- L.A. 4.2.1
-                            fmw.getDataFile().setRestricted(restricted);
-                        }
+                        cmd = new RestrictFileCommand(fmw.getDataFile(), dvRequestService.getDataverseRequest(), restricted);
+                        commandEngine.submit(cmd);
+                        
+                        
+//                        fmw.setRestricted(restricted);
+//                        if (workingVersion.isDraft() && !fmw.getDataFile().isReleased()) {
+//                            // We do not really need to check that the working version is 
+//                            // a draft here - it must be a draft, if we've gotten this
+//                            // far. But just in case. -- L.A. 4.2.1
+//                            fmw.getDataFile().setRestricted(restricted);
+//                        }
                     }
                 }
             }

src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java

@@ -10,7 +10,9 @@
 import edu.harvard.iq.dataverse.dataset.DatasetThumbnail;
 import edu.harvard.iq.dataverse.engine.command.Command;
 import edu.harvard.iq.dataverse.engine.command.exception.CommandException;
+import edu.harvard.iq.dataverse.engine.command.exception.CommandExecutionException;
 import edu.harvard.iq.dataverse.engine.command.impl.DeleteDataFileCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.RestrictFileCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UpdateDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UpdateDatasetThumbnailCommand;
 import edu.harvard.iq.dataverse.ingest.IngestRequest;
@@ -557,10 +559,18 @@ public String init() {
         if (mode == FileEditMode.UPLOAD) {
             JH.addMessage(FacesMessage.SEVERITY_INFO, getBundleString("dataset.message.uploadFiles"));
         }
+        
+        if (settingsService.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)){
+            JH.addMessage(FacesMessage.SEVERITY_WARN, getBundleString("dataset.message.publicInstall"));
+        }   
+        
         return null;
+        
     }
 
 
+    
+    
     private void msg(String s){
         System.out.println(s);
     }
@@ -697,10 +707,11 @@ public boolean isShowAccessPopup() {
     }
 
     public void setShowAccessPopup(boolean showAccessPopup) {} // dummy set method
-    
-    public void restrictFiles(boolean restricted) {
+     
+    public void restrictFiles(boolean restricted) throws CommandException{
+
         // since we are restricted files, first set the previously restricted file list, so we can compare for
-        // determinin whether to show the access popup
+        // determining whether to show the access popup
         previouslyRestrictedFiles = new ArrayList();
         for (FileMetadata fmd : workingVersion.getFileMetadatas()) {
             if (fmd.isRestricted()) {
@@ -720,13 +731,17 @@ public void restrictFiles(boolean restricted) {
                     fileNames = fileNames.concat(", " + fmd.getLabel());
                 }
             }
-            fmd.setRestricted(restricted);
-            if (workingVersion.isDraft() && !fmd.getDataFile().isReleased()) {
-                // We do not really need to check that the working version is 
-                // a draft here - it must be a draft, if we've gotten this
-                // far. But just in case. -- L.A. 4.2.1
-                fmd.getDataFile().setRestricted(restricted);
-            }
+            //fmd.setRestricted(restricted);
+            Command cmd;
+            cmd = new RestrictFileCommand(fmd.getDataFile(), dvRequestService.getDataverseRequest(), restricted);
+            commandEngine.submit(cmd);
+                       
+//            if (workingVersion.isDraft() && !fmd.getDataFile().isReleased()) {
+//                // We do not really need to check that the working version is 
+//                // a draft here - it must be a draft, if we've gotten this
+//                // far. But just in case. -- L.A. 4.2.1
+//                  fmd.getDataFile().setRestricted(restricted);              
+//            }
         }
         if (fileNames != null) {
             String successMessage = getBundleString("file.restricted.success");
@@ -1165,7 +1180,7 @@ public String save() {
             StringBuilder saveError = new StringBuilder();
 
             for (FileMetadata fileMetadata : fileMetadatas) {
-
+          
                 if (fileMetadata.getDataFile().getCreateDate() == null) {
                     fileMetadata.getDataFile().setCreateDate(updateTime);
                     fileMetadata.getDataFile().setCreator((AuthenticatedUser) session.getUser());