-
Notifications
You must be signed in to change notification settings - Fork 541
Iqss/6489 - Direct uploads to S3 using presigned URLs #6490
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 115 commits
Commits
Show all changes
164 commits
Select commit
Hold shift + click to select a range
34a2453
typos, fix for empty string
qqmyers e353ccc
use String return value
qqmyers 7a0f302
filter custom roles
qqmyers 2910168
Don't open input stream when not needed.
qqmyers 7ed30fb
simplify
qqmyers 08a0e7e
add @Override
qqmyers e5fed4a
override getReadChannel as well
qqmyers 7070e26
Merge tag 'v4.10' into v4.10-tdl
qqmyers 6ea2524
TDL image and release name changes
qqmyers d7571ac
text changes
qqmyers 7998929
TDRDV-46
qqmyers 247dd32
TDRDV-46
qqmyers 2ddfd84
Merge tag 'v4.16' into v4.16-tdl
qqmyers 1f8b528
TDL custom strings
qqmyers 19ebd02
new fav icons for TDL
qqmyers 4e1327a
more TDL icons
qqmyers 51739f6
Merge pull request #6254 from IQSS/develop
kcondon c64481f
Merge tag 'v4.17' into v4.17-tdl
qqmyers 18c7f31
Merge tag 'v4.17' into v4.17-tdl
qqmyers 07e6a0a
#6137 make DataCite API URL for MDC configurable
feac172
doc/install updates for #6137
3404453
cut/paste issue
47f2ffe
two commas
qqmyers a6e657c
add note in admin guide
qqmyers 59519df
consistent naming for jvm option
qqmyers 4c1a2e2
add logging of MDC citation get URL and tweak docs #6137
pdurbin e1d953c
separate MDC logging and display
ad784d7
Doc updates for :DisplayMDCMetrics
29c9bac
use SettingsWrapper per comments
bdaa0cc
closed ) in docs
djbrooke 110cd1c
remove blank line per comment
qqmyers c7504b1
Merge branch 'develop' of
qqmyers f8dada8
Revert "add logging of MDC citation get URL and tweak docs #6137"
qqmyers 5fff437
Revert "consistent naming for jvm option"
qqmyers 77b99f3
Revert "add note in admin guide"
qqmyers 750ed11
Revert "two commas"
qqmyers 88a90fb
Revert "cut/paste issue"
qqmyers 92e5160
Revert "doc/install updates for #6137"
qqmyers ebff193
Revert "#6137 make DataCite API URL for MDC configurable"
qqmyers d774717
version update to add -dev
qqmyers a6600fe
initial presigned upload uri api
qqmyers ef9190f
adding transfer of file name, mimetype and md5 hash
qqmyers bc96ef7
update plugin version
qqmyers 95aeea0
using profiles to get correct compile flags
qqmyers 6e0c28e
Merge branch 'v4.17-tdl' of
qqmyers 99c8b25
Merge branch 'develop' of https://github.com/TexasDigitalLibrary/data…
qqmyers bf504ab
Merge branch 'v4.17-tdl' into develop
qqmyers bddccd7
add nulls for missing params recently added
qqmyers b465234
fix logic, adjust indent
qqmyers 4095466
bugfix, pickup bucket from properties
qqmyers 118aa71
Merge pull request #6369 from IQSS/develop
kcondon a91d370
Merge pull request #6390 from IQSS/develop
kcondon 024e4c8
Merge tag 'v4.18.1' into v4.18.1-tdl
qqmyers 606326f
Merge remote-tracking branch 'IQSS/develop' into IQSS/6138
qqmyers 84297e5
use newer poi
qqmyers 0bd7e22
Merge pull request #6394 from QualitativeDataRepository/IQSS/6389
kcondon 76242e9
Merge pull request #6414 from QualitativeDataRepository/IQSS/6411
kcondon 1ccec27
Handle expired or multiple tokens per user
qqmyers 0b8b2a8
missing keys
qqmyers 08f6849
multistore implementation (built on direct upload)
qqmyers 2d104f6
enforce permissions on upload url
qqmyers 83dcec4
enforce permissions on upload url
qqmyers 5eb7fa8
missing keys
qqmyers c393181
error msg fix
qqmyers 6c2e220
check if store allows direct upload
qqmyers 08ad964
capture file list in js
qqmyers 43f78ce
toggle auto upload based on directupload property
qqmyers 14e43bd
implement proxying
qqmyers 9c01f28
implement proxy
qqmyers 0fe9918
further updates to direct upload files
qqmyers 518e2d2
add driver to upload url response
qqmyers 076c75a
fix proxy url replacement
qqmyers 2f7102d
single file, no hash upload via GUI minimally working
qqmyers d8de35f
send ID without dataset path
qqmyers c1db7a5
add md5
qqmyers 707709f
bug: switch to calling uploadFinished at oncomplete
qqmyers 25c1739
fix exist test for direct access
qqmyers dd1cca0
debug constraint violations
qqmyers a02980b
typo, clear file list after upload complete
qqmyers cafcfee
use new method
qqmyers 8859e1d
catch empty contentType not just null
qqmyers 25da677
List any constraint violations
qqmyers e07b2df
try adding multifile support, fix repeat uploads
qqmyers 407ed1c
just pop for multifile
qqmyers 36e409f
control store via dataverse.storagedriver param
qqmyers 0911571
fix temporary(which is also final) file delete for direct upload
qqmyers 150d52e
progress bar styling
qqmyers 92dbb2e
remove drag/drop message once file(s) are selected in direct upload
qqmyers 216a7de
bugfix apply styles to progress element to make it appear
qqmyers 6f56da2
bugfix for failing s3 delete
qqmyers 1895611
bugfix - temporarily set datafile owner for location calculation
qqmyers ca6b4d4
bugfix: use direct storage for delete
qqmyers db1d985
Merge branch 'multistore' into develop
qqmyers ff1488e
Merge remote-tracking branch 'IQSS/develop' into develop
qqmyers a7829c5
Merge branch 'IQSS/6485' into develop
qqmyers e68c23f
Merge remote-tracking branch 'IQSS/develop' into develop
qqmyers 5d4ce21
cleanup
qqmyers d72bf84
remove tdl customizations
qqmyers b9eb0c3
docs
qqmyers f6af668
pom version
qqmyers 86d7ee6
add tabular and fits metadata processing to direct uploads
qqmyers b373a72
docs update
qqmyers 93de5ce
fix tests
qqmyers 0cda1d0
test fixes
qqmyers 14c7134
define 'file' store type for default test
qqmyers d41b27b
broken logic for normal upload
qqmyers 588dec8
fix create mode
qqmyers 06f305d
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers 9812ebc
Merge branch 'IQSS/6485' into IQSS/6489
qqmyers 0125133
Merge branch 'IQSS/6485' into IQSS/6489
qqmyers 3e3f82b
Merge branch 'IQSS/6485' into IQSS/6489
qqmyers f5d63da
Merge branch 'IQSS/6485' into IQSS/6489
qqmyers 60c7789
Merge branch 'IQSS/6485' into IQSS/6489
qqmyers d4f84cb
Merge branch 'IQSS/6485' into IQSS/6489
qqmyers 4b0528b
fix merge issue
qqmyers 019862b
direct upload release notes
qqmyers ddfeb3e
Disallow 'placeholder' storageIO ~clones
qqmyers de4c5d0
bug fix for exporting
qqmyers 2e5009c
typo
djbrooke 93d325f
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers e92cf2a
Merge remote-tracking branch 'origin/IQSS/6489' into IQSS/6489
qqmyers e6aee06
get the effective storage driver
qqmyers cc3a2fd
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers 447bd27
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers 91f4275
don't wrap file
qqmyers 0052eef
give default selection a value
qqmyers 48d176e
typo
qqmyers 94abdda
use non-null indicator for default
qqmyers 17936da
using S3 tag for temp files
qqmyers 34c8ee2
ignore tag issues on Minio
qqmyers 5c721ff
new progress color
qqmyers 2a78a68
remove obsolete storage default string
qqmyers 66a48ee
remove unused methods and tests of them
qqmyers 560c973
make max file size limit per store
qqmyers 84f57a3
get max file size after dataset is initialized
qqmyers 500094f
remove systemconfig default methods
qqmyers e7d8d60
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers 9113612
restore test fix
qqmyers d3c2539
document per store file upload size limits
qqmyers abbebd8
doc of new features
qqmyers 5cd8744
avoid resetting mimetype for normal upload
qqmyers d600593
Update 6489-release-notes.md
kcondon 0f4b5f0
Update 6489-release-notes.md
kcondon e235d8c
initialize storageDriver to null
qqmyers a8c8e00
set alreadyExists(DVObject) false to allow direct upload
qqmyers a5dbdf2
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers aca7fa5
Merge remote-tracking branch 'origin/IQSS/6489' into IQSS/6489
qqmyers 5584adc
remove debug logging
qqmyers 00c62d2
retry to give S3 time to 'index' new objects
qqmyers 2d28a71
cors configuration example
landreev ab4f6c9
fix duplicate id issue with direct upload
qqmyers 187797f
Merge remote-tracking branch 'origin/IQSS/6489' into IQSS/6489
qqmyers 55abdab
Merge remote-tracking branch 'IQSS/develop' into IQSS/6489
qqmyers b1221c1
update script with version changes (per discussion)
qqmyers be03ead
limit parallel requests (to 4 at the moment)
qqmyers c2cf275
handle progress bars, avoid calling AllUploadsFinished more than once
qqmyers df5ec27
Add support for drag and drop
qqmyers f358295
cut/paste error and adding an error handler
qqmyers 3226d43
more error handling
qqmyers 9668c5f
increasing retries and adding logging for testing
qqmyers 21f7361
add error handling and pre/post S3 logging
qqmyers a15b83d
tighten up on async issues
qqmyers 3a9e811
remove debug log message
qqmyers 29ac854
only warn if a retry occurred
qqmyers File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Dataverse 4.19 | ||
|
|
||
| Update Geospatial Metadata Block | ||
|
|
||
| This update adds commas separating the values entered into Geographic Coverage. | ||
|
|
||
| - `wget https://github.com/IQSS/dataverse/releases/download/v4.19/geospatial.tsv` | ||
| - `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @geospatial.tsv -H "Content-type: text/tab-separated-values"` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| ## Notes for Dataverse Installation Administrators | ||
|
|
||
| ### Run ReExportall | ||
|
|
||
| We made changes to the JSON Export in this release (#6246). If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process. We've included this in the detailed instructions below. | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
| xxx. (Optional) Run ReExportall to update JSON Exports | ||
|
|
||
| <http://guides.dataverse.org/en/4.19/admin/metadataexport.html?highlight=export#batch-exports-through-the-api> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo. "uplod". Also, do we consider unzipping zip files to be part of ingest? Is processing of FITS files part of ingest? (I thought special handling of FITS files is broken right now: #5919.) My sense is that when we say "ingest" we are usually only talking about tabular files, not zip files or FITS files. Please see http://guides.dataverse.org/en/4.18.1/user/tabulardataingest/ingestprocess.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you want to call the rest of it? Processing on upload?
For FITS, this branch should now be just-as-broken as the other. I just assured that the same processing is triggered when there is no temp file - handing the code a stream to the final file instead.