DD-2153 Direct upload support

[janvanmansum]

Fixes DD-2153

Description of changes

Adds client-side support for Dataverse “S3 Direct Upload” by introducing a response model for upload URL negotiation and new Dataset API methods to request upload URLs and register a prestaged upload.

Notify

@DANS-KNAW/core-systems

[codecov]

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 21.76%. Comparing base (fdb3b50) to head (d0a2d8b).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...in/java/nl/knaw/dans/lib/dataverse/DatasetApi.java	0.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master      #67      +/-   ##
============================================
- Coverage     21.85%   21.76%   -0.10%     
  Complexity       71       71              
============================================
  Files            53       53              
  Lines           961      965       +4     
  Branches         59       59              
============================================
  Hits            210      210              
- Misses          743      747       +4     
  Partials          8        8              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds client-side support for Dataverse “S3 Direct Upload” by introducing a response model for upload URL negotiation and new Dataset API methods to request upload URLs and register a prestaged upload.

Changes:

• Add DirectUploadURLs model to represent Dataverse direct-upload URL responses.
• Extend DatasetApi with getUploadUrls(...) and addFile(PrestagedFile ...) to support direct upload workflows.
• Update StorageDriver model to include a message field (Dataverse compatibility).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
lib/src/main/java/nl/knaw/dans/lib/dataverse/model/dataset/DirectUploadURLs.java	New model for direct upload URL response payload.
lib/src/main/java/nl/knaw/dans/lib/dataverse/model/StorageDriver.java	Adds message field; introduces unused import and unclear comment.
lib/src/main/java/nl/knaw/dans/lib/dataverse/DatasetApi.java	Adds direct upload endpoints and prestaged-file add method (header consistency issue).
lib/src/main/java/nl/knaw/dans/lib/dataverse/AbstractApi.java	Adds unused import.

Comments suppressed due to low confidence (2)

lib/src/main/java/nl/knaw/dans/lib/dataverse/model/StorageDriver.java:25

• The comment // For older versions of Dataverse (6.7)? is speculative (question mark) and doesn’t explain what behavior/response shape is being handled. Please replace it with a concrete explanation (e.g., what endpoint returns message and in which versions) or remove the comment if it’s not needed.

    // For older versions of Dataverse (6.7)?
    private String message;

lib/src/main/java/nl/knaw/dans/lib/dataverse/DatasetApi.java:537

• getUploadUrls uses the HttpClientWrapper.get(..., parameters, outputClass) overload which does not send extraHeaders (e.g., X-Dataverse-invocationID). For consistency with other DatasetApi calls and to preserve request correlation, consider using the overload that accepts headers and pass extraHeaders.

    public DataverseHttpResponse<DirectUploadURLs> getUploadUrls(long fileSize) throws IOException, DataverseException {
        return httpClientWrapper.get(subPath("uploadurls"), params(Map.of("size", List.of(String.valueOf(fileSize)))), DirectUploadURLs.class);
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Class name DirectUploadURLs doesn’t follow the naming style used elsewhere in this repo (e.g., baseUrl, persistentUrl, DatasetApi use Url/Api rather than all-caps acronyms). Consider renaming to DirectUploadUrls (and file name/imports accordingly) to keep naming consistent.

Suggested change

	public class DirectUploadURLs {
	public class DirectUploadUrls {

[Copilot]

New response model DirectUploadURLs is added but there’s no corresponding JSON fixture + deserialization/round-trip test like the other model classes in lib/src/test/resources/model/**. Adding a test will help ensure the urls map and optional fields (complete/abort) stay compatible with Dataverse responses.

[Copilot]

addFile(PrestagedFile) posts with emptyMap() headers, which drops extraHeaders (notably X-Dataverse-invocationID) that are included in the other addFile(...) overload. This makes behavior inconsistent and may break request tracing/correlation; please pass extraHeaders here as well.

Suggested change

	return httpClientWrapper.post(subPath("add"), builder.build(), params(emptyMap()), emptyMap(), FileList.class);
	return httpClientWrapper.post(subPath("add"), builder.build(), params(emptyMap()), extraHeaders, FileList.class);

[Copilot]

lombok.Getter is imported but not used in this class. Please remove the unused import to avoid warnings or potential build failures when using strict linting/static analysis.

Suggested change

import lombok.Getter;

[Copilot]

com.fasterxml.jackson.annotation.JsonIgnore is imported but not used anywhere in this class. Please remove the unused import (or add the missing annotation if it was intended).

Suggested change

import com.fasterxml.jackson.annotation.JsonIgnore;