[BUG] repository-s3: S3 Outposts PutObject fails with 501 after #20345 routing fix due to CRC64NVME checksum header

[DeleOladokunTR]

Describe the bug

PR #20345 fixed S3 Outposts endpoint routing so the repository-s3 plugin can now correctly reach the Outposts S3 data plane. However, actual snapshot writes still fail with HTTP 501 "A header you provided implies functionality that is not implemented."

Root cause: AWS SDK Java v2 bundled in repository-s3 3.5.0 defaults requestChecksumCalculation to supported, which automatically appends a x-amz-checksum-crc64nvme header to all PutObject requests. S3 Outposts firmware does not implement CRC64NVME and rejects these requests with 501.

Workarounds attempted:

• Setting AWS_REQUEST_CHECKSUM_CALCULATION=when_required (env var) — no effect
• Setting -Daws.requestChecksumCalculation=when_required (JVM property) — blocked by OpenSearch security manager

Result: S3 Outposts is still not usable as a snapshot repository in OpenSearch 3.5.0 despite PR #20345 being merged.

Related component

No response

To Reproduce

1. Deploy OpenSearch 3.5.0 with repository-s3 plugin on a Kubernetes cluster running on an AWS Outpost rack.

2. Register a snapshot repository using an S3 Outposts access point ARN:

PUT /_snapshot/outposts-repo?verify=false
{
"type": "s3",
"settings": {
"bucket": "arn:aws:s3-outposts:us-east-1::outpost//accesspoint/",
"region": "us-east-1",
"base_path": "snapshots"
}
}

→ Returns {"acknowledged":true} (routing fix from #20345 works)

3. Trigger any snapshot:

PUT /_snapshot/outposts-repo/test-snap?wait_for_completion=true
{
"ignore_unavailable": true,
"include_global_state": false
}

→ Fails with 501 (see error message in Additional Details below)

Expected behavior

Snapshot writes to S3 Outposts access point ARNs should succeed. The SDK should either default to a checksum algorithm supported by S3 Outposts (e.g. CRC32), or the repository-s3 plugin should expose a setting such as request_checksum_calculation to let operators override the SDK default — for example:

s3.client.default.request_checksum_calculation: when_required

Additional Details

Plugin version: repository-s3 3.5.0 (bundles AWS SDK Java v2.32.29)

Exact error message:
Unable to upload object [snapshots/meta-.dat] using a single upload
A header you provided implies functionality that is not implemented
(Service: S3, Status Code: 501, Request ID: ...)

Environment:

• OS: Bottlerocket OS 1.55.0 (aws-k8s-1.31)
• OpenSearch version: 3.5.0
• Deployment: Kubernetes (EKS on AWS Outpost rack)
• Node instances: r5.xlarge / c5.xlarge (Nitro-based)
• S3 target: S3 Outposts access point ARN

Related issues: #19897, #19900
Related PR: #20345 (introduced routing fix, but write path remains broken)

Suggested fix: Expose request_checksum_calculation as a repository-s3 client setting, defaulting to when_required for Outposts-style endpoints, or document a supported workaround.

[github-actions]

@DeleOladokunTR Please reply to this comment with the relevant component to ensure it gets properly triaged:

• Build
• Clients
• Cluster Manager
• Extensions
• Indexing:Performance
• Indexing:Replication
• Indexing
• Libraries
• Other
• Plugins
• Search:Aggregations
• Search:Performance
• Search:Query Capabilities
• Search:Query Insights
• Search:Relevance
• Search:Remote Search
• Search:Resiliency
• Search:Searchable Snapshots
• Search
• Storage:Durability
• Storage:Performance
• Storage:Remote
• Storage:Snapshots
• Storage

Reply with the component name, and we'll label this issue appropriately.

[DeleOladokunTR]

Storage:Snapshots

[DeleOladokunTR]

### Additional findings after further investigation — all workarounds exhausted:

requestChecksumCalculation(WHEN_REQUIRED) is already in the 3.5.0 source

Looking at S3Service.java at the 3.5.0 tag, buildClient() already calls:

github-bug-report-opensearch-s3-outposts.md

[andrross]

Setting -Daws.requestChecksumCalculation=when_required (JVM property) — blocked by OpenSearch security manager

@DeleOladokunTR To my knowledge the Java agent in OpenSearch does not enforce any restrictions on setting system properties. Can you provide more details about the error you're seeing here? You should be able to provide this JVM arg by adding it to the OPENSEARCH_JAVA_OPTS environment variable.

[DeleOladokunTR]

Thank you for the correction, you're right, the JVM property is not blocked by the security manager. I apologize for the inaccuracy in my original report.

I re-tested with a fresh OpenSearch 3.5.0 pod (single-node, repository-s3 3.5.0 plugin) with the property set via OPENSEARCH_JAVA_OPTS:

OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m -Daws.requestChecksumCalculation=when_required"

Confirmed: The property is accepted, visible in /proc/<pid>/cmdline. No security manager error.

However, I also discovered the property is redundant, S3Service.java already calls:

builder.requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
       .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED);

I then traced through the AWS SDK v2.32.29 source (bundled in the plugin) and confirmed that with WHEN_REQUIRED + the S3 service model's requestChecksumRequired: false for PutObject, HttpChecksumUtils.isHttpChecksumCalculationNeeded() returns false — no flexible checksum header is sent. This was verified via HTTP wire-level logging (Apache HttpClient debug). No x-amz-checksum-* headers appear in the request.

Root Cause Correction, NOT CRC64NVME

My original diagnosis was wrong. The 501 is NOT caused by CRC64NVME checksums.

I enabled HTTP wire logging and captured the full PutObject request headers. The S3 Outposts error response XML explicitly identifies the offending header:

<Error>
  <Code>NotImplemented</Code>
  <Message>A header you provided implies functionality that is not implemented</Message>
  <Header>x-amz-acl</Header>
</Error>

The actual root cause is x-amz-acl: private , S3 Outposts does not support the ACL header.

The plugin always sends this header because S3BlobStore.initCannedACL() defaults to ObjectCannedACL.PRIVATE when the canned_acl setting is empty/null. There is no way to suppress it to null (which would cause the SDK to omit the header).

Additionally, x-amz-storage-class: STANDARD is also sent by default via initStorageClass(), which is wrong, S3 Outposts only supports the OUTPOSTS storage class.

Working Fix (Settings Only, No Code Change)

Setting canned_acl: "bucket-owner-full-control" + storage_class: "OUTPOSTS" resolves the 501:

PUT _snapshot/my-s3-outpost-repo
{
  "type": "s3",
  "settings": {
    "bucket": "arn:aws:s3-outposts:us-east-1:ACCOUNT:outpost/OUTPOST_ID/accesspoint/AP_NAME",
    "region": "us-east-1",
    "canned_acl": "bucket-owner-full-control",
    "storage_class": "OUTPOSTS"
  }
}

Verified: Full DR cycle tested, snapshot creation (SUCCESS), index deletion, restore from S3 Outposts snapshot, data integrity confirmed (all documents recovered).

Suggested Improvement

The plugin could benefit from supporting a "none" or empty ACL option that passes null to the SDK (omitting the x-amz-acl header entirely), since S3 Outposts doesn't support ACLs. The current bucket-owner-full-control workaround works but is a functional detour.

Similarly, documenting storage_class: "OUTPOSTS" as required for S3 Outposts would help users.

Apologies for the earlier misdirection about CRC64NVME. The workaround table from my previous comment is no longer relevant, the issue was never about checksums.

Tested on: OpenSearch 3.5.0 + repository-s3 3.5.0, EKS on AWS Outpost rack (Bottlerocket 1.55.0), S3 Outposts access point ARN.

[cwperks]

Similarly, documenting storage_class: "OUTPOSTS" as required for S3 Outposts would help users.

+1, this would be good to add on https://github.com/opensearch-project/documentation-website.