s3: add Content-MD5 header in S3Uploader for compatibility options by GMHDBJD · Pull Request #65457 · pingcap/tidb

GMHDBJD · 2026-01-07T07:27:33Z

What problem does this PR solve?

Issue Number: close #65331

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
tested on KS3, before this PR when cleanup import jobs we meet error. after this PR there is no such error.
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2026-01-07T07:28:02Z

Hi @GMHDBJD. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2026-01-07T08:00:48Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.6554%. Comparing base (7f129e5) to head (d0a523e).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #65457        +/-   ##
================================================
- Coverage   77.8399%   77.6554%   -0.1846%     
================================================
  Files          1983       1907        -76     
  Lines        542760     530455     -12305     
================================================
- Hits         422484     411927     -10557     
+ Misses       118616     118522        -94     
+ Partials       1660          6      -1654

Flag	Coverage Δ
integration	`41.5396% <0.0000%> (-6.6481%)`	⬇️
unit	`76.7726% <100.0000%> (+0.3087%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`56.7974% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`48.7449% <ø> (-12.3017%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

D3Hunter · 2026-01-16T07:46:13Z

pkg/objstore/s3store/s3.go

 	input := buildPutObjectInput(options, file, []byte("check"))
-	_, err = svc.PutObject(ctx, input)
+	var optFns []func(*s3.Options)
+	if options.Provider != "" && options.Provider != "aws" {


please merge master, we can also check through s3Compatible now

# Conflicts: # pkg/objstore/s3store/s3.go

Copilot

Pull request overview

This PR ensures S3-compatible object stores (e.g., KS3) receive a Content-MD5 header on uploads, fixing compatibility issues encountered during import job cleanup (missing Content-Md5 header errors).

Changes:

Updated s3Client.CheckPutAndDeleteObject and s3Client.PutObject to pass a conditional client option that adds Content-MD5 when s3Compatible is true.
Propagated the s3Compatible flag into multipartWriter and configured both the low-level multipart writer (UploadPart) and the high-level manager.Uploader (MultipartUploader) to use the withContentMD5 option for compatible backends.
Centralized checksum behavior using the new withContentMD5 helper, which disables the AWS flexible checksum stack and installs AddContentChecksumMiddleware instead.

Copilot · 2026-01-19T03:45:04Z

pkg/objstore/s3store/client.go

+	var optFns []func(*s3.Options)
+	if c.s3Compatible {
+		optFns = []func(*s3.Options){withContentMD5}
+	}
+	_, err := c.svc.PutObject(ctx, input, optFns...)


The new s3Compatible branch and use of withContentMD5 in PutObject are not covered by tests; existing tests in client_test.go only exercise the default (AWS) path where s3Compatible remains false. Consider adding unit tests that construct an s3Client with s3Compatible set to true and assert that PutObject invokes the S3API mock with the expected client option (e.g., that at least one option is passed, or that withContentMD5 is used).

Copilot · 2026-01-19T03:45:05Z

pkg/objstore/s3store/client.go

+	var optFns []func(*s3.Options)
+	if u.s3Compatible {
+		optFns = []func(*s3.Options){withContentMD5}
+	}
+	uploadResult, err := u.svc.UploadPart(ctx, partInput, optFns...)


The multipart upload writer path now conditionally applies withContentMD5 when s3Compatible is true, but there are no tests covering this new behavior; current tests only validate multipartWriter against the default configuration. Please add tests that create a multipartWriter with s3Compatible set to true and verify that UploadPart is called on the S3API mock with an additional client option (ensuring the compatibility-specific checksum middleware is exercised).

GMHDBJD · 2026-01-19T04:34:12Z

/retest

tiprow · 2026-01-19T04:34:35Z

@GMHDBJD: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2026-01-19T06:55:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/objstore/OWNERS~~ [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2026-01-19T06:55:19Z

[LGTM Timeline notifier]

Timeline:

2026-01-08 08:18:22.929141804 +0000 UTC m=+863058.747450236: ☑️ agreed by lance6716.
2026-01-19 06:55:18.044468442 +0000 UTC m=+390545.658425298: ☑️ agreed by D3Hunter.

…ingcap#65457) close pingcap#65331

storage: add s3Compatible flag to S3Uploader for compatibility options

f50dba5

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 7, 2026

lance6716 approved these changes Jan 8, 2026

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 8, 2026

Merge remote-tracking branch 'upstream/master' into fixChecksum

0270181

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Jan 14, 2026

GMHDBJD added 2 commits January 15, 2026 10:37

Merge remote-tracking branch 'upstream/master' into fixChecksum

a3d0ed4

update

cceebad

ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 15, 2026

update

2b8e39b

D3Hunter reviewed Jan 16, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/master' into fixChecksum

cc85ade

# Conflicts: # pkg/objstore/s3store/s3.go

Copilot AI review requested due to automatic review settings January 19, 2026 03:35

Copilot started reviewing on behalf of GMHDBJD January 19, 2026 03:35 View session

Copilot AI reviewed Jan 19, 2026

View reviewed changes

s3store: add UT for Content-MD5 option

d0a523e

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 19, 2026

D3Hunter approved these changes Jan 19, 2026

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 19, 2026

ti-chi-bot bot merged commit 7670ef5 into pingcap:master Jan 19, 2026
29 of 31 checks passed

hawkingrei pushed a commit to hawkingrei/tidb that referenced this pull request Jan 19, 2026

s3: add Content-MD5 header in S3Uploader for compatibility options (p…

d77e62b

…ingcap#65457) close pingcap#65331

D3Hunter mentioned this pull request Mar 13, 2026

br: upgrade AWS sdk from v1 to v2 #66634

Merged

13 tasks

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Mar 13, 2026

s3: add Content-MD5 header in S3Uploader for compatibility options (p…

29a853e

…ingcap#65457) close pingcap#65331

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3: add Content-MD5 header in S3Uploader for compatibility options#65457

s3: add Content-MD5 header in S3Uploader for compatibility options#65457
ti-chi-bot[bot] merged 7 commits intopingcap:masterfrom
GMHDBJD:fixChecksum

GMHDBJD commented Jan 7, 2026

Uh oh!

tiprow bot commented Jan 7, 2026

Uh oh!

codecov bot commented Jan 7, 2026 •

edited

Loading

Uh oh!

D3Hunter Jan 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

GMHDBJD commented Jan 19, 2026

Uh oh!

tiprow bot commented Jan 19, 2026

Uh oh!

ti-chi-bot bot commented Jan 19, 2026

Uh oh!

ti-chi-bot bot commented Jan 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

GMHDBJD commented Jan 7, 2026

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Jan 7, 2026

Uh oh!

codecov bot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

D3Hunter Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

GMHDBJD commented Jan 19, 2026

Uh oh!

tiprow bot commented Jan 19, 2026

Uh oh!

ti-chi-bot bot commented Jan 19, 2026

Uh oh!

ti-chi-bot bot commented Jan 19, 2026

[LGTM Timeline notifier]

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Jan 7, 2026 •

edited

Loading