Error from s3 upgrade: "api error InvalidPart: One or more of the specified parts could not be found"

[nguyengg]

Full error message:

cause: operation error S3: CompleteMultipartUpload, https response error StatusCode: 400, RequestID: R5B20AV083Y6HD7S, HostID: E6JXPySCHvlrX/ehR/jWufF4HnhcfVKrAfB0FF3Vjvc97d1iS0VaG03yAviRHlybylla+7sNQBE=, api error InvalidPart: One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.

Issue vanishes when downgrading to github.com/aws/aws-sdk-go-v2/service/s3@v1.72.3 (https://github.com/aws/aws-sdk-go-v2/blob/main/service/s3/CHANGELOG.md#v1723-2025-01-14).

Looks like in the next version (https://github.com/aws/aws-sdk-go-v2/blob/main/service/s3/CHANGELOG.md#v1730-2025-01-15), checksum validation is enabled but the xy3.Upload method does not account for it.

[nguyengg]

After adding good old log.Printf debugging.

Using go get github.com/aws/aws-sdk-go-v2/service/s3@v1.72.3 (success):

part #1: types.CompletedPart{ChecksumCRC32:(*string)(nil), ChecksumCRC32C:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000046e10), PartNumber:(*int32)(0x1400000e954), noSmithyDocumentSerde:document.NoSerde{}}
part #2: types.CompletedPart{ChecksumCRC32:(*string)(nil), ChecksumCRC32C:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000982130), PartNumber:(*int32)(0x140009860bc), noSmithyDocumentSerde:document.NoSerde{}}
part #3: types.CompletedPart{ChecksumCRC32:(*string)(nil), ChecksumCRC32C:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x140009821b0), PartNumber:(*int32)(0x1400098611c), noSmithyDocumentSerde:document.NoSerde{}}
part #4: types.CompletedPart{ChecksumCRC32:(*string)(nil), ChecksumCRC32C:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000047870), PartNumber:(*int32)(0x1400000ec2c), noSmithyDocumentSerde:document.NoSerde{}}
part #5: types.CompletedPart{ChecksumCRC32:(*string)(nil), ChecksumCRC32C:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000982220), PartNumber:(*int32)(0x1400098617c), noSmithyDocumentSerde:document.NoSerde{}}
part #6: types.CompletedPart{ChecksumCRC32:(*string)(nil), ChecksumCRC32C:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000047700), PartNumber:(*int32)(0x1400000eb7c), noSmithyDocumentSerde:document.NoSerde{}}

Using go get github.com/aws/aws-sdk-go-v2/service/s3@v1.75.4 (failure):

part #1: types.CompletedPart{ChecksumCRC32:(*string)(0x140001ff230), ChecksumCRC32C:(*string)(nil), ChecksumCRC64NVME:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x140001ff240), PartNumber:(*int32)(0x1
400000f714), noSmithyDocumentSerde:document.NoSerde{}}
part #2: types.CompletedPart{ChecksumCRC32:(*string)(0x14000047bc0), ChecksumCRC32C:(*string)(nil), ChecksumCRC64NVME:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000047bd0), PartNumber:(*int32)(0x1
400000f254), noSmithyDocumentSerde:document.NoSerde{}}
part #3: types.CompletedPart{ChecksumCRC32:(*string)(0x140008920b0), ChecksumCRC32C:(*string)(nil), ChecksumCRC64NVME:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x140008920c0), PartNumber:(*int32)(0x1
40001cc2fc), noSmithyDocumentSerde:document.NoSerde{}}
part #4: types.CompletedPart{ChecksumCRC32:(*string)(0x140001ff160), ChecksumCRC32C:(*string)(nil), ChecksumCRC64NVME:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x140001ff170), PartNumber:(*int32)(0x1
400000f694), noSmithyDocumentSerde:document.NoSerde{}}
part #5: types.CompletedPart{ChecksumCRC32:(*string)(0x14000892030), ChecksumCRC32C:(*string)(nil), ChecksumCRC64NVME:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000892040), PartNumber:(*int32)(0x1
40001cc1bc), noSmithyDocumentSerde:document.NoSerde{}}
part #6: types.CompletedPart{ChecksumCRC32:(*string)(0x14000229000), ChecksumCRC32C:(*string)(nil), ChecksumCRC64NVME:(*string)(nil), ChecksumSHA1:(*string)(nil), ChecksumSHA256:(*string)(nil), ETag:(*string)(0x14000229010), PartNumber:(*int32)(0x1
4000220d94), noSmithyDocumentSerde:document.NoSerde{}}

Look like ChecksumCRC32 starts being returned.

[nguyengg]

https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html#dataintegrity-sdk-compat

SDK for Go V2 (1.x) | Yes | CRC32

That checks out, looks like CRC32 is enabled by default.

[nguyengg]

"Fixed" with aws/aws-sdk-go-v2#2960 (comment) by making checksum required only when CreateMultipartUpload requests it with:

	c.client = s3.NewFromConfig(cfg, func(options *s3.Options) {
		options.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
	})

[nguyengg]

Opening back up to track fixes.

Some data for debugging:

# UploadPart using CRC32 FullObject
PUT [redacted]
Host: s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.4 m/E,U,Z
Content-Length: 233718
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 3b3400ba-24b7-4a49-aa1c-fc5c3aca62c6
Amz-Sdk-Request: attempt=1; max=3
Authorization: [redacted]
Signature=[redacted]
Content-Type: application/octet-stream
X-Amz-Checksum-Crc32: 0P3sOg==
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20250207T004845Z
X-Amz-Sdk-Checksum-Algorithm: CRC32

# UploadPart using CRC32 Composite
PUT [redacted]
Host: s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.4 m/E,U,Z
Content-Length: 233718
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c5878e7b-67b5-4d08-a914-c54cd6949648
Amz-Sdk-Request: attempt=1; max=3
Authorization: [redacted]
Signature=[redacted]
Content-Type: application/octet-stream
X-Amz-Checksum-Crc32: 0P3sOg==
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20250207T004834Z
X-Amz-Sdk-Checksum-Algorithm: CRC32

# UploadPart without any checksum algorithm
PUT [redacted]
Host: s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.4 m/E,Z
Content-Length: 233763
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 6eef4462-bbe9-4fd6-a266-c4ff79c7ebf1
Amz-Sdk-Request: attempt=1; max=3
Authorization: [redacted
Signature=[redacted]
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20250207T005111Z
X-Amz-Decoded-Content-Length: 233718
X-Amz-Trailer: x-amz-checksum-crc32