Checksum `x-amz-checksum-crc32` seems to be added INSIDE my files

[jgaucher-cs]

Describe the bug

When uploading a local file to my s3 storage, some kind of checksum seems to be added directory inside the contents of my file:

326d # added by boto3

# my file contents ...

0 # added by boto3
x-amz-checksum-crc32:6da4RA== # added by boto3

Regression Issue

• Select this option if this issue appears to be a regression.

It does not happen with boto3==1.35.41

Expected Behavior

The uploaded file contents should not be modified.

Current Behavior

The uploaded file contents are modified.

Reproduction Steps

Create a dummy file locally:

echo toto > /tmp/toto.txt

Install boto3:

pip install boto3==1.36.16

Init s3 client and upload file:

import boto3
s3_session = boto3.session.Session()
client = s3_session.client(
    service_name="s3",
    aws_access_key_id="my-access",
    aws_secret_access_key="my-secret",
    endpoint_url="https://my-url",
    region_name="my-region",
)
client.upload_file("/tmp/toto.txt", "my-bucket", "my-folder/toto.txt")

# Now read uploaded file contents:
print(client.get_object(Bucket="my-bucket", Key="my-folder/toto.txt")["Body"].read())
# returns: b'5\r\ntoto\n\r\n0\r\nx-amz-checksum-crc32:+H0IvQ==\r\n\r\n'

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.36.16

Environment details (OS name and version, etc.)

Linux Ubuntu 22

[jgaucher-cs]

It seems to be resolved if I add this at the top of my code:

import os
os.environ["AWS_REQUEST_CHECKSUM_CALCULATION"] = "when_required"
os.environ["AWS_RESPONSE_CHECKSUM_VALIDATION"] = "when_required"

[khushail]

Hi @jgaucher-cs , thanks for reaching out. This change was recently announced by Python team, related to Announcement: S3 default integrity change -

In AWS SDK for Python v1.36.0, we released changes to the S3 client that adopts new default integrity protections. For more information on default integrity behavior, please refer to the official [SDK documentation](https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html)

The workaround you suggested has also been mentioned to bypass the default checksum.

Hope that clarifies your questions. Please feel free to reach out if this does not help.

Thanks

[jgaucher-cs]

Hi @khushail, thank you for your response. The official SDK documentation says:

Amazon S3 independently calculates a checksum on the server side and validates it against the provided value before durably storing the object and its checksum in the object's metadata.

In my case, the checksum is not stored in the object's metadata, it's stored inside the file contents, making it corrupt and unreadable (e.g. if it's a Python script, it cannot be run anymore). Why is that ?

[ZeniT21]

I also have this problem. Have any ideas?