[Bug]: _delta_log not written with external S3 provider

[AntoineRoll]

What happened?

I'm trying to use the delta-rs library to write a Delta table on OVH object storage.

When attempting to execute the following code:

import pandas as pd
import os
from deltalake import write_deltalake

os.environ["AWS_REGION"] = "gra"
os.environ["AWS_ENDPOINT_URL"] = "https://s3.gra.io.cloud.ovh.net"
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""

df = pd.DataFrame({'x': [1, 2, 3]})

write_deltalake('s3://opendatacatalog-storage-staging-writer/bronze/test', df)

I'm getting the following error:
write_deltalake_rust(
OSError: Generic S3 error
↳ Error performing PUT https://s3.gra.io.cloud.ovh.net/opendatacatalog-storage-staging-writer/bronze/test/_delta_log/00000000000000000000.json in 1.985295278s, after 10 retries, max_retries
↳ 10, retry_timeout
↳ 180s - Server returned non-2xx status code
↳ 501 Not Implemented

With RUST_LOG=debug I only see those kind of additional lines:

[2025-11-02T19:35:56Z INFO object_store::client::retry] Encountered server error, backing off for 0.44229764 seconds, retry 10 of 10
[2025-11-02T19:35:57Z DEBUG hyper_util::client::legacy::pool] reuse idle connection for ("https", s3.gra.io.cloud.ovh.net)
[2025-11-02T19:35:57Z DEBUG hyper_util::client::legacy::pool] pooling idle connection for ("https", s3.gra.io.cloud.ovh.net)

My main problem is that this is only happening on writes of delta log. The parquet files are correctly written. My rust knowledge is way too limited to debug this, but I've successfully uploaded the delta log using AWS CLI, which leaves me stuck. The issue is likely within the vendor OVHCloud but I can't replicate it outside of the deltalake lib.

Their S3 limitations are described here; https://help.ovhcloud.com/csm/en-gb-public-cloud-storage-s3-limitations?id=kb_article_view&sysparm_article=KB0047378 but I don't see anything that could cause this.

Any help to narrow down the problem would be greatly appreciated.

Expected behavior

• Delta tables should be correctly written in external S3 vendor
• There should not be a write error of the transaction log after the data is correctly written
• The logs should be more verbose regarding the exact request that failed

Operating System

Linux

Binding

Python

Bindings Version

1.2.1

Steps to reproduce

1. Setup an OVHCloud Object Storage bucket
2. Run:

import pandas as pd
import os
from deltalake import write_deltalake

BUCKET_NAME = "testXYZ"

os.environ["AWS_REGION"] = "gra"
os.environ["AWS_ENDPOINT_URL"] = "https://s3.gra.io.cloud.ovh.net"
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""

df = pd.DataFrame({'x': [1, 2, 3]})

write_deltalake(f's3://{BUCKET_NAME}/test', df)

Relevant logs

Traceback (most recent call last):
  File "/home/antoine/Documents/OpenDataCatalog/debug_s3.py", line 13, in <module>
    write_deltalake('s3a://opendatacatalog-storage-staging-writer/bronze/test', df)
  File "/home/antoine/Documents/OpenDataCatalog/.venv/lib/python3.11/site-packages/deltalake/writer/writer.py", line 147, in write_deltalake
    write_deltalake_rust(
OSError: Generic S3 error
          ↳ Error performing PUT https://s3.gra.io.cloud.ovh.net/opendatacatalog-storage-staging-writer/bronze/test/_delta_log/00000000000000000000.json in 4.637281632s, after 10 retries, max_retries
           ↳ 10, retry_timeout
            ↳ 180s  - Server returned non-2xx status code
             ↳ 501 Not Implemented

[ion-elgreco]

Your S3 store seems to not support put-if-absent, can you replicate by using the obstore library and then doing: https://developmentseed.org/obstore/latest/api/put/ and then using putmode create, this will likely fail and overwrite should work.

[AntoineRoll]

Hi @ion-elgreco , thanks for the response. It does behave exactly as you described !
Using this code:

import obstore
from obstore.store import S3Store

obstore.put(
    store=S3Store(
        bucket="opendatacatalog-storage-staging-writer",
        endpoint="https://s3.gra.io.cloud.ovh.net",
        region="gra",
        access_key_id="",
        secret_access_key=""
    ),
    path="bronze/test/toto.txt",
    file="toto".encode("utf-8"),
    mode="overwrite"
)

No issue with overwrite mode, but when using create

Getting a similar trace from deltalake.

  File "/home/antoine/Documents/OpenDataCatalog/pipelines/main.py", line 6, in <module>
    obstore.put(
obstore.exceptions.GenericError: Generic S3 error: Error performing PUT https://s3.gra.io.cloud.ovh.net/opendatacatalog-storage-staging-writer/bronze/test/toto.txt in 6.809579331s, after 10 retries, max_retries: 10, retry_timeout: 180s  - Server returned non-2xx status code: 501 Not Implemented: 

Debug source:
Generic {
    store: "S3",
    source: RetryError(
        RetryErrorImpl {
            method: PUT,
            uri: Some(
                https://s3.gra.io.cloud.ovh.net/opendatacatalog-storage-staging-writer/bronze/test/toto.txt,
            ),
            retries: 10,
            max_retries: 10,
            elapsed: 6.809579331s,
            retry_timeout: 180s,
            inner: Status {
                status: 501,
                body: None,
            },
        },
    ),
} ```

[AntoineRoll]

I'm assuming that based on this spec https://github.com/delta-io/delta/blob/master/PROTOCOL.md#creation-of-new-log-entries it is not existing, nor desirable to have an option to overwrite the PutMode? S3's "Create mode" should always be used if I am correct?

[ion-elgreco]

@AntoineRoll try setting in the storage_options:

'copy_if_not_exists': 'true'

This uses the old log store for S3 stores that don't support put if absent but do support copy if not exists

[AntoineRoll]

Thanks a lot. The value shouldn't be 'true' but rather 'header: cf-copy-destination-if-none-match: *'
The following code has worked for me:

import pandas as pd
import os
from deltalake import write_deltalake

BUCKET_NAME = "testXYZ"

os.environ["AWS_REGION"] = "gra"
os.environ["AWS_ENDPOINT_URL"] = "https://s3.gra.io.cloud.ovh.net"
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""

df = pd.DataFrame({'x': [1, 2, 3]})

write_deltalake(f's3://{BUCKET_NAME}/test', df, storage_options={'copy_if_not_exists': 'header: cf-copy-destination-if-none-match: *'})

I will try to reach out to the S3 vendor, such that they can implement it, but at least now I know which feature to request.

Once again, thank you so much @ion-elgreco you have greatly helped me

[ion-elgreco]

@AntoineRoll perfect! Glad you found the right header value