write_deltalake is not creating checkpoints

[yefetBenTili]

Delta-rs version: 0.10.0

Binding:

Environment:
Cloud provider: AWS
OS: macOs
Other:

---

We have a Delta Lake on S3 with over 2TB of data, which we write to daily. using we use write_deltalake (writing new partitions every day with partition filters)

We noticed a significant decline in read performance after a few weeks. which led to further investigation I discovered that no checkpoint files were being written. Currently, I am at over 4000 transaction JSON files, and no checkpoint file is there.

As far as I know, Delta's default behavior includes checkpointing after the 10th version. Is there a way to enforce this or trigger it manually?

    write_deltalake(
        df
        mode="overwrite",
        schema=config.persrec_history_schema,
        storage_options={"AWS_S3_ALLOW_UNSAFE_RENAME": "True"},
        partition_by=[*partition_dict.keys()],
        partition_filters= partiton_filters],
    )

[djouallah]

I use this

dt = DeltaTable(Path_Delta,storage_options=storage_options) 
      if len(dt.file_uris()) >= 9 :
            dt.optimize.compact()
            dt.vacuum(retention_hours=172,dry_run=False,  enforce_retention_duration=False)
            dt.create_checkpoint()

[ion-elgreco]

@yefetBenTili we are already tracking this here: #913

[ion-elgreco]

Closing it since we track it in #913

[slanton-a]

Saw the conversion on the other issue, but I still don't understand. Should I expect write_deltalake to create a checkpoint automatically? or do I need to manually call create_checkpoint?

[djouallah]

@slanton-a with the latest version, it is automatic by default every 100 commits