[Bug]: Docs describe `partition_filters` parameter to `write_deltalake` that doesn't exist

[zyd14]

What happened?

The docs for writing to delta tables describe a partition_filters parameter for overwriting specific partitions with deltalake, but when I look at the source code I don't see this parameter in v1.2.1 (latest as of time of writing) or in main. There are references in Github Issues to partition_filters going back to 2023, so it seems like it was a thing at some point.

Did partition_filters get removed from the API? If so, is it still possible to overwrite a list of partitions without overwriting the whole table?

Expected behavior

I expected partition_filters to be a valid parameter to write_deltalake given the description in "Overwriting a partition" section of the docs

Operating System

None

Binding

Python

Bindings Version

v1.2.1, also main

Steps to reproduce

Look at the source code here

Relevant logs

[zyd14]

After digging in a bit further I think this has been replaced with the predicate argument. I can submit a PR to remove the partition_filters example if that's desirable?

[ion-elgreco]

@zyd14 if you can update the docs that would be very helpful!

[zyd14]

Sure thing I can do that! I'll try to get a PR up tonight.

[wernerdaehn]

Ran into the same problem. But the use case is the same, I want to write the data into a single partition only, potentially into a table that is not partitioned yet. I have two containers, containerA is writing partition 1, containerB partition 2.
The predicate argument is for a different purpose, isn't it?

[zyd14]

@wernerdaehn maybe someone else can chime in but my take from the docs was that predicate is a generalization of partition_filters written in a way that it can apply to other operations where predicate pushdown applies. But it's entirely possible I'm wrong about that, I haven't gotten a chance to test it yet... Should be doing that later today.

[wernerdaehn]

@zyd14 I am also not feeling comfortable stating something. I am arguing based on my assumption how thinks must work and I can be wrong here. I also did not find the code yet, where the partition filter got turned into the predicate - that would help.
But my understanding is, predicate is a filter to optimize the file reading. So if the predicate is a filter on a column and that happens to be the partition clause, the writer knows only a single partition is impacted.
So from that point of view we might be safe.

If the table was not partitioned, the predicate will not suddenly partition it. The way I read the docs was however, with the predicate filter it will be partitioned. Maybe predicate and partition_by must be set?

Anyway, let's see how it goes. The point I wanted to raise with my comment was mostly to document those aspects also.

[zyd14]

@wernerdaehn yeah I'll make a PR today to update the docs once I confirm the behavior. The reason I think predicate is just a generalization is that I think a predicate could also be used with z-ordered tables to do file skipping.

I don't think partition_filters or predicate are meant to introduce partitioning to an unpartitioned table.

[ion-elgreco]

Predicate is to replace a partition. Partition by is to set which columns a table needs to be partitioned on

[zyd14]

@wernerdaehn I did some testing just now, it does seem that predicate works similarly to partition_filters and will replace the partition that matches the predicate given. If the data being written and its predicate don't match a partition in the target table a new partition will be created. As ion-elgreco said partition_by is more useful when creating a table with partitioning... predicate doesn't require that the table is partitioned, it's a more generalized way of saying "overwrite rows that match the given predicate filter". However, if the table is partitioned and the predicate is for a partition column, then it will only overwrite the matching partition (as expected).

I hope that helps, I have a draft PR up (#3912) that removes the partition_filters references from the docs for write_deltalake. If you think any examples could be added that would help clarify usage let me know.