Environment
Delta-rs version: v0.18.1
Binding: rust
Bug
What happened: In my project, there is a service that runs on a scheduled basis to compact and optimise delta tables. After upgrading delta-rs to v0.18.1 (from v0.17.1) we see an error saying Delta(Arrow { source: SchemaError("Could not find column keys") }). There is no column called 'keys' in our datasets so I am do not understand why we get this error. The error comes from deltalake-core-0.18.1/src/operations/cast.rs:150.
What you expected to happen:
No schema errors should be reported when upgrading to a new version of delta-rs.
How to reproduce it:
Not entirely sure how to reproduce it since we do not have a column called 'keys' in our datasets.
More details:
This error does not occur in the version we run in production which is v0.17.1 (commit hash 25962a0). The look back window on the compactor service is 2 days. So after 2 days of running the new version, we no longer see the SchemaError which leads me to believe that the v0.18.1 cannot optimise tables that contain data that was inserted by v0.17.1.
Environment
Delta-rs version: v0.18.1
Binding: rust
Bug
What happened: In my project, there is a service that runs on a scheduled basis to compact and optimise delta tables. After upgrading delta-rs to v0.18.1 (from v0.17.1) we see an error saying
Delta(Arrow { source: SchemaError("Could not find column keys") }). There is no column called 'keys' in our datasets so I am do not understand why we get this error. The error comes fromdeltalake-core-0.18.1/src/operations/cast.rs:150.What you expected to happen:
No schema errors should be reported when upgrading to a new version of delta-rs.
How to reproduce it:
Not entirely sure how to reproduce it since we do not have a column called 'keys' in our datasets.
More details:
This error does not occur in the version we run in production which is v0.17.1 (commit hash 25962a0). The look back window on the compactor service is 2 days. So after 2 days of running the new version, we no longer see the SchemaError which leads me to believe that the v0.18.1 cannot optimise tables that contain data that was inserted by v0.17.1.