Skip to content

[SPARK-56391] [SQL] Support unwrap string type to date type in UnwrapCastInBinaryComparison#55257

Open
daijy wants to merge 4 commits intoapache:masterfrom
daijy:unwrapstringtodate
Open

[SPARK-56391] [SQL] Support unwrap string type to date type in UnwrapCastInBinaryComparison#55257
daijy wants to merge 4 commits intoapache:masterfrom
daijy:unwrapstringtodate

Conversation

@daijy
Copy link
Copy Markdown

@daijy daijy commented Apr 8, 2026

What changes were proposed in this pull request?

Improve the existing UnwrapCastInBinaryComparison to unwrap string to date, ie. cast(date_key_string as date)=date_literal_constant => date_key_string=cast(date_literal_constant, string)

Why are the changes needed?

Spark fail to push date_key_string=date_literal_constant condition to iceberg table, this works fine for parquet table (date_key is a string partition column):

explain select date_key, count(*) from roku.fact_device_log_amoeba_events_hourly where date_key=date_add(current_date, -10) group by date_key;

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[date_key#25], functions=[count(1)])
   +- Exchange hashpartitioning(date_key#25, 200), ENSURE_REQUIREMENTS, [plan_id=16]
      +- HashAggregate(keys=[date_key#25], functions=[partial_count(1)])
         +- Filter (cast(date_key#25 as date) = 2026-03-28)
            +- BatchScan spark_catalog.roku.fact_device_log_amoeba_events_hourly[date_key#25] spark_catalog.roku.fact_device_log_amoeba_events_hourly (branch=null) [filters=date_key IS NOT NULL, groupedBy=] RuntimeFilters: []

The reason is Spark add a cast on the left side to cast string to date. The cast expression pushed to iceberg data source which is not able to process. A workaround is to add cast on right side, ie. cast date to string.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

Was this patch authored or co-authored using generative AI tooling?

claude-4.5-sonnet

@daijy daijy force-pushed the unwrapstringtodate branch from c701e3b to d2bb836 Compare April 8, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants