Skip to content

Prevent optimizer from pruning UnnestExec.#5282

Merged
alamb merged 2 commits intoapache:mainfrom
vincev:unnest-fix
Feb 16, 2023
Merged

Prevent optimizer from pruning UnnestExec.#5282
alamb merged 2 commits intoapache:mainfrom
vincev:unnest-fix

Conversation

@vincev
Copy link
Copy Markdown
Contributor

@vincev vincev commented Feb 14, 2023

Which issue does this PR close?

Closes #5281.

Rationale for this change

When running an aggregate count on an unnested column the aggregate_statistics optimizer removes UnnestExec from the ExecutionPlan.

What changes are included in this PR?

UnnestExec::statistics returns default statistics instead of returning child statistics.

Are these changes tested?

Tests for the fix is here

Are there any user-facing changes?

Aggregate count on unnested column now works as expected.

@github-actions github-actions bot added the core Core DataFusion crate label Feb 14, 2023
Copy link
Copy Markdown
Member

@jackwener jackwener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you @vincev

@vincev
Copy link
Copy Markdown
Contributor Author

vincev commented Feb 16, 2023

Thank you @jackwener, @alamb

@alamb alamb merged commit afb73be into apache:main Feb 16, 2023
@vincev vincev deleted the unnest-fix branch February 16, 2023 18:34
@ursabot
Copy link
Copy Markdown

ursabot commented Feb 16, 2023

Benchmark runs are scheduled for baseline = ac2222a and contender = afb73be. afb73be is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

jiangzhx pushed a commit to jiangzhx/arrow-datafusion that referenced this pull request Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimizer prunes UnnestExec on aggregate count

4 participants