Minor: Add tests showing aggregate behavior for NaNs#10634
Minor: Add tests showing aggregate behavior for NaNs#10634comphead merged 3 commits intoapache:mainfrom
Conversation
| query RRRRI | ||
| select min(col_f64_nan), max(col_f64_nan), avg(col_f64_nan), sum(col_f64_nan), count(col_f64_nan) from float_table; | ||
| ---- | ||
| -128.2 32768.3 10889.133333333333 32667.4 3 |
There was a problem hiding this comment.
it is surprising to me that the aggregates on f32 seem to handle Nan consistently with postgres, but the aggregates on f64 do not (I would expect max(col_f64_nan), avg(col_f64_nan), sum(col_f64_nan) to all return NaN
| query RRRRI | ||
| select min(col_f32), max(col_f32), avg(col_f32), sum(col_f32), count(col_f32) from float_table; | ||
| ---- | ||
| -128.2 32768.3 10889.13359451294 32667.40078353882 3 |
There was a problem hiding this comment.
What concerns me is below:
DataFusion CLI v38.0.0
> CREATE TABLE float_table (
col_f32 FLOAT,
col_f32_nan FLOAT,
col_f64 DOUBLE,
col_f64_nan DOUBLE
) as VALUES
( -128.2, -128.2, -128.2, -128.2 ),
( 32768.3, arrow_cast('NAN','Float32'), 32768.3, 32768.3 ),
( 27.3, 27.3, 27.3, arrow_cast('NAN','Float64') );
0 row(s) fetched.
Elapsed 0.028 seconds.
> select min(col_f32), max(col_f32), avg(col_f32), sum(col_f32), count(col_f32) from float_table;
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| MIN(float_table.col_f32) | MAX(float_table.col_f32) | AVG(float_table.col_f32) | SUM(float_table.col_f32) | COUNT(float_table.col_f32) |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| -128.2 | 32768.3 | 10889.13359451294 | 32667.40078353882 | 3 |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
but in both duckdb and postgres
>
> select -128.2 col_f32, -128.2 col_f32_nan, -128.2 col_f64, -128.2 col_f64_nan union all
> select 32768.3, 'NaN'::DOUBLE PRECISION, 32768.3, 32768.3 union all
> select 27.3, 27.3, 27.3, 'NaN'::DOUBLE PRECISION) x
> ;
┌──────────────┬──────────────┬────────────────────┬───────────────┬────────────────┐
│ min(col_f32) │ max(col_f32) │ avg(col_f32) │ sum(col_f32) │ count(col_f32) │
│ decimal(6,1) │ decimal(6,1) │ double │ decimal(38,1) │ int64 │
├──────────────┼──────────────┼────────────────────┼───────────────┼────────────────┤
│ -128.2 │ 32768.3 │ 10889.133333333333 │ 32667.4 │ 3 │
└──────────────┴──────────────┴────────────────────┴───────────────┴────────────────┘
There was a problem hiding this comment.
AVG is incorrect precision
There was a problem hiding this comment.
AVG is incorrect precision
One reason the AVG result may different is that your example used single precision (f32). When I run the same query with double precision (f64) DataFusion gets the same 10889.133333333333 as duckdb
> select min(col_f64), max(col_f64), avg(col_f64), sum(col_f64), count(col_f64) from float_table;
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| MIN(float_table.col_f64) | MAX(float_table.col_f64) | AVG(float_table.col_f64) | SUM(float_table.col_f64) | COUNT(float_table.col_f64) |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
| -128.2 | 32768.3 | 10889.133333333333 | 32667.4 | 3 |
+--------------------------+--------------------------+--------------------------+--------------------------+----------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
comphead
left a comment
There was a problem hiding this comment.
lgtm thanks @alamb
however I think there is some issue with UNION ALL and type derivation
> select avg(a) from (select -128.2::float a union all select 32768.3 union all select 27.3);
+--------------------+
| AVG(a) |
+--------------------+
| 10889.133334350587 |
+--------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
> select avg(a) from (select -128.2::float a union all select 32768.3::float union all select 27.3::float);
+-------------------+
| AVG(a) |
+-------------------+
| 10889.13359451294 |
+-------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
I expect those numbers to be the same. And that what duck db does
|
Filed #10688 |
* Minor: Add tests showing aggregate behavior for NaNs * Fix NaN setup
Which issue does this PR close?
Related to #10627 and #8031
Rationale for this change
While looking at #10627 from @westonpace I couldn't find any existing tests for how Nans worked in aggregates.
What changes are included in this PR?
Add a test that documents how Nans work today in the basic aggregates
Are these changes tested?
Only tests
Are there any user-facing changes?
No