[SPARK-55969][SQL] regr_r2 should treat first param as dependent variable

### What changes were proposed in this pull request?

The `regr_r2` function currently is swapping the two parameters when processing the results. This causes issues only in few special cases, as otherwise the result does not depend from the order of its parameters.

### Why are the changes needed?

With special cases, the result is wrong.

### Does this PR introduce _any_ user-facing change?

Yes, fixes the result of regr_r2 in the special cases in which either the dependent or independent variable has always the same value.

### How was this patch tested?

Added UTs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #54901 from mgaido91/SPARK-55969.

Authored-by: Marco Gaido <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>

Author: mgaido91

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala

@@ -147,8 +147,10 @@ case class RegrR2(y: Expression, x: Expression) extends PearsonCorrelation(y, x,
   override def prettyName: String = "regr_r2"
   override val evaluateExpression: Expression = {
     val corr = ck / sqrt(xMk * yMk)
-    If(xMk === 0.0, Literal.create(null, DoubleType),
-      If(yMk === 0.0, Literal.create(1.0, DoubleType), corr * corr))
+    // In PearsonCorrelation, x and y are swapped, so here xMk refers to the dependent variable
+    // and yMk to the independent variable
+    If(yMk === 0.0, Literal.create(null, DoubleType),
+      If(xMk === 0.0, Literal.create(1.0, DoubleType), corr * corr))
   }
   override protected def withNewChildrenInternal(
       newLeft: Expression, newRight: Expression): RegrR2 =

sql/core/src/test/resources/sql-tests/analyzer-results/linear-regression.sql.out

@@ -407,3 +407,29 @@ Aggregate [k#x], [k#x, regr_intercept(cast(y#x as double), cast(x#x as double))
             +- Project [k#x, y#x, x#x]
                +- SubqueryAlias testRegression
                   +- LocalRelation [k#x, y#x, x#x]
+
+
+-- !query
+SELECT regr_r2(k, x) FROM testRegression where k=2
+-- !query analysis
+Aggregate [regr_r2(cast(k#x as double), cast(x#x as double)) AS regr_r2(k, x)#x]
++- Filter (k#x = 2)
+   +- SubqueryAlias testregression
+      +- View (`testRegression`, [k#x, y#x, x#x])
+         +- Project [cast(k#x as int) AS k#x, cast(y#x as int) AS y#x, cast(x#x as int) AS x#x]
+            +- Project [k#x, y#x, x#x]
+               +- SubqueryAlias testRegression
+                  +- LocalRelation [k#x, y#x, x#x]
+
+
+-- !query
+SELECT regr_r2(y, k) FROM testRegression where k=2
+-- !query analysis
+Aggregate [regr_r2(cast(y#x as double), cast(k#x as double)) AS regr_r2(y, k)#x]
++- Filter (k#x = 2)
+   +- SubqueryAlias testregression
+      +- View (`testRegression`, [k#x, y#x, x#x])
+         +- Project [cast(k#x as int) AS k#x, cast(y#x as int) AS y#x, cast(x#x as int) AS x#x]
+            +- Project [k#x, y#x, x#x]
+               +- SubqueryAlias testRegression
+                  +- LocalRelation [k#x, y#x, x#x]

sql/core/src/test/resources/sql-tests/inputs/linear-regression.sql

@@ -50,3 +50,7 @@ SELECT regr_intercept(y, x) FROM testRegression;
 SELECT regr_intercept(y, x) FROM testRegression WHERE x IS NOT NULL AND y IS NOT NULL;
 SELECT k, regr_intercept(y, x) FROM testRegression GROUP BY k;
 SELECT k, regr_intercept(y, x) FROM testRegression WHERE x IS NOT NULL AND y IS NOT NULL GROUP BY k;
+
+-- SPARK-55969: regr_r2 should treat first param as dependent variable
+SELECT regr_r2(k, x) FROM testRegression where k=2;
+SELECT regr_r2(y, k) FROM testRegression where k=2;

sql/core/src/test/resources/sql-tests/results/linear-regression.sql.out

@@ -274,3 +274,19 @@ SELECT k, regr_intercept(y, x) FROM testRegression WHERE x IS NOT NULL AND y IS
 struct<k:int,regr_intercept(y, x):double>
 -- !query output
 2	1.1547344110854496
+
+
+-- !query
+SELECT regr_r2(k, x) FROM testRegression where k=2
+-- !query schema
+struct<regr_r2(k, x):double>
+-- !query output
+1.0
+
+
+-- !query
+SELECT regr_r2(y, k) FROM testRegression where k=2
+-- !query schema
+struct<regr_r2(y, k):double>
+-- !query output
+NULL