Merge branch 'main' into iceberg-rust

Author: mbutrovich

.github/actions/java-test/action.yaml

@@ -62,7 +62,32 @@ runs:
     - name: Run Maven compile
       shell: bash
       run: |
-        ./mvnw -B compile test-compile scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ inputs.maven_opts }}
+        ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ inputs.maven_opts }}
+
+    - name: Setup Node.js
+      uses: actions/setup-node@v6
+      with:
+        node-version: '24'
+
+    - name: Install prettier
+      shell: bash
+      run: |
+        npm install -g prettier
+
+    - name: Run prettier
+      shell: bash
+      run: |
+        npx prettier "**/*.md" --write
+
+    - name: Mark workspace as safe for git
+      shell: bash
+      run: |
+        git config --global --add safe.directory "$GITHUB_WORKSPACE"
+
+    - name: Check for any local git changes (such as generated docs)
+      shell: bash
+      run: |
+        ./dev/ci/check-working-tree-clean.sh
 
     - name: Run all tests
       shell: bash

.github/pull_request_template.md

@@ -10,7 +10,7 @@ Closes #.
 
 <!--
  Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
- Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
+ Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
 -->
 
 ## What changes are included in this PR?

.github/workflows/pr_markdown_format.yml

@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Check Markdown Formatting
+
+concurrency:
+  group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+  cancel-in-progress: true
+
+on:
+  pull_request:
+    paths:
+      - '**.md'
+
+jobs:
+  prettier-check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: '24'
+
+      - name: Install prettier
+        run: npm install -g prettier
+
+      - name: Check markdown formatting
+        run: |
+          # if you encounter error, run prettier locally and commit changes using instructions at:
+          #
+          # https://datafusion.apache.org/comet/contributor-guide/development.html#how-to-format-md-document
+          #
+          prettier --check "**/*.md"

CHANGELOG.md

@@ -19,4 +19,4 @@ under the License.
 
 # Apache DataFusion Comet Changelog
 
-Comprehensive changelogs for each release are available [here](dev/changelog).
+Comprehensive changelogs for each release are available [here](dev/changelog).

README.md

@@ -34,7 +34,7 @@ Apache DataFusion Comet is a high-performance accelerator for Apache Spark, buil
 performance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the
 Spark ecosystem without requiring any code changes.
 
-Comet also accelerates Apache Iceberg, when performing Parquet scans from Spark. 
+Comet also accelerates Apache Iceberg, when performing Parquet scans from Spark.
 
 [Apache DataFusion]: https://datafusion.apache.org
 
@@ -44,7 +44,7 @@ Comet also accelerates Apache Iceberg, when performing Parquet scans from Spark.
 
 Comet delivers a performance speedup for many queries, enabling faster data processing and shorter time-to-insights.
 
-The following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in Parquet format 
+The following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in Parquet format
 using a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)
 for details of the environment used for these benchmarks.
 
@@ -66,16 +66,16 @@ The following charts shows how much Comet currently accelerates each query from
 
 ![](docs/source/_static/images/benchmark-results/0.11.0/tpch_queries_speedup_abs.png)
 
-These benchmarks can be reproduced in any environment using the documentation in the 
-[Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage 
+These benchmarks can be reproduced in any environment using the documentation in the
+[Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage
 you to run your own benchmarks.
 
 Results for our benchmark derived from TPC-DS are available in the [benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmark-results/tpc-ds.html).
 
 ## Use Commodity Hardware
 
 Comet leverages commodity hardware, eliminating the need for costly hardware upgrades or
-specialized hardware accelerators, such as GPUs or FPGA. By maximizing the utilization of commodity hardware, Comet 
+specialized hardware accelerators, such as GPUs or FPGA. By maximizing the utilization of commodity hardware, Comet
 ensures cost-effectiveness and scalability for your Spark deployments.
 
 ## Spark Compatibility
@@ -102,7 +102,7 @@ To get started with Apache DataFusion Comet, follow the
 [DataFusion Slack and Discord channels](https://datafusion.apache.org/contributor-guide/communication.html) to connect
 with other users, ask questions, and share your experiences with Comet.
 
-Follow [Apache DataFusion Comet Overview](https://datafusion.apache.org/comet/user-guide/overview.html) to get more detailed information 
+Follow [Apache DataFusion Comet Overview](https://datafusion.apache.org/comet/user-guide/overview.html) to get more detailed information
 
 ## Contributing
 

benchmarks/README.md

@@ -102,4 +102,3 @@ $SPARK_HOME/bin/spark-submit \
     --queries /opt/datafusion-benchmarks/tpcds/queries-spark \
     --iterations 1
 ```
-

common/src/main/scala/org/apache/comet/CometConf.scala

@@ -676,11 +676,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
-  val COMET_EXPR_ALLOW_INCOMPATIBLE: ConfigEntry[Boolean] =
-    conf("spark.comet.expression.allowIncompatible")
+  val COMET_EXEC_STRICT_FLOATING_POINT: ConfigEntry[Boolean] =
+    conf("spark.comet.exec.strictFloatingPoint")
       .category(CATEGORY_EXEC)
-      .doc("Comet is not currently fully compatible with Spark for all expressions. " +
-        s"Set this config to true to allow them anyway. $COMPAT_GUIDE.")
+      .doc(
+        "When enabled, fall back to Spark for floating-point operations that may differ from " +
+          s"Spark, such as when comparing or sorting -0.0 and 0.0. $COMPAT_GUIDE.")
       .booleanConf
       .createWithDefault(false)
 

dev/benchmarks/README.md

@@ -19,7 +19,7 @@ under the License.
 
 # Comet Benchmarking Scripts
 
-This directory contains scripts used for generating benchmark results that are published in this repository and in 
+This directory contains scripts used for generating benchmark results that are published in this repository and in
 the Comet documentation.
 
 For full instructions on running these benchmarks on an EC2 instance, see the [Comet Benchmarking on EC2 Guide].

dev/benchmarks/comet-tpcds.sh

@@ -40,7 +40,7 @@ $SPARK_HOME/bin/spark-submit \
     --conf spark.executor.extraClassPath=$COMET_JAR \
     --conf spark.plugins=org.apache.spark.CometPlugin \
     --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
-    --conf spark.comet.expression.allowIncompatible=true \
+    --conf spark.comet.expression.Cast.allowIncompatible=true \
     --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
     --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain \
     tpcbench.py \

dev/benchmarks/comet-tpch.sh

@@ -41,7 +41,7 @@ $SPARK_HOME/bin/spark-submit \
     --conf spark.plugins=org.apache.spark.CometPlugin \
     --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
     --conf spark.comet.exec.replaceSortMergeJoin=true \
-    --conf spark.comet.expression.allowIncompatible=true \
+    --conf spark.comet.expression.Cast.allowIncompatible=true \
     --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
     --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain \
     tpcbench.py \