Skip to content

Commit bce2ae0

Browse files
authored
docs: update Iceberg docs to reflect capabilities (#3961)
1 parent decd85a commit bce2ae0

2 files changed

Lines changed: 6 additions & 11 deletions

File tree

docs/source/contributor-guide/roadmap.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@ helpful to have a roadmap for some of the major items that require coordination
2626

2727
### Iceberg Integration
2828

29-
Iceberg tables reads are now fully native, powered by a scan operator backed by Iceberg-rust ([#2528]). We anticipate
30-
major improvements expected in the next few releases, including bringing Iceberg table format V3 features (_e.g._,
29+
Reads of Iceberg tables with Parquet data files are fully native and enabled by default, powered by a scan operator
30+
backed by Iceberg-rust ([#2528]). We anticipate major improvements in the next few releases, including bringing Iceberg table format V3 features (_e.g._,
3131
encryption) to the reader.
3232

3333
[#2528]: https://github.com/apache/datafusion-comet/pull/2528

docs/source/user-guide/latest/iceberg.md

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,20 +17,17 @@
1717
under the License.
1818
-->
1919

20-
# Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)
21-
22-
**Note: Iceberg integration is a work-in-progress.**
20+
# Accelerating Apache Iceberg Parquet Scans using Comet
2321

2422
## Native Reader
2523

26-
Comet's fully-native Iceberg integration does not require modifying Iceberg source
27-
code. Instead, Comet relies on reflection to extract `FileScanTask`s from Iceberg, which are
24+
Comet's native Iceberg reader relies on reflection to extract `FileScanTask`s from Iceberg, which are
2825
then serialized to Comet's native execution engine (see
2926
[PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
3027

3128
The example below uses Spark's package downloader to retrieve Comet 0.14.0 and Iceberg
32-
1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, and 1.10. The key configuration
33-
to enable fully-native Iceberg is `spark.comet.scan.icebergNative.enabled=true`.
29+
1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, and 1.10. The native
30+
Iceberg reader is enabled by default.
3431

3532
```shell
3633
$SPARK_HOME/bin/spark-shell \
@@ -43,7 +40,6 @@ $SPARK_HOME/bin/spark-shell \
4340
--conf spark.plugins=org.apache.spark.CometPlugin \
4441
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
4542
--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
46-
--conf spark.comet.scan.icebergNative.enabled=true \
4743
--conf spark.comet.explainFallback.enabled=true \
4844
--conf spark.memory.offHeap.enabled=true \
4945
--conf spark.memory.offHeap.size=2g
@@ -120,7 +116,6 @@ $SPARK_HOME/bin/spark-shell \
120116
--conf spark.plugins=org.apache.spark.CometPlugin \
121117
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
122118
--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
123-
--conf spark.comet.scan.icebergNative.enabled=true \
124119
--conf spark.comet.explainFallback.enabled=true \
125120
--conf spark.memory.offHeap.enabled=true \
126121
--conf spark.memory.offHeap.size=2g

0 commit comments

Comments
 (0)