Skip to content

Flaky arrow_table_all_data_types: decimal column has unstable arrow precision/scale #3918

@burnash

Description

@burnash

tests/cases.py::arrow_table_all_data_types generates the decimal column from random.uniform(0, 100), which lets PyArrow infer a different decimal128(precision, scale) on each call. Two independent calls to the fixture can therefore produce two arrow tables with non-equal schemas.

Failed CI: https://github.com/dlt-hub/dlt/actions/runs/25451561231/job/74781487637

E       OSError: LanceError(Arrow): C Data interface error: Invalid: Decimal value does not fit in precision 5. Detail: Python exception: Traceback (most recent call last):
E         File "/home/runner/work/dlt/dlt/dlt/destinations/impl/lance/utils.py", line 48, in <genexpr>
E           return pa.RecordBatchReader.from_batches(cast_schema, (b.cast(cast_schema) for b in source))
E                                                                  ^^^^^^^^^^^^^^^^^^^
E         File "pyarrow/table.pxi", line 3358, in pyarrow.lib.RecordBatch.cast
E         File "pyarrow/array.pxi", line 1161, in pyarrow.lib.Array.cast
E         File "/home/runner/work/dlt/dlt/.venv/lib/python3.11/site-packages/pyarrow/compute.py", line 414, in cast
E           return call_function("cast", [arr], options, memory_pool)
E                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E         File "pyarrow/_compute.pyx", line 604, in pyarrow._compute.call_function
E         File "pyarrow/_compute.pyx", line 399, in pyarrow._compute.Function.call
E         File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
E         File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
E       pyarrow.lib.ArrowInvalid: Decimal value does not fit in precision 5
E       , /home/runner/work/lance/lance/rust/lance-datafusion/src/utils.rs:51:31

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdestinationIssue with a specific destination

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions