Environment
Delta-rs version:
deltalake==0.16.0
pyarrow==11.0.0
Binding: Python
Environment: Ubuntu 22.04.4 LTS
Bug
What happened: Merge throws an error when there is an arrow datatype decimal128 present in any of the columns. Small numbers e.g. 35 work. 0 doesn't work, 10**20 doesn't work
What you expected to happen: Execute a merge
How to reproduce it:
import pyarrow as pa
import deltalake as dl
from deltalake import DeltaTable
from pathlib import Path
breaking_number = 0 # also doesnt work with something large like 10**20
# Simplified schema with only one column
schema = pa.schema([("decimalcol", pa.decimal128(38, 0))
, ("key", pa.int32())])
# Create dummy data
data = [{'decimalcol': breaking_number, 'key': 1}]
table = pa.Table.from_pylist(data, schema=schema)
# Define file path
path = Path('testdir')
dl.write_deltalake(path, table, mode="overwrite")
table = pa.Table.from_pylist(data, schema=schema)
existing_table = DeltaTable(path)
(
existing_table.merge(
source=table,
predicate= "s.key = t.key",
source_alias="s",
target_alias="t") # would require a predicate (primary key) to merge
.when_matched_update_all()
.when_not_matched_insert_all()
.execute()
)
More details:
Environment
Delta-rs version:
deltalake==0.16.0
pyarrow==11.0.0
Binding: Python
Environment: Ubuntu 22.04.4 LTS
Bug
What happened: Merge throws an error when there is an arrow datatype decimal128 present in any of the columns. Small numbers e.g. 35 work. 0 doesn't work, 10**20 doesn't work
What you expected to happen: Execute a merge
How to reproduce it:
More details: