Skip to content

Merge breaks using logical datatype decimal128 #2343

@Tommel71

Description

@Tommel71

Environment

Delta-rs version:
deltalake==0.16.0
pyarrow==11.0.0

Binding: Python

Environment: Ubuntu 22.04.4 LTS


Bug

What happened: Merge throws an error when there is an arrow datatype decimal128 present in any of the columns. Small numbers e.g. 35 work. 0 doesn't work, 10**20 doesn't work

What you expected to happen: Execute a merge

How to reproduce it:

import pyarrow as pa
import deltalake as dl
from deltalake import DeltaTable
from pathlib import Path

breaking_number = 0 # also doesnt work with something large like 10**20

# Simplified schema with only one column
schema = pa.schema([("decimalcol", pa.decimal128(38, 0))
                    , ("key", pa.int32())])
# Create dummy data
data = [{'decimalcol': breaking_number, 'key': 1}]
table = pa.Table.from_pylist(data, schema=schema)
# Define file path
path = Path('testdir')

dl.write_deltalake(path, table, mode="overwrite")


table = pa.Table.from_pylist(data, schema=schema)

existing_table = DeltaTable(path)

(
    existing_table.merge(
        source=table,
        predicate= "s.key = t.key",
        source_alias="s",
        target_alias="t")  # would require a predicate (primary key) to merge
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute()
)

More details:

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions