Skip to content

Error when reading delta table with IDENTITY column #2152

@Ghodot

Description

@Ghodot

Environment

Delta-rs version: 0.15.1

Binding: Python 3.8

Environment: Local

  • Cloud provider: Azure
  • OS: Linux
  • Other: External table mounted on adls gen 2 mount

Bug

What happened:
I'm trying to read an external table containing an IDENTITY column. However I encounter the following error when running locally :

    dt = DeltaTable(path)
  File "/home/***/Documents/***/venv/lib/python3.8/site-packages/deltalake/table.py", line 396, in __init__
    self._table = RawDeltaTable(
_internal.DeltaError: Delta protocol violation: Kernel: Invalid url: data did not match any variant of untagged enum MetadataValue at line 1 column 181

If I recreate the table without the Identity column, it works correctly

What you expected to happen:
The table is read correctly in python

How to reproduce it:
Create a table with an identity :

CREATE TABLE spark_catalog.data_model.d_dates (
  ID_D_DATE BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),
  TXT_DateKey STRING
) USING delta LOCATION 'dbfs:/mnt/adls_gen2_mount/d_dates' TBLPROPERTIES ('delta.minReaderVersion' = '1','delta.minWriterVersion' = '3')

Mount the adls endpoint locally or copy the folder to your local computer.
Read the folder using DeltaTable(local_path)

More details:
I tried changing the minWriterVersion / switching between GENERATED ALWAYS and GENERATED BY DEFAULT but none worked.
Except for local access, the table works just fine when working with SQL or spark.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions