Error while writing Pandas DataFrame to Delta Lake (S3)

[vinniepsychosis]

write_deltalake isn't working as expected

I encountered an error while calling write_deltalake method:

ValueError : you must provide schema if data is iterable

Even though it worked perfectly with a PyArrow Table, it didn't work for Pandas DataFrame.

How to reproduce it:

# To reproduce it, just try to write a Pandas DataFrame to an S3 Bucket

write_deltalake(
    <your_s3_url>,
    data = df,
    storage_options = storage_options,
    overwrite_schema = True,
    mode = 'overwrite'
)

[ion-elgreco]

@vinamrgrover please provide the deltalake version you're using, also share a minimal reproducible example

[vinniepsychosis]

Version : 0.15.0

I already shared a reproducible example above

[ion-elgreco]

@vinamrgrover a reproducible example should also include a sample dataframe..

[ion-elgreco]

You are likely passing something that is not a pandas dataframe, I can write a pandas dataframe with write_deltalake.

import pandas as pd
from deltalake import write_deltalake
df = pd.DataFrame({'foo':['test']})
write_deltalake('test_table_PATH', data=df, mode='overwrite', overwrite_schema=True)

[vinniepsychosis]

Please don't mark it as completed, I better know I have passed Pandas DataFrame in write_deltalake method. I think the issue is with the _has_pandas variable within writer.py module. Try executing this in a new virtual environment

[topagarwal]

I am also having similar issue. I read data from deltaWriter, converted to pandas dataframe, filtered some data and then writing it back to a different S3 location. When writing it gives me ValueError, schema of Data and Table does not match. I verified by printing schema of both, they are same. Not sure why its not letting me write.

[ion-elgreco]

I am also having similar issue. I read data from deltaWriter, converted to pandas dataframe, filtered some data and then writing it back to a different S3 location. When writing it gives me ValueError, schema of Data and Table does not match. I verified by printing schema of both, they are same. Not sure why its not letting me write.

Which version are you using? Also create an MRE so we can reproduce it

[topagarwal]

I'm using Delta lake v 0.17.0
Here are the steps i did:

1. Read in the DeltaTable from existing S3 location. dt = DeltaTable("s3://mylocation/")
2. Converted it to pyarrow table. arrow_table = dt.to_pyarrow_table()
3. Filtered the arrow table and selected specific columns of interest
4. Converted arrow table to pandas data frame. df = arrow_table.to_pandas()
5. Writing panda dataframe back to existing new delta table. Table is empty at this point.
6. write_deltalake("s3://test_sample_process/", df, mode="overwrite"). also tried it with schema_mode="overwrite"

To trouble shoot it I looked into deltalake/writer.py file. The Exception is thrown form ln 351. It is trying to sort the data schema and table schema and then match them. It is using pyarrow engine.

The output of matching is

python/3.12.7/lib/python3.12/site-packages/deltalake/writer.py", line 351, in write_deltalake
    raise ValueError(
ValueError: Schema of data does not match table schema


Data schema:
namespace: string
ki_record_name: string
work_center: string
kt_config: string
kt_parameters: string
mi_updated_at: timestamp[us, tz=UTC]
mi_updated_by: string

Table Schema:

namespace: string
ki_record_name: string
work_center: string not null
kt_config: string
kt_parameters: string
mi_updated_at: timestamp[us, tz=UTC] not null
  -- field metadata --
  comment: '"The time this record was updated"'
mi_updated_by: string not null
  -- field metadata --
  comment: '"The process that updated this record"'

I couldn't isolate the difference in 2 schemas other than table schema as comment and not null defined. The field names and data types are same for both. Wondering what I am missing here.