Datafusion error: External error: Failed to get a credential from UnityCatalog client configuration.

[djouallah]

Environment

Delta-rs version: 0.25

Binding: Python

Environment:

• OS: Windows 10

---

Bug

trying to read a table from UC using azure databricks

import os
from  deltalake import DeltaTable
os.environ["DATABRICKS_WORKSPACE_URL"] = endpoint
os.environ["DATABRICKS_ACCESS_TOKEN"]  = token
os.environ["UNITY_ALLOW_HTTP_URL"]     = "true"
dt = DeltaTable("uc://databricks_ne.aemo.xx")

got this error :

DeltaError: Generic error: Datafusion error: External error: Failed to get a credential from UnityCatalog client configuration.

[ion-elgreco]

cc @omkar-foss @hntd187

[ion-elgreco]

@djouallah have you tried passing the config through storage_options?

[hntd187]

If you are in azure, don't allow http urls. What kind of token are you providing? A personal access token?

[djouallah]

Yes PAT, btw credentials vending works fine with daft

[omkar-foss]

DeltaError: Generic error: Datafusion error: External error: Failed to get a credential from UnityCatalog client configuration.

Hello, this seems to be happening because get_credentials() is returning None and thus causing an error (in code here).

Correct me if I'm wrong - I suppose we may need to enable the "azure" feature for catalog-unity crate here so that the right azure credentials function gets called. (Unable to confirm this as I don't have an Azure Databricks account)

[ion-elgreco]

@omkar-foss I don't see any features explicitly set in the python crate for UC, and the uc crate default is only to use aws

[omkar-foss]

@omkar-foss I don't see any features explicitly set in the python crate for UC, and the uc crate default is only to use aws

Yes that's correct @ion-elgreco, so what I'm thinking is will it help if we add azure also to the defaults list for uc crate. Because right now on my local I see None returned for azure credentials when azure is not added to default features, works fine when added.

[omkar-foss]

take

[omkar-foss]

Yes PAT, btw credentials vending works fine with daft

Right, I've tested this out with Azure Databricks. Tried both delta-rs and daft with same table, daft works fine, delta-rs fails as the azure credentials don't get read (can see this warning in logs as well).

PR fixes this. I've tested this PR with Azure Databricks. @djouallah this should help resolve the issue :)

Note for Azure Databricks + delta-rs: Ensure deletionVectors feature is disabled for your delta table, or else delta-rs fails to load the table. Because Azure Databricks may auto-enable deletion vectors for your workspace.