Error when sending lineage from Databricks to Marquez

[ganeshnikumbh]

Hi ,

I am getting below error when sending the lineage data from databricks to marquez. I am reading a parquet file in azure adls and writing back into parquet in azure adls. I have also attached the lineage output from databricks. I found the issues is in api/src/main/java/marquez/common/models/NamespaceName.java file, In the regular expression we need to add the '@' symbol as well. Can you please fix this and rebuild the docker images?

error:

marquez-api | ERROR [2021-12-06 12:08:16,460] marquez.api.OpenLineageResource: Unexpected error while processing request
marquez-api | ! java.lang.IllegalArgumentException: namespace 'abfss://landingzone@.dfs.core.windows.net' must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), dashes (-), colons (:), slashes (/) or dots (.) with a maximum length of 1024 characters.
marquez-api | ! at com.google.common.base.Preconditions.checkArgument(Preconditions.java:407)
marquez-api | ! at marquez.common.models.NamespaceName.(NamespaceName.java:41)
marquez-api | ! at marquez.common.models.NamespaceName.of(NamespaceName.java:52)
marquez-api | ! at marquez.db.OpenLineageDao.toDatasetId(OpenLineageDao.java:270)
marquez-api | ! at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)
marquez-api | ! at org.jdbi.v3.core.internal.exceptions.Unchecked.lambda$function$4(Unchecked.java:76)
marquez-api | ! at org.jdbi.v3.sqlobject.DefaultMethodHandler.invoke(DefaultMethodHandler.java:50)
marquez-api | ! at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.lambda$invoke$0(SqlObjectInitData.java:132)
marquez-api | ! at org.jdbi.v3.core.internal.Invocations.invokeWith(Invocations.java:44)
marquez-api | ! at org.jdbi.v3.core.internal.Invocations.invokeWith(Invocations.java:26)
marquez-api | ! at org.jdbi.v3.core.LazyHandleSupplier.lambda$invokeInContext$1(LazyHandleSupplier.java:77)
marquez-api | ! at org.jdbi.v3.core.internal.Invocations.invokeWith(Invocations.java:44)
marquez-api | ! at org.jdbi.v3.core.internal.Invocations.invokeWith(Invocations.java:26)
marquez-api | ! at org.jdbi.v3.core.LazyHandleSupplier.invokeInContext(LazyHandleSupplier.java:76)
marquez-api | ! at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.call(SqlObjectInitData.java:138)
marquez-api | ! at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.invoke(SqlObjectInitData.java:132)
marquez-api | ! at org.jdbi.v3.sqlobject.SqlObjectFactory.lambda$attach$2(SqlObjectFactory.java:110)
marquez-api | ! at com.sun.proxy.$Proxy123.toDatasetId(Unknown Source)
marquez-api | ! at marquez.db.OpenLineageDao.updateBaseMarquezModel(OpenLineageDao.java:133)

[wslulciuc]

Fixed #1925

[gage-russell]

Why was this closed? --

#1925 addresses maximum length of 1024 characters.

But the issue is that@ is not included in the regex check for namespace.