Skip to content

[BUG] Source connection_url not long enough for Snowflake connection URLs #1888

@collado-mike

Description

@collado-mike

A common Snowflake connection url, reported e.g., from the OpenLineage Airflow or Spark integrations can include a lot of unnecessary querystring parameters - e.g.,

snowflake://12345.us-east-1.snowflakecomputing.com/TestRole?extra__snowflake__account=12345&extra__snowflake__database=MYDATABASE&extra__snowflake__insecure_mode=false&extra__snowflake__region=us-east-1&extra__snowflake__role=TestRole&extra__snowflake__warehouse=DEMOWH&extra__snowflake__aws_access_key_id=ABCDEFGHIJKLMNOP&extra__snowflake__aws_secret_access_key=ABCDEFGHIJKLMNOP

A URL such as this results in an exception when inserting into Postgres

org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2674)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2364)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:354)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:484)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:404)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:162)
	at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:151)
	at jdk.internal.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:118)
	at jdk.proxy2/jdk.proxy2.$Proxy43.execute(Unknown Source)
	at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:31)
	at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1784)
	... 88 common frames omitted

We should increase the column length for the connection_url column or consider switching to a TEXT field

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions