A common Snowflake connection url, reported e.g., from the OpenLineage Airflow or Spark integrations can include a lot of unnecessary querystring parameters - e.g.,
snowflake://12345.us-east-1.snowflakecomputing.com/TestRole?extra__snowflake__account=12345&extra__snowflake__database=MYDATABASE&extra__snowflake__insecure_mode=false&extra__snowflake__region=us-east-1&extra__snowflake__role=TestRole&extra__snowflake__warehouse=DEMOWH&extra__snowflake__aws_access_key_id=ABCDEFGHIJKLMNOP&extra__snowflake__aws_secret_access_key=ABCDEFGHIJKLMNOP
A URL such as this results in an exception when inserting into Postgres
org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2674)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2364)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:354)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:484)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:404)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:162)
at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:151)
at jdk.internal.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:118)
at jdk.proxy2/jdk.proxy2.$Proxy43.execute(Unknown Source)
at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:31)
at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1784)
... 88 common frames omitted
We should increase the column length for the connection_url column or consider switching to a TEXT field
A common Snowflake connection url, reported e.g., from the OpenLineage Airflow or Spark integrations can include a lot of unnecessary querystring parameters - e.g.,
A URL such as this results in an exception when inserting into Postgres
We should increase the column length for the
connection_urlcolumn or consider switching to aTEXTfield