Error: value too long for type character varying(255)

[prachim-collab]

Error in Db migration script.

Message    : ERROR: value too long for type character varying(255)
Location   : marquez/db/migration/V48__dataset_symlinks.sql (/usr/src/app/file:/usr/src/app/datakin-api.jar!/marquez/db/migration/V48__dataset_symlinks.sql)
Line       : 16
Statement  : INSERT INTO dataset_symlinks (dataset_uuid, name, namespace_uuid, is_primary, created_at, updated_at)
SELECT d.uuid, d.name, d.namespace_uuid, TRUE, d.created_at, d.updated_at FROM datasets d

V48__dataset_symlinks.sql Db migration script is throwing error while it inserts data in dataset_symlinks table

[boring-cyborg]

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

[pawel-big-lebowski]

@prachim-collab You're right 😢 This happens because of:

marquez/api/src/main/resources/marquez/db/migration/V40__alter_tables_resize_namespace_and_connection_url.sql

Line 3 in db92c4f

ALTER TABLE datasets ALTER COLUMN name TYPE VARCHAR;

The solution to this would be:

• modify V48__dataset_symlinks.sql and change column type to varchar for those who failed on migration,
• create extra migration step to alter column for those who succeeded migration so that they can have longer names in future.

[prachim-collab]

@prachim-collab You're right 😢 This happens because of:

marquez/api/src/main/resources/marquez/db/migration/V40__alter_tables_resize_namespace_and_connection_url.sql

Line 3 in db92c4f

ALTER TABLE datasets ALTER COLUMN name TYPE VARCHAR;

The solution to this would be:

• modify V48__dataset_symlinks.sql and change column type to varchar for those who failed on migration,

The change to varchar should take effect for all old and new rows automatically.

• create extra migration step to alter column for those who succeeded migration so that they can have longer names in future.

The insert statement in V48__dataset_symlinks.sql would have inserted partial data successfully so we also need to modify the insert to fail gracefully on duplicate keys and continue inserting missing data.