Skip to content

Serialize :date column type in cursor to ISO8601 string format#652

Merged
Mangara merged 3 commits into
Shopify:mainfrom
afuno:feature/add_date_in_enumerators
Dec 4, 2025
Merged

Serialize :date column type in cursor to ISO8601 string format#652
Mangara merged 3 commits into
Shopify:mainfrom
afuno:feature/add_date_in_enumerators

Conversation

@afuno

@afuno afuno commented Dec 2, 2025

Copy link
Copy Markdown
Contributor

Problem

When using job-iteration with ActiveRecord models that have a :date column in their primary key or as a cursor column, the cursor value is not serialized to a string. This causes issues when:

  1. Using Sidekiq with strict_args! mode enabled (recommended best practice).
  2. Using composite primary keys that include a date column (common in partitioned tables).
  3. Using maintenance_tasks gem which relies on job-iteration internally.

The error message users see:

Job arguments to MaintenanceTasks::TaskJob must be native JSON types, but Fri, 03 Oct 2025 is a Date.

Root Cause

The column_value method in both ActiveRecordEnumerator and ActiveRecordBatchEnumerator handles :datetime type columns by converting them to strings, but does not handle :date type columns:

ActiveRecordEnumerator (checks column type):

when :datetime
  value.strftime(SQL_DATETIME_WITH_NSEC) # serialized
else
  value # returns Date object

ActiveRecordBatchEnumerator (checks value class):

return value unless value.is_a?(Time)

Solution

Add :date type handling in both enumerators, serializing Date values to ISO8601 format ("2025-10-03"):

  • ActiveRecordEnumerator#column_value: Added when :date case.
  • ActiveRecordBatchEnumerator#column_value: Added when Date case.

The ISO8601 format was chosen because:

  1. It's JSON-serializable (a string).
  2. PostgreSQL/MySQL correctly compare ISO8601 date strings with date columns.
  3. It's consistent with the existing datetime serialization approach.

Testing

Added tests that verify:

  1. Date columns are serialized to ISO8601 format in cursor.
  2. Cursor with date value can be used to resume iteration.

Backward Compatibility

This change is fully backward compatible. Existing cursors with datetime columns continue to work as before. New cursors with date columns will now work correctly.

- Enhanced `column_value` method to serialize `Date` objects as ISO8601 strings.
- Added handling for `:date` column type in `active_record_enumerator`.
- Introduced new tests covering serialization of `Date` values and cursor resumption.
- Updated test setup to include `Event` model and its related schema.
@afuno afuno marked this pull request as ready for review December 2, 2025 09:35

@Mangara Mangara left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, it looks good! Can you add a change log entry under "unreleased"?

- Added entry referencing PR Shopify#652 that resolves ISO8601 serialization issue for `Date` columns in ActiveRecord enumerators.
@afuno

afuno commented Dec 2, 2025

Copy link
Copy Markdown
Contributor Author

Thanks for the contribution, it looks good! Can you add a change log entry under "unreleased"?

Done, added information about this to the changelog.

@afuno

afuno commented Dec 4, 2025

Copy link
Copy Markdown
Contributor Author

@Mangara Hi. Anything else needed for CI and PR merge?

@Mangara Mangara merged commit e1dff0a into Shopify:main Dec 4, 2025
41 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants