feat(templates): add tojson_safe Jinja2 filter (closes #580)#581
Merged
Conversation
Standard `tojson` raises `Object of type datetime is not JSON serializable` when a row contains `datetime` / `date` / `Decimal` / `UUID` — common for BigQuery `TIMESTAMP`, Postgres `numeric` / `uuid`. Add a `tojson_safe` filter that calls `json.dumps(..., default=..., ensure_ascii=False)` with a fallback that ISO-encodes datetimes and str-encodes Decimal / UUID. Register it on both `drt.templates.renderer` and `drt.destinations.staged_upload`'s local Jinja2 environments. Other types still raise `TypeError`, matching `json.dumps` semantics. The default `tojson` filter is unchanged — opt-in only, fully backwards-compatible. Unblocks REST API templates using BigQuery `TIMESTAMP` columns end-to-end without `CAST(... AS STRING)` workarounds in model SQL. - `drt/templates/renderer.py`: `_json_default`, `tojson_safe`, `_build_env()` helper. - `drt/destinations/staged_upload.py`: import + register `tojson_safe` in the local `_render` env. - `tests/unit/test_renderer.py`: 7 new tests (datetime / date / time / Decimal / UUID / nested-row / `ensure_ascii=False` / unknown-type TypeError / strict `tojson` still fails — non-breaking guard). - `docs/connectors/rest-api.md`: new "Serializing datetime / Decimal / UUID columns" section. - `CHANGELOG.md`: entry under `[Unreleased]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…er_template Drop the `_build_env()` helper introduced in the previous commit and register the filter inline inside `render_template`, mirroring the existing pattern in `staged_upload._render`. Functionally identical; CodeQL was flagging the new `_build_env` location as a "new alert" for `py/jinja2/autoescape-false`, while the project already tolerates 4 pre-existing open alerts of the same rule (drt templates output JSON/text, never HTML — the alert is a known false positive). Keeping the `Environment(...)` call inside `render_template` lets CodeQL track the alert as the same pre-existing one relocating rather than a brand new one in changed code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #580. Add a
tojson_safeJinja2 filter that toleratesdatetime/date/time/Decimal/UUID— types that the standardtojsonrejects withObject of type datetime is not JSON serializable. Unblocks BigQueryTIMESTAMPand Postgresnumeric/uuidcolumns flowing through REST APIbody_templaterendering withoutCAST(... AS STRING)workarounds in model SQL.tojson_safecallsjson.dumps(value, default=_json_default, ensure_ascii=False)with a small whitelist:datetime/date/time→ ISO 8601 (obj.isoformat())Decimal/UUID→str(obj)TypeError, matchingjson.dumpsdrt.templates.rendereranddrt.destinations.staged_upload's local Jinja2 envs.tojsonfilter is unchanged — strictly opt-in, no behaviour change for existing templates. A regression test (test_tojson_strict_still_fails_on_datetime) locks this in.Files
_json_default,tojson_safe,_build_env()helper_render[Unreleased]entryUsage
Test plan
pytest tests/unit/test_renderer.py— 9 tests pass (2 existing + 7 new)pytestfull suite — 1254 passed, 1 skippedruff checkon touched files — cleanmypyon touched files — cleanmake check-i18n— cleantojsonstill rejectsdatetime🤖 Generated with Claude Code