Skip to content

feat(templates): add tojson_safe Jinja2 filter (closes #580)#581

Merged
masukai merged 2 commits into
mainfrom
feat/580-tojson-safe-filter
May 28, 2026
Merged

feat(templates): add tojson_safe Jinja2 filter (closes #580)#581
masukai merged 2 commits into
mainfrom
feat/580-tojson-safe-filter

Conversation

@masukai

@masukai masukai commented May 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #580. Add a tojson_safe Jinja2 filter that tolerates datetime / date / time / Decimal / UUID — types that the standard tojson rejects with Object of type datetime is not JSON serializable. Unblocks BigQuery TIMESTAMP and Postgres numeric / uuid columns flowing through REST API body_template rendering without CAST(... AS STRING) workarounds in model SQL.

  • tojson_safe calls json.dumps(value, default=_json_default, ensure_ascii=False) with a small whitelist:
    • datetime / date / time → ISO 8601 (obj.isoformat())
    • Decimal / UUIDstr(obj)
    • anything else → TypeError, matching json.dumps
  • Registered on both drt.templates.renderer and drt.destinations.staged_upload's local Jinja2 envs.
  • The default tojson filter is unchanged — strictly opt-in, no behaviour change for existing templates. A regression test (test_tojson_strict_still_fails_on_datetime) locks this in.

Files

Usage

body_template: |
  {
    "name":     {{ row.name | tojson_safe }},
    "metadata": {{ row | tojson_safe }}
  }

Test plan

  • pytest tests/unit/test_renderer.py — 9 tests pass (2 existing + 7 new)
  • pytest full suite — 1254 passed, 1 skipped
  • ruff check on touched files — clean
  • mypy on touched files — clean
  • make check-i18n — clean
  • Non-breaking guard: existing tojson still rejects datetime

🤖 Generated with Claude Code

Standard `tojson` raises `Object of type datetime is not JSON
serializable` when a row contains `datetime` / `date` / `Decimal` /
`UUID` — common for BigQuery `TIMESTAMP`, Postgres `numeric` / `uuid`.

Add a `tojson_safe` filter that calls `json.dumps(..., default=...,
ensure_ascii=False)` with a fallback that ISO-encodes datetimes and
str-encodes Decimal / UUID. Register it on both `drt.templates.renderer`
and `drt.destinations.staged_upload`'s local Jinja2 environments. Other
types still raise `TypeError`, matching `json.dumps` semantics.

The default `tojson` filter is unchanged — opt-in only, fully
backwards-compatible. Unblocks REST API templates using BigQuery
`TIMESTAMP` columns end-to-end without `CAST(... AS STRING)` workarounds
in model SQL.

- `drt/templates/renderer.py`: `_json_default`, `tojson_safe`,
  `_build_env()` helper.
- `drt/destinations/staged_upload.py`: import + register `tojson_safe`
  in the local `_render` env.
- `tests/unit/test_renderer.py`: 7 new tests (datetime / date / time /
  Decimal / UUID / nested-row / `ensure_ascii=False` / unknown-type
  TypeError / strict `tojson` still fails — non-breaking guard).
- `docs/connectors/rest-api.md`: new "Serializing datetime / Decimal /
  UUID columns" section.
- `CHANGELOG.md`: entry under `[Unreleased]`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment thread drt/templates/renderer.py Fixed
@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…er_template

Drop the `_build_env()` helper introduced in the previous commit and
register the filter inline inside `render_template`, mirroring the
existing pattern in `staged_upload._render`. Functionally identical;
CodeQL was flagging the new `_build_env` location as a "new alert" for
`py/jinja2/autoescape-false`, while the project already tolerates 4
pre-existing open alerts of the same rule (drt templates output
JSON/text, never HTML — the alert is a known false positive). Keeping
the `Environment(...)` call inside `render_template` lets CodeQL track
the alert as the same pre-existing one relocating rather than a brand
new one in changed code.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@masukai masukai merged commit b4f7552 into main May 28, 2026
8 checks passed
@masukai masukai deleted the feat/580-tojson-safe-filter branch May 28, 2026 08:14
@github-actions github-actions Bot locked and limited conversation to collaborators May 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add tojson_safe Jinja2 filter for datetime/Decimal/UUID-safe JSON encoding in REST API body_template

2 participants