Skip to content

Add parentRun alias to LineageEvent RunFacet to support older OpenLin…#2130

Merged
collado-mike merged 2 commits into
mainfrom
fix/openlineage_parent_run_alias
Sep 21, 2022
Merged

Add parentRun alias to LineageEvent RunFacet to support older OpenLin…#2130
collado-mike merged 2 commits into
mainfrom
fix/openlineage_parent_run_alias

Conversation

@collado-mike

Copy link
Copy Markdown
Collaborator

Problem

Prior to OpenLineage/OpenLineage#1037 , all Airflow OpenLineage events reported their parent runs as parentRun rather than the spec-defined parent facet. This change adds an alias to support those older events.

Unfortunately, jackson seems to prefer the alias field name rather than the target (or maybe it's ordered on some condition I don't understand), so if an event defines both, the parentRun facet will be used. This is confirmed in a unit test. The OpenLineage integration does use the same value for both parentRun and parent facets, so this should not cause any issues.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

@boring-cyborg boring-cyborg Bot added api API layer changes docs labels Sep 21, 2022
@collado-mike collado-mike requested a review from fm100 September 21, 2022 19:00
…eage events

Signed-off-by: Michael Collado <collado.mike@gmail.com>
Signed-off-by: Michael Collado <collado.mike@gmail.com>
@collado-mike collado-mike force-pushed the fix/openlineage_parent_run_alias branch from 6490f57 to d083146 Compare September 21, 2022 19:01

@fm100 fm100 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@collado-mike collado-mike enabled auto-merge (squash) September 21, 2022 19:32
@collado-mike collado-mike merged commit 0aaa489 into main Sep 21, 2022
@collado-mike collado-mike deleted the fix/openlineage_parent_run_alias branch September 21, 2022 19:34
jonathanpmoraes referenced this pull request in nubank/NuMarquez Feb 6, 2025
…… (#2130)

* Add parentRun alias to LineageEvent RunFacet to support older OpenLineage events

Signed-off-by: Michael Collado <collado.mike@gmail.com>

* Add changelog update

Signed-off-by: Michael Collado <collado.mike@gmail.com>

Signed-off-by: Michael Collado <collado.mike@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api API layer changes docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants