Skip to content

Fill run state of the parent run when it is created by child run#2296

Merged
wslulciuc merged 2 commits into
MarquezProject:mainfrom
fm100:bugfix/parent-run-start-time
Dec 8, 2022
Merged

Fill run state of the parent run when it is created by child run#2296
wslulciuc merged 2 commits into
MarquezProject:mainfrom
fm100:bugfix/parent-run-start-time

Conversation

@fm100

@fm100 fm100 commented Dec 8, 2022

Copy link
Copy Markdown
Collaborator

Signed-off-by: Minkyu Park minkyu.park.200@gmail.com

Problem

Airflow integration is not sending the run event for the DAG but it is enclosed as a parent facet of task runs events, and when the marquez receives OpenLineage event with parent facet, it creates parent job and run if they don't exist. However, the created parent run is missing its run state, and it makes a little weird when getting the runs for the DAG job. For example when the marquez receives the START event, the the parent run is being created and its status is RUNNING but it does not have a startedAt information.

Solution

This PR adds a run state to the parent run at its creation time, honoring the way that it determines the state of the parent run. If the parent run is created with the START event, its run state becomes RUNNING and it adds started_at to the database. If the parent run is created with the COMPLETE event, its run state becomes COMPLETE and it adds ended_at to the database.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com>
@fm100 fm100 requested a review from wslulciuc December 8, 2022 00:44
@boring-cyborg boring-cyborg Bot added the api API layer changes label Dec 8, 2022

@wslulciuc wslulciuc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 💯 🥇

@wslulciuc wslulciuc enabled auto-merge (squash) December 8, 2022 10:49
@wslulciuc wslulciuc merged commit 1bbcb6f into MarquezProject:main Dec 8, 2022
jonathanpmoraes referenced this pull request in nubank/NuMarquez Feb 6, 2025
Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com>

Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com>
Co-authored-by: Willy Lulciuc <willy@datakin.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api API layer changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants