Fill run state of the parent run when it is created by child run#2296
Merged
wslulciuc merged 2 commits intoDec 8, 2022
Conversation
Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com>
jonathanpmoraes
referenced
this pull request
in nubank/NuMarquez
Feb 6, 2025
Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com> Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com> Co-authored-by: Willy Lulciuc <willy@datakin.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Minkyu Park minkyu.park.200@gmail.com
Problem
Airflow integration is not sending the run event for the DAG but it is enclosed as a
parentfacet of task runs events, and when the marquez receives OpenLineage event withparentfacet, it creates parent job and run if they don't exist. However, the created parent run is missing its run state, and it makes a little weird when getting the runs for the DAG job. For example when the marquez receives theSTARTevent, the the parent run is being created and its status isRUNNINGbut it does not have astartedAtinformation.Solution
This PR adds a run state to the parent run at its creation time, honoring the way that it determines the state of the parent run. If the parent run is created with the
STARTevent, its run state becomesRUNNINGand it addsstarted_atto the database. If the parent run is created with theCOMPLETEevent, its run state becomesCOMPLETEand it addsended_atto the database.Checklist
CHANGELOG.mdwith details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sqldatabase schema migration according to Flyway's naming convention (if relevant)