Feat/job to job lineage

[yanlibert]

Problem

Currently, Marquez displays the parent jobs and their child jobs independently with no lineage connection. For exemple, a Airflow task initiated in an Airflow DAG is not visually linked in the UI to its parent, although the information is present in the ParentRunFacet.

Closes: #2484

Solution / Open Questions

This solution mainly serves as the trigger of a larger discussion towards full parent/child hierarchy handling in Marquez. Adding a Job to Job representation in the lineage could be a great step toward a retrieving a more complete picture of the data landscape in general.

For now, the proposed modifications only focuses on the backend API, adding the Job UUID along with the parent name to the job metadata returned.

This could be used by the Front End to display the parent node as a short-term solution.

If we want to go further, this work leads to implementation questions:

• Should the lineage query also return the parent jobs?
• Similar to jobs inputs and outputs, should the parents and children appear in the Job model?
• How to model the hierarchy in the Front? A simple node to node relationship like jobs and datasets, or another graphical representation?

One-line summary:

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[netlify]

✅ Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
🔨 Latest commit	b430377
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/65d5e65f2d69c5000871e23f

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (a9b0b3e) 84.45% compared to head (b430377) 84.42%.

Files	Patch %	Lines
.../src/main/java/marquez/service/LineageService.java	33.33%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2752      +/-   ##
============================================
- Coverage     84.45%   84.42%   -0.04%     
- Complexity     1416     1417       +1     
============================================
  Files           251      251              
  Lines          6447     6458      +11     
  Branches        291      291              
============================================
+ Hits           5445     5452       +7     
- Misses          850      854       +4     
  Partials        152      152              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wslulciuc]

Can't hide that I'm late getting a review in for this, @yanlibert, but better late than never! Great work! Marquez support for job-to-job relationships has always been on our radar; this just gets us closer to making the lineage query changes to make it happen.