Add job tagging to API by davidsharp7 · Pull Request #2774 · MarquezProject/marquez

davidsharp7 · 2024-03-17T09:34:35Z

Problem

Currently we have no tag support for jobs.

Closes: #2756

Solution

Adding new table jobs_tag_mapping to store the relationship between jobs and tags.
Update jobs meta to return the tags as a list.
Add POST/DELETE end points to add and delete tags.

One-line summary:

Add job tagging to API

Checklist

You've signed-off your work
Your changes are accompanied by tests (if relevant)
Your change contains a small diff and is self-contained
You've updated any relevant documentation (if relevant)
You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
You've included a header in any source code files (if relevant)

Signed-off-by: sharpd <davidsharp7@gmail.com>

netlify · 2024-03-17T09:34:50Z

✅ Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
🔨 Latest commit	`bf6621e`
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/663a4ce80f1807000836ffb4

Signed-off-by: sharpd <davidsharp7@gmail.com>

codecov · 2024-03-19T21:13:12Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.57%. Comparing base (ae794a9) to head (10703c1).

❗ Current head 10703c1 differs from pull request most recent head bf6621e. Consider uploading reports for the commit bf6621e to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2774      +/-   ##
============================================
+ Coverage     84.47%   84.57%   +0.09%     
- Complexity     1430     1441      +11     
============================================
  Files           251      251              
  Lines          6462     6503      +41     
  Branches        299      302       +3     
============================================
+ Hits           5459     5500      +41     
  Misses          850      850              
  Partials        153      153

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: sharpd <davidsharp7@gmail.com>

phixMe · 2024-04-23T16:44:43Z

+      NOW(),
+      NOW(),
+      :tagName,
+      'No Description'


Is there a reason we would want to hardcode this in the SQL, instead of in our application code to be a bit more flexible?

description is a nullable column from what I can see, I'm not sure we need to default it at all?

Yeah I can leave as null - was just being explicit more than anything but it is inconsistent with dataset/dataset field tagging so makes sense to set to null.

davidjgoss · 2024-04-23T16:42:37Z

+    Job job =
+        jobService
+            .findJobByName(namespaceName.getValue(), jobName.getValue())
+            .orElseThrow(() -> new JobNotFoundException(jobName));


I guess this should be quite unlikely given updateJobTags would presumably throw if the job didn't exist? A case of, we get an optional back so we should do something beyond an unqualified get() on it?

Yeah understand what you mean -it will fall over before it gets to this point so why bother? Mainly as that seems to be the de-facto pattern for a lot of the code i.e

execute something -> retrieve object (job, dataset etc) else throw an error.

Thanks for following our code style!

davidjgoss · 2024-04-23T16:55:16Z

  @Getter private final JobType type;
  @Getter private final Set<DatasetId> inputs;
  @Getter private final Set<DatasetId> outputs;
+  @Getter @NonNull private final Set<String> tags;


Disparity between @NonNull here and @Nullable in the constructor. The builder also doesn't specify - I think nullable makes sense though? For backwards-compatibility, especially.

davidjgoss · 2024-04-23T16:57:00Z

  @Getter private final ImmutableMap<String, Object> facets;
  @Nullable private UUID currentVersion;
  @Getter @Nullable private ImmutableList<String> labels;
+  @Getter private final ImmutableSet<TagName> tags;


Should we mark this @Nullable here? Would be consistent with other fields.

davidjgoss · 2024-04-23T16:59:27Z

  @Nullable private final Run latestRun;
  @Getter private final Map<String, Object> facets;
  @Nullable private final UUID currentVersion;
+  @Getter @NonNull private final Set<String> tags;


This class extends JobMeta which already defines tags as a field - I don't think we need to define it again here, but rather pass it through into the super ctor?

Signed-off-by: sharpd <davidsharp7@gmail.com>

…quez into api/add_job_tags

Signed-off-by: sharpd <davidsharp7@gmail.com>

davidjgoss

LGTM. Not sure why webpack build is failing, as it runs okay for me locally. Could be it's maxing out resources?

davidsharp7 · 2024-04-25T17:45:56Z

It's a 137 exit code which is OOM so yes you out of resource. Did this a couple of weeks ago and then came right.

Signed-off-by: sharpd <davidsharp7@gmail.com>

wslulciuc · 2024-05-07T02:22:10Z

+    Job job =
+        jobService
+            .findJobByName(namespaceName.getValue(), jobName.getValue())
+            .orElseThrow(() -> new JobNotFoundException(jobName));


Thanks for following our code style!

Signed-off-by: sharpd <davidsharp7@gmail.com>

wslulciuc

Excited to see tagging support for jobs! (And soon in the Web UI 🚀 )

* add job tagging to API Signed-off-by: sharpd <davidsharp7@gmail.com> * fix various tests that were failing Signed-off-by: sharpd <davidsharp7@gmail.com> * fix merge commit Signed-off-by: sharpd <davidsharp7@gmail.com> * fix liniting Signed-off-by: sharpd <davidsharp7@gmail.com> * update based on pr feedback Signed-off-by: sharpd <davidsharp7@gmail.com> * lint tag test code Signed-off-by: sharpd <davidsharp7@gmail.com> * lint tag code Signed-off-by: sharpd <davidsharp7@gmail.com> * fix log typo Signed-off-by: sharpd <davidsharp7@gmail.com> * fix logging Signed-off-by: sharpd <davidsharp7@gmail.com> * update based on PR feedback. Signed-off-by: sharpd <davidsharp7@gmail.com> * correct db field error Signed-off-by: sharpd <davidsharp7@gmail.com> --------- Signed-off-by: sharpd <davidsharp7@gmail.com> Co-authored-by: Willy Lulciuc <willy@datakin.com>

add job tagging to API

0d44966

Signed-off-by: sharpd <davidsharp7@gmail.com>

boring-cyborg Bot added api API layer changes client/java labels Mar 17, 2024

fix various tests that were failing

d114c3f

Signed-off-by: sharpd <davidsharp7@gmail.com>

Merge branch 'MarquezProject:main' into api/add_job_tags

fd76893

davidsharp7 marked this pull request as ready for review March 19, 2024 22:51

davidsharp7 and others added 7 commits March 21, 2024 18:32

Merge branch 'MarquezProject:main' into api/add_job_tags

655bf3a

Merge branch 'MarquezProject:main' into api/add_job_tags

02e7484

Merge branch 'main' into api/add_job_tags

58f40cf

fix merge commit

1d3f819

Signed-off-by: sharpd <davidsharp7@gmail.com>

fix liniting

057bf14

Signed-off-by: sharpd <davidsharp7@gmail.com>

Merge branch 'MarquezProject:main' into api/add_job_tags

4d1550a

Merge branch 'MarquezProject:main' into api/add_job_tags

da04b41

phixMe reviewed Apr 23, 2024

View reviewed changes

davidjgoss reviewed Apr 23, 2024

View reviewed changes

phixMe reviewed Apr 23, 2024

View reviewed changes

Comment thread api/src/test/java/marquez/api/TagResourceIntegrationTest.java Outdated

davidsharp7 and others added 6 commits April 24, 2024 06:25

Merge branch 'main' into api/add_job_tags

3f429f9

update based on pr feedback

911549b

Signed-off-by: sharpd <davidsharp7@gmail.com>

Merge branch 'api/add_job_tags' of https://github.com/davidsharp7/mar…

2750d66

…quez into api/add_job_tags

lint tag test code

2b8ae9a

Signed-off-by: sharpd <davidsharp7@gmail.com>

lint tag code

b73c6b3

Signed-off-by: sharpd <davidsharp7@gmail.com>

fix log typo

ca332d6

Signed-off-by: sharpd <davidsharp7@gmail.com>

davidjgoss reviewed Apr 25, 2024

View reviewed changes

davidsharp7 and others added 4 commits April 26, 2024 16:58

fix logging

322ed34

Signed-off-by: sharpd <davidsharp7@gmail.com>

Merge branch 'MarquezProject:main' into api/add_job_tags

d127354

Merge branch 'MarquezProject:main' into api/add_job_tags

a64c2f6

Merge branch 'MarquezProject:main' into api/add_job_tags

4517062

Merge branch 'MarquezProject:main' into api/add_job_tags

f726170

davidsharp7 self-assigned this May 2, 2024

MarquezProject locked and limited conversation to collaborators May 3, 2024

MarquezProject unlocked this conversation May 3, 2024

davidsharp7 requested a review from wslulciuc May 5, 2024 05:32

wslulciuc reviewed May 7, 2024

View reviewed changes

davidsharp7 and others added 3 commits May 7, 2024 15:04

Merge branch 'MarquezProject:main' into api/add_job_tags

4065cb5

update based on PR feedback.

4c18168

Signed-off-by: sharpd <davidsharp7@gmail.com>

correct db field error

10703c1

Signed-off-by: sharpd <davidsharp7@gmail.com>

wslulciuc approved these changes May 7, 2024

View reviewed changes

Merge branch 'main' into api/add_job_tags

bf6621e

wslulciuc added this to the 0.47.0 milestone May 7, 2024

wslulciuc enabled auto-merge (squash) May 7, 2024 15:55

wslulciuc merged commit 00b6d35 into MarquezProject:main May 7, 2024

Conversation

davidsharp7 commented Mar 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Checklist

Uh oh!

netlify Bot commented Mar 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for peppy-sprite-186812 canceled.

Uh oh!

codecov Bot commented Mar 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidsharp7 Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidsharp7 Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidjgoss left a comment

Choose a reason for hiding this comment

Uh oh!

davidsharp7 commented Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wslulciuc left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davidsharp7 commented Mar 17, 2024 •

edited

Loading

netlify Bot commented Mar 17, 2024 •

edited

Loading

codecov Bot commented Mar 19, 2024 •

edited

Loading

davidsharp7 Apr 24, 2024 •

edited

Loading

davidsharp7 Apr 25, 2024 •

edited

Loading

davidsharp7 commented Apr 25, 2024 •

edited

Loading