Use io.openlineage.server.* pkg and add class Metadata#2853
Merged
Conversation
Signed-off-by: wslulciuc <willy@datakin.com>
✅ Deploy Preview for peppy-sprite-186812 canceled.
|
phixMe
reviewed
Jul 12, 2024
| @Builder | ||
| @ToString | ||
| @EqualsAndHashCode | ||
| public static final class Dataset { |
Member
There was a problem hiding this comment.
I think it's worth including column lineage as well as the integration type.
Member
Author
There was a problem hiding this comment.
Given this PR has been out for a while, and the Metadata class isn't yet used within the codebase, I'll add column lineage facet extraction as follow up.
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
…`, and more Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Member
Author
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2853 +/- ##
============================================
- Coverage 83.08% 82.15% -0.93%
- Complexity 1500 1504 +4
============================================
Files 265 268 +3
Lines 6888 7258 +370
Branches 320 325 +5
============================================
+ Hits 5723 5963 +240
- Misses 1007 1134 +127
- Partials 158 161 +3 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
…sOnly()` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
phixMe
approved these changes
Oct 9, 2024
phixMe
left a comment
phixMe
left a comment
Member
There was a problem hiding this comment.
Great work! This is going to improve the write path and improve interacting with and enhancing the core events with updates to the OL Spec.
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
jonathanpmoraes
referenced
this pull request
in nubank/NuMarquez
Feb 6, 2025
* Use `io.openlineage.server.*` pkg and add class `Metadata` Signed-off-by: wslulciuc <willy@datakin.com> * Add back `models.BaseEvent` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add `MetadataTest.testNewRun()` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add `testNewRunWithParent()`, `testNewRunWithNominalStartAndEndTime()`, and more Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add `Metadata.Run.ParentRun.Job.Id` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add `testNewRunWithIO()` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Assert IO dataset name and source Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add `testNewRunWithInputDatasetsOnly()`, `testNewRunWithOutputDatasetsOnly()` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Define `Metadata.Job.location` as `URL` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add assertions for facets and improved readability Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Cleanup `Metadata.Job` instantiation Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Update test suite Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * continued: Update test suite Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * continued: Update test suite Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Clean up docs for class `Metadata` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Apply spotless Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Add copyright header Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> --------- Signed-off-by: wslulciuc <willy@datakin.com> Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR removes the OpenLineage models that were first introduced in the PoC in favor of
io.openlineage.server.*defined specifically for consumers. This PR also introduces classesMetadataandVersionId. Below, we outlined more specifically on their importance and usage.class
MetadataWrapper class for parsing:
OpenLineage.RunEventOpenLineage.JobEventOpenLineage.DatasetEventMethods:
Metadata.Run.forEvent(OpenLineage.RunEvent):Metadata.RunMetadata.Job.forEvent(OpenLineage.JobEvent):Metadata.JobMetadata.Dataset.forEvent(OpenLineage.DatasetEvent):Metadata.Datasetclass
VersionIdFactory class for
VersionIds.Methods:
VersionId.forJob():VersionIdVersionId.forDataset():VersionIdCloses: #1650