Skip to content

Commit c4fdaea

Browse files
pawel-big-lebowskipawel-big-lebowski
committed
extend existing proposal with comment suggestion
Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
1 parent 258f38c commit c4fdaea

1 file changed

Lines changed: 92 additions & 85 deletions

File tree

‎proposals/2045-column-lineage-endpoint.md‎

Lines changed: 92 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Proposal: Column lineage endpoint proposal
22

3-
Author(s): @julienledem
3+
Author(s): @julienledem, @pawel-big-lebowski
44

55
Created: 20022-08-18
66

@@ -15,129 +15,136 @@ Dicussion: [column lineage endpoint issue #2045](https://github.com/MarquezProje
1515

1616
### Existing elements
1717

18-
- OpenLineage defines a [column-level lineage facet]- (https://github.com/OpenLineage/OpenLineage/blob/ff0d87d30ed6c9fe39472788948266a6d3190585/spec/facets/ColumnLineageDatasetFacet.md).
18+
- OpenLineage defines a [column-level lineage facet]- (https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json).
1919
- Marquez has a lineage endpoint `GET /api/v1/lineage` that returns the current lineage graph connected to a job or a dataset
2020

21+
### Column lineage characteristics and general assumptions
22+
23+
Column level lineage is a different lineage graph due to a different node granularity - kind of zoomed-in view of existing lineage. Instead of datasets and jobs being lineage graph nodes, each dataset field becomes a node. Additionally, there are edges between dataset fields, instead of datasets itself. Thus, enriching existing lineage with column lineage information would not be sufficient. That’s why we propose another API endpoint with column lineage graph.
24+
25+
Upstream and downstream edges do have different characteristics. An output dataset is always produced by a single version of input dataset (one upstream), while a single input datset version can have multiple output dataset versions. Lineage graph can be then easily flooded by downstream subgraph which blurs the overall view. That's why we consider an upstream column lineage as a default one. Downstream lineage will be returned only when requested explicitly.
26+
2127
### New Elements
22-
We propose to add the following:
23-
- Add column lineage to the lineage endpoint
24-
- A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column.
25-
- Point-in-time upstream (dataset or column level) lineage given a version of a dataset.
28+
29+
We propose the following changes:
30+
31+
- Add column lineage to the dataset resource endpoint. Column lineage will NOT be added to existing `/lineage` endpoint as it may be a heavy database operation run on each lineage graph's node which we want to avoid. Based on that, column level lineage get be requested per dataset in separate requests when required.
32+
- A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column.
33+
- Point-in-time upstream (dataset or column level) lineage given a version of a dataset.
2634

2735
## Proposal
2836

29-
### add column lineage to existing endpoint
30-
In the GET /lineage api, add column lineage to DATASET nodes' data
31-
```diff
32-
{
33-
"id": "dataset:food_delivery:public.categories",
34-
"type": "DATASET",
35-
"data": {
36-
"type": "DATASET",
37-
"id": {
38-
"namespace": "food_delivery",
39-
"name": "public.categories"
40-
},
41-
"type": "DB_TABLE",
42-
...
43-
"fields": [{
44-
...
45-
}],
46-
> columnLineage: {
47-
> "a": {
48-
> inputFields: [
49-
> {namespace: "ns", name: "name", "field": "a"},
50-
> ... other inputs
51-
> ],
52-
> transformationDescription: "identical",
53-
> transformationType: "IDENTITY"
54-
> },
55-
> "b": ... other output fields
56-
> }
57-
},
58-
"inEdges": [{
59-
"origin": "job:food_delivery:etl_orders.etl_categories",
60-
"destination": "dataset:food_delivery:public.categories"
61-
}],
62-
"outEdges": [{
63-
"origin": "dataset:food_delivery:public.categories",
64-
"destination": "job:food_delivery:etl_orders.etl_orders_7_days"
65-
}]
66-
}
67-
```
37+
### Add column lineage to existing datasets endpoint
38+
39+
In the `GET /api/v1/namespaces/{namespace}/datasets` api, add column lineage facet to returned dataset resource.
6840

69-
### add a column-level-lineage endpoint:
41+
### Add a column-level-lineage endpoint:
7042

43+
New endpoints to retrieve a column lineage of a single field or a whole dataset will be added:
44+
```
45+
GET /column-lineage?nodeId=dataset:{namespace}:{dataset}
46+
GET /column-lineage?nodeId=datasetField:{namespace}:{dataset}:{field}
47+
```
48+
For example:
7149
```
72-
GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days&column=a
50+
GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days
51+
GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a
7352
```
74-
`column` is a ne parameter that must be a column in the schema of the provided dataset `nodeId`.
7553

76-
The logic is layered on the existing lineage endpoint, filtering down to the datasets that contribute to that column.
77-
It only returns dataset nodes.
54+
Although creating a new endpoint, we would like to reuse existing data structures with a new `NodeType.FIELD` introduced.
7855

79-
```diff
56+
The logic returns dataset field node:
57+
58+
```
59+
GET /column-lineage?nodeId=datasetField:db1:table1:a
60+
...
8061
{
8162
graph: [
8263
{
83-
"id": "dataset:db1:table2",
84-
"type": "DATASET",
85-
data: {
86-
namespace: "DB1",
87-
name: "table2",
88-
> columnLineage: {
89-
> "a": {
90-
> inputFields: [
91-
> {namespace: "DB1", name: "table1, "field": "a"}
92-
> ],
93-
> transformationDescription: "identical",
94-
> transformationType: "IDENTITY"
95-
> },
96-
> "b": ... other output fields
97-
> }
64+
"id": "datasetField:db1:table1:a",
65+
"type": "DATASET_FIELD",
66+
"data": {
67+
"namespace": "DB1",
68+
"name": "table2",
69+
"field": "a",
70+
"type": "integer",
71+
"transformationDescription": "identical",
72+
"transformationType": "IDENTITY",
73+
"inputFields": [
74+
{ "namespace": "DBA", "name": "tableA", "field": "columnA"},
75+
{ "namespace": "DBB", "name": "tableB", "field": "columnB"},
76+
{ "namespace": "DBC", "name": "tableC", "field": "columnC"}
77+
]
78+
"inEdges": [
79+
{
80+
"origin": "datasetField:db1:table1:a",
81+
"destination": "datasetField:DBA:tableA:columnA"
82+
},
83+
{
84+
"origin": "datasetField:db1:table1:a",
85+
"destination": "datasetField:DBB:tableB:columnB"
86+
},
87+
{
88+
"origin": "datasetField:db1:table1:a",
89+
"destination": "datasetField:DBB:tableB:columnC"
90+
}
91+
],
9892
},
9993
...
94+
# Input fields, present within "inEdges", can be also returned within a graph due to a `depth` parameter greate than 0.
10095
}
10196
]
10297
}
10398
```
10499

100+
The `depth` parameter controls how many edges, from a given dataset field, shall be returned. The default is set to `0`. In case of default equal `1`, each `inputField` will be returned as a separate node within a response graph with `inputFields` used to produce it. Please note that extending depth may increase the graph size and affect request performance.
101+
102+
The endpoints above fetches upstream column-lineage for given dataset field or all fields within a dataset. Downstream column lineage is turned off by default. However, this can be turned on with an extra `withDownstream` parameter like:
103+
104+
```
105+
GET /column-lineage?nodeId=datasetField:food_delivery:public.delivery_7_days:a&withDownstream=true
106+
107+
```
108+
This will include `outEdges` within the returned node of the graph.
109+
110+
105111
### Point in time upstream lineage
106-
return historical upstream lineage from a given Dataset version.
107-
This adds the version element to the nodeId in both the existing `/api/v1/lineage` and newly proposed `/api/v1/column-lineage` endpoint
112+
113+
Point in time lineage for newly proposed `/api/v1/column-lineage` endpoint:
108114
```
109-
GET /lineage?nodeId=dataset:food_delivery:public.delivery_7_days:{version}
110-
GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days:{version}&column=a
115+
GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&datasetVersion=123e4567-e89b-12d3-a456-426614174000
116+
GET /column-lineage?nodeId=dataset_field:food_delivery:public.delivery_7_days:a&lineageAt=1661846242
111117
```
112-
This returns only upstream lineage in this current proposal. This is because upstream lineage is well defined to a specific version while downstream lineage is not. The data payload would add a version field.
118+
119+
Point in time can be controlled by:
120+
* **datasetVersion** - uuid of a specific dataset version,
121+
* **lineageAt** - which contains a unix timestamp.
122+
123+
When **lineageAt** specified, the latest dataset version before timestamp will be found. Regardles **datasetVersion** or **lineageAt** parameters applied, responses will be the same as below:
124+
113125
```diff
114126
{
115127
graph: [
116128
{
117-
< "id": "dataset:db1:table2",
118-
> "id": "dataset:db1:table2#{VERSION UUID}",
119-
"type": "DATASET",
120-
data: {
121-
namespace: "DB1",
122-
name: "table2",
123-
> version: "{VERSION UUID}"
124-
...
125-
}
126-
}
127-
]
129+
< "id": "datasetField:db1:table1:a",
130+
> "id": "datasetField:db1:table1:a#{VERSION UUID}",
131+
"type": "DATASET_FIELD",
132+
"data": {
133+
....
128134
}
129135
```
130136

131137
## Implementation
132138

133-
### columne lineage facet in lineage
134-
Adding the columnLineage facet requires a formatting of existing facet data.
139+
### columne lineage facet in dataset resource endpoint
140+
Adding the columnLineage facet requires a formatting of existing facet data (work in progress).
135141
### column lineage endpoint
136142
The `/column-lineage` endpoint leverages the `/lineage` endpoint and then filters down the payload to return the expected result.
137143
### point-in-time upstream lineage
144+
138145
The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version.
139146
Dataset version -> run that produced it -> consumed Dataset Versions.
140147

141148
## Next Steps
142149

143-
Review of this proposal and production of detailed design for the implementation, in particular for the point in time lineage which might affect the dabtabase schema.
150+
Review of this proposal and production of detailed design for the implementation.:

0 commit comments

Comments
 (0)
⚡