Skip to content

Commit 63e1056

Browse files
wslulciucwslulciuc
authored
Merge branch 'main' into main
2 parents e309364 + 9ab3d45 commit 63e1056

71 files changed

Lines changed: 1414 additions & 451 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.circleci/api-load-test.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
set -e
1515

1616
# Build version of Marquez
17-
readonly MARQUEZ_VERSION=0.48.0-SNAPSHOT
17+
readonly MARQUEZ_VERSION=0.50.0-SNAPSHOT
1818
# Fully qualified path to marquez.jar
1919
readonly MARQUEZ_JAR="api/build/libs/marquez-api-${MARQUEZ_VERSION}.jar"
2020

‎.circleci/db-migration.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
# Version of PostgreSQL
1414
readonly POSTGRES_VERSION="14"
1515
# Version of Marquez
16-
readonly MARQUEZ_VERSION=0.47.0
16+
readonly MARQUEZ_VERSION=0.49.0
1717
# Build version of Marquez
1818
readonly MARQUEZ_BUILD_VERSION="$(git log --pretty=format:'%h' -n 1)" # SHA1
1919
readonly POSTGRES_PORT=5432

‎.env.example‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ API_PORT=5000
22
API_ADMIN_PORT=5001
33
WEB_PORT=3000
44
POSTGRES_PORT=5432
5-
TAG=0.47.0
5+
TAG=0.49.0

‎CHANGELOG.md‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,63 @@
11
# Changelog
22

3-
## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.47.0...HEAD)
3+
## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.48.0...HEAD)
4+
5+
## [0.49.0](https://github.com/MarquezProject/marquez/compare/0.48.0...0.49.0) - 2024-08-07
6+
7+
### Added
8+
9+
* API: Job-to-Job lineage [`#2752`](https://github.com/MarquezProject/marquez/pull/2752) [@yanlibert](https://github.com/yanlibert)
10+
*Intended in part to spur a larger discussion of full parent/child hierarchy handling in Marquez. Changes only the backend API, adding the Job UUID along with the parent name to the Job metadata returned.*
11+
12+
### Fixed
13+
14+
* Web: security updates [`#2864`](https://github.com/MarquezProject/marquez/pull/2864) [@phixMe](https://github.com/phixMe)
15+
*Resolves `critical` security issues found using NPM's `audit` command.*
16+
* Web: encode Job name in API requests [`#2866`](https://github.com/MarquezProject/marquez/pull/2866) [@dolfinus](https://github.com/dolfinus)
17+
*Urlencodes Job, Dataset, tag and field names while sending an API request.*
18+
19+
## [0.48.0](https://github.com/MarquezProject/marquez/compare/0.47.0...0.48.0) - 2024-07-30
20+
21+
### Added
22+
23+
* API: add endpoint method and path to metrics name [`#2850`](https://github.com/MarquezProject/marquez/pull/2850) [@JDarDagran](https://github.com/JDarDagran)
24+
*In the metrics endpoint, there was information gathered containing the SQL Object name and method name. This introduces labels (DAO name, DAO method, endpoint method, endpoint path) and adds more information about endpoints.*
25+
* API: add paging to dataset versions panel [`#2855`](https://github.com/MarquezProject/marquez/pull/2855) [@davidsharp7](https://github.com/davidsharp7)
26+
*Adds Datasets paging.*
27+
* API: add paging on Jobs panel [`#2852`](https://github.com/MarquezProject/marquez/pull/2852) [@davidsharp7](https://github.com/davidsharp7)
28+
*Adds Job-level paging of Runs.*
29+
* API: add Dataset schema versions [`#2763`](https://github.com/MarquezProject/marquez/pull/2763) [@davidjgoss](https://github.com/davidjgoss)
30+
*Adds Dataset schema versions to the model and enables writing to it.*
31+
* Docker: make db port configurable via `POSTGRES_PORT` [`#2751`](https://github.com/MarquezProject/marquez/pull/2751) [@merobi-hub](https://github.com/merobi-hub)
32+
*Adds support for easy db port reassignment.*
33+
* Java: allow customization of Apache HTTP in Java client [`#2822`](https://github.com/MarquezProject/marquez/pull/2822) [@davidjgoss](https://github.com/davidjgoss)
34+
*Allows customization of Apache HTTP in Java client.*
35+
* Web: add Job tagging to UI [`#2837`](https://github.com/MarquezProject/marquez/pull/2837) [@davidsharp7](https://github.com/davidsharp7)
36+
*Adds Job tagging to the UI.*
37+
* Web: source code facets [`#2833`](https://github.com/MarquezProject/marquez/pull/2833) [@phixMe](https://github.com/phixMe)
38+
*Adds typedef and rendering of the `sourceCode` facet for a Job if available.*
39+
40+
### Fixed
41+
* API: Dataset query to get only the latest facet for each version [`#2859`](https://github.com/MarquezProject/marquez/pull/2859) [@sophiely](https://github.com/sophiely)
42+
*The facet partition is ranked by Dataset version and facet name so as we can take only the most recent facet for each Dataset UUID and type.*
43+
* API: optimize column lineage query performance [`#2821`](https://github.com/MarquezProject/marquez/pull/2821) [@vinhnemo](https://github.com/vinhnemo)
44+
*Adds a filter condition to the CTE `dataset_fields_view` in [ColumnLineageDao.java](https://github.com/MarquezProject/marquez/blob/d6ac3e6435748cada4e08516250feee48ed9c0fa/api/src/main/java/marquez/db/ColumnLineageDao.java#L187).*
45+
* Web: deduplicate the versions displayed [`#2854`](https://github.com/MarquezProject/marquez/pull/2854) [@namyyys](https://github.com/namyyys)
46+
*Excludes the symlinks from the result of the query displaying the version history in order to exclude duplicate versions.*
47+
* Web: clean up issues highlighted by some Spark Integration Data [`#2856`](https://github.com/MarquezProject/marquez/pull/2856) [@phixMe](https://github.com/phixMe)
48+
*Fixes numerous issues in our interfaces related to some OpenLineage Spark events.*
49+
* Web: remove limit from assertion evaluation [`#2844`](https://github.com/MarquezProject/marquez/pull/2844) [@phixMe](https://github.com/phixMe)
50+
*Fixes bug where our status indicator was the wrong color.*
51+
* Web: bring Dataset tags into line with Job Tags [`#2841`](https://github.com/MarquezProject/marquez/pull/2841) [@davidsharp7](https://github.com/davidsharp7)
52+
*Brings Dataset tags into line with Job tags.*
53+
* Web: fix scroll issues for drawer and home pages [`#2820`](https://github.com/MarquezProject/marquez/pull/2820) [@phixMe](https://github.com/phixMe)
54+
*Scrolling improvements for drawer and home pages.*
55+
* Web: fix search endpoint parameters [`#2818`](https://github.com/MarquezProject/marquez/pull/2818) [@Nisarg-Chokshi](https://github.com/Nisarg-Chokshi)
56+
*The search API parameters were not getting updated correctly on changing the filter and sort options.*
57+
58+
### Removed
59+
* Web: DRY paging [`#2832`](https://github.com/MarquezProject/marquez/pull/2832) [@phixMe](https://github.com/phixMe)
60+
*Removes repeated code for paging on lineage events, jobs and datasets.*
461

562
## [0.47.0](https://github.com/MarquezProject/marquez/compare/0.46.0...0.47.0) - 2024-05-17
663

‎README.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<div align="center">
22
<img src="./docs/assets/images/marquez-logo.png" width="500px" />
33
<a href="https://lfaidata.foundation/projects">
4-
<img src="./docs/assets/images/lfaidata-project-badge-incubation-black.png" width="125px" />
4+
<img src="./docs/assets/images/lfaidata-project-badge-graduate-black.png" width="125px" />
55
</a>
66
</div>
77

@@ -22,7 +22,7 @@ Marquez is an open source **metadata service** for the **collection**, **aggrega
2222

2323
## Status
2424

25-
Marquez is an [LF AI & Data Foundation](https://lfaidata.foundation/projects/marquez) incubation project under active development, and we'd love your help!
25+
Marquez is an [LF AI & Data Foundation](https://lfaidata.foundation/projects/marquez) Graduated project under active development, and we'd love your help!
2626

2727
## Adopters
2828

@@ -31,6 +31,7 @@ Want to be added? Send a pull request our way!
3131
* [Astronomer](https://astronomer.io)
3232
* [Datakin](https://datakin.com)
3333
* [Northwestern Mutual](https://www.northwesternmutual.com)
34+
* [Ilum](https://ilum.cloud)
3435

3536
## Try it!
3637

@@ -96,9 +97,9 @@ Versions of Marquez are compatible with OpenLineage unless noted otherwise. We e
9697

9798
| **Marquez** | **OpenLineage** | **Status** |
9899
|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------|---------------|
99-
| [`UNRELEASED`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased) | [`1-0-5`](https://openlineage.io/spec/1-0-5/OpenLineage.json) | `CURRENT` |
100-
| [`0.46.0`](https://github.com/MarquezProject/marquez/blob/0.46.0/CHANGELOG.md#0460---2024-03-15) | [`1-0-5`](https://openlineage.io/spec/1-0-5/OpenLineage.json) | `RECOMMENDED` |
101-
| [`0.45.0`](https://github.com/MarquezProject/marquez/blob/0.45.0/CHANGELOG.md#0450---2024-03-07) | [`1-0-5`](https://openlineage.io/spec/1-0-0/OpenLineage.json) | `MAINTENANCE` |
100+
| [`UNRELEASED`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `CURRENT` |
101+
| [`0.49.0`](https://github.com/MarquezProject/marquez/blob/0.49.0/CHANGELOG.md#0490---2024-08-07) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `RECOMMENDED` |
102+
| [`0.48.0`](https://github.com/MarquezProject/marquez/blob/0.45.0/CHANGELOG.md#0480---2024-08-05) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `MAINTENANCE` |
102103

103104
> **Note:** The [`openlineage-python`](https://pypi.org/project/openlineage-python) and [`openlineage-java`](https://central.sonatype.com/artifact/io.openlineage/openlineage-java) libraries will a higher version than the OpenLineage [specification](https://github.com/OpenLineage/OpenLineage/tree/main/spec) as they have different version requirements.
104105
@@ -179,7 +180,7 @@ Marquez listens on port `8080` for all API calls and port `8081` for the admin i
179180
* Website: https://marquezproject.ai
180181
* Source: https://github.com/MarquezProject/marquez
181182
* Chat: [MarquezProject Slack](https://bit.ly/Marquez_Slack_invite)
182-
* Twitter: [@MarquezProject](https://twitter.com/MarquezProject)
183+
* X: [@MarquezProject](https://twitter.com/MarquezProject)
183184

184185
## Contributing
185186

@@ -191,4 +192,4 @@ If you discover a vulnerability in the project, please open an issue and attach
191192

192193
----
193194
SPDX-License-Identifier: Apache-2.0
194-
Copyright 2018-2023 contributors to the Marquez project.
195+
Copyright 2018-2024 contributors to the Marquez project.

‎api/build.gradle‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -130,8 +130,8 @@ publishing {
130130
def snapshotsRepoUrl = 'https://oss.sonatype.org/content/repositories/snapshots/'
131131
url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl
132132
credentials {
133-
username = System.getenv('SONATYPE_NEXUS_USERNAME')
134-
password = System.getenv('SONATYPE_NEXUS_PASSWORD')
133+
username = System.getenv('OSSRH_USERNAME')
134+
password = System.getenv('OSSRH_PASSWORD')
135135
}
136136
}
137137
}

‎api/src/main/java/marquez/MarquezApp.java‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,10 @@
3535
import marquez.common.Utils;
3636
import marquez.db.DbMigration;
3737
import marquez.jobs.DbRetentionJob;
38+
import marquez.logging.DelegatingSqlLogger;
39+
import marquez.logging.LabelledSqlLogger;
3840
import marquez.logging.LoggingMdcFilter;
41+
import marquez.service.DatabaseMetrics;
3942
import marquez.tracing.SentryConfig;
4043
import marquez.tracing.TracingContainerResponseFilter;
4144
import marquez.tracing.TracingSQLLogger;
@@ -57,7 +60,9 @@ public final class MarquezApp extends Application<MarquezConfig> {
5760

5861
// Monitoring
5962
private static final String PROMETHEUS = "prometheus";
63+
private static final String PROMETHEUS_V2 = "prometheus_v2";
6064
private static final String PROMETHEUS_ENDPOINT = "/metrics";
65+
private static final String PROMETHEUS_ENDPOINT_V2 = "/v2beta/metrics";
6166

6267
public static void main(final String[] args) throws Exception {
6368
new MarquezApp().run(args);
@@ -73,7 +78,9 @@ public void initialize(@NonNull Bootstrap<MarquezConfig> bootstrap) {
7378
// Enable metric collection for prometheus.
7479
CollectorRegistry.defaultRegistry.register(
7580
new DropwizardExports(bootstrap.getMetricRegistry()));
81+
DatabaseMetrics.registry.register(new DropwizardExports(bootstrap.getMetricRegistry()));
7682
DefaultExports.initialize(); // Add metrics for CPU, JVM memory, etc.
83+
DefaultExports.register(DatabaseMetrics.registry);
7784

7885
// Enable variable substitution with environment variables.
7986
bootstrap.setConfigurationSourceProvider(
@@ -162,7 +169,8 @@ private Jdbi newJdbi(
162169
.installPlugin(new SqlObjectPlugin())
163170
.installPlugin(new PostgresPlugin())
164171
.installPlugin(new Jackson2Plugin());
165-
SqlLogger sqlLogger = new InstrumentedSqlLogger(env.metrics());
172+
SqlLogger sqlLogger =
173+
new DelegatingSqlLogger(new LabelledSqlLogger(), new InstrumentedSqlLogger(env.metrics()));
166174
if (isSentryEnabled(config)) {
167175
sqlLogger = new TracingSQLLogger(sqlLogger);
168176
}
@@ -197,6 +205,9 @@ private void registerServlets(@NonNull Environment env) {
197205

198206
// Expose metrics for monitoring.
199207
env.servlets().addServlet(PROMETHEUS, new MetricsServlet()).addMapping(PROMETHEUS_ENDPOINT);
208+
env.servlets()
209+
.addServlet(PROMETHEUS_V2, new MetricsServlet(DatabaseMetrics.registry))
210+
.addMapping(PROMETHEUS_ENDPOINT_V2);
200211
}
201212

202213
private void registerFilters(@NonNull Environment env, MarquezContext marquezContext) {

‎api/src/main/java/marquez/api/DatasetResource.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,11 @@ public Response listVersions(
133133
final List<DatasetVersion> datasetVersions =
134134
datasetVersionService.findAllWithRun(
135135
namespaceName.getValue(), datasetName.getValue(), limit, offset);
136-
return Response.ok(new DatasetVersions(datasetVersions)).build();
136+
137+
final int totalCount =
138+
datasetVersionService.countDatasetVersions(
139+
namespaceName.getValue(), datasetName.getValue());
140+
return Response.ok(new DatasetVersions(datasetVersions, totalCount)).build();
137141
}
138142

139143
@Timed
@@ -301,5 +305,8 @@ static class DatasetVersions {
301305
@NonNull
302306
@JsonProperty("versions")
303307
List<DatasetVersion> value;
308+
309+
@JsonProperty("totalCount")
310+
int totalCount;
304311
}
305312
}

‎api/src/main/java/marquez/api/JobResource.java‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,8 @@ public Response listRuns(
238238

239239
final List<Run> runs =
240240
runService.findAll(namespaceName.getValue(), jobName.getValue(), limit, offset);
241-
return Response.ok(new Runs(runs)).build();
241+
final int totalCount = jobService.countJobRuns(namespaceName.getValue(), jobName.getValue());
242+
return Response.ok(new Runs(runs, totalCount)).build();
242243
}
243244

244245
@Path("/jobs/runs/{id}")
@@ -329,5 +330,8 @@ public static class Runs {
329330
@NonNull
330331
@JsonProperty("runs")
331332
List<Run> value;
333+
334+
@JsonProperty("totalCount")
335+
int totalCount;
332336
}
333337
}

‎api/src/main/java/marquez/db/Columns.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ private Columns() {}
9292

9393
/* JOB ROW COLUMNS */
9494
public static final String PARENT_JOB_NAME = "parent_job_name";
95+
public static final String PARENT_JOB_UUID = "parent_job_uuid";
9596
public static final String SIMPLE_NAME = "simple_name";
9697
public static final String SYMLINK_TARGET_UUID = "symlink_target_uuid";
9798

0 commit comments

Comments
 (0)
⚡