Merge branch 'main' into fix-symlinks-table

Author: pawel-big-lebowski

.circleci/get-jdk17.sh

@@ -14,11 +14,9 @@
 #
 # Usage: $ ./get-jdk17.sh
 
-set -e
-
 wget -qO - https://adoptium.jfrog.io/adoptium/api/gpg/key/public | sudo apt-key add -
 sudo add-apt-repository --yes https://adoptium.jfrog.io/adoptium/deb
-sudo apt-get update && sudo apt-get install temurin-17-jdk
+sudo apt-get update --allow-releaseinfo-change && sudo apt-get install --yes temurin-17-jdk
 sudo update-alternatives --set java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java
 sudo update-alternatives --set javac /usr/lib/jvm/temurin-17-jdk-amd64/bin/javac
 java -version

.github/workflows/test-chart.yaml

@@ -16,15 +16,15 @@ jobs:
           fetch-depth: 0
 
       - name: Setup Helm
-        uses: azure/setup-helm@v2.2
+        uses: azure/setup-helm@v3.4
 
       - name: Setup Python
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v4
         with:
           python-version: 3.7
 
       - name: Setup chart-testing
-        uses: helm/chart-testing-action@v2.3.0
+        uses: helm/chart-testing-action@v2.3.1
 
       - name: Run chart-testing (list-changed)
         id: list-changed

.gitignore

@@ -21,11 +21,10 @@ bin/
 build/
 clients/java/out/
 dist/
-integrations/airflow/tests/integration/integration-requirements.txt
-integrations/airflow/tests/integration/requirements.txt
-examples/airflow/whl
-examples/airflow/requirements.txt
+examples/airflow/dags/
 examples/airflow/openlineage.env
+examples/airflow/requirements.txt
+examples/airflow/whl
 out/*
 venv
 

CHANGELOG.md

@@ -42,11 +42,11 @@
 
 * Add support for `parentRun` facet as reported by older Airflow OpenLineage versions [`#2130`](https://github.com/MarquezProject/marquez/pull/2130) [@collado-mike](https://github.com/collado-mike)  
     *Adds a `parentRun` alias to the `LineageEvent` `RunFacet`.*
-* Add fix and tests for handling Airflow DAGs with dots and task groups [`2126`](https://github.com/MarquezProject/marquez/pull/2126) [@collado-mike](https://github.com/collado-mike) [@wslulciuc](https://github.com/wslulciuc)  
+* Add fix and tests for handling Airflow DAGs with dots and task groups [`#2126`](https://github.com/MarquezProject/marquez/pull/2126) [@collado-mike](https://github.com/collado-mike) [@wslulciuc](https://github.com/wslulciuc)  
     *Fixes a recent change that broke how Marquez handles DAGs with dots and tasks within task groups and adds test cases to validate.*
-* Fix version bump in `docker/up.sh` [`2129`](https://github.com/MarquezProject/marquez/pull/2129) [@wslulciuc](https://github.com/wslulciuc)  
+* Fix version bump in `docker/up.sh` [`#2129`](https://github.com/MarquezProject/marquez/pull/2129) [@wslulciuc](https://github.com/wslulciuc)  
     *Defines a `VERSION` variable to bump on a release.*
-* Use `clean` when running `shadowJar` in Dockerfile [`2145`](https://github.com/MarquezProject/marquez/pull/2145) [@wslulciuc](https://github.com/wslulciuc)  
+* Use `clean` when running `shadowJar` in Dockerfile [`#2145`](https://github.com/MarquezProject/marquez/pull/2145) [@wslulciuc](https://github.com/wslulciuc)  
     *Ensures the directory `api/build/libs/` is cleaned before building the JAR again and updates `.dockerignore` to ignore `api/build/*`.*
 * Fix bug that caused a single run event to create multiple jobs [`#2162`](https://github.com/MarquezProject/marquez/pull/2162) [@collado-mike](https://github.com/collado-mike)  
     *Checks to see if a run with the given ID already exists and uses the pre-associated job if so.*

CODE_QUALITY_AND_SECURITY.md

@@ -0,0 +1,30 @@
+# Code Quality and Security Assurance Statement
+
+The authors of Marquez are committed to providing secure software of the highest quality possible. To this end, we employ a number of tools and methodologies to ensure that our design, build, maintenance and testing practices maximize efficiency and minimize risk.
+
+The specific security and analysis methodologies that we employ include but are not limited to: 
+
+## Security
+
+- Participation in the [OpenSSF Best Practices Badge Program](https://bestpractices.coreinfrastructure.org/en/projects/5106) for Free/Libre and FLOSS projects to ensure that we follow current best practices for quality and security
+- Use of [HTTPS](https://en.wikipedia.org/wiki/HTTPS) for network communication 
+- Support for multiple cryptographic algorithms (through the use of HTTPS)
+- Separate storage of authentication credentials according to best practices
+- Use of secure protocols for network communication (through the use of HTTPS)
+- Up-to-date support for TLS/SSL (through the use of [OpenSSL](https://www.openssl.org/))
+- Performance of TLS certificate verification by default before sending HTTP headers with private information (through the use of OpenSSL and HTTPS)
+- Distribution of the software via cryptographically signed releases (on the [PyPI](https://pypi.org/) and [Maven](https://mvnrepository.com/) package repositories)
+- Use of [GitHub](https://github.com/) Issues for vulnerability reporting and tracking
+
+## Analysis
+
+- Use of [PMD](https://pmd.github.io/) and [Spotless](https://github.com/diffplug/spotless) for Java code linting on pull requests and builds
+- Use of [Flake8](https://flake8.pycqa.org/en/latest/) and [Pytest](https://docs.pytest.org/en/7.2.x/) for Python code linting on pull requests and builds
+- Use of GitHub Issues for bug reporting and tracking
+
+## Contact
+
+For more information about our approach to quality and security, feel free to reach out to the Marquez development team:
+
+- Slack: [Marquezproject.slack.com](http://bit.ly/MarquezSlack)
+- Twitter: [@MarquezProject](https://twitter.com/MarquezProject)

COMMITTERS.md

@@ -1,5 +1,5 @@
 # Marquez Committers
-The Marquez Committers are the group of people who can accept Pull Request to Marquez.
+The Marquez Committers are the group of people who can accept Pull Requests to Marquez.
 They take responsibility for guiding new pull requests into the main branch.
 
 
@@ -26,13 +26,13 @@ They take responsibility for guiding new pull requests into the main branch.
 ## Emeritus
 
 The following people are no longer working on the Marquez project.
-However they have been a committer in the past and through their
+However, they have been committers in the past and, through their
 contributions, we have a strong foundation to build on.
 
 | Name             | Handle                      |
 | ---------------- | ----------------------------|
 
 # Becoming a Committer
 
-A Contributor may become a Committer by a majority approval of the
-existing Committers. (per the project [charter](https://wiki.lfaidata.foundation/download/attachments/18481434/Marquez%20Project%20Technical%20Charter%20Final_Adopted%2005.21.20.pdf?version=1&modificationDate=1591718661000&api=v2))
+A Contributor may become a Committer by the approval of a majority of the
+existing Committers (as per the project [charter](https://wiki.lfaidata.foundation/download/attachments/18481434/Marquez%20Project%20Technical%20Charter%20Final_Adopted%2005.21.20.pdf?version=1&modificationDate=1591718661000&api=v2)).

CONTRIBUTING.md

@@ -46,7 +46,7 @@ We use [spotless](https://github.com/diffplug/spotless) to format our code. This
 $ ./gradlew spotlessApply
 ```
 
-> **Note:** To make formatting code simple, we recommend installing a [plugin](https://github.com/google/google-java-format#intellij-android-studio-and-other-jetbrains-ides) for your favorite IDE. We also us [Lombok](https://projectlombok.org). Though not required, you might want to install the [plugin](https://projectlombok.org/setup/overview) as well.
+> **Note:** To make formatting code simple, we recommend installing a [plugin](https://github.com/google/google-java-format#intellij-android-studio-and-other-jetbrains-ides) for your favorite IDE. We also use [Lombok](https://projectlombok.org). Though not required, you might want to install the [plugin](https://projectlombok.org/setup/overview), as well.
 
 # `.git/hooks`
 
@@ -94,7 +94,7 @@ act pull_request --reuse --verbose
 
 # Troubleshooting
 
-There is an issue within the _act_ tool that prevents the _kind_ cluster from being deleted after execution the action.
+There is an issue within the _act_ tool that prevents the _kind_ cluster from being deleted after execution of the action.
 When this condition exists, you will experience the error below.
 
 ```bash
@@ -122,12 +122,12 @@ $ ./gradlew publishToMavenLocal
 1. [Fork](https://github.com/MarquezProject/marquez/fork) and clone the repository
 2. Make sure all tests pass locally: `./gradlew :api:test`
 3. Create a new [branch](#branching): `git checkout -b feature/my-cool-new-feature`
-4. Make change on your cool new branch
+4. Make a change on your cool new branch
 5. Write a test for your change
-6. Make sure `.java` files are formatted: `./gradlew spotlessJavaCheck`
+6. Make sure `.java` files are formatted: `./gradlew spotlessJavaCheck`
 7. Make sure `.java` files contain a [copyright and license header](#copyright--license)
 8. Make sure to [sign you work](#sign-your-work)
-9. Push change to your fork and [submit a pull request](https://github.com/MarquezProject/marquez/compare)
+9. Push the change to your fork and [submit a pull request](https://github.com/MarquezProject/marquez/compare)
 10. Work with project maintainers to get your change reviewed and merged into the `main` branch
 11. Delete your branch
 
@@ -137,17 +137,17 @@ To ensure your pull request is accepted, follow these guidelines:
 * Do your best to have a [well-formed commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) for your change
 * [Keep diffs small](https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6) and self-contained
 * If your change fixes a bug, please [link the issue](https://help.github.com/articles/closing-issues-using-keywords) in your pull request description
-* Any changes to the API reference requires [regenerating](#api-docs) the static `openapi.html` file.
+* Any changes to the API reference require [regenerating](#api-docs) the static `openapi.html` file.
 
 > **Note:** A pull request should generally contain only one commit (use `git commit --amend` and `git push --force` or [squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) existing commits into one).
 
 # Branching
 
-* Use a _group_ at the beginning of your branch names
+* Use a _group_ at the beginning of your branch names:
 
   ```
   feature  Add or expand a feature
-  bug      Fix a bug
+  bug      Fix a bug
   proposal Propose a change
   ```
 
@@ -156,7 +156,7 @@ To ensure your pull request is accepted, follow these guidelines:
   ```
   feature/my-cool-new-feature
   bug/my-bug-fix
-  bug/my-other-bug-fix
+  bug/my-other-bug-fix
   proposal/my-proposal
   ```
 
@@ -167,18 +167,18 @@ To ensure your pull request is accepted, follow these guidelines:
 # Dependencies
 
 We use [renovate](https://github.com/renovatebot/renovate) to manage dependencies for most of our project modules,
-with a couple of exceptions. Renovate automatically detects new dependency versions, and opens pull
-requests to upgrade dependencies in accordance to the [configured rules](https://github.com/MarquezProject/marquez/blob/main/renovate.json).
+with a couple of exceptions. Renovate automatically detects new dependency versions and opens pull
+requests to upgrade dependencies in accordance with the [configured rules](https://github.com/MarquezProject/marquez/blob/main/renovate.json).
 
-The following dependencies are managed manually
+The following dependencies are managed manually:
 
 * _Web code_ - it is challenging to programmatically validate web content
 * _Spark versions_ - the internal query plans parsed by the Spark OpenLineage integration are not stable across Spark versions
 * _Gradle_ - this tool orchestrates the entire build pipeline and was excluded to ensure stability
 
 # Sign Your Work
 
-The _sign-off_ is a simple line at the end of the message for a commit. All commits needs to be signed. Your signature certifies that you wrote the patch or otherwise have the right to contribute the material (see [Developer Certificate of Origin](https://developercertificate.org)):
+The _sign-off_ is a simple line at the end of the message for a commit. All commits need to be signed. Your signature certifies that you wrote the patch or otherwise have the right to contribute the material (see [Developer Certificate of Origin](https://developercertificate.org)):
 
 ```
 This is my commit message
@@ -208,42 +208,42 @@ $ redoc-cli serve spec/openapi.yml
 
 Then browse to: http://localhost:8080
 
-> **Note:** To bundle or serve the API docs, please install [redoc-cli](https://www.npmjs.com/package/redoc-cli).
-
-# `COPYRIGHT` / `LICENSE`
-
-We use [SPDX](https://spdx.dev) for copyright and license information. The following license header **must** be included in all `java,` `bash`, and `py` source files:
-
-`java`
-
-```
-/*
- * Copyright 2018-2022 contributors to the Marquez project
- * SPDX-License-Identifier: Apache-2.0
- */
-```
-
-`bash`
-
-```
-#!/bin/bash
-#
-# Copyright 2018-2022 contributors to the Marquez project
-# SPDX-License-Identifier: Apache-2.0
-```
-
-`py`
-
-```
-# Copyright 2018-2022 contributors to the Marquez project
-# SPDX-License-Identifier: Apache-2.0
+> **Note:** To bundle or serve the API docs, please install [redoc-cli](https://www.npmjs.com/package/redoc-cli).
+
+# `COPYRIGHT` / `LICENSE`
+
+We use [SPDX](https://spdx.dev) for copyright and license information. The following license header **must** be included in all `java,` `bash`, and `py` source files:
+
+`java`
+
+```
+/*
+ * Copyright 2018-2022 contributors to the Marquez project
+ * SPDX-License-Identifier: Apache-2.0
+ */
+```
+
+`bash`
+
+```
+#!/bin/bash
+#
+# Copyright 2018-2022 contributors to the Marquez project
+# SPDX-License-Identifier: Apache-2.0
+```
+
+`py`
+
+```
+# Copyright 2018-2022 contributors to the Marquez project
+# SPDX-License-Identifier: Apache-2.0
 ```
 
 # Resources
 
 * [How to Contribute to Open Source](https://opensource.guide/how-to-contribute)
 * [Using the Fork-and-Branch Git Workflow](https://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow)
 * [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
-* [Keep a Changelog](https://keepachangelog.com)
+* [Keeping a Changelog](https://keepachangelog.com)
 * [Code Review Developer Guide](https://google.github.io/eng-practices/review)
 * [Signing Commits](https://docs.github.com/en/github/authenticating-to-github/signing-commits)