Merge branch 'MarquezProject:main' into feature/lineage_event_created_at_indexed

Author: prachim-collab

.circleci/get-jdk17.sh

@@ -14,11 +14,9 @@
 #
 # Usage: $ ./get-jdk17.sh
 
-set -e
-
 wget -qO - https://adoptium.jfrog.io/adoptium/api/gpg/key/public | sudo apt-key add -
 sudo add-apt-repository --yes https://adoptium.jfrog.io/adoptium/deb
-sudo apt-get update && sudo apt-get install temurin-17-jdk
+sudo apt-get update --allow-releaseinfo-change && sudo apt-get install --yes temurin-17-jdk
 sudo update-alternatives --set java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java
 sudo update-alternatives --set javac /usr/lib/jvm/temurin-17-jdk-amd64/bin/javac
 java -version

.env.example

@@ -1,4 +1,4 @@
 API_PORT=5000
 API_ADMIN_PORT=5001
 WEB_PORT=3000
-TAG=0.27.0
+TAG=0.28.0

.github/workflows/headerchecker.yml

@@ -31,7 +31,7 @@ jobs:
       - name: Check for headers
         run: |
           ok=1
-          readarray -t files <<<"$(jq -r '.[]' <<<'${{ steps.files.outputs.all }}')"
+          readarray -t files <<<"$(jq -r '.[]' <<<'${{ steps.files.outputs.added_modified }}')"
           for file in ${files[@]}; do 
             if [[ ($file == *".java") ]]; then
               if ! grep -q Copyright "$file"; then

.github/workflows/test-chart.yaml

@@ -16,15 +16,15 @@ jobs:
           fetch-depth: 0
 
       - name: Setup Helm
-        uses: azure/setup-helm@v2.2
+        uses: azure/setup-helm@v3.4
 
       - name: Setup Python
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v4
         with:
           python-version: 3.7
 
       - name: Setup chart-testing
-        uses: helm/chart-testing-action@v2.3.0
+        uses: helm/chart-testing-action@v2.3.1
 
       - name: Run chart-testing (list-changed)
         id: list-changed

CHANGELOG.md

@@ -1,6 +1,44 @@
 # Changelog
 
-## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.27.0...HEAD)
+## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.28.0...HEAD)
+
+### Added
+
+* Column-lineage endpoints supports point-in-time requests [`#2265`](https://github.com/MarquezProject/marquez/pull/2265) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
+    *Enable requesting `column-lineage` endpoint by a dataset version, job version or dataset field of a specific dataset version.*
+
+### Fixed
+
+* Allow null column type in column-lineage [`#2272`](https://github.com/MarquezProject/marquez/pull/2272) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
+* Include error message for JSON processing exception [`#2271`](https://github.com/MarquezProject/marquez/pull/2271) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)  
+    *In case of JSON processing exceptions Marquez API should return exception message to a client.*
+* Fix column lineage when multiple jobs write to same dataset [`#2289`](https://github.com/MarquezProject/marquez/pull/2289) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
+    *The fix deprecates the way fields `transformationDescription` and `transformationType` are returned. The depracated way of returning those fields will be removed in 0.30.0.*
+
+## [0.28.0](https://github.com/MarquezProject/marquez/compare/0.27.0...0.28.0) - 2022-11-21
+
+### Added
+
+* Optimize current runs query for lineage API [`#2211`](https://github.com/MarquezProject/marquez/pull/2211) [@prachim-collab](https://github.com/prachim-collab)   
+    *Add a simpler, alternate `getCurrentRuns` query that gets only simple runs from the database without the additional data from tables such as `run_args`, `job_context`, `facets`, etc., which required extra table joins.*
+* Add Code Quality, DCO and Governance docs to project [`#2237`](https://github.com/MarquezProject/marquez/pull/2237) [`#2241`](https://github.com/MarquezProject/marquez/pull/2241) [@merobi-hub](https://github.com/MarquezProject/marquez/commits?author=merobi-hub)  
+    *Adds a number of standard governance and procedure docs to the project.*
+* Add possibility to soft-delete namespaces [`#2244`](https://github.com/MarquezProject/marquez/pull/2244) [@mobuchowski](https://github.com/mobuchowski)  
+    *Adds the ability to "hide" inactive namespaces. The namespaces are undeleted when a relevant OL event is received.*
+* Add search service proposal [`#2203`](https://github.com/MarquezProject/marquez/pull/2203) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)  
+    *Proposes using ElasticSearch as a pluggable search service to enhance the search feature in Marquez and adding the ability to turn it off, as well. Includes ideas about what should be indexed and the requirements for the interface.*
+
+### Fixed
+
+* Show facets even when dataset has no fields [`#2214`](https://github.com/MarquezProject/marquez/pull/2214) [@JDarDagran](https://github.com/JDarDagran)  
+    *Changes the logic in the `DatasetInfo` component to always show facets so that dataset facets are visible in the UI even if no dataset fields have been set.* 
+* Appreciate column prefix when given for `ended_at` [`#2231`](https://github.com/MarquezProject/marquez/pull/2231) [@fm100](https://github.com/fm100)  
+    *The `ended_at` column was always null when querying if `columnPrefix` was given for the mapper. Now, `columnPrefix` is included when checking for column existence.*
+* Fix bug keeping jobs from being properly deleted [`#2244`](https://github.com/MarquezProject/marquez/pull/2244) [@mobuchowski](https://github.com/mobuchowski)  
+    *It wasn't possible to delete jobs created from events that had a `ParentRunFacet`. Now it's possible.*
+* Fix symlink table column length ['#2217'](https://github.com/MarquezProject/marquez/pull/2217) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)  
+    *The dataset's name column in the `dataset_symlinks` table was shorter than the column in the datasets table. Changes the existing V48 migration script to allow proper migration for users who did not upgrade yet, and adds an extra migration script to extend the column length for users who did upgrade but did not experience the issues.*
+
 
 ## [0.27.0](https://github.com/MarquezProject/marquez/compare/0.26.0...0.27.0) - 2022-10-24
 
@@ -42,11 +80,11 @@
 
 * Add support for `parentRun` facet as reported by older Airflow OpenLineage versions [`#2130`](https://github.com/MarquezProject/marquez/pull/2130) [@collado-mike](https://github.com/collado-mike)  
     *Adds a `parentRun` alias to the `LineageEvent` `RunFacet`.*
-* Add fix and tests for handling Airflow DAGs with dots and task groups [`2126`](https://github.com/MarquezProject/marquez/pull/2126) [@collado-mike](https://github.com/collado-mike) [@wslulciuc](https://github.com/wslulciuc)  
+* Add fix and tests for handling Airflow DAGs with dots and task groups [`#2126`](https://github.com/MarquezProject/marquez/pull/2126) [@collado-mike](https://github.com/collado-mike) [@wslulciuc](https://github.com/wslulciuc)  
     *Fixes a recent change that broke how Marquez handles DAGs with dots and tasks within task groups and adds test cases to validate.*
-* Fix version bump in `docker/up.sh` [`2129`](https://github.com/MarquezProject/marquez/pull/2129) [@wslulciuc](https://github.com/wslulciuc)  
+* Fix version bump in `docker/up.sh` [`#2129`](https://github.com/MarquezProject/marquez/pull/2129) [@wslulciuc](https://github.com/wslulciuc)  
     *Defines a `VERSION` variable to bump on a release.*
-* Use `clean` when running `shadowJar` in Dockerfile [`2145`](https://github.com/MarquezProject/marquez/pull/2145) [@wslulciuc](https://github.com/wslulciuc)  
+* Use `clean` when running `shadowJar` in Dockerfile [`#2145`](https://github.com/MarquezProject/marquez/pull/2145) [@wslulciuc](https://github.com/wslulciuc)  
     *Ensures the directory `api/build/libs/` is cleaned before building the JAR again and updates `.dockerignore` to ignore `api/build/*`.*
 * Fix bug that caused a single run event to create multiple jobs [`#2162`](https://github.com/MarquezProject/marquez/pull/2162) [@collado-mike](https://github.com/collado-mike)  
     *Checks to see if a run with the given ID already exists and uses the pre-associated job if so.*

CODE_QUALITY_AND_SECURITY.md

@@ -0,0 +1,30 @@
+# Code Quality and Security Assurance Statement
+
+The authors of Marquez are committed to providing secure software of the highest quality possible. To this end, we employ a number of tools and methodologies to ensure that our design, build, maintenance and testing practices maximize efficiency and minimize risk.
+
+The specific security and analysis methodologies that we employ include but are not limited to: 
+
+## Security
+
+- Participation in the [OpenSSF Best Practices Badge Program](https://bestpractices.coreinfrastructure.org/en/projects/5106) for Free/Libre and FLOSS projects to ensure that we follow current best practices for quality and security
+- Use of [HTTPS](https://en.wikipedia.org/wiki/HTTPS) for network communication 
+- Support for multiple cryptographic algorithms (through the use of HTTPS)
+- Separate storage of authentication credentials according to best practices
+- Use of secure protocols for network communication (through the use of HTTPS)
+- Up-to-date support for TLS/SSL (through the use of [OpenSSL](https://www.openssl.org/))
+- Performance of TLS certificate verification by default before sending HTTP headers with private information (through the use of OpenSSL and HTTPS)
+- Distribution of the software via cryptographically signed releases (on the [PyPI](https://pypi.org/) and [Maven](https://mvnrepository.com/) package repositories)
+- Use of [GitHub](https://github.com/) Issues for vulnerability reporting and tracking
+
+## Analysis
+
+- Use of [PMD](https://pmd.github.io/) and [Spotless](https://github.com/diffplug/spotless) for Java code linting on pull requests and builds
+- Use of [Flake8](https://flake8.pycqa.org/en/latest/) and [Pytest](https://docs.pytest.org/en/7.2.x/) for Python code linting on pull requests and builds
+- Use of GitHub Issues for bug reporting and tracking
+
+## Contact
+
+For more information about our approach to quality and security, feel free to reach out to the Marquez development team:
+
+- Slack: [Marquezproject.slack.com](http://bit.ly/MarquezSlack)
+- Twitter: [@MarquezProject](https://twitter.com/MarquezProject)

COMMITTERS.md

@@ -1,5 +1,5 @@
 # Marquez Committers
-The Marquez Committers are the group of people who can accept Pull Request to Marquez.
+The Marquez Committers are the group of people who can accept Pull Requests to Marquez.
 They take responsibility for guiding new pull requests into the main branch.
 
 
@@ -26,13 +26,13 @@ They take responsibility for guiding new pull requests into the main branch.
 ## Emeritus
 
 The following people are no longer working on the Marquez project.
-However they have been a committer in the past and through their
+However, they have been committers in the past and, through their
 contributions, we have a strong foundation to build on.
 
 | Name             | Handle                      |
 | ---------------- | ----------------------------|
 
 # Becoming a Committer
 
-A Contributor may become a Committer by a majority approval of the
-existing Committers. (per the project [charter](https://wiki.lfaidata.foundation/download/attachments/18481434/Marquez%20Project%20Technical%20Charter%20Final_Adopted%2005.21.20.pdf?version=1&modificationDate=1591718661000&api=v2))
+A Contributor may become a Committer by the approval of a majority of the
+existing Committers (as per the project [charter](https://wiki.lfaidata.foundation/download/attachments/18481434/Marquez%20Project%20Technical%20Charter%20Final_Adopted%2005.21.20.pdf?version=1&modificationDate=1591718661000&api=v2)).

CONTRIBUTING.md

@@ -46,7 +46,7 @@ We use [spotless](https://github.com/diffplug/spotless) to format our code. This
 $ ./gradlew spotlessApply
 ```
 
-> **Note:** To make formatting code simple, we recommend installing a [plugin](https://github.com/google/google-java-format#intellij-android-studio-and-other-jetbrains-ides) for your favorite IDE. We also us [Lombok](https://projectlombok.org). Though not required, you might want to install the [plugin](https://projectlombok.org/setup/overview) as well.
+> **Note:** To make formatting code simple, we recommend installing a [plugin](https://github.com/google/google-java-format#intellij-android-studio-and-other-jetbrains-ides) for your favorite IDE. We also use [Lombok](https://projectlombok.org). Though not required, you might want to install the [plugin](https://projectlombok.org/setup/overview), as well.
 
 # `.git/hooks`
 
@@ -94,7 +94,7 @@ act pull_request --reuse --verbose
 
 # Troubleshooting
 
-There is an issue within the _act_ tool that prevents the _kind_ cluster from being deleted after execution the action.
+There is an issue within the _act_ tool that prevents the _kind_ cluster from being deleted after execution of the action.
 When this condition exists, you will experience the error below.
 
 ```bash
@@ -122,12 +122,12 @@ $ ./gradlew publishToMavenLocal
 1. [Fork](https://github.com/MarquezProject/marquez/fork) and clone the repository
 2. Make sure all tests pass locally: `./gradlew :api:test`
 3. Create a new [branch](#branching): `git checkout -b feature/my-cool-new-feature`
-4. Make change on your cool new branch
+4. Make a change on your cool new branch
 5. Write a test for your change
-6. Make sure `.java` files are formatted: `./gradlew spotlessJavaCheck`
+6. Make sure `.java` files are formatted: `./gradlew spotlessJavaCheck`
 7. Make sure `.java` files contain a [copyright and license header](#copyright--license)
 8. Make sure to [sign you work](#sign-your-work)
-9. Push change to your fork and [submit a pull request](https://github.com/MarquezProject/marquez/compare)
+9. Push the change to your fork and [submit a pull request](https://github.com/MarquezProject/marquez/compare)
 10. Work with project maintainers to get your change reviewed and merged into the `main` branch
 11. Delete your branch
 
@@ -137,17 +137,17 @@ To ensure your pull request is accepted, follow these guidelines:
 * Do your best to have a [well-formed commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) for your change
 * [Keep diffs small](https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6) and self-contained
 * If your change fixes a bug, please [link the issue](https://help.github.com/articles/closing-issues-using-keywords) in your pull request description
-* Any changes to the API reference requires [regenerating](#api-docs) the static `openapi.html` file.
+* Any changes to the API reference require [regenerating](#api-docs) the static `openapi.html` file.
 
 > **Note:** A pull request should generally contain only one commit (use `git commit --amend` and `git push --force` or [squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) existing commits into one).
 
 # Branching
 
-* Use a _group_ at the beginning of your branch names
+* Use a _group_ at the beginning of your branch names:
 
   ```
   feature  Add or expand a feature
-  bug      Fix a bug
+  bug      Fix a bug
   proposal Propose a change
   ```
 
@@ -156,7 +156,7 @@ To ensure your pull request is accepted, follow these guidelines:
   ```
   feature/my-cool-new-feature
   bug/my-bug-fix
-  bug/my-other-bug-fix
+  bug/my-other-bug-fix
   proposal/my-proposal
   ```
 
@@ -167,18 +167,18 @@ To ensure your pull request is accepted, follow these guidelines:
 # Dependencies
 
 We use [renovate](https://github.com/renovatebot/renovate) to manage dependencies for most of our project modules,
-with a couple of exceptions. Renovate automatically detects new dependency versions, and opens pull
-requests to upgrade dependencies in accordance to the [configured rules](https://github.com/MarquezProject/marquez/blob/main/renovate.json).
+with a couple of exceptions. Renovate automatically detects new dependency versions and opens pull
+requests to upgrade dependencies in accordance with the [configured rules](https://github.com/MarquezProject/marquez/blob/main/renovate.json).
 
-The following dependencies are managed manually
+The following dependencies are managed manually:
 
 * _Web code_ - it is challenging to programmatically validate web content
 * _Spark versions_ - the internal query plans parsed by the Spark OpenLineage integration are not stable across Spark versions
 * _Gradle_ - this tool orchestrates the entire build pipeline and was excluded to ensure stability
 
 # Sign Your Work
 
-The _sign-off_ is a simple line at the end of the message for a commit. All commits needs to be signed. Your signature certifies that you wrote the patch or otherwise have the right to contribute the material (see [Developer Certificate of Origin](https://developercertificate.org)):
+The _sign-off_ is a simple line at the end of the message for a commit. All commits need to be signed. Your signature certifies that you wrote the patch or otherwise have the right to contribute the material (see [Developer Certificate of Origin](https://developercertificate.org)):
 
 ```
 This is my commit message
@@ -208,42 +208,42 @@ $ redoc-cli serve spec/openapi.yml
 
 Then browse to: http://localhost:8080
 
-> **Note:** To bundle or serve the API docs, please install [redoc-cli](https://www.npmjs.com/package/redoc-cli).
-
-# `COPYRIGHT` / `LICENSE`
-
-We use [SPDX](https://spdx.dev) for copyright and license information. The following license header **must** be included in all `java,` `bash`, and `py` source files:
-
-`java`
-
-```
-/*
- * Copyright 2018-2022 contributors to the Marquez project
- * SPDX-License-Identifier: Apache-2.0
- */
-```
-
-`bash`
-
-```
-#!/bin/bash
-#
-# Copyright 2018-2022 contributors to the Marquez project
-# SPDX-License-Identifier: Apache-2.0
-```
-
-`py`
-
-```
-# Copyright 2018-2022 contributors to the Marquez project
-# SPDX-License-Identifier: Apache-2.0
+> **Note:** To bundle or serve the API docs, please install [redoc-cli](https://www.npmjs.com/package/redoc-cli).
+
+# `COPYRIGHT` / `LICENSE`
+
+We use [SPDX](https://spdx.dev) for copyright and license information. The following license header **must** be included in all `java,` `bash`, and `py` source files:
+
+`java`
+
+```
+/*
+ * Copyright 2018-2022 contributors to the Marquez project
+ * SPDX-License-Identifier: Apache-2.0
+ */
+```
+
+`bash`
+
+```
+#!/bin/bash
+#
+# Copyright 2018-2022 contributors to the Marquez project
+# SPDX-License-Identifier: Apache-2.0
+```
+
+`py`
+
+```
+# Copyright 2018-2022 contributors to the Marquez project
+# SPDX-License-Identifier: Apache-2.0
 ```
 
 # Resources
 
 * [How to Contribute to Open Source](https://opensource.guide/how-to-contribute)
 * [Using the Fork-and-Branch Git Workflow](https://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow)
 * [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
-* [Keep a Changelog](https://keepachangelog.com)
+* [Keeping a Changelog](https://keepachangelog.com)
 * [Code Review Developer Guide](https://google.github.io/eng-practices/review)
 * [Signing Commits](https://docs.github.com/en/github/authenticating-to-github/signing-commits)