Skip to content

Ensure job data in lineage query is not null or empty #2253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wslulciuc merged 9 commits into from
Dec 13, 2022
Merged

Ensure job data in lineage query is not null or empty #2253

Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b40de1e
Ensure job data in lineage query is not null or empty
wslulciuc Nov 18, 2022
7058bba
continued: Ensure job data in lineage query is not null or empty
wslulciuc Dec 5, 2022
7744c57
Merge branch 'main' into bug/bind-list-error-lineage-dao
wslulciuc Dec 5, 2022
7eb4393
Merge branch 'main' into bug/bind-list-error-lineage-dao
wslulciuc Dec 12, 2022
41594b8
Add toLineageWithOrphanDataset() to build orphan graph
wslulciuc Dec 12, 2022
231139d
continued: Add toLineageWithOrphanDataset() to build orphan graph
wslulciuc Dec 12, 2022
5e4aeea
continued: Add toLineageWithOrphanDataset() to build orphan graph
wslulciuc Dec 12, 2022
5cd3207
Return orphan graph on failed lookup for job when dataset nodeID prov…
wslulciuc Dec 12, 2022
14be3bc
Fix broken test for 404 on nodeID not found
wslulciuc Dec 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions api/src/main/java/marquez/service/LineageService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import com.google.common.base.Functions;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
Expand Down Expand Up @@ -48,14 +49,27 @@ public LineageService(LineageDao delegate, JobDao jobDao) {

// TODO make input parameters easily extendable if adding more options like 'withJobFacets'
public Lineage lineage(NodeId nodeId, int depth, boolean withRunFacets) {
log.debug("Attempting to get lineage for node '{}' with depth '{}'", nodeId.getValue(), depth);
Optional<UUID> optionalUUID = getJobUuid(nodeId);
if (optionalUUID.isEmpty()) {
throw new NodeIdNotFoundException("Could not find node");
throw new NodeIdNotFoundException(String.format("Node '%s' not found!", nodeId.getValue()));
}
Comment thread
wslulciuc marked this conversation as resolved.
UUID job = optionalUUID.get();

log.debug("Attempting to get lineage for job '{}'", job);
Set<JobData> jobData = getLineage(Collections.singleton(job), depth);

// Ensure job data is not empty, an empty set cannot be passed to LineageDao.getCurrentRuns() or
// LineageDao.getCurrentRunsWithFacets().
if (jobData.isEmpty()) {
// Log warning, then return an empty lineage graph; a graph must contain at most one
// job->dataset relationship.
log.warn(
"Failed to get lineage for job '{}' associated with node '{}', returning empty graph...",
job,
nodeId.getValue());
return toLineage(Sets.newHashSet(), Sets.newHashSet());
Comment thread
wslulciuc marked this conversation as resolved.
Outdated
}

List<Run> runs =
withRunFacets
? getCurrentRunsWithFacets(
Expand Down
⚡