Skip to content

bug_triage.md

Latest commit

 

History

History
161 lines (119 loc) · 8.04 KB

bug_triage.md

File metadata and controls

161 lines (119 loc) · 8.04 KB

Bug Triage Guide

This guide describes how we prioritize and triage bugs in the Comet project. The goal is to ensure that the most impactful bugs — especially correctness issues that produce wrong results — are identified and addressed before less critical issues.

Priority Labels

Every bug should have exactly one priority label. When filing or triaging a bug, apply the appropriate label from the table below.

Label Color Description Examples
priority:critical Red Data corruption, silent wrong results, security vulnerabilities Wrong aggregation results, FFI data corruption, incorrect cast output
priority:high Orange Crashes, panics, segfaults, major functional breakage affecting production workloads Native engine panic, JVM segfault, NPE on supported code path
priority:medium Yellow Functional bugs, performance regressions, broken features that have workarounds Missing expression support, writer feature gaps, excessive spilling
priority:low Green Minor issues, test-only failures, tooling, CI flakes, cosmetic issues Flaky CI test, build script edge case, documentation generator bug

How to Choose a Priority

Use this decision tree:

  1. Can this bug cause silent wrong results? If yes → priority:critical. These are the most dangerous bugs because users may not notice the incorrect output.
  2. Does this bug crash the JVM or native engine? If yes → priority:high. Crashes are disruptive but at least visible to the user.
  3. Does this bug break a feature or cause significant performance degradation? If yes → priority:medium. The user can work around it (e.g., falling back to Spark) but it impacts the value of Comet.
  4. Everything else → priority:low. Test failures, CI issues, tooling, and cosmetic problems.

Escalation Triggers

A bug should be escalated to a higher priority if:

  • A priority:high crash is discovered to also produce wrong results silently in some cases → escalate to priority:critical
  • A priority:medium bug is reported by multiple users or affects a common workload → consider escalating to priority:high
  • A priority:low CI flake is blocking PR merges consistently → escalate to priority:medium

Area Labels

Area labels indicate which subsystem is affected. A bug may have multiple area labels. These help contributors find bugs in their area of expertise.

Label Description
area:writer Native writer (Parquet and other formats)
area:shuffle Shuffle (JVM and native)
area:aggregation Hash aggregates, aggregate expressions
area:scan Data source scan (Parquet, CSV, Iceberg)
area:expressions Expression evaluation
area:ffi Arrow FFI / JNI boundary
area:ci CI/CD, GitHub Actions, build tooling

The following pre-existing labels also serve as area indicators: spark 4, spark sql tests.

Triage Process

Every new issue is automatically labeled with requires-triage when it is opened. This makes it easy to find issues that have not yet been triaged by filtering on that label. Once an issue has been triaged, remove the requires-triage label and apply the appropriate priority and area labels.

For New Issues

When a new bug is filed:

  1. Reproduce or verify the issue if possible. If the report lacks reproduction steps, ask the reporter for more details.
  2. Assess correctness impact first. Ask: "Could this produce wrong results silently?" This is more important than whether it crashes.
  3. Apply a priority label using the decision tree above.
  4. Apply area labels to indicate the affected subsystem(s).
  5. Apply good first issue if the fix is likely straightforward and well-scoped.
  6. Remove the requires-triage label to indicate triage is complete.

For Existing Bugs

Periodically review open bugs to ensure priorities are still accurate:

  • Has a priority:medium bug been open for a long time with user reports? Consider escalating.
  • Has a priority:high bug been fixed by a related change? Close it.
  • Are there clusters of related bugs that should be tracked under an EPIC?

Prioritization Principles

  1. Correctness over crashes. A bug that silently returns wrong results is worse than one that crashes, because crashes are at least visible.
  2. User-reported over test-only. A bug hit by a real user on a real workload takes priority over one found only in test suites.
  3. Core path over experimental. Bugs in widely-used expressions and operators take priority over bugs in experimental features.
  4. Production safety over feature completeness. Fixing a data corruption bug is more important than adding support for a new expression.

Common Bug Categories

Correctness Bugs (priority:critical)

These are bugs where Comet produces different results than Spark without any error or warning. Examples include:

  • Incorrect cast behavior (e.g., negative zero to string)
  • Aggregate functions ignoring configuration (e.g., ignoreNulls)
  • Data corruption in FFI boundary (e.g., boolean arrays with non-zero offset)
  • Type mismatches between partial and final aggregation stages

When fixing correctness bugs, always add a regression test that verifies the output matches Spark.

Crash Bugs (priority:high)

These are bugs where the native engine panics, segfaults, or throws an unhandled exception. Common patterns include:

  • All-scalar inputs: Some expressions assume at least one columnar input and panic when all inputs are literals (e.g., when ConstantFolding is disabled)
  • Type mismatches: Downcasting to the wrong Arrow array type
  • Memory safety: FFI boundary issues, unaligned arrays, GlobalRef lifecycle

Aggregate Planning Bugs

Several bugs relate to how Comet plans hash aggregates across stage boundaries. The key issue is that Spark's AQE may materialize a Comet partial aggregate but then run the final aggregate in Spark (or vice versa), and the intermediate formats may not be compatible. See the EPIC #2892 for the full picture.

Native Writer Bugs

The native Parquet writer has a cluster of known test failures tracked as individual issues (#3417–#3430). These are lower priority since the native writer is still maturing, but they should be addressed before the writer is promoted to production-ready status.

How to Help with Triage

Triage is a valuable contribution that doesn't require writing code. You can help by:

  • Reviewing new issues and suggesting a priority label
  • Reproducing reported bugs and adding details
  • Identifying duplicate issues
  • Linking related issues together
  • Testing whether old bugs have been fixed by recent changes
⚡