Skip to content

CI: check-generated-code failing on main due to uv 0.10.5+ pylock.toml filtering change #3032

@jiridanek

Description

@jiridanek

Summary

The check-generated-code CI job has been failing on main (and all PRs) since ~Feb 23 2026. The job regenerates lock files via bash ci/generate_code.sh and then checks for uncommitted changes. Six pylock.toml files are regenerated differently by CI than what is committed, causing the check to fail.

This is not caused by any PR — it is a repo-wide issue triggered by a behavior change in uv 0.10.5 (released 2026-02-23).

Failing files

M codeserver/ubi9-python-3.12/uv.lock.d/pylock.cpu.toml
M jupyter/datascience/ubi9-python-3.12/uv.lock.d/pylock.cpu.toml
M jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml
M jupyter/trustyai/ubi9-python-3.12/uv.lock.d/pylock.cpu.toml
M runtimes/datascience/ubi9-python-3.12/uv.lock.d/pylock.cpu.toml
M runtimes/rocm-tensorflow/ubi9-python-3.12/pylock.toml

Example CI run on main: https://github.com/opendatahub-io/notebooks/actions/runs/22481514656/job/65121631498

Example CI run on a PR: https://github.com/opendatahub-io/notebooks/actions/runs/22482916427/job/65125645371

Root cause

Our CI workflow (.github/workflows/code-quality.yaml) installs uv with version: "latest" via astral-sh/setup-uv@v7. The pylocks_generator.sh script runs uv pip compile --universal --format pylock.toml, and a recent uv release changed what --universal includes in pylock.toml output.

What changed in uv

astral-sh/uv#18081 — "Filter pylock.toml wheels by tags and requires-python" (merged 2026-02-18, released in uv 0.10.5 on 2026-02-23).

This PR unified the wheel filtering logic between uv.lock and pylock.toml. Previously, pylock.toml in --universal mode included all resolved wheels with no filtering. After this PR, even in --universal mode, uv now:

  1. Filters by requires-python — removes wheels for Python versions outside the project's requirement (e.g., cp39, cp310, cp311 wheels when requires-python >= "3.12").
  2. Filters by platform marker disjointness — removes wheels for platforms provably unreachable based on a dependency's marker expression (e.g., macOS/Windows wheels for Linux-only deps, s390x wheels for packages with architecture-restricting markers).

Full timeline of --universal + pylock.toml behavior

uv version Date PR Impact on --universal pylock.toml
< 0.9.16 No filtering at all — ALL wheels dumped into output
0.9.16 2025-12-06 #16956 Added tag filtering, but explicitly skipped in --universal mode (tags.is_none_or(...) passes all wheels through). Only filters when NOT using --universal.
0.9.22 2026-01-06 #17317 Extended architecture filtering (ppc64le, s390x, riscv64, etc.) but only for uv.lock, not pylock.toml.
0.10.5 2026-02-23 #18081 Breaking change for us. Extracted is_wheel_unreachable() from uv.lock code path and reused it in pylock.toml. Now filters even with --universal based on requires-python and marker disjointness.

What the diff looks like

The regenerated pylocks remove wheels for:

  • macOS and Windows platforms (e.g., cp312-cp312-macosx_*, cp312-cp312-win_*)
  • Older Python versions not matching requires-python (cp39, cp310, cp311)
  • s390x architecture for some Red Hat-index packages (greenlet, grpcio, msgpack, wrapt) where markers make it unreachable

This is arguably the correct behavior — these wheels were never needed in our Linux-only container builds.

Why it doesn't fail locally

Developers with uv < 0.10.5 (e.g., uv 0.10.4 from Homebrew) generate pylocks that include all wheels. CI with uv 0.10.5+ generates pylocks with the filtered set, producing a diff.

Note about pylocks_generator.sh comment

Line 276-278 of scripts/pylocks_generator.sh says:

# The behavior has changed in uv 0.9.17 (https://github.com/astral-sh/uv/pull/16956)

Minor correction: PR #16956 was released in uv 0.9.16 (not 0.9.17), and it did NOT change --universal behavior for pylock.toml — the tag filtering is explicitly bypassed in universal mode. The actual change affecting our --universal output is PR #18081 in uv 0.10.5.

Suggested fix

Regenerate the pylocks with uv >= 0.10.5:

# Update uv first
# e.g.: brew upgrade uv, or: pip install -U uv, or: curl -LsSf https://astral.sh/uv/install.sh | sh

# Then regenerate
bash ci/generate_code.sh

# Commit the updated pylocks
git add -A '*.toml'
git commit -m "chore: regenerate pylock.toml files with uv 0.10.5+ filtering"

Alternatively (or additionally), pin the uv version in .github/workflows/code-quality.yaml to avoid future surprises from version: "latest":

- uses: astral-sh/setup-uv@v7
  with:
    version: "0.10.6"  # pin to avoid breaking on new releases

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    ✅Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions