Skip to content

Commit 9bb8983

Browse files
Titus-von-Koellerclaude
Titus-von-Koeller
and
claude
committed
ci: add nightly integration tests for transformers, accelerate, peft
Adds a new nightly workflow that runs each downstream library's own bnb-specific test suite against the latest main bnb wheel (from the continuous-release_main pre-release). Catches breakage in HF downstream integrations before it reaches users. Architecture: - Three parallel test jobs (transformers on T4, accelerate and peft on A10 to match each project's own CI) - Each produces JUnit XML uploaded as an artifact - Consolidated report job downloads all XMLs, generates a markdown summary, writes to $GITHUB_STEP_SUMMARY, and uploads artifacts for inspection - Slack posting is stubbed out until SLACK_API_TOKEN is provisioned Triggers: - workflow_dispatch (manual, available after merge) - pull_request (runs automatically when workflow/report script changes) - schedule (commented out; enable in follow-up PR once stable) The report script (scripts/integration_test_report.py) parses JUnit XML, produces a markdown summary, and can post Slack Block Kit messages with threaded per-suite failure details (diffusers-style consolidated report). Full design rationale and implementation plan in agents/integration_tests_guide.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 74994ef commit 9bb8983

3 files changed

Lines changed: 1119 additions & 0 deletions

File tree

‎.github/workflows/tests-integration-nightly.yml‎

Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
name: Integration Tests (Downstream)
2+
3+
# Nightly smoke tests: run the bnb-specific test suites from transformers,
4+
# accelerate, and peft against the latest main-branch bnb wheel. Catches
5+
# downstream breakage before it reaches users.
6+
#
7+
# bnb is installed from the `continuous-release_main` pre-release which
8+
# python-package.yml publishes on every push to main — no duplicate build.
9+
#
10+
# See agents/integration_tests_guide.md for background.
11+
12+
on:
13+
workflow_dispatch:
14+
pull_request:
15+
paths:
16+
- '.github/workflows/tests-integration-nightly.yml'
17+
- 'scripts/integration_test_report.py'
18+
# schedule:
19+
# - cron: "30 3 * * *" # enable once stable; runs after python-package + tests-nightly
20+
21+
concurrency:
22+
group: ${{ github.workflow }}-${{ github.ref }}
23+
cancel-in-progress: true
24+
25+
env:
26+
PYTHON_VERSION: "3.10"
27+
TORCH_VERSION: "2.9.1"
28+
PYPI_INDEX: "https://download.pytorch.org/whl/cu128"
29+
BNB_WHEEL_URL: "https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-manylinux_2_24_x86_64.whl"
30+
31+
jobs:
32+
# ─── Downstream test jobs ─────────────────────────────────────────────────
33+
# Each job:
34+
# 1. Installs torch, then bnb from the continuous-release wheel
35+
# 2. Installs the downstream lib (latest release from PyPI)
36+
# 3. Clones the matching version tag for the test files
37+
# 4. Runs the library's bnb-specific tests with --junitxml
38+
# 5. Uploads the XML + full log as an artifact for the report job
39+
#
40+
# Runner matching rationale (see integration_tests_guide.md):
41+
# transformers CI runs on T4 → we use T4
42+
# accelerate / peft CI runs on L4 → closest bnb equivalent is A10
43+
# This reduces spurious failures from expected values calibrated on their runners.
44+
45+
test-transformers:
46+
name: Transformers bnb tests
47+
if: github.repository == 'bitsandbytes-foundation/bitsandbytes'
48+
runs-on: bandb-aws-g4dn-4xlarge-plus-use1-public-80 # T4
49+
steps:
50+
- name: Show GPU information
51+
run: nvidia-smi
52+
53+
- uses: actions/checkout@v4
54+
55+
- name: Setup Python
56+
uses: actions/setup-python@v5
57+
with:
58+
python-version: ${{ env.PYTHON_VERSION }}
59+
60+
- name: Install torch + bnb (from continuous-release)
61+
run: |
62+
pip install torch==${TORCH_VERSION} --index-url ${PYPI_INDEX}
63+
pip install --force-reinstall --no-deps ${BNB_WHEEL_URL}
64+
65+
- name: Install transformers and clone matching tag
66+
run: |
67+
pip install transformers accelerate
68+
TRANSFORMERS_VERSION=$(pip show transformers | awk '/^Version:/ {print $2}')
69+
echo "Installed transformers v${TRANSFORMERS_VERSION}"
70+
git clone --depth=1 --branch "v${TRANSFORMERS_VERSION}" \
71+
https://github.com/huggingface/transformers.git /tmp/transformers
72+
73+
- name: Show environment
74+
run: |
75+
pip list
76+
python -m torch.utils.collect_env
77+
78+
- name: Run transformers bnb tests
79+
working-directory: /tmp/transformers
80+
env:
81+
RUN_SLOW: "1"
82+
CUDA_VISIBLE_DEVICES: "0"
83+
run: |
84+
mkdir -p ${GITHUB_WORKSPACE}/reports
85+
python -m pytest tests/quantization/bnb/ \
86+
-v \
87+
-k "not MultiGpu and not multi_gpu" \
88+
--junitxml=${GITHUB_WORKSPACE}/reports/transformers.xml \
89+
-o junit_logging=all \
90+
2>&1 | tee ${GITHUB_WORKSPACE}/reports/transformers.log
91+
92+
- name: Upload JUnit XML and log
93+
if: always()
94+
uses: actions/upload-artifact@v4
95+
with:
96+
name: reports-transformers
97+
path: reports/
98+
retention-days: 7
99+
100+
test-accelerate:
101+
name: Accelerate bnb tests
102+
if: github.repository == 'bitsandbytes-foundation/bitsandbytes'
103+
runs-on: bandb-aws-g5-4xlarge-plus-use1-public-80 # A10
104+
steps:
105+
- name: Show GPU information
106+
run: nvidia-smi
107+
108+
- uses: actions/checkout@v4
109+
110+
- name: Setup Python
111+
uses: actions/setup-python@v5
112+
with:
113+
python-version: ${{ env.PYTHON_VERSION }}
114+
115+
- name: Install torch + bnb (from continuous-release)
116+
run: |
117+
pip install torch==${TORCH_VERSION} --index-url ${PYPI_INDEX}
118+
pip install --force-reinstall --no-deps ${BNB_WHEEL_URL}
119+
120+
- name: Install accelerate and clone matching tag
121+
run: |
122+
pip install accelerate transformers
123+
ACCELERATE_VERSION=$(pip show accelerate | awk '/^Version:/ {print $2}')
124+
echo "Installed accelerate v${ACCELERATE_VERSION}"
125+
git clone --depth=1 --branch "v${ACCELERATE_VERSION}" \
126+
https://github.com/huggingface/accelerate.git /tmp/accelerate
127+
128+
- name: Show environment
129+
run: |
130+
pip list
131+
python -m torch.utils.collect_env
132+
133+
- name: Run accelerate bnb tests
134+
working-directory: /tmp/accelerate
135+
env:
136+
RUN_SLOW: "1"
137+
CUDA_VISIBLE_DEVICES: "0"
138+
run: |
139+
mkdir -p ${GITHUB_WORKSPACE}/reports
140+
python -m pytest tests/test_quantization.py \
141+
-s -v \
142+
-k "not multi_device" \
143+
--junitxml=${GITHUB_WORKSPACE}/reports/accelerate.xml \
144+
-o junit_logging=all \
145+
2>&1 | tee ${GITHUB_WORKSPACE}/reports/accelerate.log
146+
147+
- name: Upload JUnit XML and log
148+
if: always()
149+
uses: actions/upload-artifact@v4
150+
with:
151+
name: reports-accelerate
152+
path: reports/
153+
retention-days: 7
154+
155+
test-peft:
156+
name: PEFT bnb tests
157+
if: github.repository == 'bitsandbytes-foundation/bitsandbytes'
158+
runs-on: bandb-aws-g5-4xlarge-plus-use1-public-80 # A10
159+
steps:
160+
- name: Show GPU information
161+
run: nvidia-smi
162+
163+
- uses: actions/checkout@v4
164+
165+
- name: Setup Python
166+
uses: actions/setup-python@v5
167+
with:
168+
python-version: ${{ env.PYTHON_VERSION }}
169+
170+
- name: Install torch + bnb (from continuous-release)
171+
run: |
172+
pip install torch==${TORCH_VERSION} --index-url ${PYPI_INDEX}
173+
pip install --force-reinstall --no-deps ${BNB_WHEEL_URL}
174+
175+
- name: Install peft and clone matching tag
176+
run: |
177+
pip install peft transformers accelerate datasets
178+
PEFT_VERSION=$(pip show peft | awk '/^Version:/ {print $2}')
179+
echo "Installed peft v${PEFT_VERSION}"
180+
git clone --depth=1 --branch "v${PEFT_VERSION}" \
181+
https://github.com/huggingface/peft.git /tmp/peft
182+
183+
- name: Show environment
184+
run: |
185+
pip list
186+
python -m torch.utils.collect_env
187+
188+
- name: Run peft bnb tests
189+
working-directory: /tmp/peft
190+
env:
191+
IS_GITHUB_CI: "1"
192+
CUDA_VISIBLE_DEVICES: "0"
193+
run: |
194+
mkdir -p ${GITHUB_WORKSPACE}/reports
195+
python -m pytest \
196+
-m "single_gpu_tests and bitsandbytes" \
197+
tests/test_gpu_examples.py tests/test_common_gpu.py \
198+
-v \
199+
--junitxml=${GITHUB_WORKSPACE}/reports/peft.xml \
200+
-o junit_logging=all \
201+
2>&1 | tee ${GITHUB_WORKSPACE}/reports/peft.log
202+
203+
- name: Upload JUnit XML and log
204+
if: always()
205+
uses: actions/upload-artifact@v4
206+
with:
207+
name: reports-peft
208+
path: reports/
209+
retention-days: 7
210+
211+
# ─── Consolidated report ──────────────────────────────────────────────────
212+
# Runs after all three test jobs finish (success or failure).
213+
# Downloads the JUnit XMLs, runs our report script, writes to the job
214+
# summary, and uploads the consolidated report as an artifact.
215+
# Slack posting is intentionally disabled until the SLACK_API_TOKEN secret
216+
# is provisioned — we'll iterate on the Slack payload locally against real
217+
# artifacts first.
218+
219+
report:
220+
name: Consolidated report
221+
needs: [test-transformers, test-accelerate, test-peft]
222+
if: always() && github.repository == 'bitsandbytes-foundation/bitsandbytes'
223+
runs-on: ubuntu-22.04
224+
steps:
225+
- uses: actions/checkout@v4
226+
227+
- name: Setup Python
228+
uses: actions/setup-python@v5
229+
with:
230+
python-version: ${{ env.PYTHON_VERSION }}
231+
232+
- name: Download all report artifacts
233+
uses: actions/download-artifact@v4
234+
with:
235+
path: artifacts
236+
pattern: reports-*
237+
238+
- name: Consolidate XMLs into reports/
239+
run: |
240+
mkdir -p reports
241+
# Each artifact lands in artifacts/reports-<suite>/ — flatten to reports/<suite>.xml
242+
find artifacts -name '*.xml' -exec cp {} reports/ \;
243+
find artifacts -name '*.log' -exec cp {} reports/ \;
244+
ls -la reports/
245+
246+
- name: Generate consolidated report
247+
run: |
248+
python scripts/integration_test_report.py \
249+
--reports-dir reports/ \
250+
--output consolidated_report.md \
251+
--dry-run
252+
253+
- name: Write to job summary
254+
if: always()
255+
run: |
256+
cat consolidated_report.md >> $GITHUB_STEP_SUMMARY
257+
258+
- name: Upload consolidated report
259+
if: always()
260+
uses: actions/upload-artifact@v4
261+
with:
262+
name: consolidated-report
263+
path: |
264+
consolidated_report.md
265+
reports/
266+
retention-days: 14
267+
268+
# TODO: enable once SLACK_API_TOKEN secret is configured
269+
# - name: Post to Slack
270+
# if: always()
271+
# env:
272+
# SLACK_API_TOKEN: ${{ secrets.SLACK_API_TOKEN }}
273+
# run: |
274+
# pip install slack_sdk
275+
# python scripts/integration_test_report.py \
276+
# --reports-dir reports/ \
277+
# --slack-channel bnb-ci-nightly

0 commit comments

Comments
 (0)
⚡