[Bugfix] Zero block_table row tail to fix concurrent variable-length prefill non-determinism (#39589)

[parasol-aser]

Summary

Fixes #39589 — non-deterministic output from concurrent
/v1/completions requests at temperature=0 when prompt lengths differ,
on V1 + FlashInfer. The reporter observed up to 7 distinct output
variants across 20 runs and confirmed the bug was not explained by FP
non-determinism (VLLM_BATCH_INVARIANT=1 did not help), by prefix
caching (repros with --no-enable-prefix-caching), or by the prior
zero-init recycled-block fix (#39283).

Root cause

BlockTable rows are only written up to num_blocks_per_row[row_idx]
and the tail is never cleared. When a persistent-batch row slot is
reused by a shorter request (or becomes the tgt of move_row), stale
block IDs from the previous occupant remain in the tail.

FlashInfer's _copy_page_indices_kernel
(vllm/v1/attention/backends/flashinfer.py) reads
block_table[req_idx, 0:num_blocks] where num_blocks comes from
num_blocks_np = ceil(seq_lens_np / page_size). If that value ever
exceeds the row's logical length (which can happen across concurrent
variable-length prefills under persistent batching), the kernel picks
up block IDs that still point at another live request's KV, and
attention happily reads from the wrong request. This matches every
aspect of the reported symptom: divergence starts at token 0, scales
with concurrency, is insensitive to batch-invariant and prefix caching,
and produces many distinct output variants (not a binary FP flip).

Fix

Zero the tail of the row (up to max_num_blocks_per_req) in:

• BlockTable.append_row: after writing [start:end], zero
  [end:max_num_blocks_per_req]. Comment references [Bug]: KV Cache Read/Write Index Corruption Under Concurrent Prefill of Variable-Length Sequences (vLLM V1, FlashInfer) #39589 so a future
  reader does not "optimize" it away.
• BlockTable.move_row: after copying src → tgt, zero
  [num_blocks:max_num_blocks_per_req] on tgt so the previous (longer)
  occupant of tgt cannot leak.

The change is local, CPU-side (numpy), and paid per-request per-step on
a few hundred to a few thousand int32 stores — negligible against H2D
copy and kernel launch cost. append_row also switches
if not block_ids to if len(block_ids) == 0 so numpy-array arguments
(which raise on truthiness) are handled cleanly.

clear_row already zeros its prefix, swap_row goes through a full
buffer swap, and commit_block_table copies the whole row H2D — all
remain correct under the stronger invariant and need no further
changes.

Why not Hypotheses B / C / D

Following the investigation plan, hypotheses B (DCP seq_lens_np /
paged_kv_last_page_len inconsistency), C (persistent-batch reorder
without row permutation), and D (V1 analog of #36580 slot-mapping pad
bug) were either not the reporter's configuration (no DCP, no cascade),
shown to be internally consistent in current V1 code, or already fixed
upstream. None of them explain the reporter's symptom on their own and
none are touched here.

Duplicate-work check

Per AGENTS.md §1:

gh pr list --repo vllm-project/vllm --state open --search "39589 in:body"
gh pr list --repo vllm-project/vllm --state open --search "block_table append_row tail"

Both return no open PRs as of 2026-04-11. This is not duplicating open
work.

AI assistance disclosure

AI assistance (Claude) was used to investigate, write, and test this
change. The submitter reviewed every changed line and ran the tests
listed below.

Test plan

• tests/v1/worker/test_block_table.py — new unit tests for the
  tail-zero invariant across append_row, add_row, move_row,
  clear_row, including:
  • Single-append, multi-append, and empty-append behavior.
  • Row reuse with a shorter new occupant (the [Bug]: KV Cache Read/Write Index Corruption Under Concurrent Prefill of Variable-Length Sequences (vLLM V1, FlashInfer) #39589 shape).
  • move_row tail clearing when tgt previously held a longer row.
  • clear_row prefix zeroing and no-touch of sibling rows.
  • Hybrid-blocks (use_hybrid_blocks=True) path.
  • Randomized fuzz over 200-step sequences with 4 rows / 16 blocks.
  • Literal transcriptions of PLAN.md §5.1.1/5.1.2/5.1.3 named tests.
• tests/v1/e2e/general/test_concurrent_prefill_determinism.py —
  end-to-end determinism regression at temperature=0 across 20
  trials of a variable-length-prefill batch on
  Qwen/Qwen2.5-0.5B-Instruct with enforce_eager=True.
• .venv/bin/python -m pytest tests/v1/worker/test_block_table.py -v
• .venv/bin/python -m pytest tests/v1/worker/ tests/v1/attention/ -v
• .venv/bin/python -m pytest tests/v1/ -k flashinfer -v
• Upstream reproducer: 200 runs of reporter's repro_minimal.py
  against a local server; expected 0/200 divergences (vs.
  11/200+ before the fix).
• pre-commit run --all-files and
  pre-commit run mypy-3.10 --all-files --hook-stage manual.

Closes #39589

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[gemini-code-assist]

Code Review

This pull request addresses a non-determinism issue (#39589) where stale block IDs in the BlockTable could leak during row reuse or movement, causing FlashInfer kernels to read incorrect KV data. The fix ensures that the tail of each row in the block table is zeroed out during append_row and move_row operations. Comprehensive unit and regression tests have been added to verify this invariant. Review feedback suggests optimizing the move_row implementation by copying the entire row in a single operation and adding a check to prevent redundant work if the source and target indices are identical.

[gemini-code-assist]

Since the invariant established by this PR ensures that the tail of every row (past num_blocks_per_row) is always zero, move_row can be simplified and optimized by copying the entire row from src to tgt in a single contiguous operation. This avoids two separate sliced assignments and is more efficient for the CPU cache and memory controller.

Additionally, if src == tgt, the operation can be skipped entirely.

Suggested change

	block_table_np[tgt, :num_blocks] = block_table_np[src, :num_blocks]
	# Zero the tail of tgt so stale block IDs from tgt's previous
	# (longer) occupant cannot leak past num_blocks_per_row[tgt].
	# See vllm-project/vllm#39589.
	block_table_np[tgt, num_blocks : self.max_num_blocks_per_req] = 0
	if src == tgt:
	return
	block_table_np[tgt, :] = block_table_np[src, :]
	self.num_blocks_per_row[tgt] = num_blocks