test(rate-limiter): toggle convergence-time probe + TTL-expiry cycle

[gandhipratik203]

Summary

Two integration tests covering rate-limiter behaviour under runtime mode toggle via the admin API. Marked draft because they raise design questions about the rate limiter's toggle-latency contract that need team input before a final shape lands.

The rate limiter is the only stateful plugin we ship — counter state lives in Redis, mode is embedded in a per-worker cached plugin instance. That makes its response to a runtime mode toggle eventually consistent, not instantaneous, in a way the other supported plugins (Output Length Guard, Secrets Detection) aren't.

What's in this PR

Two tests, both currently passing:

TestRateLimiterToggleAfterTTLExpiry — correctness contract

tests/integration/test_rate_limiter_dynamic_behavior.py

Three-phase cycle: enforce → wait past Redis counter TTL → disabled → wait → enforce. Validates that re-enforcement works after a quiet period long enough for counters to age out. Tagged @pytest.mark.slow because of two ~75s sleeps.

TestRateLimiterToggleLiveState::test_measure_convergence_time_for_mode_toggle — latency probe

tests/integration/test_rate_limiter_toggle_live_state.py

Flips mode, then polls every second until traffic actually reflects the new mode, recording the elapsed time and the per-poll trajectory. No threshold-based assertion on the convergence time — the test fails only if convergence never happens within MAX_CONVERGENCE_S. Useful both as a regression pin (sudden slowdown caught) and as a measurement tool.

Sample observation

One local run, single replica × 24 workers, post-current-main image:

enforce → disabled : converged in 22.4s
  t= 0.00s   4/6 allowed,  2/6 blocked
  t= 2.84s   2/6 allowed,  4/6 blocked
  t= 7.20s   1/6 allowed,  5/6 blocked  (got worse before better)
  t=11.30s   0/6 allowed,  6/6 blocked
  t=15.42s   5/6 allowed,  1/6 blocked  (oscillates)
  t=20.86s   4/6 allowed,  2/6 blocked
  t=22.37s   6/6 allowed,  0/6 blocked  ← converged

disabled → enforce : converged in 9.7s

The ~22s disabled-convergence with mid-trajectory oscillation is suspicious — looks like pub/sub-driven invalidation is correct for most workers within a few seconds, but a fraction takes much longer to converge, possibly relying on the 30s TTL-based cache safety net rather than the pub/sub eviction. Convergence happens; just not as fast as expected.

Why this is different from stateless plugins

Stateless plugins decide based only on the current request — the moment a worker's mode flips to disabled, every subsequent request reflects it. Rate limiter decides based on (mode, persistent counter state). When mode flips, the counter is still hot in Redis, so any worker still serving requests under stale-mode-enforce will still block them. The convergence time is a property of how long it takes every worker's plugin-manager cache to rebuild with the new mode.

Open design questions for review

1. Toggle latency expectation. What's the SLA we want to commit to? "Within N seconds" feels right, but what's N? Operators flipping a plugin off in an incident response need to know.
2. Counter window on disabled → enforce. Today we don't reset the window — flipping back to enforce on an already-hot bucket immediately blocks. Is that the right call? Some operators might expect a fresh window on re-enable. Either is defensible.
3. Should disabled mode actually reset Redis state? Today it stops checking counters but doesn't clear them. Same question from the other direction.
4. Eventual-consistency contract for stateful plugins more generally. Rate limiter is the only stateful one today, but cpex-plugins#49 has follow-ups (quota accounting, cascade limits) that would be similarly stateful. Worth deciding the convergence-time SLA at the framework level once.

What this PR is not

Not a code fix. The rate limiter works; the question is how it behaves under toggle and what we want that behaviour to be. The tests document current behaviour and provide a measurement harness for any future fix.

How to run

Prerequisites

• Local docker-compose stack from this repo brought up via make compose-up (3 gateway replicas behind nginx on :8080, Redis, Postgres, fast-test-server registered).
• Gateway image built from current main — run make docker-build if your local image is stale.
• Default dev credentials and JWT secret from your local .env (the .env.example defaults work for this test).
• uv sync --extra plugins so cpex-rate-limiter is importable in the test env.

Run

# convergence probe — ~1 minute, emits the trajectory on every run
uv run pytest tests/integration/test_rate_limiter_toggle_live_state.py \
  --with-integration -s -v

# TTL-expiry cycle — ~3 minutes due to two ~75s sleeps; tagged @pytest.mark.slow
uv run pytest \
  tests/integration/test_rate_limiter_dynamic_behavior.py::TestRateLimiterToggleAfterTTLExpiry \
  --with-integration -s -v

Both tests skip cleanly if http://localhost:8080/health isn't reachable.

---

Update — empirical evidence and layered coverage

Since the original draft, this branch has grown into a layered toggle-behaviour story. Adding here, not editing the above — the original sections remain the right snapshot of the problem statement and findings at the time the PR was opened.

Layered toggle-behaviour coverage now on this branch

Layer	Pin	File
A	reload_tenant → manager.shutdown invocation	tests/unit/mcpgateway/plugins/framework/test_tenant_plugin_manager_tool_scoped.py
A-init	reload_tenant → new manager.initialize invocation	same
B-0	manager-mediated plugin wipe via direct factory.reload_tenant	tests/integration/test_plugin_manager_wipe_on_disable.py
B-0.5	publisher↔subscriber round-trip via real Redis pub/sub	same
C (existing)	end-to-end convergence probe via HTTP+MCP	tests/integration/test_rate_limiter_toggle_live_state.py
Multi-replica probe	end-to-end convergence via Redis pub/sub only, no HTTP amplification	tests/integration/test_rate_limiter_toggle_via_redis_pubsub.py

Empirical answer to design question #3

Question #3 asked whether disabled mode should clear Redis counter state. The answer landed on "yes, clear them," and the implementation lives on cpex-plugins branch feat/rate-limiter-wipe-on-disable-only. Running the new multi-replica pub/sub probe against a wipe-enabled gateway image (3 replicas × 24 workers, plain docker-compose stack):

Convergence trajectory (counter_exists per poll):
  t=  0.00s  counter_exists=True
  t=  0.11s  counter_exists=False  ← converged

✓ Wipe converged in 0.11s

vs. the ~22s baseline observed in the existing live-state probe at the top of this PR — a ~170× improvement on the disabled-direction convergence. Note the wipe doesn't accelerate the cache rebuild; it makes the rebuild's user-visible latency irrelevant — a stale-mode worker with empty counters cannot block any request.

Pre-conditions discovered while empirically running

Two things had to be in place for the wipe path to actually fire from a clean stack — worth flagging on the cpex-plugins wipe-on-disable PR description as deployment caveats:

1. Plugin must be loaded for the wipe to fire. plugins/config.yaml ships rate-limiter with mode: "disabled", and the framework loader skips disabled plugins. The probe first toggles to enforce via the admin API so the plugin is genuinely loaded.
2. At least one cached manager required. factory.invalidate_all() iterates only cached managers; a fresh gateway with zero traffic has zero cached managers, so the broadcast triggers no reload_tenant and no plugin.shutdown. The probe sends a 20-request warm-up burst before the disable to ride gunicorn's load balancer across multiple workers.

Status of original design questions

• Configure Renovate #1 (toggle latency SLA): with wipe-on-disable enabled, the enforce → disabled direction converges in ~0.1s. The user-visible SLA could plausibly be "< 1 second" instead of "within N seconds" — but the disabled → enforce direction is unchanged by the wipe and still depends on cache rebuild propagation. Worth team input on whether one SLA covers both.
• Open Source release of MCP Context Forge - MCP Gateway #2 (fresh window on re-enable): transitively answered by Update Makefile and pyproject.toml with packaging steps #3 — wipe-on-disable means re-enable always starts with empty counters, so the operator gets a fresh window for free.
• Update Makefile and pyproject.toml with packaging steps #3 (disabled mode resets Redis state): answered yes, implementation in cpex-plugins linked above. This PR's tests are the framework-side coverage of that contract.
• Add testing documentation #4 (eventual-consistency contract for stateful plugins): still open at the framework level, separate from this PR.

[araujof]

DO NOT MERGE before #3754 is merged.
Touches files under mcpgateway/plugins

[gandhipratik203]

@araujof Noted. Won't merge with mcpgateway/plugins changes. Will be removing the files under mcpgateway/plugins. So the PR will only have the changes for tests/integration.