feat(rate-limiter): pluggable algorithms with Rust-backed execution engine, benchmarks, and validation

[gandhipratik203]

Summary

Consolidates and extends the rate limiter plugin with a Rust-backed execution engine, carrying forward the Python-side foundation work from #3783 (pluggable algorithms, tenant isolation, correctness fixes) and replacing the Python algorithm/backend implementation with a high-performance Rust engine for both memory and redis backends.

The Rust engine is the preferred execution path — all algorithm execution, backend dispatch, result aggregation, and response construction live in Rust when the rate_limiter_rust PyO3 extension is installed. Python retains ownership of plugin lifecycle, hook integration, request-context extraction, and config validation. The full Python algorithm and backend implementation is retained as a fallback when the Rust extension is unavailable or when RATE_LIMITER_FORCE_PYTHON=1 is set.

The Rust engine exposes a high-level check() API that reduces the Python-Rust boundary to a single call returning pre-built response dicts. This keeps the existing plugin integration model intact while reducing request-path overhead and preserving shared-counter semantics for multi-instance deployments.

---

Gaps closed

Python foundation

Gap 1 (HIGH) — No algorithm choice: only fixed_window was available — no way to handle boundary bursts or bursty workloads. Fixed by introducing a strategy pattern with three selectable algorithms (fixed_window, sliding_window, token_bucket) configurable via the algorithm: field. All three run on both memory and redis backends; the Redis backend was extended with atomic Lua scripts for sliding_window and token_bucket to maintain cluster-wide enforcement across instances.

test_fixed_window_burst_at_boundary (xfail) documents the boundary burst as a known trade-off for users who stay on fixed_window; sliding_window eliminates it entirely.

Gap 2 (HIGH) — by_tenant cross-throttle: tenant_id=None fell back to a shared "tenant:default" bucket, causing unrelated users to throttle each other across the entire deployment. Fixed in two places: (1) the plugin now skips the by_tenant check entirely when tenant_id is None instead of inventing a phantom bucket; (2) mcpgateway/auth.py now propagates request.state.team_id → global_context.tenant_id unconditionally via _propagate_tenant_id(), called at all four return paths in get_current_user(), so single-team API tokens correctly populate tenant context for rate limiting regardless of the include_user_info setting.

Gap 3 (MEDIUM) — Sliding window memory leak in sweep(): stale-but-non-empty timestamp lists were never evicted from _store because sweep() only removed empty lists. Fixed by embedding the window size in each store key ("{key}:{window}") so staleness is computable at sweep time, and rewriting sweep() to evict any key where all timestamps fall outside the window.

Rust execution engine

Gap 4 (HIGH) — Python hot path overhead: every request still paid Python-side rate evaluation costs, including per-dimension orchestration, repeated backend calls, and response metadata construction via individual PyO3 attribute accesses. Fixed by introducing a Rust RateLimiterEngine with a single check() call per hook invocation. The engine builds dimension keys internally from its pre-parsed config, evaluates all active dimensions in one batch, and returns pre-built Python dicts for HTTP headers and plugin metadata — eliminating ~18-25 PyO3 boundary crossings per request (depending on the number of active dimensions).

Gap 5 (HIGH) — Rust acceleration was limited to the in-memory backend: Redis-backed deployments, which are the correctness-critical path for multi-instance enforcement, still relied on the Python backend. Fixed by adding a Redis backend to the Rust engine. Rust now owns the Redis connection and executes batch Lua scripts directly, preserving shared-counter behavior across replicas.

Gap 6 (MEDIUM) — Rate-limit computation, response shaping, and plugin policy handling were still interleaved in the Python implementation. Fixed by making the Python wrapper policy-only: it validates config, normalizes context, and invokes the engine. Algorithm execution, backend dispatch, result aggregation, and response dict construction now all live in Rust. The Python hot path is reduced to a single check() call plus result dispatch.

Gap 7 (LOW) — The Rust+Redis path needed end-to-end proof under a real multi-instance deployment. Validated with the backend-correctness load test against 3 gateways behind nginx with Redis shared state. Result: 60 allowed, 60 rate-limited, 49.6% blocked, matching the expected ~50% shared-counter behavior.

---

Additional hardening

• _parse_rate robustness — was rate.split("/") with no error handling; now uses maxsplit=1 + try/except with a clear message showing the bad input
• prompt_pre_fetch normalisation — prompt_id now normalised with .strip().lower(), consistent with how tool_pre_invoke handles tool names
• _normalised_by_tool pre-computed — by_tool key lowercasing was re-done on every hook call; now computed once at init in _validate_config
• Unconditional tenant propagation — _propagate_tenant_id() copies request.state.team_id → global_context.tenant_id at every get_current_user() return path, not just inside _inject_userinfo_instate() which is gated by include_user_info (default False)
• Sliding-window Retry-After >= 1 — int truncation of (oldest_ts + window - now) could produce 0 when the oldest timestamp plus the window rounded down to int(now), telling clients to retry immediately while still over limit; fixed on both memory and Redis paths with max(1, reset_in)
• Token-bucket first-request memory/Redis parity — the memory path hard-coded time_to_full = window on first request, while Redis derived it from tokens_needed / refill_rate, causing metadata divergence between backends (e.g. 60 vs ~6 for a 10/m limit); both paths now use tokens_needed / refill_rate
• Config pre-parsing at startup — rate strings are parsed once during plugin initialization so the request path does not re-parse limits on every hook invocation
• Rolling-upgrade compatibility — the Rust Redis backend uses the same key format as the Python Redis backend so both implementations share counters correctly during mixed deployments
• Async Redis execution path — Redis-backed Rust evaluation now uses a multiplexed async Redis connection and an async PyO3 boundary so Python async hooks can await the Redis path directly
• Rust engine preferred with Python fallback — when the rate_limiter_rust PyO3 extension is installed, the Rust engine handles all rate evaluation; when unavailable or when RATE_LIMITER_FORCE_PYTHON=1 is set, the plugin falls back to the full Python algorithm and backend implementation
• RATE_LIMITER_FORCE_PYTHON env var — allows operators to force the Python backend for A/B comparisons or debugging; Makefile targets benchmark-rate-limiter-capacity-rust and benchmark-rate-limiter-capacity-python exercise both paths
• Reduced lock contention on memory hot path — the Rust MemoryStore now acquires one read lock on the outer map + one write lock on the per-key state per steady-state request (previously two outer read locks), eliminating a redundant lock cycle
• Response construction in Rust — HTTP headers and plugin metadata dicts are built inside the Rust engine after GIL reacquisition, replacing ~18-25 individual PyO3 attribute accesses from the Python side (depending on dimension count)

---

Architecture

The plugin now delegates all rate evaluation to Rust:

• Python owns plugin lifecycle, hook integration, config validation, and request-context extraction
• Rust owns dimension selection, algorithm execution, backend dispatch, result aggregation, and response dict construction
• Backend selection is an engine concern:
  • memory → Rust in-process store via the existing sync engine path
  • redis → Rust async Redis execution via a multiplexed connection + batch Lua execution
• When the rate_limiter_rust PyO3 extension is installed, Rust handles all rate evaluation; otherwise the plugin falls back to the Python backend

Plugin internals: request flow, wrapper responsibilities, Rust engine, backends, and response shaping

┌──────────────────────────────────────────────────────────────────────┐
│                         RateLimiterPlugin                           │
│                                                                      │
│  Hooks:  tool_pre_invoke            prompt_pre_fetch                │
│          (tool name + context)      (prompt_id + context)           │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              │  Python responsibilities:
                              │  - validate config (at init)
                              │  - normalize by_tool keys (at init)
                              │  - extract user / tenant / tool|prompt
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     RustRateLimiterEngine wrapper                    │
│                        (PyO3 boundary)                               │
│                                                                      │
│  Python calls the Rust engine once per request:                     │
│  - check(user, tenant, tool, now_unix) for memory                   │
│  - await check_async(user, tenant, tool, now_unix) for redis        │
│                                                                      │
│  Returns: (allowed, headers_dict, meta_dict)                        │
│  Rust builds dimension keys, evaluates, and constructs response     │
│  dicts internally — no per-attribute PyO3 accesses needed.          │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        RateLimiterEngine (Rust)                      │
│                                                                      │
│  Owns:                                                               │
│  - parsed config (by_user, by_tenant, by_tool)                      │
│  - dimension key construction (user:X, tenant:Y, tool:Z)            │
│  - backend selection + dispatch                                      │
│  - algorithm execution                                               │
│  - per-dimension result calculation                                  │
│  - aggregated EvalResult (most restrictive dimension)               │
│  - HTTP header dict construction (X-RateLimit-*, Retry-After)       │
│  - plugin metadata dict construction (remaining, reset_in, dims)    │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
              ┌───────────────┴────────────────┐
              │                                │
              ▼                                ▼
┌───────────────────────────────┐   ┌──────────────────────────────────┐
│      Memory backend (Rust)    │   │       Redis backend (Rust)      │
│                               │   │                                  │
│  in-process counters          │   │  owns multiplexed Redis conn   │
│  fixed_window                 │   │  batch Lua execution            │
│  sliding_window               │   │  shared counters across replicas│
│  token_bucket                 │   │  compatible key format with     │
│  single read-lock hot path    │   │  Python Redis backend           │
└───────────────────────────────┘   └──────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     Python result dispatch                           │
│                                                                      │
│  allowed -> PromptPrehookResult / ToolPreInvokeResult + headers     │
│  blocked -> PluginViolation(429) + headers + meta                   │
│                                                                      │
│  (headers and meta are pre-built dicts from Rust — no conversion)   │
└──────────────────────────────────────────────────────────────────────┘

---

Test results

Validation for this PR is intentionally comprehensive: unit tests, Rust tests, micro-benchmarks, hook-path A/B comparisons, multi-instance Redis correctness, sustained load tests, algorithm comparison, and Fyre VM deployment validation are all included below.

The highest-signal correctness and unit-test summaries remain visible. The more detailed benchmark and deployment sections are preserved in expandables so the data stays in the PR without dominating the main narrative.

Test results summary

#	Area	Headline result
1	Backend correctness	49.6% blocked vs expected ~50%, 0 infra failures
2	Rust micro-benchmarks	17-21 ns single-key, 49 ns three-dimension
3	Hook-path comparison	1.7x-1.9x for most 3-dim memory paths, 28x for sliding_window
4	Redis capacity benchmark	p99 60 ms vs 100 ms, 28% less gateway memory
5	Multi-user scale test	47.6% blocked, 0 infra failures across 28,650 requests
6	Algorithm comparison	All 3 algorithms at ~47-49% blocked across ~28,600 requests each
7	Fyre VM validation	Rust engine validated on x86 Fyre
8	Unit test suite	93 plugin tests passed, 47/47 Rust tests passed

1. Backend correctness (multi-instance Redis)

Multi-instance Redis correctness matched the expected shared-counter behavior: 49.6% blocked, 0 infrastructure failures, and an exact 60 allowed / 60 rate-limited split in the single-user validation run.

Full backend correctness results

Topology: nginx → 3 gateways → shared Redis, 1 user at 60 req/min against a 30/min limit.

Metric	Result
Total tool calls	121
Allowed	60
Rate-limited	60
Blocked percentage	49.6%
Infrastructure failures	0
Avg response time	49.7 ms
p90 / p99	76 ms / 88 ms

Verdict: REDIS BACKEND — limit correctly enforced. The shared Redis counter produced the expected ~50% blocked rate across three gateway instances. Zero infrastructure failures.

2. Rust micro-benchmarks (criterion, Apple Silicon M-series, release build)

Criterion benchmarks on Apple Silicon show nanosecond-scale in-process memory-backend costs: 17-21 ns for single-key operations, 49 ns for three-dimension evaluation, and 16-67 µs in the contention scenarios included here.

Full micro-benchmark results

In-process memory backend, direct MemoryStore calls (no PyO3 overhead):

Benchmark	Median latency	Notes
fixed_window/single_key	19 ns	single dimension, window reset each iteration
token_bucket/single_key	21 ns
sliding_window/single_key	17 ns
fixed_window/three_dims	49 ns	user + tenant + tool in one call
fixed_window/hot_counter	16 ns	counter near limit (steady-state path)
fixed_window/blocked_path	16 ns	counter past limit (reject path)
fixed_window/many_keys_10k	185 ns	HashMap with 10,000 distinct keys
fixed_window/concurrent_2t	22 µs	2 threads via std::thread::scope
fixed_window/concurrent_4t	35 µs	4 threads
fixed_window/concurrent_8t	67 µs	8 threads

3. Hook-path comparison (Python pytest-benchmark, memory backend)

Measured per-hook Python→Rust round-trip overhead via pytest-benchmark, comparing the Rust engine check() path against the older Python-only algorithm+backend stack. Numbers reflect the full hook path including PyO3 crossing, dict construction, and Python result dispatch.

Full hook-path comparison results

All benchmarks: fixed_window, sliding_window, token_bucket × single dimension, three dimensions.

Config	Python mean	Rust mean	Speedup
fixed_window / 1 dim	5.1 µs	2.8 µs	1.8×
fixed_window / 3 dim	13.2 µs	7.2 µs	1.8×
sliding_window / 1 dim	14.3 µs	2.9 µs	4.9×
sliding_window / 3 dim	81.3 µs	2.9 µs	28×
token_bucket / 1 dim	5.6 µs	2.8 µs	2.0×
token_bucket / 3 dim	14.9 µs	7.8 µs	1.9×

The sliding_window improvement is the most dramatic because the Python implementation walks and filters timestamp lists in pure Python on every check, while Rust uses a VecDeque with amortized O(1) front pops.

4–8. Remaining validation

Redis capacity benchmark, multi-user scale, algorithm comparison, Fyre VM, unit tests

4. Redis capacity benchmark — p99 latency 60 ms (vs 100 ms Python), 28% less gateway RSS. Three gateways, shared Redis, 100 users at 0.25 rps for 5 minutes.

5. Multi-user scale test — 47.6% blocked across 28,650 requests with 0 infrastructure failures. 100 concurrent users, 5-minute sustained load.

6. Algorithm comparison — All three algorithms (fixed_window, sliding_window, token_bucket) converged to ~47-49% blocked across ~28,600 requests each, confirming equivalent enforcement behavior.

7. Fyre VM validation — Rust engine validated on x86 Fyre VM deployment.

8. Unit test suite — 93 Python plugin tests passed, 47/47 Rust tests passed.

[lucarlig]

I found one issue to address before merge: the new tenant_id propagation still does not make by_tenant limits work for session-authenticated users.

[lucarlig]

https://docs.rs/pyo3-log/latest/pyo3_log/
please add logs with pyo3_logs

[jonpspri]

GTG

[dima-zakharov]

Github show that no new commits after review.

[brian-hussey]

Approving as this has undergone several rounds of review and has now passed all CI checks.