fix: tighten secrets detection coverage and add focused benchmarking

[lucarlig]

Summary

This updates secret handling so secret-like values are owned by the dedicated secrets detection plugin instead of the PII filter, strengthens provider-specific secret coverage, improves Rust plugin logging, adds a focused Rust-vs-Python benchmarking workflow for the secrets detector, and documents the broader intrinsic-shape heuristics that can still catch longer secret-like values.

What changed

• enabled SecretsDetection in plugins/config.yaml
• removed AWS and API-key secret handling from the PII filter in both Python and Rust so those patterns are only handled by secrets detection
• improved secrets detection coverage with stronger provider-specific patterns, including GitHub tokens and Stripe secret keys
• kept broader generic API-key heuristics opt-in and documented their false-positive tradeoffs, detection limits, and the fact that longer intrinsic-shape heuristics such as base64_24 can still catch assignment-style values when the value itself looks secret-like
• preserved safe defaults when operators provide a partial enabled: map so omitted heuristics do not silently turn back on
• added Rust-side logging for the secrets detection plugin via pyo3_log
• improved Python fallback logging so Rust scan failures preserve full exception context
• added a focused Locust scenario for secret detection at tests/loadtest/locustfile_secret_detection.py
• added make load-test-secret-detection-compare and tests/loadtest/run_secret_detection_compare.sh to compare the Rust-backed and forced Python fallback paths on the same focused workload
• added targeted gitleaks:allow annotations to intentional secret-shaped test fixtures in the Python and Rust secrets detection tests and the focused secret-detection Locust file so secret scanning stays clean without changing detector behavior
• updated secrets detection and PII filter docs to clarify ownership, warnings, capabilities, and limits

Validation

• uv run pytest tests/unit/plugins/test_secrets_detection.py
• cargo test in plugins_rust/secrets_detection
• cargo test in plugins_rust/pii_filter
• make rust-check
• focused gitleaks detect --no-git scans for tests/unit/plugins/test_secrets_detection.py, plugins_rust/secrets_detection/src, and tests/loadtest/locustfile_secret_detection.py
• make test
• live /rpc verification against the compose UI stack with the Rust plugin enabled for blocking and redaction scenarios

End-to-End Benchmark

Focused secret-detection benchmark using the new Locust scenario against the full compose stack:

• aggregate throughput: Rust 252.32 RPS vs Python 236.70 RPS (+6.2%)
• aggregate average latency: Rust 333.33 ms vs Python 358.22 ms (-7.5%)
• aggregate p95 latency: Rust 520 ms vs Python 580 ms (-11.5%)
• aggregate p99 latency: Rust 590 ms vs Python 760 ms (-28.8%)

Per-request-type improvements from the same focused run:

• clean prompt requests: Rust 125.44 RPS vs Python 118.04 RPS
• secret-blocked prompt requests: Rust 126.88 RPS vs Python 118.65 RPS

Closes #3741

[crivetimihai]

Thanks @lucarlig. Good separation of concerns — moving secret patterns out of PII filter into the dedicated secrets detection plugin. Benchmark results show solid Rust improvement (~7% latency, ~29% at p99).

[lucarlig]

we will be using rust plugin for 1.0

[dima-zakharov]

Looks good

[dima-zakharov]

all blocking issues resolved. Broad heuristics like generic_api_key_assignment are properly disabled by default with clear warnings when enabled.
Recommendation
Ready to merge

[lucarlig]

removed release-fix label blocked by #3860 and #3840

[jonpspri]

Review Summary

Clean separation of concerns — secret-style credential detection (AWS keys, API keys) is properly moved from the PII filter into the dedicated secrets_detection plugin. New provider-specific patterns (GitHub tokens, Stripe secret keys, generic API key assignment) are well-tested with both positive and negative cases across Python and Rust.

Key Positives

• Secure defaults: is_enabled() now defaults to false (was true), and all broad heuristic patterns (jwt_like, hex_secret_32, base64_24, generic_api_key_assignment) default to disabled in code — matching the documented opt-in behavior. Production config.yaml explicitly enables the ones needed.
• Config merging: The _merge_enabled_patterns field validator prevents partial YAML configs from silently enabling broad heuristics.
• Python/Rust parity: All 11 patterns and their defaults are in sync across both implementations.
• Test coverage: 221 Python tests and 59 Rust secrets_detection tests pass, covering new patterns, config edge cases, Rust log bridge, fallback exception logging, and broad-pattern warnings.
• Rust logging: pyo3-log bridge gives operators visibility into Rust-side scan behavior without separate log infrastructure.

Minor Notes (non-blocking)

• The e2e test changes (test_mcp_session_isolation.py, test_mcp_access_matrix.py, mcp_test_helpers.py, fuzz/conftest.py) are tangential to secrets detection — consider splitting in future PRs for cleaner history.
• Linters (ruff, bandit, cargo test, bash -n) all clean on changed files.