chore(rust): restructure Rust plugins as independent crates

[crivetimihai]

Note: This PR was re-created from #2961 due to repository maintenance. Your code and branch are intact. @lucarlig please verify everything looks good.

Closes #2730
Closes #2906

---

🚀 Summary

Restructures Rust plugins as independent crates with proper workspace architecture and consolidates CI testing to eliminate duplication. The PII filter plugin testing is migrated to the new structure as the reference implementation.

Key changes:

• Rust plugins now live as independent subcrates in plugins_rust/ workspace
• Each plugin has its own Cargo.toml, Makefile, and isolated build system
• Unified parameterized testing approach for both Python and Rust implementations
• Removed duplicate Rust-specific CI workflow in favor of consolidated pytest CI
• PII filter tests merged into single parameterized suite

---

🧪 Checks

• make lint passes
• make test passes
• CHANGELOG updated (if user-facing)

---

📓 Notes

Architecture Decision

This PR implements Option 2: Fully Independent Plugin Crates from issue #2730:

• Each Rust plugin is a standalone crate with its own PyO3 bindings
• Plugins can be built, tested, and distributed independently
• Clear ownership and isolation per plugin
• Natural fit for pip distribution via maturin

Testing Strategy

Implements Python-based integration testing shared between Rust and Python implementations:

• Single parameterized test suite in tests/unit/mcpgateway/plugins/plugins/pii_filter/test_pii_filter.py
• Tests run against both implementations using @pytest.mark.parametrize
• Graceful fallback when Rust plugins unavailable
• CI enforces Rust plugin availability via REQUIRE_RUST_PLUGINS=1

CI Consolidation

• Removed separate rust-plugins.yml workflow
• Rust environment setup integrated into main pytest.yml
• All tests run through single parameterized pytest suite
• Eliminates duplicate test execution and maintenance burden

This PR has 3 independent parts; found issues with:

License check was broken → added commit to fix it
Exfil plugin Rust version had bugs → needed to be addressed
Main plugin refactoring

[lucarlig]

looks same as before, LGTM

[lucarlig]

rebased today, quite a few conflicts in CI

[cafalchio]

thanks luca. LGTM

[crivetimihai]

Review Summary

Rebased onto main (clean, no conflicts) and added one fix commit:

fix: align pii_filter version and authors with other Rust plugin crates

• plugins_rust/pii_filter/Cargo.toml: version 1.0.0-RC-1 → 1.0.0-RC-2 (matches pyproject.toml), authors → "ContextForge Contributors" (matches other crates)
• plugins_rust/pii_filter/pyproject.toml: authors aligned

Review Findings

Architecture & Design — The restructuring from a workspace-level Cargo.toml to independent crates is well-executed. Each plugin has its own build/test/publish lifecycle. Import path breaking change is properly documented in MIGRATION-RUST-IMPORTS.md. CI consolidation into the main pytest workflow eliminates duplicate test runs.

Security — No issues. Changes are structural (import paths, build system). Detection logic unchanged.

Performance — No concerns. Rust implementations use LazyLock, Cow<str>, RegexSet — all appropriate patterns.

Tests — All 355 Python tests pass (114 Rust-specific properly skipped without compiled Rust). Parametrized test approach correctly covers both implementations. Differential coverage for changed lines is adequate (encoded_exfil_detector.py at 95%, BSN pattern changes fully covered).

Logic — BSN pattern improvement (context-aware vs bare \d{9}) reduces false positives. Import path migration consistent across all three plugins. License checker, test_tool_service, and metrics test fixes are all correct.

Minor Follow-up Items (non-blocking)

1. plugins/pii_filter/pii_filter_rust.py is now dead code — no longer imported by the main plugin after the direct Rust import change. Can be removed in a follow-up.
2. once_cell = "1.21" dependency in pii_filter Cargo.toml appears unused (code uses std::sync::LazyLock). Can be cleaned up.