feat: replace the internal plugin framework with CPEX

[araujof]

Summary

Replace the internal plugin framework (mcpgateway/plugins/framework/, mcpgateway/plugins/tools/, plugin_templates/) with the CPEX external package (cpex>=0.1.0rc1).

Closes: #3753

What changed

• Added cpex as external dependency in pyproject.toml
• Migrated all imports: from mcpgateway.plugins.framework → from cpex.framework, from mcpgateway.plugins.tools → from cpex.tools
• Moved get_plugin_manager() singleton factory to mcpgateway/plugins/__init__.py (bridges cpex with gateway-specific hook policies)
• Migrated PluginMode enums: ENFORCE → SEQUENTIAL, PERMISSIVE → TRANSFORM, ENFORCE_IGNORE_ERROR → SEQUENTIAL + on_error: ignore (see feat: replace the internal plugin framework with CPEX #3754 (comment))
• Adapted to cpex API changes:
  • PluginViolation.http_status_code / .http_headers → violation.details["http_status_code"] / details["http_headers"]
  • PromptPosthookPayload.name → .prompt_id; ToolPreInvokePayload.arguments → .args
• Deleted internal framework: mcpgateway/plugins/framework/, mcpgateway/plugins/tools/, plugin_templates/ directory, tests/unit/mcpgateway/plugins/framework/
• Created acceptance tests (tests/acceptance/plugins/test_cpex_contract.py) verifying cpex API surface, behavioral contracts, serialization, and settings compatibility
• Updated YAML configs across plugins and test fixtures
• Updated documentation under docs/docs/using/plugins/, plugins/README.md, plugins/AGENTS.md, llms/plugins-llms.md

Test results

• Unit tests pass
• Acceptance tests pass
• Performance profiling shows no regression
• flake8, bandit, pylint all clean

[araujof]

Ready for review. Before merging, need to update CPEX to a tagged release. Currently targeting feature branch feat/cf_syncin CPEX repo, which contains backports of in-tree changes to the plugin framework made in the CF repo.

[araujof]

@jonpspri @cafalchio we need to branch and update the cpex-* plugin packages with cpex imports for the corresponding tests to pass.

1. The published cpex-* plugin packages (0.1.0–0.2.0) import from mcpgateway.plugins.framework, which this branch removed in favor of cpex.framework. These packages need new releases that import from cpex.framework.
2. Until those packages are published, any test that imports them will fail at collection time.

@cafalchio feat/cpex is tracking a branch of cpex where I back-ported the in-tree changes to the plugin framework made after the creation of cpex. See the PR description for the documentation. I tagged it 0.1.0.dev11.

[araujof]

@msureshkumar88 thanks for the re-review. All remaining items are addressed.

The missing <0.2 upper bound means a cpex 0.2.0 release with breaking API changes would silently break production

Added. Now "cpex>=0.1.0,<0.2". Worth noting that exclude-newer-package and uv.lock already prevent surprise upgrades in CI/container builds, but the upper bound is good defensive practice for bare pip install deployments.

API clients or existing DB rows with mode="enforce_ignore_error" will fail Pydantic validation

Added as a deprecated enum member:

ENFORCE_IGNORE_ERROR = "enforce_ignore_error"  # Deprecated: use SEQUENTIAL + on_error=ignore

This ensures existing DB rows and API clients serialize/deserialize correctly while guiding users toward the new pattern.

gateway_plugin_manager.py:427-434 uses a sequential for loop while invalidate_all was correctly fixed to use asyncio.gather

Fixed — invalidate_team now uses the same asyncio.gather pattern as invalidate_all, with per-context failure logging and a summary debug line.

When HMAC is configured and an unsigned message arrives, this is invisible in production (debug is filtered)

Promoted to WARNING:

_logger.warning("Plugin invalidation: received unsigned message while HMAC is configured — expected only during rolling deploy")

_sign_message and _verify_and_extract have no dedicated unit tests

Added TestHMACSigningVerification class with 9 tests covering:

• Sign + verify roundtrip
• Tampered signature → rejected
• Tampered payload → rejected
• Missing payload key in envelope → treated as unsigned
• Unsigned accepted when no key configured
• Unsigned accepted with warning when key is set
• Invalid JSON → returns None
• _sign_message passthrough when no key
• Signed envelope extraction when receiver has no key (rolling deploy)

start_plugin_invalidation_listener probe failure at line 539 still logs at WARNING

Keeping as-is for now. The probe failure is a connectivity issue (Redis down), not a security event. Escalating to ERROR would trigger noise in environments where Redis is optional. The exponential backoff + WARNING→ERROR escalation on the listener loop itself already handles the "persistent failure" case.

[araujof]

@brian-hussey @jonpspri This PR is ready for final review and merge.

CI is passing, prior review feedback has been addressed, e2e tests performed by @cafalchio @msureshkumar88 didn't find any regressions, documentation and migration guides are included, and the acceptance test suite validates the cpex contract surface. The massive net deletion demonstrates successful extraction of the framework into a reusable package without functionality loss.

Key changes

The internal framework code that was extracted into cpex is removed, and all plugins, tests, services, and routers are migrated to import from the external package. A gateway-specific bridge layer in mcpgateway/plugins/__init__.py and gateway_plugin_manager.py wires cpex into the application's DI, RBAC, and observability layers.

Area	Summary
Dependency	Added cpex>=0.1.0,<0.2.0 to pyproject.toml
Import migration	from mcpgateway.plugins.framework → from cpex.framework; from mcpgateway.plugins.tools → from cpex.tools
Enum migration	ENFORCE → SEQUENTIAL, PERMISSIVE → TRANSFORM, ENFORCE_IGNORE_ERROR → SEQUENTIAL + on_error: ignore
API adaptations	PluginViolation.http_status_code → violation.details["http_status_code"]; PromptPosthookPayload.name → .prompt_id; ToolPreInvokePayload.arguments → .args
Deleted	Internal framework (~35k LOC), plugin cookiecutter templates, all internal framework unit tests
Added	Acceptance test suite (tests/acceptance/plugins/test_cpex_contract.py), migration guide (docs/.../migration-to-cpex.md), Playwright smoke tests for plugin admin UI
DB migration	Added on_error column to tool_plugin_bindings (with Alembic migration and check constraint)
Docs	Updated plugin docs, AGENTS.md, llms guidance, plugins README, CHANGELOG (with migration guide)

Notes

• This PR ads a compatibility layer to ensure modes are fully backwards compatible to legacy CF plugin modes.
• A migration guide was created to help users migrate to the new plugin execution and error modes.

[msureshkumar88]

❌ No unit tests for HMAC signing/verification paths

_sign_message and _verify_and_extract (mcpgateway/plugins/__init__.py:118–155) have no dedicated unit tests. These functions are the core of the security hardening added in this PR — they are exactly the paths that need regression coverage.

Missing test cases:

1. Tampered signature — _verify_and_extract must return None and log an error when sig does not match the payload:

   def test_verify_and_extract_tampered_sig(monkeypatch):
       monkeypatch.setenv("JWT_SECRET_KEY", "test-secret")
       signed = _sign_message('{"tool": "x"}')
       envelope = json.loads(signed)
       envelope["sig"] = envelope["sig"][:-1] + ("0" if envelope["sig"][-1] != "0" else "1")
       assert _verify_and_extract(json.dumps(envelope)) is None

2. Missing payload key in envelope — a dict with sig but no payload must not raise; should return None or fall through to the plain-message path:

   def test_verify_and_extract_missing_payload_key(monkeypatch):
       monkeypatch.setenv("JWT_SECRET_KEY", "test-secret")
       bad_envelope = json.dumps({"sig": "abc123"})
       # Should not raise KeyError; return None or the raw string
       result = _verify_and_extract(bad_envelope)
       assert result is None or isinstance(result, str)

3. Round-trip: sign → verify — happy path with key configured:

   def test_sign_verify_roundtrip(monkeypatch):
       monkeypatch.setenv("JWT_SECRET_KEY", "test-secret")
       payload = '{"tool": "y", "args": {}}'
       assert _verify_and_extract(_sign_message(payload)) == payload

4. No key configured — _sign_message returns payload unchanged, _verify_and_extract accepts both plain and envelope forms:

   def test_sign_no_key(monkeypatch):
       monkeypatch.delenv("JWT_SECRET_KEY", raising=False)
       payload = '{"tool": "z"}'
       assert _sign_message(payload) == payload
       assert _verify_and_extract(payload) == payload

These are the exact failure modes the HMAC hardening was meant to prevent. Without them, a future refactor that breaks hmac.compare_digest or the envelope parsing would pass all existing tests.

[msureshkumar88]

Thank you for the quick turnaround on the requested changes — all four items from the last review have been addressed:

• cpex>=0.1.0,<0.2 upper bound added
• PluginBindingMode.ENFORCE_IGNORE_ERROR added to the schema enum
• invalidate_team parallelized with asyncio.gather
• HMAC unsigned message log level escalated to WARNING when key is configured

One item remains before merge — unit tests for _sign_message and _verify_and_extract. See the latest review for the specific cases needed.

[msureshkumar88]

Documentation Review

Good migration guide and mode compatibility work overall. Found a few documentation gaps worth tracking — suggesting these go into a follow-up task given the merge-blocker concerns already in flight.

Findings

High

• llms/plugins-llms.md: PromptPosthookPayload(name: str, ...) still shows the old .name field — was renamed to .prompt_id in this PR (Step 3 of migration guide).

Medium

• migration-to-cpex.md troubleshooting: two entries say "See Step 4" but Step 4 is "Update Tool Plugin Bindings (API)" — payload fields are Step 3. Broken anchor slug too.
• migration-to-cpex.md scaffolding section uses cpex bootstrap; every other doc (lifecycle.md, llms/plugins-llms.md, index.md) uses mcpplugins bootstrap. One should be authoritative.
• index.md and lifecycle.md mode tables only list 3 modes (sequential, transform, disabled). New native modes (concurrent, audit, fire_and_forget) are described in the migration guide only — new users won't find them in the main reference docs.
• index.md config schema table has no row for the new on_error field (valid values: fail, ignore, disable).

Low

• migration-to-cpex.md checklist: pytest tests/unit/mcpgateway/plugins/ -v — the framework/ subtree was deleted by this PR; checklist should point to tests/acceptance/plugins/ or clarify what remains.
• plugins/README.md and plugins/AGENTS.md are not in the changed files — if they contain old import paths or mode names, they may be stale.

Suggestion

The stale .name field in llms/plugins-llms.md is the only one that could actively mislead users writing plugins. The rest are reference gaps rather than correctness issues.

Worth tracking the full set in a follow-up docs issue so this PR can move forward on its current blockers (cpex tagged release, cpex-plugins freeze).

[lucarlig]

@araujof still have references to old import:

rg -n "mcpgateway\.plugins\.framework\.external\.mcp\.server\.runtime" plugins/external
plugins/external/cedar/run-server.sh
27:    API_SERVER_SCRIPT="$(python -c 'import mcpgateway.plugins.framework.external.mcp.server.runtime as server; print(server.__file__)')"

plugins/external/clamav_server/README.md
20:- Use the gateway runtime: `mcpgateway.plugins.framework.external.mcp.server.runtime`.

plugins/external/clamav_server/run.sh
8:exec python -m mcpgateway.plugins.framework.external.mcp.server.runtime

plugins/external/opa/run-server.sh
28:    API_SERVER_SCRIPT="$(python -c 'import mcpgateway.plugins.framework.external.mcp.server.runtime as server; print(server.__file__)')"

plugins/external/llmguard/run-server.sh
27:    API_SERVER_SCRIPT="$(python -c 'import mcpgateway.plugins.framework.external.mcp.server.runtime as server; print(server.__file__)')"

[araujof]

Thanks @lucarlig @msureshkumar88 !

Fixed:

1. 5 stale import paths in plugins/external/ — updated mcpgateway.plugins.framework.external.mcp.server.runtime →
   cpex.framework.external.mcp.server.runtime
2. Stale .name field in llms/plugins-llms.md → now shows .prompt_id
3. 2 broken anchor links in migration-to-cpex.md — pointed to Step 4, now correctly point to Step 3
4. Inconsistent CLI command in migration-to-cpex.md — cpex bootstrap → mcpplugins bootstrap

@msureshkumar88 The comments on HMAC unit testing were already addressed: We have 9 tests covering all 4 scenarios you requested plus 5 additional edge cases.

At this point, if you could run a last end to end battery of checks, that would be awesome!
I think we can push any additional minor and doc changes into a separate PR.

cc: @brian-hussey

[msureshkumar88]

Final E2E Battery — Approved ✅

Ran a full end-to-end check pass before approving.

Local test results

Suite	Result
Full unit suite ()	16,957 passed, 111 skipped, 2 xfailed
Plugin unit tests ()	518 passed, 33 skipped
HMAC + acceptance (, )	116 passed
Bandit scan ()	0 issues
Ruff lint ()	All checks passed

Previously open items — all resolved

• HMAC unit tests ( / ): full coverage in test_plugin_runtime_management.py — round-trip, tampered sig, tampered payload, unsigned-without-key, unsigned-when-key-configured, invalid JSON ✅
• Stale import paths in cedar/run-server.sh, llmguard/run-server.sh, opa/run-server.sh, clamav_server/run.sh, clamav_server/README.md: all updated to cpex.framework.external.mcp.server.runtime ✅
• llms/plugins-llms.md → field: fixed ✅
• Migration guide anchor links: corrected to point to Step 3 ✅
• cpex>=0.1.0,<0.2 upper bound: present ✅
• ENFORCE_IGNORE_ERROR in PluginBindingMode enum as deprecated alias: present ✅
• asyncio.gather in invalidate_team (gateway_plugin_manager.py:420): implemented ✅
• WARNING log for unsigned messages when HMAC configured (__init__.py:153): escalated ✅

Other checks

• Alembic: single head (926d3e07d098) — clean ✅
• CI checks currently queued (re-triggered by recent commits), not failing ✅

The framework extraction is clean — 139 files deleted, net code reduction, no regressions in the full test suite. All review blockers resolved. Good to merge.

[msureshkumar88]

Final E2E Battery — Approved

Ran a full end-to-end check pass before approving.

Local test results

Suite	Result
Full unit suite (tests/unit/)	16,957 passed, 111 skipped, 2 xfailed
Plugin unit tests (tests/unit/mcpgateway/plugins/)	518 passed, 33 skipped
HMAC + acceptance (test_plugin_runtime_management, test_cpex_contract)	116 passed
Bandit scan (mcpgateway/plugins/)	0 issues
Ruff lint (mcpgateway/plugins/)	All checks passed

Previously open items — all resolved

• HMAC unit tests for _sign_message / _verify_and_extract: full coverage in test_plugin_runtime_management.py — round-trip, tampered sig, tampered payload, unsigned-without-key, unsigned-when-key-configured, invalid JSON ✅
• Stale import paths in cedar/run-server.sh, llmguard/run-server.sh, opa/run-server.sh, clamav_server/run.sh, clamav_server/README.md: all updated to cpex.framework.external.mcp.server.runtime ✅
• llms/plugins-llms.md .name → .prompt_id field: fixed ✅
• Migration guide anchor links: corrected to point to Step 3 ✅
• cpex>=0.1.0,<0.2 upper bound: present in pyproject.toml ✅
• ENFORCE_IGNORE_ERROR in PluginBindingMode enum as deprecated alias: present ✅
• asyncio.gather in invalidate_team (gateway_plugin_manager.py:420): implemented ✅
• WARNING log for unsigned messages when HMAC configured (init.py:153): escalated ✅

Other checks

• Alembic: single head (926d3e07d098) — clean ✅
• CI checks currently queued (re-triggered by recent commits), not failing ✅

The framework extraction is clean — 139 files deleted, net code reduction, no regressions in the full test suite. All review blockers resolved. Good to merge.

[brian-hussey]

Approved based on others' reviews.
All issues addressed. Merging.