[CHORE][NOTIFICATIONS]: Investigate and test support for notifications/tools/list_changed signal for dynamic tool discovery

[crivetimihai]

🔧 Chore Summary

Investigate, test, and validate ContextForge's support for the MCP notifications/tools/list_changed signal for dynamic tool discovery. While the NotificationService and MCPSessionPool integration code exists, it has not been end-to-end tested or validated against real MCP servers that emit these notifications. Additionally, the feature needs to be made more configurable (e.g., enable/disable per gateway, configurable debounce, admin UI controls).

Key questions:

1. Does CF currently listen for notifications/tools/list_changed (and resources/, prompts/) post-discovery?
2. Does receiving this notification auto-trigger a tools/list refresh to sync the registry?
3. Is this behavior reliable, configurable, and observable?

---

🧱 Area Affected

• Other: MCP notification handling, gateway refresh, session pool integration

---

⚙️ Context / Rationale

Current state of the code:

• mcpgateway/services/notification_service.py — A NotificationService exists with:

  • Debounced refresh queue (default 5s) to prevent notification storms
  • Flag merging during debounce window (e.g., tools + resources notifications merge)
  • Per-gateway capability tracking (GatewayCapabilities dataclass)
  • create_message_handler() that returns a callback for ClientSession
  • Background worker that calls _refresh_gateway_tools_resources_prompts() on the GatewayService
  • Module-level singleton pattern (init_notification_service / get_notification_service)

• mcpgateway/services/mcp_session_pool.py — Integration points:

  • _create_session() passes message_handler=message_handler to ClientSession (line ~1149)
  • Pool init calls init_notification_service() with configurable debounce
  • start_pool_notification_service() wires up the GatewayService reference
  • register_gateway_capabilities() / unregister_gateway() lifecycle management

What needs validation:

• The message_handler callback is passed to ClientSession, but it's unclear if the MCP SDK actually delivers ServerNotification objects for list_changed events through this callback in all transport modes (SSE, Streamable HTTP)
• The refresh path (_refresh_gateway_tools_resources_prompts) needs to be tested end-to-end with a real server that dynamically adds/removes tools
• Edge cases: What happens if the notification arrives during an in-flight discovery? During pool cleanup? When multiple workers receive the same notification?

---

📋 Acceptance Criteria

• E2E test: Create or use a test MCP server that dynamically adds/removes tools and emits notifications/tools/list_changed
• Validate SSE transport: Confirm notifications flow through message_handler callback with SSE transport
• Validate Streamable HTTP transport: Confirm notifications flow through with streamable HTTP transport
• Validate registry sync: Confirm that after receiving the notification, CF calls tools/list and updates the tool registry
• Validate debounce: Confirm that rapid notifications are debounced correctly (flag merging works)
• Configurability: Make the feature configurable:
  • Global enable/disable for notification-driven refresh (env var / config)
  • Per-gateway enable/disable
  • Configurable debounce interval (already partially exists, but needs config.py/env var support)
• Observability: Ensure notification metrics are exposed (already exists in get_metrics(), but verify integration with observability stack)
• Admin UI: Surface notification status and metrics in the admin UI (optional, stretch goal)
• Documentation: Document the feature behavior and configuration options
• CI passes with no regressions

---

🧩 Additional Context

Relevant files:

• mcpgateway/services/notification_service.py — Core notification handling
• mcpgateway/services/mcp_session_pool.py — Session pool integration
• mcpgateway/services/gateway_service.py — Gateway refresh logic
• mcpgateway/common/models.py — Data models
• mcpgateway/cache/session_registry.py — Session registry
• docs/docs/architecture/adr/034-centralized-notification-service.md — ADR for this feature
• tests/unit/mcpgateway/services/test_notification_service.py — Existing unit tests (mock-based)

MCP Spec reference: The notifications/tools/list_changed signal is part of the MCP specification for dynamic tool discovery. When a server's available tools change, it should send this notification so clients can call tools/list to get the updated list.