[TESTING][PLUGINS]: Test, harden and document retry with exponential backoff plugin

[madhu-mohan-jaishankar]

🧪 Summary

Sub-issue of #3735 — Test, harden, and document the Retry with Backoff plugin (plugins/retry_with_backoff/) to bring it to production quality for the 1.0.0 release.

This plugin currently operates as an advisory-only plugin: it annotates retry policy metadata (max retries, backoff parameters, retry-triggering status codes) on tool_post_invoke and resource_post_fetch hooks for downstream orchestration consumption, but does not itself perform retries.

---

🧱 Area Affected

• Plugin: plugins/retry_with_backoff/ — Resilient retry logic with exponential backoff for transient failures

---

⚙️ Context / Rationale

The plugin is listed as Priority 1 — Critical path in the parent issue. It is one of three plugins blocking for 1.0.0 along with Secrets Detection and Output Length Guard.

Current state:

• Plugin exists and annotates retry policy metadata via tool_post_invoke and resource_post_fetch
• No dedicated plugin-level unit tests exist (only tests/unit/mcpgateway/utils/test_retry_manager.py covers the utility layer)
• README is brief; configuration is documented but limited
• Known TODOs: per-tool/resource config overrides, jitter strategy hints, dynamic tuning based on payload size

---

📋 Acceptance Criteria

Testing

• Unit tests: Create tests/unit/plugins/test_retry_with_backoff.py covering:
  • tool_post_invoke returns correct metadata with default config
  • resource_post_fetch returns correct metadata including retry_on_status
  • Custom config values are reflected in metadata output
  • Invalid config values are rejected by RetryPolicyConfig validation (e.g., negative max_retries)
  • Empty / zero-value config edge cases
• Integration tests: Test the plugin registered and active within the gateway pipeline:
  • Plugin loads and activates without errors
  • Metadata is present on tool call responses when plugin is active
  • Plugin coexists correctly with other active plugins
• Edge cases:
  • retry_on_status with empty list
  • max_retries: 0 (no retries)
  • backoff_base_ms: 0 / max_backoff_ms: 0
  • Extremely large backoff values

Documentation

• README hardening: Expand plugins/retry_with_backoff/README.md to include:
  • Full configuration reference with valid ranges and defaults
  • Explanation of advisory-only design and downstream usage contract
  • Example payloads showing what metadata is added to tool call responses
  • Guidance for downstream orchestrators on how to consume the metadata
  • Jitter strategy hints (roadmap note or implementation if done)
• Inline docstrings: Verify all public methods have complete docstrings meeting interrogate standards
• Config docs: Ensure plugins/config.yaml has a complete example entry for this plugin

Hardening

• Config validation: Confirm RetryPolicyConfig rejects or clamps out-of-range values (e.g., backoff_base_ms > max_backoff_ms)
• Missing config keys: Verify partial configs (missing keys) fall back cleanly to defaults
• Error handling: Plugin exceptions must not propagate to caller — confirm failure mode is fail-open with appropriate log
• Logging: Ensure metadata annotation is logged at DEBUG level, no sensitive data in logs
• Per-tool/resource overrides (stretch goal): Add support for per-tool config overrides as noted in README TODOs
• Jitter strategy (stretch goal): Add jitter: true/false config field and document behavior

Overall

• CI passes with no regressions
• All new tests added to the appropriate test suite
• Plugin manifest (plugin-manifest.yaml) is accurate and complete

---

📁 Key Files

• plugins/retry_with_backoff/retry_with_backoff.py — Plugin implementation
• plugins/retry_with_backoff/README.md — Plugin documentation
• plugins/retry_with_backoff/plugin-manifest.yaml — Plugin manifest
• plugins/config.yaml — Plugin configuration
• tests/unit/mcpgateway/utils/test_retry_manager.py — Existing retry utility tests (reference, not same layer)
• plugins/AGENTS.md — Plugin development guidelines