Test any MCP server for spec compliance. 88-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the MCP specification. Works against HTTP endpoints (https://my-server.com/mcp) and stdio servers (npx @modelcontextprotocol/server-filesystem /tmp) alike. CLI, MCP server, and programmatic API.
Built and maintained by Yaw Labs.
MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.
This tool solves that:
- 88 tests across 8 categories — transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. No gaps. (HTTP runs all 85 transport-applicable tests; stdio runs ~75 — HTTP-specific tests like CORS, TLS, session headers, and rate limiting are gated out.)
- Capability-driven — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
- Graded scoring — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
- CI-ready —
--strictmode exits with code 1 on required test failures. Drop it into any pipeline. - Spec-referenced — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
- Three interfaces — CLI for humans, MCP server for AI assistants, programmatic API for integration.
- Published methodology — the testing methodology and rule catalog are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.
Remote HTTP server:
npx @yawlabs/mcp-compliance test https://my-server.com/mcpLocal stdio server (the vast majority of MCP servers on npm):
# Pass the command directly, Inspector-style
npx @yawlabs/mcp-compliance test npx @modelcontextprotocol/server-filesystem /tmp
# Or a local build
npx @yawlabs/mcp-compliance test node ./dist/server.js
# With env vars
npx @yawlabs/mcp-compliance test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-githubInstall globally:
npm install -g @yawlabs/mcp-compliance
mcp-compliance test https://my-server.com/mcpThat's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.
# Terminal output with colors and grade
mcp-compliance test https://my-server.com/mcp
# JSON / SARIF for scripting + GitHub Code Scanning
mcp-compliance test https://my-server.com/mcp --format json
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif
# Strict mode for CI — exits 1 on required-test failure
mcp-compliance test https://my-server.com/mcp --strict
# Auth (shorthand or full header)
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"
# Focus the run
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
mcp-compliance test https://my-server.com/mcp --verbosePass the command and its args as positional arguments (MCP Inspector-style). Use -- to disambiguate when the target needs flags that collide with ours.
# npm-distributed stdio servers
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-compliance test uvx mcp-server-git
# Local build
mcp-compliance test node ./dist/server.js
# With env vars (repeatable -E, or --env-file)
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
mcp-compliance test --env-file .env -- node ./server.js
# Set working directory
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js
# Target uses a flag that collides with ours — use `--` to separate
mcp-compliance test --verbose -- node ./server.js --verboseOn Windows, npx and other .cmd shims are handled automatically by spawning through the shell.
| Option | Applies to | Description |
|---|---|---|
--format <format> |
both | Output format: terminal, json, sarif, github, or markdown (default: terminal) |
--config <path> |
both | Load defaults from a config file (default: mcp-compliance.config.json in cwd) |
--output <file> |
both | Write a local SVG badge to the given path after the run |
--list |
both | Print test IDs that would run given current filters, then exit (no connection) |
--transport <kind> |
both | Filter by http or stdio (only used with --list when no target is provided) |
--strict |
both | Exit with code 1 on any required test failure (for CI) |
--min-grade <grade> |
both | Exit with code 1 if grade is below this threshold (A–F) |
-H, --header <h> |
HTTP | Add header to all requests, format "Key: Value" (repeatable) |
--auth <token> |
HTTP | Shorthand for -H "Authorization: <token>" |
-E, --env <var> |
stdio | Set env var for stdio command, format "KEY=VALUE" (repeatable) |
--env-file <path> |
stdio | Load env vars from a file (one KEY=VALUE per line) |
--cwd <dir> |
stdio | Working directory for the stdio command |
--timeout <ms> |
both | Request timeout in milliseconds (default: 15000) |
--preflight-timeout <ms> |
HTTP | Preflight connectivity check timeout (HTTP only) |
--retries <n> |
both | Number of retries for failed tests (default: 0) |
--only <items> |
both | Only run tests matching these categories or test IDs (comma-separated) |
--skip <items> |
both | Skip tests matching these categories or test IDs (comma-separated) |
--concurrency <n> |
both | Max parallel-safe tests in flight (default: 1; raising reduces wall time but can perturb timing-sensitive servers) |
--verbose |
both | Print each test result as it runs (also forwards stdio stderr) |
GitHub Action (drop into any .github/workflows/*.yml):
- uses: YawLabs/mcp-compliance@v0
with:
target: 'node ./dist/server.js' # or a URL like https://my-server.com/mcp
format: github # ::error / ::warning annotations on the PR
strict: 'true' # exit non-zero if any required test fails
min-grade: 'A' # also exit if grade slipsManual CLI invocation:
# GitHub Actions: emits ::error / ::warning annotations inline on the PR
mcp-compliance test https://my-server.com/mcp --format github --strict
# Slack/Linear/PR comment: drop the body straight into a comment
mcp-compliance test https://my-server.com/mcp --format markdown > report.md
# HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages)
mcp-compliance test https://my-server.com/mcp --format html > report.html
# Block release if grade slips below B
mcp-compliance test https://my-server.com/mcp --min-grade B
# Preview which tests will run before connecting (handy for --only/--skip authoring)
mcp-compliance test --list --transport stdio --skip security
# Diff two runs — exit 1 if anything that was passing is now failing
mcp-compliance test https://my-server.com/mcp --format json > current.json
mcp-compliance diff baseline.json current.json
# Watch mode for stdio dev loop — re-runs on file changes in cwd
mcp-compliance test --watch -- node ./dist/server.js
# Latency benchmark
mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4Docker:
docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcpmcp-compliance initInteractive prompts walk you through transport (http/stdio), command/url, env vars, timeout, and strict mode — then write a mcp-compliance.config.json you can commit.
Check in a mcp-compliance.config.json so CI and your dev loop can run mcp-compliance test with no arguments. Supported locations (searched in order): mcp-compliance.config.json, .mcp-compliancerc.json, .mcp-compliancerc, and the "mcp-compliance" field of package.json. Pass --config <path> to load an explicit file.
HTTP:
{
"target": {
"type": "http",
"url": "https://my-server.com/mcp",
"headers": { "Authorization": "Bearer tok123" }
},
"timeout": 20000,
"strict": true
}stdio:
{
"target": {
"type": "stdio",
"command": "node",
"args": ["./dist/server.js"],
"env": { "LOG_LEVEL": "error" }
},
"skip": ["security"],
"strict": true
}Precedence: CLI flags > config file > defaults. Any field can be overridden on the command line.
mcp-compliance badge https://my-server.com/mcpRuns the compliance suite, publishes the report to mcp.hosting, and prints the markdown embed for your README. The badge image reflects the real grade (A–F) and links to the full report.
| Option | Description |
|---|---|
-H, --header <header> |
Add header to all requests, format "Key: Value" (repeatable) |
--auth <token> |
Shorthand for -H "Authorization: <token>" |
--timeout <ms> |
Request timeout in milliseconds (default: 15000) |
--no-publish |
Skip publishing; print a local badge markdown only |
--output <file> |
Also write a local SVG badge to the given path |
Reports are kept for 90 days from last submission; resubmitting the same URL overwrites the previous report. Auth headers are stripped client-side before upload. Private/loopback URLs (localhost, 127.0.0.1, 192.168.*, etc.) trigger an interactive confirmation before publishing, and are rejected by the server in any case.
A delete token is returned at publish time and stored at ~/.mcp-compliance/tokens.json (mode 0600). Use it to take a report down:
mcp-compliance unpublish https://my-server.com/mcpStdio servers can't be published (no public URL to key on), but you can commit a local SVG reflecting the real grade:
mcp-compliance test node ./dist/server.js --output badge.svg
mcp-compliance badge npx -y @modelcontextprotocol/server-filesystem /tmp --output badge.svgThen embed it in your README:
The test command never publishes — use it for CI, debugging, and local iteration. badge is the only command that publishes to mcp.hosting.
Transport (16 tests)
HTTP-only (13):
- transport-post — Server accepts HTTP POST requests (required)
- transport-content-type — Responds with application/json or text/event-stream (required)
- transport-notification-202 — Notifications return exactly 202 Accepted
- transport-content-type-reject — Rejects non-JSON request Content-Type
- transport-session-id — Enforces MCP-Session-Id after initialization
- transport-session-invalid — Returns 404 for unknown session ID
- transport-get — GET returns SSE stream or 405
- transport-delete — DELETE accepted or returns 405
- transport-batch-reject — Rejects JSON-RPC batch requests (required)
- transport-content-type-init — Initialize response has valid content type
- transport-get-stream — GET with session returns SSE or 405
- transport-concurrent — Handles concurrent requests
- transport-sse-event-field — SSE responses include required event: message field
stdio-only (3):
- stdio-framing — Newline-delimited JSON framing (required)
- stdio-unicode — UTF-8 unicode roundtrip preserves non-ASCII payloads
- stdio-unknown-method-recovers — Returns -32601 for unknown methods and keeps serving
Lifecycle (21 tests)
- lifecycle-init — Initialize handshake succeeds (required)
- lifecycle-proto-version — Returns valid YYYY-MM-DD protocol version (required)
- lifecycle-server-info — Includes serverInfo with name
- lifecycle-capabilities — Returns capabilities object (required)
- lifecycle-jsonrpc — Response is valid JSON-RPC 2.0 (required)
- lifecycle-ping — Responds to ping method (required)
- lifecycle-instructions — Instructions field is valid string if present
- lifecycle-id-match — Response ID matches request ID (required)
- lifecycle-string-id — Supports string request IDs (JSON-RPC 2.0)
- lifecycle-version-negotiate — Handles unknown protocol version gracefully
- lifecycle-reinit-reject — Rejects second initialize request
- lifecycle-logging — logging/setLevel accepted (required if logging capability declared)
- lifecycle-completions — completion/complete accepted (required if completions capability declared)
- lifecycle-cancellation — Handles cancellation notifications
- lifecycle-progress — Handles progress notifications gracefully
- lifecycle-list-changed — Accepts listChanged notifications for declared capabilities
- lifecycle-progress-token — Supports progress tokens in requests via SSE
- lifecycle-sampling-capability — Advisory check for server-side use of the client sampling capability
- lifecycle-roots-capability — Advisory check for server-side use of the client roots capability
- lifecycle-elicitation-capability — Advisory check for the 2025-11-25 client elicitation capability
- lifecycle-meta-tolerance — Server ignores unknown
_metafields on incoming requests
Tools (4 tests)
- tools-list — tools/list returns valid array (required if tools capability declared)
- tools-call — tools/call responds with correct format
- tools-pagination — tools/list supports cursor-based pagination
- tools-content-types — Tool content items have valid types
Resources (5 tests)
- resources-list — resources/list returns valid array (required if resources capability declared)
- resources-read — resources/read returns content items
- resources-templates — resources/templates/list works or returns method-not-found
- resources-pagination — resources/list supports cursor-based pagination
- resources-subscribe — Resource subscribe/unsubscribe (required if subscribe capability declared)
Prompts (3 tests)
- prompts-list — prompts/list returns valid array (required if prompts capability declared)
- prompts-get — prompts/get returns valid messages
- prompts-pagination — prompts/list supports cursor-based pagination
Error Handling (10 tests)
- error-unknown-method — Returns JSON-RPC error for unknown method (required)
- error-method-code — Uses correct -32601 error code
- error-invalid-jsonrpc — Handles malformed JSON-RPC (required)
- error-invalid-json — Handles invalid JSON body
- error-missing-params — Returns error for tools/call without name
- error-parse-code — Returns -32700 for invalid JSON
- error-invalid-request-code — Returns -32600 for invalid request
- tools-call-unknown — Returns error for nonexistent tool name
- error-capability-gated — Rejects methods for undeclared capabilities
- error-invalid-cursor — Handles invalid pagination cursor gracefully
Schema Validation (6 tests)
- tools-schema — All tools have valid name and inputSchema (required if tools capability declared)
- tools-annotations — Tool annotations are valid if present
- tools-title-field — Tools include title field (2025-11-25)
- tools-output-schema — Tools with outputSchema are valid (2025-11-25)
- prompts-schema — Prompts have valid name field (required if prompts capability declared)
- resources-schema — Resources have valid uri and name (required if resources capability declared)
Security (23 tests)
- security-auth-required — Rejects unauthenticated requests
- security-www-authenticate — 401 responses include WWW-Authenticate header
- security-auth-malformed — Rejects malformed auth credentials
- security-tls-required — Enforces HTTPS/TLS
- security-session-entropy — Session IDs are high-entropy
- security-session-not-auth — Session ID does not bypass auth
- security-oauth-metadata — Protected Resource Metadata endpoint exists (RFC 9728)
- security-token-in-uri — Rejects auth tokens in query string
- security-cors-headers — CORS headers are restrictive
- security-origin-validation — Validates Origin header for DNS rebinding protection
- security-command-injection — Resists command injection in tool params
- security-sql-injection — Resists SQL injection in tool params
- security-path-traversal — Resists path traversal in tool params
- security-ssrf-internal — Resists SSRF to internal networks
- security-oversized-input — Handles oversized inputs gracefully
- security-extra-params — Rejects or ignores extra tool params
- security-tool-schema-defined — All tools define inputSchema
- security-tool-rug-pull — Tool definitions are stable across calls
- security-tool-description-poisoning — Tool descriptions free of injection patterns
- security-tool-cross-reference — Tools do not reference other tools by name
- security-error-no-stacktrace — Error responses do not leak stack traces
- security-error-no-internal-ip — Error responses do not leak internal IPs
- security-rate-limiting — Rate limiting is enforced
| Grade | Score |
|---|---|
| A | 90-100 |
| B | 75-89 |
| C | 60-74 |
| D | 40-59 |
| F | 0-39 |
Required tests are worth 70% of the score, optional tests 30%. See the full scoring algorithm in the methodology doc.
# GitHub Actions example
- name: MCP Compliance Check
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict# With JSON output for parsing
- name: MCP Compliance Check
run: |
npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format json > compliance.json
cat compliance.json | jq '.grade'# With retries for flaky network conditions
- name: MCP Compliance Check
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000# SARIF output for GitHub Code Scanning
- name: MCP Compliance Check
run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif
- name: Upload SARIF
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: compliance.sarifThis package also exposes an MCP server with 3 tools that can be used from Claude Code, Cursor, or any MCP client.
Claude Code (one-liner):
claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance mcpOr create .mcp.json in your project root:
macOS / Linux / WSL:
{
"mcpServers": {
"mcp-compliance": {
"command": "npx",
"args": ["-y", "@yawlabs/mcp-compliance", "mcp"]
}
}
}Windows:
{
"mcpServers": {
"mcp-compliance": {
"command": "cmd",
"args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance", "mcp"]
}
}
}Tip: This file is safe to commit — it contains no secrets.
Restart your MCP client and approve the server when prompted.
- mcp_compliance_test — Run the full 88-test suite against a URL or stdio command. Supports auth, custom headers, env vars, timeout, retries, and category/test filtering. Returns grade, score, and detailed results.
- mcp_compliance_badge — Get the badge markdown/HTML for a server. Supports auth and custom headers.
- mcp_compliance_explain — Explain what a specific test ID checks and why it matters.
All tools have MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) so MCP clients can skip confirmation dialogs for safe operations.
import { runComplianceSuite } from '@yawlabs/mcp-compliance';
const report = await runComplianceSuite('https://my-server.com/mcp');
console.log(`Grade: ${report.grade} (${report.score}%)`);
// With options
const report2 = await runComplianceSuite('https://my-server.com/mcp', {
headers: { 'Authorization': 'Bearer tok123' },
timeout: 30000,
retries: 1,
only: ['transport', 'lifecycle'],
});
// Live progress for streaming UIs (e.g. server-sent-events to a browser)
await runComplianceSuite('https://my-server.com/mcp', {
onTestComplete: (result) => {
// result has the full TestResult: id, name, category, required,
// passed, details, durationMs, specRef. Push it to your client.
sendToClient(result);
},
});The JSON output of the test suite is a stable, versioned contract. Every report includes a schemaVersion field at the top level. The full JSON Schema lives at schemas/report.v1.json and is shipped with the npm package.
Consumer guidance:
- Pin against
schemaVersion. Reject reports with an unknown version rather than guessing at the shape. - The schema validates with any Draft 2020-12 validator (e.g.
ajv). - Within a major version, additions are non-breaking. Renames, removals, or type changes bump the version.
- Two runs against the same server produce equivalent grade, score, and per-test pass/fail (modulo timings/timestamps).
The testing methodology is published openly so the grading is auditable:
- Testing methodology — test execution model, scoring algorithm, all 88 test rules with pass/fail criteria (CC BY 4.0)
- Machine-readable rule catalog — JSON Schema-compliant catalog for programmatic consumption
- Why
mcp-compliance— the problem, existing alternatives, what this tool does differently - Fixing common failures — recipes for the most frequent test failures with code snippets
- Spec version migration policy — how this tool evolves with MCP spec releases
- mcp.hosting external API — public submit/retrieve/badge/delete endpoints used by
mcp-compliance badgeand any custom integrations - Enterprise tier (draft) — paid tier structure for organizations with scheduled/private/audit-track compliance needs
- Performance deep-dive — why the suite is sequential and what parallel execution would cost
- Spec PR drafts — our proposed MCP spec clarifications for ambiguous cases we've hit
- mcp.hosting integration spec — the contract between this engine and the mcp.hosting platform: URL surfaces, data flow, storage model, badge API, leaderboard, router integration
The methodology is not an authoritative conformance standard — it's one tool's choices, published so they can be inspected, adopted, or forked. The official MCP specification defines what servers must do; this document describes how @yawlabs/mcp-compliance verifies it.
- Node.js 18+
git clone https://github.com/YawLabs/mcp-compliance.git
cd mcp-compliance
npm install
npm run build
npm testDevelopment commands:
| Command | Description |
|---|---|
npm run build |
Compile with tsup |
npm run dev |
Watch mode |
npm test |
Run tests (vitest) |
npm run lint |
Check with Biome |
npm run lint:fix |
Auto-fix with Biome |
npm run typecheck |
TypeScript type checking |
npm run test:ci |
Build + test (CI-safe) |
- mcp.hosting — Hosted MCP server infrastructure
- MCP Specification
- Testing methodology
- Yaw Labs
MIT — see LICENSE.
{ "schemaVersion": "1", // bumped on breaking changes to the report shape "specVersion": "2025-11-25", // MCP spec version tested against "toolVersion": "0.10.0", // mcp-compliance version that produced the report "url": "...", "timestamp": "...", "grade": "A", "score": 92.5, "tests": [ ... ], // ... }