api-testing-core

API testing CLIs overview

Overview

This workspace ships five Rust API testing CLIs (api-rest, api-gql, api-grpc, api-websocket, api-test) plus the shared library crate api-testing-core. The canonical Rust CLI contracts are the source of truth: flags, defaults, exit codes, and on-disk artifacts (history files, reports, results).

Detailed command specs live with each binary:

• crates/api-rest/README.md
• crates/api-gql/README.md
• crates/api-grpc/README.md
• crates/api-websocket/README.md
• crates/api-test/README.md

This README focuses on the shared repository layout, cross-CLI concepts, and the api-testing-core surface area.

Binaries and command surface mapping

• api-rest (REST)
  • api-rest call (default)
  • api-rest history
  • api-rest report
• api-gql (GraphQL)
  • api-gql call (default)
  • api-gql history
  • api-gql report
  • api-gql schema
• api-test (suite runner)
  • api-test run
  • api-test summary
• api-grpc (gRPC unary)
  • api-grpc call (default)
  • api-grpc history
  • api-grpc report
  • api-grpc report-from-cmd
• api-websocket (WebSocket scripted sessions)
  • api-websocket call (default)
  • api-websocket history
  • api-websocket report
  • api-websocket report-from-cmd

Notes:

• api-test executes REST/GraphQL/gRPC/WebSocket cases through shared core runners (no shelling out to scripts or the binaries).
• api-rest/api-gql/api-grpc/api-websocket also provide report-from-cmd as a Rust-only convenience for replaying saved call snippets.

api-testing-core scope

api-testing-core is a library crate used by all five CLIs. Key modules:

• config: setup dir discovery for REST/GraphQL/gRPC/WebSocket configs.
• env_file: .env parsing + key normalization helpers.
• cli_*: shared CLI helpers (endpoint resolution, history I/O, report args, CLI utilities).
• history, report, markdown, redact, cmd_snippet: shared report/history rendering and snippet parsing.
• rest: request schema, runner, expect/cleanup logic, report rendering.
• graphql: schema/vars loading, auth/JWT resolution, runner, expect/allow-errors, report rendering, mutation detection.
• grpc: unary request schema, transport runner, expect logic, report rendering.
• websocket: request schema, scripted runner, expect logic, report rendering.
• suite: suite schema v1, path resolution, filters, safety gates, auth integration, runner, cleanup, results, summary, JUnit (including type: websocket).

Transport decision and reuse matrix

• Transport decision:
  • gRPC selected: grpcurl adapter for unary MVP (api-testing-core::grpc::runner).
  • gRPC rejected for MVP: native dynamic invocation path (higher complexity for the same unary delivery goal).
  • WebSocket selected: native Rust tungstenite transport (api-testing-core::websocket::runner).
  • WebSocket rejected for MVP: external adapter shell-out (websocat-style).
• Reuse matrix:
  • unchanged: suite selection/filtering, run directory/artifact envelope, summary/JUnit/results rendering.
  • additive grpc: suite schema defaults/case validation, type: grpc runner branch, gRPC endpoint/token resolution.
  • additive websocket: suite schema defaults/case validation, type: websocket runner branch, WS endpoint/token resolution.
• Evidence commands:
  • cargo test -p nils-api-testing-core --test suite_rest_graphql_matrix
  • cargo test -p nils-api-testing-core --test suite_runner_loopback
  • cargo test -p nils-api-testing-core --test suite_runner_grpc_matrix
  • cargo test -p nils-api-testing-core --test suite_runner_websocket_matrix
  • cargo test -p nils-api-test suite_schema

Shared terminology

• Setup dir
  • Protocol-specific config directory.
  • Canonical locations: REST setup/rest, GraphQL setup/graphql, gRPC setup/grpc, WebSocket setup/websocket.
• Config dir
  • CLI arg --config-dir <dir> that seeds setup-dir discovery.
  • Discovery searches upward for known config files; fallback uses the canonical setup/<tool> when applicable.
• Env preset
  • --env <name> selects a base URL from endpoints.env (+ optional .local overrides).
  • REST: REST_URL_<ENV_KEY>; GraphQL: GQL_URL_<ENV_KEY>; gRPC: GRPC_URL_<ENV_KEY>.
  • WebSocket: WS_URL_<ENV_KEY>.
  • If the value looks like http(s)://... or ws(s)://..., it is treated as a direct URL (like --url).
• Token/JWT profile
  • REST: --token <name> or REST_TOKEN_NAME selects REST_TOKEN_<NAME>.
  • GraphQL: --jwt <name> or GQL_JWT_NAME selects GQL_JWT_<NAME>.
  • gRPC: --token <name> or GRPC_TOKEN_NAME selects GRPC_TOKEN_<NAME>.
  • WebSocket: --token <name> or WS_TOKEN_NAME selects WS_TOKEN_<NAME>.
  • REST fallback: ACCESS_TOKEN, then SERVICE_TOKEN if no profile is selected.
  • GraphQL fallback: ACCESS_TOKEN, then SERVICE_TOKEN if no profile is selected.
• History
  • REST: <setup_dir>/.rest_history, GraphQL: <setup_dir>/.gql_history, gRPC: <setup_dir>/.grpc_history, WebSocket: <setup_dir>/.ws_history.
  • Enabled by default, can be disabled, and supports rotation/size limits.
• Report
  • Markdown artifact (usually under <repo>/docs/) capturing request/operation, response, and optional assertions.
  • Redacts common secret fields by default, with opt-out flags for debugging.
• Suite
  • JSON manifest (version: 1) that drives api-test run and defines cases, defaults, auth, and cleanup.

Canonical repo layouts

The CLIs support the following repository layouts.

Layout A: App repo with setup/ + tests/ (recommended)

<repo>/
  setup/
    rest/
      endpoints.env
      endpoints.local.env        # optional (local override)
      tokens.env
      tokens.local.env           # optional (local tokens; do not commit)
    graphql/
      endpoints.env
      endpoints.local.env        # optional (local override)
      jwts.env
      jwts.local.env             # optional (local tokens; do not commit)
      schema.env                 # optional (sets GQL_SCHEMA_FILE)
      schema.local.env           # optional (local override)
      schema.graphql             # or: schema.gql / schema.graphqls / api.graphql / api.gql
      operations/                # optional (login.graphql, shared ops, etc.)
    grpc/
      endpoints.env
      endpoints.local.env        # optional (local override)
      tokens.env
      tokens.local.env           # optional (local tokens; do not commit)
      requests/                  # *.grpc.json request definitions
    websocket/
      endpoints.env
      endpoints.local.env        # optional (local override)
      tokens.env
      tokens.local.env           # optional (local tokens; do not commit)
      requests/                  # *.ws.json or *.websocket.json request definitions
  tests/
    api/
      suites/
        <name>.suite.json
  out/
    api-test-runner/             # suite runner output base dir

Layout B: Suites under setup/ (fallback)

api-test resolves --suite <name> to:

• <repo>/tests/api/suites/<name>.suite.json (preferred)
• <repo>/setup/api/suites/<name>.suite.json (fallback)

Layout C: Custom suites directory

• API_TEST_SUITES_DIR=<path> overrides the suites directory for --suite <name>.

Quickstart examples

api-rest

Run a request:

api-rest call --env staging setup/rest/requests/health.request.json

Write a report:

api-rest report --case health --request setup/rest/requests/health.request.json --run

Generate a report from a saved snippet:

api-rest history --command-only | api-rest report-from-cmd --stdin

api-gql

Run an operation:

api-gql call --env staging setup/graphql/operations/health.graphql

Write a report:

api-gql report --case health --op setup/graphql/operations/health.graphql --run

Resolve and print schema:

api-gql schema --cat

api-grpc

Run a unary request:

api-grpc call --env staging setup/grpc/requests/health.grpc.json

Write a report:

api-grpc report --case health --request setup/grpc/requests/health.grpc.json --run

Generate a report from a saved snippet:

api-grpc history --command-only | api-grpc report-from-cmd --stdin

api-test

Run a suite (always emits results JSON to stdout):

api-test run --suite smoke

Use an explicit suite file:

api-test run --suite-file tests/api/suites/smoke.suite.json

Write results JSON + JUnit:

api-test run --suite smoke --out out/api-test-runner/results.json --junit out/api-test-runner/junit.xml

Render a Markdown summary:

api-test summary --in out/api-test-runner/results.json --out out/api-test-runner/summary.md

api-websocket

Run a scripted request:

api-websocket call --env staging setup/websocket/requests/health.ws.json

Write a report:

api-websocket report --case health --request setup/websocket/requests/health.ws.json --run

Generate a report from a saved snippet:

api-websocket history --command-only | api-websocket report-from-cmd --stdin

Suite runner behavior (high-level)

• Results JSON is always emitted to stdout. --out writes an additional copy.
• Output directory base defaults to <repo>/out/api-test-runner (override via API_TEST_OUTPUT_DIR).
• Each run creates <output_dir>/<run_id>/ where run_id is YYYYMMDD-HHMMSSZ.
• Per-case artifacts include <case>.response.json and <case>.stderr.log, referenced in results JSON.
• Exit code is 2 when any case fails; otherwise 0.
• Write safety is two-step:
  • Case must set allowWrite: true.
  • Writes must be enabled via --allow-writes, API_TEST_ALLOW_WRITES_ENABLED=true, or env: local.
• Optional suite auth can derive tokens from secret JSON in API_TEST_AUTH_JSON.
• Optional cleanup steps run after the main case; cleanup failures mark the case failed.

Suite runner environment variables

• API_TEST_OUTPUT_DIR: override the base output directory.
• API_TEST_SUITES_DIR: override the suites directory used by --suite.
• API_TEST_ALLOW_WRITES_ENABLED: enable write-capable cases.
• API_TEST_REST_URL: override REST base URL for all REST/rest-flow cases.
• API_TEST_GQL_URL: override GraphQL endpoint URL for all GraphQL cases.
• API_TEST_GRPC_URL: override gRPC target for all gRPC cases.
• API_TEST_WS_URL: override WebSocket target URL for all websocket cases.
• API_TEST_AUTH_JSON: credentials JSON used by suite auth (default key name).
• GITHUB_STEP_SUMMARY: when set, api-test summary appends Markdown output (disable via --no-github-summary).

Docs

• Docs index