Skip to content

Feat: Add API audit tooling for Meshery Server and Meshery Cloud#700

Draft
PragalvaXFREZ wants to merge 6 commits intomeshery:masterfrom
PragalvaXFREZ:feat/api-audit
Draft

Feat: Add API audit tooling for Meshery Server and Meshery Cloud#700
PragalvaXFREZ wants to merge 6 commits intomeshery:masterfrom
PragalvaXFREZ:feat/api-audit

Conversation

@PragalvaXFREZ
Copy link
Copy Markdown
Member

@PragalvaXFREZ PragalvaXFREZ commented Apr 3, 2026

Notes for Reviewers

This PR fixes #

Summary

This PR introduces a new API audit tool in meshery-schemas for checking how the bundled OpenAPI spec lines up with real route registrations and handler implementations in Meshery Server and Meshery Cloud.

The tool is designed to answer these questions:

  • What endpoints exist in the bundled OpenAPI spec?
  • Which endpoints are actually registered in Meshery Server and/or Meshery Cloud?
  • How schema-backed, complete, and schema-driven are those implementations?

What The Audit Checks

The audit combines multiple inputs:

  • meshery-schemas/_openapi_build/merged_openapi.yml as the authoritative bundled OpenAPI spec
  • Meshery Server router and handlers when MESHERY_REPO is provided
  • Meshery Cloud router and handlers when CLOUD_REPO is provided

For each endpoint, the tool reports:

  • Coverage: overlap, server underlap, or schema underlap
  • Endpoint status: active, deprecated, unimplemented, or cloud-only
  • Whether the endpoint is schema-backed
  • Schema completeness
  • Whether the handler is schema-driven
  • x-internal tagging information

It also supports combined analysis of Meshery Server and Meshery Cloud in one run, so the final summary and sheet output can show platform-specific results side by side.

Capabilities Included In This PR

  • A new Python audit entrypoint in build/scripts/api-audit.py
  • Make targets for setup, dry-run audit, refresh, update, and cloud-only audit flows
  • Router analysis for both Gorilla Mux and Echo-based route definitions
  • A Go AST-based helper in build/scripts/analyze_handlers/main.go for handler-level analysis
  • Platform-aware summary output for Meshery Server and Meshery Cloud
  • Optional Google Sheet update support for storing audit snapshots

The Go helper is used to extract:

  • per-handler schema import usage
  • request and response types
  • transitive aliases from local models to schema packages
  • JSON struct field names from handlers, local models, and schema models

Dependencies And Prerequisites

Before using the audit flow:

  • Ensure go is installed if you want AST-based handler analysis

Python dependencies are installed by make api-audit-setup, which is already a prerequisite of make api-audit and make api-audit-update:

  • pyyaml
  • gspread
  • google-auth

If go is unavailable, the script falls back without AST analysis and handler classification becomes limited.

Recommended Environment Setup

For the most complete audit run, set:

  • MESHERY_REPO to the local Meshery Server repo path
  • CLOUD_REPO to the local Meshery Cloud repo path
  • SHEET_ID when you want to write results to the Google Sheet
  • GOOGLE_CREDENTIALS_JSON or GOOGLE_APPLICATION_CREDENTIALS when sheet updates are needed
  • OPENAPI_SPEC_PATH only if you want to override the default bundled spec path

Main Commands

  • make api-audit runs the audit in dry-run mode and prints the summary without updating the sheet
    Screenshot:
image
  • make api-audit-update runs the audit and updates the Google Sheet when SHEET_ID and Google credentials are configured

Signed commits

  • Yes, I signed my commits.

Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Go-based AST analysis tool to replace regex-based heuristics for API auditing, along with new Makefile targets to manage the audit workflow. Key feedback includes addressing a missing Makefile dependency (api-audit-update-check), ensuring the api-audit-update target correctly handles Google Sheets parameters, improving error handling in file traversal, and adopting a more robust approach for parsing JSON struct tags.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an API audit tool to meshery-schemas that compares the bundled OpenAPI spec (_openapi_build/merged_openapi.yml) against route registrations and handler implementations in Meshery Server and Meshery Cloud, with optional Google Sheet snapshot updates.

Changes:

  • Add Makefile targets to set up and run the API audit (dry-run and sheet-update flows).
  • Introduce a Python audit entrypoint (build/scripts/api-audit.py) that parses OpenAPI + router definitions and can update a Google Sheet.
  • Add a Go AST helper (build/scripts/analyze_handlers/main.go) to extract handler I/O types, schema import usage, and JSON field names for deeper completeness checks.

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 4 comments.

File Description
Makefile Adds api-audit* targets for setup, auditing, refresh, and (intended) sheet-update flows.
build/scripts/api-audit.py Implements the audit pipeline (spec parsing, router parsing for Gorilla Mux + Echo, optional sheet diff/update).
build/scripts/analyze_handlers/main.go Provides Go AST-based analysis to support handler classification and field-level completeness cross-checks.
.gitignore Ignores the audit virtualenv and Python bytecode/cache artifacts.

Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants