Feat: Add API audit tooling for Meshery Server and Meshery Cloud#700
Feat: Add API audit tooling for Meshery Server and Meshery Cloud#700PragalvaXFREZ wants to merge 6 commits intomeshery:masterfrom
Conversation
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a Go-based AST analysis tool to replace regex-based heuristics for API auditing, along with new Makefile targets to manage the audit workflow. Key feedback includes addressing a missing Makefile dependency (api-audit-update-check), ensuring the api-audit-update target correctly handles Google Sheets parameters, improving error handling in file traversal, and adopting a more robust approach for parsing JSON struct tags.
There was a problem hiding this comment.
Pull request overview
Adds an API audit tool to meshery-schemas that compares the bundled OpenAPI spec (_openapi_build/merged_openapi.yml) against route registrations and handler implementations in Meshery Server and Meshery Cloud, with optional Google Sheet snapshot updates.
Changes:
- Add Makefile targets to set up and run the API audit (dry-run and sheet-update flows).
- Introduce a Python audit entrypoint (
build/scripts/api-audit.py) that parses OpenAPI + router definitions and can update a Google Sheet. - Add a Go AST helper (
build/scripts/analyze_handlers/main.go) to extract handler I/O types, schema import usage, and JSON field names for deeper completeness checks.
Reviewed changes
Copilot reviewed 2 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| Makefile | Adds api-audit* targets for setup, auditing, refresh, and (intended) sheet-update flows. |
| build/scripts/api-audit.py | Implements the audit pipeline (spec parsing, router parsing for Gorilla Mux + Echo, optional sheet diff/update). |
| build/scripts/analyze_handlers/main.go | Provides Go AST-based analysis to support handler classification and field-level completeness cross-checks. |
| .gitignore | Ignores the audit virtualenv and Python bytecode/cache artifacts. |
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Signed-off-by: Pragalva Sapkota <sapkotapragalva@gmail.com>
Notes for Reviewers
This PR fixes #
Summary
This PR introduces a new API audit tool in
meshery-schemasfor checking how the bundled OpenAPI spec lines up with real route registrations and handler implementations in Meshery Server and Meshery Cloud.The tool is designed to answer these questions:
What The Audit Checks
The audit combines multiple inputs:
meshery-schemas/_openapi_build/merged_openapi.ymlas the authoritative bundled OpenAPI specMESHERY_REPOis providedCLOUD_REPOis providedFor each endpoint, the tool reports:
x-internaltagging informationIt also supports combined analysis of Meshery Server and Meshery Cloud in one run, so the final summary and sheet output can show platform-specific results side by side.
Capabilities Included In This PR
build/scripts/api-audit.pybuild/scripts/analyze_handlers/main.gofor handler-level analysisThe Go helper is used to extract:
Dependencies And Prerequisites
Before using the audit flow:
gois installed if you want AST-based handler analysisPython dependencies are installed by
make api-audit-setup, which is already a prerequisite ofmake api-auditandmake api-audit-update:pyyamlgspreadgoogle-authIf
gois unavailable, the script falls back without AST analysis and handler classification becomes limited.Recommended Environment Setup
For the most complete audit run, set:
MESHERY_REPOto the local Meshery Server repo pathCLOUD_REPOto the local Meshery Cloud repo pathSHEET_IDwhen you want to write results to the Google SheetGOOGLE_CREDENTIALS_JSONorGOOGLE_APPLICATION_CREDENTIALSwhen sheet updates are neededOPENAPI_SPEC_PATHonly if you want to override the default bundled spec pathMain Commands
make api-auditruns the audit in dry-run mode and prints the summary without updating the sheetScreenshot:
make api-audit-updateruns the audit and updates the Google Sheet whenSHEET_IDand Google credentials are configuredSigned commits