CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file. The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[3.0.5] - 2026-04-21

Fixed (3.0.5)

• Latency Metrics: Make processingTime consistently represent server-side processing time only, excluding any configured responseDelayMs simulation from persisted log data.
• Performance: Cache compiled JSON Schema validators so repeated requests and stable hot-reload configurations do not trigger unnecessary recompilation on the webhook path.
• Lifecycle: Reset webhook-manager singleton state during test teardown and recreate the sync-service limiter after shutdown so repeated initialize/stop/start flows remain stable.
• Lifecycle: Cache the active SyncService limiter stop promise so disconnect cleanup can be retried safely after a partial stop failure, and so a later start() drains stale limiter cleanup before creating a replacement scheduler.
• Shutdown Ordering: Drain the HTTP listener before stopping SyncService and closing DuckDB so in-flight requests and readiness probes do not race read-model teardown.
• DuckDB Lifecycle: Drain both pooled and in-use DuckDB connections before resetting the singleton so repeated DB teardown and rebuild flows do not leave stale handles behind.
• DuckDB Reset Coordination: Keep the reset gate for new DuckDB callers while allowing already-queued writes and transactions to drain first, preventing reset deadlocks in the serialized write path.
• DuckDB Init Invalidation: Prevent an in-flight DuckDB initializer from republishing a stale singleton after resetDbInstance() clears teardown state, and allow clean reinitialization even when that late initializer ultimately fails.
• Contracts & Docs: Align dataset, output, OpenAPI, README, architecture, and API reference documentation with the finalized processingTime semantics and response-delay behavior.
• Worker Cleanup: Remove the custom script executor's void cleanup chaining and keep message/error/exit settlement deterministic even when worker listener cleanup throws.
• Tests: Add regression coverage for latency semantics, validator-cache reuse and memoized schema cache keys, DuckDB reset coordination with active reads plus queued write/transaction drain paths, malformed JSON sanitation persistence, restart-safe integration harness cleanup, shutdown-only sync error suppression, and spawned-process close-path resilience.
• Tests: Add focused regression coverage for SyncService stop/start retry paths and custom-script worker cleanup failure handling, and drive the touched files to 100% focused unit coverage before rerunning the stress suite.
• Tests: Make setupTestApp() fail fast when node:fs/promises.mkdtemp() is mocked incorrectly, and keep in-process integration suites on a real temp-directory implementation so APIFY_LOCAL_STORAGE_DIR isolation stays intact.
• Tooling: Remove the unused cross-env dependency and keep npm run test:stress on a direct Node + Jest invocation with --expose-gc, which works after a native Windows dependency install regenerates the local toolchain.

[3.0.4] - 2026-04-18

Fixed (3.0.4)

• Apify: Relax the dataset storage schema for body and responseBody so webhook events can be stored when request or response payloads are captured as any JSON-compatible values instead of only strings and objects.
• Tests: Add a regression test that keeps the Actor dataset schema aligned with the webhook payload shapes persisted by the runtime.

[3.0.3] - 2026-04-18

Fixed (3.0.3)

• Documentation: Convert README asset and document links to absolute GitHub URLs so the Apify Store listing renders images and documentation links correctly outside the repository context.

[3.0.2] - 2026-04-17

Fixed (3.0.2)

• Apify: Restore valid Actor input schema metadata by adding the missing nested alert channel descriptions required by Apify schema validation.
• Apify: Reorder signatureVerificationSecret so it appears directly above signatureVerification in the Actor input UI while preserving the same backward-compatible runtime mapping.
• CI/CD: Validate the Actor input schema with apify validate-schema alongside the web server schema so Apify build-time schema errors are caught before release.
• CI/CD: Split npm validation from npm publishing so id-token: write is granted only during release publishing, and keep Docker PR validation tags limited to the single synthetic PR tag.
• CI/CD: Normalize known flaky external link handling in the link check workflow by matching stable URL origin-and-path keys instead of full query-string URLs.
• Tooling: Stop hardcoding the Apify CLI package version inside the validation script by invoking the project-installed apify binary instead.
• Tooling: Add @apify/input_secrets as an explicit development dependency and align sync-version test fixtures with the production HTTP status key style.

[3.0.1] - 2026-04-15

Fixed (3.0.1)

• CI/CD: Make the release publish workflow use an explicit Jest invocation that force-exits after the suite completes.
• CI/CD: Make the release-only npm and Docker workflows report successful pull request checks without publishing artifacts, so Dependabot PRs do not remain stuck waiting on required checks.
• CI/CD: Make the required Link Check workflow run on every pull request so dependency-only PRs no longer remain stuck in an expected state when path filters skip the job.
• Apify: Move the webhook signing secret to a top-level signatureVerificationSecret input with isSecret: true, while keeping runtime compatibility with older nested secret values.
• Forwarding: Implement strict Content-Length parsing and enhance fallback logic in ForwardingService.
• GitHub/NPM: Update CI/CD, package.json and package-lock.json to use the latest versions of various dependencies.

[3.0.0] - 2026-04-02

Added (3.0.0)

• Standby Web Server Contract: Added .actor/web_server_schema.json and wired it through .actor/actor.json with usesStandbyMode: true, turning the Actor into a documented long-lived web service.
• Expanded HTTP Surface: Added or formalized routes for dashboard, runtime info, log queries, log detail, payload retrieval, replay, streaming, health, readiness, and system metrics.
• Webhook Signature Verification: Added provider-aware verification for Stripe, Shopify, GitHub, Slack, and custom HMAC integrations.
• Standalone Self-Hosting Track: Added a runtime-standalone target in the main Dockerfile for running the product outside the default Apify image path.
• Environment Bootstrapping: Added .env.example and project-level .env loading support for local CLI and self-hosted workflows.
• Documentation Set: Added docs/api-reference.md, docs/architecture.md, multiple operational playbooks, roadmap notes, and expanded publication/release guidance.

Improved (3.0.0)

• Architecture: Refactored the runtime into a clearer modular monolith split across routes, middleware, services, repositories, consts, and utils.
• DuckDB Integration: Migrated to @duckdb/node-api with cached instance management, pooled connections, and serialized writes for better stability under load.
• Log Querying: Upgraded /logs with richer filters, range parsing, sort controls, signature filters, and cursor-based pagination.
• Hot Reloading: Improved runtime config refresh with both Apify key-value-store polling and local filesystem watching.
• Operator Controls: Expanded input schema support for replay retries/timeouts, memory overrides, redaction, alerting, forwarding, verification, and response simulation.
• Quality Tooling: Added coverage matrix tooling, schema validation scripts, version synchronization helpers, and stronger CI/release automation.

Security (3.0.0)

• Access Control: Hardened management and ingress authentication behavior when authKey is configured.
• Traffic Protection: Split rate limiting between management endpoints and per-webhook ingestion flows.
• Forwarding Safety: Added recursion detection to block self-referential forwarding loops.
• Data Protection: Strengthened sensitive-header and body-path redaction options for captured traffic.

Testing (3.0.0)

• Test Pyramid Restructure: Reorganized tests into explicit unit, integration, and e2e suites.
• Harness Depth: Added reusable helpers for DB lifecycle management, process harnessing, payload fixtures, middleware testing, signature assertions, and mock orchestration.
• Coverage Depth: Added broad route, repository, middleware, service, utility, and script coverage for the new modular runtime.

[2.8.7] - 2026-01-15

Fixed (2.8.7)

• Removed redundant dataset_csv and dataset_json views from output schema (native "Storage" tab should be used for downloads).
• Documentation: Updated Stripe Playbook with correct official IP range links.

[2.8.6] - 2026-01-15

Fixed (2.8.6)

• Removed clean parameter from output schema views to resolve Apify Console type validation error.

[2.8.5] - 2026-01-14

Fixed (2.8.5)

• Documentation: Fixed bugs in output_schema.json views (parameter types and path formatting).
• Documentation: Updated download links to use clean=true.

[2.8.4] - 2026-01-14

Improved (2.8.4)

• Documentation: Consolidated "Self-Hosting" sections in README.md into a single, comprehensive guide.
• Documentation: Highlighted the "Activity-Based Retention" policy with a prominent alert block for better user visibility.

[2.8.3] - 2026-01-13

Fixed (2.8.3)

• CLI Compatibility: The Actor now correctly respects the INPUT environment variable when running via npx, overriding any local INPUT.json artifacts. This restores full stateless CLI functionality (e.g., INPUT='{...}' npx ...).
• NPM Publishing: Fixed an authentication issue in the CI/CD pipeline by correctly balancing OIDC provenance with legacy setup-node requirements.

Improved (2.8.3)

• Startup UX: Clarified startup logs to distinguish between "Initializing" (0 -> N webhooks) and "Scaling Up" (N -> M webhooks).
• Log Noise: Silenced the "Refreshed retention" log for insignificant updates (< 5 mins), preventing console spam during quick restarts.

[2.8.2] - 2026-01-13

Added (2.8.2)

• UX: Automatic INPUT.json creation for npx users (Zero-Conf Hot-Reload).

Security (2.8.2)

• NPM: Switched to OIDC Trusted Publishing for verified package provenance.

Improved (2.8.2)

• Type Safety: Removed all @ts-ignore directives in favor of proper JSDoc casting.
• Documentation: Clarified npx usage and fixed broken anchor links.

[2.8.1] - 2026-01-11

Fixed (2.8.1)

• UI: Fixed a regression where {{VERSION}} placeholders in index.html were not being fully replaced, causing raw template strings to be visible.

Improved (2.8.1)

• UX: The /log-stream SSE endpoint now sends an immediate : connected comment upon connection, preventing browsers from showing a "loading" state indefinitely while waiting for the first log event.

DevOps (2.8.1)

• Docker Verification: Added a comprehensive local_docker_testing.md guide and a new verify-docker CI/CD job. This ensures production-only bugs (like SSE compression issues) are caught in the container environment before deployment.

[2.8.0] - 2026-01-11

Added (2.8.0)

• Robust Paginated Replay Search: Implemented "Deep Search" for the /replay endpoint. It now defaults to checking the 1000 most recent items (fast path) but automatically paginates through older history if the target event is not found, preventing 404s for valid older events while avoiding OOM crashes.
• SSRF Protection: Added a shared src/utils/ssrf.js utility with DNS resolution and IP range validation. Applied this protection to both HTTP Forwarding and Replay APIs to prevent internal network scanning.
• Community Standards: Added CODE_OF_CONDUCT.md, CONTRIBUTING.md, SECURITY.md, LICENSE, and GitHub Issue Templates to meet open-source best practices.

Improved (2.8.0)

• Configuration Refactoring: Centralized input validation and default value logic (including maxPayloadSize and rateLimitPerMinute) into src/utils/config.js (coerceRuntimeOptions). This ensures consistent behavior between initial startup and hot-reloading.
• Test Suite overhaul:
  • Achieved >90% Statement/Line coverage and >80% Branch coverage (144 tests total).
  • Enforced strict type checking (removed @ts-nocheck) and standardized Jest mocks across all test files.
  • Added specific test suites for SSRF protection, Config Validation, and Edge Cases (coverage_gaps.test.js).
• Security Hardening:
  • Header Stripping: Automatically strips hop-by-hop headers and sensitive headers (Authorization, Cookie) during forwarding.
  • Input Sanitization: Enhanced error handling to prevent information leakage and sanitized req.query inputs.
• Code Quality: Applied Prettier formatting project-wide (trailing commas, consistent indentation) and resolved all ESLint/TypeScript errors.

Fixed (2.8.0)

• Replay Memory Safety: Prevented dataset.getData() from loading the entire dataset into memory by enforcing pagination limit (1000 items/page).
• Hot-Reload Stability: Fixed potential crashes during script re-compilation and ensured stale schemas are cleared on failure.
• API Robustness:
  • Added req.forcedStatus coercion validation.
  • Fixed handling of "all retries exhausted" in replay logic to prevent undefined errors.
  • Ensures correct handling of disconnected clients in SSE (/log-stream).

Refactored (2.7.2)

• Code Reusability: Centralized system constants and type definitions into src/consts.js and src/typedefs.js for better maintainability.
• Test Infrastructure: Standardized mock generation (createDatasetMock) and utility helpers to reduce test boilerplate and improve stability.

[2.7.1] - 2026-01-08

Improved (2.7.1)

• Quality Assurance: Achieved >90% test coverage for Lines/Statements and implemented strict type checking across the entire test suite.
• Reliability: Hardened forwarding retry logic and platform limit handling with comprehensive new test cases.
• Maintainability: Removed all // @ts-nocheck directives and standardized Jest mocks for long-term stability.

[2.7.0] - 2025-12-31

Added (2.7.0)

• Dynamic Infrastructure Scaling & Hot-Reloading:
  • Implemented Actor Hot-Reloading: Configuration changes (Auth Keys, Allowed IPs, Scripts, Schemas) now apply in real-time via Actor.on('input') without Actor restarts.
  • Implemented urlCount Reconciliation: The Actor now automatically generates missing webhooks on restart (or hot-reload) if the count is increased, preserving existing IDs.
  • Implemented Retention Synchronization: Existing webhooks are now automatically extended if the retentionHours setting is increased.
• Enterprise-Grade Rate Limiting:
  • Implemented LRU (Least Recently Used) eviction strategy for superior client protection under load.
  • Added strict IP validation for proxy headers (X-Forwarded-For/X-Real-IP) to prevent spoofing and malformed data propagation.
  • Added robust validation for limit, windowMs, and maxEntries parameters.
• Security & Privacy Hardening:
  • Enhanced Privacy (PII Masking): Integrated an IP masking helper to obfuscate sensitive client data in logs (e.g., 192.168.1.****).
  • XSS Protection: Implemented escapeHtml sanitization for all user-controlled output in the "Locked" auth screen and error pages.
  • Universal Auth UI: Created a unified landing page with hardened auth detection and zero-leak link generation.
• Enterprise Integration Suite:
  • Forwarding Security: Automatically strip sensitive headers (Authorization, Cookie, etc.) during real-time forwarding and replay.
  • Forwarding Controls: Added forwardHeaders toggle for granular control over header transmission.
  • Documentation Playbooks: Added new guides for Revenue Recovery, Low-Code Bridge, and Legacy Migration.
  • Example Saturation: Added 3+ comprehensive end-to-end input/output examples to satisfying Apify Quality Score requirements.

Improved (2.7.0)

• Auth Security: Eliminated all authKey leakage by removing automatic query parameter propagation in management links and API responses.
• Replay Accuracy: Prioritized nanoid id over timestamps in event lookup to eliminate collisions during high-concurrency replays.
• Architectural Reliability:
  • Background Pruning: Moved hit cleanup to a non-blocking background interval (60s).
  • Middleware Orchestration: Guaranteed immediate response delivery before racing background tasks against a 10s timeout.
• Comprehensive Verification: Expanded test suite to 85 tests (17 files), achieving full coverage of security, reliability, dynamic scaling, hot-reloading, and edge case scenarios with zero memory/timer leaks.
• Repository Health: Removed heavy binary assets and updated .gitignore for a leaner, faster repository.

[2.6.0] - 2025-12-27

Added (2.6.0)

• Management Rate Limiting: Implemented a memory-efficient rate limiter for /info, /logs, and /replay endpoints to prevent brute-force attacks on API keys.
• Sensitive Data Masking: Added opt-in masking for sensitive headers (Authorization, Cookie, etc.) in captured logs to enhance user privacy.
• Resource Offloading: Dataset schemas and processing have been optimized for better platform performance.
• Detailed Log Views: Added "Full Payloads" view to the Apify Dataset for easier inspection of headers and bodies in the console.

Improved (2.6.0)

• SSE Scalability: Refactored Server-Sent Events to use a high-performance global heartbeat mechanism, significantly reducing memory overhead per concurrent listener.
• Input Schema Quality: Added detailed tooltips, grouping, and prefill examples for all v2.0+ features.
• Documentation: Major README overhaul with new troubleshooting guides, professional usage examples, and performance metrics.

[2.5.0] - 2025-12-26

• Standby Mode Enabled: Formally added "usesStandbyMode": true to actor.json for superior performance and persistence.
• QA Success Logic: The Actor now yields an immediate "Server Ready" result to the dataset on startup. This ensures compliance with Apify's automated QA tests (which require a result within 5 minutes).
• Test & Exit: Added a hidden testAndExit input to allow automated health checks to complete and exit cleanly.
• Readiness Probes: Implemented explicit handling for Apify's x-apify-container-server-readiness-probe header in the root endpoint.

Fixed (2.5.0)

• Resolved "Under maintenance" flag by ensuring the Actor does not timeout during automated platform tests.
• Improved version consistency across all project manifest files.

[2.4.2] - 2025-12-22

Added (2.4.2)

• Stress Testing: Added a comprehensive stress test suite to verify the system's stability under high load.
• Documentation: Added missing Pricing, FAQ, Support, Security & Permissions, and Privacy sections to README.md.
• Schema Quality: Populated dataset_schema.json with concrete example values for all fields.
• Reliability: Implemented retry logic with exponential backoff (3 attempts) for both HTTP Forwarding and the /replay API.

Fixed (2.4.2)

• Stress Testing: Fixed a memory leak in the stress test suite.
• Stress Testing: Fixed a timeout issue in the stress test suite.

[2.4.1] - 2025-12-22

Fixed (2.4.1)

• ESM Compatibility: Fixed eventsource import in demo_cli.js to support latest named exports.
• Version Sync: Synchronized project version across all manifests.

[2.4.0] - 2025-12-22

Added (2.4.0)

• Comprehensive Test Suite: 15+ Automated tests covering unit, integration, and E2E scenarios.
• Testing Framework: Integrated Jest and Supertest with full ESM/VM support.
• Architectural Polish: Refactored main.js to decouple the Express app from the server listener for professional testability.

[2.3.1] - 2025-12-22

Fixed (2.3.1)

• Edge Case Hardening: Added input sanitization and hard-caps for /logs API.
• Safety: Added object validation for custom response headers to prevent runtime crashes.
• Maintenance: Minor documentation cleanup.

[2.3.0] - 2025-12-22

Added (2.3.0)

• v2.2 Comprehensive Robustness Update:
  • Global Express error handling to catch malformed bodies and unhandled exceptions.
  • Hardened state persistence with try-catch and validation logic.
  • Standardized timeouts (10s) for HTTP Forwarding and Replay APIs.
  • Improved error reporting for Replay API (distinguishes between timeouts and target rejections).
  • Resilient initialization to survive storage-layer transients.

[2.1.1] - 2025-12-22

Fixed (2.1.1)

• Hotfix: Added missing editor fields to input_schema.json to resolve Apify platform validation errors.

[2.1.0] - 2025-12-22

Added (2.1.0)

• v2.1 Custom Scripting: Allow users to provide JavaScript snippets for advanced data transformation before logging.
• Prepared submission for the $1M Challenge with a unified enterprise feature set.

[2.0.0] - 2025-12-21

Added (2.0.0)

• v2.0 Enterprise Features:
  • Security: CIDR IP Whitelisting and API Key/Bearer Auth support.
  • Mocking: Custom response bodies, status codes, and headers.
  • Simulation: Configurable response delay (latency simulation up to 10s).
  • Workflows: Real-time HTTP Request Forwarding (pipe webhooks to other APIs).
  • Replay API: Endpoint to resend captured events to any destination.
  • Validation: JSON Schema validation with professional error reporting.
• New "Enterprise" sections in input schema for easier configuration.

Improved (2.0.0)

• Middleware refactored into a high-performance pipeline.
• Upgraded dependencies: ajv for validation and ip-range-check for security.

[1.1.0] - 2025-12-21

Added (1.1.0)

• Narrated walkthrough video integrated into README for Quality Score boost.
• 5+ comprehensive FAQs added to documentation.
• CSV Output Format preview table in README.
• Example values added to Dataset Schema for improved platform documentation.
• Developer Support Guarantee (24h response time) added.

Improved (1.1.0)

• Input Schema sections and tooltips polished for better UX.
• [CRITICAL] Safety checks added to logging middleware to prevent data loss.
• GitHub repository synchronized with finalized production assets.

[1.0.0] - 2025-12-20

Added (1.0.0)

• Initial release of Webhook Debugger & Logger.
• Standby mode support for sub-10ms response times.
• Real-time event streaming via Server-Sent Events (SSE).
• Dynamic webhook URL generation (1-10 IDs per run).
• /logs API with advanced filtering (method, status, ID).
• Response status code override via ?__status=XXX.
• Robust body parsing for JSON, XML, and URL-encoded form data.
• Payload size limit (configurable, default 10MB).
• Detailed integration guides for Zapier and Make.
• Table view configuration for Apify Dataset.
• CSV/JSON export links in Output tab.
• PPE (Pay-per-Event) pricing support.