Cloud Connect - API Gateway Traffic Forwarder

This project is a robust, serverless system designed to observe all HTTP traffic for a given AWS API Gateway (REST API) and forward a structured representation of each request and response to an arbitrary HTTP destination.

The solution dynamically attaches to an existing API Gateway, making it a flexible tool for traffic analysis, logging, or webhook-style integrations.

Architecture

The system is architected around a decoupled, asynchronous pattern using SQS to ensure resilience and handle traffic spikes without impacting the performance of the primary API.

Data Flow

Core Components

• Proxy Lambda: This is the heart of the capture mechanism. The existing API Gateway's integrations are dynamically updated to point to this Lambda. It receives the request, forwards it to the original backend URL, captures the response, constructs the final JSON payload, and sends it to an SQS queue.
• SQS Queue: Acts as a durable, elastic buffer between capturing and forwarding. This decoupling prevents data loss and ensures that slowdowns or failures at the final destination do not impact the primary API's availability.
• S3 Bucket & SQS Extended Client Library: To handle HTTP requests/responses that exceed the SQS message size limit of 256KB, the Lambdas use the extended client library. Large payloads are automatically stored in S3, and a reference is sent through SQS.
• Forwarder Lambda: This function is triggered by new messages in the SQS queue. It retrieves the full payload (from S3 if necessary), and makes an HTTP POST request to the configured destination URL.
• Dead-Letter Queue (DLQ): If the Forwarder Lambda fails to process a message after several retries (e.g., due to a malformed payload or a persistent error from the destination), the message is moved to a DLQ for manual inspection.
• CloudWatch Alarms: Alarms are configured to monitor key operational metrics. An alert is sent via SNS (email) if:
  • Any message enters the DLQ (Critical).
  • Messages in the main queue become too old (Warning).
  • The Proxy or Forwarder Lambdas experience errors (Warning).

File Structure

The repository is organized to clearly separate the core solution, example code, and utility scripts.

.
├── traffic-forwarder/      # The core traffic forwarding solution
│   ├── lambdas/            # Source code for Lambda functions
│   │   ├── proxy_lambda/
│   │   └── forwarder_lambda/
│   └── terraform/          # Terraform IaC for the core solution
│
├── example-apigw/          # Terraform IaC for a standalone example API
│
├── scripts/                # Deployment and utility scripts
│   ├── deploy.sh
│   └── manage_example_api.sh
│
└── README.md

Setup and Prerequisites

Required Tools

Ensure the following tools are installed and configured on your system:

• AWS CLI: Authenticated to your AWS account.
• Terraform (>= 1.0): For deploying the infrastructure.
• jq: A command-line JSON processor used by the deployment script.
• Python 3 & pip: Required for packaging the Lambda function dependencies locally during deployment.

AWS Credentials

Configure your AWS credentials. The simplest way is to run aws configure and follow the prompts.

Deployment Instructions

The deployment is a two-step process: first, deploy the example API that we will monitor, and second, deploy the traffic forwarder solution to attach to it.

Step 1: Deploy the Example API Gateway

This creates a sample REST API with a /users endpoint that proxies to https://jsonplaceholder.typicode.com. The deployment is managed by an interactive script. You will need the API Gateway ID which is an output of the script as the input to step 2.

# Navigate to the scripts directory
cd scripts

# Run the management script for the example API
./manage_example_api.sh deploy

Step 2: Deploy the Traffic Forwarder Solution

This script will inspect the API Gateway you just created, discover its routes, and deploy the Lambda functions and SQS queue to capture its traffic.

# Navigate to the scripts directory
cd ../scripts

# Run the deployment script
./deploy.sh

The script will prompt you for the following information:

1. API Gateway ID: Enter the ID from the previous step.
2. AWS Region: The region to deploy to.
3. Project Name: A unique name for your deployment resources.
4. Destination URL: The target URL where traffic payloads will be sent.
5. Alarm Email: The email address to receive CloudWatch alarm notifications.

On subsequent runs, the script will remember your previous inputs and offer them as defaults.

Destroying the Infrastructure

To clean up all resources, you can run the deployment script with the --destroy flag.

# Destroy the traffic forwarder solution
./scripts/deploy.sh --destroy

# Destroy the example API
cd ../example-apigw
terraform destroy --auto-approve

Example Usage and Expected Behavior

1. After deploying both stacks, find the invoke_url for the example API by running terraform output in the example-apigw directory.
2. Make a request to one of its endpoints:

   curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"name": "morpheus", "job": "leader"}' \
     "YOUR_API_GATEWAY_INVOKE_URL/api/v1/users"

3. Expected Behavior:
   • You will receive a response from the upstream API.
   • Simultaneously, the Proxy Lambda will capture the request and response, and the Forwarder Lambda will send a JSON payload to your configured destination URL.

The payload sent to your destination will look like this:

{
  "from": "your-api-gateway-id.execute-api.eu-west-1.amazonaws.com",
  "endpoint": "/api/v1/users",
  "method": "POST",
  "body": {
    "name": "morpheus",
    "job": "leader"
  },
  "request_headers": {
    "Content-Type": "application/json",
    "Authorization": "[REDACTED]"
  },
  "response": {
    "name": "morpheus",
    "job": "leader",
    "id": "573",
    "createdAt": "2024-01-01T12:00:00.000Z"
  },
  "response_headers": {
    "Content-Type": "application/json; charset=utf-8"
  },
  "status": 201
}

Notes on Assumptions and Trade-offs

This solution was built with the following assumptions and design trade-offs in mind:

• Target API is a REST API: The discovery script is designed for AWS API Gateway v1 (REST API). It can be extended to support v2 (HTTP API), but the resource discovery logic would need to be adapted.
• HTTP/S Integrations Only: The system is designed to intercept traffic going to public HTTP/S backends. It correctly preserves the original backend URL for each path. It does not currently support intercepting traffic for other integration types like AWS_PROXY to another Lambda or mock integrations.
• Full Body Capture: The requirement is to capture the entire request and response body. For very large payloads (e.g., file uploads), this could have cost implications for S3 storage and data transfer.
• Asynchronous Forwarding: The choice of SQS means that the destination receives the traffic payload with a slight delay (typically seconds). This is a trade-off for resilience. A synchronous, in-line approach would have lower latency but would be less resilient to destination failures.
• Security and Data Redaction: The Proxy Lambda automatically redacts common sensitive headers like Authorization and Cookie. This is a critical security measure but is not exhaustive. The list of redacted headers should be reviewed and customized for a specific use case.
• Deployment via Local Machine: The deploy.sh script relies on local tools (AWS CLI, Terraform). In a production environment, this entire process would ideally be managed by a CI/CD pipeline (e.g., GitHub Actions, AWS CodePipeline).
• Error Handling at Destination: The system guarantees "at-least-once" delivery to the SQS queue. The Forwarder Lambda will retry on failure. If the destination endpoint does not respond with a 2xx status code, the Lambda will error, and after retries, the message will go to the DLQ. The destination should be idempotent to handle potential duplicate deliveries.