Nginx Cookie-Based Routing System

This project implements a cookie-based routing system using Nginx and Express.js, deployed on Kubernetes (Minikube). The system routes traffic to different Express server instances based on a tag_cookie value.

📋 Requirements

Docker (for building images) # v28.0.4+
Minikube (for local Kubernetes cluster) # v1.36.0+
kubectl (for managing Kubernetes resources) # v1.33.2+
Node.js 22+ (for local development) # v22.15.0+
curl (for testing) # v8.7.1+

Architecture Overview

Browser → Minikube Ingress → Nginx Service → Express Services
                                  ↓
                            Cookie-based routing:
                            - No cookie → express-server-main
                            - tag_cookie=a → express-server-a
                            - tag_cookie=b → express-server-b
                            - tag_cookie=c → express-server-c

Install Prerequisites (macOS)

# Install Docker Desktop
# Download from: https://www.docker.com/products/docker-desktop

# Install minikube
brew install minikube

# Install kubectl
brew install kubectl

# Verify installations
docker --version
minikube version
kubectl version --client

Quick Start

1. Start Minikube

# Start minikube cluster
minikube start

# Enable ingress addon
minikube addons enable ingress

# Verify cluster is running
kubectl cluster-info

2. Prepare Dependencies

Important: Before running the deployment script, ensure dependencies are installed:

# Navigate to project directory
cd /path/to/nginx-cookie-routing

# Install Express dependencies to generate package-lock.json
cd express
npm install
cd ..

# Set Docker environment to use minikube's Docker daemon
eval $(minikube docker-env)

3. Deploy the System

# Clone/navigate to project directory
cd /path/to/nginx-cookie-routing

# Run the deployment script
./deploy.sh

Note: If you encounter Docker build errors related to package-lock.json, run:

cd express && npm install && cd .. && ./deploy.sh

The deploy script will:

Build Docker images for Express and Nginx
Deploy all Kubernetes resources (namespaces, RBAC, main services)
Optionally create example ephemeral deployments (a, b, c)
Wait for deployments to be ready
Show testing instructions

4. Access the Application

Choose one of these methods:

Option A: Port Forward (Recommended for testing)

# Forward local port 8080 to nginx service
kubectl port-forward service/nginx-service 8080:80

Option B: Minikube Tunnel

# In a separate terminal, run:
minikube tunnel

# Then access via ingress
# Add to /etc/hosts: $(minikube ip) nginx-cookie-routing.local
echo "$(minikube ip) nginx-cookie-routing.local" | sudo tee -a /etc/hosts

5. Test Cookie Routing

Manually

# Test default routing (no cookie) - should go to 'main'
curl http://localhost:8080/

# Test health endpoints
curl http://localhost:8080/healthcheck
curl http://localhost:8080/version

# Test tag-specific routing (requires ephemeral deployments)
# First create the ephemeral instances:
./ephemeral.sh --tag=a
./ephemeral.sh --tag=b  
./ephemeral.sh --tag=c

# Then test them:
curl -v -H "Cookie: tag_cookie=a" http://localhost:8080/
curl -v -H "Cookie: tag_cookie=b" http://localhost:8080/
curl -v -H "Cookie: tag_cookie=c" http://localhost:8080/

Full suite

./test.sh

Ephemeral Deployments

After completing the initial setup with ./deploy.sh, you can create temporary deployments for testing specific versions, commits, or branches using the ephemeral.sh script.

1. Usage

The ephemeral.sh script allows you to deploy and manage temporary Express server instances:

# Deploy an ephemeral tag (e.g., git commit hash) and test it
./ephemeral.sh --tag=1234abcd
curl -H "Cookie: tag_cookie=1234abcd" http://localhost:8080/

# Deploy a branch, and test it
./ephemeral.sh --tag=feature-login
curl -H "Cookie: tag_cookie=feature-login" http://localhost:8080/

# Delete an ephemeral deployment
./ephemeral.sh --tag=1234abcd --delete

# Get help
./ephemeral.sh --help

2. What Ephemeral Deployments Create

Each ephemeral deployment creates:

Service: express-server-<tag> on port 3000
Deployment: express-server-<tag> with 1 replica
Labels: type=ephemeral for easy identification
Environment: EPHEMERAL=true to distinguish from permanent deployments

3. Routing to Ephemeral Instances

The nginx configuration has been updated to support dynamic routing:

Ephemeral instances are routed directly to service names: express-server-<tag>:3000
Fallback behavior: If an ephemeral service doesn't exist, requests are directed to an error page

4. Dynamic Routing Behavior

Note: All tag-specific routing requires ephemeral deployments to be created first using ./ephemeral.sh --tag=<tag>.

# These use dynamic routing (ephemeral deployments)
curl -H "Cookie: tag_cookie=a" http://localhost:8080/      # → express-server-a
curl -H "Cookie: tag_cookie=b" http://localhost:8080/      # → express-server-b  
curl -H "Cookie: tag_cookie=c" http://localhost:8080/      # → express-server-c
curl -H "Cookie: tag_cookie=abc123" http://localhost:8080/ # → express-server-abc123:3000
curl -H "Cookie: tag_cookie=pr-456" http://localhost:8080/ # → express-server-pr-456:3000

# No cookie or invalid service → fallback to error page
curl http://localhost:8080/                                    # → express-server-main

5. Monitoring Ephemeral Deployments

# List all ephemeral deployments
kubectl get deployments -l type=ephemeral

# Check logs for a specific ephemeral tag
kubectl logs -l tag=abc123 -f

# Monitor pod status
kubectl get pods -l type=ephemeral

# Check the Docker image being used by the pod (table format with image)
kubectl get pods -l type=ephemeral -o custom-columns="NAME:.metadata.name,STATUS:.status.phase,IMAGE:.spec.containers[*].image"

# Get detailed pod information including image
kubectl describe pod -l type=ephemeral

# Check service endpoints
kubectl get endpoints | grep express-server

6. Best Practices

Tag naming: Use alphanumeric characters, hyphens, and underscores only
Cleanup: Always delete ephemeral deployments when done testing
Resource limits: Ephemeral deployments have the same resource limits as permanent ones
Testing: Verify the ephemeral service is responding before sharing with others
Monitoring: Check logs if routing doesn't work as expected

Integration with CI/CD

Ephemeral deployments are perfect for:

Pull request testing: Deploy feature branch code for review
Commit verification: Test specific commits before merging
Feature demonstrations: Show features to stakeholders
Load testing: Test new instances under load

GitHub Actions integration example:

# In your CI pipeline
COMMIT_HASH=$(git rev-parse --short HEAD)
./ephemeral.sh --tag=$COMMIT_HASH

# Run tests against the ephemeral instance
curl -H "Cookie: tag_cookie=$COMMIT_HASH" http://localhost:8080/healthcheck

# Clean up after tests
./ephemeral.sh --tag=$COMMIT_HASH --delete

Troubleshooting

This section covers common issues encountered during development and their solutions.

Ephemeral Deployment Issues

Service not found errors

# Check if the service was created
kubectl get service express-server-<tag>

# Check if the deployment is ready
kubectl get deployment express-server-<tag>

# View nginx logs for routing errors
kubectl logs -l app=nginx-router

Fallback to main service

If nginx can't reach an ephemeral service, it falls back to the main service. Check the response headers:

curl -v -H "Cookie: tag_cookie=<tag>" http://localhost:8080/
# Look for X-Routed-To header to see which service was used

Nginx Configuration Issues

Problem: Multiple `if` statements causing routing failures

Symptom: Nginx fails to route properly or gives unexpected responses Cause: Multiple if statements with proxy_pass don't work well together in nginx Solution: Use map directives instead of if statements

# ❌ Don't do this - multiple if statements
if ($cookie_tag_cookie = "a") {
    proxy_pass http://express_a;
}
if ($cookie_tag_cookie = "b") {
    proxy_pass http://express_b;
}

# ✅ Do this - use map directive
map $cookie_tag_cookie $final_backend {
    default "http://express_main";
    "a" "http://express_a";
    "b" "http://express_b";
}
location / {
    proxy_pass $final_backend;
}

Problem: "no resolver defined" error for dynamic routing

Symptom: 502 Bad Gateway with log message "no resolver defined to resolve express-server-abc123" Cause: Nginx needs a DNS resolver when using variables in proxy_pass Solution: Add resolver directive pointing to Kubernetes DNS

server {
    # DNS resolver for dynamic service discovery
    resolver 10.96.0.10 valid=10s;
    
    location / {
        proxy_pass $dynamic_backend;  # This now works with variables
    }
}

Problem: Service name resolution failures

Symptom: 503 Service Unavailable with "could not be resolved (3: Host not found)" Cause: Simple service names don't resolve from nginx pod Investigation:

# Test DNS resolution from nginx pod
kubectl exec deployment/nginx-router -- nslookup express-server-abc123
# Result: NXDOMAIN - doesn't resolve

kubectl exec deployment/nginx-router -- nslookup express-server-abc123.default.svc.cluster.local  
# Result: Success - resolves to service IP

Solution: Use fully qualified domain names (FQDN) for ephemeral services

# ❌ Simple name - doesn't resolve
proxy_pass http://express-server-$cookie_tag_cookie:3000;

# ✅ FQDN - resolves correctly  
proxy_pass http://express-server-$cookie_tag_cookie.default.svc.cluster.local:3000;

Cookie Naming Issues

Problem: Cookie name inconsistency

Symptom: Routing doesn't work, cookies aren't detected Cause: Mixed use of tag_cookie (underscore) consistently Investigation:

# Check what nginx sees
curl -v -H "Cookie: tag_cookie=abc123" http://localhost:8080/
# Nginx variable: $cookie_tag_cookie (underscore, not hyphen)

Solution: Always use underscore in cookie names to match nginx variable naming

# ✅ Correct - matches $cookie_tag_cookie
curl -H "Cookie: tag_cookie=abc123" http://localhost:8080/

# ❌ Wrong - nginx won't detect this  
curl -H "Cookie: tag_cookie=abc123" http://localhost:8080/

Docker and Kubernetes Issues

Problem: nginx deployment not picking up configuration changes

Symptom: Changes to nginx.conf don't take effect Cause: Docker image not rebuilt or pod not restarted Solution: Always rebuild image and restart deployment

# Rebuild nginx image
eval $(minikube docker-env)
docker build --no-cache -t nginx-cookie-router:latest nginx/

# Restart nginx deployment
kubectl rollout restart deployment/nginx-router

# Wait for restart to complete
kubectl wait --for=condition=available --timeout=60s deployment/nginx-router

Problem: Port forwarding keeps stopping

Symptom: curl: (7) Failed to connect to localhost port 8080 Cause: Port forwarding process dies during testing Solution: Restart port forwarding as needed

# Check if port forwarding is running
ps aux | grep "port-forward" | grep -v grep

# Restart port forwarding
kubectl port-forward service/nginx-service 8080:80 --address=0.0.0.0 > /dev/null 2>&1 &

Problem: Service connectivity issues from nginx

Symptom: Nginx can't reach ephemeral services Investigation:

# Test connectivity from nginx pod
kubectl exec -it deployment/nginx-router -- wget -qO- http://express-server-abc123.default.svc.cluster.local:3000/

Solution: Ensure service exists and is running

# Check service exists
kubectl get service express-server-abc123

# Check pod is running
kubectl get pods -l tag=abc123

# Check service endpoints
kubectl get endpoints express-server-abc123

Debugging Techniques

Essential debugging commands

# Check nginx logs for routing errors
kubectl logs deployment/nginx-router --tail=10

# Test service connectivity from nginx
kubectl exec deployment/nginx-router -- wget -qO- http://service-name:3000/

# Check DNS resolution from nginx
kubectl exec deployment/nginx-router -- nslookup service-name.default.svc.cluster.local

# View response headers for routing debug info
curl -I -H "Cookie: tag_cookie=abc123" http://localhost:8080/

Debug headers interpretation

# Example response headers
X-Routed-To: express-server-abc123     # Which service nginx routed to
X-Tag-Cookie: abc123               # Cookie value nginx detected  
X-Routing-Type: dynamic                # Routing method (upstream vs dynamic)

Performance Considerations

Connection header importance

The nginx configuration includes this critical setting:

proxy_set_header Connection "";

Why this matters:

Prevents HTTP 426 (Upgrade Required) errors
Enables proper HTTP/1.1 keep-alive connections
Avoids forwarding problematic client Connection headers
Essential for Express.js/Node.js backends

Resolver caching

resolver 10.96.0.10 valid=10s;

Caches DNS lookups for 10 seconds
Reduces load on Kubernetes DNS
Balances between performance and service discovery freshness

Common Error Patterns

Error Code	Log Message	Likely Cause	Solution
502	"no resolver defined"	Missing resolver directive	Add `resolver 10.96.0.10;`
503	"could not be resolved"	Service doesn't exist or wrong name	Check service exists, use FQDN
502	"upstream" errors	Predefined service down	Check permanent deployments
Connection refused	Port forwarding issue	Restart `kubectl port-forward`

Best Practices Learned

Always use FQDN for dynamic services - Simple names don't resolve reliably
Use map directives, not if statements - More reliable routing logic
Include DNS resolver for dynamic routing - Required for variable-based proxy_pass
Consistent cookie naming - Use underscores to match nginx variables
Rebuild and restart after config changes - Don't assume changes are picked up
Add debug headers - Essential for troubleshooting routing issues
Test connectivity from nginx pod - Direct way to verify service accessibility

Name		Name	Last commit message	Last commit date
Latest commit History 20 Commits
.github/workflows		.github/workflows
express		express
k8s		k8s
nginx		nginx
.gitignore		.gitignore
FLOW.md		FLOW.md
PLAN_1.md		PLAN_1.md
PLAN_2.md		PLAN_2.md
PLAN_3.md		PLAN_3.md
PLAN_4.md		PLAN_4.md
PLAN_5.md		PLAN_5.md
README.md		README.md
cleanup.sh		cleanup.sh
deploy.sh		deploy.sh
ephemeral.sh		ephemeral.sh
test.sh		test.sh

Folders and files

Latest commit

History

Repository files navigation

Nginx Cookie-Based Routing System

📋 Requirements

Architecture Overview

Install Prerequisites (macOS)

Quick Start

1. Start Minikube

2. Prepare Dependencies

3. Deploy the System

4. Access the Application

5. Test Cookie Routing

Ephemeral Deployments

1. Usage

2. What Ephemeral Deployments Create

3. Routing to Ephemeral Instances

4. Dynamic Routing Behavior

5. Monitoring Ephemeral Deployments

6. Best Practices

Integration with CI/CD

Troubleshooting

Ephemeral Deployment Issues

Service not found errors

Fallback to main service

Nginx Configuration Issues

Problem: Multiple if statements causing routing failures

Problem: "no resolver defined" error for dynamic routing

Problem: Service name resolution failures

Cookie Naming Issues

Problem: Cookie name inconsistency

Docker and Kubernetes Issues

Problem: nginx deployment not picking up configuration changes

Problem: Port forwarding keeps stopping

Problem: Service connectivity issues from nginx

Debugging Techniques

Essential debugging commands

Debug headers interpretation

Performance Considerations

Connection header importance

Resolver caching

Common Error Patterns

Best Practices Learned

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Problem: Multiple `if` statements causing routing failures

Packages