(@aws-cdk/aws-eks-v2-alpha): kubectl handler masks error messages with "Response object is too long" when kubectl output exceeds 4KB

[derlopes-amzn]

Describe the bug

The CDK kubectl handler Lambda (lib/kubectl-handler/apply/init.py) passes raw kubectl error output as the exception message without truncation. When kubectl apply fails with a validation error, Kubernetes includes the entire object being validated in its error output. This often exceeds CloudFormation's 4KB custom resource response limit, causing CloudFormation to replace the actual error with the unhelpful message: Response object is too long.

The real error is only visible in the Lambda's CloudWatch logs.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

When kubectl apply fails, the actual error message (e.g., strict decoding error: unknown field "spec.terminationGracePeriod") should be visible in the CloudFormation stack events, not hidden behind Response object is too long.

Current Behavior

CloudFormation shows only Response object is too long. because the kubectl handler raises the full kubectl output (which includes the entire K8s object JSON) as the exception. The CloudFormation custom resource framework serializes this into the response Reason field, exceeding the 4KB response payload limit.

Error in CloudWatch logs (truncated for readability):

Exception: b'The request is invalid: patch: Invalid value: "{...entire object JSON...}": strict decoding error: unknown field "spec.terminationGracePeriod"'

Reproduction Steps

import * as eks from '@aws-cdk/aws-eks-v2-alpha';

// Given an existing EKS cluster
const cluster = new eks.Cluster(this, 'Cluster', { ... });

// Apply a manifest with an intentionally invalid field
cluster.addManifest('test', {
  apiVersion: 'karpenter.sh/v1',
  kind: 'NodePool',
  metadata: { name: 'test' },
  spec: {
    // This field doesn't exist at spec level (should be spec.template.spec)
    terminationGracePeriod: '48h',
    template: {
      spec: {
        nodeClassRef: { group: 'eks.amazonaws.com', kind: 'NodeClass', name: 'default' },
        requirements: [{ key: 'karpenter.sh/capacity-type', operator: 'In', values: ['on-demand'] }],
      },
    },
    limits: { cpu: '100' },
    disruption: { consolidationPolicy: 'WhenEmpty', consolidateAfter: '1h' },
  },
});

Deploy, CloudFormation reports Response object is too long instead of the schema validation error.

Possible Solution

Truncate the error message in lib/kubectl-handler/apply/init.py line 89:

# Current
raise Exception(output)

# Fix
MAX_ERROR_LENGTH = 2048  # Leave room for other response fields within 4KB limit
error_msg = output.decode('utf-8', errors='replace')[:MAX_ERROR_LENGTH]
raise Exception(error_msg)

The same issue exists in line 93 (raise Exception(f'Operation failed after {maxAttempts} attempts: {output}')).

Additional Information/Context

The patch/__init__.py handler has a similar pattern and would benefit from the same fix.

AWS CDK Library version (aws-cdk-lib)

@aws-cdk/aws-eks-v2-alpha

AWS CDK CLI version

2.x

Node.js Version

18.x

OS

Amazon Linux 2

Language

TypeScript

Language Version

No response

Other information

The 4KB limit is a hard constraint on CloudFormation custom resource responses. Kubernetes validation errors are particularly verbose because they include the full object being validated in the error message, making this a common issue for any non-trivial manifest.

[ashoknarayan]

I'd like to work on this issue. The proposed truncation approach makes sense, and I can implement it along with tests for large error outputs. Please let me know if there are any specific considerations before I start.

[pahud]

Hi @derlopes-amzn,

┌──────────────────────┐     ┌──────────────────────┐     ┌──────────────────────────┐
│ kubectl fails with   │ ──> │ Handler raises raw    │ ──> │ CFN response > 4KB       │
│ verbose K8s error    │     │ output (no truncation)│     │ "Response object is too  │
│                      │     │                       │     │  long"                   │
└──────────────────────┘     └──────────────────────┘     └──────────────────────────┘

Thanks for the detailed report. This is a valid bug — the kubectl() helper raises the raw subprocess output without truncation, which exceeds CloudFormation's 4KB custom resource response limit when Kubernetes returns verbose validation errors.

In my view, the fix is straightforward: decode and truncate the error output to ~2048 chars before raising, applied consistently across the apply, patch, and helm handlers in both aws-eks and aws-eks-v2.

I see PR #37384 is already in progress to address this.

Labeling as p2 and bug.