Commit 7d82072
authored
fix(s3-deployment): optimize memory usage for large files (#34020)
### Issue # (if applicable)
Closes #34002.
### Reason for this change
The original fix for issue #22661 (PR #33698) introduced a regression where the S3 deployment Lambda would read entire files into memory to check if they're JSON. This approach works fine for small files but causes Lambda timeouts and memory issues with large files (10MB+). This is particularly problematic for customers deploying large assets to S3 buckets.
### Description of changes
The S3 deployment Lambda handler was reading entire files into memory to check if they're JSON, causing timeouts and memory issues with large files (10MB+).
This change optimizes the S3 deployment Lambda handler to process files more efficiently by:
1. Adding an early return when there are no markers to replace
2. Processing all files line by line, which is much more memory-efficient than loading the full JSON in memory
3. Adding an optional `escape` parameter to the `Source.jsonData` method in order to control JSON escaping
4. Using the `jsonEscape` flag in `MarkersConfig` to control when special JSON escaping is needed
These changes ensure that:
- Files without markers are processed instantly
- Files with markers are processed line by line, minimizing memory usage
- Special JSON escaping is only applied when explicitly requested
The implementation is backward compatible with the experience before the PR #33698 was merged, as it maintains the existing behavior by default but provides an opt-in mechanism for JSON escaping when needed. The opt-in mechanism is required for users who were benefitting from the escaping mechanism introduced by the PR #33698.
### Describe any new or updated permissions being added
No new or updated IAM permissions are required for this change.
### Description of how you validated changes
- Created an integration test (`integ.bucket-deployment-large-file.ts`) that reproduces the issue with large files
- Implemented local testing to verify the fix with both small and large files
- Added memory limit assertions to ensure memory usage stays within acceptable bounds
- Conducted performance testing with isolated test runs to measure memory usage across various file types and sizes
The integration test specifically validates that large files (10MB+) can be successfully deployed without memory issues, ensuring the fix works in real-world scenarios.
### Checklist
- [ ] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)
----
*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*1 parent ccd8de7 commit 7d82072
File tree
600 files changed
+487184
-84489
lines changed- packages
- @aws-cdk-testing/framework-integ/test
- aws-appconfig/test/integ.configuration.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- aws-codepipeline-actions/test
- integ.cross-account-pipeline-cfn-action.js.snapshot
- asset.9a1fcb4a7ecba81ad70e9d3fb241f6794497da945dae5f25924e4dd002b65f2d
- integ.cross-account-pipeline-sfn-action.js.snapshot
- asset.9a1fcb4a7ecba81ad70e9d3fb241f6794497da945dae5f25924e4dd002b65f2d
- integ.pipeline-commands.js.snapshot
- integ.pipeline-ecr-image-scan-action.js.snapshot
- asset.97484721f29e34bf38d7a459804dd2d2a8dea6f8c27d7531e215bf4274fbc895.bundle
- integ.pipeline-elastic-beanstalk-deploy.js.snapshot
- aws-dynamodb/test/integ.import-source.js.snapshot
- aws-ecs/test/ec2/integ.environment-file.js.snapshot
- aws-lambda-nodejs/test
- integ.bundling-user.js.snapshot
- integ.dependencies-pnpm.js.snapshot
- asset.78d43382e519a4bce684a187408c96399e462c8a1ac7dfb615458896dd6918ed
- .pnpm-store/v3/files
- 00
- 03
- 04
- 05
- 09
- 0d
- 0e
- 0f
- 10
- 11
- 13
- 14
- 15
- 17
- 18
- 19
- 1b
- 24
- 27
- 2e
- 2f
- 31
- 33
- 34
- 36
- 39
- 3a
- 3b
- 3c
- 3d
- 40
- 41
- 43
- 46
- 4a
- 4c
- 4f
- 51
- 56
- 57
- 5b
- 5c
- 5d
- 5f
- 62
- 63
- 64
- 67
- 69
- 6a
- 6c
- 73
- 76
- 78
- 7b
- 7c
- 7d
- 7e
- 80
- 81
- 82
- 83
- 86
- 87
- 8e
- 93
- 97
- 99
- 9a
- 9d
- a0
- a1
- a5
- a6
- a8
- a9
- ab
- ac
- ae
- af
- b0
- b1
- b2
- b5
- b6
- b7
- b9
- ba
- bc
- bd
- c1
- c2
- c5
- c6
- c8
- c9
- ca
- cb
- cc
- cd
- d1
- d3
- d4
- d8
- d9
- dd
- de
- e2
- e3
- e4
- e6
- e7
- eb
- f0
- f1
- f8
- fb
- node_modules
- .pnpm
- asynckit
- lib
- axios
- dist
- browser
- esm
- node
- lib
- adapters
- cancel
- core
- defaults
- env
- classes
- helpers
- platform
- browser
- classes
- common
- node
- classes
- combined-stream
- lib
- delayed-stream
- lib
- follow-redirects
- form-data
- lib
- mime-db
- mime-types
- proxy-from-env
- asset.97484721f29e34bf38d7a459804dd2d2a8dea6f8c27d7531e215bf4274fbc895.bundle
- aws-lambda/test
- integ.bundling.docker-opts.js.snapshot
- integ.bundling.js.snapshot
- asset.c504585dae89ce8e394d7744ccae054318e7d1aff14afb545438d173d51dc97d.bundle
- aws-s3-assets/test
- integ.assets.bundling.docker-opts.js.snapshot
- integ.assets.file-bundling.lit.js.snapshot
- aws-s3-deployment/test
- integ.bucket-deployment-big-response.js.snapshot
- integ.bucket-deployment-cloudfront.js.snapshot
- asset.4fe0aba5e672b596d0f72505a9eec502f98d46906bb30fae2511fbdc1df4956f
- integ.bucket-deployment-data.js.snapshot
- asset.86adb64db7c5ffae02bd6d739a8750b0539cc6e022932e141f141904aa5eda1a
- asset.d389e273abc3a151d7ee50a64dfd8c1d10d6e72117ca68f7f241f4ba2672484c.bundle
- integ.bucket-deployment-deployed-bucket.js.snapshot
- asset.4fe0aba5e672b596d0f72505a9eec502f98d46906bb30fae2511fbdc1df4956f
- integ.bucket-deployment-large-file.js.snapshot
- asset.0465042b3f7892b88ecfeaad39e4ba0a16d2c51b36c2152d82b0d9c5098563cd
- asset.1175d01d4ddecbd7be39bdc306dabe248b6ada0281089ae680473c5ca7e13e67/my-json
- asset.44e9c4d7a5d3fd2d677e1a7e416b2b56f6b0104bd5eff9cac5557b4c65a9dc61
- asset.97484721f29e34bf38d7a459804dd2d2a8dea6f8c27d7531e215bf4274fbc895.bundle
- asset.9a1fcb4a7ecba81ad70e9d3fb241f6794497da945dae5f25924e4dd002b65f2d
- asset.c71e65c14767615eb12046a47812c3f60d82c33e90ec926c2ab13dca2df294f6/my-json
- asset.cbc7dae5477d6aa98463e36140d5833273cc3caca06c731d848799f30237fefe
- asset.dd350b0d251cf3c4abfd488c198fbea912a4964f401a3e01c183ca2f04072e76
- integ.bucket-deployment-signcontent.js.snapshot
- asset.4fe0aba5e672b596d0f72505a9eec502f98d46906bb30fae2511fbdc1df4956f
- integ.bucket-deployment.js.snapshot
- asset.4fe0aba5e672b596d0f72505a9eec502f98d46906bb30fae2511fbdc1df4956f
- aws-servicecatalog/test
- integ.nested-stack-in-product-stack.js.snapshot
- asset.9a1fcb4a7ecba81ad70e9d3fb241f6794497da945dae5f25924e4dd002b65f2d
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.product.encrypted.asset.js.snapshot
- asset.9a1fcb4a7ecba81ad70e9d3fb241f6794497da945dae5f25924e4dd002b65f2d
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.product.js.snapshot
- integ.two-products.js.snapshot
- custom-resources/test/custom-resource-config
- integ.custom-resource-config-lambda-python-runtime.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.custom-resource-config-lambda-runtime.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.custom-resource-config-logGroup.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.custom-resource-config-logRetention.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.custom-resource-config-removalPolicy.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- integ.custom-resource-config-undefined-log.js.snapshot
- asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
- pipelines/test/integ.cross-account-pipeline-action.js.snapshot
- asset.4fe0aba5e672b596d0f72505a9eec502f98d46906bb30fae2511fbdc1df4956f
- asset.9a1fcb4a7ecba81ad70e9d3fb241f6794497da945dae5f25924e4dd002b65f2d
- @aws-cdk/custom-resource-handlers
- lib/aws-s3-deployment/bucket-deployment-handler
- test/aws-s3-deployment/bucket-deployment-handler
- aws-cdk-lib
- aws-s3-deployment
- lib
- rosetta/aws_s3_deployment
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
600 files changed
+487184
-84489
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
This file was deleted.
This file was deleted.
0 commit comments