Experimental flag to boost 'package' command performance

[eyalroth]

Which issue(s) does this change fix?

#4053

Why is this change necessary?

sam package (and deploy) can take an extremely long time.

This happens because the command zips every lambda resource in a template, even if all the lambdas point to the same local path. For instance, with NodeJS lambdas, there are many files involved due to the nature of npm/node, and that puts a serious dent on the disk (especially so in CI environments where SSD is not guarenteed).

How does it address the issue?

Normally, sam build will create a separate directory for each lambda resource in the template. However, there is an experimental flag -- SAM_CLI_BETA_BUILD_PERFORMANCE -- which creates only one shared directory for all the lambdas. This boosts sam build performance tremendously, and it works because in most cases, all lambdas in a project share the same code (just a different handler endpoint).

This PR adds a new flag -- SAM_CLI_BETA_PACKAGE_PERFORMANCE -- that is aimed to work in conjunction with the build performance flag.

All that it does is add a cache that is currently only used by ResourceZip + S3Uploader. The uploader checks if the local path was previously uploaded, and if so, it simply returns the cached upload url.

What side effects does this change have?

Should have no effect as the changes are only enabled when using the new flag.

Mandatory Checklist

PRs will only be reviewed after checklist is complete

• Add input/output type hints to new functions/methods
• Write design document if needed (Do I need to write a design document?)
• Write/update unit tests
• Write/update integration tests
• Write/update functional tests if needed
• make pr passes
• make update-reproducible-reqs if dependencies were changed
• Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[roger-zhangg]

Hi @eyalroth thanks for the PR, I see currently there are few errors in CI, do you mind helping to fix them? The implementation looks well-designed. Before merging, please review the testing practices in our development guide and add appropriate test coverage for these changes. Once the tests are in place, this will be ready to merge

[eyalroth]

Thank you @roger-zhangg!

Turns out that when I initially ran make pr, it actually failed but I didn't notice.

It failed regardless of my changes, but because I was running with Python 3.12 (default installed on my MacOS). It took me some time to realize I need to install Python 3.9 and setup the virtual env with it. Only then the tests actually worked. It might be worth updating the development guide with these requirements in mind.

I fixed/added relevant tests, amended the original commit, rebased on latest develop and force-pushed.

make pr passes except for test_import_module_proxy.py test, which I think is not related to my changes and would fail on my machine regardless.

[roger-zhangg]

LGTM, thank you for your contribution!