Skip to content

feat: added configuration variables to handle EC2 metadata service#2385

Merged
rtyler merged 4 commits intodelta-io:mainfrom
mightyshazam:imds-fix
Apr 12, 2024
Merged

feat: added configuration variables to handle EC2 metadata service#2385
rtyler merged 4 commits intodelta-io:mainfrom
mightyshazam:imds-fix

Conversation

@mightyshazam
Copy link
Copy Markdown
Contributor

Description

Rusoto and the AWS SDK have different behavior when interacting with the ec2 metadata service. Despite the lower default timeout in the AWS SDK, it still takes longer to abandon the IMDS credential provider. This PR add a setting AWS_EC2_METADATA_DISABLED to disable the provider. It can be provided as an environment variable or isolated to the s3 storage settings. Additionally, I added AWS_EC2_METADATA_TIMEOUT to represent the timeout for the imds credential provider in milliseconds with a default value of 100.
This does not completely eliminate the regression, but it provides opt-in/opt-out options to mitigate the issues in exceptional cases.

Related Issue(s)

Documentation

Rust SDK Environment Variables

@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Apr 4, 2024
@echai58
Copy link
Copy Markdown

echai58 commented Apr 9, 2024

Hi, bump on reviewing this 🙂 , this is blocking my ability to upgrade past 0.16.1 which has some important bug fixes/features.

@echai58
Copy link
Copy Markdown

echai58 commented Apr 11, 2024

@mightyshazam Hi, I tried running against a local build with this commit, and it seems like passing "AWS_EC2_METADATA_DISABLED": "true" via the storage_options still raises the failed to load region from IMDS err warning and slow reads, whereas setting the environment variable prevents this behavior.

@rtyler rtyler merged commit 64b3e54 into delta-io:main Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/rust Issues for the Rust crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Very slow s3 connection after 0.16.1

3 participants