CredentialUnavailableError raised from `ImdsCredential` with no real connection issue

[bonsairobo]

• Package Name: azure.identity
• Package Version: master
• Operating System: Linux
• Python Version: 3

Describe the bug

Here:

azure-sdk-for-python/sdk/identity/azure-identity/azure/identity/_credentials/imds.py

Line 60 in e918edd

def _request_token(self, *scopes, **kwargs): # pylint:disable=unused-argument

The code assumes that if gets an Exception other than HttpResponseError, then it must be some connectivity issue. This is probably not valid, as we have tested our connection to 169.254.169.254 immediately after getting the "no managed identity endpoint found" error, and there was no connection problem.

The larger issue is that the original exception is dropped, so the user can't even know what went wrong. Rather, they get a useless CredentialUnavailableError. It would be much nicer if you wrapped the original error so the user knows what happened.

Furthermore, after the first presumed connection failure with IMDS, the _request_token method will set _endpoint_available to False, which makes it always raise CredentialUnavailableError on further calls; there is no hope for recovering.

To Reproduce
Steps to reproduce the behavior:

1. Create a SecretClient with a ManagedIdentity credential.
2. We use azure.keyvault.secrets.SecretClient.get_secret and occasionally we get:

azure.identity._exceptions.CredentialUnavailableError: ManagedIdentityCredential authentication unavailable, no managed identity endpoint found.

We retry on this error, but it never recovers, and we eventually timeout at a higher layer.

Expected behavior

I think you have a few options:

1. Actually test TCP connections with IMDS instead of relying on exception handling at the HTTP layer.
2. Wrap the exception into the CredentialUnavailableError instead of dropping it so users can see what happened.

[chlowell]

Sure, we can find a way to expose the error. I'm also curious what exactly is happening here. Is your application deployed to a VM?

after the first presumed connection failure with IMDS, the _request_token method will set _endpoint_available to False, which makes it always raise CredentialUnavailableError on further calls; there is no hope for recovering.

This is a deliberate part of the design. It's awkward in some cases (pod identity) and we've discussed alternatives but have kept this behavior because the application may be running on a platform without IMDS, in which case we don't want to keep timing out waiting for responses from it. This behavior is per instance though, so if it's problematic for your application, one (admittedly not great) workaround is to create another credential instance.

[bonsairobo]

Thanks for the quick reply!

Yes our application is running on a VM. It's a pretty simple Python script that fetches a secret from a key vault.

This is a deliberate part of the design. It's awkward in some cases (pod identity) and we've discussed alternatives but have kept this behavior because the application may be running on a platform without IMDS, in which case we don't want to keep timing out waiting for responses from it.

I see. That seems reasonable then. It would be better to avoid the blanket except Exception though.

one (admittedly not great) workaround is to create another credential instance

We have discussed this workaround. It should work fine for us. We just can't be sure our app will be able to recover from whatever is causing the CredentialUnavailableError until we can see the original error. So that kind of fix would be appreciated!