azure-cosmos: SDK routes requests through public IP even with private endpoint and enable_endpoint_discovery=False

[KrzysztofKasprowicz]

Describe the bug

When using azure.cosmos.aio.CosmosClient to connect to an Azure Cosmos DB account exclusively via a private endpoint (public network access disabled), the SDK intermittently sends requests through the public IP of the Cosmos DB account, resulting in 403 Forbidden errors:

azure.cosmos.exceptions.CosmosHttpResponseError: (Forbidden)
Request originated from IP <public-ip> through public internet.
This is blocked by your Cosmos DB account firewall settings.

Key finding: Setting enable_endpoint_discovery=False does not fully resolve the issue — the 403 errors persist.

Root cause analysis

I traced through the SDK source code (v4.15.0) and identified the endpoint discovery mechanism as a likely contributor:

1. The _GlobalEndpointManager periodically (every 5 min) calls _GetDatabaseAccount() on the gateway.
2. The gateway responds with writableLocations / readableLocations containing public regional endpoint URLs (e.g., https://account-northeurope.documents.azure.com:443/), even when the request arrives via a private endpoint.
3. LocationCache.get_regional_routing_contexts_by_loc() (_location_cache.py:61-83) stores these public URLs verbatim as RegionalRoutingContext objects.
4. resolve_service_endpoint() (_location_cache.py:335-375) selects endpoints from this cache, potentially routing requests to the public URL.

Setting enable_endpoint_discovery=False should prevent this, but the 403 errors continue — suggesting there may be additional code paths in the SDK that bypass the EnableEndpointDiscovery flag and still attempt connections to public endpoints.

To Reproduce

1. Create a Cosmos DB account with:
   • Public network access: Disabled
   • A private endpoint configured in a VNet
2. Deploy an application in the same VNet (e.g., Azure Container Apps with VNet integration)
3. Initialize the async client:

from azure.cosmos.aio import CosmosClient

client = CosmosClient(
    "https://myaccount.documents.azure.com:443/",
    credential=my_aad_credential,
    enable_endpoint_discovery=False,  # does NOT prevent the issue
)

4. Perform read/write operations over an extended period (hours)
5. Intermittently, requests fail with 403 Forbidden — "Request originated from IP through public internet"

DNS verification: Independent DNS monitoring confirms the application's OS-level DNS always resolves the Cosmos FQDN to the correct private IP via the privatelink.documents.azure.com CNAME chain. The public IP in the error message does not match what the OS resolves — it appears the SDK is connecting to a different endpoint entirely.

Expected behavior

When enable_endpoint_discovery=False is set, the SDK should only use the endpoint URL provided at client initialization. No requests should be routed to gateway-discovered regional endpoints, especially public ones.

Even when enable_endpoint_discovery=True (default), if the client was initialized with a private endpoint URL, the SDK should not route requests to public regional endpoints that are unreachable due to firewall rules.

Additional context

• Environment: Azure Container Apps with VNet integration, private endpoint to Cosmos DB
• Authentication: Azure AD (DefaultAzureCredential) — async variant
• Cosmos DB: Single-region account, North Europe, public access disabled
• The error is intermittent — most requests succeed (via private IP), but periodically requests hit the public IP
• The documentdb-dotnet-sdk/2.14.0 in the error message is from the Cosmos DB server-side, not the Python client

Setup

• OS: Linux (container on Azure Container Apps)
• Python: 3.12
• azure-cosmos: 4.15.0
• azure-identity: latest (async DefaultAzureCredential)

[KrzysztofKasprowicz]

Additional findings after deeper investigation

We've conducted extensive validation on our side:

DNS Resolution: We deployed a DNS resolution monitor that resolved the Cosmos DB FQDN every 30 seconds using socket.gethostbyname_ex. Over several days of monitoring, DNS always resolved correctly through the private endpoint chain:

dev-***.documents.azure.com → dev-***.privatelink.documents.azure.com → 10.x.x.x (private IP)

There was never a single DNS mis-resolution observed.

VNet Routing: Our Azure Container Apps environment has VNet integration, and the private endpoint is correctly provisioned with a private DNS zone. All other Azure services using private endpoints from the same VNet work without issues.

enable_endpoint_discovery=False: We applied this setting to both CosmosClient initialization paths (master key and AAD credential). The 403 errors persist even with endpoint discovery disabled.

Source code trace with enable_endpoint_discovery=False

We traced through the SDK source code (v4.15.0) and confirmed that with EnableEndpointDiscovery=False:

1. update_location_cache() in _location_cache.py:464-477 correctly gates the population of regional endpoints behind the EnableEndpointDiscovery check — so account_read_regional_routing_contexts_by_location and account_write_regional_routing_contexts_by_location remain empty.
2. get_preferred_regional_routing_contexts() falls back to the default endpoint when no regional endpoints are populated.
3. resolve_service_endpoint() should always return the default endpoint.
4. Background refresh (_refresh_database_account_and_health) still runs every 5 minutes and calls _GetDatabaseAccount() + perform_on_database_account_read(), but the location cache update is gated.

Despite this, the SDK intermittently sends requests that Cosmos DB sees as originating from our public outbound IP (not the private endpoint IP). This suggests either:

• There is an additional code path in the SDK that bypasses the EnableEndpointDiscovery flag and routes to a different endpoint
• Or the issue is at a lower layer (HTTP transport, connection pooling, or Azure networking infrastructure)

Questions for the SDK team

1. Are there known issues with private endpoint traffic being routed through public IPs in the Python SDK's HTTP transport layer (aiohttp)?
2. Does the SDK's AsyncPipelineClient or its retry policies ever modify the request URL in a way that could bypass the EnableEndpointDiscovery=False setting?
3. Could the background _GetDatabaseAccount() call (which still happens even with discovery disabled) somehow affect routing for subsequent requests?
4. Is there any DNS caching or connection pooling behavior in the SDK's transport layer that could cause intermittent routing to a public IP even when the OS-level DNS correctly resolves to the private IP?

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AbhinavTrips @bambriz @Pilchie @pjohari-ms @simorenoh.

[kashifkhan]

Thank you for the feedback @KrzysztofKasprowicz . Im routing this over to the right team to help you further