Pin azure-ai-ml==1.31.0 to fix inference server crash#3856
Open
Chakradhar886 wants to merge 17 commits intomainfrom
Open
Pin azure-ai-ml==1.31.0 to fix inference server crash#3856Chakradhar886 wants to merge 17 commits intomainfrom
Chakradhar886 wants to merge 17 commits intomainfrom
Conversation
azure-ai-ml is only used in notebook cells, not in score.py. Its 100+ transitive dependencies conflict with the inference server. Also replace azure-keyvault metapackage with azure-keyvault-secrets (the only sub-package score.py imports).
bac28af to
addf2c9
Compare
Replace openmpi4.1.0-ubuntu20.04:latest with openmpi4.1.0-ubuntu22.04:latest to resolve EolImageBrownouts deployment failure.
|
@Chakradhar886 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
minimal-py312-inference has pre-installed Python 3.12 that conflicts with conda env python=3.10, crashing the inference server (502). openmpi4.1.0-ubuntu22.04 is a clean base without this conflict.
Previous failed CI runs left soft-deleted vaults in the 0-10000 range. Increase to 0-100000 to reduce collision probability.
openmpi4.1.0-ubuntu22.04 requires azureml-defaults (not just azureml-inference-server-http) to set up the full serving stack. This matches the pattern used by all other working openmpi deployments in the repo (e.g. model-1/environment/conda.yaml).
- python=3.10 -> 3.12 (match minimal-py312-inference image) - Keep azureml-inference-server-http (correct for minimal images) - Remove unused azure-ai-ml - azure-keyvault -> azure-keyvault-secrets (matches score.py import) - Revert image back to minimal-py312-inference
Align SDK env.yml with the working CLI version which pins azureml-inference-server-http>=0.7.7,<1. The unpinned version may resolve to a breaking v1.x release. Also remove azure-ai-ml which is not used by score.py.
Azure changed the default for new Key Vaults to enable RBAC authorization, which causes access policies to be ignored. The CLI version already sets --enable-rbac-authorization false. Add the equivalent enable_rbac_authorization=False to the SDK notebook's VaultProperties. Also revert all env.yml changes - the root cause was never the container environment.
Replace openmpi4.1.0-ubuntu20.04 with openmpi4.1.0-ubuntu22.04 in all three notebooks. Using openmpi (not minimal-py312) because model-1/model-2 conda files use azureml-defaults which requires the openmpi image family.
Standard_DS3_v2 requires 8 vCPUs per instance, exceeding the available quota when multiple notebooks run concurrently. Standard_DS2_v2 halves the requirement to 4 vCPUs per instance.
The OutOfQuota errors (694/700 vCPUs) affect ALL notebooks in the subscription, including ones we didn't touch (binary-payloads). This is a shared CI infrastructure issue, not fixable by changing VM size. Reverting to keep the PR focused on actual bug fixes.
sdgilley
approved these changes
Apr 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
azure-ai-ml 1.32.0 (released 2026-03-16) introduces transitive dependency conflicts with azureml-inference-server-http inside the minimal-py312-inference container, causing a 502 liveness probe failure.
Description
Checklist