Skip to content

Commit 6e0f3e2

Browse files
kushagraThaparCopilot
kushagraThapar
and
Copilot
authored
Cosmos CI build optimizations (#48260)
* Optimize Cosmos CI: reduce PR matrix, increase parallelism, skip shade 1. PR-conditional emulator matrix (16 → 11 jobs): Drops redundant JDK variants for Spark/Kafka in PR builds. Full matrix on main merges. Dropped for PRs (5 jobs, ~5 agent hours saved): - Spark 3.3 Java 11 (keeping Java 8) - Spark 3.4 Java 8 (keeping Java 11) - Spark 3.5/Scala 2.12 Java 8 (keeping Java 17) - Spark 4.0/Scala 2.13 Java 17 (keeping Java 21) - Kafka Java 11 (keeping Java 17) 2. Increase BuildParallelization from 1 to 2 in all stages (Build, TestEmulator, TestVNextEmulator). 3. Skip maven-shade-plugin for non-Spark/non-Kafka emulator jobs: Core emulator, long emulator, and encryption jobs don't need Spark/Kafka uber JARs. Adding -Dshade.skip=true saves ~90s of shade plugin execution per Spark module × 5 modules = ~7-8 min per non-Spark job (5 jobs × 7 min = ~35 min agent time saved). 4. Remove outdated comment about emulator download time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix: also skip maven-antrun-plugin when shade is skipped The antrun 03-repack phase expects shade output (native .jnilib/.so files in target/tmp/). When -Dshade.skip=true, the shade output doesn't exist and antrun fails with 'Could not find file'. Add -Dmaven.antrun.skip=true alongside -Dshade.skip=true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix: pass shade.skip to both build AND test steps via AdditionalArgs The test step runs 'clean verify' which recompiles everything from scratch, including Spark shade. Our BuildOptions only affected the build step. Add -Dshade.skip=true -Dmaven.antrun.skip=true to AdditionalArgs for non-Spark jobs so it flows into TestOptions too. Keep BuildOptions for the build step as well (both steps need it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add BuildOptions plumbing to skip shade in Build stage unit test jobs Add BuildOptions parameter through ci.yml → ci.tests.yml → build-and-test.yml pipeline chain. Defaults to empty string (no behavior change for other SDKs). Cosmos Build stage sets BuildOptions to '-Dshade.skip=true -Dmaven.antrun.skip=true' to skip Spark/Kafka uber JAR creation during unit test matrix jobs, saving ~14 min per job. The release artifact deploy step is unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add per-job ProjectListOverride for Spark/Kafka emulator jobs Each Spark emulator job previously compiled ALL 14 modules including other Spark versions it doesn't test, wasting ~11 min per job on unnecessary shade+compile. Changes: - generate-project-list.ps1: Check for ProjectListOverride env var at the top. If set, use it directly and skip normal computation. Defaults to empty (no behavior change for other SDKs). - Emulator matrix JSONs: Add ProjectListOverride for each Spark and Kafka job with only the modules they need (core + their specific Spark/Kafka module). Example: Spark 3.5/2.13 job previously built 14 modules (41 min test step). Now builds only 6 modules, saving ~11 min per Spark job. Estimated savings: ~11 min × 9 Spark jobs + ~5 min × 2 Kafka jobs = ~109 min agent time per full CI run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * No-op: trigger Cosmos CI for build optimization validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add ProjectListOverride for emulator/encryption jobs to skip Spark compilation Emulator, Long Emulator, and Encryption jobs were compiling all 14 cosmos modules including 7 Spark modules (Scala compilation ~10-16 min) despite only running core emulator tests. Add ProjectListOverride to limit these jobs to only the modules they actually test: - Emulator/Long Emulator: azure-cosmos, azure-cosmos-test, azure-cosmos-tests - Encryption: adds azure-cosmos-encryption Also reverts the no-op TestSuiteBase trigger commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Guard ProjectListOverride from being used in FromSource runs FromSource runs use ClientFromSourcePom.xml to build all track 2 libraries and rely on the computed project list for -pl/-amd test scoping. Allowing ProjectListOverride to take effect during FromSource runs would restrict the test scope incorrectly. Check $env:TESTFROMSOURCE and skip the override when it is 'true'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent aa1a3b1 commit 6e0f3e2

7 files changed

Lines changed: 221 additions & 35 deletions

File tree

‎eng/pipelines/scripts/generate-project-list.ps1‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,29 @@ $additionalModulesList = @()
44

55
. "${PSScriptRoot}/../../common/scripts/common.ps1"
66

7+
# If ProjectListOverride is set (e.g., from matrix variables), use it directly
8+
# to avoid building unnecessary modules in jobs that only test a subset.
9+
# Do not honor ProjectListOverride for FromSource runs — FromSource builds use
10+
# ClientFromSourcePom.xml which builds all libraries and the project list must
11+
# be computed from the artifacts, not overridden.
12+
if ($env:PROJECTLISTOVERRIDE -and $env:PROJECTLISTOVERRIDE -notlike '*ProjectListOverride*') {
13+
if ($env:TESTFROMSOURCE -eq 'true') {
14+
Write-Host "Ignoring ProjectListOverride for FromSource run (TestFromSource=true)"
15+
} else {
16+
$projects = $env:PROJECTLISTOVERRIDE
17+
Write-Host "Using ProjectListOverride = $projects"
18+
Write-Host "##vso[task.setvariable variable=ProjectList;]$projects"
19+
Write-Host "##vso[task.setvariable variable=ArtifactsList;]$projects"
20+
Write-Host "##vso[task.setvariable variable=AdditionalModulesList;]"
21+
22+
$sha256 = new-object -TypeName System.Security.Cryptography.SHA256Managed
23+
$utf8 = new-object -TypeName System.Text.UTF8Encoding
24+
$projectListSha256 = [Convert]::ToBase64String($sha256.ComputeHash($utf8.GetBytes($projects)))
25+
Write-Host "##vso[task.setvariable variable=ProjectListSha256;]$projectListSha256"
26+
return
27+
}
28+
}
29+
730
if ($env:ARTIFACTSJSON -and $env:ARTIFACTSJSON -notlike '*ArtifactsJson*') {
831
$artifacts = $env:ARTIFACTSJSON | ConvertFrom-Json
932
foreach ($artifact in $artifacts) {

‎eng/pipelines/templates/jobs/ci.tests.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ parameters:
4040
- name: BuildParallelization
4141
type: string
4242
default: '2C'
43+
- name: BuildOptions
44+
type: string
45+
default: ''
4346
- name: AdditionalLintingOptions
4447
type: string
4548
default: ''
@@ -145,6 +148,7 @@ jobs:
145148
TestOptions: ${{ parameters.TestOptions }}
146149
TestParallelization: ${{ parameters.TestParallelization }}
147150
BuildParallelization: ${{ parameters.BuildParallelization }}
151+
BuildOptions: ${{ parameters.BuildOptions }}
148152
TestEnvVars: ${{ parameters.EnvVars }}
149153
VersionOverride: ${{ parameters.VersionOverride }}
150154

‎eng/pipelines/templates/jobs/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ parameters:
4040
- name: BuildParallelization
4141
type: string
4242
default: '2C'
43+
- name: BuildOptions
44+
type: string
45+
default: ''
4346
- name: TestGoals
4447
type: string
4548
default: $(TestGoals)
@@ -503,6 +506,7 @@ jobs:
503506
TestOptions: ${{ parameters.TestOptions }}
504507
TestParallelization: ${{ parameters.TestParallelization }}
505508
BuildParallelization: ${{ parameters.BuildParallelization }}
509+
BuildOptions: ${{ parameters.BuildOptions }}
506510
AdditionalLintingOptions: ${{ parameters.AdditionalLintingOptions }}
507511
TimeoutInMinutes: ${{ parameters.TimeoutInMinutes }}
508512
EnvVars: ${{ parameters.EnvVars }}

‎eng/pipelines/templates/stages/cosmos-emulator-matrix-pr.json‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
{
2+
"displayNames": {
3+
"clean verify": ""
4+
},
5+
"matrix": {
6+
"Agent": {
7+
"windows-2022": {
8+
"OSVmImage": "env:WINDOWSVMIMAGE",
9+
"Pool": "env:WINDOWSPOOL"
10+
}
11+
},
12+
"TestGoals": "clean verify",
13+
"EmulatorConfig": {
14+
"Emulator Only Integration Tests Tcp - Java 8": {
15+
"ProfileFlag": "-Pemulator",
16+
"PROTOCOLS": "[\"Tcp\"]",
17+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
18+
"JavaTestVersion": "1.8",
19+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true -Dshade.skip=true -Dmaven.antrun.skip=true",
20+
"BuildOptions": "-Dshade.skip=true -Dmaven.antrun.skip=true",
21+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests"
22+
},
23+
"Emulator Only Integration Tests Tcp - Java 17": {
24+
"ProfileFlag": "-Pemulator",
25+
"PROTOCOLS": "[\"Tcp\"]",
26+
"DESIRED_CONSISTENCIES": "[\"Strong\"]",
27+
"JavaTestVersion": "1.17",
28+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true -Dshade.skip=true -Dmaven.antrun.skip=true",
29+
"BuildOptions": "-Dshade.skip=true -Dmaven.antrun.skip=true",
30+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests"
31+
},
32+
"Long Emulator Only Integration Tests Tcp - Java 8": {
33+
"ProfileFlag": "-Plong-emulator",
34+
"PROTOCOLS": "[\"Tcp\"]",
35+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
36+
"JavaTestVersion": "1.8",
37+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true -Dshade.skip=true -Dmaven.antrun.skip=true",
38+
"BuildOptions": "-Dshade.skip=true -Dmaven.antrun.skip=true",
39+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests"
40+
},
41+
"Long Emulator Only Integration Tests Tcp - Java 17": {
42+
"ProfileFlag": "-Plong-emulator",
43+
"PROTOCOLS": "[\"Tcp\"]",
44+
"DESIRED_CONSISTENCIES": "[\"Strong\"]",
45+
"JavaTestVersion": "1.17",
46+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true -Dshade.skip=true -Dmaven.antrun.skip=true",
47+
"BuildOptions": "-Dshade.skip=true -Dmaven.antrun.skip=true",
48+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests"
49+
},
50+
"Encryption Emulator Only Integration Tests": {
51+
"ProfileFlag": "-Pencryption-integration",
52+
"PROTOCOLS": "[\"Tcp\"]",
53+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
54+
"JavaTestVersion": "1.8",
55+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true -Dshade.skip=true -Dmaven.antrun.skip=true",
56+
"BuildOptions": "-Dshade.skip=true -Dmaven.antrun.skip=true",
57+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-encryption,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests"
58+
},
59+
"Spark 3.3 Integration Tests targeting Cosmos Emulator - Java 8'": {
60+
"ProfileFlag": "-Dspark-e2e_3-3",
61+
"PROTOCOLS": "[\"Tcp\"]",
62+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
63+
"JavaTestVersion": "1.8",
64+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -Dhadoop.home.dir=D:/Hadoop -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true",
65+
"BuildOptions": "",
66+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests,com.azure.cosmos.spark:azure-cosmos-spark_3,com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12"
67+
},
68+
"Spark 3.4 Integration Tests targeting Cosmos Emulator - Java 11'": {
69+
"ProfileFlag": "-Dspark-e2e_3-4",
70+
"PROTOCOLS": "[\"Tcp\"]",
71+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
72+
"JavaTestVersion": "1.11",
73+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -Dhadoop.home.dir=D:/Hadoop -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true",
74+
"BuildOptions": "",
75+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests,com.azure.cosmos.spark:azure-cosmos-spark_3,com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12"
76+
},
77+
"Spark 3.5, Scala 2.12 Integration Tests targeting Cosmos Emulator - Java 17'": {
78+
"ProfileFlag": "-Dspark-e2e_3-5_2-12",
79+
"PROTOCOLS": "[\"Tcp\"]",
80+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
81+
"JavaTestVersion": "1.17",
82+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -Dhadoop.home.dir=D:/Hadoop -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true",
83+
"BuildOptions": "",
84+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests,com.azure.cosmos.spark:azure-cosmos-spark_3,com.azure.cosmos.spark:azure-cosmos-spark_3-5,com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12"
85+
},
86+
"Spark 3.5, Scala 2.13 Integration Tests targeting Cosmos Emulator - Java 17'": {
87+
"ProfileFlag": "-Dspark-e2e_3-5_2-13",
88+
"PROTOCOLS": "[\"Tcp\"]",
89+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
90+
"JavaTestVersion": "1.17",
91+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -Dhadoop.home.dir=D:/Hadoop -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true",
92+
"BuildOptions": "",
93+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests,com.azure.cosmos.spark:azure-cosmos-spark_3,com.azure.cosmos.spark:azure-cosmos-spark_3-5,com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-13"
94+
},
95+
"Spark 4.0, Scala 2.13 Integration Tests targeting Cosmos Emulator - Java 21'": {
96+
"ProfileFlag": "-Dspark-e2e_4-0_2-13",
97+
"PROTOCOLS": "[\"Tcp\"]",
98+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
99+
"JavaTestVersion": "1.21",
100+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -Dhadoop.home.dir=D:/Hadoop -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true",
101+
"BuildOptions": "",
102+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests,com.azure.cosmos.spark:azure-cosmos-spark_3,com.azure.cosmos.spark:azure-cosmos-spark_4-0_2-13"
103+
},
104+
"Kafka Integration Tests targeting Cosmos Emulator - Java 17": {
105+
"ProfileFlag": "-Pkafka-emulator",
106+
"PROTOCOLS": "[\"Tcp\"]",
107+
"DESIRED_CONSISTENCIES": "[\"Session\"]",
108+
"JavaTestVersion": "1.17",
109+
"AdditionalArgs": "-DACCOUNT_HOST=https://localhost:8081/ -DCOSMOS.AZURE_COSMOS_DISABLE_NON_STREAMING_ORDER_BY=true",
110+
"BuildOptions": "",
111+
"ProjectListOverride": "com.azure:azure-cosmos,com.azure:azure-cosmos-test,com.azure:azure-cosmos-tests,com.azure.cosmos.kafka:azure-cosmos-kafka-connect"
112+
}
113+
}
114+
}
115+
}

0 commit comments

Comments
 (0)
⚡