Intermittent BlobContainerClientTest.Tags test failure due to HTTP status code 202 and `FindBlobsByTags` not returning the expected tags

[ahsonkhan]

From an unrelated PR: #5772

On ValidateLive Win2022_x86_with_unit_test_winHttp
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3954909&view=logs&j=1cbef409-5ce5-5885-71aa-60f628bf5123&t=614bdaae-21d9-5723-b317-b9e0e4565c8b&l=434709

Test:

azure-sdk-for-cpp/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp

Line 846 in 1143b6b

TEST_F(BlobContainerClientTest, Tags)

2024-07-11T22:26:16.9327118Z 118: [2024-07-11T22:26:16.9307174Z T: 4604] INFO  : HTTP status code 200 won't be retried.
2024-07-11T22:26:17.9426083Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(954): error: Expected: (numPages1) > (2), actual: 1 vs 2
2024-07-11T22:26:17.9428592Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(955): error: Expected: (numPages2) > (2), actual: 1 vs 2
2024-07-11T22:26:17.9430137Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(956): error: Expected equality of these values:
2024-07-11T22:26:17.9431451Z 118:   findResults.size()
2024-07-11T22:26:17.9432319Z 118:     Which is: 0
2024-07-11T22:26:17.9433120Z 118:   blobNames.size()
2024-07-11T22:26:17.9433879Z 118:     Which is: 6
2024-07-11T22:26:17.9434978Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(957): error: Expected equality of these values:
2024-07-11T22:26:17.9435737Z 118:   findResults2.size()
2024-07-11T22:26:17.9436448Z 118:     Which is: 0
2024-07-11T22:26:17.9437421Z 118:   blobNames.size()
2024-07-11T22:26:17.9438407Z 118:     Which is: 6
2024-07-11T22:26:17.9439666Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(961): error: Expected equality of these values:
2024-07-11T22:26:17.9440254Z 118:   blobNames
2024-07-11T22:26:17.9440757Z 118:     Which is: { "blobQdSWwHWvd0", "blobQdSWwHWvd00", "blobQdSWwHWvd01", "blobQdSWwHWvd02", "blobQdSWwHWvd03", "blobQdSWwHWvd04" }
2024-07-11T22:26:17.9441428Z 118:   findResults
2024-07-11T22:26:17.9441804Z 118:     Which is: {}
2024-07-11T22:26:17.9442446Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(962): error: Expected equality of these values:
2024-07-11T22:26:17.9442942Z 118:   blobNames
2024-07-11T22:26:17.9443442Z 118:     Which is: { "blobQdSWwHWvd0", "blobQdSWwHWvd00", "blobQdSWwHWvd01", "blobQdSWwHWvd02", "blobQdSWwHWvd03", "blobQdSWwHWvd04" }
2024-07-11T22:26:17.9443911Z 118:   findResults2
2024-07-11T22:26:17.9444291Z 118:     Which is: {}

See detailed logs attached
ValidateLive Win2022_x86_with_unit_test_winHttp-BlobContainerClientTest.Tags.txt

[ahsonkhan]

This test has failed twice, recently (in the last 2 weeks).
https://dev.azure.com/azure-sdk/internal/_test/analytics?definitionId=1770&contextType=build

[Jinming-Hu]

This is server-side issue. We can increase this value to reduce the possibility of failing. However, it will increase the running time of this testcase.

azure-sdk-for-cpp/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp

Line 907 in 9ccd206

for (int i = 0; i < 30; ++i)

[ahsonkhan]

Is the server-side issue, something that we need to request to be fixed?
Is there some way we increase the determinism/reliability of the test and it's expectations without increasing the test run time?

Having flaky tests, that intermittently fail, ultimately defeats the purpose of tests passing/failing, and increases the signal-to-noise ratio when a test does "really" fail which requires investigation to fix a product/test bug.

[ahsonkhan]

This failed again here, on ValidateLive Ubu2204_x64_with_unit_test:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=4307597&view=logs&j=ad8416ac-7692-5d0d-c5ac-2197f7bd6cb0&t=b3cb4762-4006-5438-9ee2-dc6a405af617&l=365110

It has failed twice in the last 2 weeks, so it is still fairly rare (99%+ pass rate)

2024-11-06T22:19:05.1266483Z 163: [2024-11-06T22:19:05.1227343Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Creating a new session.
2024-11-06T22:19:05.1267641Z 163: [2024-11-06T22:19:05.1227575Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Re-using connection from the pool.
2024-11-06T22:19:05.1268781Z 163: [2024-11-06T22:19:05.1227920Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: No Host in request headers. Adding it
2024-11-06T22:19:05.1269886Z 163: [2024-11-06T22:19:05.1228148Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Send request without payload
2024-11-06T22:19:05.1270639Z 163: [2024-11-06T22:19:05.1230188Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Parse server response
2024-11-06T22:19:05.1271869Z 163: [2024-11-06T22:19:05.1263068Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Request completed. Moving response out of session and session to response.
2024-11-06T22:19:05.1272613Z 163: [2024-11-06T22:19:05.1263515Z T: 139804527198464] DEBUG : Moving connection to pool...
2024-11-06T22:19:05.1273278Z 163: [2024-11-06T22:19:05.1263726Z T: 139804527198464] DEBUG : Clean thread running. Won't start a new one.
2024-11-06T22:19:05.1273924Z 163: [2024-11-06T22:19:05.1264324Z T: 139804527198464] INFO  : HTTP/1.1 Response (3ms) : 200 OK
2024-11-06T22:19:05.1274472Z 163: access-control-allow-origin : REDACTED
2024-11-06T22:19:05.1274957Z 163: access-control-expose-headers : REDACTED
2024-11-06T22:19:05.1275423Z 163: content-type : application/xml
2024-11-06T22:19:05.1275859Z 163: date : Wed, 06 Nov 2024 22:19:05 GMT
2024-11-06T22:19:05.1276361Z 163: server : Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
2024-11-06T22:19:05.1276840Z 163: transfer-encoding : chunked
2024-11-06T22:19:05.1277358Z 163: x-ms-client-request-id : 19b5490f-cbad-432d-ac43-226a85a5cef6
2024-11-06T22:19:05.1277903Z 163: x-ms-request-id : 56abb744-d01e-000d-4c99-30064d000000
2024-11-06T22:19:05.1278376Z 163: x-ms-version : 2024-08-04
2024-11-06T22:19:05.1278951Z 163: [2024-11-06T22:19:05.1264542Z T: 139804527198464] INFO  : HTTP status code 200 won't be retried.
2024-11-06T22:19:06.1272509Z 163: /mnt/vss/_work/1/s/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp:975: Failure
2024-11-06T22:19:06.1274026Z 163: Expected: (numPages1) > (2), actual: 1 vs 2
2024-11-06T22:19:06.1276160Z 163: /mnt/vss/_work/1/s/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp:976: Failure
2024-11-06T22:19:06.1277334Z 163: Expected: (numPages2) > (2), actual: 1 vs 2
2024-11-06T22:19:06.1279599Z 163: /mnt/vss/_work/1/s/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp:977: Failure
2024-11-06T22:19:06.1280857Z 163: Expected equality of these values:
2024-11-06T22:19:06.1281898Z 163:   findResults.size()
2024-11-06T22:19:06.1282698Z 163:     Which is: 0
2024-11-06T22:19:06.1283477Z 163:   blobNames.size()
2024-11-06T22:19:06.1284248Z 163:     Which is: 6
2024-11-06T22:19:06.1285350Z 163: /mnt/vss/_work/1/s/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp:978: Failure
2024-11-06T22:19:06.1286601Z 163: Expected equality of these values:
2024-11-06T22:19:06.1287373Z 163:   findResults2.size()
2024-11-06T22:19:06.1288115Z 163:     Which is: 0
2024-11-06T22:19:06.1288841Z 163:   blobNames.size()
2024-11-06T22:19:06.1289591Z 163:     Which is: 6
2024-11-06T22:19:06.1290608Z 163: /mnt/vss/_work/1/s/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp:982: Failure
2024-11-06T22:19:06.1291539Z 163: Expected equality of these values:
2024-11-06T22:19:06.1292304Z 163:   blobNames
2024-11-06T22:19:06.1293210Z 163:     Which is: { "blobxchCSnQcqC", "blobxchCSnQcqC0", "blobxchCSnQcqC1", "blobxchCSnQcqC2", "blobxchCSnQcqC3", "blobxchCSnQcqC4" }
2024-11-06T22:19:06.1293979Z 163:   findResults
2024-11-06T22:19:06.1294695Z 163:     Which is: {}
2024-11-06T22:19:06.1295698Z 163: /mnt/vss/_work/1/s/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp:983: Failure
2024-11-06T22:19:06.1296491Z 163: Expected equality of these values:
2024-11-06T22:19:06.1297227Z 163:   blobNames
2024-11-06T22:19:06.1298121Z 163:     Which is: { "blobxchCSnQcqC", "blobxchCSnQcqC0", "blobxchCSnQcqC1", "blobxchCSnQcqC2", "blobxchCSnQcqC3", "blobxchCSnQcqC4" }
2024-11-06T22:19:06.1298870Z 163:   findResults2
2024-11-06T22:19:06.1299589Z 163:     Which is: {}
2024-11-06T22:19:06.1300612Z 163: [2024-11-06T22:19:06.1269144Z T: 139804527198464] DEBUG : Moving connection to pool...
2024-11-06T22:19:06.1301814Z 163: [2024-11-06T22:19:06.1269476Z T: 139804527198464] DEBUG : Clean thread running. Won't start a new one.
2024-11-06T22:19:06.1303052Z 163: [2024-11-06T22:19:06.1272851Z T: 139804527198464] INFO  : HTTP Request : DELETE https://***.blob.core.windows.net/blobcontainerclienttesttags?restype=container
2024-11-06T22:19:06.1304066Z 163: authorization : REDACTED
2024-11-06T22:19:06.1305183Z 163: user-agent : azsdk-cpp-storage-blobs/12.14.0-beta.1 (Linux 6.8.0-1017-azure x86_64 #20~22.04.1-Ubuntu SMP Tue Oct 22 20:42:07 UTC 2024 Cpp/-1)
2024-11-06T22:19:06.1306181Z 163: x-ms-client-request-id : 00955ca4-10df-4be1-bd5f-223dc650b467
2024-11-06T22:19:06.1307045Z 163: x-ms-date : Wed, 06 Nov 2024 22:19:06 GMT
2024-11-06T22:19:06.1307847Z 163: x-ms-version : 2024-08-04
2024-11-06T22:19:06.1308862Z 163: [2024-11-06T22:19:06.1273154Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Creating a new session.
2024-11-06T22:19:06.1309916Z 163: [2024-11-06T22:19:06.1273446Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Re-using connection from the pool.
2024-11-06T22:19:06.1310964Z 163: [2024-11-06T22:19:06.1273975Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: No Host in request headers. Adding it
2024-11-06T22:19:06.1312168Z 163: [2024-11-06T22:19:06.1274344Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Send request without payload
2024-11-06T22:19:06.1313104Z 163: [2024-11-06T22:19:06.1277103Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Parse server response
2024-11-06T22:19:06.1386789Z 163: [2024-11-06T22:19:06.1382629Z T: 139804527198464] DEBUG : [CURL Transport Adapter]: Request completed. Moving response out of session and session to response.
2024-11-06T22:19:06.1388102Z 163: [2024-11-06T22:19:06.1383197Z T: 139804527198464] DEBUG : Moving connection to pool...
2024-11-06T22:19:06.1389195Z 163: [2024-11-06T22:19:06.1383456Z T: 139804527198464] DEBUG : Clean thread running. Won't start a new one.
2024-11-06T22:19:06.1390337Z 163: [2024-11-06T22:19:06.1384274Z T: 139804527198464] INFO  : HTTP/1.1 Response (11ms) : 202 Accepted
2024-11-06T22:19:06.1391399Z 163: content-length : 0
2024-11-06T22:19:06.1392187Z 163: date : Wed, 06 Nov 2024 22:19:05 GMT
2024-11-06T22:19:06.1393075Z 163: server : Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
2024-11-06T22:19:06.1393987Z 163: x-ms-client-request-id : 00955ca4-10df-4be1-bd5f-223dc650b467
2024-11-06T22:19:06.1394903Z 163: x-ms-request-id : 41e98114-f01e-0078-5699-306d61000000
2024-11-06T22:19:06.1395707Z 163: x-ms-version : 2024-08-04
2024-11-06T22:19:06.1396678Z 163: [2024-11-06T22:19:06.1384564Z T: 139804527198464] INFO  : HTTP status code 202 won't be retried.
2024-11-06T22:19:06.1397825Z 163: [  FAILED  ] BlobContainerClientTest.Tags (30554 ms)

 
Detailed logs:
ValidateLive Ubu2204_x64_with_unit_test-azure-storage-blobs.BlobContainerClientTest.Tags.txt