[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

[Mark-Simulacrum]

We have seen this error several times recently:
on:

• 2023-10-10T01:05:38.498Z
• 2023-10-23T01:03:25.895Z
• 2023-10-26T01:08:47.479Z
• 2023-11-04T00:59:29.566Z
• 2023-12-26 (25?)
• 2023-12-30
• 2024-03-29T00:50:30.097Z
• 2024-03-29T00:58:30.285Z
• 2024-03-30T00:48:14.868Z to 2024-03-30T01:10:33.716Z many failures

See below for a capture from the most recent occurence (2023-10-23). This appears to be a spurious error, unfortunately we don't yet have request IDs or similar to investigate further, and depending on when the failure occurs they may not be available. This may be due to new client software (e.g., awscli) or due to upstream changes, it's not clear.

Both failures happened using awscli 1.22.34-1 from Ubuntu 22.04 archives, starting just after we landed 19ed30b (which rebuilt the docker image). But we were using that awscli previously as well.

I'm not sure whether underlying openssl versions or similar changed across that time range.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                             message                                                                                                                                                                                                                                             |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1698022319473 | upload failed: release/docs/core/core_arch/arm_shared/neon/generated/fn.vtstq_s16.html to s3://static-rust-lang-org/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vtstq_s16.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vtstq_s16.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                                     |
| 1698022321511 | upload failed: release/docs/core/core_arch/arm_shared/neon/generated/fn.vzipq_s32.html to s3://static-rust-lang-org/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vzipq_s32.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vzipq_s32.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                                     |
| 1698022931768 | upload failed: release/docs/nightly-rustc/rustc_target/abi/call/loongarch/sidebar-items.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/abi/call/loongarch/sidebar-items.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/abi/call/loongarch/sidebar-items.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                         |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/sidebar-items.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/sidebar-items.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/sidebar-items.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)        |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_ohos/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_ohos/fn.target.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_ohos/fn.target.html".                                             |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_gnu/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_gnu/fn.target.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_gnu/fn.target.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)              |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/wasm32_unknown_emscripten/index.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/wasm32_unknown_emscripten/index.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/wasm32_unknown_emscripten/index.html".                                                         |
| 1698023003878 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_sun_solaris/index.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_sun_solaris/index.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_sun_solaris/index.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                            |
| 1698023003878 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/fn.target.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/fn.target.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)              |
| 1698023005895 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_unknown_l4re_uclibc/sidebar-items.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_l4re_uclibc/sidebar-items.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_l4re_uclibc/sidebar-items.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)  |
| 1698023005895 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_uwp_windows_gnu/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_uwp_windows_gnu/fn.target.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_uwp_windows_gnu/fn.target.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[Mark-Simulacrum]

We disabled debug logging because it was rather excessive, but this happened again today:

running "aws" "s3" "sync" "--storage-class" "INTELLIGENT_TIERING" "--delete" "--only-show-errors" "/codebuild/output/src2664974370/src/release/docs/" "s3://static-rust-lang-org/doc/nightly/"
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustVersionMinor.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustVersionMinor.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustVersionMinor.html".
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrepareThinLTOInternalize.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrepareThinLTOInternalize.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrepareThinLTOInternalize.html".
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMSetGlobalConstant.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMSetGlobalConstant.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMSetGlobalConstant.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrintStatistics.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrintStatistics.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrintStatistics.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/foreigntype.ModuleBuffer.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/foreigntype.ModuleBuffer.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/foreigntype.ModuleBuffer.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

We might want to consider an in-code retry of all aws s3 commands or something...

[Mark-Simulacrum]

Happened again today:

upload failed: release/docs/alloc/macro.vec!.html to s3://static-rust-lang-org/doc/nightly/alloc/macro.vec!.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/alloc/macro.vec%21.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
[...] dozens more [...]
upload failed: release/docs/nightly-rustc/rustc_passes/errors/struct.OnlyHasEffectOn.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_passes/errors/struct.OnlyHasEffectOn.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_passes/errors/struct.OnlyHasEffectOn.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
Error: failed command:"aws" "s3" "sync" "--storage-class" "INTELLIGENT_TIERING" "--delete" "--only-show-errors" "/codebuild/output/src2335245485/src/release/docs/" "s3://static-rust-lang-org/doc/nightly/"

[Mark-Simulacrum]

I manually enabled S3 access logs for static-rust-lang-org. My hope is that gives us request IDs without changing the client, but given the error is an SSL error I'm not super optimistic it'll be tied to any particular request.

[Mark-Simulacrum]

Failed again today:

2024-01-04T01:08:33.019Z upload failed: release/docs/nightly-rustc/src/cargo/core/resolver/context.rs.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/src/cargo/core/resolver/context.rs.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/src/cargo/core/resolver/context.rs.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

Looking at S3 logs (now that we have those), I'm not seeing any references to this file, though there are uploads around the same time from the codebuild job. I don't see any references to this particular file path or obvious errors for PUT requests.

So unfortunately no real luck from having those logs...

[Mark-Simulacrum]

Occurred again today:

upload failed: release/docs/core/arch/x86/fn._mm512_cvtepu8_epi64.html to s3://static-rust-lang-org/doc/nightly/core/arch/x86/fn._mm512_cvtepu8_epi64.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/core/arch/x86/fn._mm512_cvtepu8_epi64.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

[Mark-Simulacrum]

upload failed: release/docs/nightly-rustc/trait.impl/rustc_hir/intravisit/nested_filter/trait.NestedFilter.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/trait.impl/rustc_hir/intravisit/nested_filter/trait.NestedFilter.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/trait.impl/rustc_hir/intravisit/nested_filter/trait.NestedFilter.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)