TLS handshake fails in hardened containers unless TLS 1.3 disabled; succeeds with GODEBUG overrides

[wandergeek]

In hardened container environments (Wolfi/MSFT-FIPS-like), Terraform provider initialization fails with a TLS handshake error. Setting either GODEBUG=tls13=0 or GODEBUG=tlshandshake=1 makes the run succeed, suggesting a TLS 1.3 handshake/curve negotiation issue in the plugin runtime path. The provider local v2.5.3 works; v2.6.1 fails in these environments.

Environment

• Terraform CLI: 1.13.5 (also observed with 1.14.0)
• Provider: hashicorp/local v2.6.1 (linux/arm64); v2.5.3 succeeds
• terraform-provider-local deps in go.mod:
  • github.com/hashicorp/terraform-plugin-go v0.29.0
  • github.com/hashicorp/terraform-plugin-framework v1.16.1
• Container: Wolfi/MSFT-FIPS-like base (not necessarily strict FIPS mode; /proc/sys/crypto/fips_enabled absent)
• Go in container: go1.25.4 linux/arm64
• OS/Arch: Linux arm64
• Crypto availability in container:
  • Ed25519 OK
  • RSA 2048 OK
  • ECDSA P-256 OK

Steps to Reproduce

1. In a Wolfi/MSFT-FIPS-like container, use minimal Terraform config with local v2.6.1:

terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "2.6.1"
    }
  }
}

resource "local_file" "test" {
  filename = "./test.txt"
  content  = "hello"
}

2. Run terraform init and terraform plan/apply.
3. Observe TLS handshake failure and "Plugin did not respond" while loading provider schema.
4. Retry with:
   • export GODEBUG=tls13=0 → succeeds reliably
   • Or export GODEBUG=tlshandshake=1 → also succeeds reliably

Expected Behavior

• Provider mTLS handshake and schema loading succeed in hardened containers without requiring GODEBUG changes.

Actual Behavior

• Handshake fails unless TLS 1.3 is disabled or tlshandshake logging is enabled.

Logs / Diagnostics

• Terraform GRPC/TRACE shows channel READY then immediate TLS failure (or EOF) on first handshake attempt.
• With GODEBUG=tls13=0, runs complete successfully.
• Example logs: https://gist.github.com/wandergeek/5fca2e5d9361c5ae4f9a4a316a76bc09

Additional Context

• The provider code does not implement TLS; handshake is owned by the plugin runtime. The success of TLS 1.2 and the sensitivity to tlshandshake=1 (timing perturbation) point to a TLS 1.3 negotiation quirk, likely curve/keyshare preference (e.g., X25519 vs P-256) or a timing-specific path in hardened libc/crypto?

Proposed Direction (for discussion)

• Prefer P-256 in CurvePreferences (e.g., tls.CurveP256 first) for the provider RPC client/server TLS config.
• Ensure a robust TLS 1.2 fallback is available (MinVersion: tls.VersionTLS12), and consider retrying once on tls: handshake failure with a slightly different TLS config.
• If the appropriate place for this change is the framework or another shared component, please redirect; happy to refile.

Workarounds

• export GODEBUG=tls13=0
• Or export GODEBUG=tlshandshake=1

[SBGoods]

Closing as this is a provider-specific issue (see: hashicorp/terraform-provider-local#461 (comment))