bug: $ENV and $secret SSL cert references not working in Stream TLS mode

[horsley]

Current Behavior

In stream TLS (ssl_certificate_by_lua), cert/key configured as $ENV://... or $secret://... are not resolved and SSL handshake fails. HTTP works.

Expected Behavior

$ENV:// and $secret:// references should be resolved to PEM content, and TLS handshake succeeds.

Error Logs

2026/01/22 09:55:30 [info] 35#35: *15086 client 127.0.0.1:54858 connected to 0.0.0.0:443
2026/01/22 09:55:30 [debug] 35#35: *15087 stream [lua] ssl.lua:314: get_status_request_ext(): parsing status request extension ... 
2026/01/22 09:55:30 [debug] 35#35: *15087 stream [lua] ssl.lua:317: get_status_request_ext(): no contains status request extension
2026/01/22 09:55:30 [debug] 35#35: *15087 stream [lua] radixtree_sni.lua:170: match_and_set(): sni: abc.sgw.woa.com
2026/01/22 09:55:30 [info] 35#35: *15087 stream [lua] radixtree_sni.lua:210: match_and_set(): debug - matched: {"value":{"snis":["*.sgw.woa.com"],"cert":"$ENV://cert","key":"$ENV://key","id":"arr_1","type":"server","status":1},"modifiedIndex":1769045058,"key":"/ssls/arr_1","clean_handlers":{}}, context: ssl_client_hello_by_lua*, client: 127.0.0.1, server: 0.0.0.0:443
2026/01/22 09:55:30 [debug] 35#35: *15088 stream [lua] ssl.lua:214: create_obj_fun(): parsing cert for sni: abc.sgw.woa.com
2026/01/22 09:55:30 [error] 35#35: *15088 stream [lua] init.lua:173: ssl_phase(): failed to fetch ssl config: failed to parse PEM cert: PEM_read_bio_X509_AUX() failed, context: ssl_certificate_by_lua*, client: 127.0.0.1, server: 0.0.0.0:443
2026/01/22 09:55:30 [info] 35#35: *15086 SSL_do_handshake() failed (SSL: error:0A000179:SSL routines::cert cb error) while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:443
2026/01/22 09:55:30 [info] 35#35: *15086 stream [lua] init.lua:1235: stream_log_phase(): enter stream_log_phase while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:443

I tried to debug the problem by adding log to radixtree_sni.lua

      local new_ssl_value = secret.fetch_secrets(matched_ssl.value, true, matched_ssl.value, "")
                              or matched_ssl.value

      ok, err = _M.set_cert_and_key(sni, new_ssl_value)
      if not ok then
+         log_ssl_value_meta(sni, new_ssl_value)
          return false, err
      end

+local function is_secret_ref(val)
+    local core_string      = core.string
+    return core_string.has_prefix(val, "$ENV://") or
+           core_string.has_prefix(val, "$env://") or
+           core_string.has_prefix(val, "$secret://") or
+           core_string.has_prefix(val, "$SECRET://")
+end
+
+local function log_ssl_value_meta(sni, value)
+    if type(value) ~= "table" then
+        core.log.info("ssl value meta for sni: ", sni, " type=", type(value))
+        return
+    end
+
+    local cert = value.cert
+    if type(cert) == "string" then
+        core.log.info("ssl cert meta for sni: ", sni,
+                      " ref=", is_secret_ref(cert),
+                      " len=", #cert,
+                      " has_pem=", str_find(cert, "BEGIN CERTIFICATE", 1) ~= nil)
+    else
+        core.log.info("ssl cert meta for sni: ", sni, " type=", type(cert))
+    end


+
+    local key = value.key
+    if type(key) == "string" then
+        core.log.info("ssl key meta for sni: ", sni,
+                      " ref=", is_secret_ref(key),
+                      " len=", #key,
+                      " has_pem=", str_find(key, "BEGIN ", 1) ~= nil)
+    else
+        core.log.info("ssl key meta for sni: ", sni, " type=", type(key))
+end

debug log:

 2026/01/22 11:11:24 [info] 352#352: *402472 stream [lua] radixtree_sni.lua:59: log_ssl_value_meta(): ssl cert meta for sni: abc.sgw.woa.com ref=true len=11 has_pem=false,
  context: ssl_certificate_by_lua*, client: 127.0.0.1, server: 0.0.0.0:443
 2026/01/22 11:11:24 [info] 352#352: *402472 stream [lua] radixtree_sni.lua:69: log_ssl_value_meta(): ssl key meta for sni: abc.sgw.woa.com ref=true len=10 has_pem=false,
  context: ssl_certificate_by_lua*, client: 127.0.0.1, server: 0.0.0.0:443

Steps to Reproduce

1. Configure SSL object with cert "$ENV://cert" and key "$ENV://key"
   (or use "$secret:////" with secret config)
2. Start APISIX with a stream TLS listener
3. Connect to the stream TLS port using SNI that matches the SSL object

Environment

• APISIX version (run apisix version): 3.13.0
• Operating system (run uname -a): Linux horsleyli-25jtg44f4a 5.4.241-1-tlinux4-0023.1 change: added doc of how to load plugin. #1 SMP Tue May 6 19:24:58 CST 2025 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.27.1.2
  built by gcc 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)
  built with OpenSSL 3.4.1 11 Feb 2025
  TLS SNI support enabled
  configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.3.2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl3/include' --add-module=../ngx_devel_kit-0.3.3 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.28 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.37 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.20 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.16 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl3/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl3/lib' --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../ngx_multi_upstream_module-1.3.2 --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../apisix-nginx-module-1.19.2 --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../apisix-nginx-module-1.19.2/src/stream --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../apisix-nginx-module-1.19.2/src/meta --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../wasm-nginx-module-0.7.0 --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.mKRjeKfFeV/openresty-1.27.1.2/../lua-resty-events-0.2.0 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --with-http_v3_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): not using etcd
• APISIX Dashboard version, if relevant: not using dashboard
• Plugin runner version, for issues related to plugin runners: not related
• LuaRocks version, for installation issues (run luarocks --version):not related

[horsley]

Proposed fix

Align stream init with http init:

• call core.env.init() in stream_init()
• call apisix_secret.init_worker() in stream_init_worker()

Patch

diff --git a/deps/apisix/apisix/init.lua b/deps/apisix/apisix/init.lua
--- a/deps/apisix/apisix/init.lua
+++ b/deps/apisix/apisix/init.lua
@@
 function _M.stream_init(args)
     core.log.info("enter stream_init")

     core.resolver.init_resolver(args)
+    core.env.init()

     if core.config.init then
         local ok, err = core.config.init()
         if not ok then
             core.log.error("failed to load the configuration: ", err)
         end
     end

     xrpc.init()
 end

@@
 function _M.stream_init_worker()
     core.log.info("enter stream_init_worker")
     local seed, err = core.utils.get_seed_from_urandom()
@@
     if core.config.init_worker then
         local ok, err = core.config.init_worker()
         if not ok then
             core.log.error("failed to init worker process of ", core.config.type,
                            " config center, err: ", err)
         end
     end

     plugin.init_worker()
     xrpc.init_worker()
+    apisix_secret.init_worker()
     router.stream_init_worker()
     require("apisix.http.service").init_worker()
     apisix_upstream.init_worker()

———

Verification

After applying this patch on our side, stream TLS with $ENV:// cert/key succeeds and handshake completes normally.

[Baoyuantop]

Hi @horsley, thank you for your report. Welcome to submitting a PR to fix this issue.

[suryaparua-official]

Hi, i'd like to work on this issue and submit a PR.

[suryaparua-official]

I’ve added a Stream TLS test covering $ENV:// certificate resolution
in t/stream-node/tls.t. Please let me know if any adjustments are needed.

[vijayalakshmipokala]

I would like to work on this issue. please assign it to me if it's available