help request: how can i work it correctly？about upstream config

[LeeZBug]

Description

I created two routes:
One is associated with an upstream,
The other directly configured with the upstream during route creation.

But the ‘‘associated upstream’’ routes can't work correctly，this is my configuration（by admin api）：
route config：

{
  "total": 2,
  "list": [
    {
      "modifiedIndex": 78,
      "value": {
        "status": 1,
        "uri": "/qywx/*",
        "create_time": 1758693580,
        "methods": [
          "GET",
          "POST",
          "PUT",
          "DELETE",
          "PATCH",
          "HEAD",
          "OPTIONS",
          "CONNECT",
          "TRACE",
          "PURGE"
        ],
        "upstream": {
          "hash_on": "vars",
          "type": "roundrobin",
          "scheme": "https",
          "nodes": [
            {
              "weight": 0,
              "priority": 0,
              "host": "qyapi.weixin.qq.com",
              "port": 443
            }
          ],
          "pass_host": "pass"
        },
        "priority": 0,
        "id": "00000000000000000015",
        "plugins": {
          "proxy-rewrite": {
            "regex_uri": [
              "/qywx/*",
              "/$1"
            ],
            "use_real_request_uri_unsafe": false,
            "host": "qyapi.weixin.qq.com"
          }
        },
        "name": "qyapi.weixin.qq.com1",
        "update_time": 1758704358,
        "host": "api.gateway.com",
        "enable_websocket": false
      },
      "createdIndex": 16,
      "key": "/apisix/routes/00000000000000000015"
    },
    {
      "modifiedIndex": 77,
      "value": {
        "status": 1,
        "uri": "/qywx2/*",
        "create_time": 1758702981,
        "upstream_id": "00000000000000000053",
        "host": "api.gateway.com",
        "priority": 0,
        "plugins": {
          "proxy-rewrite": {
            "regex_uri": [
              "/qywx2/*",
              "/$1"
            ],
            "use_real_request_uri_unsafe": false,
            "host": "qyapi.weixin.qq.com"
          }
        },
        "name": "qyapi.weixin.qq.com2",
        "update_time": 1758703415,
        "id": "00000000000000000072",
        "enable_websocket": false
      },
      "createdIndex": 73,
      "key": "/apisix/routes/00000000000000000072"
    }
  ]
}

upstream config：

{
  "total": 1,
  "list": [
    {
      "modifiedIndex": 71,
      "value": {
        "id": "00000000000000000053",
        "create_time": 1758700908,
        "type": "roundrobin",
        "scheme": "https",
        "pass_host": "pass",
        "hash_on": "vars",
        "name": "qyapi.weixin.qq.com",
        "update_time": 1758702673,
        "nodes": [
          {
            "weight": 0,
            "priority": 0,
            "host": "qyapi.weixin.qq.com",
            "port": 443
          }
        ],
        "tls": {
          "verify": false
        }
      },
      "createdIndex": 54,
      "key": "/apisix/upstreams/00000000000000000053"
    }
  ]
}

work correctly

can‘t work correctly：

And log is different：

work correctly：

10.10.21.63 - - [24/Sep/2025:09:13:35 +0000] api.gateway.com "POST /qywx/cgi-bin/webhook/send?key=16294052-87ec-404e-aba5-xxxx HTTP/2.0" 200 27 0.279 "-" "Reqable/2.33.12" 101.226.141.58:443 200 0.277 "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=16294052-87ec-404e-aba5-xxxx"

can't work correctly：

2025/09/24 09:13:49 [error] 54#54: *371231 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/ngx/ssl.lua:448: attempt to get length of local 'pem' (a nil value)
stack traceback:
coroutine 0:
/usr/local/openresty/lualib/ngx/ssl.lua: in function 'parse_pem_cert'
/usr/local/apisix/apisix/ssl.lua:216: in function 'create_obj_fun'
/usr/local/apisix/apisix/core/lrucache.lua:95: in function 'cert_cache'
/usr/local/apisix/apisix/ssl.lua:222: in function 'fetch_cert'
/usr/local/apisix/apisix/upstream.lua:392: in function 'set_upstream'
/usr/local/apisix/apisix/init.lua:562: in function 'handle_upstream'
/usr/local/apisix/apisix/init.lua:748: in function 'http_access_phase'
access_by_lua(nginx.conf:424):2: in main chunk, client: 10.10.21.63, server: _, request: "POST /qywx2/cgi-bin/webhook/send?key=16294052-87ec-404e-xxx HTTP/2.0", host: "api.gateway.com"
10.10.21.63 - - [24/Sep/2025:09:13:49 +0000] api.gateway.com "POST /qywx2/cgi-bin/webhook/send?key=16294052-87ec-404e-aba5-xxxx HTTP/2.0" 500 249 0.000 "-" "Reqable/2.33.12" - - - "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=16294052-87ec-404e-aba5-xxxx "

Environment

• APISIX version (run apisix version): apisix:3.13.0-debian （run by docker）
• Operating system (run uname -a): Linux fyitiplwng02 5.15.0-25-generic feature: get the consumer id by auth plugin like key-auth. #25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.27.1.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): etcd:3.5.11
• APISIX Dashboard version, if relevant: UI Commit SHA 70712bd33f55f7979d4cb73a898e9778e0fbfe8b
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

Hi @LeeZBug, APISIX 3.13.0 has a issue when processing referenced upstreams with the tls field: even if only tls.verify=false is set and no client certificate field is configured, the client certificate parsing process will be incorrectly executed, and nil will be passed to ngx.ssl.parse_pem_cert, triggering a runtime error. Inline upstreams work correctly because they don't contain the tls field, thus avoiding the flaw.

This issue has been fixed in #12527, waiting for the version release.

You can remove the tls related configuration in the upstream configuration and try again.

[LeeZBug]

Hi @LeeZBug, APISIX 3.13.0 has a issue when processing referenced upstreams with the tls field: even if only tls.verify=false is set and no client certificate field is configured, the client certificate parsing process will be incorrectly executed, and nil will be passed to ngx.ssl.parse_pem_cert, triggering a runtime error. Inline upstreams work correctly because they don't contain the tls field, thus avoiding the flaw.

This issue has been fixed in #12527, waiting for the version release.

You can remove the tls related configuration in the upstream configuration and try again.

Does this issue only occur when I create the upstream resource? Because I noticed that specifying upstream ocnfig in routes — even with tls.verify=falseand no client certificate — doesn't cause this problem."

[Baoyuantop]

You can verify it with the latest master branch code.

[Baoyuantop]

If there is still a problem, please open it again.