bug: Internal Server Error when call an API created with Admin UI to https upstream

[narate]

Current Behavior

When create routes with admin ui to https upstream then call an api via apisix then apisix response Internal server error.

Expected Behavior

Got response from upstream server

Error Logs

2025/08/07 11:38:48 [error] 50#50: *83823 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/ngx/ssl.lua:448: attempt to get length of local 'pem' (a nil value)
stack traceback:
coroutine 0:
        /usr/local/openresty/lualib/ngx/ssl.lua: in function 'parse_pem_cert'
        /usr/local/apisix/apisix/ssl.lua:216: in function 'create_obj_fun'
        /usr/local/apisix/apisix/core/lrucache.lua:95: in function 'cert_cache'
        /usr/local/apisix/apisix/ssl.lua:222: in function 'fetch_cert'
        /usr/local/apisix/apisix/upstream.lua:392: in function 'set_upstream'
        /usr/local/apisix/apisix/init.lua:562: in function 'handle_upstream'
        /usr/local/apisix/apisix/init.lua:748: in function 'http_access_phase'
        access_by_lua(nginx.conf:436):2: in main chunk, client: 192.168.65.1, server: _, request: "GET /get HTTP/1.1", host: "127.0.0.1"

Steps to Reproduce

1. Run APISIX via Docker
2. Access to Admin UI
3. Create Routes with Name: Get, URI: /get, upstream: nodes: httpbun.com:443, scheme: https, pass_host: node then Add
4. Access API via curl http://127.0.0.1:9080/get will get an Internal Server Error

Investigate

1. Inspect the payload of route creating

the upstream tls verify set to false

Work around fix

Remove upstream tls verify

Required:

• curl
• jq

export ADMIN_KEY=your_admin_key_here
export ROUTE_ID=route_id
curl -s -H "X-API-KEY: $ADMIN_KEY" http://127.0.0.1:9180/apisix/admin/routes/$ROUTE_ID | jq '.value | del(.upstream.tls, .create_time)' > get.patch.json
curl -s -H "X-API-KEY: $ADMIN_KEY" http://127.0.0.1:9180/apisix/admin/routes/$ROUTE_ID -X PUT -d @get.patch.json

Now we can call an API via APISIX with expected response

Environment

• APISIX version 3.13.0
• Operating system : Docker image apache/apisix:3.13.0-ubuntu, Linux e58f105a038a 6.10.14-linuxkit #1 SMP Sat May 17 08:28:57 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

[SkyeYoung]

Hello, I checked the code and documentation and found the following facts:

TL;DR: upstream.tls.verify is defined in schema, but the value is not referenced in the documentation and code

1. The document does not include this field in the upstream.
2. upstream.tls.verify is not judged in code
   

   apisix/apisix/upstream.lua

   Lines 385 to 407 in a5d06b3

   	if (scheme == "https" or scheme == "grpcs") and up_conf.tls then
   	local client_cert, client_key
   	if up_conf.tls.client_cert_id then
   	client_cert = api_ctx.upstream_ssl.cert
   	client_key = api_ctx.upstream_ssl.key
   	else
   	client_cert = up_conf.tls.client_cert
   	client_key = up_conf.tls.client_key
   	end
   	-- the sni here is just for logging
   	local sni = api_ctx.var.upstream_host
   	local cert, err = apisix_ssl.fetch_cert(sni, client_cert)
   	if not ok then
   	return 503, err
   	end
   	local key, err = apisix_ssl.fetch_pkey(sni, client_key)
   	if not ok then
   	return 503, err
   	end

3. However, the upstream_schema referenced by admin ui during development contains the tls.verify field, and the default value is also false.
4. APISIX also verifies upstream.tls.verify when accepting the upstream part.

So we need to discuss that since upstream.tls.verify actually exists in schema,

1. Should we add judgments on this field in apisix?
2. If No, should the property be removed in the schema?

cc @juzhiyuan

[SkyeYoung]

It was determined that relevant code logic and documentation needed to be added to APISIX.

I will start processing later.