help request: Health checks only start for the first accessed route; other routes' upstreams are not checked

[liu1004010308]

Description

Hi team,
I'm using APISIX and have defined multiple routes with different upstreams. However, I've noticed that only the upstream associated with the first accessed route has active health checks running. Other upstreams do not seem to initiate health checks
Here is a summary of the issue:
I have two routes, each with a unique host, uri, vars, and upstream configuration (defined inline).
When I access the first route, its upstream starts health checking as expected.
When I access the second route — confirmed via logs that the request matches and the upstream is used — APISIX does not initiate health checks for that upstream.

two routes like this:
one route:
{
"create_time": 1744595151,
"id": "00000000000000000070",
"enable_websocket": true,
"hosts": [
"yun.test.com.cn"
],
"uris": [
"/path1/path2*"
],
"methods": [
"GET",
"POST",
"PUT",
"PATCH",
"OPTIONS",
"HEAD",
"DELETE",
"CONNECT",
"TRACE",
"PURGE"
],
"desc": "",
"update_time": 1744713210,
"plugins": {
"proxy-rewrite": {
"headers": {
"Upgrade": "$http_upgrade",
"X-Real-IP": "$remote_addr",
"X-Forwarded-For": "$proxy_add_x_forwarded_for",
"Connection": "upgrade",
"Host": "$host"
},
"use_real_request_uri_unsafe": true
}
},
"vars": [
[
"server_port",
"==",
9002
]
],
"priority": 0,
"upstream": {
"scheme": "http",
"hash_on": "vars",
"type": "roundrobin",
"timeout": {
"send": 7200,
"connect": 60,
"read": 7200
},
"pass_host": "pass",
"nodes": {
"10.13.90.23:10004": 50,
"10.13.90.24:10004": 50
},
"checks": {
"active": {
"https_verify_certificate": true,
"type": "tcp",
"timeout": 5,
"unhealthy": {
"interval": 5,
"http_failures": 3,
"tcp_failures": 2,
"timeouts": 3,
"http_statuses": [
429,
404,
500,
501,
502,
503,
504,
505
]
},
"concurrency": 10,
"healthy": {
"interval": 2,
"http_statuses": [
200,
302
],
"successes": 1
},
"http_path": "/"
},
"passive": {
"healthy": {
"http_statuses": [
200,
201
],
"successes": 3
},
"type": "http",
"unhealthy": {
"timeouts": 7,
"http_statuses": [
500
],
"http_failures": 3,
"tcp_failures": 3
}
}
},
"keepalive_pool": {
"idle_timeout": 60,
"size": 320,
"requests": 1000
}
},
"status": 1
}

another route
{
"create_time": 1744595151,
"id": "00000000000000000074",
"enable_websocket": true,
"hosts": [
"10.13.90.22"
],
"uris": [
"*"
],
"methods": [
"GET",
"POST",
"PUT",
"PATCH",
"OPTIONS",
"HEAD",
"DELETE",
"CONNECT",
"TRACE",
"PURGE"
],
"desc": "",
"update_time": 1744713210,
"plugins": {
"proxy-rewrite": {
"headers": {
"Upgrade": "$http_upgrade",
"X-Real-IP": "$remote_addr",
"X-Forwarded-For": "$proxy_add_x_forwarded_for",
"Connection": "upgrade",
"Host": "$host"
},
"use_real_request_uri_unsafe": true
}
},
"vars": [
[
"server_port",
"==",
9003
]
],
"priority": 0,
"upstream": {
"scheme": "http",
"hash_on": "vars",
"type": "roundrobin",
"timeout": {
"send": 7200,
"connect": 60,
"read": 7200
},
"pass_host": "pass",
"nodes": {
"10.13.90.23:10005": 50,
"10.13.90.24:10005": 50
},
"checks": {
"active": {
"https_verify_certificate": true,
"type": "tcp",
"timeout": 5,
"unhealthy": {
"interval": 5,
"http_failures": 3,
"tcp_failures": 2,
"timeouts": 3,
"http_statuses": [
429,
404,
500,
501,
502,
503,
504,
505
]
},
"concurrency": 10,
"healthy": {
"interval": 2,
"http_statuses": [
200,
302
],
"successes": 1
},
"http_path": "/"
},
"passive": {
"healthy": {
"http_statuses": [
200,
201
],
"successes": 3
},
"type": "http",
"unhealthy": {
"timeouts": 7,
"http_statuses": [
500
],
"http_failures": 3,
"tcp_failures": 3
}
}
},
"keepalive_pool": {
"idle_timeout": 60,
"size": 320,
"requests": 1000
}
},
"status": 1
}

debug log like this:

Environment

• APISIX version (run apisix version):3.11
• Operating system (run uname -a): Linux centos79 3.10.0-1160.el7.x86_64 change: added doc of how to load plugin. #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[liu1004010308]

Could someone please help me take a look at this issue?

[liu1004010308]

@Baoyuantop Thank you, please let me know once it's confirmed.

[Baoyuantop]

I've noticed that only the upstream associated with the first accessed route has active health checks running

Have you changed the order of visits to confirm this?

[liu1004010308]

@Baoyuantop
yes, I have changed the order.
For example: the first time it requests route1, it keeps checking route1.
If I take route1 offline and access route2, it then keeps probing route2.

[Baoyuantop]

Hi @liu1004010308, by APISIX's design, health checks only start when upstream is requested; if upstream is configured but not requested, it does not trigger the start of a health check. Did you confirm that all your upstreams are being accessed?

[Baoyuantop]

Hi @liu1004010308, any update?

[Baoyuantop]

If there is still a problem, please open it again.