bug: apisix route will not sync

[chisheun-snsoft]

Current Behavior

After checking on the apisix logs and dashboard, the issue had been found that the apisix will not update to get the latest config. The restarted pods with new ip addresses were not in the registered upstream.

example:
pods current IP addresses:
10.62.9.32
10.62.45.130
10.62.13.39
apisix logs IP addresses:
10.62.32.40
10.62.41.229
10.62.40.40

Expected Behavior

the ip addresses should be aligned with the the ip addresses showing in the logs also align with the ip addresses registered in upstream.

Error Logs

checking from the ingress controller logs. Found the error logs of

2025-06-04 11:52:36.782	2025-06-04T11:52:36+08:00 error apisix/stream_route.go:95 failed to list stream_routes: function disabled
		2025-06-04 11:52:34.717
		2025-06-04 11:52:34.717
		2025-06-04 11:52:32.567
		2025-06-04 11:52:32.567
		2025-06-04 11:52:30.297
		2025-06-04 11:52:30.297
		2025-06-04 11:52:28.197
		2025-06-04 11:52:28.197

checking from the etcd logs. Found the warning logs of

2025-06-04 15:31:13.364	{"level":"warn","ts":"2025-06-04T07:31:13.364214Z","caller":"etcdserver/raft.go:416","msg":"leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk","to":"39e3f227d2d33a6d","heartbeat-interval":"100ms","expected-duration":"200ms","exceeded-duration":"56.00703ms"}
		2025-06-04 15:31:13.312

Steps to Reproduce

1. restart pods and check on the new pods ip addresses
2. check the ip addresses of the specific route from the pods on the logs from the apisix pods
3. check on the registered ip addresses in the apisix dashboard

Environment

• APISIX version (run apisix version): 3.12.0-debian
• Operating system (run uname -a): Linux apisix-679c58fc9f-fh6bf 5.10.236-228.935.amzn2.x86_64 change: added doc of how to load plugin. #1 SMP Tue May 6 04:10:50 UTC 2025 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.27.1.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): N/A

[chisheun-snsoft]

reattached the error logs

[Baoyuantop]

How did you deploy APISIX? Can you provide an example of a minimally reproducible problem?

[Baoyuantop]

Hi @chisheun-snsoft, any updates?

[Baoyuantop]

This issue is related to the current version of the apisix ingress controller, which will be discontinued soon, and we plan to release a new version of ingress controller with these issues resolved. More details can be found at https://lists.apache.org/thread/r8ydjz4nq35xokov2lc6sfhppybqwgrn

[wylie22]

hi, is this issue declared as current version known bug? we installed apisix via Helm, with the following components enabled: etcd, apisix, apisix-ingress-controller, and dashboard. we tried multiple times to replicate the issue, however it is difficult because this issue only occur intermittently. what else information do you need us to provide from our side to assist in fixing this bug?

[chisheun-snsoft]

@Baoyuantop
this attached files are the values.yaml and the route.yaml

values.txt

apisix-route.txt

[chisheun-snsoft]

This issue is related to the current version of the apisix ingress controller, which will be discontinued soon, and we plan to release a new version of ingress controller with these issues resolved. More details can be found at https://lists.apache.org/thread/r8ydjz4nq35xokov2lc6sfhppybqwgrn

Hi @Baoyuantop , may I confirm that the latest release of the ingress controller version "2.0.0-rc1" is able to solve these issues?

[Baoyuantop]

You can test it, mock-etcd has been removed in the new version.

[chisheun-snsoft]

@Baoyuantop

Based on this upgrade guide, does it means the mock-etcd is only removed in the apisix standalone version?