help request: Delayed and DEADLINE_EXCEEDED APISIX requests

[Adrianckwai]

Description

Hi Apisix team,

We have encountered an issue, where our service A request to service B via APISIX using GRPC scheme. We noticed that our APISIX will delayed or sometimes unable to receive request, before upstream to our service B. This issue happened intermittently and has been impacting our production environment. Hope we can get help on this. Thanks!

Trace for delayed APISIX request:

Trace for DEADLINE_EXCEEDED request:

Error msg from service A:
{"level":50,"time":1748229084607,"instance_id":"1","trace_id":"885e97e4c157355063e62855bd85483e","span_id":"a9942b384a76f518","trace_flags":1,"msg"/:"Error in grpcClientCall: 4 DEADLINE_EXCEEDED: Deadline exceeded after 5.000s,name resolution: 0.001s,metadata filters: 0.001s,LB pick: 0.004s,remote_addr=10.253.24.104:80 Error"}

APISIX route config:
$ k get ar -n fpms-nt auth-apiserver-internal-route -o yaml

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: fpms-nt:apisix.apache.org/ApisixRoute:fpms-nt/auth-apiserver-internal-route
  creationTimestamp: "2025-04-17T06:52:15Z"
  generation: 24
  name: auth-apiserver-internal-route
  namespace: fpms-nt
  resourceVersion: "141266233"
  uid: 51c15c9d-0d7f-42cd-9062-d12c1718696f
spec:
  http:
  - backends:
    - serviceName: auth-apiserver-cs
      servicePort: 80
      weight: 0
    - serviceName: auth-apiserver-ss
      servicePort: 80
      weight: 100
    match:
      hosts:
      - fpms-nt-internal.fpms100.me
      - fpms-nt-internal.fpms10.me
      paths:
      - /auth.*
    name: auth-apiserver-internal-route
    plugins:
    - enable: true
      name: prometheus
    - config:
        sampler:
          name: always_on
          options:
            fraction: 0.1
            root:
              name: always_on
              options:
                fraction: 0.1
      enable: true
      name: opentelemetry
    - config:
        count: 5000
        key: http_service
        key_type: var
        policy: redis
        redis_host: prod-fpms-nt-apisix-valkey-sg.7odqjm.ng.0001.apse1.cache.amazonaws.com
        redis_port: 6379
        rejected_code: 429
        time_window: 1
      enable: true
      name: limit-count
status:
  conditions:
  - message: Sync Successfully
    observedGeneration: 18
    reason: ResourcesSynced
    status: "True"
    type: ResourcesAvailable

APISIX upstream config:
$ k get apisixupstream -n fpms-nt auth-apiserver-ss -o yaml

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: fpms-nt:apisix.apache.org/ApisixUpstream:fpms-nt/auth-apiserver-ss
  creationTimestamp: "2025-04-17T06:52:15Z"
  generation: 1
  name: auth-apiserver-ss
  namespace: fpms-nt
  resourceVersion: "142996228"
  uid: d72751e7-8cb9-4dc7-92db-79cc42cc6aa4
spec:
  loadbalancer:
    type: least_conn
  scheme: grpc
status:
  conditions:
  - message: not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable

Environment

• APISIX version (run apisix version): 3.12.0
• Operating system (run uname -a): Linux apisix-588d5777db-t6ncn 6.1.134-152.225.amzn2023.x86_64 change: added doc of how to load plugin. #1 SMP PREEMPT_DYNAMIC Wed May 7 09:10:59 UTC 2025 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.27.1.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): {"etcdserver":"3.5.10","etcdcluster":"3.5.0"}
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

Hi @Adrianckwai, are there any error logs in apisix? Specifically "proxy retry timeout" related messages.

[Adrianckwai]

Hi @Baoyuantop , thanks for replying to this issue. I've search on our APISIX logs matching with our DEADLINE_EXCEEDED error.

We can't seems to find any error relate to "proxy retry timeout". However, I did search some error logs on APISIX. Not sure if this helps, error logs as below:-

2025-06-05 14:51:11.033 | 2025/06/05 06:51:11 [error] 75#75: *12460472 connect() failed (111: Connection refused) while connecting to upstream, client: 10.215.31.42 |  
-- | -- | --
  |   | 2025-06-05 14:51:11.103 | 2025/06/05 06:51:11 [error] 53#53: *12626652 connect() failed (111: Connection refused) while connecting to upstream, client: 10.215.33.86 |  
  |   | 2025-06-05 14:51:11.147 | 2025/06/05 06:51:11 [error] 57#57: *12461929 connect() failed (111: Connection refused) while connecting to upstream, client: 10.215.26.201 |  
  |   | 2025-06-05 14:52:52.727 | 2025/06/05 06:52:52 [error] 68#68: *12689362 upstream prematurely closed connection while reading response header from upstream, client: 10.215.42.57 |  
  |   | 2025-06-05 14:53:03.034 | 2025/06/05 06:53:03 [error] 53#53: *12468474 upstream prematurely closed connection while reading response header from upstream, client: 10.215.27.104 |  
  |   | 2025-06-05 14:53:18.819 | 2025/06/05 06:53:18 [error] 67#67: *12666852 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: z9usp48cmgk5y97vn33opa32jyqo5qfb.bybit-aws.com, context: ssl_client_hello_by_lua*, client: 10.215.50.73, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 14:55:41.520 | 2025/06/05 06:55:41 [error] 53#53: *12649001 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: stream.bybit.com, context: ssl_client_hello_by_lua*, client: 10.215.50.73, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 14:55:57.368 | 2025/06/05 06:55:57 [error] 64#64: *12493037 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:57.678 | 2025/06/05 06:55:57 [error] 72#72: *12488619 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:57.731 | 2025/06/05 06:55:57 [error] 49#49: *12476802 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:57.777 | 2025/06/05 06:55:57 [error] 59#59: *12443007 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:58.069 | 2025/06/05 06:55:58 [error] 53#53: *12648234 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:58.457 | 2025/06/05 06:55:58 [error] 75#75: *12517358 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:58.550 | 2025/06/05 06:55:58 [error] 51#51: *12467750 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:55:58.635 | 2025/06/05 06:55:58 [error] 68#68: *12419561 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.190 | 2025/06/05 06:56:03 [error] 63#63: *12483526 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.221 | 2025/06/05 06:56:03 [error] 72#72: *12488619 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.224 | 2025/06/05 06:56:03 [error] 51#51: *12467529 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.224 | 2025/06/05 06:56:03 [error] 49#49: *12706658 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.299 | 2025/06/05 06:56:03 [error] 72#72: *12434231 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.318 | 2025/06/05 06:56:03 [error] 67#67: *12709506 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:03.318 | 2025/06/05 06:56:03 [error] 67#67: *12480876 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.851 | 2025/06/05 06:56:22 [error] 53#53: *12478384 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.885 | 2025/06/05 06:56:22 [error] 71#71: *12422350 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.886 | 2025/06/05 06:56:22 [error] 49#49: *12489321 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.886 | 2025/06/05 06:56:22 [error] 67#67: *12483589 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.932 | 2025/06/05 06:56:22 [error] 72#72: *12636799 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.980 | 2025/06/05 06:56:22 [error] 49#49: *12707263 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:22.980 | 2025/06/05 06:56:22 [error] 72#72: *12690935 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:56:40.412 | 2025/06/05 06:56:40 [error] 62#62: *12436579 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: z9usp48cmgk5y97vn33opa32jyqo5qfb.bybit-aws.com, context: ssl_client_hello_by_lua*, client: 10.215.48.34, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 14:57:07.883 | 2025/06/05 06:57:07 [error] 58#58: *12438757 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: stream.bybit.com, context: ssl_client_hello_by_lua*, client: 10.215.48.34, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 14:57:12.472 | 2025/06/05 06:57:12 [error] 57#57: *12582484 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:57:12.956 | 2025/06/05 06:57:12 [error] 57#57: *12450145 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:57:12.957 | 2025/06/05 06:57:12 [error] 69#69: *12652937 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:57:13.137 | 2025/06/05 06:57:13 [error] 72#72: *12434231 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:57:13.457 | 2025/06/05 06:57:13 [error] 54#54: *12494668 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:57:13.935 | 2025/06/05 06:57:13 [error] 50#50: *12446714 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:57:14.136 | 2025/06/05 06:57:14 [error] 59#59: *12482832 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.292 | 2025/06/05 06:58:27 [error] 57#57: *12646715 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.296 | 2025/06/05 06:58:27 [error] 61#61: *12680460 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.310 | 2025/06/05 06:58:27 [error] 55#55: *12662705 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.310 | 2025/06/05 06:58:27 [error] 53#53: *12491037 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.347 | 2025/06/05 06:58:27 [error] 61#61: *12650383 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.393 | 2025/06/05 06:58:27 [error] 58#58: *12692867 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:27.395 | 2025/06/05 06:58:27 [error] 54#54: *12498682 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:58:29.748 | 2025/06/05 06:58:29 [error] 57#57: *12497831 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: z9usp48cmgk5y97vn33opa32jyqo5qfb.bybit-aws.com, context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 14:59:29.264 | 2025/06/05 06:59:29 [error] 49#49: *12446753 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:59:29.313 | 2025/06/05 06:59:29 [error] 54#54: *12687442 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:59:29.313 | 2025/06/05 06:59:29 [error] 72#72: *12658708 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:59:29.736 | 2025/06/05 06:59:29 [error] 53#53: *12495950 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:59:30.093 | 2025/06/05 06:59:30 [error] 52#52: *12677652 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 14:59:30.138 | 2025/06/05 06:59:30 [error] 63#63: *12596496 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:00:06.578 | 2025/06/05 07:00:06 [error] 49#49: *12652452 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:00:06.744 | 2025/06/05 07:00:06 [error] 50#50: *12461190 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:00:07.018 | 2025/06/05 07:00:07 [error] 59#59: *12535662 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:00:07.393 | 2025/06/05 07:00:07 [error] 52#52: *12666678 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:00:07.448 | 2025/06/05 07:00:07 [error] 65#65: *12487897 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:00:07.511 | 2025/06/05 07:00:07 [error] 50#50: *12596157 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:02:02.355 | 2025/06/05 07:02:02 [error] 61#61: *12553287 [lua] init.lua:185: http_ssl_client_hello_phase(): failed to find SNI: please check if the client requests via IP or uses an outdated protocol. If you need to report an issue, provide a packet capture file of the TLS handshake., context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:02:03.077 | 2025/06/05 07:02:03 [error] 73#73: *12524043 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: 13.214.46.213, context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:02:03.992 | 2025/06/05 07:02:03 [error] 66#66: *12496096 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: 13.214.46.213, context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:03:06.301 | 2025/06/05 07:03:06 [error] 58#58: *12511929 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: stream.bybit.com, context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:04:38.944 | 2025/06/05 07:04:38 [error] 60#60: *12464202 [lua] grpc-web.lua:124: phase_func(): request method: `HEAD` invalid, client: 10.215.49.189, server: _, request: "HEAD /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:04:42.164 | 2025/06/05 07:04:42 [error] 72#72: *12477716 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:04:42.166 | 2025/06/05 07:04:42 [error] 60#60: *12464202 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:04:42.227 | 2025/06/05 07:04:42 [error] 52#52: *12541490 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:04:43.595 | 2025/06/05 07:04:43 [error] 73#73: *12554009 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:04:44.466 | 2025/06/05 07:04:44 [error] 72#72: *12526505 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:04:44.539 | 2025/06/05 07:04:44 [error] 61#61: *12555594 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:05:39.954 | 2025/06/05 07:05:39 [error] 50#50: *12529027 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: socket.3932719.com, context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:06:48.775 | 2025/06/05 07:06:48 [error] 49#49: *12532260 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:48.778 | 2025/06/05 07:06:48 [error] 54#54: *12518036 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:48.901 | 2025/06/05 07:06:48 [error] 51#51: *12528388 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:49.104 | 2025/06/05 07:06:49 [error] 51#51: *12528388 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:49.105 | 2025/06/05 07:06:49 [error] 49#49: *12701048 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:51.643 | 2025/06/05 07:06:51 [error] 59#59: *12550721 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:51.842 | 2025/06/05 07:06:51 [error] 55#55: *12717579 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:51.844 | 2025/06/05 07:06:51 [error] 49#49: *12522283 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:51.964 | 2025/06/05 07:06:51 [error] 51#51: *12516191 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:52.045 | 2025/06/05 07:06:52 [error] 67#67: *12478759 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:59.591 | 2025/06/05 07:06:59 [error] 67#67: *12523705 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:59.593 | 2025/06/05 07:06:59 [error] 50#50: *12546587 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:59.593 | 2025/06/05 07:06:59 [error] 59#59: *12492387 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:59.981 | 2025/06/05 07:06:59 [error] 63#63: *12532303 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:06:59.984 | 2025/06/05 07:06:59 [error] 57#57: *12487741 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:00.090 | 2025/06/05 07:07:00 [error] 50#50: *12546781 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:13.637 | 2025/06/05 07:07:13 [error] 57#57: *12519387 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:13.769 | 2025/06/05 07:07:13 [error] 71#71: *12512627 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:13.771 | 2025/06/05 07:07:13 [error] 64#64: *12526004 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:13.871 | 2025/06/05 07:07:13 [error] 49#49: *12527408 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:13.955 | 2025/06/05 07:07:13 [error] 63#63: *12523115 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.48.34, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:14.043 | 2025/06/05 07:07:14 [error] 54#54: *12696922 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.50.73, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:33.366 | 2025/06/05 07:07:33 [error] 75#75: *12520057 [lua] grpc-web.lua:124: phase_func(): request method: `GET` invalid, client: 10.215.49.189, server: _, request: "GET /cms/CMS.FrontendService/GetPageResource HTTP/1.1", host: "fpms-nt.fpms10.me" |  
  |   | 2025-06-05 15:07:43.166 | 2025/06/05 07:07:43 [error] 62#62: *12540096 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: merryblue.llc, context: ssl_client_hello_by_lua*, client: 10.215.49.189, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:09:12.813 | 2025/06/05 07:09:12 [error] 60#60: *12731891 [lua] init.lua:205: http_ssl_client_hello_phase(): failed to match any SSL certificate by SNI: z9usp48cmgk5y97vn33opa32jyqo5qfb.bybit-aws.com, context: ssl_client_hello_by_lua*, client: 10.215.50.73, server: 0.0.0.0:9443 |  
  |   | 2025-06-05 15:09:58.817 | 10.215.49.189 - - [05/Jun/2025:07:09:58 +0000] [1749107398.000] Service: [-] Uri:[/Public/css/errorCss.css] [Request_time: 0.000] [Upstream_response_time: -] [Upstream_status: -] fpms-nt.fpms10.me "GET /Public/css/errorCss.css HTTP/1.1" 404 47 0.000 "-" "Mozilla/5.0 (Linux; Android 15; I2201 Build/AP3A.240905.015.A2; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/132.0.6834.90 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/496.0.0.45.65;IABMV/1;] FBNV/5" CF-"Xy1-kMK-JFJdDh5Cp5gAVK1a7_27CVCdepk6r3pQ2aXxIJDnJGABuQ==" TIP-"101.36.112.147" TraceId-"-" "-" "- - - "http://fpms-nt.fpms10.me" "Body: -"

Thanks in advanced!

[Baoyuantop]

As you can see from the logs:

• connect() failed (111: Connection refused) error, indicating that APISIX was unable to connect to the backend service.
• upstream prematurely closed connection error, indicating that the backend service closed the connection abnormally while processing the request.

My personal deduction from this information:

1. The backend gRPC service may be intermittently unavailable, are there any checks?
2. Recommended to turn on health check for upstream to detect and remove unhealthy nodes in a timely manner.

[Baoyuantop]

Hi @Adrianckwai, any update?

[github-actions]

Due to lack of the reporter's response this issue has been labeled with "no response". It will be close in 3 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.