ExternalService handler retry mechanism failing with 500

[DiogoFKT]

In the third retry attempt the HTTPClient gets recreated with Connection header defined as close however we see in logs failure to manipulate this restricted header (logs below) so I believe this static JDK property is not taking effect

Logs

2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.correlation.CorrelationHandler:101 handleRequest - Request Headers: 'Accept=text/html, image/gif, image/jpeg, *; q=.2, /; q=.2
X-Correlation-Id=AZ14VnjLdCSC7ZOQXbdZlw
Cache-Control=no-cache
Pragma=no-cache
User-Agent=Java/1.8.0_461
Connection=keep-alive
CorID=
X-Forwarded-For=
Content-Type=application/x-www-form-urlencoded
Content-Length=191
Host=stg-okta-oauth..com
'
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.correlation.CorrelationHandler:104 handleRequest - CorrelationHandler.handleRequest ends.
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.r.m.PathPrefixServiceHandler:71 handleRequest - PathPrefixServiceHandler.handleRequest starts.
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.config.Config$AbstractConfigImpl:641 loadJsonMapConfigWithSpecificConfigLoader - Trying to load pathPrefixService with extension yaml, yml or json by using default loading method.
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.config.Config$AbstractConfigImpl:498 getConfigStream - Unable to load config from externalized folder for pathPrefixService.yml in F:\L4J\eadp-dmz-gateway\config
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.config.Config$AbstractConfigImpl:511 getConfigStream - Trying to load config from classpath directory for file pathPrefixService.yml
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.config.Config$AbstractConfigImpl:523 getConfigStream - Config loaded from default folder for pathPrefixService.yml
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.handler.config.HandlerUtils:39 findServiceEntry - findServiceEntry for /v0/oauth2//v1/token
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.handler.config.HandlerUtils:57 findServiceEntry - serviceEntry not found!
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.m.PathPrefixServiceHandler:100 pathPrefixService - serviceEntry is null and endpoint is set to = 'unknown@post'
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.handler.AuditAttachmentUtil:18 populateAuditAttachmentField - AuditInfo is null, creating a new one and inserting the key-value pair 'endpoint:unknown@post'
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.r.m.PathPrefixServiceHandler:73 handleRequest - PathPrefixServiceHandler.handleRequest ends.
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG com.networknt.audit.AuditHandler:120 handleRequest - AuditHandler.handleRequest starts.
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE com.networknt.audit.AuditHandler:234 auditHeader - header name = X-Correlation-Id header value = AZ14VnjLdCSC7ZOQXbdZlw
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE com.networknt.audit.AuditHandler:234 auditHeader - header name = X-Traceability-Id header value = null
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE com.networknt.audit.AuditHandler:234 auditHeader - header name = caller_id header value = null
2026-04-10T13:00:26.187-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR com.networknt.audit.AuditHandler:286 auditRequest - Unknown audit key client_id in audit.yml
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR com.networknt.audit.AuditHandler:286 auditRequest - Unknown audit key user_id in audit.yml
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR com.networknt.audit.AuditHandler:286 auditRequest - Unknown audit key scope_client_id in audit.yml
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR com.networknt.audit.AuditHandler:286 auditRequest - Unknown audit key endpoint in audit.yml
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR com.networknt.audit.AuditHandler:286 auditRequest - Unknown audit key serviceId in audit.yml
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG com.networknt.audit.AuditHandler:201 handleRequest - AuditHandler.handleRequest ends.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.security.UnifiedSecurityHandler:52 handleRequest - UnifiedSecurityHandler.handleRequest starts.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.security.UnifiedSecurityHandler:87 verifyUnifiedSecurity - Skip request path base on anonymousPrefixes for /v0/oauth2//v1/token
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.security.UnifiedSecurityHandler:70 handleRequest - UnifiedSecurityHandler.handleRequest ends.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG com.networknt.header.HeaderHandler:73 handleRequest - HeaderHandler.handleRequest starts.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE com.networknt.header.HeaderHandler:107 handleRequest - found with requestPath = /v0/oauth2//v1/token prefix = /v0/oauth2
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE com.networknt.header.HeaderHandler:119 lambda$handleRequest$4 - remove request header transfer-encoding
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE com.networknt.header.HeaderHandler:119 lambda$handleRequest$4 - remove request header Expect
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG com.networknt.header.HeaderHandler:164 handleRequest - HeaderHandler.handleRequest ends.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.RequestInterceptorInjectionHandler:76 handleRequest - injectionContentRequired = true appliedBodyInjectionPathPrefix = true method = POST requestComplete = false requiresContinueResponse = false
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.RequestInterceptorInjectionHandler:79 handleRequest - Trying to read body
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.RequestInterceptorInjectionHandler:258 saveBufferAndResetUndertowConnector - saveBufferAndResetUndertowConnector is called.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.RequestInterceptorInjectionHandler:268 updateContentLength - original content length in request headers = 191
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.RequestInterceptorInjectionHandler:274 updateContentLength - update content length in request headers = 191
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.RequestTransformerInterceptor:69 handleRequest - RequestTransformerInterceptor.handleRequest starts.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.RequestTransformerInterceptor:273 handleRequest - RequestTransformerInterceptor.handleRequest ends.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.body.RequestBodyInterceptor:75 handleRequest - RequestBodyInterceptor.handleRequest starts.
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.body.RequestBodyInterceptor:84 handleRequest - request body exists in exchange attachment = true
2026-04-10T13:00:26.188-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.body.RequestBodyInterceptor:89 handleRequest - Attach request body requirement is met and the byte buffer pool exists.
2026-04-10T13:00:26.222-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.body.RequestBodyInterceptor:99 handleRequest - contentType = application/x-www-form-urlencoded request body = grant_type=client_credentials&client_id=&client_secret=&scope=
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.body.RequestBodyInterceptor:107 handleRequest - Request body was attached to exchange
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.body.RequestBodyInterceptor:114 handleRequest - RequestBodyInterceptor.handleRequest ends.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.RequestInterceptorInjectionHandler:132 handleRequest - Exchange response started status = false
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.token.limit.TokenLimitHandler:119 handleRequest - TokenLimitHandler.handleRequest starts.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:127 handleRequest - client address
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:132 handleRequest - request path /v0/oauth2//v1/token matches with one of the 6 patterns.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:135 handleRequest - requestBodyString = grant_type=client_credentials&client_id=&client_secret=&scope=
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:146 handleRequest - No authorization header found. Will obtain credentials from body.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:173 handleRequest - client credentials key = ::
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:195 handleRequest - client is configured as Legacy, bypass the token limit.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.token.limit.TokenLimitHandler:205 handleRequest - legacy client cache key :: has NO token cached, calling next handler.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.proxy.ExternalServiceHandler:96 handleRequest - ExternalServiceHandler.handleRequest starts.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:106 handleRequest - original requestPath = /v0/oauth2//v1/token
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:128 handleRequest - found matched pathHostMapping = /v0/oauth2/ host = https://nonprod1-okta-oauth..com
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:136 handleRequest - endpoint = /v0/oauth2/@post
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:145 handleRequest - rewritten requestPath = /oauth2//v1/token
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.handler.AuditAttachmentUtil:23 populateAuditAttachmentField - AuditInfo is not null, inserting the key-value pair 'endpoint:/v0/oauth2/@post'
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.handler.AuditAttachmentUtil:26 populateAuditAttachmentField - AuditInfo already contains the field 'endpoint'! Replacing the value 'unknown@post' with '/v0/oauth2/@post'.
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:160 handleRequest - External Service Request Info: host = 'https://nonprod1-okta-oauth..com', method = 'POST', requestPath = '/oauth2//v1/token', queryString = ''
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = Accept value = text/html, image/gif, image/jpeg, *; q=.2, /; q=.2
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = X-Correlation-Id value = AZ14VnjLdCSC7ZOQXbdZlw
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = Cache-Control value = no-cache
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = Pragma value = no-cache
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = User-Agent value = Java/1.8.0_461
2026-04-10T13:00:26.223-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = CorID value = 78566F66AC1D01AF9C6C9E27AD928D57,1:1,0,0,,,AgAAA6NIQgAAAAFGAAAAAQAAABFqYXZhLnV0aWwuSGFzaE1hcAAAAAVIQgAAAAJGAAAAAgAAABBqYXZhLmxhbmcuU3RyaW5nAAxJc0F1dG9UcmFjZWRIQgAAAANFAAAAAgCEMTpOZXcgdHJhbnNhY3Rpb24gcGF0aCBkaXNjb3ZlcmVkOkJhY2tlbmRzfGdyc2RvY3VtZW50aW5kZXggKGluc3RhbmNlIGhyc3EwMXMpIG9uIHNxaHJzcTAxcy5zdW5saWZlY29ycC5jb20tNTE0NDYgKE1TIFNRTCBTZXJ2ZXIgREIpSEIAAAAERQAAAAIAClR4blRyYWNlSWRIQgAAAAVFAAAAAgAkNzc5RUQxMzZBQzFEMDFBRjlDNkM5RTI3RDkxQTQ4NjQ1NDk4SEIAAAAGRQAAAAIAE0NhbGxlciBDb21wb25lbnQgSURIQgAAAAdFAAAAAgAQNkY2NjBDQTM5QjkyMDAwN0hCAAAACEUAAAACAA9DYWxsZXJUaW1lc3RhbXBIQgAAAAlFAAAAAgANMTc3NTg0MDQyNjE4NUhCAAAACkUAAAACABFVcHN0cmVhbUdVSURDYWNoZUhCAAAAC0YAAAADAAAAE2phdmEudXRpbC5BcnJheUxpc3QAAAAKSEIAAAAMRQAAAAIAIDBGQ0ZCQjRGNjYxRDQ4REE4RkIwMjg3MDAzODA0NzM1SEIAAAANRQAAAAIAIDc4NTE1MTNFQUMxRDAxQUY5QzZDOUUyNzlCRTZBMkMwSEIAAAAORQAAAAIAIDc4NTE1MDk5MEE4MDkzMTdCRTE1QjlFNkU5MTc0RUQ4SEIAAAAPRQAAAAIAIDc4NERERDE3QUMxRDAxQUY5QzZDOUUyN0NCRUU2NjlGSEIAAAAQRQAAAAIAIDc4NEMyOEQ2QUMxRDAxQUY5QzZDOUUyNzIzMzAxQTJGSEIAAAARRQAAAAIAIDc4NEMyN0VBMEE4MENFNjFFODhCMzk0M0M2M0NBRTRCSEIAAAASRQAAAAIAIDczQTQ0RTQyOEMwRjQ0MUFCQjY0RkVGQkFFMjIzODMzSEIAAAATRQAAAAIAIDc4NDZGNkRDQUMxRDAxQUY5QzZDOUUyN0E5NjdEMDAySEIAAAAURQAAAAIAIDc4NDZGNjAxMEE4MDFGRjU3NEQ2QkVFNDBGRURCNEIxSEIAAAAVRQAAAAIAIDc4NDM5QUY3QUMxRDAxQUY5QzZDOUUyNzlCOTgzRkE2
2026-04-10T13:00:26.224-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = X-Forwarded-For value =
2026-04-10T13:00:26.224-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:313 copyHeaders - Copy header key = Content-Type value = application/x-www-form-urlencoded
2026-04-10T13:00:26.224-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.proxy.ExternalServiceHandler:422 handleBufferedRequestBody - request body = grant_type=client_credentials&client_id=&client_secret=&scope=
2026-04-10T13:00:27.620-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw WARN c.n.proxy.ExternalServiceHandler:251 handleRequest - Attempt 1 failed (HTTP/1.1 header parser received no bytes). Retrying...
2026-04-10T13:00:27.620-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw INFO c.n.proxy.ExternalServiceHandler:236 handleRequest - Attempt 1 failed. Retrying with 'Connection: close' to force fresh connection.
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR c.n.exception.ExceptionHandler:83 handleRequest - ExceptionHandler error processing - path: /v0/oauth2//v1/token
java.lang.IllegalArgumentException: restricted header name: "Connection"
at java.net.http/jdk.internal.net.http.common.Utils.newIAE(Utils.java:350)
at java.net.http/jdk.internal.net.http.HttpRequestBuilderImpl.checkNameAndValue(HttpRequestBuilderImpl.java:110)
at java.net.http/jdk.internal.net.http.HttpRequestBuilderImpl.header(HttpRequestBuilderImpl.java:126)
at java.net.http/jdk.internal.net.http.HttpRequestBuilderImpl.header(HttpRequestBuilderImpl.java:43)
at com.networknt.proxy.ExternalServiceHandler.handleRequest(ExternalServiceHandler.java:239)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.token.limit.TokenLimitHandler.handleRequest(TokenLimitHandler.java:207)
at com.networknt.handler.Handler.next(Handler.java:251)
at com.networknt.handler.RequestInterceptorInjectionHandler.handleRequest(RequestInterceptorInjectionHandler.java:135)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.handler.ResponseInterceptorInjectionHandler.handleRequest(ResponseInterceptorInjectionHandler.java:109)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.header.HeaderHandler.handleRequest(HeaderHandler.java:165)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.security.UnifiedSecurityHandler.handleRequest(UnifiedSecurityHandler.java:71)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.audit.AuditHandler.next(AuditHandler.java:461)
at com.networknt.audit.AuditHandler.handleRequest(AuditHandler.java:202)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.metrics.APMMetricsHandler.handleRequest(APMMetricsHandler.java:70)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.router.middleware.PathPrefixServiceHandler.handleRequest(PathPrefixServiceHandler.java:74)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.correlation.CorrelationHandler.handleRequest(CorrelationHandler.java:106)
at com.networknt.handler.Handler.next(Handler.java:227)
at com.networknt.handler.Handler.next(Handler.java:253)
at com.networknt.exception.ExceptionHandler.handleRequest(ExceptionHandler.java:81)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:418)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:938)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2691)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2670)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1615)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
at java.base/java.lang.Thread.run(Thread.java:1583)
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw ERROR c.networknt.handler.LightHttpHandler:121 setExchangeStatus - {"statusCode":500,"code":"ERR10010","message":"RUNTIME_EXCEPTION","description":"Unexpected runtime exception","severity":"ERROR"}
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.networknt.handler.LightHttpHandler:125 setExchangeStatus - java.base/java.lang.Thread.getStackTrace(Thread.java:2451)
com.networknt.handler.LightHttpHandler.setExchangeStatus(LightHttpHandler.java:118)
com.networknt.handler.LightHttpHandler.setExchangeStatus(LightHttpHandler.java:73)
com.networknt.exception.ExceptionHandler.handleRequest(ExceptionHandler.java:95)
io.undertow.server.Connectors.executeRootHandler(Connectors.java:418)
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:938)
org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2691)
org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2670)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1615)
org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
java.base/java.lang.Thread.run(Thread.java:1583)
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.ResponseInterceptorInjectionHandler:127 requiresContentSinkConduit - requiresContentSinkConduit: requiredContent true, pathPrefix true and isNotCompressed true
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.ResponseInterceptorInjectionHandler:102 lambda$handleRequest$1 - created a ModifiableContentSinkConduit instance com.networknt.handler.conduit.ModifiableContentSinkConduit@42faafc2
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:120 terminateWrites - terminating writes with interceptors length = 3
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.h.c.ModifiableContentSinkConduit:127 terminateWrites - Executing interceptor class com.networknt.restrans.ResponseTransformerInterceptor
2026-04-10T13:00:27.621-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.ResponseTransformerInterceptor:92 handleRequest - ResponseTransformerInterceptor.handleRequest starts.
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.ResponseTransformerInterceptor:200 handleRequest - ResponseTransformerInterceptor.handleRequest ends.
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.h.c.ModifiableContentSinkConduit:127 terminateWrites - Executing interceptor class com.networknt.restrans.ResponseCacheInterceptor
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.ResponseCacheInterceptor:72 handleRequest - ResponseCacheInterceptor.handleRequest starts.
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.r.ResponseCacheInterceptor:75 handleRequest - Skip on error code 500. ResponseCacheInterceptor.handleRequest ends.
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.h.c.ModifiableContentSinkConduit:127 terminateWrites - Executing interceptor class com.networknt.body.ResponseBodyInterceptor
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.body.ResponseBodyInterceptor:63 handleRequest - ResponseBodyInterceptor.handleRequest starts.
2026-04-10T13:00:27.622-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.body.ResponseBodyInterceptor:74 handleRequest - Attach response body requirement is met and the byte buffer pool exists.
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.body.ResponseBodyInterceptor:81 handleRequest - contentType = application/json response body = {"statusCode":500,"code":"ERR10010","message":"RUNTIME_EXCEPTION","description":"Unexpected runtime exception","severity":"ERROR"}
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.body.ResponseBodyInterceptor:93 handleRequest - ResponseBodyInterceptor.handleRequest ends.
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:143 terminateWrites - Next conduit is: io.undertow.server.protocol.http.ServerFixedLengthStreamSinkConduit
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:311 updateContentLength - PooledByteBuffer array added up length = 130
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:321 updateContentLength - The next conduit is ServerFixedLengthStreamSinkConduit and reset the length.
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:339 updateContentLength - reset ServerFixedLengthStreamSinkConduit length = 130
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:178 http1Write - buffer position 0 and buffer limit 130
2026-04-10T13:00:27.655-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:183 http1Write - Before write buffer position: 0
2026-04-10T13:00:27.656-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.h.c.ModifiableContentSinkConduit:188 http1Write - After write buffer position: 130
2026-04-10T13:00:27.656-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.metrics.AbstractMetricsHandler:157 setOrUnknownTag - endpoint = /v0/oauth2/@post
2026-04-10T13:00:27.656-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.metrics.AbstractMetricsHandler:157 setOrUnknownTag - clientId =
2026-04-10T13:00:27.656-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw TRACE c.n.m.AbstractMetricsHandler$MetricsExchangeCompletionListener:331 exchangeEvent - metricName = response_time{endpoint=/v0/oauth2/@post, clientId=, port=8443, host=, api=<SERVICE_ID>, env=stg, addr=}, commonTags = {"port":"8443","host":"","api":"<SERVICE_ID>","env":"stg","addr":""} tags = {"endpoint":"/v0/oauth2/@post","clientId":""}
2026-04-10T13:00:27.656-0400 [XNIO-1 task-4] <SERVICE_ID> AZ14VnjLdCSC7ZOQXbdZlw DEBUG c.n.exception.ExceptionHandler:120 handleRequest - ExceptionHandler completed - path: /v0/oauth2//v1/token

[stevehu]

Yes. The root cause is the Connection: close retry path, not the downstream call itself.

In light-4j/ingress-proxy/src/main/java/com/networknt/proxy/ExternalServiceHandler.java:40, the code tries to enable restricted headers with:

System.setProperty("jdk.httpclient.allowRestrictedHeaders", "host,connection");

and later on retry it does this in light-4j/ingress-proxy/src/main/java/com/networknt/proxy/ExternalServiceHandler.java:228:

HttpRequest.newBuilder(request, (name, value) -> true)
.header("Connection", "close")
.build();

The failure in issue 2740:

java.lang.IllegalArgumentException: restricted header name: "Connection"

means the JDK HttpClient did not honor that property. The important part is that Oracle’s java.net.http docs explicitly say there is no guarantee this property can be set programmatically with System.setProperty(). It is intended to be
provided as a JVM/system networking property at startup or in conf/net.properties, not reliably changed at runtime. Source:

• Oracle java.net.http module docs: jdk.httpclient.allowRestrictedHeaders and “No guarantee is provided that property values can be set programatically with System.setProperty()”
  https://docs.oracle.com/en/java/javase/24/docs/api/java.net.http/module-summary.html
• OpenJDK CSR for the property
  https://bugs.openjdk.org/browse/JDK-8213696

So the actual sequence is:

1. First attempt fails with HTTP/1.1 header parser received no bytes
2. Retry code tries to force a fresh connection with Connection: close
3. JDK rejects that restricted header
4. Handler throws IllegalArgumentException
5. Request returns 500

A few concrete points:

• The design doc is directionally fine, but the implementation depends on a JDK behavior that is only safe when the property is set before the HTTP client classes initialize.
• The current static block is not a reliable fix.
• The issue body mentions “third retry recreates HttpClient”, but the current code in light-4j does not recreate the client anymore. The current failure is earlier than that.

How to fix it

1. Operational fix:
   Start the JVM with:

   -Djdk.httpclient.allowRestrictedHeaders=connection,host

   or set it in JDK conf/net.properties.

2. Code fix:
   Remove the misleading static block in light-4j/ingress-proxy/src/main/java/com/networknt/proxy/ExternalServiceHandler.java:40. It suggests the handler can enable this dynamically, which is not reliable.

3. Defensive fix:
   Wrap the retry-request rebuild so failure to set Connection does not become a 500. If .header("Connection", "close") throws, log a clear warning and fall back to a normal retry instead of crashing the handler.

4. Documentation fix:
   Update the retry design and handler config docs to state that Connection: close retry requires JVM startup property configuration.

What I would recommend

• Short term: set the JVM flag in the test environment and verify the issue disappears.
• Medium term: patch ExternalServiceHandler to fail gracefully if restricted headers are unavailable.
• Long term: keep the shared client approach; do not go back to recreating HttpClient per retry unless you are deliberately accepting the resource cost described in the design doc.

[stevehu]

Root cause was exactly what the issue suggested: the retry code assumes System.setProperty("jdk.httpclient.allowRestrictedHeaders", ...) enables Connection: close, but the JDK does not reliably honor that property when set programmatically
after startup. When the first downstream call fails, the retry builder throws IllegalArgumentException: restricted header name: "Connection", and that local exception becomes the 500.

The fix does three things:

• removes the misleading static System.setProperty(...) block
• moves retry request construction into buildRetryRequest(...)
• catches restricted-header rejection and falls back to a normal retry request instead of throwing

So after this change:

• if the JVM was started with -Djdk.httpclient.allowRestrictedHeaders=connection,host, retry can still use Connection: close
• if not, retries still proceed, just without the forced fresh-connection hint
• the handler no longer fails with ERR10010 because of local retry-header construction