MultithreadedHttpClient leaks pooled connections on non-200 responses, cascaded WMS store blocks permanently (cannot create Jira issue)

GeoServer GeoServer User
,
1

I tried to report this in the GEOT Jira, but my new account has no permission
to create issues. Could a maintainer grant me access (Jira user: [gebruikersnaam])
or create the ticket on my behalf? Full details below.

Environment:

  • GeoServer [versie], gt-http-commons [versie]
  • httpclient5 5.4.4, httpcore5 5.4.3
  • Eclipse Temurin 21.0.12.1, Apache Tomcat 11, Windows Server 2019

Problem:
With a cascaded WMS store using HTTP connection pooling, every response from the
remote server with a status other than 200 permanently consumes one connection
from the pool. Once max connections is reached, every subsequent request to that
store waits 180 s (default connection request timeout) and fails with
ConnectionRequestTimeoutException. The store never recovers on its own; only
saving the store or restarting GeoServer helps. In one occurrence the store was
unavailable for more than 26 hours. Blocked requests hold Tomcat threads, which
can eventually affect all services.

Root cause:
In MultithreadedHttpClient.get(URL, Map) and post(…), an IOException is thrown
for non-200 responses without closing the response, so the leased connection is
never returned to the PoolingHttpClientConnectionManager:

response = executeMethod(getMethod);
if (200 != response.getStatusCode()) {
    throw new IOException("Server returned HTTP error code " + ...);
}

Proposed fix (get() and post()):

if (200 != response.getStatusCode()) {
    response.dispose();
    throw new IOException("Server returned HTTP error code " + ...);
}

Evidence:

  • Affected store: max connections = 12, read timeout 60 s, connect timeout 30 s.
    The 180 s wait is unrelated to the store settings.
  • 09:11:28 to 09:12:19: the remote server returned HTTP 500 exactly twelve
    times. After the twelfth error, no further requests reached the remote server.
  • 09:15:12: first ConnectionRequestTimeoutException:
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint
    at org.geotools.http.commons.MultithreadedHttpClient.executeMethod(MultithreadedHttpClient.java:200)
    at org.geotools.http.commons.MultithreadedHttpClient.get(MultithreadedHttpClient.java:238)
    at org.geotools.ows.wms.map.WMSCoverageReader.getMap(WMSCoverageReader.java:329)
  • Thread dump at 09:18:38: 209 threads, 93 of them parked in StrictConnPool
    lease (acquireEndpoint), zero threads reading from a socket. The connections
    are leased but unused, i.e. leaked.

Workaround:
Disable “Use HTTP connection pooling” on the cascaded WMS/WMTS store.
Applied to 26 stores in production.

Related reports with the same symptom:

Full log excerpt and thread dump are available on request.