[Geoserver-devel] [JIRA] (GEOS-10856) geoserver monitor plugin - scaling troubles

GeoServer GeoServer Developer
1

Hans Yperman created an issue

GeoServer / ImprovementGEOS-10856

geoserver monitor plugin - scaling troubles

Issue Type:

ImprovementImprovement

Affects Versions:

2.22.1

Assignee:

Unassigned

Components:

Monitoring

Created:

07/Feb/23 2:08 PM

Environment:

When a request is done, it is sent to a post processor thread. We found this thread unable to keep up with incoming requests.

Specifically, ~100 reqs/second come in, but a failed DNS reverse lookup takes multiple seconds. After a day of runtime, I found ~300K PostProcessTask objects in the heap. This has multiple interesting consequences.

  1. Memory leaks, of course. As the PostProcessTask contains the HttpServletRequest / Reponse, and these are big objects, we quickly talk about multiple gigabytes.

  2. After a HttpServletRequest/Response is serviced, tomcat will reuse some backing buffers for new requests. But as these objects are still held on by the post processor, fragments of unrelated requests can turn up in the post processing.

  3. The post process executor service is hardcoded to 2 threads, which is simply not enough. This should be configurable.

  4. ReverseDNSPostProcessor has a reverseLookupCache, but AFAIK it is only read, never written. Apart from that, it overlaps with the JDK cache and has no TTL. So it seems useless. I propose replacing it with a TTL-based cache, caching only the lookup FAILURES, as these are not cached by the JDK.

As a workaround, I configured it to not do reverse DNS lookups in monitoring.properties

Priority:

MediumMedium

Reporter:

Hans Yperman

Add Comment

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100214-sha1:c33f0bb)

Atlassian logo