[Geoserver-users] hdf5 high native memory usage

GeoServer GeoServer User
1

We found a bug where geoserver grows basically unbounded (32GB+) when using hdf5, while the java heap and native heaps stay small (2GB and 300MB resp). The bug still exists in the current version of netcdf-java. The main symptom is unfortunately not noticeable with JVM tooling: a lot of mmapped segments of around 64K each appear when you look on linux with pmap.

I hacked a fix in the cdm-4.6.15.jar library. A test scenario that crashed the server in minutes now runs for an hour straight without growing the process RSS size.

I logged a bug in both projects (the geoserver one is pretty minimal).

https://github.com/Unidata/netcdf-java/issues/1138

https://osgeo-org.atlassian.net/browse/GEOS-10831

image001.png

···

Hans Yperman

Department IT

Vlaams Instituut voor de Zee vzw

InnovOcean Campus, Jacobsenstraat 1

8400 Oostende, België

:phone:+32 (0) 59 33 61 13

:e-mail: hans.yperman@…1715…

www.vliz.be

2

I expect we will need to wait on netcdf for a fix; thank you for reporting the issue to them.

image001.png

···

–
Jody Garnett

3

As noted here, upgrading the NetCDF library will pose some challenge:
https://osgeo-org.atlassian.net/browse/GEOS-10831

See also:

Cheers

Andrea

Mostra testo citato

Il mar 24 gen 2023, 07:42 Jody Garnett <jody.garnett@anonymised.com> ha scritto:

I expect we will need to wait on netcdf for a fix; thank you for reporting the issue to them.

–
Jody Garnett

On Fri, Jan 20, 2023 at 3:21 PM Hans Yperman <hans.yperman@anonymised.com> wrote:

We found a bug where geoserver grows basically unbounded (32GB+) when using hdf5, while the java heap and native heaps stay small (2GB and 300MB resp). The bug still exists in the current version of netcdf-java. The main symptom is unfortunately not noticeable with JVM tooling: a lot of mmapped segments of around 64K each appear when you look on linux with pmap.

I hacked a fix in the cdm-4.6.15.jar library. A test scenario that crashed the server in minutes now runs for an hour straight without growing the process RSS size.

I logged a bug in both projects (the geoserver one is pretty minimal).

https://github.com/Unidata/netcdf-java/issues/1138

https://osgeo-org.atlassian.net/browse/GEOS-10831

Hans Yperman

Department IT

Vlaams Instituut voor de Zee vzw

InnovOcean Campus, Jacobsenstraat 1

8400 Oostende, België

:phone:+32 (0) 59 33 61 13

:e-mail: hans.yperman@anonymised.com

www.vliz.be

Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users