Hi,
I wonder if there are binaries from the ErMapper/Leica/ERDAS ECWJPEG2000 SDK of version 3.x somewhere behind the screen. Growing memory consumption is typical for all applications which use that SDK version and during the use lots of panning and zooming occur. WMS usage is surely of that kind. Reason is that the SDK tries to optimize the situation and builds temporary indexes/overviews/pyramid layers into memory for speeding up the next visit to the same area which is usually a good idea for desktop GIS. If this is the case I fear there is no way to fix the issue because what happens is done by closed source code outside both Java and GDAL which is also there in the middle. I am remembering that there is a hard coded timeout of 10 minutes for these temporary structures and knowing that helps sometimes with debugging because memory consumption goes down again after 10 minutes of idle time if such can be arranged.
If some other driver than GDAL ECWJP2 driver based on SDK 3.x is in use then you can forget this message.
-Jukka Rahkonen-
···
Lähettäjä: Dominik Mikiewicz [mailto:dominikmikiewicz@anonymised.com]
Lähetetty: 25. kesäkuuta 2014 9:53
Vastaanottaja: geoserver-devel@lists.sourceforge.net
Aihe: [Geoserver-devel] gdal / jp2 memory leak
Hi all,
I am having problems with memory leaking when serving jp2.
The whole setup is running on windows server 2k12 x64.
I have deployed geoserver under tomcat 7 – 4 separate instances accessed through a load balancer.
Geoserver / tomcat quickly consumes memory until it crashes and I am wondering whether this is a matter of misconfiguration of my tomcat services or maybe it is related to geoserver, gdal, etc.
Each tomcat runs as a windows service with the following configuration:
· Startup: auto
· User: local system account
· Java jre 7 x64
· Startup params:
-Xmx2048m
-Xms2048m
-XX:PermSize=512m
-XX:MaxPermSize=512
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseParallelGC
-Dsun.rmi.dgc.client.gcInterval=600000
-Dsun.rmi.dgc.server.gcInterval=600000
Geoserver 2.5.1 is deployed from a war file.
Each raster layer has a corresponding store
SUGGESTED_TILE_SIZE: 512,512
USE_JAI_IMAGEREAD: true
USE_MULTITHREADING: false
GDAL 1.11 and ecw gdal plugin from http://www.gisinternals.com/sdk/
Rasters I am trying to serve vary in size – from 200mb to 40gb, but the high memory consumption can be observed on all the files
In a matter of 5 – 10 minutes using just an OpenLayers client with one layer (more layers displayed makes the crash much quicker) I am able to make geoserver instances eat up all the available memory and then start reporting ‘unable to allocate temporary buffer ‘ java.lang.RuntimeException (tomcat then crashes).
Load balancer is set to use round robin so all the instances get similar throughput.
If I understand well, tomcat/geoserver should not take more than 2g of memory, but it seems to completely ignore this setting and quickly goes up to 14gb on each instance. What makes it worse is a fact that it does not release the memory
All the ideas appreciated
thanx
dom
Ps. Below is the log:
18 Jun 17:54:27 ERROR [geoserver.ows] -
java.lang.RuntimeException: Unable to allocate temporary buffer.
at org.gdal.gdal.gdalJNI.Dataset_ReadRaster__SWIG_0(Native Method)
at org.gdal.gdal.Dataset.ReadRaster(Dataset.java:178)
at it.geosolutions.imageio.gdalframework.GDALImageReader.readDatasetRaster(GDALImageReader.java:344)
at it.geosolutions.imageio.gdalframework.GDALImageReader.read(GDALImageReader.java:952)
at com.sun.media.jai.imageioimpl.ImageReadOpImage.computeTile(ImageReadOpImage.java:697)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.PlanarImage.getExtendedData(PlanarImage.java:2440)
at javax.media.jai.ScaleOpImage.computeTile(ScaleOpImage.java:1101)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.PlanarImage.getExtendedData(PlanarImage.java:2440)
at com.sun.media.jai.opimage.MosaicOpImage.computeTile(MosaicOpImage.java:432)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.opimage.BandSelectOpImage.computeTile(BandSelectOpImage.java:120)
at com.sun.media.jai.opimage.BandSelectOpImage.getTile(BandSelectOpImage.java:144)
at javax.media.jai.RenderedOp.getTile(RenderedOp.java:2257)
at com.sun.imageio.plugins.jpeg.JPEGImageWriter.writeOnThread(Unknown Source)
at com.sun.imageio.plugins.jpeg.JPEGImageWriter.write(Unknown Source)
at org.geotools.image.ImageWorker.writeJPEG(ImageWorker.java:2977)
at org.geoserver.wms.map.JPEGMapResponse.formatImageOutputStream(JPEGMapResponse.java:69)
at org.geoserver.wms.map.RenderedImageMapResponse.write(RenderedImageMapResponse.java:122)
at org.geoserver.ows.Dispatcher.response(Dispatcher.java:919)
at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:276)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:74)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:70)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:70)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:45)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:49)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:68)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:72)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:68)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:72)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:53)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:68)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:72)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:82)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:68)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:52)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:72)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:134)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:75)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:43)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2442)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2431)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)