[Geoserver-users] making a very large imagemosaic - store creation is killed- more resources needed?

Adam_Steer · March 19, 2017, 10:26pm

Hi all

I’m attempting to create a very large imagemosaic in Geoserver 2.10.2.

The process goes like:

1. add the MODIS sinusoidal projection (http://spatialreference.org/ref/sr-org/6974/geoserver/) to user_projections and netcdf_projections
2. use CreateIndexer.jar (http://geoserver.geo-solutions.it/downloads/releases/2.8.x-netcdf/CreateIndexer.jar) to generate an indexer.xml and _auxiliary.xml for the underlying files
3. check that permissions are all correct on the data store (a directory of many netCDF files)
4. check that my datastore.propoerties file is all correct
5. use the imagemosaic web UI create a store

What happens? Geoserver dutifully trundles off, tells me it’s busy and starts writing to the relevant database tables.

The tables have data, including geometries:

--
select * from bare_soil limit 10;
fid | the_geom | imageindex | location | time
-----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------------------------------------------------------------------------------+---------------------
1 | 01030000000100000005000000D2F3B5D18E1673C100E0D6EF17F86CC0D2F3B5D18E1673C13964C84576F83041A3754DF1F80672C13964C84576F83041A3754DF1F80672C100E0D6EF17F86CC0D2F3B5D18E1673C100E0D6EF17F86CC0 | 80 | /local/geoserver_data/gdata/fr5/FC.v302.MCD43A4/FC.v302.MCD43A4.h00v08.2000.005.nc | 2000-02-18 11:00:00
2 | 01030000000100000005000000D2F3B5D18E1673C100E0D6EF17F86CC0D2F3B5D18E1673C13964C84576F83041A3754DF1F80672C13964C84576F83041A3754DF1F80672C100E0D6EF17F86CC0D2F3B5D18E1673C100E0D6EF17F86CC0 | 81 | /local/geoserver_data/gdata/fr5/FC.v302.MCD43A4/FC.v302.MCD43A4.h00v08.2000.005.nc | 2000-02-26 11:00:00

…and the geometries actually hold data (whether they are correct I have to check - but for the moment I dangerously assume that they are):

select st_astext(the_geom) from bare_soil limit 10;
st_astext
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
POLYGON((-20015341.106922 -231.752921981504,-20015341.106922 1112182.27258898,-18902927.081411 1112182.27258898,-18902927.081411 -231.752921981504,-20015341.106922 -231.752921981504))
POLYGON((-20015341.106922 -231.752921981504,-20015341.106922 1112182.27258898,-18902927.081411 1112182.27258898,-18902927.081411 -231.752921981504,-20015341.106922 -231.752921981504))

….but no store is ever created.

There are just over 9k tiles, in 4D (ie each tile has X, Y, and time axes). The DB tables get to about 51mb for each ‘layer’ publishable from the tiles.

The only clue I have found in catalina.out is this:

[gce.imagemosaic] - Unable to close indexingnull

…which leads to a post about geoserver needing an EPSG: code: https://osgeo-org.atlassian.net/browse/GEOS-7201 - but this is apparently resolved.

Geoserver’s own log (geoserver-home/logs/geoserver.log) shows me that the mosaic construction is cancelled (see below).

Do I need to:

a. give geoserver more resources?
b. somehow give tomcat more time before it kills processes (no idea if this is a setting I can change or if it’s even relevant)
c. provide more detail?

It’s a single geoserver instance, with Xms 8g, Xmx 8g, G1Gc garbage collection, on a 16GB/8CPU VM. I can make some swap space (it’s on SSD), and expand the heap size if needed - but that might be silly Java management.

Thoughts/suggestions welcome!

Thanks

Adam

—
(geoserver.log output)
2017-03-18 20:52:36,788 WARN [gce.imagemosaic] - Failure occurred while collecting the granules
org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled
  at org.apache.commons.io.DirectoryWalker.checkIfCancelled(DirectoryWalker.java:395)
  at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:368)
  at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
  at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.<init>(ImageMosaicDirectoryWalker.java:114)
  at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:197)
  at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:450)
  at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1310)
  at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:532)
  at org.geotools.gce.imagemosaic.ImageMosaicReader.<init>(ImageMosaicReader.java:479)
  at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:502)
  at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:97)
  at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1493)
  at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1423)
  at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:147)
  at com.sun.proxy.$Proxy13.getGridCoverageReader(Unknown Source)
  at org.geoserver.web.data.layer.NewLayerPageProvider.getItemsInternal(NewLayerPageProvider.java:95)
  at org.geoserver.web.data.layer.NewLayerPageProvider.getItems(NewLayerPageProvider.java:61)
  at org.geoserver.web.wicket.GeoServerDataProvider.fullSize(GeoServerDataProvider.java:243)
  at org.geoserver.web.wicket.GeoServerTablePanel$PagerDelegate.updateMatched(GeoServerTablePanel.java:583)
  at org.geoserver.web.wicket.GeoServerTablePanel$PagerDelegate.<init>(GeoServerTablePanel.java:576)
  at org.geoserver.web.wicket.GeoServerTablePanel.<init>(GeoServerTablePanel.java:176)
  at org.geoserver.web.wicket.GeoServerTablePanel.<init>(GeoServerTablePanel.java:97)
  at org.geoserver.web.data.layer.NewLayerPage$1.<init>(NewLayerPage.java:105)
  at org.geoserver.web.data.layer.NewLayerPage.<init>(NewLayerPage.java:105)
  at org.geoserver.web.data.store.CoverageStoreNewPage.onSuccessfulSave(CoverageStoreNewPage.java:86)
  at org.geoserver.web.data.store.CoverageStoreNewPage.onSave(CoverageStoreNewPage.java:74)
  at org.geoserver.web.data.store.AbstractCoverageStorePage$1.onSubmit(AbstractCoverageStorePage.java:122)
  at org.apache.wicket.ajax.markup.html.form.AjaxSubmitLink$1.onSubmit(AjaxSubmitLink.java:110)
  at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:215)
  at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1307)
  at org.apache.wicket.markup.html.form.Form.process(Form.java:974)
  at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:795)
  at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:171)
  at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155)
  at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:588)
  at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
  at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
  at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248)
  at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234)
  at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:865)
  at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
  at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:265)
  at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:222)
  at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:293)
  at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:261)
  at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
  at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
  at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:158)
  at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
  at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
  at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
  at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
  at org.geoserver.flow.controller.IpBlacklistFilter.doFilter(IpBlacklistFilter.java:94)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
  at org.geoserver.flow.ControlFlowCallback.doFilter(ControlFlowCallback.java:252)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
  at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
  at org.geoserver.security.filter.GeoServerUserNamePasswordAuthenticationFilter.doFilter(GeoServerUserNamePasswordAuthenticationFilter.java:116)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:157)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
  at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
  at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:152)
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:87)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
  at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
  at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)
  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
  at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2517)
  at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2506)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.lang.Thread.run(Thread.java:745)

Daniele_Romagnoli2 · March 20, 2017, 11:25am

Hi Adam,

can you provide us additional logging? (looking for something which cancel the directory walking if available).

In general, when dealing with similar big datasets, configuring stores via REST would be better.

You can take a look on these page to get some information on that topic:
http://docs.geoserver.org/stable/en/user/rest/examples/curl.html#uploading-and-modifying-a-image-mosaic
http://geoserver.geo-solutions.it/edu/en/multidim/rest/index.html

As an instance, you could start configuring a store on top of a set of sample granules and then harvest additional ones.

Please, let us know.

Cheers,

Daniele

···

On Sun, Mar 19, 2017 at 11:26 PM, Adam Steer <Adam.Steer@anonymised.com> wrote:

Hi all

I’m attempting to create a very large imagemosaic in Geoserver 2.10.2.

The process goes like:

add the MODIS sinusoidal projection (http://spatialreference.org/ref/sr-org/6974/geoserver/) to user_projections and netcdf_projections

use CreateIndexer.jar (http://geoserver.geo-solutions.it/downloads/releases/2.8.x-netcdf/CreateIndexer.jar) to generate an indexer.xml and _auxiliary.xml for the underlying files

check that permissions are all correct on the data store (a directory of many netCDF files)

check that my datastore.propoerties file is all correct

use the imagemosaic web UI create a store

What happens? Geoserver dutifully trundles off, tells me it’s busy and starts writing to the relevant database tables.

The tables have data, including geometries:

–
select * from bare_soil limit 10;
fid | the_geom | imageindex | location | time
-----±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------±-----------±-----------------------------------------------------------------------------------±--------------------
1 | 01030000000100000005000000D2F3B5D18E1673C100E0D6EF17F86CC0D2F3B5D18E1673C13964C84576F83041A3754DF1F80672C13964C84576F83041A3754DF1F80672C100E0D6EF17F86CC0D2F3B5D18E1673C100E0D6EF17F86CC0 | 80 | /local/geoserver_data/gdata/fr5/FC.v302.MCD43A4/FC.v302.MCD43A4.h00v08.2000.005.nc | 2000-02-18 11:00:00
2 | 01030000000100000005000000D2F3B5D18E1673C100E0D6EF17F86CC0D2F3B5D18E1673C13964C84576F83041A3754DF1F80672C13964C84576F83041A3754DF1F80672C100E0D6EF17F86CC0D2F3B5D18E1673C100E0D6EF17F86CC0 | 81 | /local/geoserver_data/gdata/fr5/FC.v302.MCD43A4/FC.v302.MCD43A4.h00v08.2000.005.nc | 2000-02-26 11:00:00

…and the geometries actually hold data (whether they are correct I have to check - but for the moment I dangerously assume that they are):

select st_astext(the_geom) from bare_soil limit 10;
st_astext

POLYGON((-20015341.106922 -231.752921981504,-20015341.106922 1112182.27258898,-18902927.081411 1112182.27258898,-18902927.081411 -231.752921981504,-20015341.106922 -231.752921981504))
POLYGON((-20015341.106922 -231.752921981504,-20015341.106922 1112182.27258898,-18902927.081411 1112182.27258898,-18902927.081411 -231.752921981504,-20015341.106922 -231.752921981504))

….but no store is ever created.

There are just over 9k tiles, in 4D (ie each tile has X, Y, and time axes). The DB tables get to about 51mb for each ‘layer’ publishable from the tiles.

The only clue I have found in catalina.out is this:

[gce.imagemosaic] - Unable to close indexingnull

…which leads to a post about geoserver needing an EPSG: code: https://osgeo-org.atlassian.net/browse/GEOS-7201 - but this is apparently resolved.

Geoserver’s own log (geoserver-home/logs/geoserver.log) shows me that the mosaic construction is cancelled (see below).

Do I need to:

a. give geoserver more resources?
b. somehow give tomcat more time before it kills processes (no idea if this is a setting I can change or if it’s even relevant)
c. provide more detail?

It’s a single geoserver instance, with Xms 8g, Xmx 8g, G1Gc garbage collection, on a 16GB/8CPU VM. I can make some swap space (it’s on SSD), and expand the heap size if needed - but that might be silly Java management.

Thoughts/suggestions welcome!

Thanks

Adam

—
(geoserver.log output)
2017-03-18 20:52:36,788 WARN [gce.imagemosaic] - Failure occurred while collecting the granules
org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled
at org.apache.commons.io.DirectoryWalker.checkIfCancelled(DirectoryWalker.java:395)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:368)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:197)
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:450)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1310)
at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:532)
at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:479)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:502)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:97)
at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1493)
at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1423)
at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:147)
at com.sun.proxy.$Proxy13.getGridCoverageReader(Unknown Source)
at org.geoserver.web.data.layer.NewLayerPageProvider.getItemsInternal(NewLayerPageProvider.java:95)
at org.geoserver.web.data.layer.NewLayerPageProvider.getItems(NewLayerPageProvider.java:61)
at org.geoserver.web.wicket.GeoServerDataProvider.fullSize(GeoServerDataProvider.java:243)
at org.geoserver.web.wicket.GeoServerTablePanel$PagerDelegate.updateMatched(GeoServerTablePanel.java:583)
at org.geoserver.web.wicket.GeoServerTablePanel$PagerDelegate.(GeoServerTablePanel.java:576)
at org.geoserver.web.wicket.GeoServerTablePanel.(GeoServerTablePanel.java:176)
at org.geoserver.web.wicket.GeoServerTablePanel.(GeoServerTablePanel.java:97)
at org.geoserver.web.data.layer.NewLayerPage$1.(NewLayerPage.java:105)
at org.geoserver.web.data.layer.NewLayerPage.(NewLayerPage.java:105)
at org.geoserver.web.data.store.CoverageStoreNewPage.onSuccessfulSave(CoverageStoreNewPage.java:86)
at org.geoserver.web.data.store.CoverageStoreNewPage.onSave(CoverageStoreNewPage.java:74)
at org.geoserver.web.data.store.AbstractCoverageStorePage$1.onSubmit(AbstractCoverageStorePage.java:122)
at org.apache.wicket.ajax.markup.html.form.AjaxSubmitLink$1.onSubmit(AjaxSubmitLink.java:110)
at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:215)
at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1307)
at org.apache.wicket.markup.html.form.Form.process(Form.java:974)
at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:795)
at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:171)
at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155)
at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:588)
at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248)
at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234)
at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:865)
at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:265)
at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:222)
at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:293)
at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:261)
at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:158)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.flow.controller.IpBlacklistFilter.doFilter(IpBlacklistFilter.java:94)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.flow.ControlFlowCallback.doFilter(ControlFlowCallback.java:252)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerUserNamePasswordAuthenticationFilter.doFilter(GeoServerUserNamePasswordAuthenticationFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:157)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:152)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:87)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2517)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2506)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-users mailing list
Geoserver-users@anonymised.com.382…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Adam_Steer · March 20, 2017, 1:14pm

Hi Daniele

can you provide us additional logging? (looking for something which cancel the directory walking if available).

I’ll take a look - it may be that I’ve missed an out-of-memory kill and should check more generic system logs.

But first I’ll try this:

You can take a look on these page to get some information on that topic:
http://docs.geoserver.org/stable/en/user/rest/examples/curl.html#uploading-and-modifying-a-image-mosaic
http://geoserver.geo-solutions.it/edu/en/multidim/rest/index.html
As an instance, you could start configuring a store on top of a set of sample granules and then harvest additional ones.

It seems a lot saner to build a small index then add more granules - and i didn’t know geoserver could do that!

Iterating through the files (or file subsets, say by year) would probably avoid memory issues, and give me more time to make coffee...

Please, let us know.

Will do, thanks for the suggestions.

Regards

Adam

[Geoserver-users] making a very large imagemosaic - store creation is killed- more resources needed?

select st_astext(the_geom) from bare_soil limit 10; st_astext

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

select st_astext(the_geom) from bare_soil limit 10;
st_astext

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.