[Geoserver-users] Image Mosaic REST harvesting - failure after time passes

Mike_Grogan · May 1, 2015, 1:40am

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting / creating new image mosaics via REST … that seems to be occurring after time passes.

What I mean is that harvesting & creating new mosaics works fine most of the time. However, it seems to fail after some amount of time passes. For instance, if I have not created a new mosaic via REST for a day or 2, and I try to create one, it will fail. If I then try to create several more via REST, again, they all fail.

If I restart Tomcat or postgres and then run the very same REST call (via gsconfig in python) against the very same data, voila, it is successful. After restart, I can then go on creating multiple new mosaics via REST over the course of many hours.

Let some time pass, though, and I’ll start seeing failures again.

Yes, this is a troubleshooting nightmare.

My initial thoughts have been this has something to do with stale connections via JNDI … but I have validation turned on, both in JNDI Tomcat Resource and in datastore.properties. I have tried leaving in the eviction settings (shown in some of the JNDI Image Mosaic examples) and removing them … but this still doesn’t help.

What’s odd is that I have no problems accessing existing mosaic indexes backed via JNDI. It just seems the ability to create new ones goes bad after some time passes (usually 24 hours or so).

Again, JNDI has been a suspicion. I kind of also am asking myself if something is somehow getting garbage collected that shouldn’t be (I know that sounds strange) … or something else is somehow disappearing over time that would prevent successful mosaic creation via REST.

Does this sound familiar to anyone? Any pointers?

The stack trace from when this happens is below.

Thanks!

30 Apr 19:02:45 INFO [catalog.rest] - PUT file, mimetype: application/zip
30 Apr 19:02:45 INFO [catalog.rest] - Auto-configuring coverage store: sppub–mikegtest20150430–gust
Apr 30, 2015 7:02:45 PM org.geotools.jdbc.JDBCJNDIDataStoreFactory createDataSource
WARNING: Using java:comp/env/jdbc/stormprint instead of jdbc/stormprint would avoid an unnecessary JNDI lookup
30 Apr 19:02:45 WARN [geotools.jdbc] - Failure occurred while looking up the primary key with finder: org.geotools.jdbc.HeuristicPrimaryKeyFinder@anonymised.com
org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:252)
at org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getPrimaryKeys(AbstractJdbc2DatabaseMetaData.java:3057)
at org.geotools.jdbc.HeuristicPrimaryKeyFinder.getPrimaryKey(HeuristicPrimaryKeyFinder.java:49)
at org.geotools.jdbc.CompositePrimaryKeyFinder.getPrimaryKey(CompositePrimaryKeyFinder.java:52)
at org.geotools.jdbc.JDBCDataStore.getPrimaryKey(JDBCDataStore.java:1066)
at org.geotools.jdbc.JDBCFeatureSource.(JDBCFeatureSource.java:94)
at org.geotools.jdbc.JDBCDataStore.createFeatureSource(JDBCDataStore.java:910)
at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:388)
at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:353)
at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:103)
at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.extractBasicProperties(GTDataStoreGranuleCatalog.java:258)
at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.createType(GTDataStoreGranuleCatalog.java:559)
at org.geotools.gce.imagemosaic.catalog.CachingDataStoreGranuleCatalog.createType(CachingDataStoreGranuleCatalog.java:103)
at org.geotools.gce.imagemosaic.RasterManager.createStore(RasterManager.java:1263)
at org.geotools.gce.imagemosaic.ImageMosaicReader.createCoverage(ImageMosaicReader.java:1114)
at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.updateConfiguration(ImageMosaicConfigHandler.java:845)
at org.geotools.gce.imagemosaic.ImageMosaicWalker.handleFile(ImageMosaicWalker.java:210)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.handleFile(ImageMosaicDirectoryWalker.java:98)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:367)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:354)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1223)

at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:505)
at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:457)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:212)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)
at org.geoserver.catalog.rest.CoverageStoreFileResource.handlePut(CoverageStoreFileResource.java:208)
at org.restlet.Finder.handle(Finder.java:298)
at org.geoserver.rest.BeanDelegatingRestlet.handle(BeanDelegatingRestlet.java:38)
at org.restlet.Filter.doHandle(Filter.java:105)
at org.restlet.Filter.handle(Filter.java:134)
at org.restlet.Router.handle(Router.java:444)
at org.geoserver.rest.RESTDispatcher$1.handle(RESTDispatcher.java:205)
at com.noelios.restlet.ext.servlet.ServletConverter.service(ServletConverter.java:129)
at org.geoserver.rest.RESTDispatcher.handleRequestInternal(RESTDispatcher.java:87)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
30 Apr 19:02:45 ERROR [imagemosaic.catalog] - org.geotools.jdbc.JDBCFeatureSource cannot be cast to org.geotools.data.simple.SimpleFeatureStore

java.lang.ClassCastException: org.geotools.jdbc.JDBCFeatureSource cannot be cast to org.geotools.data.simple.SimpleFeatureStore
at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.removeGranules(GTDataStoreGranuleCatalog.java:319)
at org.geotools.gce.imagemosaic.catalog.CachingDataStoreGranuleCatalog.removeGranules(CachingDataStoreGranuleCatalog.java:233)
at org.geotools.gce.imagemosaic.catalog.GranuleCatalogStore.removeGranules(GranuleCatalogStore.java:88)
at org.geotools.gce.imagemosaic.CatalogManager.updateCatalog(CatalogManager.java:406)
at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.updateConfiguration(ImageMosaicConfigHandler.java:937)
at org.geotools.gce.imagemosaic.ImageMosaicWalker.handleFile(ImageMosaicWalker.java:210)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.handleFile(ImageMosaicDirectoryWalker.java:98)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:367)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:354)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1223)
at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:505)
at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:457)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:212)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)
at org.geoserver.catalog.rest.CoverageStoreFileResource.handlePut(CoverageStoreFileResource.java:208)
at org.restlet.Finder.handle(Finder.java:298)
at org.geoserver.rest.BeanDelegatingRestlet.handle(BeanDelegatingRestlet.java:38)
at org.restlet.Filter.doHandle(Filter.java:105)
at org.restlet.Filter.handle(Filter.java:134)
at org.restlet.Router.handle(Router.java:444)
at org.geoserver.rest.RESTDispatcher$1.handle(RESTDispatcher.java:205)
at com.noelios.restlet.ext.servlet.ServletConverter.service(ServletConverter.java:129)
at org.geoserver.rest.RESTDispatcher.handleRequestInternal(RESTDispatcher.java:87)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
30 Apr 19:02:45 ERROR [imagemosaic.Utils] - org.geotools.jdbc.JDBCFeatureSource cannot be cast to org.geotools.data.simple.SimpleFeatureStore
java.lang.ClassCastException: org.geotools.jdbc.JDBCFeatureSource cannot be cast to org.geotools.data.simple.SimpleFeatureStore
at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.addGranules(GTDataStoreGranuleCatalog.java:350)
at org.geotools.gce.imagemosaic.catalog.CachingDataStoreGranuleCatalog.addGranules(CachingDataStoreGranuleCatalog.java:85)
at org.geotools.gce.imagemosaic.catalog.GranuleCatalog.addGranule(GranuleCatalog.java:55)
at org.geotools.gce.imagemosaic.catalog.GranuleCatalogStore.addGranules(GranuleCatalogStore.java:56)
at org.geotools.gce.imagemosaic.CatalogManager.updateCatalog(CatalogManager.java:409)
at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.updateConfiguration(ImageMosaicConfigHandler.java:937)
at org.geotools.gce.imagemosaic.ImageMosaicWalker.handleFile(ImageMosaicWalker.java:210)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.handleFile(ImageMosaicDirectoryWalker.java:98)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:367)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:354)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1223)
at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:505)
at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:457)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:212)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)
at org.geoserver.catalog.rest.CoverageStoreFileResource.handlePut(CoverageStoreFileResource.java:208)
at org.restlet.Finder.handle(Finder.java:298)
at org.geoserver.rest.BeanDelegatingRestlet.handle(BeanDelegatingRestlet.java:38)
at org.restlet.Filter.doHandle(Filter.java:105)
at org.restlet.Filter.handle(Filter.java:134)
at org.restlet.Router.handle(Router.java:444)
at org.geoserver.rest.RESTDispatcher$1.handle(RESTDispatcher.java:205)
at com.noelios.restlet.ext.servlet.ServletConverter.service(ServletConverter.java:129)
at org.geoserver.rest.RESTDispatcher.handleRequestInternal(RESTDispatcher.java:87)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
30 Apr 19:02:45 WARN [gce.imagemosaic] - Failure occurred while collecting the granules
org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled
at org.apache.commons.io.DirectoryWalker.checkIfCancelled(DirectoryWalker.java:395)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:368)
at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)
at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:354)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1223)
at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:505)
at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:457)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:212)
at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)
at org.geoserver.catalog.rest.CoverageStoreFileResource.handlePut(CoverageStoreFileResource.java:208)
at org.restlet.Finder.handle(Finder.java:298)
at org.geoserver.rest.BeanDelegatingRestlet.handle(BeanDelegatingRestlet.java:38)
at org.restlet.Filter.doHandle(Filter.java:105)
at org.restlet.Filter.handle(Filter.java:134)
at org.restlet.Router.handle(Router.java:444)

at org.geoserver.rest.RESTDispatcher$1.handle(RESTDispatcher.java:205)
at com.noelios.restlet.ext.servlet.ServletConverter.service(ServletConverter.java:129)
at org.geoserver.rest.RESTDispatcher.handleRequestInternal(RESTDispatcher.java:87)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)

at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
30 Apr 19:02:45 ERROR [geoserver.rest] - Could not aquire reader for coverage.
30 Apr 19:02:45 ERROR [geoserver.rest] -
org.geoserver.rest.RestletException
at org.geoserver.catalog.rest.CoverageStoreFileResource.handlePut(CoverageStoreFileResource.java:211)
at org.restlet.Finder.handle(Finder.java:298)
at org.geoserver.rest.BeanDelegatingRestlet.handle(BeanDelegatingRestlet.java:38)
at org.restlet.Filter.doHandle(Filter.java:105)
at org.restlet.Filter.handle(Filter.java:134)
at org.restlet.Router.handle(Router.java:444)
at org.geoserver.rest.RESTDispatcher$1.handle(RESTDispatcher.java:205)
at com.noelios.restlet.ext.servlet.ServletConverter.service(ServletConverter.java:129)
at org.geoserver.rest.RESTDispatcher.handleRequestInternal(RESTDispatcher.java:87)

at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)

at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Andrea_Aime4 · May 1, 2015, 1:45pm

On Fri, May 1, 2015 at 3:40 AM, Mike Grogan <d.michael.grogan@anonymised.com>
wrote:

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting /
creating new image mosaics via REST ... that seems to be occurring after
time passes.

By the description this is similar to a typical connection pool issue with
lack of use for several hours, the connections are closed by the database
end, the connection pool has no way to know, and grabbing the connection
ends up having to time out, with everything locked up (if you have a
validation query, otherwise it fails in other ways).

Do you have the validation query setup in your JNDI connection pool?
Also, what about the followoing params, that we are setting up in our
normal data stores, that are there to deal with the issues
of connections not being used for several hours? (sorry for pasting code,
today is a holiday in Italy, have to be expedient as I'm
getting out of the house):

Boolean testWhileIdle = (Boolean) TEST_WHILE_IDLE.lookUp(params);
        if(testWhileIdle != null) {
            dataSource.setTestWhileIdle(testWhileIdle);
        }

Integer timeBetweenEvictorRuns = (Integer)
TIME_BETWEEN_EVICTOR_RUNS.lookUp(params);
if(timeBetweenEvictorRuns != null && timeBetweenEvictorRuns > 0) {

dataSource.setTimeBetweenEvictionRunsMillis(timeBetweenEvictorRuns * 1000l);
}

        Integer minEvictableTime = (Integer)
MIN_EVICTABLE_TIME.lookUp(params);
        if(minEvictableTime != null) {
            dataSource.setMinEvictableIdleTimeMillis(minEvictableTime *
1000l);
        }

        Integer evictorTestsPerRun = (Integer)
EVICTOR_TESTS_PER_RUN.lookUp(params);
        if(evictorTestsPerRun != null) {
            dataSource.setNumTestsPerEvictionRun(evictorTestsPerRun);
        }

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

Mike_Grogan · May 1, 2015, 1:53pm

Holiday? Why are you answering today? Ha ha …

Thanks for this. I will review and write back. Please, enjoy the holiday and the weekend.

Mike

···

On Fri, May 1, 2015 at 9:45 AM, Andrea Aime <andrea.aime@anonymised.com.> wrote:

On Fri, May 1, 2015 at 3:40 AM, Mike Grogan <d.michael.grogan@anonymised.com> wrote:

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting / creating new image mosaics via REST … that seems to be occurring after time passes.

By the description this is similar to a typical connection pool issue with lack of use for several hours, the connections are closed by the database
end, the connection pool has no way to know, and grabbing the connection ends up having to time out, with everything locked up (if you have a validation query, otherwise it fails in other ways).

Do you have the validation query setup in your JNDI connection pool?
Also, what about the followoing params, that we are setting up in our normal data stores, that are there to deal with the issues
of connections not being used for several hours? (sorry for pasting code, today is a holiday in Italy, have to be expedient as I’m
getting out of the house):

Boolean testWhileIdle = (Boolean) TEST_WHILE_IDLE.lookUp(params);
if(testWhileIdle != null) {
dataSource.setTestWhileIdle(testWhileIdle);
}

Integer timeBetweenEvictorRuns = (Integer) TIME_BETWEEN_EVICTOR_RUNS.lookUp(params);
if(timeBetweenEvictorRuns != null && timeBetweenEvictorRuns > 0) {
dataSource.setTimeBetweenEvictionRunsMillis(timeBetweenEvictorRuns * 1000l);
}

Integer minEvictableTime = (Integer) MIN_EVICTABLE_TIME.lookUp(params);
if(minEvictableTime != null) {
dataSource.setMinEvictableIdleTimeMillis(minEvictableTime * 1000l);
}

Integer evictorTestsPerRun = (Integer) EVICTOR_TESTS_PER_RUN.lookUp(params);
if(evictorTestsPerRun != null) {
dataSource.setNumTestsPerEvictionRun(evictorTestsPerRun);
}

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Mike_Grogan · May 1, 2015, 4:32pm

Ok, now my answers … hope your holiday is going well …

For JNDI, I was initially using the JNDI PostGIS Example settings from http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_datastore.html, so my configuration looked like:

So, I followed the textbook example for JNDI for GeoServer, including a validation query. I was experiencing the described problem under this example configuration.

I then started experimenting with the configuration, and found that if I manually killed the db connections in postgres: “SELECT pg_terminate_backend(PID#);”, GeoServer would absolutely not recover from this … for display … for anything … it was just choking on any JNDI-backed stores. If I removed both of the eviction settings in the last few rows (keeping validation), I could kill the connections as much as I wanted, and GeoServer would recover.

So, what I ended up with now is:

As you can see, I added testOnBorrow and testOnReturn.

Looking at your code, which seems to be in https://github.com/geotools/geotools/blob/12.x/modules/library/jdbc/src/main/java/org/geotools/jdbc/JDBCDataStoreFactory.java (for my future reference), maybe I should be adding in a testWhileIdle to the JNDI configuration?

Thoughts?

Thanks again,

Mike

···

On Fri, May 1, 2015 at 9:45 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, May 1, 2015 at 3:40 AM, Mike Grogan <d.michael.grogan@anonymised.com> wrote:

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting / creating new image mosaics via REST … that seems to be occurring after time passes.

By the description this is similar to a typical connection pool issue with lack of use for several hours, the connections are closed by the database
end, the connection pool has no way to know, and grabbing the connection ends up having to time out, with everything locked up (if you have a validation query, otherwise it fails in other ways).

Do you have the validation query setup in your JNDI connection pool?
Also, what about the followoing params, that we are setting up in our normal data stores, that are there to deal with the issues
of connections not being used for several hours? (sorry for pasting code, today is a holiday in Italy, have to be expedient as I’m
getting out of the house):

Boolean testWhileIdle = (Boolean) TEST_WHILE_IDLE.lookUp(params);
if(testWhileIdle != null) {
dataSource.setTestWhileIdle(testWhileIdle);
}

Integer timeBetweenEvictorRuns = (Integer) TIME_BETWEEN_EVICTOR_RUNS.lookUp(params);
if(timeBetweenEvictorRuns != null && timeBetweenEvictorRuns > 0) {
dataSource.setTimeBetweenEvictionRunsMillis(timeBetweenEvictorRuns * 1000l);
}

Integer minEvictableTime = (Integer) MIN_EVICTABLE_TIME.lookUp(params);
if(minEvictableTime != null) {
dataSource.setMinEvictableIdleTimeMillis(minEvictableTime * 1000l);
}

Integer evictorTestsPerRun = (Integer) EVICTOR_TESTS_PER_RUN.lookUp(params);
if(evictorTestsPerRun != null) {
dataSource.setNumTestsPerEvictionRun(evictorTestsPerRun);
}

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Mike_Grogan · May 1, 2015, 7:17pm

On May 1, 2015 3:16 PM, "Mike Grogan" <d.michael.grogan@anonymised.com> wrote:

Oh and I meant to say that I am experiencing the problem under the second
configuration I posted as well.

Thanks,

Mike
On May 1, 2015 12:32 PM, "Mike Grogan" <d.michael.grogan@anonymised.com> wrote:

Ok, now my answers .... hope your holiday is going well ....

For JNDI, I was initially using the JNDI PostGIS Example settings from
http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_datastore.html,
so my configuration looked like:

<Resource
        name="jdbc/stormprint"
        auth="Container"
        type="javax.sql.DataSource"
        driverClassName="org.postgresql.Driver"
        url="jdbc:postgresql://localhost/stormprint"
        username="XXXXXXXXXXXXXXXXX"
        password="XXXXXXXXXXXXXXXXX"
        maxActive="20"
        maxIdle="10"
        maxWait="10000"
        minEvictableIdleTimeMillis="300000"
        timeBetweenEvictionRunsMillis="300000"
        validationQuery="SELECT 1"/>

So, I followed the textbook example for JNDI for GeoServer, including a
validation query. I was experiencing the described problem under this
example configuration.

I then started experimenting with the configuration, and found that if I
manually killed the db connections in postgres: "SELECT
pg_terminate_backend(PID#);", GeoServer would absolutely not recover from
this ... for display ... for anything ... it was just choking on any
JNDI-backed stores. If I removed both of the eviction settings in the last
few rows (keeping validation), I could kill the connections as much as I
wanted, and GeoServer would recover.

So, what I ended up with now is:

<Resource
  name="jdbc/stormprint"
  auth="Container"
  type="javax.sql.DataSource"
  driverClassName="org.postgresql.Driver"
  url="jdbc:postgresql://localhost/stormprint"
  username="XXXXXXXXXXXXXXXXXXXXXXXXX"
  password="XXXXXXXXXXXXXXXXXXXXXXXXX"
  maxActive="20"
  maxIdle="10"
  maxWait="10000"
  testOnBorrow="true"
  testOnReturn="true"
  validationQuery="SELECT 1"
  validationQueryTimeout="2"/>

As you can see, I added testOnBorrow and testOnReturn.

Looking at your code, which seems to be in
https://github.com/geotools/geotools/blob/12.x/modules/library/jdbc/src/main/java/org/geotools/jdbc/JDBCDataStoreFactory.java
(for my future reference), maybe I should be adding in a testWhileIdle to
the JNDI configuration?

Thoughts?

Thanks again,

Mike

On Fri, May 1, 2015 at 9:45 AM, Andrea Aime <andrea.aime@anonymised.com
> wrote:

On Fri, May 1, 2015 at 3:40 AM, Mike Grogan <d.michael.grogan@anonymised.com>
wrote:

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting /
creating new image mosaics via REST ... that seems to be occurring after
time passes.

By the description this is similar to a typical connection pool issue
with lack of use for several hours, the connections are closed by the
database
end, the connection pool has no way to know, and grabbing the connection
ends up having to time out, with everything locked up (if you have a
validation query, otherwise it fails in other ways).

Do you have the validation query setup in your JNDI connection pool?
Also, what about the followoing params, that we are setting up in our
normal data stores, that are there to deal with the issues
of connections not being used for several hours? (sorry for pasting
code, today is a holiday in Italy, have to be expedient as I'm
getting out of the house):

Boolean testWhileIdle = (Boolean) TEST_WHILE_IDLE.lookUp(params);
        if(testWhileIdle != null) {
            dataSource.setTestWhileIdle(testWhileIdle);
        }

        Integer timeBetweenEvictorRuns = (Integer)
TIME_BETWEEN_EVICTOR_RUNS.lookUp(params);
        if(timeBetweenEvictorRuns != null && timeBetweenEvictorRuns > 0)
{

dataSource.setTimeBetweenEvictionRunsMillis(timeBetweenEvictorRuns * 1000l);
        }

        Integer minEvictableTime = (Integer)
MIN_EVICTABLE_TIME.lookUp(params);
        if(minEvictableTime != null) {
            dataSource.setMinEvictableIdleTimeMillis(minEvictableTime *
1000l);
        }

        Integer evictorTestsPerRun = (Integer)
EVICTOR_TESTS_PER_RUN.lookUp(params);
        if(evictorTestsPerRun != null) {
            dataSource.setNumTestsPerEvictionRun(evictorTestsPerRun);
        }

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be confidential
or proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

Bart_van_den_Eijnde1 · May 4, 2015, 4:18am

Mike,

are tomcat and postgres on the same machine or is there a firewall between them?

I have seen Cisco(?) firewalls silently sever idle connections after an hour or so; the situation could not be recovered by connection validation. Note that this was app-schema WFS output and I have not used imagemosaic, but if this is a firewall problem a similar workaround for the database connection might help. Here is a JNDI Context that shuts down idle connections after 5 to 10 minutes:
http://docs.geoserver.org/latest/en/user/data/app-schema/data-stores.html#jndi

"Firewall timeouts can silently sever idle connections to the database and cause GeoServer to hang. If there is a firewall between GeoServer and the database, a connection pool configured to shut down idle connections before the firewall can drop them will prevent GeoServer from hanging. This JNDI connection pool is configured to shut down idle connections after 5 to 10 minutes."

Kind regards,
Ben.

On 01/05/15 13:40, Mike Grogan wrote:

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting /
creating new image mosaics via REST ... that seems to be occurring after
time passes.

What I mean is that harvesting & creating new mosaics works fine most of
the time. However, it seems to fail after some amount of time passes. For
instance, if I have not created a new mosaic via REST for a day or 2, and I
try to create one, it will fail. If I then try to create several more via
REST, again, they all fail.

If I restart Tomcat or postgres and then run the very same REST call (via
gsconfig in python) against the very same data, voila, it is successful.
After restart, I can then go on creating multiple new mosaics via REST over
the course of many hours.

Let some time pass, though, and I'll start seeing failures again.

Yes, this is a troubleshooting nightmare.

My initial thoughts have been this has something to do with stale
connections via JNDI ... but I have validation turned on, both in JNDI
Tomcat Resource and in datastore.properties. I have tried leaving in the
eviction settings (shown in some of the JNDI Image Mosaic examples) and
removing them ... but this still doesn't help.

What's odd is that I have no problems accessing existing mosaic indexes
backed via JNDI. It just seems the ability to create new ones goes bad
after some time passes (usually 24 hours or so).

Again, JNDI has been a suspicion. I kind of also am asking myself if
something is somehow getting garbage collected that shouldn't be (I know
that sounds strange) ... or something else is somehow disappearing over
time that would prevent successful mosaic creation via REST.

Does this sound familiar to anyone? Any pointers?

The stack trace from when this happens is below.

Thanks!

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Mike_Grogan · May 4, 2015, 5:16pm

Ben,

Same machine; localhost connection.

So, I’m not sure where the idle connection is getting severed, if it even is. Everything seems to be suggesting that it is … though … with the passage of time seeming to cause the problem and database (JDBC) issues in the error logs.

Thanks,

Mike

···

On Mon, May 4, 2015 at 12:18 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Mike,

are tomcat and postgres on the same machine or is there a firewall between them?

I have seen Cisco(?) firewalls silently sever idle connections after an hour or so; the situation could not be recovered by connection validation. Note that this was app-schema WFS output and I have not used imagemosaic, but if this is a firewall problem a similar workaround for the database connection might help. Here is a JNDI Context that shuts down idle connections after 5 to 10 minutes:
http://docs.geoserver.org/latest/en/user/data/app-schema/data-stores.html#jndi

“Firewall timeouts can silently sever idle connections to the database and cause GeoServer to hang. If there is a firewall between GeoServer and the database, a connection pool configured to shut down idle connections before the firewall can drop them will prevent GeoServer from hanging. This JNDI connection pool is configured to shut down idle connections after 5 to 10 minutes.”

Kind regards,
Ben.

On 01/05/15 13:40, Mike Grogan wrote:

(GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)

I am experiencing another hard to track down issue with harvesting /
creating new image mosaics via REST … that seems to be occurring after
time passes.

What I mean is that harvesting & creating new mosaics works fine most of
the time. However, it seems to fail after some amount of time passes. For
instance, if I have not created a new mosaic via REST for a day or 2, and I
try to create one, it will fail. If I then try to create several more via
REST, again, they all fail.

If I restart Tomcat or postgres and then run the very same REST call (via
gsconfig in python) against the very same data, voila, it is successful.
After restart, I can then go on creating multiple new mosaics via REST over
the course of many hours.

Let some time pass, though, and I’ll start seeing failures again.

Yes, this is a troubleshooting nightmare.

My initial thoughts have been this has something to do with stale
connections via JNDI … but I have validation turned on, both in JNDI
Tomcat Resource and in datastore.properties. I have tried leaving in the
eviction settings (shown in some of the JNDI Image Mosaic examples) and
removing them … but this still doesn’t help.

What’s odd is that I have no problems accessing existing mosaic indexes
backed via JNDI. It just seems the ability to create new ones goes bad
after some time passes (usually 24 hours or so).

Again, JNDI has been a suspicion. I kind of also am asking myself if
something is somehow getting garbage collected that shouldn’t be (I know
that sounds strange) … or something else is somehow disappearing over
time that would prevent successful mosaic creation via REST.

Does this sound familiar to anyone? Any pointers?

The stack trace from when this happens is below.

Thanks!

–
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

[Geoserver-users] Image Mosaic REST harvesting - failure after time passes

--

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

--

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.