Hi List,
I have created a GeoServer setup with 10 separate Tomcat processes each running Geoserver (with HAProxy as a load balancer, running on Solaris x86). This works great most of the time, but sometimes on startup some of the instances are failing (see stack trace below).
From what I am seeing, it looks like each instance is reading its config from the shared data directory, and also each instance is reading and then writing out 3 files (wms.xml, wfs.xml and wcs.xml). Then there is a race - if another instance starts up, it reads only part of one of those files and gets upset. If I add a 5 second delay between starting each instance, the problem goes away (except on startup, which ignores my delay). Without a delay, between 0 and 4 instances fail to initialize.
Has anyone else run into this issue? And does anyone know why (and where!) these files are being written back out (without any changes)?
Thanks
Jack
2014-09-08 15:26:15,213 WARN [support.DisposableBeanAdapter] - Invocation of destroy method failed on bean with name ‘geoServerLoader’: org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name ‘metaTileCache’: Singleton bean creation not allowed while the singletons of this factory
are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
2014-09-08 15:26:15,225 ERROR [context.ContextLoader] - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘wmsServiceInterceptor-kmlReflector’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-kml-2.5.jar!/applicationContext.xml]: Cannot resolve reference to bean ‘wms’ while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘wms’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-wms-2.5.jar!/applicationContext.xml]: Cannot resolve reference to bean ‘geoServer’ while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘geoServer’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-main-2.5.jar!/applicationContext.xml]: Initialization of bean failed; nested exception is java.lang.RuntimeException: java.lang.NullPointerException
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:329)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:107)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1035)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:939)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
2014-09-08 15:19:46,975 WARN [turbojpeg.TurboJPEGMapResponse] - The turbo jpeg encoder is available for usage
2014-09-08 15:26:15,213 WARN [support.DisposableBeanAdapter] - Invocation of destroy method failed on bean with name ‘geoServerLoader’: org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name ‘metaTileCache’: Singleton bean creation not allowed while the singletons of this factory
are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
2014-09-08 15:26:15,225 ERROR [context.ContextLoader] - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘wmsServiceInterceptor-kmlReflector’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-kml-2.5.jar!/applicationContext.xml]: Cannot resolve reference to bean ‘wms’ while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘wms’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-wms-2.5.jar!/applicationContext.xml]: Cannot resolve reference to bean ‘geoServer’ while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘geoServer’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-main-2.5.jar!/applicationContext.xml]: Initialization of bean failed; nested exception is java.lang.RuntimeException: java.lang.NullPointerException
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:329)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:107)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1035)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:939)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:607)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:388)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:293)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
at org.geoserver.platform.GeoServerContextLoaderListener.contextInitialized(GeoServerContextLoaderListener.java:22)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1083)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1880)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘wms’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-wms-2.5.jar!/applicationContext.xml]: Cannot resolve reference to bean ‘geoServer’ while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘geoServer’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-main-2.5.jar!/applicationContext.xml]: Initialization of bean failed; nested exception is java.lang.RuntimeException: java.lang.NullPointerException
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:329)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:107)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:630)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1035)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:939)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:323)
… 31 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘geoServer’ defined in URL [jar:file:/opt1/geoserver/apache-tomcat-7.0.54/instance4/webapps/geoserver/WEB-INF/lib/gs-main-2.5.jar!/applicationContext.xml]: Initialization of bean failed; nested exception is java.lang.RuntimeException: java.lang.NullPointerException
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:527)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:323)
… 43 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at org.geoserver.config.GeoServerLoader.postProcessBeforeInitialization(GeoServerLoader.java:135)
at org.geoserver.config.GeoServerLoaderProxy.postProcessBeforeInitialization(GeoServerLoaderProxy.java:58)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:394)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1448)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
… 49 more
Caused by: java.lang.NullPointerException
at org.geoserver.config.GeoServerLoader.loadServices(GeoServerLoader.java:724)
at org.geoserver.config.GeoServerLoader.readConfiguration(GeoServerLoader.java:649)
at org.geoserver.config.DefaultGeoServerLoader.loadGeoServer(DefaultGeoServerLoader.java:72)
at org.geoserver.config.GeoServerLoader.postProcessBeforeInitialization(GeoServerLoader.java:129)
… 53 more