Hi devs,
I’ll be starting the 2.27.0 release train on Monday.
We have two outstanding pull requests:
The former should be a no brainer.
As for the latest, it should fix the problem Jody mentioned in the last PMC meeting, and also allow GeoServer Cloud to transition to the GeoServer Loader with no custom code.
Please give it a final test with your data directories and we’re good to go with the release.
Cheers,
Gabe
Hi Gabriel,
those two are now in.
I seem to remember we had a startup issue reported during the last PSC meeting, was that addressed by [GEOS-11790] Data Directory Loader Consistency Issues by groldan · Pull Request #8449 · geoserver/geoserver · GitHub?
Cheers
Andrea
One more from me: [GEOS-11792] Fix welcome page listing all services via sevice caps by jodygarnett · Pull Request #8455 · geoserver/geoserver · GitHub
Simple fix, some code was being too smart for its own good, annoying but not critical.
It also looks like @sikeoka has a PR addressing feedback from CSP testing: [GEOS-11788] Apply feedback from CSP testing by sikeoka · Pull Request #8443 · geoserver/geoserver · GitHub
Hi all
I had a look at the loading issue again, by the looks of the changes in the PR, I don’t think it was addressed.
As a reminder, the full stack trace is here:
We see a circularity, the catalog bean is being created, which in turn triggers the GeoServerLoaderProxy,
which calls upon the DataDirectoryGeoServerLoader to actually populate the catalog contents, which
in turn does an extension lookup for EntityResolverProvider, which needs GeoServerImpl (constructor
injection), which in turn has the catalog injected as a property.
Now one might think that would always trigger a circularity, but in my case, the bean creation works fine,
the EntityResolverProvider is injected a GeoServerImpl that is in state of creation, and still does not have
the catalog property set in. The catalog property is actually injected after the calog is loaded, and there
are no issues.
Specifically, I see the following bean creation chain:
geoServer (id 7611) → localWorkspaceCatalog → advertisedCatalog → accessRuleDao → rawCatalog → GeoServerLoaderProxy.postProcessBeforeInitialization → DataDirectoryGeoServerLoader.loadCatalog → entityResolverProvider creation (which is given a GeoServer instance with id 7611) as part of the DataDirectoryGeoServerLoader pre-init phase.
I have attached a customized logging profile that tracks bean creation, which can be used to compare startup sequences.
To run a test, copy the release data directory, add the logging profile and configure it in logging.xml, then run docker
with something like this (adapt the position of your local data dir):
docker run -it --rm -p8080:8080 --mount type=bind,src=/home/aaime/tmp/spring-bean-debug,target=/opt/geoserver_data --env INSTALL_EXTENSIONS=true --env STABLE_EXTENSIONS="ysld,ogcapi-features" [docker.osgeo.org/geoserver:2.27.x](http://docker.osgeo.org/geoserver:2.27.x)
Running the above, I see the following bean and catalog initialization order:
30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wcs20Extension'
30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wcs20ServiceTarget'
**30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'geoServer'**
**30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'localWorkspaceCatalog'**
30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'advertisedCatalog'
30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'accessRulesDao'
30 043 13:43:03 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'rawCatalog'
**30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'entityResolverProvider'**
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wmsCatalogValidator'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'sldHandler'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'sldPackageHandler'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'ysldHandler'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'ysldUomMapper'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'gwcZoomContextFinder'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'gwcGridSetBroker'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'gwcXmlConfig'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'gwcAppCtx'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'gwcXmlConfigResourceProvider'
30 043 13:43:04 CONFIG [gwc.config] - Will look for 'geowebcache.xml' in directory '/opt/geoserver_data/gwc'.
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'GWCGeoServerConfigurationProvider'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'GWCGeoServerRESTConfigurationProvider'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'localWorkspaceCatalog'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'virtualTableCallback'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'cascadedStoredQueryCallback'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wmtsLoader'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wfsLoader'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wmsLoader'
30 043 13:43:04 DEBUG [support.DefaultListableBeanFactory] - Creating shared instance of singleton bean 'wcsLoader'
30 043 13:43:04 CONFIG [geoserver.config] - Loading catalog /opt/geoserver_data
30 043 13:43:05 CONFIG [config.datadir] - Catalog and configuration loader uses 16 threads out of 16 available cores.
30 043 13:43:05 CONFIG [config.datadir] - Loaded style raster
30 043 13:43:05 CONFIG [config.datadir] - Loaded style line
30 043 13:43:05 CONFIG [config.datadir] - Loaded style green
30 043 13:43:05 CONFIG [config.datadir] - Loaded style burg
30 043 13:43:05 CONFIG [config.datadir] - Loaded style polygon
30 043 13:43:05 CONFIG [config.datadir] - Loaded style cite_lakes
30 043 13:43:05 CONFIG [config.datadir] - Loaded style poi
30 043 13:43:05 CONFIG [config.datadir] - Loaded style simple_roads
I’ve also attached the full startup logs to this mail.
Jody (or anyone else having startup problems, really) can you share similar logs for your failing startup case?
SPRING_BEAN_LOGGING.xml (3.63 KB)
geoserver.log (157 KB)
Hi Andrea and team,
Thanks for your detailed analysis of the issue. After looking at your logs and doing additional investigation,
I think I’ve found the root cause of the circular dependency problem.
The problem is exactly in the preLoadExtensions(EntityResolverProvider.class) call in DataDirectoryGeoServerLoader.initializeDependencies().
This call is trying to get all instances of EntityResolverProvider during catalog initialization, creating a circular dependency chain:
I’ve confirmed that this call is unnecessary during catalog loading. There’s no code path during catalog loading that requires EntityResolverProvider:
In your debug logs, EntityResolverProvider seems to be creating fine, but in Jody’s environment (and potentially others), it’s causing a circular
dependency exception. This likely happens due to differences in how Spring resolves beans in different environments or subtle timing variations.
I couldn’t reproduce it on a variety of linux and macos machines though, but the strange thing is while debugging the startup process, I’ll get the
EntityResolverProvider instantiated with a null argument instead of the GeoServer bean.
I’ve created this pull request that removes the preloading call with a detailed comment explaining why it’s safe:
// EntityResolverProvider is intentionally not preloaded to avoid circular dependency with GeoServer bean
// The EntityResolverProvider isn’t used during catalog loading, and ResourcePoolInitializer
// will properly set it in the ResourcePool after catalog loading is complete
This doesn’t change how EntityResolverProvider is used after startup. It’s still properly set by ResourcePoolInitializer after the catalog is loaded:
gs.getCatalog().getResourcePool().setEntityResolverProvider(resolverProvider);
Jody, does this fix resolve the startup issue for you? If you can test with the updated code, that would be great confirmation.
Regards,
Gabriel
@groldan I have only ever encountered the problem using the docker nightly 2.27.x. So I am not in a good position to tell you if your pull-request resolves the issue.
ok, so I’m pretty confident about the fix.
As we’re stuck on the release I’ll merge it and let you test the nightly again
Happy to test.
I do not think we are stuck on release, with have a fallback setting in the update instructions, and we always have the option of defaulting to off ( for both startup and CSP). The no workspace issue is only annoying not a blocker.
Let’s test and coordinate today.
Jody