3.0-RC: Feedback Cannot Create Blob

GS 3 RC, Tomcat11, Debian Trixie. Trying to create a blob, directory as V2 re security (where blob been created ok) .

using cach inside the geoserver/data, ie /var/lib/tomcat11/webapps/geoserver/data/cache fails

Message dump (large) is java.lang.IllegalArgumentException: No configuration found capable of saving FileBlobStore\[id:nvme, enabled:true, baseDirectory:/var/lib/tomcat11/webapps/geoserver/data, fileSystemBlockSize:4096\]
  at org.geowebcache.storage.BlobStoreAggregator.addBlobStore(BlobStoreAggregator.java:249)
  at org.geoserver.gwc.GWC.addBlobStore(GWC.java:2387)
  at org.geoserver.gwc.web.blob.BlobStorePage.save(BlobStorePage.java:215)
  at org.geoserver.gwc.web.blob.BlobStorePage$SaveLink.onSubmit(BlobStorePage.java:316)
  at org.apache.wicket.ajax.markup.html.form.AjaxSubmitLink$1.onSubmit(AjaxSubmitLink.java:112)
  at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:249)
  at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1390)
  at org.apache.wicket.markup.html.form.Form.process(Form.java:1022)
  at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:842)
  at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:202)
  at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:147)
  at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:630)
  at org.apache.wicket.core.request.handler.ListenerRequestHandler.internalInvoke(ListenerRequestHandler.java:300)
  at org.apache.wicket.core.request.handler.ListenerRequestHandler.invoke(ListenerRequestHandler.java:274)
  at org.apache.wicket.core.request.handler.ListenerRequestHandler.invokeListener(ListenerRequestHandler.java:222)
  at org.apache.wicket.core.request.handler.ListenerRequestHandler.respond(ListenerRequestHandler.java:202)
  at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:913)
  at org.apache.wicket.request.RequestHandlerExecutor.execute(RequestHandlerExecutor.java:63)
  at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:294)
  at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:255)
  at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:277)
  at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:208)
  at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
  at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:649)
  at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:710)
  at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:162)
  at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
  at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:49)
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:866)
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1000)
  at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:903)
  at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:649)
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:874)
  at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:710)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:128)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:70)
  at org.geoserver.rest.SuffixStripFilter.doFilterInternal(SuffixStripFilter.java:79)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:67)
  at org.geoserver.ows.HTTPHeadersCollector.doFilter(HTTPHeadersCollector.java:48)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:67)
  at org.geoserver.filters.HTTPMethodFilter.doFilter(HTTPMethodFilter.java:34)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:67)
  at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:180)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:67)
  at org.geoserver.monitor.MonitorFilter.doFilter(MonitorFilter.java:103)
  at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:67)
  at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:41)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:39)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:235)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:67)
  at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:71)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:88)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:67)
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:71)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:88)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
  at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:52)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:67)
  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:245)
  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:239)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:71)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:88)
  at org.geoserver.security.filter.GeoServerUserNamePasswordAuthenticationFilter.doFilter(GeoServerUserNamePasswordAuthenticationFilter.java:119)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:67)
  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:115)
  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:106)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:71)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:88)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:67)
  at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilterInternal(GeoServerSecurityContextPersistenceFilter.java:66)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
  at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:71)
  at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:88)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:237)
  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:195)
  at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:139)
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:355)
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:272)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.filters.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:116)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:46)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:47)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:41)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:199)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:165)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:77)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:113)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:83)
  at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:654)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:72)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1778)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
  at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:946)
  at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:480)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:57)
  at java.base/java.lang.Thread.run(Thread.java:840)

Thanks for the testing and feedback! I do not see anything odd about your Fileblob store.Can you double check permissions on your /var/lib/tomcat11/webapps/geoserver/data/cache folder?

Notes trying to reproduce your issue:

  • When I start up GeoServer 3 there are no blobstores configured.

    aside: That is actually a useful configuration as just having tiles with the correct headers trusts downstream caches (such as browser cache) to keep generated tiles for the correct amount of time.

  • When I set one up FileBlobStore is an option, did you configure a blob store?

    I chose the geoserver data directory: gwc/tmp folder as I am testing in docker. Often your system would have a useful temp folder configured?

  • And then use the blobstore:

  • And turn on diskquota so it tracks space used:

drwxr-xr-x 2 tomcat tomcat 4096 Apr 21 13:22 cache

I go to the create blobstore page, enter all the required parameters, and press save. the error appears. cannot use the blobstore as one has not been created.

on return to the blobstore page, all parameters are there, except for the Base Directory, its now blank

Hi,

I noticed you are using the internal data directory, withing the Tomcat geoServer deployment.
Wondering if this is a restriction Tomcat 11 is placing? Having an internal data dir used to be feasible, and it’s ok for a throaway test, but we strongly recommend using an external data directory for any serious usage.
Maybe that will also solve the permission issue?

Cheers
Andrea

Hi,

Actually its not the intended directory, the operational one is /mnt/nvme/geoserver, the device is an nvme. This is identical to the v2 setup, currently running in another system. The initial directory on V3 was set to /mnt/nvme/geoserver The idea of putting down the war goserver directory was to see if it made a difference, it didnt :frowning:

At the moment my many layers all use the BlobFile and so this is a bit of a show stopper.

Tks

Aimee

Hi,

out of curiosity I tried GeoServer 3 RC on a Tomcat 11.0.21, JDK 17, and created a file blobstore without problems:

I’ve then attached this blob storage to a layer group and produced some tiles, no issue found:

Since the error message talks about a permission issue, I’m wondering if it’s coming from a different moving part. How are you deploying GeoServer? Wondering if maybe the Tomcat 11 docker image is not running Tomcat with root privileges anymore, which could explain why you cannot write on random folders? Just an idea, I invite you to explore your setup for whatever is the reason for the tightneted security (the error message is clear, GeoServer does not have write permissions).

Cheers
Andrea

Hi
the set up is WAR based, no dockers used.
This is the operational V2 blob directory
ls -lt /mnt/nvme/
drwxr-xr-x 6 tomcat tomcat 4096 Oct 7 2024 cache
and the version 3

ls -lt /mnt/nvme/
dwxr-xr-x 6 tomcat tomcat 4096 Apr 23 08:24 cache

Base Error
java.lang.IllegalArgumentException: No configuration found capable of saving FileBlobStore id:nvme, enabled:true,

suggests it a configuration error , not permissions?
I replaced the various gwc xml files from the v2 setup, however even that did not
show the blobfile definition in the gui, completely blank, much to my surprise.
One point i noticed in the v2 version, there is a directory named diskquota_page_store_hsql, is this the same in v3?

Yes, this is the first inetrpreatation of the error message. However that’s a generic message, the code is going through configuration storage options and checking if each one can store the config.
And none of the ones available says “yes”… internally some of these check if the target directory exists and is writable, hence the suggestion to check permissions.

In my case and Jody’s, this does not happen, and really, should not happen, but I don’t know what’s happening on your end.

You say it’s a war deployment. Ok, is it a vanilla one? Are you adding any customization on top of it? Particular extensions or community modules, any other manual change?

Cheers
Andrea

Hi

Created a new VM, installed jdk17, tomcat11, installed geoserver (same one as used on the other server).

created directory /mnt/nvme/cache, owner on nvme & cache is tomcat.

Created blob, failed with a message this time, not a java exception dump.

Issue with not writable directory!

Cant change /mnt, so created /home/geoserver/nvme/cache, owner tomcat (not home still owned by root), created blob file

Hi

The broken one was not vanilla, i added all the required modules where applicable from the list on the v2 system, non afaik. only one on the v3 release is community?, not installed as not required.

the new one in my recent post is vanilla, at the moment.

Anyway i am going to cut my losses, use the new one and hope for the best!!

Nice troubleshooting, when you added other ones from the list what were they?

my thinking: I am curious if one of the other blobstores is failing to be loaded, and thus causing geowebcache to give up and return the generic message. Do you have any other blob store configured like gwc-s3?

No other blob related modules installed.

I am going to spend some time backing up, then copying on the workspace from v2, and see if it works. This will take a while, and I will get back when it works. (or not)

As v2 got a bit muddled, i will install the modules as required. The workspace has various sources such as fixed files and a postgresql database thematic version of openstreetmap,

Hi

Copied the workspace over. only glitch was the cache directory on /mnt/nvme/cache, which whatever i tried would prevent start up. changed the xml, now its on /home/geoserver/nvme/cache.

Moved the VM onto the live system, placed the VM on the nvme device, now testing & tuning.

Thanks for all your help

Aimee