[Geoserver-users] HTTP ERROR 503 on startup after moving Geoserver data directory

Steffen_Reichel · October 20, 2021, 9:51am

Hi all,

after I moved Geoserver's data directory Geoserver is not starting
anymore. When trying to access the Geoserver web interface I get an 503
error.

The GEOSERVER_DATA_DIR is pointing to the correct location. Owner and
Access rights stayed the same. My OS is Linux. It seems that there is a
problem with GWC. In the logfile I see the error (full log message at
the end)

java.lang.IllegalStateException: Could not locate a layer or layer group
with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer
configuration, the GWC configuration seems to be out of synch

There is a configuration like this in the gwc-layers folder. And that
points to a layer that does not exists anymore. My question is, as the
web interface is not starting and thus I can't really change anything
via API, can I safely delete this folder? The documentation [1] states
that these files should not be edited by hand, but does not state if
it's safe to delete them. Or could I even empty the whole gwc-layers folder?

Kind regards,
Steffen

[1]
https://docs.geoserver.org/stable/en/user/geowebcache/config.html#configuration-files

-- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --

The full log message - I added line breaks for better readability and
snipped all the jar paths:

WARN [support.XmlWebApplicationContext] - Exception encountered during
context initialization - cancelling refresh attempt:

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'geopkgGetMapOutputFormat' defined in URL [..]:

Cannot resolve reference to bean 'webMapService' while setting
constructor argument; nested exception is

org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'wmsService2': FactoryBean threw exception
on object creation; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'wmsServiceInterceptor_CachingWMS' defined in URL [..]:

Cannot resolve reference to bean 'cachingWebMapService' while setting
bean property 'advice'; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'cachingWebMapService' defined in URL [..]:

Cannot resolve reference to bean 'gwcFacade' while setting constructor
argument; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'gwcFacade' defined in URL [..]:

Cannot resolve reference to bean 'DiskQuotaMonitor' while setting
constructor argument; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'DiskQuotaMonitor' defined in URL [..]:

Invocation of init method failed; nested exception is

java.lang.IllegalStateException: Could not locate a layer or layer group
with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer
configuration, the GWC configuration seems to be out of synch

Russ · October 20, 2021, 10:16am

Are you running under Linux/Tomcat?

In the latest(?) Tomcat you need to explicitly give permission for Tomcat to rad the data folder.
It is on a file called /lib/systemd/system/tomcat.service

You need to add a line;
ReadWritePaths=<path to your data folder>

e.g.
ReadWritePaths=/data/

Russ

On 20 Oct 2021, at 10:51, Steffen Reichel <st_reichel@anonymised.com> wrote:

Hi all,

after I moved Geoserver's data directory Geoserver is not starting
anymore. When trying to access the Geoserver web interface I get an 503
error.

The GEOSERVER_DATA_DIR is pointing to the correct location. Owner and
Access rights stayed the same. My OS is Linux. It seems that there is a
problem with GWC. In the logfile I see the error (full log message at
the end)

java.lang.IllegalStateException: Could not locate a layer or layer group
with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer
configuration, the GWC configuration seems to be out of synch

There is a configuration like this in the gwc-layers folder. And that
points to a layer that does not exists anymore. My question is, as the
web interface is not starting and thus I can't really change anything
via API, can I safely delete this folder? The documentation [1] states
that these files should not be edited by hand, but does not state if
it's safe to delete them. Or could I even empty the whole gwc-layers folder?

Kind regards,
Steffen

[1]
https://docs.geoserver.org/stable/en/user/geowebcache/config.html#configuration-files

-- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --

The full log message - I added line breaks for better readability and
snipped all the jar paths:

WARN [support.XmlWebApplicationContext] - Exception encountered during
context initialization - cancelling refresh attempt:

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'geopkgGetMapOutputFormat' defined in URL [..]:

Cannot resolve reference to bean 'webMapService' while setting
constructor argument; nested exception is

org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'wmsService2': FactoryBean threw exception
on object creation; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'wmsServiceInterceptor_CachingWMS' defined in URL [..]:

Cannot resolve reference to bean 'cachingWebMapService' while setting
bean property 'advice'; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'cachingWebMapService' defined in URL [..]:

Cannot resolve reference to bean 'gwcFacade' while setting constructor
argument; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'gwcFacade' defined in URL [..]:

Cannot resolve reference to bean 'DiskQuotaMonitor' while setting
constructor argument; nested exception is

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'DiskQuotaMonitor' defined in URL [..]:

Invocation of init method failed; nested exception is

java.lang.IllegalStateException: Could not locate a layer or layer group
with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer
configuration, the GWC configuration seems to be out of synch

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Steffen_Reichel · October 20, 2021, 11:53am

Hi,

thank you for your quick answer.

On 20.10.21 12:16, Russ Hore wrote:

Are you running under Linux/Tomcat?

Jetty behind an Apache proxy on Linux

In the latest(?) Tomcat you need to explicitly give permission for Tomcat to rad the data folder.
It is on a file called /lib/systemd/system/tomcat.service

You need to add a line;
ReadWritePaths=<path to your data folder>

The path is writable. The logfile is also stored there and can be
written to. Geoserver also loads the layers, styles (at least the log
says so) etc. before crashing with the error described.

I was mainly wondering if it is safe to delete the folder (or empty the
whole gcw-layers/ folder), or not and if so, what configurations I'd
have to redo to set up GWC properly again for all layers.

Cheers,
Steffen

Russ · October 20, 2021, 12:03pm

Ah, I am not sure about Jetty but for Tomcat, it does not matter if the data folder is 777/rwxrwxrwx, if it is not listed in the tomcat.service file, it can't be written to, as far as I am aware under Tomcat9.

Russ

On 20 Oct 2021, at 12:53, Steffen Reichel <st_reichel@anonymised.com> wrote:

Hi,

thank you for your quick answer.

On 20.10.21 12:16, Russ Hore wrote:

Are you running under Linux/Tomcat?

Jetty behind an Apache proxy on Linux

In the latest(?) Tomcat you need to explicitly give permission for Tomcat to rad the data folder.
It is on a file called /lib/systemd/system/tomcat.service

You need to add a line;
ReadWritePaths=<path to your data folder>

The path is writable. The logfile is also stored there and can be
written to. Geoserver also loads the layers, styles (at least the log
says so) etc. before crashing with the error described.

I was mainly wondering if it is safe to delete the folder (or empty the
whole gcw-layers/ folder), or not and if so, what configurations I'd
have to redo to set up GWC properly again for all layers.

Cheers,
Steffen

Steffen_Reichel · October 21, 2021, 9:24am

On 20.10.21 14:03, Russ Hore wrote:

Ah, I am not sure about Jetty but for Tomcat, it does not matter if
the data folder is 777/rwxrwxrwx, if it is not listed in the
tomcat.service file, it can't be written to, as far as I am aware
under Tomcat9.

I'm 100% sure that Jetty / the Servlet can write to the data directory,
because can I see it doing it - the logfile for example is in there.

Also the mountpoint of the datafolder did not change - my question may
have been formulated a bit unprecise. I moved the data do a different
partition (was running out of space on the old one) and mounted it to
the same mountpoint as the old one. So from the point of view of Jetty
nothing changed. There is simply a different partition behind the same
mountpoint.

Thus my question still stands: Can I safely delete the offending
configuration folder in gwc-layers or will this mess up something in the
GWC configuration and make the situation worse? The error I'm getting is:

java.lang.IllegalStateException: Could not locate a layer or layer group
with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer
configuration, the GWC configuration seems to be out of synch

Kind regards,
Steffen

Russ · October 21, 2021, 10:00am

Also the mountpoint of the datafolder did not change - my question may
have been formulated a bit unprecise. I moved the data do a different

Ah I see. That makes more sense. But that means I am out of ideas.

Russ

On 21 Oct 2021, at 10:24, Steffen Reichel <st_reichel@anonymised.com> wrote:

On 20.10.21 14:03, Russ Hore wrote:

Ah, I am not sure about Jetty but for Tomcat, it does not matter if
the data folder is 777/rwxrwxrwx, if it is not listed in the
tomcat.service file, it can't be written to, as far as I am aware
under Tomcat9.

I'm 100% sure that Jetty / the Servlet can write to the data directory,
because can I see it doing it - the logfile for example is in there.

Also the mountpoint of the datafolder did not change - my question may
have been formulated a bit unprecise. I moved the data do a different
partition (was running out of space on the old one) and mounted it to
the same mountpoint as the old one. So from the point of view of Jetty
nothing changed. There is simply a different partition behind the same
mountpoint.

Thus my question still stands: Can I safely delete the offending
configuration folder in gwc-layers or will this mess up something in the
GWC configuration and make the situation worse? The error I'm getting is:

java.lang.IllegalStateException: Could not locate a layer or layer group
with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer
configuration, the GWC configuration seems to be out of synch

Kind regards,
Steffen

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Humphries_Graham · October 21, 2021, 9:58pm

Why don't you try moving the offending directory to another location on the file system and try to start Geoserver. If it is worse than before you can then move it back

-----Original Message-----
From: Steffen Reichel <st_reichel@anonymised.com>
Sent: Thursday, 21 October 2021 8:25 PM
To: geoserver-users@lists.sourceforge.net
Subject: Re: [Geoserver-users] HTTP ERROR 503 on startup after moving Geoserver data directory

On 20.10.21 14:03, Russ Hore wrote:

Ah, I am not sure about Jetty but for Tomcat, it does not matter if
the data folder is 777/rwxrwxrwx, if it is not listed in the
tomcat.service file, it can't be written to, as far as I am aware
under Tomcat9.

I'm 100% sure that Jetty / the Servlet can write to the data directory, because can I see it doing it - the logfile for example is in there.

Also the mountpoint of the datafolder did not change - my question may have been formulated a bit unprecise. I moved the data do a different partition (was running out of space on the old one) and mounted it to the same mountpoint as the old one. So from the point of view of Jetty nothing changed. There is simply a different partition behind the same mountpoint.

Thus my question still stands: Can I safely delete the offending configuration folder in gwc-layers or will this mess up something in the GWC configuration and make the situation worse? The error I'm getting is:

java.lang.IllegalStateException: Could not locate a layer or layer group with id LayerInfoImpl--fd40ab5:166e9af110c:-273f within GeoServer configuration, the GWC configuration seems to be out of synch

Kind regards,
Steffen

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ianturton.com%2Ftalks%2Ffoss4g.html%23%2F&data=04|01|graham.humphries%40stategrowth.tas.gov.au|31321d4b1f3a4fbb563508d99474dd91|64ebab8accf44b5ca2d32b4e972d96b2|0|0|637704052998412179|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=jJXinvicME%2Bj3MnzaQSr%2FWPYNireBrPpsFYO9PtCu%2B4%3D&reserved=0
- The GeoServer user list posting guidelines: https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgeoserver.org%2Fcomm%2Fuserlist-guidelines.html&data=04|01|graham.humphries%40stategrowth.tas.gov.au|31321d4b1f3a4fbb563508d99474dd91|64ebab8accf44b5ca2d32b4e972d96b2|0|0|637704052998422174|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=KRg23F%2F2xS2yvwd3V%2FJCacBxAi6EuWULDAEUQdkmDHg%3D&reserved=0

If you want to request a feature or an improvement, also see this: https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeoserver%2Fgeoserver%2Fwiki%2FSuccessfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer&data=04|01|graham.humphries%40stategrowth.tas.gov.au|31321d4b1f3a4fbb563508d99474dd91|64ebab8accf44b5ca2d32b4e972d96b2|0|0|637704052998422174|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=gEQUDw7Yq1PL5m37rYsFdQ2eSoyRUG2ruA4hwCypURw%3D&reserved=0

Geoserver-users@lists.sourceforge.net
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-users&data=04|01|graham.humphries%40stategrowth.tas.gov.au|31321d4b1f3a4fbb563508d99474dd91|64ebab8accf44b5ca2d32b4e972d96b2|0|0|637704052998422174|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=DXhOydoVNHf1FBDAamePM2YtIvohRFMM%2BU431jnsXXQ%3D&reserved=0

________________________________

CONFIDENTIALITY NOTICE AND DISCLAIMER
The information in this transmission may be confidential and/or protected by legal professional privilege, and is intended only for the person or persons to whom it is addressed. If you are not such a person, you are warned that any disclosure, copying or dissemination of the information is unauthorised. If you have received the transmission in error, please immediately contact this office by telephone, fax or email, to inform us of the error and to enable arrangements to be made for the destruction of the transmission, or its return at our cost. No liability is accepted for any unauthorised use of the information contained in this transmission.

Clint_Hastings · October 22, 2021, 1:36pm

Easier to just rename the top level directory, like adding NOT at the end.

On Thursday, October 21, 2021, 04:02:36 PM MDT, Humphries, Graham graham.humphries@anonymised.com wrote:

Why don’t you try moving the offending directory to another location on the file system and try to start Geoserver. If it is worse than before you can then move it back

-----Original Message-----
From: Steffen Reichel <st_reichel@anonymised.com>
Sent: Thursday, 21 October 2021 8:25 PM
To: geoserver-users@anonymised.comrceforge.net
Subject: Re: [Geoserver-users] HTTP ERROR 503 on startup after moving Geoserver data directory

On 20.10.21 14:03, Russ Hore wrote:

Ah, I am not sure about Jetty but for Tomcat, it does not matter if
the data folder is 777/rwxrwxrwx, if it is not listed in the
tomcat.service file, it can’t be written to, as far as I am aware
under Tomcat9.
I’m 100% sure that Jetty / the Servlet can write to the data directory, because can I see it doing it - the logfile for example is in there.

Also the mountpoint of the datafolder did not change - my question may have been formulated a bit unprecise. I moved the data do a different partition (was running out of space on the old one) and mounted it to the same mountpoint as the old one. So from the point of view of Jetty nothing changed. There is simply a different partition behind the same mountpoint.

Thus my question still stands: Can I safely delete the offending configuration folder in gwc-layers or will this mess up something in the GWC configuration and make the situation worse? The error I’m getting is:

java.lang.IllegalStateException: Could not locate a layer or layer group with id LayerInfoImpl–fd40ab5:166e9af110c:-273f within GeoServer configuration, the GWC configuration seems to be out of synch

Kind regards,
Steffen

Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:

If you want to request a feature or an improvement, also see this: https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeoserver%2Fgeoserver%2Fwiki%2FSuccessfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer&data=04%7C01%7Cgraham.humphries%40stategrowth.tas.gov.au%7C31321d4b1f3a4fbb563508d99474dd91%7C64ebab8accf44b5ca2d32b4e972d96b2%7C0%7C0%7C637704052998422174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gEQUDw7Yq1PL5m37rYsFdQ2eSoyRUG2ruA4hwCypURw%3D&reserved=0

Geoserver-users@lists.sourceforge.net
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-users&data=04%7C01%7Cgraham.humphries%40stategrowth.tas.gov.au%7C31321d4b1f3a4fbb563508d99474dd91%7C64ebab8accf44b5ca2d32b4e972d96b2%7C0%7C0%7C637704052998422174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DXhOydoVNHf1FBDAamePM2YtIvohRFMM%2BU431jnsXXQ%3D&reserved=0

CONFIDENTIALITY NOTICE AND DISCLAIMER
The information in this transmission may be confidential and/or protected by legal professional privilege, and is intended only for the person or persons to whom it is addressed. If you are not such a person, you are warned that any disclosure, copying or dissemination of the information is unauthorised. If you have received the transmission in error, please immediately contact this office by telephone, fax or email, to inform us of the error and to enable arrangements to be made for the destruction of the transmission, or its return at our cost. No liability is accepted for any unauthorised use of the information contained in this transmission.

Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:

Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users