[Geoserver-users] crash after disk full

GeoServer GeoServer User
1

After the root partition of the server where an instance of geoserver (2.8 master) is running became full, I'm no more able to run it.
Many configuration files are lost (wcs.xml, wfs.xml, wms.xml have zero size), the connections to postgis is no more working and in the geoserver log I found this WARN as last notification:

WARN [support.DisposableBeanAdapter] - Invocation of destroy method failed on bean with name 'geoServerLoader': org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'geoWebCacheExtensions': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

2 questions:

1. Is that the expected behaviour?
2. Is there some kind of recovery procedure?
3. I don't have a recent backup of the "data" directory.Is there a way to understand what files are corrupted or lost?

Thank you
  maxx

2

On Sun, Jun 14, 2015 at 11:23 PM, emmexx <emmexx@anonymised.com> wrote:

After the root partition of the server where an instance of geoserver
(2.8 master) is running became full, I'm no more able to run it.
Many configuration files are lost (wcs.xml, wfs.xml, wms.xml have zero
size), the connections to postgis is no more working and in the
geoserver log I found this WARN as last notification:

WARN [support.DisposableBeanAdapter] - Invocation of destroy method
failed on bean with name 'geoServerLoader':
org.springframework.beans.factory.BeanCreationNotAllowedException: Error
creating bean with name 'geoWebCacheExtensions': Singleton bean creation
not allowed while the singletons of this factory are in destruction (Do
not request a bean from a BeanFactory in a destroy method implementation!)

2 questions:

1. Is that the expected behaviour?

In case of disk full there is no expected behavior, we do not have tests
covering that case, nor I think the configuration subsystem is designed
to cope with such case.

Generally speaking, config wise, all we do is to write changes to
a temporary file, and then move the file over the previous one, using
an atomic rename on *nix systems, and a "delete and move" one
on Windows where atomic renames are not available.

Code here, if anyone as suggestions on how to make it better, it
would be appreciated:
https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/data/util/IOUtils.java#L480

That said... as far as I know, we only change one file at a time,
and fail in case of IO error so... if you have many files corrupted, I may
be
that you had something that insisted on making changes on the configuration
despite the errors, and ended up ruining multiple files?

2. Is there some kind of recovery procedure?

If the file contents is lost... it's lost.
For the service files (wms.xml and friends) you can find copies you
can restart from here:
https://github.com/geoserver/geoserver/tree/master/data/release

Hopefully those are the only you have ruined.

3. I don't have a recent backup of the "data" directory.Is there a way
to understand what files are corrupted or lost?

No guidelines here either, but a 0 length file is a good indicator
that the file is gone

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

3

Il 06/15/2015 09:01 AM, Andrea Aime scrisse:

That said... as far as I know, we only change one file at a time,
and fail in case of IO error so... if you have many files corrupted, I
may be
that you had something that insisted on making changes on the configuration
despite the errors, and ended up ruining multiple files?

I insisted restarting tomcat. But shouldn't geoserver stop upon finding one of the xml files empty?

    2. Is there some kind of recovery procedure?

If the file contents is lost... it's lost.
For the service files (wms.xml and friends) you can find copies you
can restart from here:
https://github.com/geoserver/geoserver/tree/master/data/release

Hopefully those are the only you have ruined.

I already copied them but the postgis password is still missing and geoserver stops anyway.

Thank you
   maxx

4

On Mon, Jun 15, 2015 at 9:09 AM, emmexx <emmexx@anonymised.com> wrote:

Il 06/15/2015 09:01 AM, Andrea Aime scrisse:
> That said... as far as I know, we only change one file at a time,
> and fail in case of IO error so... if you have many files corrupted, I
> may be
> that you had something that insisted on making changes on the
configuration
> despite the errors, and ended up ruining multiple files?

I insisted restarting tomcat. But shouldn't geoserver stop upon finding
one of the xml files empty?

Nope, normally the startup is tolerant, tries to get over failures.

>
> 2. Is there some kind of recovery procedure?
>
>
> If the file contents is lost... it's lost.
> For the service files (wms.xml and friends) you can find copies you
> can restart from here:
> https://github.com/geoserver/geoserver/tree/master/data/release
>
> Hopefully those are the only you have ruined.

I already copied them but the postgis password is still missing and
geoserver stops anyway.

Hum... I guess you also have the security directory ruined then?
GeoServer "stops"? It should reach the end of the startup and
fail to connect to postgis...

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Il 06/15/2015 09:39 AM, Andrea Aime scrisse:

Hum... I guess you also have the security directory ruined then?
GeoServer "stops"? It should reach the end of the startup and
fail to connect to postgis...

The last notification in the geoserver log is the warn I put in my OP.

After overwriting the 3 empty xml files and renaming gwc-gs.xml the last notification is:
INFO [geoserver.security] - Reloading user/groups successful for service named default

I can post all the logs if it can be useful.

bye
  maxx

6

Hi,
I’m afraid to help you one would need to have actual access to your
GeoServer installation (which requires a lot of dedicated time, not something
normally done on the mailing list).

Cheers
Andrea

···

On Mon, Jun 15, 2015 at 9:56 AM, emmexx <emmexx@anonymised.com> wrote:

Il 06/15/2015 09:39 AM, Andrea Aime scrisse:

Hum… I guess you also have the security directory ruined then?
GeoServer “stops”? It should reach the end of the startup and
fail to connect to postgis…

The last notification in the geoserver log is the warn I put in my OP.

After overwriting the 3 empty xml files and renaming gwc-gs.xml the last
notification is:
INFO [geoserver.security] - Reloading user/groups successful for service
named default

I can post all the logs if it can be useful.

bye

maxx

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

7

Il 06/15/2015 10:27 AM, Andrea Aime scrisse:

I'm afraid to help you one would need to have actual access to your
GeoServer installation (which requires a lot of dedicated time, not
something
normally done on the mailing list).

Sorry, I meant to understand how to solve the problem in general, non to help me. :slight_smile:

bye
  maxx

8

If it were me, I'd do a fresh install. If the server is important, I'd also learn the importance of backup and configuration management, maybe even surveillance :slight_smile:

/julian
________________________________________
Fra: emmexx [emmexx@anonymised.com]
Sendt: 15. juni 2015 10:34
Til: geoserver-users@lists.sourceforge.net
Emne: Re: [Geoserver-users] crash after disk full

Il 06/15/2015 10:27 AM, Andrea Aime scrisse:

I'm afraid to help you one would need to have actual access to your
GeoServer installation (which requires a lot of dedicated time, not
something
normally done on the mailing list).

Sorry, I meant to understand how to solve the problem in general, non to
help me. :slight_smile:

bye
        maxx

------------------------------------------------------------------------------
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

9

I guess you can post them somewhere on the internet and then
link them, the mailing list won’t accept large attachments

Cheers
Andrea

···

On Mon, Jun 15, 2015 at 10:34 AM, emmexx <emmexx@anonymised.com> wrote:

Il 06/15/2015 10:27 AM, Andrea Aime scrisse:

I’m afraid to help you one would need to have actual access to your
GeoServer installation (which requires a lot of dedicated time, not
something
normally done on the mailing list).

Sorry, I meant to understand how to solve the problem in general, non to
help me. :slight_smile:

bye
maxx

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

10

Il 06/15/2015 10:38 AM, Julian Hollingbery scrisse:

If it were me, I'd do a fresh install. If the server is important,
I'd also learn the importance of backup and configuration management,
maybe even surveillance:-)

I know. But...

I'm not criticizing geoserver developers but I don't know of any app that deletes or overwrites configuration files because the disk is full. Configuration files should be editable only by users and read-only by the application.

bye
  maxx

11

On Mon, Jun 15, 2015 at 11:44 AM, emmexx <emmexx@anonymised.com> wrote:

Il 06/15/2015 10:38 AM, Julian Hollingbery scrisse:
> If it were me, I'd do a fresh install. If the server is important,
> I'd also learn the importance of backup and configuration management,
> maybe even surveillance:-)

I know. But...

I'm not criticizing geoserver developers but I don't know of any app
that deletes or overwrites configuration files because the disk is full.
Configuration files should be editable only by users and read-only by
the application.

Known issue for the <service>.xml files, patches welcomed:

https://osgeo-org.atlassian.net/browse/GEOS-5867

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

12

On Mon, Jun 15, 2015 at 12:04 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Known issue for the <service>.xml files, patches welcomed:

https://osgeo-org.atlassian.net/browse/GEOS-5867

Just fixed it, the fix will be in tomorrow's nightly (and in 2.7.2, 2.6.5)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------