[Geoserver-devel] wms.xml, wfs.xml, wcs.xml wps.xml files get cleared of content making services unavailable

Issue:

We have an issue where the wms.xml, wfs.xml, wcs.xml and wps.xml files in GeoServer from time to time get cleared of content, thus resulting in no services being available.

Environment:

We have 2 environments. Both exhibit the issue.

  1. Environment 1

a. GeoServer version 2.1.x

b. Running on RHEL 5.7

c. JBoss 5.1

d. Clustered configuration

i. 4 GeoServer Instances clustered sharing global file external data directory

ii. F5 Load balancer in front of the GeoServers.

  1. Environment 2

a. GeoServer version 2.1.x

b. Running on RHEL 5.7

c. JBoss 5.1

d. Clustered configuration

i. 2 GeoServers Clustered, sharing global file external data directory

ii. F5 Load balancer in front of the GeoServers.

Theory:

We ‘think’ this may be triggered by a rest reload command.

We do a reset reload command because we use the rest interface to create new layers. The call is made via one of the clustered GeoServers, and a reload of the other GeoServers in the cluster is required in order for them to become aware of the newly created layers.

It has been our observation that a back-to-back reload command often precedes the issue occurring.

Request:

I attempted to see if this issue has been reported as bug in the Jira system, but did not find a report that I thought was a match.

Has anyone else experienced this issue?

Is there an existing bug report for this issue?

Has this issue be addressing in a newer release?

Any ideas?

Thanks in advance.

Respectfully,

Chris Moreau

Robert L Brister
Principal Software Engineer
General Dynamics Information Technology
FNMOC, Monterey, CA
Building 704, Rm 235
Ph: 831-656-5056
Cell: 408-569-5954
robert.brister@anonymised.com

UNCLASSIFIED//FOR OFFICIAL USE ONLY//

Hi Robert,

I don’t know of any outstanding jira ticket specifically for this issue. You could try a newer version and see if the problem persists.

All in all your best bet is going to be to try as best you can to reliably reproduce the problem and then find a way to have a developer reproduce the problem. Without that I am not sure what can be done.

Perhaps you can try to set up a watcher script that notifies when the files are flushed away. And perhaps in parallel to that turn on request level logging in your servlet container and try to compare timestamps to see indeed what is going on and perhaps what lead to it.

-Justin

On Wed, May 2, 2012 at 2:50 PM, Brister, Robert L <Robert.Brister@anonymised.com> wrote:

Issue:

We have an issue where the wms.xml, wfs.xml, wcs.xml and wps.xml files in GeoServer from time to time get cleared of content, thus resulting in no services being available.

Environment:

We have 2 environments. Both exhibit the issue.

  1. Environment 1

a. GeoServer version 2.1.x

b. Running on RHEL 5.7

c. JBoss 5.1

d. Clustered configuration

i. 4 GeoServer Instances clustered sharing global file external data directory

ii. F5 Load balancer in front of the GeoServers.

  1. Environment 2

a. GeoServer version 2.1.x

b. Running on RHEL 5.7

c. JBoss 5.1

d. Clustered configuration

i. 2 GeoServers Clustered, sharing global file external data directory

ii. F5 Load balancer in front of the GeoServers.

Theory:

We ‘think’ this may be triggered by a rest reload command.

We do a reset reload command because we use the rest interface to create new layers. The call is made via one of the clustered GeoServers, and a reload of the other GeoServers in the cluster is required in order for them to become aware of the newly created layers.

It has been our observation that a back-to-back reload command often precedes the issue occurring.

Request:

I attempted to see if this issue has been reported as bug in the Jira system, but did not find a report that I thought was a match.

Has anyone else experienced this issue?

Is there an existing bug report for this issue?

Has this issue be addressing in a newer release?

Any ideas?

Thanks in advance.

Respectfully,

Chris Moreau

Robert L Brister
Principal Software Engineer
General Dynamics Information Technology
FNMOC, Monterey, CA
Building 704, Rm 235
Ph: 831-656-5056
Cell: 408-569-5954
robert.brister@anonymised.com

UNCLASSIFIED//FOR OFFICIAL USE ONLY//


Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Thanks for the response Justin. I work w. Robert. We have been attempting to do just what you have recommended, unfortunately, without success. So it’s hard for us to know if newer version has addressed the issue. We’ve done testing with a trunk release from March 12, 2012 and with 2.2 beta 1, but bugs in their rest interfaces made those version a non-option for us. We also looked through Jira.

We are posting to see if anyone else has experienced this and has either found a way to reproduce it or found a way to prevent it. We’ll continue to try to reliably reproduce.

Respectfully,

Chris Moreau

Manager, Software Engineering

General Dynamics Information Technology

8989 Rio San Diego Dr., Suite 330

San Diego, CA 92108

W: (619) 684-7909

M: (619) 379-9939

···

I don’t know of any outstanding jira ticket specifically for this issue. You could try a newer version and see if the problem persists.

All in all your best bet is going to be to try as best you can to reliably reproduce the problem and then find a way to have a developer reproduce the problem. Without that I am not sure what can be done.

Perhaps you can try to set up a watcher script that notifies when the files are flushed away. And perhaps in parallel to that turn on request level logging in your servlet container and try to compare timestamps to see indeed what is going on and perhaps what lead to it.

-Justin

On Wed, May 2, 2012 at 2:50 PM, Brister, Robert L <Robert.Brister@anonymised.com> wrote:

Issue:

We have an issue where the wms.xml, wfs.xml, wcs.xml and wps.xml files in GeoServer from time to time get cleared of content, thus resulting in no services being available.

Environment:

We have 2 environments. Both exhibit the issue.

  1. Environment 1

a. GeoServer version 2.1.x

b. Running on RHEL 5.7

c. JBoss 5.1

d. Clustered configuration

i. 4 GeoServer Instances clustered sharing global file external data directory

ii. F5 Load balancer in front of the GeoServers.

  1. Environment 2

a. GeoServer version 2.1.x

b. Running on RHEL 5.7

c. JBoss 5.1

d. Clustered configuration

i. 2 GeoServers Clustered, sharing global file external data directory

ii. F5 Load balancer in front of the GeoServers.

Theory:

We ‘think’ this may be triggered by a rest reload command.

We do a reset reload command because we use the rest interface to create new layers. The call is made via one of the clustered GeoServers, and a reload of the other GeoServers in the cluster is required in order for them to become aware of the newly created layers.

It has been our observation that a back-to-back reload command often precedes the issue occurring.

Request:

I attempted to see if this issue has been reported as bug in the Jira system, but did not find a report that I thought was a match.

Has anyone else experienced this issue?

Is there an existing bug report for this issue?

Has this issue be addressing in a newer release?

Any ideas?

Thanks in advance.

Respectfully,

Chris Moreau

Robert L Brister
Principal Software Engineer
General Dynamics Information Technology
FNMOC, Monterey, CA
Building 704, Rm 235
Ph: 831-656-5056
Cell: 408-569-5954
robert.brister@anonymised.com

UNCLASSIFIED//FOR OFFICIAL USE ONLY//


Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Justin Deoliveira

OpenGeo - http://opengeo.org

Enterprise support for open source geospatial.

On Thu, May 3, 2012 at 5:50 PM, Moreau, Christopher <Christopher.Moreau@anonymised.com> wrote:

Thanks for the response Justin. I work w. Robert. We have been attempting to do just what you have recommended, unfortunately, without success. So it’s hard for us to know if newer version has addressed the issue. We’ve done testing with a trunk release from March 12, 2012 and with 2.2 beta 1, but bugs in their rest interfaces made those version a non-option for us. We also looked through Jira.

“bugs in the rest interfaces” → were they reported in jira as well?

Cheers
Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

Yes. GEOS-5059

Also got more ambiguous reports about rest issues arising from new security features, that are being addressing beta 2. Not sure of the source of that information.

Another issue reported not releated directly to rest: GEOS-5019.

Respectfully,

Chris Moreau

···

On Thu, May 3, 2012 at 5:50 PM, Moreau, Christopher <Christopher.Moreau@anonymised.com> wrote:

Thanks for the response Justin. I work w. Robert. We have been attempting to do just what you have recommended, unfortunately, without success. So it’s hard for us to know if newer version has addressed the issue. We’ve done testing with a trunk release from March 12, 2012 and with 2.2 beta 1, but bugs in their rest interfaces made those version a non-option for us. We also looked through Jira.

“bugs in the rest interfaces” → were they reported in jira as well?

Cheers

Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

Yes. GEOS-5059

Also got more ambiguous reports about rest issues arising from new security features, that are being addressing beta 2. Not sure of the source of that information.

Another issue reported not releated directly to rest: GEOS-5019.

Respectfully,

Chris Moreau

···

On Thu, May 3, 2012 at 5:50 PM, Moreau, Christopher <Christopher.Moreau@anonymised.com> wrote:

Thanks for the response Justin. I work w. Robert. We have been attempting to do just what you have recommended, unfortunately, without success. So it’s hard for us to know if newer version has addressed the issue. We’ve done testing with a trunk release from March 12, 2012 and with 2.2 beta 1, but bugs in their rest interfaces made those version a non-option for us. We also looked through Jira.

“bugs in the rest interfaces” → were they reported in jira as well?

Cheers

Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf