[Geoserver-users] PermGen space issue

GeoServer GeoServer User
1

Hi all,

I have deployed a GeoServer vertical cluster in my lab (1 master, 3 slaves) and I am getting java.lang.OutOfMemoryError: PermGen space errors when I use the Web User Interface for more than 30 mins.

Some details about my setup:

Hardware: 32 CPU cores, 32GB RAM, 1 TB HDD

OS: Debian 7
Linux geosrv 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1 x86_64 GNU/Linux

Java: Oracle JDK 7
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Tomcat: apache-tomcat-7.0.56.tar.gz

GeoServer version: 2.6.0 (the same issue has happened with 2.5.1)

Details: I have deployed a tomcat user under /home/tomcat and I have deployed 4 separate tomcat instances.
I have configured server.xml to point to different ports, and all startup without problem.
The setup is behind HAProxy and I have one backend pointing to the master GeoServer so that the UI requests go only to that instance.
This master instance hangs after I work a bit in the UI. The 3 slaves have no problem, even under stress by users.

The initialization of the master server is done through the following shell script:
export CATALINA_OPTS="-server -Xms1024M -Xmx4096M -XX:+UseParallelGC -XX:SoftRefLRUPolicyMSPerMB=36000 -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=512m"
su - tomcat -c /home/tomcat/tomcat1/bin/startup.sh

This is part of the tomcat1 log:
http://pastebin.com/f8cMQeE4

Any ideas why the PermGen goes out of memory?

Thanks,
Angelos

--
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
http://users.ntua.gr/tzotsos

2

On Fri, Jan 16, 2015 at 12:14 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

The initialization of the master server is done through the following
shell script:
export CATALINA_OPTS="-server -Xms1024M -Xmx4096M -XX:+UseParallelGC
-XX:SoftRefLRUPolicyMSPerMB=36000 -XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=512m"
su - tomcat -c /home/tomcat/tomcat1/bin/startup.sh

Hum... I have never seen a permgen error with 256m of perm gen, let alone
512...
Wondering, are you re-deploying GeoServer in the container without
restarting
tomcat?
There are a few permgen leaks happening on redeploy that we have never
managed to locate and remove, so when redeploying, it's best to
restart the container.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

3

On 01/16/2015 01:47 PM, Andrea Aime wrote:

On Fri, Jan 16, 2015 at 12:14 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

The initialization of the master server is done through the following
shell script:
export CATALINA_OPTS="-server -Xms1024M -Xmx4096M -XX:+UseParallelGC
-XX:SoftRefLRUPolicyMSPerMB=36000 -XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=512m"
su - tomcat -c /home/tomcat/tomcat1/bin/startup.sh

Hum... I have never seen a permgen error with 256m of perm gen, let alone
512...
Wondering, are you re-deploying GeoServer in the container without
restarting
tomcat?
There are a few permgen leaks happening on redeploy that we have never
managed to locate and remove, so when redeploying, it's best to
restart the container.

Cheers
Andrea

Hi Andrea, thanks for the fast answer!

GeoServer is deployed once per instance (perhaps I should delete the war file from the webapps folder?).
I do not re-deploy.
Each time the master instance hangs, I just restart the container, nothing more.

Just another detail I missed to provide: I have added the 64bit native JAI drivers in the setup.

Best,
Angelos

--
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
http://users.ntua.gr/tzotsos

4

On Fri, Jan 16, 2015 at 1:00 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

Hi Andrea, thanks for the fast answer!

GeoServer is deployed once per instance (perhaps I should delete the war
file from the webapps folder?).
I do not re-deploy.
Each time the master instance hangs, I just restart the container, nothing
more.

Just another detail I missed to provide: I have added the 64bit native JAI
drivers in the setup.

Just checking the obvious but... hum... are you positive the options are
actually set in
the virtual machine running Tomcat?
Maybe check with ps?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

On 01/16/2015 03:02 PM, Andrea Aime wrote:

On Fri, Jan 16, 2015 at 1:00 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

Hi Andrea, thanks for the fast answer!
GeoServer is deployed once per instance (perhaps I should delete the war
file from the webapps folder?).
I do not re-deploy.
Each time the master instance hangs, I just restart the container, nothing
more.

Just another detail I missed to provide: I have added the 64bit native JAI
drivers in the setup.

Just checking the obvious but... hum... are you positive the options are
actually set in
the virtual machine running Tomcat?
Maybe check with ps?

Cheers
Andrea

Hi Andrea,

This is my ps output:
tomcat 6451 1 2 14:31 ? 00:01:13 /usr/bin/java -Djava.util.logging.config.file=/home/t\
omcat/tomcat1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManag\
er -Djava.endorsed.dirs=/home/tomcat/tomcat1/endorsed -classpath /home/tomcat/tomcat1/bin/bootstrap.j\
ar:/home/tomcat/tomcat1/bin/tomcat-juli.jar -Dcatalina.base=/home/tomcat/tomcat1 -Dcatalina.home=/hom\
e/tomcat/tomcat1 -Djava.io.tmpdir=/home/tomcat/tomcat1/temp org.apache.catalina.startup.Bootstrap sta\
rt

Should the startup options appear here?

--
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
http://users.ntua.gr/tzotsos

6

On Fri, Jan 16, 2015 at 2:44 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

On 01/16/2015 03:02 PM, Andrea Aime wrote:

On Fri, Jan 16, 2015 at 1:00 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

Hi Andrea, thanks for the fast answer!

GeoServer is deployed once per instance (perhaps I should delete the war
file from the webapps folder?).
I do not re-deploy.
Each time the master instance hangs, I just restart the container,
nothing
more.

Just another detail I missed to provide: I have added the 64bit native
JAI
drivers in the setup.

Just checking the obvious but... hum... are you positive the options are

actually set in
the virtual machine running Tomcat?
Maybe check with ps?

Cheers
Andrea

Hi Andrea,

This is my ps output:
tomcat 6451 1 2 14:31 ? 00:01:13 /usr/bin/java
-Djava.util.logging.config.file=/home/t\
omcat/tomcat1/conf/logging.properties -Djava.util.logging.manager=
org.apache.juli.ClassLoaderLogManag\
er -Djava.endorsed.dirs=/home/tomcat/tomcat1/endorsed -classpath
/home/tomcat/tomcat1/bin/bootstrap.j\
ar:/home/tomcat/tomcat1/bin/tomcat-juli.jar -Dcatalina.base=/home/tomcat/tomcat1
-Dcatalina.home=/hom\
e/tomcat/tomcat1 -Djava.io.tmpdir=/home/tomcat/tomcat1/temp
org.apache.catalina.startup.Bootstrap sta\
rt

Should the startup options appear here?

Yes, they are arguments of the "java" call

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

7

On 01/16/2015 03:40 PM, Andrea Aime wrote:

On Fri, Jan 16, 2015 at 2:44 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

On 01/16/2015 03:02 PM, Andrea Aime wrote:

On Fri, Jan 16, 2015 at 1:00 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

  Hi Andrea, thanks for the fast answer!

GeoServer is deployed once per instance (perhaps I should delete the war
file from the webapps folder?).
I do not re-deploy.
Each time the master instance hangs, I just restart the container,
nothing
more.

Just another detail I missed to provide: I have added the 64bit native
JAI
drivers in the setup.

  Just checking the obvious but... hum... are you positive the options are

actually set in
the virtual machine running Tomcat?
Maybe check with ps?

Cheers
Andrea

  Hi Andrea,

This is my ps output:
tomcat 6451 1 2 14:31 ? 00:01:13 /usr/bin/java
-Djava.util.logging.config.file=/home/t\
omcat/tomcat1/conf/logging.properties -Djava.util.logging.manager=
org.apache.juli.ClassLoaderLogManag\
er -Djava.endorsed.dirs=/home/tomcat/tomcat1/endorsed -classpath
/home/tomcat/tomcat1/bin/bootstrap.j\
ar:/home/tomcat/tomcat1/bin/tomcat-juli.jar -Dcatalina.base=/home/tomcat/tomcat1
-Dcatalina.home=/hom\
e/tomcat/tomcat1 -Djava.io.tmpdir=/home/tomcat/tomcat1/temp
org.apache.catalina.startup.Bootstrap sta\
rt

Should the startup options appear here?

Yes, they are arguments of the "java" call

Cheers
Andrea

What is the preferred method to pass those parameters?
A setenv.sh in Catalina_base/bin folder?

Thanks,
Angelos

--
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
http://users.ntua.gr/tzotsos

8

On Fri, Jan 16, 2015 at 2:51 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

What is the preferred method to pass those parameters?

A setenv.sh in Catalina_base/bin folder?

How is Tomcat installed? Did you install it by hand, or is it a package?
I normally go by hand, downloading from the official web site, and then
directly
modify catalina.sh :-p

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

9

On 01/16/2015 03:51 PM, Andrea Aime wrote:

On Fri, Jan 16, 2015 at 2:51 PM, Angelos Tzotsos <gcpp.kalxas@anonymised.com>
wrote:

  What is the preferred method to pass those parameters?

A setenv.sh in Catalina_base/bin folder?

How is Tomcat installed? Did you install it by hand, or is it a package?
I normally go by hand, downloading from the official web site, and then
directly
modify catalina.sh :-p

Cheers
Andrea

Thanks Andrea, I wanted to confirm.
I added in catalina.sh and I get the following now in ps:

tomcat 7269 1 99 15:50 ? 00:00:44 /usr/bin/java -Djava.util.logging.config.file=/home/t\
omcat/tomcat1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManag\
er -server -Xms1024M -Xmx4096M -XX:+UseParallelGC -XX:SoftRefLRUPolicyMSPerMB=36000 -XX:+CMSClassUnlo\
adingEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=1024m -XX:PermSize=512m -Djava.endorsed.d\
irs=/home/tomcat/tomcat1/endorsed -classpath /home/tomcat/tomcat1/bin/bootstrap.jar:/home/tomcat/tomc\
at1/bin/tomcat-juli.jar -Dcatalina.base=/home/tomcat/tomcat1 -Dcatalina.home=/home/tomcat/tomcat1 -Dj\
ava.io.tmpdir=/home/tomcat/tomcat1/temp org.apache.catalina.startup.Bootstrap start

I will play around to see if I get any errors.

Thank you very much for your help!

Best,
Angelos

--
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
http://users.ntua.gr/tzotsos

10

I have a layer group I want to seed tiles into using the built in geowebcache in geoserver 2.6.1

It is made up of 5 other layer groups.

My question is what should I generate a cache for ?

1 - The individual layers
2 - The smaller layer groups
3 - The layer group of layer groups
4 - all of the above

I am using a custom epsg:27700 with only 8 levels

Mark Ismail
Principal IT Officer
This e-mail and any attachments are confidential. It may contain privileged information and is intended for the named recipient(s) only. It must not be distributed without consent. If you are not one of the intended recipients, please notify the sender immediately and do not disclose, distribute, or retain this email or any part of it and do not take any action based on it.

Unless expressly stated, opinions in this email are those of the individual sender, and not of Knowsley MBC. Legally binding obligations can only be created for, or be entered into on behalf of, Knowsley MBC by duly authorised officers or representatives.

Knowsley MBC excludes any liability whatsoever for any offence caused, any direct or consequential loss arising from the use, or reliance on, this e-mail or its contents. We believe but do not warrant that this e-mail and any attachments are virus free. You must therefore take full responsibility for virus checking and no responsibility is accepted for loss or damage arising from viruses or changes made to this message after it was sent. Knowsley MBC reserves the right to monitor and/or record all e-mail communications through its network in accordance with relevant legislation.

11

You will want to seed for the layer for which you will be requesting images from via WMS… In most cases that is the layergroup.

Ben

···

On Fri, Jan 16, 2015 at 9:57 AM, Ismail, Mark <mark.ismail@anonymised.com> wrote:

I have a layer group I want to seed tiles into using the built in geowebcache in geoserver 2.6.1

It is made up of 5 other layer groups.

My question is what should I generate a cache for ?

1 - The individual layers
2 - The smaller layer groups
3 - The layer group of layer groups
4 - all of the above

I am using a custom epsg:27700 with only 8 levels

Mark Ismail
Principal IT Officer
This e-mail and any attachments are confidential. It may contain privileged information and is intended for the named recipient(s) only. It must not be distributed without consent. If you are not one of the intended recipients, please notify the sender immediately and do not disclose, distribute, or retain this email or any part of it and do not take any action based on it.

Unless expressly stated, opinions in this email are those of the individual sender, and not of Knowsley MBC. Legally binding obligations can only be created for, or be entered into on behalf of, Knowsley MBC by duly authorised officers or representatives.

Knowsley MBC excludes any liability whatsoever for any offence caused, any direct or consequential loss arising from the use, or reliance on, this e-mail or its contents. We believe but do not warrant that this e-mail and any attachments are virus free. You must therefore take full responsibility for virus checking and no responsibility is accepted for loss or damage arising from viruses or changes made to this message after it was sent. Knowsley MBC reserves the right to monitor and/or record all e-mail communications through its network in accordance with relevant legislation.

New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users