[Geoserver-devel] Our build test memory filled by finalizers

[Andrea_Aime4]

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

···

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[Andrea_Aime4]

Ah, by the way, probably not relevant to this discussion, but
looking at some performance charts, it seems that we are leaking threads:

I can count 90 FileSystemWatcher, so I guess that during a build test run we are setting up and not properly cleaning up a lot of them.
We should probably open a ticket about that right?

Cheers
Andrea

(attachments)

···

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[Andrea_Aime4]

Ah hem,
I had another look and think I’ve found a smoking gun. See here:

So the MapContent has a finalize method, and contains layers, which refer to resources, which refer to the catalog,
which ends up referring to the whole application context.
The blasted thing is literally keeping up a “GeoServer worth of memory” pending for finalization.
And there are

We could just remove that finalize method… I’ve also found this article talking about alternative approaches, like
using weak references instead, which could be interesting, but requires some study:
http://www.oracle.com/technetwork/articles/java/finalization-137655.html

Wondering, anyone interesting in pursuing this?

Cheers
Andrea

(attachments)

···

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[Andrea_Aime4]

Hi,
while removing that gives some relief, I’ve also found something just blocking the finalizer for extended amounts of
time:

“Finalizer” #3 daemon prio=8 os_prio=0 tid=0x00007f1d701d3800 nid=0x76b1 runnable [0x00007f1d4a5e4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

• locked <0x00000000f57d8a48> (a java.net.SocksSocketImpl)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at java.net.Socket.connect(Socket.java:538)
  at java.net.Socket.(Socket.java:434)
  at java.net.Socket.(Socket.java:244)
  at javax.media.jai.remote.SerializableRenderedImage.connectToServer(SerializableRenderedImage.java:1169)
  at javax.media.jai.remote.SerializableRenderedImage.closeClient(SerializableRenderedImage.java:1105)
  at javax.media.jai.remote.SerializableRenderedImage.dispose(SerializableRenderedImage.java:1314)
  at javax.media.jai.remote.SerializableRenderedImage.finalize(SerializableRenderedImage.java:1259)
  at java.lang.System$2.invokeFinalize(System.java:1270)
  at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
  at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)

This is a class used to store on disk a SampleModel and ColorModel in the image mosaic, which for
some forsaken reason decides to open a socket and let it time out during finalization… we’ll have
to find another way to store the same information without using SerializableRenderedImage

Cheers
Andrea

(attachments)

···

On Wed, Feb 8, 2017 at 10:08 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ah hem,
I had another look and think I’ve found a smoking gun. See here:

So the MapContent has a finalize method, and contains layers, which refer to resources, which refer to the catalog,
which ends up referring to the whole application context.
The blasted thing is literally keeping up a “GeoServer worth of memory” pending for finalization.
And there are

We could just remove that finalize method… I’ve also found this article talking about alternative approaches, like
using weak references instead, which could be interesting, but requires some study:
http://www.oracle.com/technetwork/articles/java/finalization-137655.html

Wondering, anyone interesting in pursuing this?

Cheers

Andrea

–

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[Andrea_Aime4]

Ok,
so I’ve made a fix removing the SerializableRendererImage down in GeoTools (backwards compatibile with
silent upgrade) and then also made the code release the map content early.
Seems to be building fine now.

In case you want to have a look, it’s the second commit in this pull request:

https://github.com/geoserver/geoserver/pull/2094/commits

Cheers
Andrea

(attachments)

···

On Wed, Feb 8, 2017 at 11:17 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
while removing that gives some relief, I’ve also found something just blocking the finalizer for extended amounts of
time:

“Finalizer” #3 daemon prio=8 os_prio=0 tid=0x00007f1d701d3800 nid=0x76b1 runnable [0x00007f1d4a5e4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

• locked <0x00000000f57d8a48> (a java.net.SocksSocketImpl)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at java.net.Socket.connect(Socket.java:538)
  at java.net.Socket.(Socket.java:434)
  at java.net.Socket.(Socket.java:244)
  at javax.media.jai.remote.SerializableRenderedImage.connectToServer(SerializableRenderedImage.java:1169)
  at javax.media.jai.remote.SerializableRenderedImage.closeClient(SerializableRenderedImage.java:1105)
  at javax.media.jai.remote.SerializableRenderedImage.dispose(SerializableRenderedImage.java:1314)
  at javax.media.jai.remote.SerializableRenderedImage.finalize(SerializableRenderedImage.java:1259)
  at java.lang.System$2.invokeFinalize(System.java:1270)
  at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
  at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)

This is a class used to store on disk a SampleModel and ColorModel in the image mosaic, which for
some forsaken reason decides to open a socket and let it time out during finalization… we’ll have
to find another way to store the same information without using SerializableRenderedImage

Cheers

Andrea

–

On Wed, Feb 8, 2017 at 10:08 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ah hem,
I had another look and think I’ve found a smoking gun. See here:

So the MapContent has a finalize method, and contains layers, which refer to resources, which refer to the catalog,
which ends up referring to the whole application context.
The blasted thing is literally keeping up a “GeoServer worth of memory” pending for finalization.
And there are

We could just remove that finalize method… I’ve also found this article talking about alternative approaches, like
using weak references instead, which could be interesting, but requires some study:
http://www.oracle.com/technetwork/articles/java/finalization-137655.html

Wondering, anyone interesting in pursuing this?

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com68…> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[jive]

I am interested in pursuing this, and expect that it is my fault for trying to make MapContext easier to use for GeoTools developers (and calling them out when they leak resources by not calling dispose() ).

We already do have a dispose() method - one of the suggestions provided by your link. Happy to help untangle up to and including not making MapContext finalized.

(attachments)

···

On 8 February 2017 at 01:08, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ah hem,
I had another look and think I’ve found a smoking gun. See here:

So the MapContent has a finalize method, and contains layers, which refer to resources, which refer to the catalog,
which ends up referring to the whole application context.
The blasted thing is literally keeping up a “GeoServer worth of memory” pending for finalization.
And there are

We could just remove that finalize method… I’ve also found this article talking about alternative approaches, like
using weak references instead, which could be interesting, but requires some study:
http://www.oracle.com/technetwork/articles/java/finalization-137655.html

Wondering, anyone interesting in pursuing this?

Cheers

Andrea

---

Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

---

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[jive]

Reviewing now; what other help do you need?

(attachments)

···

On 8 February 2017 at 09:51, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ok,
so I’ve made a fix removing the SerializableRendererImage down in GeoTools (backwards compatibile with
silent upgrade) and then also made the code release the map content early.
Seems to be building fine now.

In case you want to have a look, it’s the second commit in this pull request:

https://github.com/geoserver/geoserver/pull/2094/commits

Cheers

Andrea

---

Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

---

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

On Wed, Feb 8, 2017 at 11:17 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
while removing that gives some relief, I’ve also found something just blocking the finalizer for extended amounts of
time:

“Finalizer” #3 daemon prio=8 os_prio=0 tid=0x00007f1d701d3800 nid=0x76b1 runnable [0x00007f1d4a5e4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

• locked <0x00000000f57d8a48> (a java.net.SocksSocketImpl)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at java.net.Socket.connect(Socket.java:538)
  at java.net.Socket.(Socket.java:434)
  at java.net.Socket.(Socket.java:244)
  at javax.media.jai.remote.SerializableRenderedImage.connectToServer(SerializableRenderedImage.java:1169)
  at javax.media.jai.remote.SerializableRenderedImage.closeClient(SerializableRenderedImage.java:1105)
  at javax.media.jai.remote.SerializableRenderedImage.dispose(SerializableRenderedImage.java:1314)
  at javax.media.jai.remote.SerializableRenderedImage.finalize(SerializableRenderedImage.java:1259)
  at java.lang.System$2.invokeFinalize(System.java:1270)
  at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
  at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)

This is a class used to store on disk a SampleModel and ColorModel in the image mosaic, which for
some forsaken reason decides to open a socket and let it time out during finalization… we’ll have
to find another way to store the same information without using SerializableRenderedImage

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

On Wed, Feb 8, 2017 at 10:08 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ah hem,
I had another look and think I’ve found a smoking gun. See here:

So the MapContent has a finalize method, and contains layers, which refer to resources, which refer to the catalog,
which ends up referring to the whole application context.
The blasted thing is literally keeping up a “GeoServer worth of memory” pending for finalization.
And there are

We could just remove that finalize method… I’ve also found this article talking about alternative approaches, like
using weak references instead, which could be interesting, but requires some study:
http://www.oracle.com/technetwork/articles/java/finalization-137655.html

Wondering, anyone interesting in pursuing this?

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[Andrea_Aime4]

Hi Jody,
a review will do I believe, checking a memory dump after those two fixes the finalization queue is only holding up
something like 1.5MB worth of memory, so I’d say this problem is actually gone.

The other thing that is a bit worrysome is the file watcher threads not being closed, but that does not seem to be
affecting the build all that much. I’ve created a separate ticket for it:
https://osgeo-org.atlassian.net/browse/GEOS-7983

Cheers
Andrea

(attachments)

···

On Wed, Feb 8, 2017 at 7:03 PM, Jody Garnett <jody.garnett@anonymised.com> wrote:

Reviewing now; what other help do you need?

–

–
Jody Garnett

On 8 February 2017 at 09:51, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ok,
so I’ve made a fix removing the SerializableRendererImage down in GeoTools (backwards compatibile with
silent upgrade) and then also made the code release the map content early.
Seems to be building fine now.

In case you want to have a look, it’s the second commit in this pull request:

https://github.com/geoserver/geoserver/pull/2094/commits

Cheers

Andrea

---

Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

---

Geoserver-devel mailing list
Geoserver-devel@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Wed, Feb 8, 2017 at 11:17 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
while removing that gives some relief, I’ve also found something just blocking the finalizer for extended amounts of
time:

“Finalizer” #3 daemon prio=8 os_prio=0 tid=0x00007f1d701d3800 nid=0x76b1 runnable [0x00007f1d4a5e4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

• locked <0x00000000f57d8a48> (a java.net.SocksSocketImpl)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at java.net.Socket.connect(Socket.java:538)
  at java.net.Socket.(Socket.java:434)
  at java.net.Socket.(Socket.java:244)
  at javax.media.jai.remote.SerializableRenderedImage.connectToServer(SerializableRenderedImage.java:1169)
  at javax.media.jai.remote.SerializableRenderedImage.closeClient(SerializableRenderedImage.java:1105)
  at javax.media.jai.remote.SerializableRenderedImage.dispose(SerializableRenderedImage.java:1314)
  at javax.media.jai.remote.SerializableRenderedImage.finalize(SerializableRenderedImage.java:1259)
  at java.lang.System$2.invokeFinalize(System.java:1270)
  at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
  at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)

This is a class used to store on disk a SampleModel and ColorModel in the image mosaic, which for
some forsaken reason decides to open a socket and let it time out during finalization… we’ll have
to find another way to store the same information without using SerializableRenderedImage

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

On Wed, Feb 8, 2017 at 10:08 AM, Andrea Aime <andrea.aime@anonymised.com1268…> wrote:

Ah hem,
I had another look and think I’ve found a smoking gun. See here:

So the MapContent has a finalize method, and contains layers, which refer to resources, which refer to the catalog,
which ends up referring to the whole application context.
The blasted thing is literally keeping up a “GeoServer worth of memory” pending for finalization.
And there are

We could just remove that finalize method… I’ve also found this article talking about alternative approaches, like
using weak references instead, which could be interesting, but requires some study:
http://www.oracle.com/technetwork/articles/java/finalization-137655.html

Wondering, anyone interesting in pursuing this?

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

On Tue, Feb 7, 2017 at 3:19 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
today I was checking out the build for a new PDF generation test I added,
and observed very slow build and eventually a “OOM, GC overhead exceeeded”.

I’ve investigated and found out that:

• We were already on the brink of failure
• The new test, first one to verify the PDF output, added some 30+ MB of heap usage caused by some iText internal font cache, made it overflow
  I’ve then looked at a heap dump of the surefire jvm and found this (mind, this is a dump of the surefire without my changes, I wanted to make sure

the new test was not altering things significantly, and indeed confirmed it was merely adding the last drop making the bucket overflow):

Now, the surefire JVM is allowed only 512MB of heap, the Class are the top are likely accounting the new metaspace.
So the heap is mostly filled by that ton of Finalizer, which variably refer to many of the objects you see below.

Why are we getting there? Because we are creating a very high number of objects that sport a finalize() method, sampling though the
object explorer I’ve found large numbers of FileInputStream/FileOutputStream/ZipFileInflaterInputStream/Deflater/CloseableIteratorAdapter/MapContent and many others.

That is coupled with a JVM own limitation, the finalizer thread is only one and is low priority… when the memory gets filled with finalizers
the GC is not able to push the finalizer to work more, has less and less space to work with, and eventually the JVM gives up
by throwing the dreaded “java.lang.OutOfMemoryError: GC overhead limit exceeded”.

So… what to do about it? Eh… One thing to try is to raise the finalizer thread priority… too bad that does not work on Linux unless the user
making the switch is the root user himself.
Other options:

• Reduce the usage of objects with finalizers
• Give tests more memory (now they are at 512MB)
• Run finalization explicitly using System.runFinalization() at the end of each test class (as an @AfterClass of sorts)

The first one is a lot of work with no guaranteed results due to the many JVM classes currently held by finalizers (but it would still be
a good thing to think twice before adding a finalizer onto an object that gets used a lot).

Tried the second one, giving it 768MB of heap, works.

[INFO]
[INFO] Main Module … SUCCESS [01:57 min]
[INFO] Web Feature Service Module … SUCCESS [01:17 min]
[INFO] Web Map Service Module … SUCCESS [01:43 min]

And then the third option, only 512MB of heap but manually forcing the run of finalization at end… got the GC overhead exceeded

anyways (a bit surprised about it, but oh well…). Also works, but build times suffer greatly:

[INFO] Main Module … SUCCESS [01:56 min]
[INFO] Web Feature Service Module … SUCCESS [01:14 min]
[INFO] Web Map Service Module … SUCCESS [02:33 min]
[INFO] ------------------------------------------------------------------------

So… I’d say it’s best to increase the heap size. Unless anyone else has alternative suggestions?
Sure, I could commit the work on PDF without tests (the horror…), but it’s just a matter of time
before we get back to square one…

Pull request for the PDF change and memory increase here:

https://github.com/geoserver/geoserver/pull/2094

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

[Bart_van_den_Eijnde1]

Great work, Andrea. I have seen this finalizer crapflood when running gt-netcdf unit tests in Eclipse, and I could never figure out what was causing it!

Kind regards,
Ben.

On 08/02/17 23:17, Andrea Aime wrote:

Hi,
while removing that gives some relief, I've also found something just
blocking the finalizer for extended amounts of
time:

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1d701d3800 nid=0x76b1
runnable [0x00007f1d4a5e4000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x00000000f57d8a48> (a java.net.SocksSocketImpl)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:244)
at
javax.media.jai.remote.SerializableRenderedImage.connectToServer(SerializableRenderedImage.java:1169)
at
javax.media.jai.remote.SerializableRenderedImage.closeClient(SerializableRenderedImage.java:1105)
at
javax.media.jai.remote.SerializableRenderedImage.dispose(SerializableRenderedImage.java:1314)
at
javax.media.jai.remote.SerializableRenderedImage.finalize(SerializableRenderedImage.java:1259)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)

This is a class used to store on disk a SampleModel and ColorModel in the
image mosaic, which for
some forsaken reason decides to open a socket and let it time out during
finalization... we'll have
to find another way to store the same information without using
SerializableRenderedImage

Cheers
Andrea

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand