[Geoserver-users] Suspected memory leak when using Oracle OCI drivers

GeoServer GeoServer User
1

Hello GeoServer community,

I've spent quite some time (all of last week) investigating a performance
problem regarding our mapping application built on top of GeoServer /
OpenLayers.
The memory consumption grew as time went by and never recovered / declined
anymore to the point where the operating system began to swap which resulted
in a really poor performance.
The log entry 'rendering takes >60 sec, aborting..' came up in shorter
intervals and at some point the server just didn't respond anymore.

To find the reason for that I've used different servers (Tomcat / WebLogic)
and even different JVMs (HotSpot / JRockit).
A heap dump didn't really show anything extraordinary, but looking at some
other data the native memory part was suspicious.
As it turns out the culprit seems to be the OCI drivers (or their usage).

The problem can be observed by using the built-in Layer Preview feature
(just moving around the map - panning / zooming in and out is sufficient).
As soon as I switch to Thin-Driver the application performs better and the
memory stays within some sensible boundaries (after some initial climbing).

Since the usage of OCI is mandatory for our customer the Thin-Driver option
is unfortunately not possible.

I guess Oracle is not the preferred database in the GeoServer community and
OCI is even less used to access it but has anyone some experience in dealing
with such a situation nontheless?

Should I create a ticket for that?

Thanks for any input in advance,

Daniel

==============================================================================
Datasource settings (OCI):
  Connection timeout: 20
  validate connections: true
  max connections: 10
  Primary key metadata table: geo_pk_metadata
  Loose bbox: true
  Expose primary keys: true
  Max open prepared statements: 100
  fetch size: 1000
  Estimated extends: true
  min connections: 1

==============================================================================
Environment:
  GeoServer: 2.5 final
  
  App-Server / Servlet container: WebLogic 10.3.2 (it behaves the same on
Tomcat 6)

  OS: Red Hat Enterprise Linux Server release 5.4 (Tikanga)
  
  Database: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit
Production

  JVM:
  Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
  BEA JRockit(R) (build
R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32, compiled mode)
  
  Java settings: -Xmx1024m -Xms1024m

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Suspected-memory-leak-when-using-Oracle-OCI-drivers-tp5133570.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

2

Please create a ticket, with whatever details you have.

There is no real developer community priority a preferred datastore. Comes down to what volunteers are willing to work on (or what customers are willing to pay for). In this case having a bug report to point to is a good way to start the conversation.

So you are thinking the memory link is in the native code - and thus not something that shows up in a memory profiler?

···

Jody Garnett

On Mon, Apr 7, 2014 at 11:11 PM, daniel.soneira <daniel.soneira@anonymised.com> wrote:

Hello GeoServer community,

I’ve spent quite some time (all of last week) investigating a performance
problem regarding our mapping application built on top of GeoServer /
OpenLayers.
The memory consumption grew as time went by and never recovered / declined
anymore to the point where the operating system began to swap which resulted
in a really poor performance.
The log entry ‘rendering takes >60 sec, aborting…’ came up in shorter
intervals and at some point the server just didn’t respond anymore.

To find the reason for that I’ve used different servers (Tomcat / WebLogic)
and even different JVMs (HotSpot / JRockit).
A heap dump didn’t really show anything extraordinary, but looking at some
other data the native memory part was suspicious.
As it turns out the culprit seems to be the OCI drivers (or their usage).

The problem can be observed by using the built-in Layer Preview feature
(just moving around the map - panning / zooming in and out is sufficient).
As soon as I switch to Thin-Driver the application performs better and the
memory stays within some sensible boundaries (after some initial climbing).

Since the usage of OCI is mandatory for our customer the Thin-Driver option
is unfortunately not possible.

I guess Oracle is not the preferred database in the GeoServer community and
OCI is even less used to access it but has anyone some experience in dealing
with such a situation nontheless?

Should I create a ticket for that?

Thanks for any input in advance,

Daniel

==============================================================================
Datasource settings (OCI):
Connection timeout: 20
validate connections: true
max connections: 10
Primary key metadata table: geo_pk_metadata
Loose bbox: true
Expose primary keys: true
Max open prepared statements: 100
fetch size: 1000
Estimated extends: true
min connections: 1

==============================================================================
Environment:
GeoServer: 2.5 final

App-Server / Servlet container: WebLogic 10.3.2 (it behaves the same on
Tomcat 6)

OS: Red Hat Enterprise Linux Server release 5.4 (Tikanga)

Database: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit
Production

JVM:
Java™ SE Runtime Environment (build 1.6.0_14-b08)
BEA JRockit(R) (build
R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32, compiled mode)

Java settings: -Xmx1024m -Xms1024m

–
View this message in context: http://osgeo-org.1560.x6.nabble.com/Suspected-memory-leak-when-using-Oracle-OCI-drivers-tp5133570.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_APR

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

On Mon, Apr 7, 2014 at 3:11 PM, daniel.soneira <daniel.soneira@anonymised.com>wrote:

Hello GeoServer community,

I've spent quite some time (all of last week) investigating a performance
problem regarding our mapping application built on top of GeoServer /
OpenLayers.
The memory consumption grew as time went by and never recovered / declined
anymore to the point where the operating system began to swap which
resulted
in a really poor performance.
The log entry 'rendering takes >60 sec, aborting..' came up in shorter
intervals and at some point the server just didn't respond anymore.

To find the reason for that I've used different servers (Tomcat / WebLogic)
and even different JVMs (HotSpot / JRockit).
A heap dump didn't really show anything extraordinary, but looking at some
other data the native memory part was suspicious.
As it turns out the culprit seems to be the OCI drivers (or their usage).

The problem can be observed by using the built-in Layer Preview feature
(just moving around the map - panning / zooming in and out is sufficient).
As soon as I switch to Thin-Driver the application performs better and the
memory stays within some sensible boundaries (after some initial climbing).

Since the usage of OCI is mandatory for our customer the Thin-Driver option
is unfortunately not possible.

I guess Oracle is not the preferred database in the GeoServer community and
OCI is even less used to access it but has anyone some experience in
dealing
with such a situation nontheless?

You might want to have a look at this discussion:
http://osgeo-org.1560.x6.nabble.com/Jave-heap-space-exception-on-GetFeatureInfo-request-tp5105467p5105726.html

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

4

Well, at least the native memory is growing and growing, that's why I think that is the case.
I was able to increase the (native) memory footprint by ca. 200 MB in just 5 minutes:

screenshot #1 (15:32): http://i59.tinypic.com/2rzeulg.png
screenshot #2 (15:37): http://i61.tinypic.com/xlgle9.png

The major part of the memory is categorized as "Other"
http://i59.tinypic.com/2a77p0w.png

The biggest jumps definitely occured when clicking on features (which then sends a GetFeatureInfo request).
As a sidenote the data is projected (from EPSG:4326 to EPSG:3416).

This is the URL that I've used for testing:

http://as11g:7003/geoserver/wms/reflect?layers=vzg:austria,vzg:strecken,vzg:strklassen,vzg:bst&srs=EPSG:3416&format=application/openlayers

"austria" is a shapefile-layer, all the other ones are served from the OCI DB datasource.

A java heap dump is not going to cut it in that case, I guess. Of course I can provide one if that helps.
I am by no means an expert analyzing memory leaks, fortunately I haven't had to deal with something like that, until now that is.
The screenshots are from Mission Control, which is bundled with JRockit btw.

Regards,
Daniel

On 08.04.2014 09:19, Jody Garnett wrote:

Please create a ticket, with whatever details you have.

There is no real developer community priority a preferred datastore. Comes down to what volunteers are willing to work on (or what customers are willing to pay for). In this case having a bug report to point to is a good way to start the conversation.

So you are thinking the memory link is in the native code - and thus not something that shows up in a memory profiler?
--
Jody

Jody Garnett

On Mon, Apr 7, 2014 at 11:11 PM, daniel.soneira <daniel.soneira@anonymised.com <mailto:daniel.soneira@anonymised.com>> wrote:

    Hello GeoServer community,

    I've spent quite some time (all of last week) investigating a
    performance
    problem regarding our mapping application built on top of GeoServer /
    OpenLayers.
    The memory consumption grew as time went by and never recovered /
    declined
    anymore to the point where the operating system began to swap
    which resulted
    in a really poor performance.
    The log entry 'rendering takes >60 sec, aborting..' came up in shorter
    intervals and at some point the server just didn't respond anymore.

    To find the reason for that I've used different servers (Tomcat /
    WebLogic)
    and even different JVMs (HotSpot / JRockit).
    A heap dump didn't really show anything extraordinary, but looking
    at some
    other data the native memory part was suspicious.
    As it turns out the culprit seems to be the OCI drivers (or their
    usage).

    The problem can be observed by using the built-in Layer Preview
    feature
    (just moving around the map - panning / zooming in and out is
    sufficient).
    As soon as I switch to Thin-Driver the application performs better
    and the
    memory stays within some sensible boundaries (after some initial
    climbing).

    Since the usage of OCI is mandatory for our customer the
    Thin-Driver option
    is unfortunately not possible.

    I guess Oracle is not the preferred database in the GeoServer
    community and
    OCI is even less used to access it but has anyone some experience
    in dealing
    with such a situation nontheless?

    Should I create a ticket for that?

    Thanks for any input in advance,

    Daniel

    ==============================================================================
    Datasource settings (OCI):
            Connection timeout: 20
            validate connections: true
            max connections: 10
            Primary key metadata table: geo_pk_metadata
            Loose bbox: true
            Expose primary keys: true
            Max open prepared statements: 100
            fetch size: 1000
            Estimated extends: true
            min connections: 1

    ==============================================================================
    Environment:
            GeoServer: 2.5 final

            App-Server / Servlet container: WebLogic 10.3.2 (it
    behaves the same on
    Tomcat 6)

            OS: Red Hat Enterprise Linux Server release 5.4 (Tikanga)

            Database: Oracle Database 11g Enterprise Edition Release
    11.2.0.3.0 - 64bit
    Production

            JVM:
            Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
            BEA JRockit(R) (build
    R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32, compiled mode)

            Java settings: -Xmx1024m -Xms1024m

    --
    View this message in context:
    http://osgeo-org.1560.x6.nabble.com/Suspected-memory-leak-when-using-Oracle-OCI-drivers-tp5133570.html
    Sent from the GeoServer - User mailing list archive at Nabble.com.

    ------------------------------------------------------------------------------
    Put Bad Developers to Shame
    Dominate Development with Jenkins Continuous Integration
    Continuously Automate Build, Test & Deployment
    Start a new project now. Try Jenkins in the cloud.
    http://p.sf.net/sfu/13600_Cloudbees_APR
    _______________________________________________
    Geoserver-users mailing list
    Geoserver-users@lists.sourceforge.net
    <mailto:Geoserver-users@lists.sourceforge.net>
    https://lists.sourceforge.net/lists/listinfo/geoserver-users