[Geoserver-users] WPS Process performance

GeoServer GeoServer User
1

Geoserver 2.5-RC1
Geotools 11-RC1

The GetBblAndBin process (attached) is doing a bbox query for BUILDING features (Bin)...
The query for PLUTO features (Bbl) has been commented so as to have a reasonable comparison to a WFS GetFeature request for BUILDING.

The WPS process takes in excess of 5 seconds to complete, while a WFS GetFeature is nearly instantaneous (http://msdlva-gisapp01.csc.nycnet:82/geoserver/wfs?request=GetFeature&version=1.1.0&typeName=public:BUILDING&BBOX=984450,203736,984453,203739,EPSG:2263&outputFormat=application/json).

The queries performed by the WFS and WPS are nearly identical except for the record number limiter of 30000 in the WFS.

I have included 2 debug logs attached (wps.txt, wfs.txt). Of particular concern are the following 2 lines from the WPS log where 5 seconds elapse:

28 Feb 11:37:50 DEBUG [org.geotools.jdbc] - CREATE CONNECTION
28 Feb 11:37:55 DEBUG [org.geotools.jdbc] - CLOSE CONNECTION

I am not sure what is going on here, but I guess I am not accessing the FeatureSource in an optimal manner.

Thanks in advance for any assistance.

-Tim

wps.txt (15.7 KB)

GetBblAndBin.java (7.5 KB)

Util.java (1.56 KB)

applicationContext.xml (555 Bytes)

pom.xml (1.88 KB)

wfs.txt (54 KB)

2

You may wish to limit this discussion to the user list.

If you use getFeatures( Query ) you can limit the result set produced, which should be significantly faster then allowing the database to queue up a larger result set which is then mostly ignored.

As for your delay, you did not indicate which database you were using - but in general obtaining a connection can be quite expensive which is why we use a connection pool internally to recycle connections. It may be luck of the draw which process is stuck obtaining a new connection, it is best to do your performance comparison over several requests.

···

Jody Garnett

On Sat, Mar 1, 2014 at 4:05 AM, Keane, Tim <tkeane@anonymised.com> wrote:

Geoserver 2.5-RC1
Geotools 11-RC1

The GetBblAndBin process (attached) is doing a bbox query for BUILDING features (Bin)…
The query for PLUTO features (Bbl) has been commented so as to have a reasonable comparison to a WFS GetFeature request for BUILDING.

The WPS process takes in excess of 5 seconds to complete, while a WFS GetFeature is nearly instantaneous (http://msdlva-gisapp01.csc.nycnet:82/geoserver/wfs?request=GetFeature&version=1.1.0&typeName=public:BUILDING&BBOX=984450,203736,984453,203739,EPSG:2263&outputFormat=application/json).

The queries performed by the WFS and WPS are nearly identical except for the record number limiter of 30000 in the WFS.

I have included 2 debug logs attached (wps.txt, wfs.txt). Of particular concern are the following 2 lines from the WPS log where 5 seconds elapse:

28 Feb 11:37:50 DEBUG [org.geotools.jdbc] - CREATE CONNECTION
28 Feb 11:37:55 DEBUG [org.geotools.jdbc] - CLOSE CONNECTION

I am not sure what is going on here, but I guess I am not accessing the FeatureSource in an optimal manner.

Thanks in advance for any assistance.

-Tim

Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

3

Ciao Tim,
Jody is right, you bypassing GeoServer internal machinery creating
connection pools over and over (not even sure you are dispoing them at
the end...).

To rely on GeoServer internal machinery you can follow this example:
https://github.com/afabiani/geoserver-enterprise/blob/downloadServices-latest/src/extension/wps-download/src/main/java/org/geoserver/wps/gs/DownloadProcess.java#L141

There we are trying to get ahold of the FeatureTypeInfo in GeoServer
for a layer to extract features for a custom download service.
Check the sequence of layer name --> layerInfo --> ResourceInfo

Then, if the ResourceInfo is a FeatureTypeInfo (vector layer) we go to
the right download service:

https://github.com/afabiani/geoserver-enterprise/blob/downloadServices-latest/src/extension/wps-download/src/main/java/org/geoserver/wps/gs/VectorDownload.java#L80

Sorry to be this short but I believe you can get some good indications anyway.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Sun, Mar 2, 2014 at 4:58 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

You may wish to limit this discussion to the user list.

If you use getFeatures( Query ) you can limit the result set produced, which
should be significantly faster then allowing the database to queue up a
larger result set which is then mostly ignored.

As for your delay, you did not indicate which database you were using - but
in general obtaining a connection can be quite expensive which is why we use
a connection pool internally to recycle connections. It may be luck of the
draw which process is stuck obtaining a new connection, it is best to do
your performance comparison over several requests.

Jody Garnett

On Sat, Mar 1, 2014 at 4:05 AM, Keane, Tim <tkeane@anonymised.com> wrote:

Geoserver 2.5-RC1
Geotools 11-RC1

The GetBblAndBin process (attached) is doing a bbox query for BUILDING
features (Bin)...
The query for PLUTO features (Bbl) has been commented so as to have a
reasonable comparison to a WFS GetFeature request for BUILDING.

The WPS process takes in excess of 5 seconds to complete, while a WFS
GetFeature is nearly instantaneous
(http://msdlva-gisapp01.csc.nycnet:82/geoserver/wfs?request=GetFeature&version=1.1.0&typeName=public:BUILDING&BBOX=984450,203736,984453,203739,EPSG:2263&outputFormat=application/json).

The queries performed by the WFS and WPS are nearly identical except for
the record number limiter of 30000 in the WFS.

I have included 2 debug logs attached (wps.txt, wfs.txt). Of particular
concern are the following 2 lines from the WPS log where 5 seconds elapse:

28 Feb 11:37:50 DEBUG [org.geotools.jdbc] - CREATE CONNECTION
28 Feb 11:37:55 DEBUG [org.geotools.jdbc] - CLOSE CONNECTION

I am not sure what is going on here, but I guess I am not accessing the
FeatureSource in an optimal manner.

Thanks in advance for any assistance.

-Tim

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.

http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

GeoTools is supposed to have a tips and tricks “welcome” page for GeoServer developers.
I will try and write down Simone’s notes for next time.

···

Jody Garnett

On Mon, Mar 3, 2014 at 7:31 AM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Tim,
Jody is right, you bypassing GeoServer internal machinery creating
connection pools over and over (not even sure you are dispoing them at
the end…).

To rely on GeoServer internal machinery you can follow this example:
https://github.com/afabiani/geoserver-enterprise/blob/downloadServices-latest/src/extension/wps-download/src/main/java/org/geoserver/wps/gs/DownloadProcess.java#L141

There we are trying to get ahold of the FeatureTypeInfo in GeoServer
for a layer to extract features for a custom download service.
Check the sequence of layer name → layerInfo → ResourceInfo

Then, if the ResourceInfo is a FeatureTypeInfo (vector layer) we go to
the right download service:

https://github.com/afabiani/geoserver-enterprise/blob/downloadServices-latest/src/extension/wps-download/src/main/java/org/geoserver/wps/gs/VectorDownload.java#L80

Sorry to be this short but I believe you can get some good indications anyway.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

On Sun, Mar 2, 2014 at 4:58 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

You may wish to limit this discussion to the user list.

If you use getFeatures( Query ) you can limit the result set produced, which
should be significantly faster then allowing the database to queue up a
larger result set which is then mostly ignored.

As for your delay, you did not indicate which database you were using - but
in general obtaining a connection can be quite expensive which is why we use
a connection pool internally to recycle connections. It may be luck of the
draw which process is stuck obtaining a new connection, it is best to do
your performance comparison over several requests.

Jody Garnett

On Sat, Mar 1, 2014 at 4:05 AM, Keane, Tim <tkeane@anonymised.com> wrote:

Geoserver 2.5-RC1
Geotools 11-RC1

The GetBblAndBin process (attached) is doing a bbox query for BUILDING
features (Bin)…
The query for PLUTO features (Bbl) has been commented so as to have a
reasonable comparison to a WFS GetFeature request for BUILDING.

The WPS process takes in excess of 5 seconds to complete, while a WFS
GetFeature is nearly instantaneous
(http://msdlva-gisapp01.csc.nycnet:82/geoserver/wfs?request=GetFeature&version=1.1.0&typeName=public:BUILDING&BBOX=984450,203736,984453,203739,EPSG:2263&outputFormat=application/json).

The queries performed by the WFS and WPS are nearly identical except for
the record number limiter of 30000 in the WFS.

I have included 2 debug logs attached (wps.txt, wfs.txt). Of particular
concern are the following 2 lines from the WPS log where 5 seconds elapse:

28 Feb 11:37:50 DEBUG [org.geotools.jdbc] - CREATE CONNECTION
28 Feb 11:37:55 DEBUG [org.geotools.jdbc] - CLOSE CONNECTION

I am not sure what is going on here, but I guess I am not accessing the
FeatureSource in an optimal manner.

Thanks in advance for any assistance.

-Tim

Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.

http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users