[Geoserver-devel] who said we need perf/load/stress tests?

Hi all,

having some load problems here with geoserver.

Trying to serve a ~150MB shapefile thru geoserver 1.6 and 1.7 I'm giving up.
Also tried the same dataset thru postgis, no luck.

At a first glance I thought it were the shapefile indexing stuff eating memory
and taking so long as to timeout any connection attempt. A hand made
connection (java code) setting the connection and read timeouts to infinite
lead to an "unexpected end of file from server".
But what surprised me more is the same behavior trying to fetch gml from
postgis.

Moreover, the geoserver VM memory (ran thru mvn jetty:run) grows up from ~300M
up to 540M without sending back a single feature (note the VM is ran
with -Xmx1024m).

On the bright side, geoserver 1.5 responds quite instantly and serves the data
quite well.

so that's is, I'm too tired to hunt out what's going on right now, but will do
tomorrow, in the hope it is just something wrong with my environment.

But if any of you have a chance to try out GetFeature with a recent geos
version with some reasonably sized dataset please reply with success/failure
notice.

btw, the shapefile I'm trying out is the gshhs_land.shp from
<http://www.ngdc.noaa.gov/mgg/shorelines/gshhs.html&gt;

cheers,

Gabriel.

Hint:
with geoserver 1.6.x, disabling "create spatial index" and "use memory mapped
buffer" on the shapefile config, got some results:
GML2 output format works.
GML3 output format starts to stream out but consistently breaks when served
around 70MB of gml (whole gml document is around 300MB).
It breaks with the following exception:

java.io.IOException: Error occurred encoding features
        at
org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:125)
        at
org.geoserver.wfs.WFSGetFeatureOutputFormat.write(WFSGetFeatureOutputFormat.java:137)
        at org.geoserver.ows.Dispatcher.response(Dispatcher.java:615)
....
Caused by: org.mortbay.jetty.EofException
        at org.apache.xml.serialize.BaseMarkupSerializer.characters(Unknown
Source)
        at org.geotools.xml.Encoder.start(Encoder.java:719)
        at org.geotools.xml.Encoder.encode(Encoder.java:566)
        at org.geotools.xml.Encoder.encode(Encoder.java:320)
        at
org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:122)
        ... 50 more

Gabriel

On Thursday 20 December 2007 12:11:07 am Gabriel Roldán wrote:

Hi all,

having some load problems here with geoserver.

Trying to serve a ~150MB shapefile thru geoserver 1.6 and 1.7 I'm giving
up. Also tried the same dataset thru postgis, no luck.

At a first glance I thought it were the shapefile indexing stuff eating
memory and taking so long as to timeout any connection attempt. A hand made
connection (java code) setting the connection and read timeouts to infinite
lead to an "unexpected end of file from server".
But what surprised me more is the same behavior trying to fetch gml from
postgis.

Moreover, the geoserver VM memory (ran thru mvn jetty:run) grows up from
~300M up to 540M without sending back a single feature (note the VM is ran
with -Xmx1024m).

On the bright side, geoserver 1.5 responds quite instantly and serves the
data quite well.

so that's is, I'm too tired to hunt out what's going on right now, but will
do tomorrow, in the hope it is just something wrong with my environment.

But if any of you have a chance to try out GetFeature with a recent geos
version with some reasonably sized dataset please reply with
success/failure notice.

btw, the shapefile I'm trying out is the gshhs_land.shp from
<http://www.ngdc.noaa.gov/mgg/shorelines/gshhs.html&gt;

cheers,

Gabriel.

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplac
e _______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4045,4769a548196482458217002!

Not sure I am answering your question, but here goes:

I have configured this server as follows:
http://docs.codehaus.org/display/GEOSDOC/CentOS+(Red+Hat)+5.1+Install
(I believe with GEOS 2.2.3 , but I don't know how to check Postgis is actually using it)

The following serves up the tiger dataset (many gigabytes) from Postgis if you zoom in on the states:
http://64.147.114.66:8080/geoserver/wms?bbox=-179.999908447266,-90.0,179.999908447266,83.6274185180664&styles=&Format=application/openlayers&request=GetMap&layers=sigma,topp:states&width=800&height=362&srs=EPSG:4326

So a sample getfeature would be
<wfs:GetFeature service="WFS" version="1.1.0"
xmlns:topp="http://www.openplans.org/topp&quot;
xmlns:wfs="http://www.opengis.net/wfs&quot;
xmlns:ogc="http://www.opengis.net/ogc&quot;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot;
xsi:schemaLocation="http://www.opengis.net/wfs
                     http://schemas.opengis.net/wfs/1.1.0/wfs.xsd&quot;&gt;
<wfs:Query typeName="topp:roads">
   <ogc:Filter>
      <ogc:FeatureId fid="roads.TGR48353&amp;186307642"/>
   </ogc:Filter>
   </wfs:Query>
</wfs:GetFeature>

to http://64.147.114.66:8080/geoserver/wfs

which responds very fast.

Anyway, feel free to hammer http://64.147.114.66:8080 for a while if it's of any use to you, nobody's dependent on it.

-Arne

Gabriel Roldán wrote:

Hint:
with geoserver 1.6.x, disabling "create spatial index" and "use memory mapped buffer" on the shapefile config, got some results:
GML2 output format works.
GML3 output format starts to stream out but consistently breaks when served around 70MB of gml (whole gml document is around 300MB).
It breaks with the following exception:

java.io.IOException: Error occurred encoding features
        at org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:125)
        at org.geoserver.wfs.WFSGetFeatureOutputFormat.write(WFSGetFeatureOutputFormat.java:137)
        at org.geoserver.ows.Dispatcher.response(Dispatcher.java:615)
....
Caused by: org.mortbay.jetty.EofException
        at org.apache.xml.serialize.BaseMarkupSerializer.characters(Unknown Source)
        at org.geotools.xml.Encoder.start(Encoder.java:719)
        at org.geotools.xml.Encoder.encode(Encoder.java:566)
        at org.geotools.xml.Encoder.encode(Encoder.java:320)
        at org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:122)
        ... 50 more

Gabriel

On Thursday 20 December 2007 12:11:07 am Gabriel Roldán wrote:
  

Hi all,

having some load problems here with geoserver.

Trying to serve a ~150MB shapefile thru geoserver 1.6 and 1.7 I'm giving
up. Also tried the same dataset thru postgis, no luck.

At a first glance I thought it were the shapefile indexing stuff eating
memory and taking so long as to timeout any connection attempt. A hand made
connection (java code) setting the connection and read timeouts to infinite
lead to an "unexpected end of file from server".
But what surprised me more is the same behavior trying to fetch gml from
postgis.

Moreover, the geoserver VM memory (ran thru mvn jetty:run) grows up from
~300M up to 540M without sending back a single feature (note the VM is ran
with -Xmx1024m).

On the bright side, geoserver 1.5 responds quite instantly and serves the
data quite well.

so that's is, I'm too tired to hunt out what's going on right now, but will
do tomorrow, in the hope it is just something wrong with my environment.

But if any of you have a chance to try out GetFeature with a recent geos
version with some reasonably sized dataset please reply with
success/failure notice.

btw, the shapefile I'm trying out is the gshhs_land.shp from
<http://www.ngdc.noaa.gov/mgg/shorelines/gshhs.html&gt;

cheers,

Gabriel.

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplac
e _______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4038,4769a992211806491211187!

Thanks Arne, this gives some relief, I'm still hoping it is just something
wrong on my side.

I'll be hitting it for a while, starting tomorrow. Also it makes sense if it
works well with a well configured postgis, wonder if my problems are just
shapefile indexing plus connection timeouts though.

thanks again for the tip

Gabriel

On Thursday 20 December 2007 12:45:28 am Arne Kepp wrote:

Not sure I am answering your question, but here goes:

I have configured this server as follows:
http://docs.codehaus.org/display/GEOSDOC/CentOS+(Red+Hat)+5.1+Install
(I believe with GEOS 2.2.3 , but I don't know how to check Postgis is
actually using it)

The following serves up the tiger dataset (many gigabytes) from Postgis
if you zoom in on the states:
http://64.147.114.66:8080/geoserver/wms?bbox=-179.999908447266,-90.0,179.99
9908447266,83.6274185180664&styles=&Format=application/openlayers&request=Ge
tMap&layers=sigma,topp:states&width=800&height=362&srs=EPSG:4326

So a sample getfeature would be
<wfs:GetFeature service="WFS" version="1.1.0"
  xmlns:topp="http://www.openplans.org/topp&quot;
  xmlns:wfs="http://www.opengis.net/wfs&quot;
  xmlns:ogc="http://www.opengis.net/ogc&quot;
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot;
  xsi:schemaLocation="http://www.opengis.net/wfs
                      http://schemas.opengis.net/wfs/1.1.0/wfs.xsd&quot;&gt;
  <wfs:Query typeName="topp:roads">
    <ogc:Filter>
       <ogc:FeatureId fid="roads.TGR48353&amp;186307642"/>
    </ogc:Filter>
    </wfs:Query>
</wfs:GetFeature>

to http://64.147.114.66:8080/geoserver/wfs

which responds very fast.

Anyway, feel free to hammer http://64.147.114.66:8080 for a while if
it's of any use to you, nobody's dependent on it.

-Arne

Gabriel Roldán wrote:
> Hint:
> with geoserver 1.6.x, disabling "create spatial index" and "use memory
> mapped buffer" on the shapefile config, got some results:
> GML2 output format works.
> GML3 output format starts to stream out but consistently breaks when
> served around 70MB of gml (whole gml document is around 300MB).
> It breaks with the following exception:
>
> java.io.IOException: Error occurred encoding features
> at
> org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:125)
> at
> org.geoserver.wfs.WFSGetFeatureOutputFormat.write(WFSGetFeatureOutputForm
>at.java:137) at org.geoserver.ows.Dispatcher.response(Dispatcher.java:615)
> ....
> Caused by: org.mortbay.jetty.EofException
> at
> org.apache.xml.serialize.BaseMarkupSerializer.characters(Unknown Source)
> at org.geotools.xml.Encoder.start(Encoder.java:719)
> at org.geotools.xml.Encoder.encode(Encoder.java:566)
> at org.geotools.xml.Encoder.encode(Encoder.java:320)
> at
> org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:122)
> ... 50 more
>
> Gabriel
>
> On Thursday 20 December 2007 12:11:07 am Gabriel Roldán wrote:
>> Hi all,
>>
>> having some load problems here with geoserver.
>>
>> Trying to serve a ~150MB shapefile thru geoserver 1.6 and 1.7 I'm giving
>> up. Also tried the same dataset thru postgis, no luck.
>>
>> At a first glance I thought it were the shapefile indexing stuff eating
>> memory and taking so long as to timeout any connection attempt. A hand
>> made connection (java code) setting the connection and read timeouts to
>> infinite lead to an "unexpected end of file from server".
>> But what surprised me more is the same behavior trying to fetch gml from
>> postgis.
>>
>> Moreover, the geoserver VM memory (ran thru mvn jetty:run) grows up from
>> ~300M up to 540M without sending back a single feature (note the VM is
>> ran with -Xmx1024m).
>>
>> On the bright side, geoserver 1.5 responds quite instantly and serves
>> the data quite well.
>>
>> so that's is, I'm too tired to hunt out what's going on right now, but
>> will do tomorrow, in the hope it is just something wrong with my
>> environment.
>>
>> But if any of you have a chance to try out GetFeature with a recent geos
>> version with some reasonably sized dataset please reply with
>> success/failure notice.
>>
>> btw, the shapefile I'm trying out is the gshhs_land.shp from
>> <http://www.ngdc.noaa.gov/mgg/shorelines/gshhs.html&gt;
>>
>> cheers,
>>
>> Gabriel.
>>
>> ------------------------------------------------------------------------
>>- SF.Net email is sponsored by:
>> Check out the new SourceForge.net Marketplace.
>> It's the best place to buy or sell services
>> for just about anything Open Source.
>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketp
>>lac e _______________________________________________
>> Geoserver-devel mailing list
>> Geoserver-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketpl
>ace _______________________________________________
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4045,4769ad19216926491211187!

Gabriel Roldán ha scritto:

Thanks Arne, this gives some relief, I'm still hoping it is just something wrong on my side.

I'll be hitting it for a while, starting tomorrow. Also it makes sense if it works well with a well configured postgis, wonder if my problems are just shapefile indexing plus connection timeouts though.

I was thinking about it after we talked yesterday and I think it may be a matter of the dataset not of the configuration.
The detailed shoreline database you pointed us at will probably have
single geometries with millions of coordinates.
The GML3 encoder will build a DOM node for each single feature before
encoding it in XML, so I fear there is nothing we can do about it.
The GML2 encoder is more stream oriented but the single coordinate
sequence gets turned into a string buffer before turning it into
an xml element so I guess it suffers from similar problems when
faced with a gigantic coordinate sequence. Maybe the GML2 can be
modified to stream at the coordinate level, I have to investigate
it.

The dataset Arne is referring to is made of millions of little
geometries so its well behaved in both cases.

Anyways, only a little test with a memory profiler can tell us
the truth.

Cheers
Andrea

Gabriel Roldán ha scritto:

Thanks Arne, this gives some relief, I'm still hoping it is just something wrong on my side.

I can confirm there is something wrong on my side too.
I loaded the gshhs_land data (150MB right?) and asked to generate
the gml2 out of it. The resonse was immediate:

curl "http://localhost:8080/geoserver/wfs?request=GetFeature&service=WFS&version=1.0.0&typeName=topp:gshhs_land&quot; > gshhs_land.gml
   % Total % Received % Xferd Average Speed Time Time Time Current
                                  Dload Upload Total Spent Left Speed
100 323M 0 323M 0 0 5379k 0 --:--:-- 0:01:01 --:--:-- 8374k

So with WFS 1.0, GML2 output all right, just slow because this
is trunk (on 1.6.0 you can probably get something more like 10MB/s).

The I tried GML3:
C:\Temp\test>curl "http://localhost:8080/geoserver/wfs?request=GetFeature&service=WFS&version=1.1.0&typeName=topp:gshhs_land&quot; > gshhs_land.gml

and the result is:
java.lang.OutOfMemoryError: Java heap space
  at org.geotools.geometry.jts.DefaultCoordinateSequenceTransformer.transform(DefaultCoordinateSequenceTransformer.java:128)

Ah ah, now I get it. The GML3 output defaults on lat/lon axis order, so we grab the coordinates and reproject them. In this case the coordinate
array is too big to fit the 256MB memory limit I set.

If I remove the transformation imposing a lon/lat output I get an OOM
error too anyways... so indeed the GML3 has higher memory requirements or memory leaks... not sure were, I mean, with this big geometries it's
enough to keep a few DOM elements representing features around and
kaboom.

I tried with 512 too and 1024mb too, with 1024 in the end it was
able to generate out the xml althought my PC was nearly killed by it
(XP does not like having processes that do consume that much memory,
it enters swap storm).

This is some work for xml-man (aka Justin) to solve I guess. In the
meantime we'll have to tell users to either stay away from huge
geometries, stay away from GML3, or buy lots of memory (pick your
poison) :slight_smile:
Cheers
Andrea

Hey Andrea,

thanks a lot for downloading the data and trying it out, I really appreciate
that.

It seems you got it right, now it makes complete sense to me.

aside, I didn't notice a throughput slowdown from 1.5. to 1.6 for GML2 output,
yet we didn't establish a minimum throughput as a requirement, though even if
4M/s seems like not the higher possible rate, I would be glad if it scales
well, say, serving 10 concurrent requests at 4M/s. But that's load testing,
so I hope with the new servers we can at least have a manually ran jmeter
project, as long as we can make the time to work on it, of course.

cheers,

Gabriel

On Thursday 20 December 2007 09:03:18 am Andrea Aime wrote:

Gabriel Roldán ha scritto:
> Thanks Arne, this gives some relief, I'm still hoping it is just
> something wrong on my side.

I can confirm there is something wrong on my side too.
I loaded the gshhs_land data (150MB right?) and asked to generate
the gml2 out of it. The resonse was immediate:

curl
"http://localhost:8080/geoserver/wfs?request=GetFeature&service=WFS&version
=1.0.0&typeName=topp:gshhs_land"

> gshhs_land.gml

   % Total % Received % Xferd Average Speed Time Time Time
  Current
                                  Dload Upload Total Spent Left
  Speed
100 323M 0 323M 0 0 5379k 0 --:--:-- 0:01:01 --:--:--
8374k

So with WFS 1.0, GML2 output all right, just slow because this
is trunk (on 1.6.0 you can probably get something more like 10MB/s).

The I tried GML3:
C:\Temp\test>curl
"http://localhost:8080/geoserver/wfs?request=GetFeature&service=WFS&version
=1.1.0&typeName=topp:gshhs_land"

> gshhs_land.gml

and the result is:
java.lang.OutOfMemoryError: Java heap space
  at
org.geotools.geometry.jts.DefaultCoordinateSequenceTransformer.transform(De
faultCoordinateSequenceTransformer.java:128)

Ah ah, now I get it. The GML3 output defaults on lat/lon axis order, so
we grab the coordinates and reproject them. In this case the coordinate
array is too big to fit the 256MB memory limit I set.

If I remove the transformation imposing a lon/lat output I get an OOM
error too anyways... so indeed the GML3 has higher memory requirements
or memory leaks... not sure were, I mean, with this big geometries it's
enough to keep a few DOM elements representing features around and
kaboom.

I tried with 512 too and 1024mb too, with 1024 in the end it was
able to generate out the xml althought my PC was nearly killed by it
(XP does not like having processes that do consume that much memory,
it enters swap storm).

This is some work for xml-man (aka Justin) to solve I guess. In the
meantime we'll have to tell users to either stay away from huge
geometries, stay away from GML3, or buy lots of memory (pick your
poison) :slight_smile:
Cheers
Andrea

!DSPAM:4045,476a21d6179742092453641!

On Thu, December 20, 2007 1:36 pm, Gabriel Roldán wrote:

Hey Andrea,

thanks a lot for downloading the data and trying it out, I really
appreciate that.

It seems you got it right, now it makes complete sense to me.

aside, I didn't notice a throughput slowdown from 1.5. to 1.6 for GML2
output, yet we didn't establish a minimum throughput as a requirement,
though even if 4M/s seems like not the higher possible rate, I would be
glad if it scales well, say, serving 10 concurrent requests at 4M/s.

Hum, I did test with trunk, not 1.6.x, trunk is where the slowdown is.
With 1.6.x you can get say 10mb/s on my pc, hard to get more since almost
all of the time is spent formatting doubles into decimals to respect the
xs:decimal type and the number of decimals.

On GML3 this does not happen (the type is xs:decimal) but there are other,
bigger causes of slowdown. Unfortunately I'm not able to provide any good
solution for that.

Cheers
Andrea

On Thu, December 20, 2007 3:01 pm, aaime@anonymised.com wrote:

On GML3 this does not happen (the type is xs:decimal)

Sorry, I meant the type is xs:double
Cheers
Andrea

This is some work for xml-man (aka Justin) to solve I guess. In the
meantime we'll have to tell users to either stay away from huge
geometries, stay away from GML3, or buy lots of memory (pick your
poison) :slight_smile:

Hmmm... this is quite serious. I did not see a jira opened for this one
so i am creating one:

http://jira.codehaus.org/browse/GEOS-1634

I scheduled for 1.6.1 since it will require some changes to the xml
encoder... hopefully not too profound though.

-Justin

Cheers
Andrea

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,476a21e1179935332866982!

--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org