[Geoserver-devel] PNG encoder comparison - complex stylings

[Jonathan_Moules2]

Hi List,
I’ve been testing the new PNG Encoder with JMeter compared to whatever the default on Windows is without it installed (Oracle JDK?).

I’m testing with 10 users/threads using a layergroup which is scale thresh-holded from 1:10million to 1:250; lots of different source layers and SLD’s behind it.
It’s a slightly altered version of the default basemap here: - http://maps.warwickshire.gov.uk/inspire/

Anyway, the results from the testing seem to indicate that the new encoder not only produces a larger file (as determined previously), but in several instances is slower. I’m not seeing any real speedup as you can see from the attached.

OracleJDK encoder

PNG Encoder:

6 of the 16 layers tested were slower with the new encoder, by up to 20%. The overall average is also slightly slower for the new encoder.

5 of the six were the smallest scale (most zoomed out) levels.

The times for the PNG encoder plummet when reduced to a single thread:

Might there be some Achilles heal that the new encoder isn’t optimised for?

Running on a 16-core physical machine with typical CPU use of 1%. Use rose to about 30-40% with the 10 thread tests.

Thoughts?
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Andrea_Aime4]

On Mon, Dec 23, 2013 at 6:06 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi List,
I've been testing the new PNG Encoder with JMeter compared to whatever the
default on Windows is without it installed (Oracle JDK?).

I'm testing with 10 users/threads using a layergroup which is scale
thresh-holded from 1:10million to 1:250; lots of different source layers
and SLD's behind it.
It's a slightly altered version of the default basemap here: -
http://maps.warwickshire.gov.uk/inspire/

Anyway, the results from the testing seem to indicate that the new encoder
not only produces a larger file (as determined previously), but in several
instances is slower. I'm not seeing any real speedup as you can see from
the attached.

OracleJDK encoder
!png_encoded.png|937x299

PNG Encoder:
!oracle_encoded.png|974x302

6 of the 16 layers tested were slower with the new encoder, by up to 20%.
The overall average is also slightly slower for the new encoder.

5 of the six were the smallest scale (most zoomed out) levels.

The times for the PNG encoder plummet when reduced to a single thread:

!png_encoded_1_thread.png|940x300

Might there be some Achilles heal that the new encoder isn't optimised for?

Running on a 16-core physical machine with typical CPU use of 1%. Use rose
to about 30-40% with the 10 thread tests.

Hmm... not sure. I find it hard to believe the encoder can make any
difference when the throughput is
around 8 responses _a minute_ (if I'm reading correctly, the images are
small enough that I find it hard to read them),
in my tests it does not make any visible difference until you get to around
300 responses a minute (or in other
words, at least 3 per second, which is already a rather low value, the
FOSS4G benchmarks I've
referred to in the tests are returning something like 60 responses a
second, or 3600 a minute,
with a dataset that covers the entire Spain, even if the scale rules make
for a map that's quite a bit sparser
than yours).
Wondering, how many layers do you have in that layer group?

My gut feeling, something else is going on during those tests, like more
load in the data sources,
that skewed the results, and most likely, also the cases in which results
got faster are most
likely due to environmental conditions as well... which is also sort of
confirmed by the CPU
usage you have, in my benchmark I have all CPU pegged to almost 100% with
OpenJDK,
and at least at around 60-70% with the Oracle one.

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,
I’ve put the images of the results, plus the csv files from jmeter here if you want to check: http://maps.warwickshire.gov.uk/misc/

That layer group comprises 6 other layer groups, each of which is basically a different Ordnance Survey product set.
The super zoomed out stuff (1:150,000 outward) is all OS Strategi which contains 15 layers - all Oracle. There’s also a “EU basemap” group under it which contains 3 layers (those are all local shapefiles).

(attachments)

···

Weird that I’m throwing about 4 times as many cores at the problem than you are (based on your ~4 core VM), but still max out at 60 requests a second. Co-incidence?

Sorry if I’m failing to understand something. I don’t think I’m missing anything obvious.

Cheers,
Jonathan

On 23 December 2013 17:23, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Dec 23, 2013 at 6:06 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
I’ve been testing the new PNG Encoder with JMeter compared to whatever the default on Windows is without it installed (Oracle JDK?).

I’m testing with 10 users/threads using a layergroup which is scale thresh-holded from 1:10million to 1:250; lots of different source layers and SLD’s behind it.
It’s a slightly altered version of the default basemap here: - http://maps.warwickshire.gov.uk/inspire/

Anyway, the results from the testing seem to indicate that the new encoder not only produces a larger file (as determined previously), but in several instances is slower. I’m not seeing any real speedup as you can see from the attached.

OracleJDK encoder

PNG Encoder:

6 of the 16 layers tested were slower with the new encoder, by up to 20%. The overall average is also slightly slower for the new encoder.

5 of the six were the smallest scale (most zoomed out) levels.

The times for the PNG encoder plummet when reduced to a single thread:

Might there be some Achilles heal that the new encoder isn’t optimised for?

Running on a 16-core physical machine with typical CPU use of 1%. Use rose to about 30-40% with the 10 thread tests.

Hmm… not sure. I find it hard to believe the encoder can make any difference when the throughput is
around 8 responses a minute (if I’m reading correctly, the images are small enough that I find it hard to read them),
in my tests it does not make any visible difference until you get to around 300 responses a minute (or in other
words, at least 3 per second, which is already a rather low value, the FOSS4G benchmarks I’ve
referred to in the tests are returning something like 60 responses a second, or 3600 a minute,
with a dataset that covers the entire Spain, even if the scale rules make for a map that’s quite a bit sparser
than yours).
Wondering, how many layers do you have in that layer group?

My gut feeling, something else is going on during those tests, like more load in the data sources,
that skewed the results, and most likely, also the cases in which results got faster are most
likely due to environmental conditions as well… which is also sort of confirmed by the CPU
usage you have, in my benchmark I have all CPU pegged to almost 100% with OpenJDK,
and at least at around 60-70% with the Oracle one.

Cheers
Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

[Andrea_Aime4]

On Mon, Dec 23, 2013 at 6:55 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi Andrea,
I've put the images of the results, plus the csv files from jmeter here
if you want to check: http://maps.warwickshire.gov.uk/misc/

That layer group comprises 6 other layer groups, each of which is
basically a different Ordnance Survey product set.
The super zoomed out stuff (1:150,000 outward) is all OS Strategi which
contains 15 layers - all Oracle. There's also a "EU basemap" group under it
which contains 3 layers (those are all local shapefiles).

====

The Oracle datasource is certainly slow, but it seemed odd that it was
slower with the PNG encoder. - Also, as the requests are all for the same
area, wouldn't GeoServer/Oracle have cached the results (speeding things
up)?

Oracle should cache, as should the file system cache for the shapefiles.
GeoServer only caches db connections, but never data
(at least so far).
As said, I find it hard to believe the PNG encoder has anything to do with
the discrepancies you've found, the image
encoding time for an image the size of a screen is way less than a second,
your response times, even with the
single thread, are like 3 seconds instead.

Swapping datasource:
If I test JMeter against the EU Basemap (3 local shapefiles) - PNG
Encoder, even using 50 threads, the CPU usage only goes up to 90%, with a
throughput of 63.8/s. The average response rate there is 747ms, compared to
one thread where it's just 136ms.

This is more like it. The response times of the Oracle based layers are not
slow, they are a tragedy.

So I don't seem to be able to get 100% CPU use.

It's normal, you're on Windows, so you cannot use OpenJDK. The Oracle JDK
rendering subsystem has internal synchronizations
that prevent it to scale up nicely, that's why we suggest to install a
GeoServer every 2-4 cores and load balance them instead.

For the EU basemap, the sweet spot is around 15 threads (85% CPU use) -
61.5/s; any more and geoserver starts lagging.

Yeah, see above about load balancing the GeoServer to get a higher
throughput.

Weird that I'm throwing about 4 times as many cores at the problem than
you are (based on your ~4 core VM), but still max out at 60 requests a
second. Co-incidence?

Most likely, my result is done with OpenJDK, with Oracle one I got around
40-50 instead, and the topp:states map, which has much less data, goes up
to 95 r/s

Sorry if I'm failing to understand something. I don't think I'm missing
anything obvious.

The issue is definitely with the Oracle access, but I don't know where... I
mean, if you could
serve at 60r/s all day continously (I know, not realistic, but let's have a
ballpark) you'd be
serving over 5 million requests a day.
However, when you can serve only 8 requests a minute well... you could as
well ship out paper maps instead.
Now, Oracle is indeed a constant pain, but something this slow is truly
problematic.
We recently discovered an issue with sql views over Oracle, but afaik
you're publishing straight tables
instead, no?

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,
Thanks for the detailed feedback.

So, I’ve decided to eliminate Oracle from the equation. I’m now accessing OS Strategi from the shapefiles (locally sourced). Using the exact same stylesheets as the Oracle ones.

With 1 thread from the shapefile using the PNG Encoder, the times are more reasonable, with an average of 800ms per request (still pretty slow though).
But with 10 threads, the requests take anywhere from 5 to 14 times as long to respond as they did with 1 thread - an average across the board of 7543ms per request. So the PNG Encoder doesn’t appear to be scaling nicely.
CPU usage peaks at 20%.

···

On 23 December 2013 20:08, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Dec 23, 2013 at 6:55 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Andrea,
I’ve put the images of the results, plus the csv files from jmeter here if you want to check: http://maps.warwickshire.gov.uk/misc/

That layer group comprises 6 other layer groups, each of which is basically a different Ordnance Survey product set.
The super zoomed out stuff (1:150,000 outward) is all OS Strategi which contains 15 layers - all Oracle. There’s also a “EU basemap” group under it which contains 3 layers (those are all local shapefiles).

====

The Oracle datasource is certainly slow, but it seemed odd that it was slower with the PNG encoder. - Also, as the requests are all for the same area, wouldn’t GeoServer/Oracle have cached the results (speeding things up)?

Oracle should cache, as should the file system cache for the shapefiles. GeoServer only caches db connections, but never data
(at least so far).

As said, I find it hard to believe the PNG encoder has anything to do with the discrepancies you’ve found, the image
encoding time for an image the size of a screen is way less than a second, your response times, even with the
single thread, are like 3 seconds instead.

Swapping datasource:
If I test JMeter against the EU Basemap (3 local shapefiles) - PNG Encoder, even using 50 threads, the CPU usage only goes up to 90%, with a throughput of 63.8/s. The average response rate there is 747ms, compared to one thread where it’s just 136ms.

This is more like it. The response times of the Oracle based layers are not slow, they are a tragedy.

So I don’t seem to be able to get 100% CPU use.

It’s normal, you’re on Windows, so you cannot use OpenJDK. The Oracle JDK rendering subsystem has internal synchronizations
that prevent it to scale up nicely, that’s why we suggest to install a GeoServer every 2-4 cores and load balance them instead.

For the EU basemap, the sweet spot is around 15 threads (85% CPU use) - 61.5/s; any more and geoserver starts lagging.

Yeah, see above about load balancing the GeoServer to get a higher throughput.

Most likely, my result is done with OpenJDK, with Oracle one I got around 40-50 instead, and the topp:states map, which has much less data, goes up to 95 r/s

The issue is definitely with the Oracle access, but I don’t know where… I mean, if you could
serve at 60r/s all day continously (I know, not realistic, but let’s have a ballpark) you’d be
serving over 5 million requests a day.
However, when you can serve only 8 requests a minute well… you could as well ship out paper maps instead.
Now, Oracle is indeed a constant pain, but something this slow is truly problematic.
We recently discovered an issue with sql views over Oracle, but afaik you’re publishing straight tables
instead, no?

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

Weird that I’m throwing about 4 times as many cores at the problem than you are (based on your ~4 core VM), but still max out at 60 requests a second. Co-incidence?

Sorry if I’m failing to understand something. I don’t think I’m missing anything obvious.

[Andrea_Aime4]

On Tue, Dec 24, 2013 at 1:23 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi Andrea,
Thanks for the detailed feedback.

So, I've decided to eliminate Oracle from the equation. I'm now accessing
OS Strategi from the shapefiles (locally sourced). Using the exact same
stylesheets as the Oracle ones.

With 1 thread from the shapefile using the PNG Encoder, the times are more
reasonable, with an average of 800ms per request (still pretty slow though).
But with 10 threads, the requests take anywhere from 5 to 14 times as long
to respond as they did with 1 thread - an average across the board of
7543ms per request. So the PNG Encoder doesn't appear to be scaling nicely.
CPU usage peaks at 20%.

The fact GeoServer does not scale up properly under Windows is known, but
as far as I know it has nothing to do with the
PNG encoder, it has to do with the java2d rendering engine included in the
Oracle JDK.
It is however strange that you get slightly slower results with PNGJ.

===
I then re-did the Oracle tests (still using the PNG Encoder), just for
this layer (so everything is identical except the data source).
Average times were 2000ms with 1 thread and 8355ms for 10 threads.

------------
I then did all four tests again but with the original encoder. In 3 of the
four cases, on average the PNG Encoder was faster, but with 10 cores from
shapefile the original encoder was faster (7259ms for PNG, 7543ms for
default).

Conclusions:
The data source isn't the sole factor involved in the speed degredation.
PNG encoder doesn't seem to be scaling up well. I'm wondering if rather
than it being the data (it may be), it could also be the styles?

You can see all 8 results here: http://maps.warwickshire.gov.uk/misc/

As OS Strategi is a free dataset, I can provide the shapefiles and styles
and layergroup definitions; if you copy them into one of your instances you
can probably be up and testing inside of a few minutes.

That would be interesting, yes. Where can I fetch them?
And I would need the jmeter plans too

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Andrea_Aime4]

On Tue, Dec 24, 2013 at 2:13 PM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

Conclusions:
The data source isn't the sole factor involved in the speed degredation.
PNG encoder doesn't seem to be scaling up well. I'm wondering if rather
than it being the data (it may be), it could also be the styles?

You can see all 8 results here: http://maps.warwickshire.gov.uk/misc/

As OS Strategi is a free dataset, I can provide the shapefiles and styles
and layergroup definitions; if you copy them into one of your instances you
can probably be up and testing inside of a few minutes.

That would be interesting, yes. Where can I fetch them?

Ok, found them on the OS site, along with a bunch of SLDs. Is that what
you're using, or have you customized them?
Would still be nice to have the layer group definitions.

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,
I’ve pre-packaged everything for you including the data. Layergroups, SLD’s (mine are completely different - no idea what theirs look like), and workspaces.
I think you’ll just need to change where the stores are looking to wherever you put the shapefiles, but you’re going to know better than me.

z_test is the layergroup I’m testing against (it contains the EU basemap and strategi layergroups).

http://maps.warwickshire.gov.uk/misc/strategi.zip - also includes the JMX.

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 24 December 2013 13:35, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Tue, Dec 24, 2013 at 2:13 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ok, found them on the OS site, along with a bunch of SLDs. Is that what you’re using, or have you customized them?
Would still be nice to have the layer group definitions.

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

Conclusions:

The data source isn’t the sole factor involved in the speed degredation. PNG encoder doesn’t seem to be scaling up well. I’m wondering if rather than it being the data (it may be), it could also be the styles?

You can see all 8 results here: http://maps.warwickshire.gov.uk/misc/

As OS Strategi is a free dataset, I can provide the shapefiles and styles and layergroup definitions; if you copy them into one of your instances you can probably be up and testing inside of a few minutes.

That would be interesting, yes. Where can I fetch them?

[Andrea_Aime4]

On Tue, Dec 24, 2013 at 2:43 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi Andrea,
I've pre-packaged everything for you including the data. Layergroups,
SLD's (mine are completely different - no idea what theirs look like), and
workspaces.
I think you'll just need to change where the stores are looking to
wherever you put the shapefiles, but you're going to know better than me.

z_test is the layergroup I'm testing against (it contains the EU basemap
and strategi layergroups).

http://maps.warwickshire.gov.uk/misc/strategi.zip - also includes the JMX.

Thanks. I made a very quick test with what I had handy, a GeoServer 2.4.x,
Oracle JDK 7 (which has known scalability issues) and not even the Image/IO
native PNG encoder
enabled (so, I'm using the slowest PNG encoder in the lot) with a core i7
820 (three years old CPU) I get a throughput that is 2.4 times faster than
yours
(without even trying... but I'm under Linux 64 bits, that might be a
factor):

!image.png|901x171

CPU consumption was around 60%.

Btw, This JMeter setup is a bit different that what I'm used to, the
various zoom levels are not run sequentially, in isolation, but all
together at the same time,
using 10 threads, so the real throughput is the TOTAL one, 3.4r/s, the
throughput value associated to the various zoom levels is apparently
meaningless?
The size of the output image is also a factor, it's "big" compared to the
sizes that were used for the FOSS4G benchmarks, at 1272x1261 it is
roughly 4 times bigger than the average one used in last public
benchmarking effort.

When I have time I'll run some tests with OpenJDK and PNGJ and report back,
and also have a look at profiles, to see if there is anything obvious
to optimize.

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Andrea_Aime4]

On Tue, Dec 24, 2013 at 3:55 PM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

Thanks. I made a very quick test with what I had handy, a GeoServer 2.4.x,
Oracle JDK 7 (which has known scalability issues) and not even the Image/IO
native PNG encoder
enabled (so, I'm using the slowest PNG encoder in the lot) with a core i7
820 (three years old CPU) I get a throughput that is 2.4 times faster than
yours
(without even trying... but I'm under Linux 64 bits, that might be a
factor):

!image.png|901x171

CPU consumption was around 60%.

And these are the results with the PNGJ based encoder, with Oracle JDK 7
(cpu consumption was around 50% in this test, so, it went down):

!image.png|889x167

I've also tried OpenJDK with PNGJ, in this bench it's somewhat
disappointing, getting only up to 3.3 requests/second, with CPU at around
95%...
I guess the extra rendering effort in these maps makes the slowness of the
OpenJDK renderer bottleneck the execution, in spite of its better
scalability features

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,
Thanks for your investigations. I’m on Windows 64bit, so no native Image I/O here. I guess that means we should be equal (Geoserver 2.4.3 here) but clearly yours is much faster. In theory ours is somewhat optimised - we had Simone consult on it and tell me the stuff I’d missed.
I agree the throughput is somewhat meaningless which is why I’m using ms averages. Also because that’s how long the user will have to wait for their response which is more important to me than throughput.

I chose that image size because that’s the size of a single WMS request that our web-application makes for a 1280*1024 monitor (what many of our users have). The idea of the zoom threshold was to allow the different SLD’s to take effect (though it’s only one set with the Strategi stuff, albeit with a few scale-thresholded layers).

(attachments)

···

Best,
Jonathan

On 24 December 2013 14:55, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Tue, Dec 24, 2013 at 2:43 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Andrea,
I’ve pre-packaged everything for you including the data. Layergroups, SLD’s (mine are completely different - no idea what theirs look like), and workspaces.
I think you’ll just need to change where the stores are looking to wherever you put the shapefiles, but you’re going to know better than me.

z_test is the layergroup I’m testing against (it contains the EU basemap and strategi layergroups).

http://maps.warwickshire.gov.uk/misc/strategi.zip - also includes the JMX.

Thanks. I made a very quick test with what I had handy, a GeoServer 2.4.x, Oracle JDK 7 (which has known scalability issues) and not even the Image/IO native PNG encoder
enabled (so, I’m using the slowest PNG encoder in the lot) with a core i7 820 (three years old CPU) I get a throughput that is 2.4 times faster than yours
(without even trying… but I’m under Linux 64 bits, that might be a factor):

CPU consumption was around 60%.

Btw, This JMeter setup is a bit different that what I’m used to, the various zoom levels are not run sequentially, in isolation, but all together at the same time,
using 10 threads, so the real throughput is the TOTAL one, 3.4r/s, the throughput value associated to the various zoom levels is apparently meaningless?
The size of the output image is also a factor, it’s “big” compared to the sizes that were used for the FOSS4G benchmarks, at 1272x1261 it is
roughly 4 times bigger than the average one used in last public benchmarking effort.

When I have time I’ll run some tests with OpenJDK and PNGJ and report back, and also have a look at profiles, to see if there is anything obvious
to optimize.

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

[Jonathan_Moules2]

Hmm, it doesn’t seem to be entirely specific to this layer.

I tried with two other layers (again, this is the live system I’m testing against now, so load balanced):

1. A simple polygon layer in Oracle, styled with a simple hatch fill. After testing various thread counts, my peak throughput rate there is about 13r/s (but the cost is that average response time doubles).

2. The EU_BASE layergroup that’s in the shapefile I provided. 1 thread response time is only 129ms there (finally something low! ). I can get 21 threads on there and the throughput is a respectable 57.5r/s; the response time doubles to 287ms (not really noticeable to the user in this case though).

So Oracle is doing something to cripple it (surprise!), but that’s on top of the fact that any increase to throughput comes at the cost of average response time.

I’m happy to conduct any other tests if you have them.
Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 24 December 2013 15:23, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Andrea,
Thanks for your investigations. I’m on Windows 64bit, so no native Image I/O here. I guess that means we should be equal (Geoserver 2.4.3 here) but clearly yours is much faster. In theory ours is somewhat optimised - we had Simone consult on it and tell me the stuff I’d missed.
I agree the throughput is somewhat meaningless which is why I’m using ms averages. Also because that’s how long the user will have to wait for their response which is more important to me than throughput.

I chose that image size because that’s the size of a single WMS request that our web-application makes for a 1280*1024 monitor (what many of our users have). The idea of the zoom threshold was to allow the different SLD’s to take effect (though it’s only one set with the Strategi stuff, albeit with a few scale-thresholded layers).

=====

I’ve now been running these against our live systems too (normally that’d be rather irresponsible, but it’s Xmas eve and no-one is in;’ probably the only day I can do this in the year!). Our live server, which has a few less cores (12), and three load-balanced instances is definitely much better able to handle 10 threads - I get a total average of 3316ms (2.9r/s) in that scenario (Oracle layers, PNG encoder). That uses 100% of the CPU (~30% per instance). The optimum seems to be about 8 threads; any more than that and the response time plummets.

However it’s just as slow for 1 thread as the test system was.

Also, I’m new to Jmeter so don’t know what the best plans are yet. This is a hybrid of Christians and what the internet presented and what seemed to work.

---

In relation to your using PNGJ test - Your total average response times are about 1/3rd mine (2.5s compared to my 7.5s for the same thing), while using a heck of a lot less CPU power too. So either I have something really badly configured on my install, or Windows is even more crippled than I thought.

Best,
Jonathan

[Andrea_Aime4]

On Tue, Dec 24, 2013 at 4:23 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

In relation to your using PNGJ test - Your total average response times
are about 1/3rd mine (2.5s compared to my 7.5s for the same thing), while
using a heck of a lot less CPU power too. So either I have something really
badly configured on my install, or Windows is even more crippled than I
thought.

Hmm... given that I set absolutely no performance flags on the JVM, I'd go
for the latter :-p
Well, one thing might be interesting: what's your PNG compression level, to
be found in in the WMS panel?

In the meantime I've load balanced GeoServer on my machine as well (using
HAProxy), here are some results:

2 instances, Oracle JDK 7 + PNGJ, giving 1GB or heap each:

!image.png|874x151

3 instances (still 1GB each)

!image.png|871x153

4 instances, this time each with 512MB heap, since my system was starting
to swap enough to affect the benchmarks otherwise (only have 8GB of memory):

!image.png|862x157

So it seems max throughput is reached at 3 instances on my machine, the
boost from 1 to 2 is quite significant (3.9 -> 6.5, 66% speedup), the one
from 2 to 3 less so (6.5 -> 7.9, 20% speedup)

About Oracle... not sure what to say. Do you have handy a backup of the
database that I could import locally? Running a profiler could point
at some obvious issue.

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,

Well, one thing might be interesting: what’s your PNG compression level, to be found in in the WMS panel?
It’s 50. I guess I changed it given the default is 25 apparently.

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

About Oracle… not sure what to say. Do you have handy a backup of the database that I could import locally?
Well the database is about 50GB, and any time we’ve tried to supply it in the past it’s been impossible to import (apparently Oracle tools are incompatible with themselves in relation to importing versus exporting; can’t say I’m surprised).

Interesting that 3 instances seems to be optimal; I got lucky in choosing to deploy that many then.

As noted previously, we’re hoping to do some Oracle upgrading very soon, so hopefully we’ll be able to get significant increases in speed that way.

But the renderers still slow down considerably when threaded, even for shapefiles, so the speed gains will probably be quite minor in those scenarios.
This isn’t a critical issue for us currently as we’re pre-rendering the basemap for TMS, but some of the others are still a bit slow and could benefit from a speedup.
Cheers,
Jonathan

[Andrea_Aime4]

On Mon, Dec 30, 2013 at 1:25 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi Andrea,
> Well, one thing might be interesting: what's your PNG compression level,
to be found in in the WMS panel?
It's 50. I guess I changed it given the default is 25 apparently.

Wow yes, that seems a high value. It should not make a difference for the
JDK own PNG encoder (that you
get on Windows 64 unless you install PNGJ), but it should make the PNGJ
work more (and thus spend more time).
I'd have a look at output size vs extra time performance.

> About Oracle... not sure what to say. Do you have handy a backup of the
database that I could import locally?
Well the database is about 50GB, and any time we've tried to supply it in
the past it's been impossible to import (apparently Oracle tools are
incompatible with themselves in relation to importing versus exporting;
can't say I'm surprised).

Interesting that 3 instances seems to be optimal; I got lucky in choosing
to deploy that many then.

Well well... the thing is, a core i7 820 is a 4 cores CPU + hyperthreading,
so I'm not far from my phisical capacity
anyways. On your system (12 cores, yes?) it might be that the threashold is
higher.

As noted previously, we're hoping to do some Oracle upgrading very soon,
so hopefully we'll be able to get significant increases in speed that way.

Let us know how that goes

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,

Well, one thing might be interesting: what’s your PNG compression level, to be found in in the WMS panel?

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

(attachments)

···

So, I’ve done the experiments (3 threads on a single instance, 6 loops, PNGJ encoder) and my results are as below:

It’s 50. I guess I changed it given the default is 25 apparently.

Wow yes, that seems a high value. It should not make a difference for the JDK own PNG encoder (that you
get on Windows 64 unless you install PNGJ), but it should make the PNGJ work more (and thus spend more time).
I’d have a look at output size vs extra time performance.

So it seems that the 50% compression on PNG WMS requests is on average the fastest by a slight margin (25% coming second). But considering that the 50% is significantly smaller it’s the clear winner.

Funnily enough, while the 0% compression was the winner overall (if you don’t mind 5MB images ) - it failed badly at the 1:10million scale. In fact, 100% compression won - I guess all that white space.

---

I also conducted the same test against the entire stack (down to 1:250); in that case the 50% wins on the overall average again (2098ms versus 2139ms for 25%). And that’s with a bonus file-size shrinkage of ~30kB on average.

It’s interesting to see that some layers are much better at 25% compression and others at 50%. The difference can be 200ms either way depending on the layer.

Maybe it’s worth considering changing the default in GeoServer? Assuming others can corroborate the findings.

Cheers,
Jonathan

[Andrea_Aime4]

On Thu, Jan 2, 2014 at 3:26 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi Andrea,

> Well, one thing might be interesting: what's your PNG compression level,
to be found in in the WMS panel?

It's 50. I guess I changed it given the default is 25 apparently.

Wow yes, that seems a high value. It should not make a difference for the
JDK own PNG encoder (that you
get on Windows 64 unless you install PNGJ), but it should make the PNGJ
work more (and thus spend more time).
I'd have a look at output size vs extra time performance.

So, I've done the experiments (3 threads on a single instance, 6 loops,
PNGJ encoder) and my results are as below:

!image.png|818x169

So it seems that the 50% compression on PNG WMS requests is on average the
fastest by a slight margin (25% coming second). But considering that the
50% is significantly smaller it's the clear winner.

Funnily enough, while the 0% compression was the winner overall (if you
don't mind 5MB images ) - it failed badly at the 1:10million scale. In
fact, 100% compression won - I guess all that white space.

-------
I also conducted the same test against the entire stack (down to 1:250);
in that case the 50% wins on the overall average again (2098ms versus
2139ms for 25%). And that's with a bonus file-size shrinkage of ~30kB on
average.

It's interesting to see that some layers are much better at 25%
compression and others at 50%. The difference can be 200ms either way
depending on the layer.

Maybe it's worth considering changing the default in GeoServer? Assuming
others can corroborate the findings.

Hmm... wondering how are you doing the tests? That is, do you have a public
internet connection in the middle?
That would explain why the smaller PNG results in so much better times,
even if the higher compression requires
a significant extra effort on the CPUs.

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

Hi Andrea,
I thought of that too, but these are going over the LAN.
Ping time from my machine (where jmeter was running) to the server is “<1ms”.

Download speed of a ~670MB zip file from there to my machine averaged 100MB/s (or 800Mbits/s!).

All requests were direct to GeoServer; no load balancing (thus IIS) in the middle.

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

(attachments)

···

Also, if it were the connection, I’d have expected the 0% compression with the resulting 5MB images to be by far the slowest instead of the fastest on average.

Cheers,
Jonathan

On 2 January 2014 14:38, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Thu, Jan 2, 2014 at 3:26 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Andrea,

Well, one thing might be interesting: what’s your PNG compression level, to be found in in the WMS panel?

Hmm… wondering how are you doing the tests? That is, do you have a public internet connection in the middle?
That would explain why the smaller PNG results in so much better times, even if the higher compression requires
a significant extra effort on the CPUs.

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

So, I’ve done the experiments (3 threads on a single instance, 6 loops, PNGJ encoder) and my results are as below:

It’s 50. I guess I changed it given the default is 25 apparently.

Wow yes, that seems a high value. It should not make a difference for the JDK own PNG encoder (that you
get on Windows 64 unless you install PNGJ), but it should make the PNGJ work more (and thus spend more time).
I’d have a look at output size vs extra time performance.

So it seems that the 50% compression on PNG WMS requests is on average the fastest by a slight margin (25% coming second). But considering that the 50% is significantly smaller it’s the clear winner.

Funnily enough, while the 0% compression was the winner overall (if you don’t mind 5MB images ) - it failed badly at the 1:10million scale. In fact, 100% compression won - I guess all that white space.

---

I also conducted the same test against the entire stack (down to 1:250); in that case the 50% wins on the overall average again (2098ms versus 2139ms for 25%). And that’s with a bonus file-size shrinkage of ~30kB on average.

It’s interesting to see that some layers are much better at 25% compression and others at 50%. The difference can be 200ms either way depending on the layer.

Maybe it’s worth considering changing the default in GeoServer? Assuming others can corroborate the findings.

[Andrea_Aime4]

On Thu, Jan 2, 2014 at 3:47 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi Andrea,
I thought of that too, but these are going over the LAN.
Ping time from my machine (where jmeter was running) to the server is
"<1ms".

Download speed of a ~670MB zip file from there to my machine averaged
100MB/s (or 800Mbits/s!).

I still had the test harness setup for my last test, so running another
with 50% compression was quick.
Here are the results:

PNG comparisonST_Simplify + 10000 fetch sizeST_Simplify + 10000 fetch size
+ PNG 50%Avg TimeThroughputAvg TimeThroughput10million72511,5982110,175
million76810,7983710,042,5 million67712,2575110,9912500009828,8512415,99
6250001.5745,8619924,583000001.0458,4311277,9015000063212,8580910,37TOTAL915
8,8010837,51

So, on my machine, I have a rather visible slowdown by upping the PNG
compression.
However... I don't have a network that's really usable for tests (only
100MB/s with an ADSL router
in the middle) so all my tests have JMeter and GeoServer deployed on the
same machine

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan_Moules2]

How odd. I’ll try and remember to re-run the tests when you’ve released the optimised .jar file; maybe that’s somehow affecting the results.
Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 2 January 2014 18:33, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Thu, Jan 2, 2014 at 3:47 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Andrea,

I thought of that too, but these are going over the LAN.
Ping time from my machine (where jmeter was running) to the server is “<1ms”.

Download speed of a ~670MB zip file from there to my machine averaged 100MB/s (or 800Mbits/s!).

I still had the test harness setup for my last test, so running another with 50% compression was quick.
Here are the results:

PNG comparison	ST_Simplify + 10000 fetch size		ST_Simplify + 10000 fetch size + PNG 50%
	Avg Time	Throughput	Avg Time	Throughput
10million	725	11,59	821	10,17
5 million	768	10,79	837	10,04
2,5 million	677	12,25	751	10,99
1250000	982	8,85	1241	5,99
625000	1.574	5,86	1992	4,58
300000	1.045	8,43	1127	7,90
150000	632	12,85	809	10,37
TOTAL	915	8,80	1083	7,51

So, on my machine, I have a rather visible slowdown by upping the PNG compression.
However… I don’t have a network that’s really usable for tests (only 100MB/s with an ADSL router
in the middle) so all my tests have JMeter and GeoServer deployed on the same machine

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---