[Geoserver-devel] Speeding up gif/png serving

GeoServer GeoServer Developer
1

Hi all,
during the last week I spent some time trying to get the best possible
performance out of GIF WMS serving.

Now, during my tests I noticed that GIF encoding took as much time
as pure rendering, if not more. One way to speed encoding up in a
dramatic way (5/10 times faster) is to render stuff directly on a
256 colors palette, something that you can get by creating a
buffered image with the TYPE_BYTE_INDEXED flag.

The drawback about this approach is that the TYPE_BYTE_INDEXED
provides you with a fixed 256 colors palette. Now, thinking about
this, it's not such a great problem if we allow the user to create its
own palette, or give him some tools to estimate a good one.

It would not be so difficult: take the map you want
to render, with all of the layers enabled, and render it into true colors.
Then perform a reduction to 256 colors -> the palette you get is
a stable palette and it's a good estimate of the one you need.
Of course this would be only a tool for estimation, leaving the user
free to perform the estimation in another way (for example, we could
learn to import paint shop pro/gimp palettes).

I haven't tried, but hope the same would apply to PNG images as well
(with various palettes, ranging from 256 to 65xxx colors).
Me or Simone are going to try this out on the coverage branch next week.

Cheers
Andrea

2

I fully agree with Andrea, we have been playing around the all week
with different writers and the conclusion is that if you want to speed
up things you need to use gif or png with indexed color model, if you
want to dramatically speed up thing you need to used natively in the
streaming renderer image with an index color model instead or drawing
true color rgb and then reducing.
What I am going to do next days is enabling the streaming renderer to
use directly indexed buffered image with a standard prefixed palette
of 256 color enabling that for GIF and paletted PNG and TIFF.

Next step, as Andrea pointed out, would be investigating some sort of
customization of palettes for optimal quality. The options we have
right now on the plate are as he described, let's see if someone has
some others bright ideas for step 2 (that is, after the standard
palette work fine).

Simone.

On 8/6/06, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi all,
during the last week I spent some time trying to get the best possible
performance out of GIF WMS serving.

Now, during my tests I noticed that GIF encoding took as much time
as pure rendering, if not more. One way to speed encoding up in a
dramatic way (5/10 times faster) is to render stuff directly on a
256 colors palette, something that you can get by creating a
buffered image with the TYPE_BYTE_INDEXED flag.

The drawback about this approach is that the TYPE_BYTE_INDEXED
provides you with a fixed 256 colors palette. Now, thinking about
this, it's not such a great problem if we allow the user to create its
own palette, or give him some tools to estimate a good one.

It would not be so difficult: take the map you want
to render, with all of the layers enabled, and render it into true colors.
Then perform a reduction to 256 colors -> the palette you get is
a stable palette and it's a good estimate of the one you need.
Of course this would be only a tool for estimation, leaving the user
free to perform the estimation in another way (for example, we could
learn to import paint shop pro/gimp palettes).

I haven't tried, but hope the same would apply to PNG images as well
(with various palettes, ranging from 256 to 65xxx colors).
Me or Simone are going to try this out on the coverage branch next week.

Cheers
Andrea

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions

http://www.geo-solutions.it

-------------------------------------------------------

3

On Sunday 06 August 2006 16:00, Simone Giannecchini wrote:

I fully agree with Andrea, we have been playing around the all week
with different writers and the conclusion is that if you want to speed
up things you need to use gif or png with indexed color model, if you
want to dramatically speed up thing you need to used natively in the
streaming renderer image with an index color model instead or drawing
true color rgb and then reducing.
What I am going to do next days is enabling the streaming renderer to
use directly indexed buffered image with a standard prefixed palette
of 256 color enabling that for GIF and paletted PNG and TIFF.

Next step, as Andrea pointed out, would be investigating some sort of
customization of palettes for optimal quality. The options we have
right now on the plate are as he described, let's see if someone has
some others bright ideas for step 2 (that is, after the standard
palette work fine).

that's great guys, it was just the idea i had but still hadn't the time to
accomplish.
Wouldn't it be possible, instead of using a fixed pallete, to create it at
runtime by inspecting the colors needed by the SLD object?

2c-

Gabriel.

Simone.

On 8/6/06, Andrea Aime <andrea.aime@anonymised.com> wrote:
> Hi all,
> during the last week I spent some time trying to get the best possible
> performance out of GIF WMS serving.
>
> Now, during my tests I noticed that GIF encoding took as much time
> as pure rendering, if not more. One way to speed encoding up in a
> dramatic way (5/10 times faster) is to render stuff directly on a
> 256 colors palette, something that you can get by creating a
> buffered image with the TYPE_BYTE_INDEXED flag.
>
> The drawback about this approach is that the TYPE_BYTE_INDEXED
> provides you with a fixed 256 colors palette. Now, thinking about
> this, it's not such a great problem if we allow the user to create its
> own palette, or give him some tools to estimate a good one.
>
> It would not be so difficult: take the map you want
> to render, with all of the layers enabled, and render it into true
> colors. Then perform a reduction to 256 colors -> the palette you get is
> a stable palette and it's a good estimate of the one you need.
> Of course this would be only a tool for estimation, leaving the user
> free to perform the estimation in another way (for example, we could
> learn to import paint shop pro/gimp palettes).
>
> I haven't tried, but hope the same would apply to PNG images as well
> (with various palettes, ranging from 256 to 65xxx colors).
> Me or Simone are going to try this out on the coverage branch next week.
>
> Cheers
> Andrea
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your opinions on IT & business topics through brief surveys -- and earn
> cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Gabriel Roldán (groldan@anonymised.com)
Axios Engineering (http://www.axios.es)
Tel. +34 944 41 63 84
Fax. +34 944 41 64 90

4

I thought about this problem before -- the main issue is that the
anti-aliasing causes a lot of "extra" colours to be created.

The original colour reduction algorithm was to find the first 256
colours in the image and use those colours in the colour map. This
completely failed to work after antialiasing was turned on - images
went from 16 colours to 6,000!

Most paint programs will tell you how many colours an image is using,
and you can turn antialiasing on and off.

The second problem is how to deal with transparancy, esp since
antialiasing afects transparency.

dave

5

David Blasby ha scritto:

I thought about this problem before -- the main issue is that the
anti-aliasing causes a lot of "extra" colours to be created.

The original colour reduction algorithm was to find the first 256
colours in the image and use those colours in the colour map. This
completely failed to work after antialiasing was turned on - images
went from 16 colours to 6,000!

Another problem occurs when you add a coverage on the background, SLD
won't tell you which colors are there in a satellite image.

Cheers
Andrea

6

Seems to me that one solution could be to have a per-layer 'cache' of a representative color-map.

I.e., on the first request of a particular layer, a full color-reduction (using an expensive algorithm) is used. Then the calculated color-map is attached to that layer and cached. For future requests, the cached color-map is used.

This idea could be expanded in a number of ways.

1) Rather than simply using the *first* request to generate the sample color-map, you could have a "learning mode" and a "production mode". When a layer is in "learning mode", its color-map is constantly refined, based on each image...resulting in as accurate a static color-map as possible for all viewed maps. In "production mode", the static color-map is used for all requests.

Note that for most vector layers styled with SLD, you don't need to do learning-mode at all, as all colors are available for pre-populating the color-map. (although anti-aliasing would need to be 'learned').

2) It's possible that for many geoserver instances, most layers are repeatedly requested in a particular 'stack' (i.e, most clients are map-oriented, not layer-oriented). You could expand the color-reduction caching to include one map for a particular SET of layers, rather than just one layer. This way commonly used combinations of layers (for particular clients) get the biggest speed boosts, while less-frequently used combos aren't optimized and don't waste space with their cached color-maps.

Thoughts?

--saul

Andrea Aime wrote:

David Blasby ha scritto:

I thought about this problem before -- the main issue is that the
anti-aliasing causes a lot of "extra" colours to be created.

The original colour reduction algorithm was to find the first 256
colours in the image and use those colours in the colour map. This
completely failed to work after antialiasing was turned on - images
went from 16 colours to 6,000!

Another problem occurs when you add a coverage on the background, SLD
won't tell you which colors are there in a satellite image.

Cheers
Andrea

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

7

2) It's possible that for many geoserver instances, most layers are repeatedly requested in a particular 'stack' (i.e, most clients are map-oriented, not layer-oriented). You could expand the color-reduction caching to include one map for a particular SET of layers, rather than just one layer. This way commonly used combinations of layers (for particular clients) get the biggest speed boosts, while less-frequently used combos aren't optimized and don't waste space with their cached color-maps.

This also points to the idea of letting users define a 'set' of layers, so that you can refer to a number of layers with just one layer name. I've thought for awhile this would be useful, especially when we get in to caching stuff - if we have a long string of layer then another user could copy them wrong and not get the full caching effects. If it's just one layer name, that returns 7 configured and styled layers it's much easier.

Also, we should start to capture these ideas in an RnD page...

Chris

Thoughts?

--saul

Andrea Aime wrote:

David Blasby ha scritto:

I thought about this problem before -- the main issue is that the
anti-aliasing causes a lot of "extra" colours to be created.

The original colour reduction algorithm was to find the first 256
colours in the image and use those colours in the colour map. This
completely failed to work after antialiasing was turned on - images
went from 16 colours to 6,000!

Another problem occurs when you add a coverage on the background, SLD
won't tell you which colors are there in a satellite image.

Cheers
Andrea

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:1003,44d77984206341527717022!

--
Chris Holmes
The Open Planning Project
http://topp.openplans.org