Hello,
We have a set of ECW files (LUFO2002), about 3015 files in total.
All these files have to be served as one layer.
In Geoserver 2.4.0 we have created a datastore ‘ImageMosiac’ but the performance is not that good.
What is the best approach on serving so many ECW files? Do we need to make GeoTIFF / ImagePyramid of this set?
With regards,
Lucas
Hello,
Hi Lucas
We have a set of ECW files (LUFO2002), about 3015 files in total.
All these files have to be served as one layer.
In Geoserver 2.4.0 we have created a datastore 'ImageMosiac' but the
performance is not that good.What is the best approach on serving so many ECW files? Do we need to make
GeoTIFF / ImagePyramid of this set?
I am not certain if GeoServer might not have some way to improve your
setup programmatically (ie. via setup), but as an old data man myself,
I would recommend doing something to the root of your problem:
- Having ECWs that need to be decompressed for each request
- Having a large (tiled) set of data that need to be parsed/handled
for each request.
I would suggest that you use GDAL to:
1) Make a single VRT of your lib full of ECWs (see
http://www.gdal.org/gdalbuildvrt.html)
2) Add overviews to said VRT, for instance 2 4 8 16 32 64 (depending
on your usual zooms you might even go as far as 128 or 156) (see
http://www.gdal.org/gdaladdo.html)
3) Load said VRT instead of using an imagemosaic
That way the ECWs will only be decompressed when people want to look
at near real resolutions (better than 1/2 resolution).
Best,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk
2014-03-04 9:22 GMT+01:00 Hans Gregers Petersen <gregers@anonymised.com>:
I would suggest that you use GDAL to:
1) Make a single VRT of your lib full of ECWs (see
http://www.gdal.org/gdalbuildvrt.html)
2) Add overviews to said VRT, for instance 2 4 8 16 32 64 (depending
on your usual zooms you might even go as far as 128 or 156) (see
http://www.gdal.org/gdaladdo.html)
3) Load said VRT instead of using an imagemosaicThat way the ECWs will only be decompressed when people want to look
at near real resolutions (better than 1/2 resolution).
Hi Hans, ECW is a proprietary format with internal overviews and tiling.
In my understanding it does not require the client to decompress the whole
file for each request.
Cheers
Stefano
---------------------------------------------------
41.95581N 12.52854E
On Tue, Mar 4, 2014 at 9:22 AM, Hans Gregers Petersen <gregers@anonymised.com>wrote:
I would suggest that you use GDAL to:
1) Make a single VRT of your lib full of ECWs (see
http://www.gdal.org/gdalbuildvrt.html)
2) Add overviews to said VRT, for instance 2 4 8 16 32 64 (depending
on your usual zooms you might even go as far as 128 or 156) (see
http://www.gdal.org/gdaladdo.html)
3) Load said VRT instead of using an imagemosaic
GeoServer cannot load VRT files, not that I know of, at least.
For some reason, at the imageio-ext level, we need to write custom classes
for each new format we want to support (not sure why, I would very much like
to see this limitation removed), and VRT is not one of the formats that
we have in our support list:
https://github.com/geosolutions-it/imageio-ext/tree/master/plugin/gdal
However, it is possible to create reduced resolution versions of the
dataset,
manually setup a pyramid config file, and have GeoServer use that, or
simply register each layer independently, assign a scale range to each via
SLD,
and create a layer group that would more or less act the same way as the
pyramid.
Cheers
Andrea
--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
Hi Hans, ECW is a proprietary format with internal overviews and tiling.
In my understanding it does not require the client to decompress the whole
file for each request.
Indeed it does not require a decompression of the whole file - in my
experience though the partial decompression is still slower
(especially on big ECWs) than reading an overview using GDAL. It might
be something to do with the way the OS accesses the ECW file.
Cheers,
Greg
GeoServer cannot load VRT files, not that I know of, at least.
For some reason, at the imageio-ext level, we need to write custom classes
for each new format we want to support (not sure why, I would very much like
to see this limitation removed), and VRT is not one of the formats that
we have in our support list:
https://github.com/geosolutions-it/imageio-ext/tree/master/plugin/gdal
I would have sworn that I have done it - but it cannot be so, if the
VRT driver doesn't exist. 
Maybe I emulated the thoughts behind VRTs using layer groups as you describe.
IMHO VRTs are great - and is part of my usual setup when using
MapServer, and were also very much used when I was part of a big
geodata production company too.
Best,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk
I didn’t run any specific benchmark on them but my personal experience is on the contrary a very fast behavior. Indeed I have been using them with proprietary software, may be the difference is in the ECW sw release used to access them?
I didn’t fully understand how a VRT could help in reducing access time, letting apart the big problem of format unsupported highlighted by Andrea, maybe I am wrong but I thought that when using VRT you always have to search for inner overviews browsing the ECW file. Or maybe you meant creating external overview in geotiff format?
Thanks for sharing your thoughts on this.
Cheers,
Stefano
2014-03-04 10:04 GMT+01:00 Hans Gregers Petersen <gregers@anonymised.com>:
Indeed it does not require a decompression of the whole file - in my
experience though the partial decompression is still slower
(especially on big ECWs) than reading an overview using GDAL. It might
be something to do with the way the OS accesses the ECW file.
41.95581N 12.52854E
Hi Stefano,
I didn't run any specific benchmark on them but my personal experience is on
the contrary a very fast behavior. Indeed I have been using them with
proprietary software, may be the difference is in the ECW sw release used to
access them?
I agree completely that the performance of a single ECW file, is good
(I wrote that last mail in a clumsy manner - see what I meant by "big"
below).
You can "decompress" the single wavelet very quickly as the format is
designed with this specifically in mind.
I didn't fully understand how a VRT could help in reducing access time,
letting apart the big problem of format unsupported highlighted by Andrea,
maybe I am wrong but I thought that when using VRT you always have to search
for inner overviews browsing the ECW file. Or maybe you meant creating
external overview in geotiff format?
Having a setup where one tiles into a number of ECWs is another thing
(here in DK I see a lot of 1km tiles with 10cm/pixel resolution, which
are each still reasonably big files). In outer zooms you have to
"decompress" many wavelets, and the wavelet idea is kind of spoiled
when you break it up into tiles. Instead using a VRT with (external)
overviews will reduce your reading to a single TIFFJPEG overview,
which performs very nicely. In a couple of setups, I have seen a speed
improvement using a pyramid of simple TIFFJPEG overviews over a single
very big ECW (sizes of around 10GB), it might be an OS-issue there
though (both were on Win as far as I remember).
Cheers,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk
Ok, so an VRT file is not a solution in combination with geoserver.
I'm not so familiar with this many EVW files and GDAL.
What is the best solution to look into in combination with geoserver?
Met vriendelijke groet,
Lucas Heezen
Covadis b.v.
t: 026 3616600
Geograaf 12
f: 026 3612317
6921 EW Duiven
e: heezen@anonymised.com
-----Oorspronkelijk bericht-----
Van: Hans Gregers Petersen [mailto:gregers@anonymised.com]
Verzonden: dinsdag 4 maart 2014 10:39
Aan: Stefano Iacovella
CC: Lucas Heezen - Covadis; geoserver-users@lists.sourceforge.net
Onderwerp: Re: [Geoserver-users] DataStore ImageMosaic with many ECW files
Hi Stefano,
I didn't run any specific benchmark on them but my personal experience
is on the contrary a very fast behavior. Indeed I have been using them
with proprietary software, may be the difference is in the ECW sw
release used to access them?
I agree completely that the performance of a single ECW file, is good (I wrote that last mail in a clumsy manner - see what I meant by "big"
below).
You can "decompress" the single wavelet very quickly as the format is designed with this specifically in mind.
I didn't fully understand how a VRT could help in reducing access
time, letting apart the big problem of format unsupported highlighted
by Andrea, maybe I am wrong but I thought that when using VRT you
always have to search for inner overviews browsing the ECW file. Or
maybe you meant creating external overview in geotiff format?
Having a setup where one tiles into a number of ECWs is another thing (here in DK I see a lot of 1km tiles with 10cm/pixel resolution, which are each still reasonably big files). In outer zooms you have to "decompress" many wavelets, and the wavelet idea is kind of spoiled when you break it up into tiles. Instead using a VRT with (external) overviews will reduce your reading to a single TIFFJPEG overview, which performs very nicely. In a couple of setups, I have seen a speed improvement using a pyramid of simple TIFFJPEG overviews over a single very big ECW (sizes of around 10GB), it might be an OS-issue there though (both were on Win as far as I remember).
Cheers,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk
Hi Lucas,
Ok, so an VRT file is not a solution in combination with geoserver.
Apparently not - sorry for leading you down a stray (but good) path
I'm not so familiar with this many EVW files and GDAL.
What is the best solution to look into in combination with geoserver?
I would look into "emulating" (if I may call it that) the VRT-approach
using the style Andrea described previously.
Cheers,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk
Hi Lucas,
Ok, so an VRT file is not a solution in combination with geoserver.
Apparently not - sorry for leading you down a stray (but good) path
I'm not so familiar with this many EVW files and GDAL.
What is the best solution to look into in combination with geoserver?
I would look into "emulating" (if I may call it that) the VRT-approach
using the style Andrea described previously.
Cheers,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk
Hello Hans,
I will do that, but we are also trying to get the dataset as one ECW.
-lucas
-----Oorspronkelijk bericht-----
Van: Hans Gregers Petersen [mailto:gregers@anonymised.com]
Verzonden: vrijdag 7 maart 2014 10:19
Aan: Lucas Heezen - Covadis
CC: Stefano Iacovella; geoserver-users@lists.sourceforge.net
Onderwerp: Re: [Geoserver-users] DataStore ImageMosaic with many ECW files
Hi Lucas,
Ok, so an VRT file is not a solution in combination with geoserver.
Apparently not - sorry for leading you down a stray (but good) path
I'm not so familiar with this many EVW files and GDAL.
What is the best solution to look into in combination with geoserver?
I would look into "emulating" (if I may call it that) the VRT-approach using the style Andrea described previously.
Cheers,
Greg
--
Hans Gregers Petersen
Partner, Senior Consultant
www.septima.dk