[Geoserver-users] ImageMosaic and Overviews

GeoServer GeoServer User
1

We have a set of TIF images which we use to create an image mosaic via the
ImageMosaic plugin. Each of the images have internal overviews generated by
gdaladdo. Up until geoserver 2.4.4 everything worked perfectly. However we
attempted to upgrade to geoserver 2.4.5 and now 2.5 and we have found that
the mosaic no longer recognizes the overviews. When attempting to view the
data via wms at a high level the server hangs on the first request... as if
it is attempting to render the full data set a the base level. If we make a
wms request at the base level (below the overviews) the response is
immediate and all looks good. Once we zoom back to the a level that would
use the overview the server hangs again.

Again.. these exact same datasets work perfectly in geoserver 2.4.4 and
lower.

Any help would be greatly appreciated.. as we would like to move to 2.5 to
take advantage of the new pngj encoder.

Thanks!

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

2

Ciao,
here is what I would suggest doing.

Raise the log level to verbose and make a few request to reach the
hang condition.
Then send over the logs + the properties file for the mosaic, the one
at the top dir where the mosaic has been created.

This should be enough for us to understand what is going on and see if
this is something we need to fix.

Regards,
Simone Giannecchini

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Fri, Mar 21, 2014 at 7:26 PM, bhecox <bhecox@anonymised.com> wrote:

We have a set of TIF images which we use to create an image mosaic via the
ImageMosaic plugin. Each of the images have internal overviews generated by
gdaladdo. Up until geoserver 2.4.4 everything worked perfectly. However we
attempted to upgrade to geoserver 2.4.5 and now 2.5 and we have found that
the mosaic no longer recognizes the overviews. When attempting to view the
data via wms at a high level the server hangs on the first request... as if
it is attempting to render the full data set a the base level. If we make a
wms request at the base level (below the overviews) the response is
immediate and all looks good. Once we zoom back to the a level that would
use the overview the server hangs again.

Again.. these exact same datasets work perfectly in geoserver 2.4.4 and
lower.

Any help would be greatly appreciated.. as we would like to move to 2.5 to
take advantage of the new pngj encoder.

Thanks!

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

Thanks for the follow up. Below is contents of the properties file for the
mosaic. The contents of the log after a wms request on the mosaic layer is
here geoserver_log.txt
<http://osgeo-org.1560.x6.nabble.com/file/n5130573/geoserver_log.txt&gt;

*Contents of the Properties file:*

#Sun Mar 23 21:43:24 EDT 2014
Levels=50.977131725298676,50.97548805290303
101.95426345059735,101.95097610580606 203.88185126973195,203.90195221161213
407.7637025394639,407.6484913498922 815.1008740093259,815.2969826997844
1630.2017480186519,1630.5939653995688 3260.4034960373037,3261.1879307991376
6493.636962940963,6522.375861598275
Heterogeneous=true
AbsolutePath=false
Name=test
TypeName=test
Caching=false
ExpandToRGB=false
LocationAttribute=location
CheckAuxiliaryMetadata=false
LevelsNum=8

If we zoom in to level 1 the wms request is fast as expected.. but as we
zoom out, each level gets slower and slower.. until we reach level 5 which
takes several minutes and level 6 and up which simply never return. The
source data of the mosaic totals about 15g. Thanks again for any help.

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414p5130573.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

4

Ciao,
I checked the logs together with the mosaic config and I believe what
you see is normal due to how the data is structure.

Before you think I am fool let me explain.
Your mosaic is heterogeneous (line Heterogeneous=true) which means the
granules are not all at the same resolution.
This is ok.

On the other side GeoServer _is_ using the overviews, see messages
like this in the log:

2014-03-23 21:46:38,175 DEBUG [gce.imagemosaic] - Loading level 7 with
source region: java.awt.Rectangle[x=0,y=0,width=114,height=83]
subsampling: 2,2 for
granule:file:/maps/test/../20140403/sec/conus/Wichita.tif

this means that GeoTools underneath is picking up the last available
overviews (level 7 is the 7th overviews, you have 7 + 1 levels total
as indicated in the props file)

But, check this part:

2014-03-23 21:46:38,177 DEBUG [data.shapefile] - Read lock:
file:/maps/test/test.shp by
org.geotools.data.shapefile.shp.ShapefileReader
2014-03-23 21:46:38,177 TRACE [data.shapefile] - Shapefile Reader has
been closed. Number open: 0
2014-03-23 21:46:38,177 DEBUG [data.shapefile] - Read lock:
file:/maps/test/test.shx by org.geotools.data.shapefile.shp.IndexFile
2014-03-23 21:46:38,177 TRACE [data.shapefile] - IndexFile has been
closed. Number open: 0
2014-03-23 21:46:38,177 DEBUG [data.shapefile] - Read lock:
file:/maps/test/test.dbf by
org.geotools.data.shapefile.dbf.DbaseFileReader
2014-03-23 21:46:38,177 TRACE [data.shapefile] - Dbase File Reader has
been closed. Number open: 0
2014-03-23 21:46:38,177 DEBUG [gce.imagemosaic] - Producing the final
mosaic, step 1, loop through granule collectors
2014-03-23 21:46:38,177 DEBUG [gce.imagemosaic] - Using collector with
filter:Filter.INCLUDE
2014-03-23 21:46:38,179 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Albuquerque.tif
2014-03-23 21:46:38,190 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Atlanta.tif
2014-03-23 21:46:38,190 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Billings.tif
2014-03-23 21:46:38,190 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Brownsville.tif
2014-03-23 21:46:38,191 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Charlotte.tif
2014-03-23 21:46:38,191 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Cheyenne.tif
2014-03-23 21:46:38,191 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Chicago.tif
2014-03-23 21:46:38,191 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Cincinnati.tif
2014-03-23 21:46:38,191 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Dallas-FtWorth.tif
2014-03-23 21:46:38,191 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Denver.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Detroit.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/ElPaso.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/GreatFalls.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/GreenBay.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Halifax.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Houston.tif
2014-03-23 21:46:38,192 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Jacksonville.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/KansasCity.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/KlamathFalls.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/LakeHuron.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/LasVegas.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/LosAngeles.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Memphis.tif
2014-03-23 21:46:38,193 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Miami.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Montreal.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/NewOrleans.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/NewYork.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Omaha.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Phoenix.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/SaltLakeCity.tif
2014-03-23 21:46:38,194 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/SanAntonio.tif
2014-03-23 21:46:38,195 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/SanFrancisco.tif
2014-03-23 21:46:38,195 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Seattle.tif
2014-03-23 21:46:38,195 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/StLouis.tif
2014-03-23 21:46:38,195 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/TwinCities.tif
2014-03-23 21:46:38,195 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Washington.tif
2014-03-23 21:46:38,195 DEBUG [gce.imagemosaic] - Adding to mosaic
granule file:/maps/test/../20140403/sec/conus/Wichita.tif
2014-03-23 21:46:38,219 DEBUG [gce.imagemosaic] - Producing the final
mosaic, step 2, final mosaicking

you are loading too many files at a time since your mosaic is made by
many files (correct?) and covers a large area (correct?) and you are
showing at all resolutions (correct?). Since you say that its slower
and slower as you zoom out this exactly the effect of having too many
files to load and process for the ImageMosaic.

To validate my thinking, isolat 3 granules, play with them at all
resolutions and tell me if you see a difference when yuo zoom out.
If not the you need to rethink how you show the data (I'll help), if
yes, then we have to investigate further.

Generally speaking having a mosaic that at certain scales uses more
than 15/20 files is bad for performance reasons since file descriptors
are a valuable resources and servers tend to slow down quickly as tey
exhaust them.

Let me know if this sounds reasonable.

Regards,
Simone Giannecchini

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Mon, Mar 24, 2014 at 1:53 AM, bhecox <bhecox@anonymised.com> wrote:

Thanks for the follow up. Below is contents of the properties file for the
mosaic. The contents of the log after a wms request on the mosaic layer is
here geoserver_log.txt
<http://osgeo-org.1560.x6.nabble.com/file/n5130573/geoserver_log.txt&gt;

*Contents of the Properties file:*

#Sun Mar 23 21:43:24 EDT 2014
Levels=50.977131725298676,50.97548805290303
101.95426345059735,101.95097610580606 203.88185126973195,203.90195221161213
407.7637025394639,407.6484913498922 815.1008740093259,815.2969826997844
1630.2017480186519,1630.5939653995688 3260.4034960373037,3261.1879307991376
6493.636962940963,6522.375861598275
Heterogeneous=true
AbsolutePath=false
Name=test
TypeName=test
Caching=false
ExpandToRGB=false
LocationAttribute=location
CheckAuxiliaryMetadata=false
LevelsNum=8

If we zoom in to level 1 the wms request is fast as expected.. but as we
zoom out, each level gets slower and slower.. until we reach level 5 which
takes several minutes and level 6 and up which simply never return. The
source data of the mosaic totals about 15g. Thanks again for any help.

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414p5130573.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

5

Thanks Simone. I agree, I also noticed that in the logs it appears to be
grabbing the level 7 overlay. What has confused us is that this exact same
data set and mosaic worked fine and was fast all all zoom levels when using
Geoserver 2.4.4 and lower.. it was not until version 2.4.5 and 2.5.0 that
the speed became an issue at further out zoom levels.

However, after working through this and trying different approaches.. we
agree that using a mosaic for this many files is not as efficient as other
methods. We have found we get the most performance if we add each tif as
its own geotiff store and layer and use a group layer to group the full set
together.

Thanks again for your help!

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414p5130823.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

6

Ciao,
please read below...

Regards,
Simone Giannecchini

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Tue, Mar 25, 2014 at 12:38 AM, bhecox <bhecox@anonymised.com> wrote:

Thanks Simone. I agree, I also noticed that in the logs it appears to be
grabbing the level 7 overlay. What has confused us is that this exact same
data set and mosaic worked fine and was fast all all zoom levels when using
Geoserver 2.4.4 and lower.. it was not until version 2.4.5 and 2.5.0 that
the speed became an issue at further out zoom levels.

However, after working through this and trying different approaches.. we
agree that using a mosaic for this many files is not as efficient as other
methods. We have found we get the most performance if we add each tif as
its own geotiff store and layer and use a group layer to group the full set
together.

can you elaborate a bit more on this approach as I am particularly interested.

That said, can you also tell me how the ImageMosaic layer has the
USE_JAI_IMAGEREAD parameter configured?

I know I am abusing of your time but I am trying to figure out why the
layergroup seems to perform better.

Thanks again for your help!

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414p5130823.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

7

Our full map spans the entire US.. it is a set of tifs of vfr aviation
charts. Most of the wms requests made against the service are at about the
1:1M or 1:2M level.. and often span 2 or more tifs. In the past we have
always used a separate layer for each tif and a group layer to group the
entire set until geoserver 2.4.. where we cut over to a mosaic layer...
mainly for ease of maintenance as the tifs are updated regularly. We
notices a slight degrade in performance using the mosaic, however it was
acceptable difference (i.e. group layer request would return a request in
600-800ms where the mosaic would return in 1 - 1.2 sec). We did't run into
this current problem until we attempted to upgrade to 2.4.5.

On the mosaic layer we have USE_JAI_IMAGEREAD set to true. However, we have
tested it with both true and false with the same result.

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/ImageMosaic-and-Overviews-tp5130414p5130825.html
Sent from the GeoServer - User mailing list archive at Nabble.com.