[Geoserver-users] Issues with image pyramid

GeoServer GeoServer User
1

Apologies if this is not the right forum for this question.

I’m trying to setup geoserver environment for demo purposes to display satellite imagery such as truemarble imagery (http://www.unearthedoutdoors.net/global_data/true_marble/download).

I’d like to set it up as follows
Scale > A - show one set of tiles
Scale range < A and > B higher res set of tiles Scale range < B and > C even higher res set of tiles And so on.

Initial thinking was to create an image pyramid so I took the high res (250m) TrueMarble tiles and did

gdal_translate -stats -co “TILED=YES” filename new_filename

because I was getting warnings

Warning 1: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in ascending order ERROR 1: MissingRequired:TIFF directory is missing required “ImageLength” field ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: FillEmptyTiles() failed because panByteCounts == NULL E:\MAP_TM\A1_01_34.tif : 67584|0–>2048-2048

And thought the translate would fix these issues. After translating all files, I did

gdal_retile.py -v -levels 4 -ps 2048 2048 -targetDir E:\MAP_TM A1.tif A2.tif … H3.tif H4.tif

And get above errors ending with

E:\MAP_TM\A1_02_82.tif : 165888|2048–>2048-2048 Warning 1: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in ascending order ERROR 1: MissingRequired:TIFF directory is missing required “ImageLength” field ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
More than 1000 errors or warnings have been reported. No more will be reported from now.
E:\MAP_TM\A1_02_83.tif : 167936|2048–>2048-2048 E:\MAP_TM\A1_02_84.tif : 169984|2048–>2048-2048 E:\MAP_TM\A1_02_85.tif : 172032|2048–>768-2048 E:\MAP_TM\A1_03_01.tif : 0|4096–>2048-2048

.
.
.
Finishing up with

E:\MAP_TM\A1_32_10.tif : 18432|63488–>2048-2048 Traceback (most recent call last):
File “c:\Program Files\gdal\gdal_retile.py”, line 1001, in
sys.exit(main(sys.argv))
File “c:\Program Files\gdal\gdal_retile.py”, line 911, in main
dsCreatedTileIndex = tileImage(minfo,ti)
File “c:\Program Files\gdal\gdal_retile.py”, line 349, in tileImage
createTile(minfo, offsetX, offsetY, width, height,tilename,OGRDS)
File “c:\Program Files\gdal\gdal_retile.py”, line 494, in createTile
dec.uly+offsetY*dec.scaleY)
File “c:\Program Files\gdal\gdal_retile.py”, line 256, in getDataSet
data = s_band.ReadRaster( readOffsetX,readOffsetY,readX,readY, readX,readY, self.band_type )
AttributeError: ‘NoneType’ object has no attribute ‘ReadRaster’

Am I doing something wrong?

Can I build an ‘image pyramid’ data set manually? For example, configure geoserver to use 32km TIFF for lowest resolution 1km TIFF tiles for next 500m tiles for next resolution level 250m tiles for highest resolution

Seems like it should be fairly straightforward - what am I missing?

Thanks for any help/pointers/links that can be provided?

E. Scott Stricker

2

You can do this by using a LayerGroup. It’s a little tedious because you must make a variant of your style for each level of detail (changing just the MinScaleDenominator and MaxScaleDenominator) but it can be done.

I can’t offer any insight on the GDAL issues though.

–
David Winslow
OpenGeo - http://opengeo.org/

On Nov 1, 2012 4:17 PM, “Stricker, Scott (IS)” <scott.stricker@anonymised.com508…> wrote:

Apologies if this is not the right forum for this question.

I’m trying to setup geoserver environment for demo purposes to display satellite imagery such as truemarble imagery (http://www.unearthedoutdoors.net/global_data/true_marble/download).

I’d like to set it up as follows
Scale > A - show one set of tiles
Scale range < A and > B higher res set of tiles Scale range < B and > C even higher res set of tiles And so on.

Initial thinking was to create an image pyramid so I took the high res (250m) TrueMarble tiles and did

gdal_translate -stats -co “TILED=YES” filename new_filename

because I was getting warnings

Warning 1: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in ascending order ERROR 1: MissingRequired:TIFF directory is missing required “ImageLength” field ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: FillEmptyTiles() failed because panByteCounts == NULL E:\MAP_TM\A1_01_34.tif : 67584|0–>2048-2048

And thought the translate would fix these issues. After translating all files, I did

gdal_retile.py -v -levels 4 -ps 2048 2048 -targetDir E:\MAP_TM A1.tif A2.tif … H3.tif H4.tif

And get above errors ending with

E:\MAP_TM\A1_02_82.tif : 165888|2048–>2048-2048 Warning 1: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in ascending order ERROR 1: MissingRequired:TIFF directory is missing required “ImageLength” field ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
More than 1000 errors or warnings have been reported. No more will be reported from now.
E:\MAP_TM\A1_02_83.tif : 167936|2048–>2048-2048 E:\MAP_TM\A1_02_84.tif : 169984|2048–>2048-2048 E:\MAP_TM\A1_02_85.tif : 172032|2048–>768-2048 E:\MAP_TM\A1_03_01.tif : 0|4096–>2048-2048

.
.
.
Finishing up with

E:\MAP_TM\A1_32_10.tif : 18432|63488–>2048-2048 Traceback (most recent call last):
File “c:\Program Files\gdal\gdal_retile.py”, line 1001, in
sys.exit(main(sys.argv))
File “c:\Program Files\gdal\gdal_retile.py”, line 911, in main
dsCreatedTileIndex = tileImage(minfo,ti)
File “c:\Program Files\gdal\gdal_retile.py”, line 349, in tileImage
createTile(minfo, offsetX, offsetY, width, height,tilename,OGRDS)
File “c:\Program Files\gdal\gdal_retile.py”, line 494, in createTile
dec.uly+offsetY*dec.scaleY)
File “c:\Program Files\gdal\gdal_retile.py”, line 256, in getDataSet
data = s_band.ReadRaster( readOffsetX,readOffsetY,readX,readY, readX,readY, self.band_type )
AttributeError: ‘NoneType’ object has no attribute ‘ReadRaster’

Am I doing something wrong?

Can I build an ‘image pyramid’ data set manually? For example, configure geoserver to use 32km TIFF for lowest resolution 1km TIFF tiles for next 500m tiles for next resolution level 250m tiles for highest resolution

Seems like it should be fairly straightforward - what am I missing?

Thanks for any help/pointers/links that can be provided?

E. Scott Stricker

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

On Fri, 2 Nov 2012 07:16:59 AM Stricker, Scott wrote:

Apologies if this is not the right forum for this question.

I'm trying to setup geoserver environment for demo purposes to display
satellite imagery such as truemarble imagery
(http://www.unearthedoutdoors.net/global_data/true_marble/download).

I'd like to set it up as follows
Scale > A - show one set of tiles
Scale range < A and > B higher res set of tiles Scale range < B and > C even
higher res set of tiles And so on.

Can I build an 'image pyramid' data set manually? For example, configure
geoserver to use 32km TIFF for lowest resolution 1km TIFF tiles for next
500m tiles for next resolution level 250m tiles for highest resolution

Seems like it should be fairly straightforward - what am I missing?

Thanks for any help/pointers/links that can be provided?

I build my image pyramids manually (well I've automated it using a bash script
- attached for reference).

The requirements as I understand them (mostly from trial and error):

The root directory has a .properties file and a .prj file which both have the
same name as pyramid layer.

Each level of the pyramid is in a separate directory which contains an
ImageMosaic consisting of a number of images (TIFF files), a shape file (.shp,
.shx, .dbf) created by gdaltindex, a .properties file and a .prj file. The
properties files is different from the root one while the .prj file is the
same (don't know if they have to be, but mine all are).

For example I have a layer called bathy_pyramid.

In my coverages/bathy_pyrmid directory I have:

more bathy_pyramid.properties

Name=bathy_pyramid
LevelsNum=10
LevelsDirs=1 2 4 8 16 32 64 128 256 512
Envelope2D=109.8395984985,-64.0000001972 173.9999996985,0.1604010028
Levels=2.44618395E-4,2.44618395E-4 4.89236791E-4,4.89236791E-4
9.78473581E-4,9.78473581E-4 0.00195694716,0.00195694716
0.00391389432,0.00391389432 0.00782778865,0.00782778865
0.0156555773,0.0156555773 0.0313111546,0.0313111546 0.0626223092,0.0626223092
0.125244618,0.125244618

more bathy_pyramid.prj

GEOGCS["WGS 84", DATUM["World Geodetic System 1984", SPHEROID["WGS 84",
6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST],
AXIS["Geodetic latitude", NORTH], AUTHORITY["EPSG","4326"]]

ls 512

bathy_110_174_-64_0_bathy.tif
bathy_pyramid.dbf
bathy_pyramid.prj
bathy_pyramid.properties
bathy_pyramid.shp
bathy_pyramid.shx

more 512/bathy_pyramid.properties

Name=bathy_pyramid
LocationAttribute=location
LevelsNum=1
Envelope2D=109.8395984985,-64.0000001972 173.9999996985,0.1604010028
Levels=0.125245,0.125245

Points to note:
- The Name and Envelope2D fields for all the properties files for a pyramid
must be identical.
- geoserver must have at least read access to all files and write access to
the directories (very easy to overlook when you do things manually)
- The Levels field in the root properties file is used by geoserver to
determine which level to use for the resolution requested, I'm not sure of the
algorithm but you can play with the numbers to get it to do what you want. The
Levels field in the level directories is the pixels size of the TIFFs, I think
this must match the TIFFs and all the TIFFs must have the same resolution. The
Levels field in the level properties and the corresponding value in the root
properties don't have to be the same, but in my case always are.
- The number of (space separated) entries in LevelDirs and Levels in the root
properties file must be exactly LevelsNum
- The value of the LocationAttribute field of the level properties file
matches the field name used in gdaltindex, e.g.:
gdaltindex -tileindex location bathy_pyramid.shp *.tif

Hope this helps
Gordon

update_pyramid (2.52 KB)

4

Thanks to all who replied. Magic that I turned out doing was to build the subdirectories (level directories) as imagemosaics, then create an imagepyramid store pointing to the root directory. Geoserver was very happy with that and I got exactly the results I was hoping for. Did not need to retile or translate. Very good stuff!!

E. Scott Stricker
703-561-3671

-----Original Message-----
From: Gordon Keith [mailto:gordon.keith@anonymised.com]
Sent: Thursday, November 01, 2012 6:21 PM
To: geoserver-users@lists.sourceforge.net
Subject: EXT :Re: [Geoserver-users] Issues with image pyramid

On Fri, 2 Nov 2012 07:16:59 AM Stricker, Scott wrote:

Apologies if this is not the right forum for this question.

I'm trying to setup geoserver environment for demo purposes to display
satellite imagery such as truemarble imagery
(http://www.unearthedoutdoors.net/global_data/true_marble/download).

I'd like to set it up as follows
Scale > A - show one set of tiles
Scale range < A and > B higher res set of tiles Scale range < B and >
C even higher res set of tiles And so on.

Can I build an 'image pyramid' data set manually? For example,
configure geoserver to use 32km TIFF for lowest resolution 1km TIFF
tiles for next 500m tiles for next resolution level 250m tiles for
highest resolution

Seems like it should be fairly straightforward - what am I missing?

Thanks for any help/pointers/links that can be provided?

I build my image pyramids manually (well I've automated it using a bash script
- attached for reference).

The requirements as I understand them (mostly from trial and error):

The root directory has a .properties file and a .prj file which both have the same name as pyramid layer.

Each level of the pyramid is in a separate directory which contains an ImageMosaic consisting of a number of images (TIFF files), a shape file (.shp, .shx, .dbf) created by gdaltindex, a .properties file and a .prj file. The properties files is different from the root one while the .prj file is the same (don't know if they have to be, but mine all are).

For example I have a layer called bathy_pyramid.

In my coverages/bathy_pyrmid directory I have:

more bathy_pyramid.properties

Name=bathy_pyramid
LevelsNum=10
LevelsDirs=1 2 4 8 16 32 64 128 256 512
Envelope2D=109.8395984985,-64.0000001972 173.9999996985,0.1604010028
Levels=2.44618395E-4,2.44618395E-4 4.89236791E-4,4.89236791E-4
9.78473581E-4,9.78473581E-4 0.00195694716,0.00195694716
0.00391389432,0.00391389432 0.00782778865,0.00782778865
0.0156555773,0.0156555773 0.0313111546,0.0313111546 0.0626223092,0.0626223092
0.125244618,0.125244618

more bathy_pyramid.prj

GEOGCS["WGS 84", DATUM["World Geodetic System 1984", SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH], AUTHORITY["EPSG","4326"]]

ls 512

bathy_110_174_-64_0_bathy.tif
bathy_pyramid.dbf
bathy_pyramid.prj
bathy_pyramid.properties
bathy_pyramid.shp
bathy_pyramid.shx

more 512/bathy_pyramid.properties

Name=bathy_pyramid
LocationAttribute=location
LevelsNum=1
Envelope2D=109.8395984985,-64.0000001972 173.9999996985,0.1604010028
Levels=0.125245,0.125245

Points to note:
- The Name and Envelope2D fields for all the properties files for a pyramid must be identical.
- geoserver must have at least read access to all files and write access to the directories (very easy to overlook when you do things manually)
- The Levels field in the root properties file is used by geoserver to determine which level to use for the resolution requested, I'm not sure of the algorithm but you can play with the numbers to get it to do what you want. The Levels field in the level directories is the pixels size of the TIFFs, I think this must match the TIFFs and all the TIFFs must have the same resolution. The Levels field in the level properties and the corresponding value in the root properties don't have to be the same, but in my case always are.
- The number of (space separated) entries in LevelDirs and Levels in the root properties file must be exactly LevelsNum
- The value of the LocationAttribute field of the level properties file matches the field name used in gdaltindex, e.g.:
gdaltindex -tileindex location bathy_pyramid.shp *.tif

Hope this helps
Gordon

5

Ciao Scott,
please, read below inline my comments...

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Thu, Nov 1, 2012 at 9:16 PM, Stricker, Scott (IS)
<scott.stricker@anonymised.com> wrote:

Apologies if this is not the right forum for this question.

I'm trying to setup geoserver environment for demo purposes to display
satellite imagery such as truemarble imagery
(http://www.unearthedoutdoors.net/global_data/true_marble/download).

I'd like to set it up as follows
Scale > A - show one set of tiles
Scale range < A and > B higher res set of tiles Scale range < B and > C even
higher res set of tiles And so on.

Initial thinking was to create an image pyramid so I took the high res
(250m) TrueMarble tiles and did

gdal_translate -stats -co "TILED=YES" filename new_filename

because I was getting warnings

Warning 1: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not
sorted in ascending order ERROR 1: MissingRequired:TIFF directory is missing
required "ImageLength" field ERROR 1: Bogus block size; unable to allocate a
buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: FillEmptyTiles() failed because panByteCounts == NULL
E:\MAP_TM\A1_01_34.tif : 67584|0-->2048-2048

And thought the translate would fix these issues. After translating all
files, I did

gdal_retile.py -v -levels 4 -ps 2048 2048 -targetDir E:\MAP_TM A1.tif A2.tif
... H3.tif H4.tif

And get above errors ending with

E:\MAP_TM\A1_02_82.tif : 165888|2048-->2048-2048 Warning 1:
TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in
ascending order ERROR 1: MissingRequired:TIFF directory is missing required
"ImageLength" field ERROR 1: Bogus block size; unable to allocate a buffer.
ERROR 1: Bogus block size; unable to allocate a buffer.
More than 1000 errors or warnings have been reported. No more will be
reported from now.
E:\MAP_TM\A1_02_83.tif : 167936|2048-->2048-2048 E:\MAP_TM\A1_02_84.tif :
169984|2048-->2048-2048 E:\MAP_TM\A1_02_85.tif : 172032|2048-->768-2048
E:\MAP_TM\A1_03_01.tif : 0|4096-->2048-2048

.
.
.
Finishing up with

E:\MAP_TM\A1_32_10.tif : 18432|63488-->2048-2048 Traceback (most recent call
last):
  File "c:\Program Files\gdal\gdal_retile.py", line 1001, in <module>
    sys.exit(main(sys.argv))
  File "c:\Program Files\gdal\gdal_retile.py", line 911, in main
    dsCreatedTileIndex = tileImage(minfo,ti)
  File "c:\Program Files\gdal\gdal_retile.py", line 349, in tileImage
    createTile(minfo, offsetX, offsetY, width, height,tilename,OGRDS)
  File "c:\Program Files\gdal\gdal_retile.py", line 494, in createTile
    dec.uly+offsetY*dec.scaleY)
  File "c:\Program Files\gdal\gdal_retile.py", line 256, in getDataSet
    data = s_band.ReadRaster( readOffsetX,readOffsetY,readX,readY,
readX,readY, self.band_type )
AttributeError: 'NoneType' object has no attribute 'ReadRaster'

Am I doing something wrong?

Can I build an 'image pyramid' data set manually? For example, configure
geoserver to use 32km TIFF for lowest resolution 1km TIFF tiles for next
500m tiles for next resolution level 250m tiles for highest resolution

Seems like it should be fairly straightforward - what am I missing?

Thanks for any help/pointers/links that can be provided?

You can find some generic info inside these slides http://goo.gl/TXJRS

Anyway, I would do this:
- create the single mosaics in geoserver for the various level. Put
all the images into single directories under the same root directory
- point the imagepyramid plugin to the root directory

GeoServer should create a pyramid by picking the single mosaics. One
thing, make sure the subsampling step between two levels is 2
otherwise if the jump is too high the pyramid could slow down.

E. Scott Stricker

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users