[Geoserver-users] questions about pyramids or overviews for very large image mosaics

Vincent_Schut · October 20, 2006, 1:28pm

Continuing the WCS testing saga...

We have a bunch of srtm raster data here and I thought that might be nice for some coverage mosaic testing... Creating the mosaic went OK using the mosaicIndexBuilder from geotools/ext/coverage-development. However, when trying to create pyramids for this mosaics, the pyramidbuilder gives an error:

Oct 20, 2006 3:12:43 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Requested scale factor is 2
Oct 20, 2006 3:12:43 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Acquiring a mosaic reader to mosaic /mnt/amalia/srtm3/SRTM_Data_GeoTiff/srtm3_mosaic.shp
Error while parsing JAI registry file "/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai" :
Error in registry file at line number #31
A descriptor is already registered against the name "org.geotools.Combine" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name "org.geotools.Hysteresis" under registry mode "rendered"
Error in registry file at line number #33
A descriptor is already registered against the name "org.geotools.NodataFilter" under registry mode "rendered"
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original envelope is [180°00.0'W 60°00.0'S , 180°00.0'E 60°00.0'N]
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original range is GeneralGridRange[0..432000, 0..144000]
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New dimension is (W,H)==(216000.0,72000.0)
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New matrix dimension is (cols,rows)==(72,24)
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Preparing tile (col,row)==(0,0) to file /mnt/amalia/srtm3/SRTM_Data_GeoTiff/2/mosaic_0.tiff
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Reading with grid envelope [180°00.0'W 60°00.0'S , 175°00.0'W 55°00.0'S]
Oct 20, 2006 3:12:46 PM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...
Exception in thread "PyramidBuilder" java.lang.NullPointerException
    at org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:88)
    at org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:148)
    at it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679)
    at java.lang.Thread.run(Thread.java:595)

This is using geotools 2.3 branch from today's svn. JAI and JAI-IO are from cvs couple of days ago.
This error is independent from 1) tile size (tried 500, 1000 and 3000) and 2) scaling algorithm (tried nn, avg, bil).
Now this is a pretty large mosaic, so maybe that is a problem? Earlier I tried running the pyramidBuilder on a subset, and that mostly went OK (not always, but alas the errors were erratic and not reproduceable). I'll start some testing on smaller mosaics, but in the mean time want to ask:

- should the pyramidBuilder be able to handle mosaics this large?
- any idea what could be the problem?
side note questions:
- is it also possible to achieve the same performance by adding internal overviews to the original mosaic tiles? Or will pyramids outperform internal overviews?
- and if I use internal overviews, will the imagemosaic plugin automatically use these when a lower resolution image is requested?

Cheers,
Vincent.

Simone3 · October 20, 2006, 1:57pm

On 10/20/06, Vincent Schut <schut@anonymised.com> wrote:

Continuing the WCS testing saga...

We have a bunch of srtm raster data here and I thought that might be
nice for some coverage mosaic testing... Creating the mosaic went OK
using the mosaicIndexBuilder from geotools/ext/coverage-development.

However, when trying to create pyramids for this mosaics, the
pyramidbuilder gives an error:

Oct 20, 2006 3:12:43 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Requested scale factor is 2
Oct 20, 2006 3:12:43 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Acquiring a mosaic reader to mosaic
/mnt/amalia/srtm3/SRTM_Data_GeoTiff/srtm3_mosaic.shp
Error while parsing JAI registry file
"/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai"
:
Error in registry file at line number #31
A descriptor is already registered against the name
"org.geotools.Combine" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name
"org.geotools.Hysteresis" under registry mode "rendered"
Error in registry file at line number #33
A descriptor is already registered against the name
"org.geotools.NodataFilter" under registry mode "rendered"
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original envelope is [180°00.0'W 60°00.0'S ,
180°00.0'E 60°00.0'N]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original range is GeneralGridRange[0..432000,
0..144000]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New dimension is (W,H)==(216000.0,72000.0)
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New matrix dimension is (cols,rows)==(72,24)
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Preparing tile (col,row)==(0,0) to file
/mnt/amalia/srtm3/SRTM_Data_GeoTiff/2/mosaic_0.tiff
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Reading with grid envelope [180°00.0'W 60°00.0'S ,
175°00.0'W 55°00.0'S]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...
Exception in thread "PyramidBuilder" java.lang.NullPointerException
   at
org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:88)
   at
org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:148)
   at
it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679)
   at java.lang.Thread.run(Thread.java:595)

This error is quite strange. It seems that JAI is not able to register
some of the custom operations that we have in geotools, which by the
way are not used by the PyramidBuilder. Can you give some details
about he platform where this is happening? Anyway the code that
actaully does this registration comes Martin Desruisseaux (he is
cc-ed) hence I would like to know what he thinks of this error.

This is using geotools 2.3 branch from today's svn. JAI and JAI-IO are
from cvs couple of days ago.
This error is independent from 1) tile size (tried 500, 1000 and 3000)
and 2) scaling algorithm (tried nn, avg, bil).
Now this is a pretty large mosaic, so maybe that is a problem? Earlier I
tried running the pyramidBuilder on a subset, and that mostly went OK
(not always, but alas the errors were erratic and not reproduceable).

At FOSS06 I showed people various mosaics, one had been made from Blue
marble , the base level add 1000 images, but I had another one
covering chicago where the base level had exaclty 10.000 images. I am
builing a new one for san diego with 20.000 image so, in thery 3000
should not be that much :-).

I'll start some testing on smaller mosaics, but in the mean time want to
ask:

- should the pyramidBuilder be able to handle mosaics this large?

See above.

- any idea what could be the problem?

See above.

side note questions:
- is it also possible to achieve the same performance by adding internal
overviews to the original mosaic tiles? Or will pyramids outperform
internal overviews?

Rationale of the ImagePyramid. If you have less then 300 images of a
reasonable size (up to 100 mb) usin the ImageMosaic with embedded
overviews should be fine. Beyond that it won't work ebcause you might
try to open too many files at the same time having problems very
quickly on some platforms. Pyramids are there to reduce not only the
resoulution but also the number of files to be used. Moreover you
avoid seeking into a big file and you get more parallelism.

- and if I use internal overviews, will the imagemosaic plugin
automatically use these when a lower resolution image is requested?

Yeah, I fixed a bug in the mosaic plugin just three days ago that was
giving strange results depending on the resolution. Now it work
perfectly (well, fine is probably better because it generates less
expectations ).

Cheers,
Vincent.

Ciao,
Simone.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions

http://www.geo-solutions.it

-------------------------------------------------------

Vincent_Schut · October 20, 2006, 2:17pm

-------- Original Message --------
From: "Simone Giannecchini" <simboss1@anonymised.com>
To: "Vincent Schut" <schut@anonymised.com>
Subject: Re:[Geoserver-devel] questions about pyramids or overviews for very large image mosaics
Date: 10/20/2006 03:57 PM

On 10/20/06, Vincent Schut <schut@anonymised.com> wrote:

Continuing the WCS testing saga...

We have a bunch of srtm raster data here and I thought that might be
nice for some coverage mosaic testing... Creating the mosaic went OK
using the mosaicIndexBuilder from geotools/ext/coverage-development.

However, when trying to create pyramids for this mosaics, the
pyramidbuilder gives an error:

Oct 20, 2006 3:12:43 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Requested scale factor is 2
Oct 20, 2006 3:12:43 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Acquiring a mosaic reader to mosaic
/mnt/amalia/srtm3/SRTM_Data_GeoTiff/srtm3_mosaic.shp
Error while parsing JAI registry file
"/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai"

:
Error in registry file at line number #31
A descriptor is already registered against the name
"org.geotools.Combine" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name
"org.geotools.Hysteresis" under registry mode "rendered"
Error in registry file at line number #33
A descriptor is already registered against the name
"org.geotools.NodataFilter" under registry mode "rendered"
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original envelope is [180°00.0'W 60°00.0'S ,
180°00.0'E 60°00.0'N]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original range is GeneralGridRange[0..432000,
0..144000]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New dimension is (W,H)==(216000.0,72000.0)
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New matrix dimension is (cols,rows)==(72,24)
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Preparing tile (col,row)==(0,0) to file
/mnt/amalia/srtm3/SRTM_Data_GeoTiff/2/mosaic_0.tiff
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Reading with grid envelope [180°00.0'W 60°00.0'S ,
175°00.0'W 55°00.0'S]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...
Exception in thread "PyramidBuilder" java.lang.NullPointerException
   at
org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:88)

   at
org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:148)

   at
it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679)

   at java.lang.Thread.run(Thread.java:595)

This error is quite strange. It seems that JAI is not able to register
some of the custom operations that we have in geotools, which by the
way are not used by the PyramidBuilder. Can you give some details
about he platform where this is happening? Anyway the code that
actaully does this registration comes Martin Desruisseaux (he is
cc-ed) hence I would like to know what he thinks of this error.

This is on gentoo linux, amd64. sun-jdk 1.5.

This is using geotools 2.3 branch from today's svn. JAI and JAI-IO are
from cvs couple of days ago.
This error is independent from 1) tile size (tried 500, 1000 and 3000)
and 2) scaling algorithm (tried nn, avg, bil).
Now this is a pretty large mosaic, so maybe that is a problem? Earlier I
tried running the pyramidBuilder on a subset, and that mostly went OK
(not always, but alas the errors were erratic and not reproduceable).

At FOSS06 I showed people various mosaics, one had been made from Blue
marble , the base level add 1000 images, but I had another one
covering chicago where the base level had exaclty 10.000 images. I am
builing a new one for san diego with 20.000 image so, in thery 3000
should not be that much :-).

OK, clear

I'll start some testing on smaller mosaics, but in the mean time want to
ask:

- should the pyramidBuilder be able to handle mosaics this large?

See above.

- any idea what could be the problem?

See above.

side note questions:
- is it also possible to achieve the same performance by adding internal
overviews to the original mosaic tiles? Or will pyramids outperform
internal overviews?

Rationale of the ImagePyramid. If you have less then 300 images of a
reasonable size (up to 100 mb) usin the ImageMosaic with embedded
overviews should be fine. Beyond that it won't work ebcause you might
try to open too many files at the same time having problems very
quickly on some platforms. Pyramids are there to reduce not only the
resoulution but also the number of files to be used. Moreover you
avoid seeking into a big file and you get more parallelism.

Aha, hadn't thought of the open files thing.

- and if I use internal overviews, will the imagemosaic plugin
automatically use these when a lower resolution image is requested?

Yeah, I fixed a bug in the mosaic plugin just three days ago that was
giving strange results depending on the resolution. Now it work
perfectly (well, fine is probably better because it generates less
expectations ).

Good to know. Will continue to try and get the pyramidBuilder working though.
However, first rebuilding geotools and geoserver because when trying to visualize a mosaic geoserver complains about the new input stream being not found (<ServiceException>it/geosolutions/imageio/stream/input/FileImageInputStreamExtImpl</ServiceException> </ServiceExceptionReport>). Hope rebuilding geotools and geoserver and redeploying will solve this...
Then I'll try some pyramids going with less files first, see if that changes things. I'll keep you updated.

Ciao,
Vincenzo (been in Italy recently ;-))

Cheers,
Vincent.

Ciao,
Simone.

-------------------------------------------------------------------------

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Simone3 · October 20, 2006, 2:33pm

On 10/20/06, Vincent Schut <schut@anonymised.com> wrote:

-------- Original Message --------
From: "Simone Giannecchini" <simboss1@anonymised.com>
To: "Vincent Schut" <schut@anonymised.com>
Subject: Re:[Geoserver-devel] questions about pyramids or overviews for
very large image mosaics
Date: 10/20/2006 03:57 PM
> On 10/20/06, Vincent Schut <schut@anonymised.com> wrote:
>> Continuing the WCS testing saga...
>>
>> We have a bunch of srtm raster data here and I thought that might be
>> nice for some coverage mosaic testing... Creating the mosaic went OK
>> using the mosaicIndexBuilder from geotools/ext/coverage-development.
>
>> However, when trying to create pyramids for this mosaics, the
>> pyramidbuilder gives an error:
>>
>> Oct 20, 2006 3:12:43 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Requested scale factor is 2
>> Oct 20, 2006 3:12:43 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Acquiring a mosaic reader to mosaic
>> /mnt/amalia/srtm3/SRTM_Data_GeoTiff/srtm3_mosaic.shp
>> Error while parsing JAI registry file
>> "/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai"
>>
>> :
>> Error in registry file at line number #31
>> A descriptor is already registered against the name
>> "org.geotools.Combine" under registry mode "rendered"
>> Error in registry file at line number #32
>> A descriptor is already registered against the name
>> "org.geotools.Hysteresis" under registry mode "rendered"
>> Error in registry file at line number #33
>> A descriptor is already registered against the name
>> "org.geotools.NodataFilter" under registry mode "rendered"
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Original envelope is [180°00.0'W 60°00.0'S ,
>> 180°00.0'E 60°00.0'N]
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Original range is GeneralGridRange[0..432000,
>> 0..144000]
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: New dimension is (W,H)==(216000.0,72000.0)
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: New matrix dimension is (cols,rows)==(72,24)
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Preparing tile (col,row)==(0,0) to file
>> /mnt/amalia/srtm3/SRTM_Data_GeoTiff/2/mosaic_0.tiff
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Reading with grid envelope [180°00.0'W 60°00.0'S ,
>> 175°00.0'W 55°00.0'S]
>> Oct 20, 2006 3:12:46 PM
>> it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
>> INFO: Progress is at 0.0
>> attached message is: Scaling...
>> Exception in thread "PyramidBuilder" java.lang.NullPointerException
>> at
>> org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:88)
>>
>> at
>> org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:148)
>>
>> at
>> it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679)
>>
>> at java.lang.Thread.run(Thread.java:595)
>>
>
> This error is quite strange. It seems that JAI is not able to register
> some of the custom operations that we have in geotools, which by the
> way are not used by the PyramidBuilder. Can you give some details
> about he platform where this is happening? Anyway the code that
> actaully does this registration comes Martin Desruisseaux (he is
> cc-ed) hence I would like to know what he thinks of this error.
This is on gentoo linux, amd64. sun-jdk 1.5.
>
>> This is using geotools 2.3 branch from today's svn. JAI and JAI-IO are
>> from cvs couple of days ago.
>> This error is independent from 1) tile size (tried 500, 1000 and 3000)
>> and 2) scaling algorithm (tried nn, avg, bil).
>> Now this is a pretty large mosaic, so maybe that is a problem? Earlier I
>> tried running the pyramidBuilder on a subset, and that mostly went OK
>> (not always, but alas the errors were erratic and not reproduceable).
>
> At FOSS06 I showed people various mosaics, one had been made from Blue
> marble , the base level add 1000 images, but I had another one
> covering chicago where the base level had exaclty 10.000 images. I am
> builing a new one for san diego with 20.000 image so, in thery 3000
> should not be that much :-).
OK, clear
>
>> I'll start some testing on smaller mosaics, but in the mean time want to
>> ask:
>>
>> - should the pyramidBuilder be able to handle mosaics this large?
>
> See above.
>
>> - any idea what could be the problem?
>
> See above.
>
>> side note questions:
>> - is it also possible to achieve the same performance by adding internal
>> overviews to the original mosaic tiles? Or will pyramids outperform
>> internal overviews?
>
> Rationale of the ImagePyramid. If you have less then 300 images of a
> reasonable size (up to 100 mb) usin the ImageMosaic with embedded
> overviews should be fine. Beyond that it won't work ebcause you might
> try to open too many files at the same time having problems very
> quickly on some platforms. Pyramids are there to reduce not only the
> resoulution but also the number of files to be used. Moreover you
> avoid seeking into a big file and you get more parallelism.
Aha, hadn't thought of the open files thing.
>
>> - and if I use internal overviews, will the imagemosaic plugin
>> automatically use these when a lower resolution image is requested?
>
> Yeah, I fixed a bug in the mosaic plugin just three days ago that was
> giving strange results depending on the resolution. Now it work
> perfectly (well, fine is probably better because it generates less
> expectations ).
Good to know. Will continue to try and get the pyramidBuilder working
though.
However, first rebuilding geotools and geoserver because when trying to
visualize a mosaic geoserver complains about the new input stream being
not found
(<ServiceException>it/geosolutions/imageio/stream/input/FileImageInputStreamExtImpl</ServiceException>
</ServiceExceptionReport>). Hope rebuilding geotools and geoserver and
redeploying will solve this...
Then I'll try some pyramids going with less files first, see if that
changes things. I'll keep you updated.

As I told you this error is quite strange, it is not relate at all to
the PyramidBuilder. As I said let's first see what Martin says, then
I'll look myself at it a but closer.
You running on linux right?

Simone.

Ciao,
Vincenzo (been in Italy recently ;-))
>>
>> Cheers,
>> Vincent.
>>
>
> Ciao,
> Simone.
>> -------------------------------------------------------------------------
>>
>> Using Tomcat but need to do more? Need to support web services,
>> security?
>> Get stuff done quickly with pre-integrated technology to make your
>> job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>> Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Geoserver-devel mailing list
>> Geoserver-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>
>

--
-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions

http://www.geo-solutions.it

-------------------------------------------------------

Vincent_Schut · October 20, 2006, 2:53pm

-------- Original Message --------
From: "Simone Giannecchini" <simboss1@anonymised.com>
To: "Vincent Schut" <schut@anonymised.com>
Subject: Re:[Geotools-devel] [Geoserver-devel] questions about pyramids or overviews for very large image mosaics
Date: 10/20/2006 04:33 PM

On 10/20/06, Vincent Schut <schut@anonymised.com> wrote:


-------- Original Message --------
From: "Simone Giannecchini" <simboss1@anonymised.com>
To: "Vincent Schut" <schut@anonymised.com>
Subject: Re:[Geoserver-devel] questions about pyramids or overviews for
very large image mosaics
Date: 10/20/2006 03:57 PM


On 10/20/06, Vincent Schut <schut@anonymised.com> wrote:


Continuing the WCS testing saga...

We have a bunch of srtm raster data here and I thought that might be
nice for some coverage mosaic testing... Creating the mosaic went OK
using the mosaicIndexBuilder from geotools/ext/coverage-development.

However, when trying to create pyramids for this mosaics, the
pyramidbuilder gives an error:

Oct 20, 2006 3:12:43 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Requested scale factor is 2
Oct 20, 2006 3:12:43 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Acquiring a mosaic reader to mosaic
/mnt/amalia/srtm3/SRTM_Data_GeoTiff/srtm3_mosaic.shp
Error while parsing JAI registry file
"/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai"

:
Error in registry file at line number #31
A descriptor is already registered against the name
"org.geotools.Combine" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name
"org.geotools.Hysteresis" under registry mode "rendered"
Error in registry file at line number #33
A descriptor is already registered against the name
"org.geotools.NodataFilter" under registry mode "rendered"
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original envelope is [180°00.0'W 60°00.0'S ,
180°00.0'E 60°00.0'N]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Original range is GeneralGridRange[0..432000,
0..144000]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New dimension is (W,H)==(216000.0,72000.0)
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: New matrix dimension is (cols,rows)==(72,24)
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Preparing tile (col,row)==(0,0) to file
/mnt/amalia/srtm3/SRTM_Data_GeoTiff/2/mosaic_0.tiff
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Reading with grid envelope [180°00.0'W 60°00.0'S ,
175°00.0'W 55°00.0'S]
Oct 20, 2006 3:12:46 PM
it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...
Exception in thread "PyramidBuilder" java.lang.NullPointerException
   at
org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:88)

   at
org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:148)

   at
it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679)

   at java.lang.Thread.run(Thread.java:595)

This error is quite strange. It seems that JAI is not able to register
some of the custom operations that we have in geotools, which by the
way are not used by the PyramidBuilder. Can you give some details
about he platform where this is happening? Anyway the code that
actaully does this registration comes Martin Desruisseaux (he is
cc-ed) hence I would like to know what he thinks of this error.


This is on gentoo linux, amd64. sun-jdk 1.5.


This is using geotools 2.3 branch from today's svn. JAI and JAI-IO are
from cvs couple of days ago.
This error is independent from 1) tile size (tried 500, 1000 and 3000)
and 2) scaling algorithm (tried nn, avg, bil).
Now this is a pretty large mosaic, so maybe that is a problem? Earlier I
tried running the pyramidBuilder on a subset, and that mostly went OK
(not always, but alas the errors were erratic and not reproduceable).


At FOSS06 I showed people various mosaics, one had been made from Blue
marble , the base level add 1000 images, but I had another one
covering chicago where the base level had exaclty 10.000 images. I am
builing a new one for san diego with 20.000 image so, in thery 3000
should not be that much :-).


OK, clear


I'll start some testing on smaller mosaics, but in the mean time want to
ask:

- should the pyramidBuilder be able to handle mosaics this large?


See above.

- any idea what could be the problem?


See above.

side note questions:
- is it also possible to achieve the same performance by adding internal
overviews to the original mosaic tiles? Or will pyramids outperform
internal overviews?


Rationale of the ImagePyramid. If you have less then 300 images of a
reasonable size (up to 100 mb) usin the ImageMosaic with embedded
overviews should be fine. Beyond that it won't work ebcause you might
try to open too many files at the same time having problems very
quickly on some platforms. Pyramids are there to reduce not only the
resoulution but also the number of files to be used. Moreover you
avoid seeking into a big file and you get more parallelism.


Aha, hadn't thought of the open files thing.


- and if I use internal overviews, will the imagemosaic plugin
automatically use these when a lower resolution image is requested?


Yeah, I fixed a bug in the mosaic plugin just three days ago that was
giving strange results depending on the resolution. Now it work
perfectly (well, fine is probably better because it generates less
expectations ).


Good to know. Will continue to try and get the pyramidBuilder working
though.
However, first rebuilding geotools and geoserver because when trying to
visualize a mosaic geoserver complains about the new input stream being
not found
(<ServiceException>it/geosolutions/imageio/stream/input/FileImageInputStreamExtImpl</ServiceException>
</ServiceExceptionReport>). Hope rebuilding geotools and geoserver and
redeploying will solve this...
Then I'll try some pyramids going with less files first, see if that
changes things. I'll keep you updated.

As I told you this error is quite strange, it is not relate at all to
the PyramidBuilder. As I said let's first see what Martin says, then
I'll look myself at it a but closer.
You running on linux right?

yep. Linux (gentoo) on amd64. JAI & JAI-IO from cvs (couple of days old), java 5 (sun-jdk), tomcat 5.5 (however that has nothing to do with the pyramidBuilder of course). Everything is running in native 64 bit mode.

I know the error is strange, and I know I am testing several things through each other at the same time which makes things hard to separate... Not a very fluent java developer myself (python, actually...), but I've been doing some java & JAI stuff (just creating processing chains and some simple custom operations, no real magic) myself not so long ago so I know a bit what we're talking about, and how hard it can be to solve (and find) JAI related errors. Wonderful library still, though

Reason I mention testing with less images is because I did manage to get the pyramidBuilder to run on a small subset of srtm tiles some days ago. So either something happened to the codebase or my setup in between, or it actually is somehow related to the dataset (not necessarily to it's size). I'm of course curious about Martins comments too, though.

Anyway, thanks for looking into this. I'll keep you and the list updated on my progress, and eagerly await any comments or hints.

Vincent.

Simone.

Ciao,
Vincenzo (been in Italy recently ;-))

Cheers,
Vincent.

Ciao,
Simone.

-------------------------------------------------------------------------

Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your
job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Martin_Desruisseaux · October 22, 2006, 1:33pm

Simone Giannecchini a écrit :

Error while parsing JAI registry file
"/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai"

Error in registry file at line number #31
A descriptor is already registered against the name
"org.geotools.Combine" under registry mode "rendered"

Maybe you have two gt2-coverage.jar files in your classpath? An old JAR and the new one that you just compiled from SVN?

Martin

Vincent_Schut · October 23, 2006, 10:43am

Simone Giannecchini a écrit :


Error while parsing JAI registry file
"/usr/local/src/geotools/branch-2.3.x/module/coverage/target/classes/META-INF/registryFile.jai"

Error in registry file at line number #31
A descriptor is already registered against the name
"org.geotools.Combine" under registry mode "rendered"

Maybe you have two gt2-coverage.jar files in your classpath? An old JAR and the new one that you just compiled from SVN?

  Martin

Hmm, should have thought about that myself... For some reason, eclipse included the loaded gt-2.3.0 projects as well as the maven2 repo gt-2.4.0 libraries... Or probably I ignorantly told eclipse to do so because I'm not (yet) really used to working with eclipse and it's pretty complex, especially when working with large sets of projects like geotools, combined with maven... Well, cleaned that one up.

Now for the actual error, which had not disappeared after that: I found out what the problem is.
When having a mosaic with holes (like the srtm elevation data, which is only defined for tiles that include land masses), the pyramidbuilder plugin wants to build pyramids also for the undefined tiles in the mosaic. When entering such a tile for processing, the mosaicreader cannot get at the raster source (which is undefined, of course) and returns with a null pointer. Ecco, the nullpointer exception.
As an illustration I have included part of the output for the pyramidBuilder when processing a smaller mosaic. The mosaic is only one North-South line of srtm tiles, but the part from 20 degrees north till 40 degrees north is missing (sea). The pyramidBuilder starts building from the most southern tile (not included in the output paste) and continues till it reaches 20 degrees north. Then it tries to build a pyramid for data starting at 20 degrees north, which obviously isn't there, which results in the error.

paste of the output (don't be fooled by the fact that the exception is printed before the actual notification, I double checked it and it really occurs at getting the renderedImage from the input gc):

attached message is: Reading with grid envelope [70°00.0'W 10°00.0'N , 65°00.0'W 15°00.0'N]
Oct 23, 2006 11:43:23 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...
Oct 23, 2006 11:43:23 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.5833333333333334
attached message is: Writing out...
Oct 23, 2006 11:43:37 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.625
attached message is: Preparing tile (col,row)==(0,15) to file /mnt/amalia/srtm3/geotools_mosaic_test/2/mosaic_15.tiff
Oct 23, 2006 11:43:37 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.625
attached message is: Reading with grid envelope [70°00.0'W 15°00.0'N , 65°00.0'W 20°00.0'N]
Oct 23, 2006 11:43:37 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...
Oct 23, 2006 11:43:37 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.625
attached message is: Writing out...
Exception in thread "PyramidBuilder" java.lang.NullPointerException
    at org.geotools.coverage.processing.operation.ScaledGridCoverage2D.create(ScaledGridCoverage2D.java:91)
    at org.geotools.coverage.processing.operation.Scale.doOperation(Scale.java:149)
    at it.geosolutions.utils.imagepyramid.PyramidBuilder.run(PyramidBuilder.java:679)
    at java.lang.Thread.run(Thread.java:595)
Oct 23, 2006 11:43:45 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.6666666666666666
attached message is: Preparing tile (col,row)==(0,16) to file /mnt/amalia/srtm3/geotools_mosaic_test/2/mosaic_16.tiff
Oct 23, 2006 11:43:45 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.6666666666666666
attached message is: Reading with grid envelope [70°00.0'W 20°00.0'N , 65°00.0'W 25°00.0'N]
Oct 23, 2006 11:43:45 AM it.geosolutions.utils.imagepyramid.PyramidBuilder getNotification
INFO: Progress is at 0.0
attached message is: Scaling...

Cheers,
Vincent.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel