[Geoserver-users] Temporal mosaic on multi-band GeoTiff

GeoServer GeoServer User
1

Hi folks,

is there a way to let Geoserver handle a multi-band Geotiff as a temporal mosaic? I have a large collection of GeoTiffs that each have about 1000+ bands. The temporal reference is related to the bands in the following way:

data.tif#band1: 01-2006
data.tif#band2: 02-2006
data.tif#band2: 03-2006
....
data.tif#bandN: 12-2099

Up to now I have built mosaics as described in [1]. But littering my disk with thousands of small files seems not to be a good solution in this case. Mapping timestamps into bands would be my favourite solution.

Is this possible, and if so, how can I do it?

Cheers,
Matthias

[1]: http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic_timeseries/imagemosaic_timeseries.html

2

Dear Matthias,
I am afraid this is not possible atm although it would probably be not
too difficult to achieve.

That said, I am not sure this is the best approach performance wise
since it makes the tiff directory very long and forces the tiff reader
to read a ton of metadata.
This is aprtly due a suboptimal code in the tiff read (we read all the
metadata at once in memory) but, still I would not recommend this
approach upfront.
I would rather split the tif and let GeoServer handle them as a single
ImageMosaic. You might want to group them in separate directories
under a single directory
in the end GeoServer will search recursively under the toor directory
for files that can be indexed.

Hope that helps.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Jul 16, 2014 at 9:01 PM, Matthias Müller
<Matthias_Mueller@anonymised.com> wrote:

Hi folks,

is there a way to let Geoserver handle a multi-band Geotiff as a
temporal mosaic? I have a large collection of GeoTiffs that each have
about 1000+ bands. The temporal reference is related to the bands in the
following way:

data.tif#band1: 01-2006
data.tif#band2: 02-2006
data.tif#band2: 03-2006
....
data.tif#bandN: 12-2099

Up to now I have built mosaics as described in [1]. But littering my
disk with thousands of small files seems not to be a good solution in
this case. Mapping timestamps into bands would be my favourite solution.

Is this possible, and if so, how can I do it?

Cheers,
Matthias

[1]:
http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic_timeseries/imagemosaic_timeseries.html

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

Dear Simone,

thanks, this is the definitive answer I was looking for.

Regards,
Matthias

p.s.: Performance-wise I was also thinking of free file handles and inodes that are quickly eaten up by large numbers of very small files.

Am 18.07.2014 13:19, schrieb Simone Giannecchini:

Dear Matthias,
I am afraid this is not possible atm although it would probably be not
too difficult to achieve.

That said, I am not sure this is the best approach performance wise
since it makes the tiff directory very long and forces the tiff reader
to read a ton of metadata.
This is aprtly due a suboptimal code in the tiff read (we read all the
metadata at once in memory) but, still I would not recommend this
approach upfront.
I would rather split the tif and let GeoServer handle them as a single
ImageMosaic. You might want to group them in separate directories
under a single directory
in the end GeoServer will search recursively under the toor directory
for files that can be indexed.

Hope that helps.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Jul 16, 2014 at 9:01 PM, Matthias Müller
<Matthias_Mueller@anonymised.com> wrote:

Hi folks,

is there a way to let Geoserver handle a multi-band Geotiff as a
temporal mosaic? I have a large collection of GeoTiffs that each have
about 1000+ bands. The temporal reference is related to the bands in the
following way:

data.tif#band1: 01-2006
data.tif#band2: 02-2006
data.tif#band2: 03-2006
....
data.tif#bandN: 12-2099

Up to now I have built mosaics as described in [1]. But littering my
disk with thousands of small files seems not to be a good solution in
this case. Mapping timestamps into bands would be my favourite solution.

Is this possible, and if so, how can I do it?

Cheers,
Matthias

[1]:
http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic_timeseries/imagemosaic_timeseries.html

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

Dear Matthias,
a few things.

-1- Could you open up a JIRA task for this feature as it might be
interesting to implement when we have some spare time.

-2- About file handles and iNode.
You are right having too many (small) files can be a problem as well.
But you know when a nswering email on the mailing list you never know
the level of knowledge of the people in fron tof you hence you tend to
over-simplify answers :slight_smile:

Generally speaking when we create pyramids or mosaics ourselves we
tend to balance between have too many small files by merging them and
having geotiff files that are too big (e.g. 1TB) since the geotiff
reader is inefficient (sigh.... it could be improved though with some
effort) when reading metadata (aka tiff directories) which can become
huge for extremely large files with decent tiling.

In a situation like yours merging is not a solution as you are
mosaicking not only (or not at all) spatially but rther on the time
dimension and using bands as dimensions is not yet supported. As I
suggested above, I would open a JIRA task for this as we might want to
add support for it, it would not tremendously difficult in the end and
I see the potential of this approach.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Mon, Jul 21, 2014 at 7:20 PM, Matthias Müller
<Matthias_Mueller@anonymised.com> wrote:

Dear Simone,

thanks, this is the definitive answer I was looking for.

Regards,
Matthias

p.s.: Performance-wise I was also thinking of free file handles and
inodes that are quickly eaten up by large numbers of very small files.

Am 18.07.2014 13:19, schrieb Simone Giannecchini:

Dear Matthias,
I am afraid this is not possible atm although it would probably be not
too difficult to achieve.

That said, I am not sure this is the best approach performance wise
since it makes the tiff directory very long and forces the tiff reader
to read a ton of metadata.
This is aprtly due a suboptimal code in the tiff read (we read all the
metadata at once in memory) but, still I would not recommend this
approach upfront.
I would rather split the tif and let GeoServer handle them as a single
ImageMosaic. You might want to group them in separate directories
under a single directory
in the end GeoServer will search recursively under the toor directory
for files that can be indexed.

Hope that helps.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Jul 16, 2014 at 9:01 PM, Matthias Müller
<Matthias_Mueller@anonymised.com> wrote:

Hi folks,

is there a way to let Geoserver handle a multi-band Geotiff as a
temporal mosaic? I have a large collection of GeoTiffs that each have
about 1000+ bands. The temporal reference is related to the bands in the
following way:

data.tif#band1: 01-2006
data.tif#band2: 02-2006
data.tif#band2: 03-2006
....
data.tif#bandN: 12-2099

Up to now I have built mosaics as described in [1]. But littering my
disk with thousands of small files seems not to be a good solution in
this case. Mapping timestamps into bands would be my favourite solution.

Is this possible, and if so, how can I do it?

Cheers,
Matthias

[1]:
http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic_timeseries/imagemosaic_timeseries.html

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

5

Done.

https://jira.codehaus.org/browse/GEOS-6590

Regards,
Matthias

-----Ursprüngliche Nachricht-----
Von: simboss1@anonymised.com [mailto:simboss1@anonymised.com] Im Auftrag von Simone Giannecchini
Gesendet: Dienstag, 22. Juli 2014 07:54
An: Matthias Müller
Cc: geoserver-users
Betreff: Re: [Geoserver-users] Temporal mosaic on multi-band GeoTiff

Dear Matthias,
a few things.

-1- Could you open up a JIRA task for this feature as it might be interesting to implement when we have some spare time.

-2- About file handles and iNode.
You are right having too many (small) files can be a problem as well.
But you know when a nswering email on the mailing list you never know the level of knowledge of the people in fron tof you hence you tend to over-simplify answers :slight_smile:

Generally speaking when we create pyramids or mosaics ourselves we tend to balance between have too many small files by merging them and having geotiff files that are too big (e.g. 1TB) since the geotiff reader is inefficient (sigh.... it could be improved though with some
effort) when reading metadata (aka tiff directories) which can become huge for extremely large files with decent tiling.

In a situation like yours merging is not a solution as you are mosaicking not only (or not at all) spatially but rther on the time dimension and using bands as dimensions is not yet supported. As I suggested above, I would open a JIRA task for this as we might want to add support for it, it would not tremendously difficult in the end and I see the potential of this approach.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Mon, Jul 21, 2014 at 7:20 PM, Matthias Müller <Matthias_Mueller@anonymised.com4476...> wrote:

Dear Simone,

thanks, this is the definitive answer I was looking for.

Regards,
Matthias

p.s.: Performance-wise I was also thinking of free file handles and
inodes that are quickly eaten up by large numbers of very small files.

Am 18.07.2014 13:19, schrieb Simone Giannecchini:

Dear Matthias,
I am afraid this is not possible atm although it would probably be
not too difficult to achieve.

That said, I am not sure this is the best approach performance wise
since it makes the tiff directory very long and forces the tiff
reader to read a ton of metadata.
This is aprtly due a suboptimal code in the tiff read (we read all
the metadata at once in memory) but, still I would not recommend this
approach upfront.
I would rather split the tif and let GeoServer handle them as a
single ImageMosaic. You might want to group them in separate
directories under a single directory in the end GeoServer will search
recursively under the toor directory for files that can be indexed.

Hope that helps.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Jul 16, 2014 at 9:01 PM, Matthias Müller
<Matthias_Mueller@anonymised.com> wrote:

Hi folks,

is there a way to let Geoserver handle a multi-band Geotiff as a
temporal mosaic? I have a large collection of GeoTiffs that each
have about 1000+ bands. The temporal reference is related to the
bands in the following way:

data.tif#band1: 01-2006
data.tif#band2: 02-2006
data.tif#band2: 03-2006
....
data.tif#bandN: 12-2099

Up to now I have built mosaics as described in [1]. But littering my
disk with thousands of small files seems not to be a good solution
in this case. Mapping timestamps into bands would be my favourite solution.

Is this possible, and if so, how can I do it?

Cheers,
Matthias

[1]:
http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic_times
eries/imagemosaic_timeseries.html

--------------------------------------------------------------------
---------- Want fast and easy access to all the code in your
enterprise? Index and search up to 200,000 lines of code with a free
copy of Black Duck Code Sight - the same software that powers the
world's largest code search on Ohloh, the Black Duck Open Hub! Try
it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

---------------------------------------------------------------------
--------- Want fast and easy access to all the code in your
enterprise? Index and search up to 200,000 lines of code with a free
copy of Black Duck Code Sight - the same software that powers the
world's largest code search on Ohloh, the Black Duck Open Hub! Try it
now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

----------------------------------------------------------------------
-------- Want fast and easy access to all the code in your enterprise?
Index and search up to 200,000 lines of code with a free copy of Black
Duck Code Sight - the same software that powers the world's largest
code search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users