[Geoserver-users] Image Mosaic w/ Large Amount of Granules

GeoServer GeoServer User
1

Hi,

I have an image mosaic layer with around 2100 granules that covers the whole world. Through testing, I realized that not all of my granules being displayed properly. The granules that aren’t displayed properly are returned from GeoServer as black images. If I make a smaller image mosaic layer with 30 granules in the region that was returning black images by GeoServer, this time around GeoServer is sending back the actual images correctly. Is there a recommended image mosaic configuration for large amount of granules? I tried setting the MaxAllowedTiles in the layer configuration, but that didn’t seem to fix my problem.

Thanks,

Danny

2

Hi Danny,

A couple of questions to better understand your issue:

  • Did you see any log message for those black images?

  • Can you provide some more info about your system configuration? (OS at least).

When you do very zoomed out request (as you want to see the whole word in a single view) the imageMosaic have to read all the granules involved at that level (so it has to read 2100 files). If your layer is configured to use JAI imageRead (USE_JAI_IMAGEREAD = true), then it may run out of file descriptors. Indeed, JAI ImageRead may keeps a lot of files open due to its deferred execution loading. Switching this parameter to false will let the ImageMosaic use immediate (in memory) read operations which means: open the file → read what it needs → close the file. You may also consider set allowMultithreading = true, which performs those read operations concurrently, using the configured Thread pool executor (see the Settings → Coverage Access configuration page).

However, since your mosaic is made of a big number of files you may also consider setting up an Image Pyramid on top of it.

Please, let us know.
Best regards,
Daniele

···

On Tue, Feb 3, 2015 at 12:33 AM, Danny Cheng <dcheng@anonymised.com> wrote:

Hi,

I have an image mosaic layer with around 2100 granules that covers the whole world. Through testing, I realized that not all of my granules being displayed properly. The granules that aren’t displayed properly are returned from GeoServer as black images. If I make a smaller image mosaic layer with 30 granules in the region that was returning black images by GeoServer, this time around GeoServer is sending back the actual images correctly. Is there a recommended image mosaic configuration for large amount of granules? I tried setting the MaxAllowedTiles in the layer configuration, but that didn’t seem to fix my problem.

Thanks,

Danny

Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

Hi Daniele,

After turning on verbose logging, I did see that those black images are coming back black as it think they are “No Data” values. I’m running GeoServer on SuSE Linux with 32 cores and 128GB of RAM. My image mosaic layer is configured with USE_JAIR_IMAGEREAD=false and allowMultithreading=true. Since in the log I saw “No Data”, I turned on input and output transparency in the layer configuration and now my world image seems to show up fine. It was a bit misleading because the world image tiles weren’t supposed to overlap each other and the fact that it only happened in one side of the world was misleading too.

Is the default stacking order of the image mosaic layer by fid?

Thanks,
Danny

···

Hi Danny,

A couple of questions to better understand your issue:

  • Did you see any log message for those black images?

  • Can you provide some more info about your system configuration? (OS at least).

When you do very zoomed out request (as you want to see the whole word in a single view) the imageMosaic have to read all the granules involved at that level (so it has to read 2100 files). If your layer is configured to use JAI imageRead (USE_JAI_IMAGEREAD = true), then it may run out of file descriptors. Indeed, JAI ImageRead may keeps a lot of files open due to its deferred execution loading. Switching this parameter to false will let the ImageMosaic use immediate (in memory) read operations which means: open the file → read what it needs → close the file. You may also consider set allowMultithreading = true, which performs those read operations concurrently, using the configured Thread pool executor (see the Settings → Coverage Access configuration page).

However, since your mosaic is made of a big number of files you may also consider setting up an Image Pyramid on top of it.

Please, let us know.
Best regards,
Daniele

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

On Tue, Feb 3, 2015 at 12:33 AM, Danny Cheng <dcheng@…3836…> wrote:

Hi,

I have an image mosaic layer with around 2100 granules that covers the whole world. Through testing, I realized that not all of my granules being displayed properly. The granules that aren’t displayed properly are returned from GeoServer as black images. If I make a smaller image mosaic layer with 30 granules in the region that was returning black images by GeoServer, this time around GeoServer is sending back the actual images correctly. Is there a recommended image mosaic configuration for large amount of granules? I tried setting the MaxAllowedTiles in the layer configuration, but that didn’t seem to fix my problem.

Thanks,

Danny

Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

Hi Danny,

yes, the default stacking order is by fid because the query sent to the spatial index gets back the elements sorted by fid.

If you need for a different order you can use the SORTING configuration param in the layer configuration.

You can specify attributes of the schema and the requested order.

As an instance, if you have 4D mosaic with granules available at different times and elevation with an index schema containing “time” and “elevation” attributes, you may specify a SORTING parameter like this:

time D, elevation A

in order to sort granules by time in Descending order and then by elevation in Ascending order.

Hope this helps.

Regards,

Daniele

···

On Tue, Feb 3, 2015 at 6:39 PM, Danny Cheng <dcheng@anonymised.com> wrote:

Hi Daniele,

After turning on verbose logging, I did see that those black images are coming back black as it think they are “No Data” values. I’m running GeoServer on SuSE Linux with 32 cores and 128GB of RAM. My image mosaic layer is configured with USE_JAIR_IMAGEREAD=false and allowMultithreading=true. Since in the log I saw “No Data”, I turned on input and output transparency in the layer configuration and now my world image seems to show up fine. It was a bit misleading because the world image tiles weren’t supposed to overlap each other and the fact that it only happened in one side of the world was misleading too.

Is the default stacking order of the image mosaic layer by fid?

Thanks,
Danny

From: dany.geotools@anonymised.com [mailto:dany.geotools@anonymised.com] On Behalf Of Daniele Romagnoli
Sent: Tuesday, February 03, 2015 1:12 AM
To: Danny Cheng
Cc: geoserver-users@lists.sourceforge.net
Subject: Re: [Geoserver-users] Image Mosaic w/ Large Amount of Granules

Hi Danny,

A couple of questions to better understand your issue:

  • Did you see any log message for those black images?

  • Can you provide some more info about your system configuration? (OS at least).

When you do very zoomed out request (as you want to see the whole word in a single view) the imageMosaic have to read all the granules involved at that level (so it has to read 2100 files). If your layer is configured to use JAI imageRead (USE_JAI_IMAGEREAD = true), then it may run out of file descriptors. Indeed, JAI ImageRead may keeps a lot of files open due to its deferred execution loading. Switching this parameter to false will let the ImageMosaic use immediate (in memory) read operations which means: open the file → read what it needs → close the file. You may also consider set allowMultithreading = true, which performs those read operations concurrently, using the configured Thread pool executor (see the Settings → Coverage Access configuration page).

However, since your mosaic is made of a big number of files you may also consider setting up an Image Pyramid on top of it.

Please, let us know.
Best regards,
Daniele

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

On Tue, Feb 3, 2015 at 12:33 AM, Danny Cheng <dcheng@anonymised.com> wrote:

Hi,

I have an image mosaic layer with around 2100 granules that covers the whole world. Through testing, I realized that not all of my granules being displayed properly. The granules that aren’t displayed properly are returned from GeoServer as black images. If I make a smaller image mosaic layer with 30 granules in the region that was returning black images by GeoServer, this time around GeoServer is sending back the actual images correctly. Is there a recommended image mosaic configuration for large amount of granules? I tried setting the MaxAllowedTiles in the layer configuration, but that didn’t seem to fix my problem.

Thanks,

Danny

Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.