[Geoserver-devel] Geoserver/gdal: large tif with multiple overviews - bug + workaround

Idan_Miara · January 14, 2019, 10:00am

Hi,

Due to the following bug:
https://trac.osgeo.org/gdal/ticket/5067

When I need to make multiple overviews for a large tiff I use the suggested workaround (making tif.ovr, tif.ovr.ovr, tif.ovr.ovr.ovr etc…).
This workaround works well for applications that use gdal (i.e. QGIS) but Geoserver’s internal GeoTiff reader ignores the multiple ovr files (I assume it reads only the tif+tif.ovr files) thus the performance for this map is very bad.

Workaround I found: I use the gdal-geoserver plugin and then create a trivial vrt for the tif and load it in Geoserver with the gdal plugin.
I needed to make a VRT because as far as I can tell Geoserver doesn’t register the GeoTiff format with gdal when the plugin is installed but still uses the internal GeoTiff reader).

A few questions:

Is there a plan to solve that very old gdaladdo bug? (I didn’t check with recent gdal versions but it still not marked as resolved as in 6 years)
Is it possible to make Geoserver recognize ovr.ovr files?
Can I make geoserver use the gdal plugin for GeoTiff files to avoid the trivial vrt file ?

Kind regards,
Idan

Andrea_Aime4 · January 14, 2019, 10:09am

Is it possible to make Geoserver recognize ovr.ovr files?

You can contribute changes to the code and add support for ovr.ovr files if you want to, see a guide here:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

If you want to code the improvement yourself just ask, I can give you pointers to which classes need to be modified.

Can I make geoserver use the gdal plugin for GeoTiff files to avoid the trivial vrt file ?

We’d have the code, but it has been removed years ago as it was source of unpredictable issues when deployed:
https://osgeo-org.atlassian.net/browse/GEOS-7981

Again, if you’re interested, you could spend time looking at the actual source issue and reinstate the plugin.

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Idan_Miara · January 14, 2019, 10:51am

Hi Andrea,

Thanks for very prompt reply and for your suggestions!

I’m wondering if it worth the effort to debug the current gdal-geoserver plugin.

I remember we talked in FOSS4G about the gdal 2.x support in Geoserver, is there any progress with that?

I could also reproduce the following bug (https://osgeo-org.atlassian.net/browse/GEOS-8143) for loading a vrt file but not for the internal GeoTiff loader.

Kind regards,
Idan

On Mon, 14 Jan 2019 at 12:10, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Jan 14, 2019 at 11:01 AM Idan Miara <idan@anonymised.com> wrote:

Is it possible to make Geoserver recognize ovr.ovr files?

You can contribute changes to the code and add support for ovr.ovr files if you want to, see a guide here:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

If you want to code the improvement yourself just ask, I can give you pointers to which classes need to be modified.

Can I make geoserver use the gdal plugin for GeoTiff files to avoid the trivial vrt file ?

We’d have the code, but it has been removed years ago as it was source of unpredictable issues when deployed:
https://osgeo-org.atlassian.net/browse/GEOS-7981

Again, if you’re interested, you could spend time looking at the actual source issue and reinstate the plugin.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

rouault · January 14, 2019, 10:54am

On lundi 14 janvier 2019 12:00:27 CET Idan Miara wrote:

Hi,

Due to the following bug:
#5067 (gdaladdo slowness for large files) – GDAL
When I need to make multiple overviews for a large tiff I use the suggested
workaround (making tif.ovr, tif.ovr.ovr, tif.ovr.ovr.ovr etc...).
This workaround works well for applications that use gdal (i.e. QGIS) but
Geoserver's internal GeoTiff reader ignores the multiple ovr files (I
assume it reads only the tif+tif.ovr files) thus the performance for this
map is very bad.

Workaround I found: I use the gdal-geoserver plugin and then create a
trivial vrt for the tif and load it in Geoserver with the gdal plugin.
I needed to make a VRT because as far as I can tell Geoserver doesn't
register the GeoTiff format with gdal when the plugin is installed but
still uses the internal GeoTiff reader).

A few questions:
1. Is there a plan to solve that very old gdaladdo bug? (I didn't check
with recent gdal versions but it still not marked as resolved as in 6 years)

That could be very involved to fix that due to how things work in libtiff.

You can use the workaround suggested in the ticket to create a single file
from all the .tif, .ovr, .ovr.ovr, etc.

gdal_translate process.tif process_with_ovr.tif -co COPY_SRC_OVERVIEWS YES -co
COMPRESS=DEFLATE

That will take some time, but not more than really needed.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

Andrea_Aime4 · January 14, 2019, 10:59am

Hi Andrea,

Thanks for very prompt reply and for your suggestions!

I’m wondering if it worth the effort to debug the current gdal-geoserver plugin.

I remember we talked in FOSS4G about the gdal 2.x support in Geoserver, is there any progress with that?

Not at the moment, no one has been sponsoring that work.

I could also reproduce the following bug (https://osgeo-org.atlassian.net/browse/GEOS-8143) for loading a vrt file but not for the internal GeoTiff loader.

Sorry to hear that. Anyone is free to provide a fix or sponsor one.

As Paul Ramsey said (if memory serves me well) “in open source one gets what everybody else is paying for”. Current sponsored activity does not seem to match your needs, sorry

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Idan_Miara · January 16, 2019, 10:59am

Even,
Thanks for your pointer!
It does solve the problem for me!

Andrea,
Thanks for you input!
Should I find the time to look into these issues I’ll contact you.

I’m working on a python wrapper for gdal translate/wrap/info python functions.
I’ll open source it once it’s done, hopefully it will help others with the issues I encountered.

On Mon, 14 Jan 2019 at 12:59, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Jan 14, 2019 at 11:52 AM Idan Miara <idan@anonymised.com> wrote:

Hi Andrea,

Thanks for very prompt reply and for your suggestions!

I’m wondering if it worth the effort to debug the current gdal-geoserver plugin.

I remember we talked in FOSS4G about the gdal 2.x support in Geoserver, is there any progress with that?

Not at the moment, no one has been sponsoring that work.

I could also reproduce the following bug (https://osgeo-org.atlassian.net/browse/GEOS-8143) for loading a vrt file but not for the internal GeoTiff loader.

Sorry to hear that. Anyone is free to provide a fix or sponsor one.

As Paul Ramsey said (if memory serves me well) “in open source one gets what everybody else is paying for”. Current sponsored activity does not seem to match your needs, sorry

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

rouault · January 16, 2019, 11:36am

On mercredi 16 janvier 2019 12:59:51 CET Idan Miara wrote:

Even,
Thanks for your pointer!
It does solve the problem for me!

Andrea,
Thanks for you input!
Should I find the time to look into these issues I'll contact you.

I'm working on a python wrapper for gdal translate/wrap/info python
functions.

Hum, are you aware that they are already such wrappers in recent GDAL versions with the GDAL Python bindings ? gdal.Info(), gdal.Translate(), gdal.Warp() ?

Usage examples:

github.com

OSGeo/gdal/blob/master/autotest/utilities/test_gdalinfo_lib.py

#!/usr/bin/env pytest
# -*- coding: utf-8 -*-
###############################################################################
# $Id$
#
# Project:  GDAL/OGR Test Suite
# Purpose:  test librarified gdalinfo
# Author:   Faza Mahamood <fazamhd at gmail dot com>
#
###############################################################################
# Copyright (c) 2015, Faza Mahamood <fazamhd at gmail dot com>
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included

This file has been truncated. show original

github.com

OSGeo/gdal/blob/master/autotest/utilities/test_gdal_translate_lib.py

#!/usr/bin/env pytest
# -*- coding: utf-8 -*-
###############################################################################
# $Id$
#
# Project:  GDAL/OGR Test Suite
# Purpose:  test librarified gdal_translate
# Author:   Faza Mahamood <fazamhd at gmail dot com>
#
###############################################################################
# Copyright (c) 2015, Faza Mahamood <fazamhd at gmail dot com>
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included

This file has been truncated. show original

github.com

OSGeo/gdal/blob/master/autotest/utilities/test_gdalwarp_lib.py

# ve!/usr/bin/env pytest
# -*- coding: utf-8 -*-
###############################################################################
# $Id$
#
# Project:  GDAL/OGR Test Suite
# Purpose:  test librarified gdalwarp
# Author:   Faza Mahamood <fazamhd @ gmail dot com>
#
###############################################################################
# Copyright (c) 2015, Faza Mahamood <fazamhd at gmail dot com>
# Copyright (c) 2015, Even Rouault <even.rouault at spatialys.com>
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#

This file has been truncated. show original

--
Spatialys - Geospatial professional services
http://www.spatialys.com

Idan_Miara · January 16, 2019, 11:58am

I’m using this python API, I didn’t know theses tests which do help (as the python API is not as documented as the c++ API).
I feel we’re off topic here so I’ll publish it soon in another thread in the gdal-dev list.
Thanks!

On Wed, 16 Jan 2019 at 13:36, Even Rouault <even.rouault@anonymised.com> wrote:

On mercredi 16 janvier 2019 12:59:51 CET Idan Miara wrote:

Even,
Thanks for your pointer!
It does solve the problem for me!

Andrea,
Thanks for you input!
Should I find the time to look into these issues I’ll contact you.

I’m working on a python wrapper for gdal translate/wrap/info python
functions.

Hum, are you aware that they are already such wrappers in recent GDAL versions with the GDAL Python bindings ? gdal.Info(), gdal.Translate(), gdal.Warp() ?

Usage examples:
https://github.com/OSGeo/gdal/blob/master/autotest/utilities/test_gdalinfo_lib.py
https://github.com/OSGeo/gdal/blob/master/autotest/utilities/test_gdal_translate_lib.py
https://github.com/OSGeo/gdal/blob/master/autotest/utilities/test_gdalwarp_lib.py

–
Spatialys - Geospatial professional services
http://www.spatialys.com