[Geoserver-devel] Geopackage tiles generation issues

Hi,

Given that the modules are unsupported and not working properly anyways, I’d >rather fix things to follow GDAL’s approach.

Any objection?

No objection. If two open source GIS software under the OSGeo umbrella do something in the same way we can start calling it as a de facto standard.

Even has been committing a lot to GPKG lately but mostly to the vector side. It might still be good to check if GDAL still follows the same approach as you are planning to follow.

-Jukka Rahkonen-

Andrea Aime wrote:

[Geoserver-devel] Geopackage tiles generation issues

···

Hi,

I am looking into why the raster geopackages GeoServer generates via the geopackage module are not

being read correctly by gdal and found a few issues, some straight bugs, other more like odd design decisions.

The straight bug is what’s causing GDAL not to read the files correctly, the current code, shared with mbtiles

generation, forces the request to be cast on a GWC managed gridset (and actually works only if the requested

layers are cached, which is a bit of a nonsense to me given this is a WMS request…), and uses that gridset

coordinates for tiles.

Now the problem is that in the geopackage metadata for the layer and the gridset we are declaring the requested bbox, not the

ones used for tiles generation, resulting in tile coordinates being off the expected grid.

Now, geopackage natively has the layer bounds that are separate from the tile matrix ones, but the current geotools code

forces them to be the same… that’s a limitation I’m fixing.

However, the main issue remains… if someone asks for a layer in a certain SRS, why try to coax it into a tile matrix managed

by GWC?

I believe the intention was to try and leverage GWC cache, but still, while I understand taking the opportunity if possible, in

my opinion the request of the user should be respected, and if the request is not a match either by SRS or BBOX to a tileset,

GeoServer should not force a different response.

From the same line of thought, a WMS request should not fail if it’s made against a layer that is not cached by GWC, like it

does today, with a strange error message too, e.g. asking for UTM 32N, which is not tile cached:

http://demo.geo-solutions.it/geoserver/wms/reflect?layers=nurc:mosaic&srs=EPSG:32632&format=geopackage

results in:

java.lang.IllegalArgumentException: Thread 19276 Unknown layer nurc:mosaic. Check the logfiles, it may not have loaded properly

The second thing that I find confusing is, if I asked for a result image that’s 768 pixels wide, why am I getting back a geopackage that’s

over 100000 pixels wide? That’s… insane, especially for a WMS request (large jobs should be deferred to WPS asynch requests).

Looking at the code, there is logic that generates the min and max zoom levels for the tile matrix, and it does something rather surprising…

it picks the zoom level closest to the requested one as the min zoom level, and then starts adding zoom levels until the matrix

has (stricly) more than 256 tiles in width… in my case, that means 512 tiles, or a total width of 512*256=131072 pixels!

Trying GDAL on the command line, translating from a geotiff to a geopackage, it makes what looks like a more sensible choice instead: it uses the

requested zoom level as the maximum one, and then adds overviews to it, in the cases I’ve tried, until a single tile is left (basically, a good

output with overviews). With our case we can also generate raster geopackages out of vector layers, and creating the overviews might end up

hitting badly setup styles displaying too much data, but still, that would be a problem also in random WMS requests, so I’m not too concerned…

Also, by default, GDAL generates a gridset that’s an exact match for the source data, meaning a custom one. That also makes sense to me, if someone

wants something other than the BBOX/SRS requested there are format options to choose both the target gridset and min/max zoom levels.

Given that the modules are unsupported and not working properly anyways, I’d rather fix things to follow GDAL’s approach.

Any objection?

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Le mardi 18 octobre 2016 12:13:59, Rahkonen Jukka (MML) a écrit :

Hi,

>Given that the modules are unsupported and not working properly anyways,
>I'd >rather fix things to follow GDAL's approach. Any objection?

No objection. If two open source GIS software under the OSGeo umbrella do
something in the same way we can start calling it as a de facto standard.

In the matter that would be more like "profiles" of what is allowed within the
OGC standard.

Even has been committing a lot to GPKG lately but mostly to the vector
side. It might still be good to check if GDAL still follows the same
approach as you are planning to follow.

No change in the GPKG raster approach on the GDAL side done recently. Andrea's
analysis matches well how it is designed.

Note that Brad Hards identified a possible non compliance of the GDAL writer in
some circumstances regarding a picky requirement
See (the part about Requirement 45, the issue with now vs CURRENT_TIMESTAMP
has now been fixed)
https://lists.osgeo.org/pipermail/gdal-dev/2016-August/045098.html
https://lists.osgeo.org/pipermail/gdal-dev/2016-August/045100.html

-Jukka Rahkonen-

Andrea Aime wrote:
[Geoserver-devel] Geopackage tiles generation issues

Hi,
I am looking into why the raster geopackages GeoServer generates via the
geopackage module are not being read correctly by gdal and found a few
issues, some straight bugs, other more like odd design decisions.

The straight bug is what's causing GDAL not to read the files correctly,
the current code, shared with mbtiles generation, forces the request to be
cast on a GWC managed gridset (and actually works only if the requested
layers are cached, which is a bit of a nonsense to me given this is a WMS
request...), and uses that gridset coordinates for tiles.
Now the problem is that in the geopackage metadata for the layer and the
gridset we are declaring the requested bbox, not the ones used for tiles
generation, resulting in tile coordinates being off the expected grid.
Now, geopackage natively has the layer bounds that are separate from the
tile matrix ones, but the current geotools code forces them to be the
same.. that's a limitation I'm fixing.

However, the main issue remains... if someone asks for a layer in a certain
SRS, why try to coax it into a tile matrix managed by GWC?
I believe the intention was to try and leverage GWC cache, but still, while
I understand taking the opportunity if possible, in my opinion the request
of the user should be respected, and if the request is not a match either
by SRS or BBOX to a tileset, GeoServer should not force a different
response.
From the same line of thought, a WMS request should not fail if it's made
against a layer that is not cached by GWC, like it does today, with a
strange error message too, e.g. asking for UTM 32N, which is not tile
cached:

http://demo.geo-solutions.it/geoserver/wms/reflect?layers=nurc:mosaic&srs=E
PSG:32632&format=geopackage

results in:

java.lang.IllegalArgumentException: Thread 19276 Unknown layer nurc:mosaic.
Check the logfiles, it may not have loaded properly

The second thing that I find confusing is, if I asked for a result image
that's 768 pixels wide, why am I getting back a geopackage that's over
100000 pixels wide? That's... insane, especially for a WMS request (large
jobs should be deferred to WPS asynch requests). Looking at the code,
there is logic that generates the min and max zoom levels for the tile
matrix, and it does something rather surprising... it picks the zoom level
closest to the requested one as the min zoom level, and then starts adding
zoom levels until the matrix has (stricly) more than 256 tiles in width...
in my case, that means 512 tiles, or a total width of 512*256=131072
pixels!

Trying GDAL on the command line, translating from a geotiff to a
geopackage, it makes what looks like a more sensible choice instead: it
uses the requested zoom level as the _maximum_ one, and then adds
overviews to it, in the cases I've tried, until a single tile is left
(basically, a good output with overviews). With our case we can also
generate raster geopackages out of vector layers, and creating the
overviews might end up hitting badly setup styles displaying too much
data, but still, that would be a problem also in random WMS requests, so
I'm not too concerned...

Also, by default, GDAL generates a gridset that's an exact match for the
source data, meaning a custom one. That also makes sense to me, if someone
wants something other than the BBOX/SRS requested there are format options
to choose both the target gridset and min/max zoom levels.

Given that the modules are unsupported and not working properly anyways,
I'd rather fix things to follow GDAL's approach. Any objection?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
GeoSolutions Enterprise Support Services for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
x.com

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure,
reproduction, copying, distribution, or either dissemination, either whole
or partial, is strictly forbidden except previous formal approval of the
named addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--
Spatialys - Geospatial professional services
http://www.spatialys.com

Hi Even,
thanks a lot for chiming in :slight_smile:

I have a question about GDAL behavior, I made some more tests today and the geopackage
always seems to have only one zoom level. Wondering if I just fussed around yesterday or I used
some option that I don’t remember about :slight_smile:

Looking at the docs it seems that if someone wants some overviews gdaladdo should be used to add them

Cheers
Andrea

···

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Le mercredi 19 octobre 2016 12:12:03, Andrea Aime a écrit :

Hi Even,
thanks a lot for chiming in :slight_smile:

I have a question about GDAL behavior, I made some more tests today and the
geopackage
always seems to have only one zoom level. Wondering if I just fussed around
yesterday or I used
some option that I don't remember about :slight_smile:

Looking at the docs it seems that if someone wants some overviews gdaladdo
should be used to add them

If you just do gdal_translate or gdalwarp, you will get only the maximum zoom
level filled in (entries for lower zoom levels will be reserved in
gpkg_tile_matrix though)
If you want lower zoom levels to be filled, you indeed need to run gdaladdo.
Exactly for other formats like GeoTIFF etc.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com