[Geoserver-devel] geopackage community module

Justin_Deoliveira · July 11, 2013, 3:40pm

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Francesco_Izzi1 · July 11, 2013, 8:10pm

+1 Justin

···

2013/7/11 Justin Deoliveira <jdeolive@anonymised.com>

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Francesco Izzi
CNR - IMAA
geoSDI
Direzione Tecnologie e Sviluppo

C.da S. Loja
85050 Tito Scalo - POTENZA (PZ)
Italia

phone: +39 0971427310
fax: +39 0971 427271
mob: +39 3666373172
mail: francesco.izzi@anonymised.com
skype: neofx8080

web: http://www.geosdi.org

Jukka_Rahkonen · July 11, 2013, 8:48pm

Hi,

Interesting. It would be nice to see how a geopackage file with metadata tables and some vectors looks like. Is there any sample available?

-Jukka Rahkonen-

________________________________
Justin Deoliveira wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

-Justin

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Justin_Deoliveira · July 11, 2013, 9:24pm

Hey Jukka,

You can find a sample package with the spearfish data here.

http://dev.opengeo.org/~jdeolive/geopkg/spearfish.zip

Just unzip it and you should be able to connect to the resulting .geopackage file with any sqlite client.

-Justin

···

On Thu, Jul 11, 2013 at 2:48 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com> wrote:

Hi,

Interesting. It would be nice to see how a geopackage file with metadata tables and some vectors looks like. Is there any sample available?

-Jukka Rahkonen-

Justin Deoliveira wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Andrea_Aime4 · July 12, 2013, 12:52pm

On Thu, Jul 11, 2013 at 5:40 PM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also
developed an output format for geoserver capable of producing a geopackage
file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

+1 on the module
As a curiosity, why wms? Is this going to produce tiles in the geopackage?
Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Chris_Holmes1 · July 12, 2013, 1:27pm

On Fri, Jul 12, 2013 at 8:52 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Thu, Jul 11, 2013 at 5:40 PM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also
developed an output format for geoserver capable of producing a geopackage
file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

+1 on the module
As a curiosity, why wms? Is this going to produce tiles in the geopackage?
Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

Probably should have both, like you did with KML recently.

GeoPackage is actually a few things - a package. I've been on most of the
calls in the last few months. And tried to get it split up in to multiple
specs, but that was one I didn't mange to pull off. So the two main things
it contains are tiles and vectors. So tiles contain rendered tiles. It's
similar to MBTiles, but you can have multiple tile sets, and do projections
other than web mercator.

Features is a wkb based binary format. I guess in the next testbed they're
going to think more about the service interface. Does seem like it'd be
useful on the wms side to be able to ask for tiles or features or both for
any vector layers. Rasters should be tiles, though we can also experiment
with putting raster blobs directly in sqlite tables. Originally 'rasters'
was a type as well, but we cut that out as it wasn't thought out well
enough, but left a spot where one could put it. So eventually we could
maybe do some format options to specify if you want raw data, tiles, or
both.

But doing a WFS output format would make good sense, as that would indicate
that you just want features - you'd always get a features only package when
requesting from WFS.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Justin_Deoliveira · July 12, 2013, 1:48pm

On Fri, Jul 12, 2013 at 6:52 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Thu, Jul 11, 2013 at 5:40 PM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also
developed an output format for geoserver capable of producing a geopackage
file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

+1 on the module
As a curiosity, why wms? Is this going to produce tiles in the geopackage?
Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

Yeah, the package can actually contain three types of tables, vector,
raster, and tiles. So wms seemed the best fit but certainly nothing
stopping wfs and wcs output formats.

Afaik there is nothing describing styles in the package but it would be a
useful extension i think.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Chris_Holmes1 · July 12, 2013, 1:58pm

On Fri, Jul 12, 2013 at 9:48 AM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

On Fri, Jul 12, 2013 at 6:52 AM, Andrea Aime <andrea.aime@anonymised.com
> wrote:

On Thu, Jul 11, 2013 at 5:40 PM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also
developed an output format for geoserver capable of producing a geopackage
file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

+1 on the module
As a curiosity, why wms? Is this going to produce tiles in the
geopackage? Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

Yeah, the package can actually contain three types of tables, vector,
raster, and tiles. So wms seemed the best fit but certainly nothing
stopping wfs and wcs output formats.

Afaik there is nothing describing styles in the package but it would be a
useful extension i think.

+1 on a styles extension. No discussion of this took place in the
geopackage group, but it's extensible enough to just throw in a table that
we understand. MapStory would definitely benefit from this, maybe it can
take a crack at it, as a way to pull down a full mapstory with data and
styles. Be able to move it to another server, etc.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

David_Winslow · July 12, 2013, 2:05pm

Interesting idea, having a style as a SQL table. I wonder if it could be designed so that some or all of the dynamic feature selection could be done by the database engine.

SELECT f.geometry, s.stroke, s.width, s.opacity FROM features f, styles s WHERE …

···

On Fri, Jul 12, 2013 at 9:58 AM, Chris Holmes <cholmes@anonymised.com> wrote:

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Fri, Jul 12, 2013 at 9:48 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

+1 on a styles extension. No discussion of this took place in the geopackage group, but it’s extensible enough to just throw in a table that we understand. MapStory would definitely benefit from this, maybe it can take a crack at it, as a way to pull down a full mapstory with data and styles. Be able to move it to another server, etc.

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Fri, Jul 12, 2013 at 6:52 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Thu, Jul 11, 2013 at 5:40 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Yeah, the package can actually contain three types of tables, vector, raster, and tiles. So wms seemed the best fit but certainly nothing stopping wfs and wcs output formats.

Afaik there is nothing describing styles in the package but it would be a useful extension i think.

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

+1 on the module
As a curiosity, why wms? Is this going to produce tiles in the geopackage? Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Jukka_Rahkonen · July 13, 2013, 9:19am

Hi,

It looks like the GeoPackage BLOB is pretty close to Spatialite BLOB but still so much different that no existing software can read it out of the box. Differences are in a new magic numbers, dropping out the geometry type code, and support for several alternatives for storing the feature bounding box. The latter means that there are four alternatives for how many bytes must be skipped before the WKB part begins. I am sure if using ISO 13249 WKB instead of OGC WKB will make any additional trouble but let's hope not but I can't even interpret which WKB there is included in spearfish GeoPackege sample. It may be that JTS does not read ISO WKB and that would be sad http://sourceforge.net/p/jts-topo-suite/code/878/tree/trunk/jts/java/src/com/vividsolutions/jts/io/WKBReader.java

General note about the GeoPackage specification is that I am not sure if it supports features with empty geometries or not. Features with empty geometries are not uncommon in GIS work (new parcel is created by administration and stored into database but it is not digitized yet etc.) so I hope that GeoPackage developers have not forgotten this. The Spearfish sample database does not have "not null" constraint for the geometry field so null geometry seems to be OK. Sometimes empty geometries would be better as Paul Ramsey wrote http://blog.cleverelephant.ca/2010/03/nothing-nada-zip-bupkus.html

One scenario in GeoPackage is that any SQLite capable client could just read the Geopackage BLOBs without a need to know anything more about GeoPackage. The OpenJUMP DB Query plugin http://sourceforge.net/projects/jumpdbqplugin/ seems to be almost ready for that because is can already do the same with Spatialite BLOB. The plugin does not care about stuff like geometry_columns but it just rips off the WKB part of the BLOB and sends it to JTS Wkb reader. The logic used by the plugin is to analyze the Spatialite BLOB is described in the source code as

//From http://www.gaia-gis.it/spatialite-2.1/SpatiaLite-manual.html
//Spatialite geometry blobs are WKB-like, with some specifics to
//spatialite: For our purposes, this should be good enough:
//the 39th byte must be 0x7C (marks MBR end)
//and the blob must end with 0xFE

I must say that for a non-programmer like me it is much more easy to understand the Spatialite BLOB structure from the document http://www.gaia-gis.it/gaia-sins/BLOB-Geometry.html than to understand how the Geopackage BLOB is built by reading the GeoPackage implementation specification v. 0.7.2.

Now I believe that if somebody wants just to read the WKB part from the GeoPackage BLOB it could be done like this:

1) Check if BLOB begins with "GPB" bytes (47 50 42). If yes, then it is a GeoPagkage BLOB.
2) Study the fourth byte and find the "E" bits (1, 2, and 3).
3) Skip the four srs_id bytes and based on the envelope type, either 0, 32, 48, or 64 bytes more.
4) Here starts the WKB

Thus the first byte of WKB is either byte number 9, 41, 57, or 73. Have I understood it right and counted the bytes correctly? If I have, then I suppose that DB Query Plugin would work with GeoPackage BLOBs after someone adds a few lines of code for checking the envelope flags for getting the correct numbers of bytes to skip. Flags.java by Justin seems to do this. For comparison here is the corresponding part of the Spatialite BLOB reader. tt is enough to start always from the 40th byte because there is only one way for giving envelope in Spatialite and it has a fixed length.

From \jumpdbplugin-src-0.8.2.zip\jumpdbplugin-src-0.8.2\src\org\freevoice\jumpdbqueryextension\spatialite\JumpSpatialiteDbQuery.java

//copy the byte array, removing the MBR at the front,
//and the ending OxFE byte at the end
byte wkb = new byte[blobAsBytes.length - 39];
System.arraycopy(blobAsBytes, 39, wkb, 1, blobAsBytes.length - 1 - 39);

//prepend byte-order byte
wkb[0] = blobAsBytes[1];

What do you think, could it really be that simple? Or does ISO WKB vs. OGC WKB break the simplicity?

-Jukka Rahkonen-

________________________________
Lähettäjä: Justin Deoliveira [jdeolive@anonymised.com]
Lähetetty: 12. heinäkuuta 2013 0:24
Vastaanottaja: Rahkonen Jukka
Cc: geoserver-devel@lists.sourceforge.net
Aihe: Re: [Geoserver-devel] geopackage community module

Hey Jukka,

You can find a sample package with the spearfish data here.

http://dev.opengeo.org/~jdeolive/geopkg/spearfish.zip

Just unzip it and you should be able to connect to the resulting .geopackage file with any sqlite client.

-Justin

On Thu, Jul 11, 2013 at 2:48 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com<mailto:jukka.rahkonen@anonymised.com>> wrote:
Hi,

Interesting. It would be nice to see how a geopackage file with metadata tables and some vectors looks like. Is there any sample available?

-Jukka Rahkonen-

________________________________
Justin Deoliveira wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

-Justin

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net<mailto:Geoserver-devel@anonymised.comrge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Andrea_Aime4 · July 13, 2013, 12:13pm

On Fri, Jul 12, 2013 at 3:48 PM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

+1 on the module

As a curiosity, why wms? Is this going to produce tiles in the
geopackage? Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

Yeah, the package can actually contain three types of tables, vector,
raster, and tiles. So wms seemed the best fit but certainly nothing
stopping wfs and wcs output formats.

Afaik there is nothing describing styles in the package but it would be a
useful extension i think.

Thinking a bit out loud about the existing services and what a geopackage
output might mean for them (warning, I did not have a look at what the
geopackage module does, so I'm really just thinking out loud):
* wfs wise, it's clear we want to extract raw vector data, so geopackage
with the requested feature collections inside the package should be the
natural fit
* wcs wise, the rasters seem the good fit, although a flat tiling scheme
might be of interest in case we want to allow fast extraction of a certain
subset of flat data.
And I'm also wondering about out usual n-dimensiona raster approach which
is really made of stacked rasters associated to a tuple of dimension
values,
wondering if an how that would be a fit for the geopackage construct
* wmts wise, it seems like a nice way to get not a single tile, but a
"sub-pyramid" of tiles fitting a certain bounding box and a certain set of
zoom levels. However, there is no such a thing in WMTS as a multi-tile
request, so I guess the protocol would have to be extended with a new
operation, GetTiles I suppose
* wms wise, there is plenty of interpretation. The meaning of a GetMap is
clear, we want a painted, styled map out of the request, the usual criteria
for selecting an output for WMS is that styles have a say in what you
generate. This is true for all image formats, and also true for the less
visual GeoRSS, in which the style still has a say in what you get back
depending on the scale dependencies and filters contained in the styles.

So... what is geopackage for wms?
* a package containing the same raster I would get from a image/png
request.... the strictest match, but not very useful
* a package containing tiles to represent that map, maybe for a bunch of
zoom levels around the requested one... sounds better, gives a way to look
at the map once disconnected, and yet still ends up missing the
GetFeatureInfo ability
* a package containing the ingredients to build the requested map, raw
vector, raw raster, and the associated styles. For a disconnected client
that's probably the most interesting of the outputs, provided that it can
be interpreted client side... so the styles would have to be SLD, which
would probably finally raise it from "that thing that GeoServer and Deegree
use to drive the map painter" to something actually useful :-p
Still, even in this case the styles should have a say, so I'd expect the
output package contents to only contain the vectors that are really visible
in the map (not all of them), and possibly a raster that's not only cut to
the bbox, but maybe also reduced in resolution (and this also makes me
wonder if the vectors should be generalized as well?)

In all of the above examples there is still a large elephant in the room:
all of the above requests are synchronous, and yet the package is capable
of hosting large downloads. Yet, as far as I understand it, it cannot be
generated in a streaming fashion, so there is a risk that the HTTP
connection
dies and fails while the geopackage is being written (that's especially
true for the partially connected scenario for which the geopackage is
designed for).
So... maybe WPS is overall a better fit, in that it has the ability to
support asynchronous requests, and the client is free to go its merry way
and maybe temporarily disconnect from the net while the server is preparing
the output.

Well... enough ranting I guess? :-p

Cheers
Andrea

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Chris_Holmes1 · July 13, 2013, 12:45pm

Hey Jukka, thanks for all the feedback. I’ve been working on the specification, and just send the information to the group (and I hope to use this experience to figure out how to help open up the process more, so you could join it just like an open source project instead of me having to be a go between).

···

So the reason we decided for WKB for the blob was to ensure that more existing WKB readers could make use of it. Spatialite’s blob isn’t quite wkb, at least that’s what people in the working group discussed. So if the ISO wkb doesn’t work with most readers then we definitely did something seriously wrong. If you could try it out that’d be great. Though the geopackage Justin created I believe used JTS to create the wkb, so it should be able to read it. Note also that there’s another open source geopackage implementation, at https://bitbucket.org/luciad/libgpkg

I’ll make sure the specification says what to do about features with empty geometries, if it’s not there already. I don’t remember discussions that we’ve had that have talked about it explicitly. This is great feedback.

A core design goal that I pushed a lot was to be able to have geopackage blobs be able to be read and written without needing any additional libraries, like spatialite.

There have been lots of changes since 0.7.2, to support exactly the use case you describe, so hopefully the newer version of the spec makes it easier to understand. So it should be as simple as you describe.

The variable bytes for the envelope size was based on a suggestion from Paul Ramsey. It does make the reader slightly more complicated, but it leads to a very significant space saving for points, since there’s no good reason to include an envelope on a point.

Thanks again for this Jukka, it’s quite helpful.

Chris

On Sat, Jul 13, 2013 at 5:19 AM, Rahkonen Jukka <jukka.rahkonen@anonymised.com> wrote:

Hi,

It looks like the GeoPackage BLOB is pretty close to Spatialite BLOB but still so much different that no existing software can read it out of the box. Differences are in a new magic numbers, dropping out the geometry type code, and support for several alternatives for storing the feature bounding box. The latter means that there are four alternatives for how many bytes must be skipped before the WKB part begins. I am sure if using ISO 13249 WKB instead of OGC WKB will make any additional trouble but let’s hope not but I can’t even interpret which WKB there is included in spearfish GeoPackege sample. It may be that JTS does not read ISO WKB and that would be sad http://sourceforge.net/p/jts-topo-suite/code/878/tree/trunk/jts/java/src/com/vividsolutions/jts/io/WKBReader.java

General note about the GeoPackage specification is that I am not sure if it supports features with empty geometries or not. Features with empty geometries are not uncommon in GIS work (new parcel is created by administration and stored into database but it is not digitized yet etc.) so I hope that GeoPackage developers have not forgotten this. The Spearfish sample database does not have “not null” constraint for the geometry field so null geometry seems to be OK. Sometimes empty geometries would be better as Paul Ramsey wrote http://blog.cleverelephant.ca/2010/03/nothing-nada-zip-bupkus.html

One scenario in GeoPackage is that any SQLite capable client could just read the Geopackage BLOBs without a need to know anything more about GeoPackage. The OpenJUMP DB Query plugin http://sourceforge.net/projects/jumpdbqplugin/ seems to be almost ready for that because is can already do the same with Spatialite BLOB. The plugin does not care about stuff like geometry_columns but it just rips off the WKB part of the BLOB and sends it to JTS Wkb reader. The logic used by the plugin is to analyze the Spatialite BLOB is described in the source code as

//From http://www.gaia-gis.it/spatialite-2.1/SpatiaLite-manual.html
//Spatialite geometry blobs are WKB-like, with some specifics to
//spatialite: For our purposes, this should be good enough:
//the 39th byte must be 0x7C (marks MBR end)
//and the blob must end with 0xFE

I must say that for a non-programmer like me it is much more easy to understand the Spatialite BLOB structure from the document http://www.gaia-gis.it/gaia-sins/BLOB-Geometry.html than to understand how the Geopackage BLOB is built by reading the GeoPackage implementation specification v. 0.7.2.

Now I believe that if somebody wants just to read the WKB part from the GeoPackage BLOB it could be done like this:

Check if BLOB begins with “GPB” bytes (47 50 42). If yes, then it is a GeoPagkage BLOB.

Study the fourth byte and find the “E” bits (1, 2, and 3).

Skip the four srs_id bytes and based on the envelope type, either 0, 32, 48, or 64 bytes more.

Here starts the WKB

Thus the first byte of WKB is either byte number 9, 41, 57, or 73. Have I understood it right and counted the bytes correctly? If I have, then I suppose that DB Query Plugin would work with GeoPackage BLOBs after someone adds a few lines of code for checking the envelope flags for getting the correct numbers of bytes to skip. Flags.java by Justin seems to do this. For comparison here is the corresponding part of the Spatialite BLOB reader. tt is enough to start always from the 40th byte because there is only one way for giving envelope in Spatialite and it has a fixed length.

From \jumpdbplugin-src-0.8.2.zip\jumpdbplugin-src-0.8.2\src\org\freevoice\jumpdbqueryextension\spatialite\JumpSpatialiteDbQuery.java

//copy the byte array, removing the MBR at the front,
//and the ending OxFE byte at the end
byte wkb = new byte[blobAsBytes.length - 39];
System.arraycopy(blobAsBytes, 39, wkb, 1, blobAsBytes.length - 1 - 39);

//prepend byte-order byte
wkb[0] = blobAsBytes[1];

What do you think, could it really be that simple? Or does ISO WKB vs. OGC WKB break the simplicity?

-Jukka Rahkonen-

Lähettäjä: Justin Deoliveira [jdeolive@anonymised.com]
Lähetetty: 12. heinäkuuta 2013 0:24
Vastaanottaja: Rahkonen Jukka
Cc: geoserver-devel@lists.sourceforge.net
Aihe: Re: [Geoserver-devel] geopackage community module

Hey Jukka,

You can find a sample package with the spearfish data here.

http://dev.opengeo.org/~jdeolive/geopkg/spearfish.zip

Just unzip it and you should be able to connect to the resulting .geopackage file with any sqlite client.

-Justin

On Thu, Jul 11, 2013 at 2:48 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com.mailto:[jukka.rahkonen@anonymised.com](mailto:jukka.rahkonen@anonymised.com)> wrote:
Hi,

Interesting. It would be nice to see how a geopackage file with metadata tables and some vectors looks like. Is there any sample available?

-Jukka Rahkonen-

Justin Deoliveira wrote:

Hi all,

To go along with the geopkg module just added to geotools we have also developed an output format for geoserver capable of producing a geopackage file from a wms request. I would like to add it as a community module.

The code is up in my github repo.

https://github.com/jdeolive/geoserver/compare/geopkg

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list

Geoserver-devel@lists.sourceforge.net mailto:[Geoserver-devel@anonymised.comists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)

https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Chris_Holmes1 · July 13, 2013, 1:03pm

Nice thoughts Andrea. A few responses in line. In general we considered most all of this out of scope for geopackage 1.0, but it is of interest for many. And I think a next testbed should lead to modifications of geopackage and service specs. Some people have talked about a ‘geopackage service’, but I’d rather see it just be integrated in lots of other specs.

···

On Sat, Jul 13, 2013 at 8:13 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jul 12, 2013 at 3:48 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Definitely the most straightforward.

Yeah, for 1.0 we actually ended up punting on rasters. Like we say very little about them, since we just didn’t have the expertise to do them really well, as it opens up a whole bunch of questions. So we just have some slight guidance that says you could use a feature table to store blobs. My hope was we’d get more experimentation in the market with rasters, see what people would want to do with them. So I think/hope geopackage should give us enough to come up with an approach that makes sense for raster / wcs output. I don’t think anyone had thought about the n-dimensional raster stuff though. Which makes me glad we didn’t try to say anything about it, as I’m sure it would have not included anything that smart. We just had no great raster experts in the initial group. Would be great to get you and/or Simone on the next round where we do tackle it.

Interesting. Yeah, would make sense to extend it a bit for that operation.

Yeah, given de-emphasis on raster in the spec I’d say it wouldn’t mean this - it’s much more about tiles.

The best approach I’ve seen for this is mapbox’s UTFGrids. http://www.mapbox.com/developers/utfgrid/

Yeah. Realistically though I think we’d be better served by having it be a standardized CSS, as that’s where the world is going. In the short term I think style will be something different implementations add on, a vendor specific thing.

Yeah, seems like those are the two main possibilities - give me tiles, and give me data/styles to make it myself.

Interesting, I hadn’t considered that. I wonder if in WFS we can use those new weird precomputed queries in some way to help.

That said I do hope that the core geopackage is simple enough to generate in a streaming fashion. Just write the blobs out, don’t need full sqlite. Not sure if that’s true though, would love insight from Justin if the current thing is streaming. But even if it is streaming there’s still the risk the http connection dies.

See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Thinking a bit out loud about the existing services and what a geopackage output might mean for them (warning, I did not have a look at what the geopackage module does, so I’m really just thinking out loud):

wfs wise, it’s clear we want to extract raw vector data, so geopackage with the requested feature collections inside the package should be the natural fit

Yeah, the package can actually contain three types of tables, vector, raster, and tiles. So wms seemed the best fit but certainly nothing stopping wfs and wcs output formats.

Afaik there is nothing describing styles in the package but it would be a useful extension i think.

+1 on the module
As a curiosity, why wms? Is this going to produce tiles in the geopackage? Or does the geopackage embed styles as well?
I guess the real question is, why wms and not wfs output format?

wcs wise, the rasters seem the good fit, although a flat tiling scheme might be of interest in case we want to allow fast extraction of a certain subset of flat data.
And I’m also wondering about out usual n-dimensiona raster approach which is really made of stacked rasters associated to a tuple of dimension values,
wondering if an how that would be a fit for the geopackage construct
wmts wise, it seems like a nice way to get not a single tile, but a “sub-pyramid” of tiles fitting a certain bounding box and a certain set of zoom levels. However, there is no such a thing in WMTS as a multi-tile request, so I guess the protocol would have to be extended with a new operation, GetTiles I suppose
wms wise, there is plenty of interpretation. The meaning of a GetMap is clear, we want a painted, styled map out of the request, the usual criteria for selecting an output for WMS is that styles have a say in what you generate. This is true for all image formats, and also true for the less visual GeoRSS, in which the style still has a say in what you get back depending on the scale dependencies and filters contained in the styles.

So… what is geopackage for wms?

a package containing the same raster I would get from a image/png request… the strictest match, but not very useful
a package containing tiles to represent that map, maybe for a bunch of zoom levels around the requested one… sounds better, gives a way to look at the map once disconnected, and yet still ends up missing the GetFeatureInfo ability
a package containing the ingredients to build the requested map, raw vector, raw raster, and the associated styles. For a disconnected client that’s probably the most interesting of the outputs, provided that it can be interpreted client side… so the styles would have to be SLD, which would probably finally raise it from “that thing that GeoServer and Deegree use to drive the map painter” to something actually useful :-p

Still, even in this case the styles should have a say, so I’d expect the output package contents to only contain the vectors that are really visible in the map (not all of them), and possibly a raster that’s not only cut to the bbox, but maybe also reduced in resolution (and this also makes me wonder if the vectors should be generalized as well?)

In all of the above examples there is still a large elephant in the room: all of the above requests are synchronous, and yet the package is capable
of hosting large downloads. Yet, as far as I understand it, it cannot be generated in a streaming fashion, so there is a risk that the HTTP connection
dies and fails while the geopackage is being written (that’s especially true for the partially connected scenario for which the geopackage is
designed for).
So… maybe WPS is overall a better fit, in that it has the ability to support asynchronous requests, and the client is free to go its merry way
and maybe temporarily disconnect from the net while the server is preparing the output.

Well… enough ranting I guess? :-p

Cheers

Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Andrea_Aime4 · July 13, 2013, 1:22pm

On Sat, Jul 13, 2013 at 3:03 PM, Chris Holmes <cholmes@anonymised.com> wrote:

That said I do hope that the core geopackage is simple enough to generate

in a streaming fashion. Just write the blobs out, don't need full sqlite.
Not sure if that's true though, would love insight from Justin if the
current thing is streaming. But even if it is streaming there's still the
risk the http connection dies.

That would require some native writer that just generates the output
without relying on a JDBC driver, otherwise the output is ready to be sent
only the moment we close the database connection.
As an observation, many http libraries will be rather keen to timeout on
you if you don't respond within N seconds, but there is nothing to time out
on if you keep on receiving data, yet, as you say, the possibility of the
connection just dying on you is still very real, using some protocol
allowing for resumes would be better

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Jukka_Rahkonen · July 13, 2013, 4:06pm

Hi,

One possible source for troubles came into my mind. People will not be satisfied with the delivered GeoPackages but they will want to modify and combine data from several packages. With SQLite it is very easy to add data from other SQLite data files by using the Attach database method. Now the possible problem is that srs_id in the feature tables are referring to srid column of the spatial_ref_sys table of the mother SQLite database. This means that srid X used in the attached table does not necessarily mean the same as the same srid code in the main GeoPackage. If user adds data as "create table added_data as select * from attached_table" the BLOBs are copied as is and the result can be unexpected. User should after this simple step to
- check the meaning of srid=X from the attached GeoPackage
- find the code for the same reference system from the main GeoPackage
- if corresponding srtext or auth_name/auth_srid combination is missing from the main GeoPackage, add new line into spatial_ref_sys
- if same srtext is found but with different srids, update the GeoPackage BLOBs in the table that is copied into the main db to use correct code

In the sample spearfish GeoPackege srid=auth_srid on every row (and epsg is the only auth in the table) but that will not be the case always. Spatialite BLOBs contain also references to srid code used in the local database so this is not a new issue and perhaps not so big trouble because I do not remember that anybody has complained about is in the Spatialite users forum. Perhaps because spatial_ref_sys table in Spatialite is most often created and populated by a Spatialite SQL function so the srids usually have a perfect match.

Perhaps the GeoPackage spec might have an annex about best practise to add tables into the package from other attached GeoPackeges. I wonder also that if some special SQL functions are needed in any case as listed in Annex D, why not to include ST_AsBinary and let the database engine to resolve the slight but inevidently existing GeoPackage BLOB pecularities? Then the GeoPackage clients would have two alternatives: use their own GeoPackage BLOB parser or use some basic GeoPackage SQLite extension that outputs the same WKB that comes from PostGIS and Spatialite with "select ST_AsBinary(geom)..."

Standards are funny, I never knew before today that there exists these OGC "Well Known Binaries" and ISO "(Not as) Well Known Binaries"

-Jukka Rahkonen-

________________________________________
Lähettäjä: Rahkonen Jukka [jukka.rahkonen@anonymised.com]
Lähetetty: 13. heinäkuuta 2013 12:19
Vastaanottaja: geoserver-devel@lists.sourceforge.net
Aihe: Re: [Geoserver-devel] geopackage community module