[Geoserver-devel] new community module: mbtiles

GeoServer GeoServer Developer
1

Hi everyone,

I would like to add a community module for the mbtiles format in geoserver. We also recently just started an unsupported mbtiles module in geotools, which lies the the foundations of the format. The geoserver package provides the outputformats for geoserver (this is similar to the geopackage modules in geotools and geoserver).

At this moment, a wms getmap outputformat is supported:

https://github.com/geoserver/geoserver/pull/416

This will output any getmap requests to an mbtiles database, splitting it in tiles, basically using exactly the same algorithm as the one for geopackage.

Regards
Niels

2

On Mon, Dec 9, 2013 at 2:35 PM, Niels Charlier <niels@anonymised.com> wrote:

Hi everyone,

I would like to add a community module for the mbtiles format in
geoserver. We also recently just started an unsupported mbtiles module in
geotools, which lies the the foundations of the format. The geoserver
package provides the outputformats for geoserver (this is similar to the
geopackage modules in geotools and geoserver).

At this moment, a wms getmap outputformat is supported:

https://github.com/geoserver/geoserver/pull/416

This will output any getmap requests to an mbtiles database, splitting it
in tiles, basically using exactly the same algorithm as the one for
geopackage.

Hi Niels,
question, give that GetMap can occur in any CRS, size and bbox, how is the
mapping between the request and the tile
structure in the mbtiles setup?
Also, are there overviews in the generated mbtiles output?

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

3

Andrea, mbtiles is always in wgs84 global mercator. The bounds in the request are interpreted according to the CRS specified in the request, but after that the requested CRS is ignored. The bounding box is used, but I think the size is ignored. Justin can probably answer your questions better than me, because he wrote the actual code for the mapping to tiles. I made it generic so it works for mbtiles as well as geopackage. Regards Niels

···

On 09/12/13 14:44, Andrea Aime wrote:

On Mon, Dec 9, 2013 at 2:35 PM, Niels Charlier <niels@anonymised.com> wrote:

Hi everyone,

I would like to add a community module for the mbtiles format in geoserver. We also recently just started an unsupported mbtiles module in geotools, which lies the the foundations of the format. The geoserver package provides the outputformats for geoserver (this is similar to the geopackage modules in geotools and geoserver).

At this moment, a wms getmap outputformat is supported:

https://github.com/geoserver/geoserver/pull/416

This will output any getmap requests to an mbtiles database, splitting it in tiles, basically using exactly the same algorithm as the one for geopackage.

Hi Niels,
question, give that GetMap can occur in any CRS, size and bbox, how is the mapping between the request and the tile
structure in the mbtiles setup?
Also, are there overviews in the generated mbtiles output?Technical Lead

4

On Mon, Dec 9, 2013 at 7:30 AM, Niels Charlier <niels@anonymised.com> wrote:

On 09/12/13 14:44, Andrea Aime wrote:

On Mon, Dec 9, 2013 at 2:35 PM, Niels Charlier <niels@anonymised.com> wrote:

Hi everyone,

I would like to add a community module for the mbtiles format in
geoserver. We also recently just started an unsupported mbtiles module in
geotools, which lies the the foundations of the format. The geoserver
package provides the outputformats for geoserver (this is similar to the
geopackage modules in geotools and geoserver).

At this moment, a wms getmap outputformat is supported:

https://github.com/geoserver/geoserver/pull/416

This will output any getmap requests to an mbtiles database, splitting it
in tiles, basically using exactly the same algorithm as the one for
geopackage.

Hi Niels,
question, give that GetMap can occur in any CRS, size and bbox, how is the
mapping between the request and the tile
structure in the mbtiles setup?
Also, are there overviews in the generated mbtiles output?Technical Lead

Andrea, mbtiles is always in wgs84 global mercator. The bounds in the
request are interpreted according to the CRS specified in the request, but
after that the requested CRS is ignored. The bounding box is used, but I
think the size is ignored. Justin can probably answer your questions better
than me, because he wrote the actual code for the mapping to tiles. I made
it generic so it works for mbtiles as well as geopackage.

Yeah, the mbtiles spec mandates spherical mercator, so any wms request
bounds will have to be reprojected to epsg:3857. To get around the mercator
pole issue from what I have seen typically the bottom and top 5 degrees are
clipped off.. so that may be a heuristic we want to apply to the case where
the user requests "the world" in geographic. Once the bbox is reprojected
it is aligned to the built in google mercator tile matrix provided by gwc
choosing a "best fit".

As for overviews the mbtiles structure has no explicit notion of them but
it is a multi level pyramid of tiles so as you go up zoom levels you get
more detailed tiles, so I guess you could look at that as an implicit
overview. Not sure if that answers the question.

Regards
Niels

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

5

On Mon, Dec 9, 2013 at 5:42 PM, Justin Deoliveira <jdeolive@anonymised.com

wrote:

Yeah, the mbtiles spec mandates spherical mercator, so any wms request
bounds will have to be reprojected to epsg:3857. To get around the mercator
pole issue from what I have seen typically the bottom and top 5 degrees are
clipped off.. so that may be a heuristic we want to apply to the case where
the user requests "the world" in geographic. Once the bbox is reprojected
it is aligned to the built in google mercator tile matrix provided by gwc
choosing a "best fit".

I'm a bit weirded out about the CRS lockin, but I see it's indeed in the
spec. I guess it will have never any use for polar/high latitude mapping...
:-p
Good idea to cut the request area to the -85/85 latitude range (and
eventually throw a meaningful exception is the requested area is fully
outside it?)

As for overviews the mbtiles structure has no explicit notion of them but
it is a multi level pyramid of tiles so as you go up zoom levels you get
more detailed tiles, so I guess you could look at that as an implicit
overview. Not sure if that answers the question.

Ah yes, I really meant, is just one zoom level returned, or do you generate
also the lower zoom levels for faster display in output?

Anyways, +1 on the module, I was just being curious.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

6

On Mon, Dec 9, 2013 at 9:47 AM, Andrea Aime <andrea.aime@anonymised.com>wrote:

On Mon, Dec 9, 2013 at 5:42 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yeah, the mbtiles spec mandates spherical mercator, so any wms request
bounds will have to be reprojected to epsg:3857. To get around the mercator
pole issue from what I have seen typically the bottom and top 5 degrees are
clipped off.. so that may be a heuristic we want to apply to the case where
the user requests "the world" in geographic. Once the bbox is reprojected
it is aligned to the built in google mercator tile matrix provided by gwc
choosing a "best fit".

I'm a bit weirded out about the CRS lockin, but I see it's indeed in the
spec. I guess it will have never any use for polar/high latitude mapping...
:-p
Good idea to cut the request area to the -85/85 latitude range (and
eventually throw a meaningful exception is the requested area is fully
outside it?)

Indeed. This is actually an area where i think geopackage can be useful if
it gets any traction since it doesn't have this limitation.

As for overviews the mbtiles structure has no explicit notion of them but
it is a multi level pyramid of tiles so as you go up zoom levels you get
more detailed tiles, so I guess you could look at that as an implicit
overview. Not sure if that answers the question.

Ah yes, I really meant, is just one zoom level returned, or do you
generate also the lower zoom levels for faster display in output?

Anyways, +1 on the module, I was just being curious.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

7

Hi,

EPSG-registry says that 3857 is valid in World between 85.06°S and 85.06°N
http://www.epsg-registry.org/report.htm?type=selection&entity=urn:ogc:def:crs:EPSG::3857&reportDetail=short&style=urn:uuid:report-style:default-with-code&style_name=OGP%20Default%20With%20Code&title=EPSG:3857

MapProxy extents and breaks MBTiles spec and allows all gridsets, see thread http://lists.osgeo.org/pipermail/mapproxy/2013-November/001796.html

-Jukka Rahkonen-

________________________________
Justin Deoliveira wrote:

On Mon, Dec 9, 2013 at 9:47 AM, Andrea Aime <andrea.aime@anonymised.com<mailto:andrea.aime@anonymised.com>> wrote:
On Mon, Dec 9, 2013 at 5:42 PM, Justin Deoliveira <jdeolive@anonymised.com<mailto:jdeolive@anonymised.com>> wrote:
Yeah, the mbtiles spec mandates spherical mercator, so any wms request bounds will have to be reprojected to epsg:3857. To get around the mercator pole issue from what I have seen typically the bottom and top 5 degrees are clipped off.. so that may be a heuristic we want to apply to the case where the user requests "the world" in geographic. Once the bbox is reprojected it is aligned to the built in google mercator tile matrix provided by gwc choosing a "best fit".

I'm a bit weirded out about the CRS lockin, but I see it's indeed in the spec. I guess it will have never any use for polar/high latitude mapping... :-p
Good idea to cut the request area to the -85/85 latitude range (and eventually throw a meaningful exception is the requested area is fully outside it?)

Indeed. This is actually an area where i think geopackage can be useful if it gets any traction since it doesn't have this limitation.

As for overviews the mbtiles structure has no explicit notion of them but it is a multi level pyramid of tiles so as you go up zoom levels you get more detailed tiles, so I guess you could look at that as an implicit overview. Not sure if that answers the question.

Ah yes, I really meant, is just one zoom level returned, or do you generate also the lower zoom levels for faster display in output?

Anyways, +1 on the module, I was just being curious.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313<tel:%2B39%200584%20962313>
fax: +39 0584 1660272<tel:%2B39%200584%201660272>
mob: +39 339 8844549<tel:%2B39%20%C2%A0339%208844549>

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com<mailto:jdeolive@anonymised.com>
@j_deolive<https://twitter.com/j_deolive&gt;

8

On Mon, Dec 9, 2013 at 6:08 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com>wrote:

Hi,

EPSG-registry says that 3857 is valid in World between 85.06°S and 85.06°N

http://www.epsg-registry.org/report.htm?type=selection&entity=urn:ogc:def:crs:EPSG::3857&reportDetail=short&style=urn:uuid:report-style:default-with-code&style_name=OGP%20Default%20With%20Code&title=EPSG:3857

MapProxy extents and breaks MBTiles spec and allows all gridsets, see
thread http://lists.osgeo.org/pipermail/mapproxy/2013-November/001796.html

I guess that's fine because MapProxy knows which SRS each sqlite file is in.
But in an output format, where would you register that info so that the
information is mantained after the file is registered on the client?
I guess some new tables would be needed?

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

9

+1 on the module here!

···

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Alessio Fabiani
@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

On Mon, Dec 9, 2013 at 6:16 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Dec 9, 2013 at 6:08 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com> wrote:

Sponsored by Intel(R) XDK
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Hi,

EPSG-registry says that 3857 is valid in World between 85.06°S and 85.06°N
http://www.epsg-registry.org/report.htm?type=selection&entity=urn:ogc:def:crs:EPSG::3857&reportDetail=short&style=urn:uuid:report-style:default-with-code&style_name=OGP%20Default%20With%20Code&title=EPSG:3857

MapProxy extents and breaks MBTiles spec and allows all gridsets, see thread http://lists.osgeo.org/pipermail/mapproxy/2013-November/001796.html

I guess that’s fine because MapProxy knows which SRS each sqlite file is in.
But in an output format, where would you register that info so that the information is mantained after the file is registered on the client?
I guess some new tables would be needed?

Cheers

Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

10

On Mon, Dec 9, 2013 at 10:16 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Mon, Dec 9, 2013 at 6:08 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com>wrote:

Hi,

EPSG-registry says that 3857 is valid in World between 85.06°S and 85.06°N

http://www.epsg-registry.org/report.htm?type=selection&entity=urn:ogc:def:crs:EPSG::3857&reportDetail=short&style=urn:uuid:report-style:default-with-code&style_name=OGP%20Default%20With%20Code&title=EPSG:3857

MapProxy extents and breaks MBTiles spec and allows all gridsets, see
thread
http://lists.osgeo.org/pipermail/mapproxy/2013-November/001796.html

I guess that's fine because MapProxy knows which SRS each sqlite file is
in.
But in an output format, where would you register that info so that the
information is mantained after the file is registered on the client?
I guess some new tables would be needed?

There is actually already a metadata table in MBTiles that is just a set of
key/value pairings. So if we were to extend the spec i imagine stuff like
srs would go there. Regardless our plan is to stick with strictly what is
supported by the spec but if there is overwhelming feedback that additional
projections are wanted it should be fairly straight forward to accommodate.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

------------------------------------------------------------------------------
Sponsored by Intel(R) XDK
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!

http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

11

Justin Deoliveira wrote:

There is actually already a metadata table in MBTiles that is just a set of key/value pairings. So if we were to extend the spec i imagine stuff like srs would go there. Regardless our plan is to stick with strictly what is supported by the spec but if there is overwhelming feedback that additional projections are wanted it should be fairly straight forward to accommodate.

Hi,

Applications made by/for Finnish administration do not use spherical mercator tiles so we do not have much use for MBTiles. However, storing and delivering tiles in some kind of file database feels interesting. What would you store as key/value pair for making the client to understand the tiling schema well enough? Could it be something like the whole <gridSet> of GeoWebCache config file as in http://geowebcache.org/docs/current/configuration/layers/projections.html or what?

Some other name than MBTiles (MapBox tiles?) would be needed for the derivative. Could it be DBTiles? Or GTTiles or GSTiles like in DataBase, GeoTools or GeoServer, respectively?

-Jukka Rahkonen-

12

On Mon, Dec 9, 2013 at 2:34 PM, Rahkonen Jukka <jukka.rahkonen@anonymised.com>wrote:

Justin Deoliveira wrote:

> There is actually already a metadata table in MBTiles that is just a set
of key/value pairings. So if we were to extend the spec i imagine stuff
like srs would go there. Regardless our plan is to stick with strictly what
is supported by the spec but if there is overwhelming feedback that
additional projections are wanted it should be fairly straight forward to
accommodate.

Hi,

Applications made by/for Finnish administration do not use spherical
mercator tiles so we do not have much use for MBTiles. However, storing and
delivering tiles in some kind of file database feels interesting. What
would you store as key/value pair for making the client to understand the
tiling schema well enough? Could it be something like the whole <gridSet>
of GeoWebCache config file as in
http://geowebcache.org/docs/current/configuration/layers/projections.htmlor what?

Some other name than MBTiles (MapBox tiles?) would be needed for the
derivative. Could it be DBTiles? Or GTTiles or GSTiles like in DataBase,
GeoTools or GeoServer, respectively?

Well I believe this is what the tile portion of the GeoPackage spec
addresses. It has no restriction on projection and as you state it has a
metadata table that describes the tile matrix structure. So from our
perspective it seems like a pretty good fit for folks who need more than
spherical mercator.

-Jukka Rahkonen-

------------------------------------------------------------------------------
Sponsored by Intel(R) XDK
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!

http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;