[Geoserver-devel] Flipping the relationship between GeoServer and GeoWebCache upside down

GeoServer GeoServer Developer
1

Hi,
I’m fiddling with a case in which it appears it would make sense to reverse the relationship
between GWC and GS.

Currently we have a WMS layer, that the integrated GWC treats as the source, and can tile cache
in different projections/formats.

But what if… we had the source as a GWC TileLayer instead? Say, a ArcGIS or TMS on disk cache?
Well, we could then create a reader than mosaics and rescales the tiles to run the WMS… but what
about the tile layer that GWC automatically creates out of it? It would be redundant, and it would
double cache the tiles.
So we don’t want it… or do we? What if we want to provide as GWC tiles some projection/format/grid set
that the original tile layer does not support?
It would seem we would want it, but transparently fall back on the original GWC tile layer in case
the projection/format/gridset matches the original tile layer.
And what if the tile layer supports a gridset that we don’t want to expose, but one that we would like
to cache locally, because maybe we are talking to a remote service, like, say we are cascading another
WMTS service, and we don’t want to repeat the requests over and over?
In this case things would also be interesteing in that we should try to play browser, and consider
the caching headers (time to live, etag, and so on).

Any thought on how to integrate things in a clear, linear fashion? :slight_smile:

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Hi,
adding another take on this. Let's say that instead of a GWC TileLayer we
have GeoTools
tile client work as the source https://github.com/geotools/geotools/pull/777
(basically any tile layer could be reworked to have a tile client as its
source).
And let's also say, we create a grid reader wrapping the tile client,
making this a natural
data source for GeoServer, which can then be tile-cached.

Now, there are hiccups with this approach too:
a) In case GWC is caching tiles in the same format and gridset as the
origin, we would have
  an un-necessary overhead
b) The tile client now only has a cache in memory
c) There could be a mismatch between the format/gridset served by GWC, and
the one
    of the origin tile service

We could address a) by adding the tile client as a reader metadata, the GWC
integration
could recognize it, and fetch te tiles directly from the tile client in
that case... simple optimization.

Handling b) could be done by making the tile client cache pluggable, and
making it collaborate
with the GWC one (if the tiles we are searching for are already there, we
use them, if
not, we fetch and add them. And we would have disk quota handling, pretty
much mandatory if what we
are trying to cascade and cache is a large remote tile cache (e.g., OSM,
Google, Bing)
There is a hiccup here too though, our cache should somehow behave like a
browser,
if the tile from the remote service has HTTP headers/etag, we should
somehow handle them too,
which would point to a different tile caching strategy than the GWC one (or
else, GWC is often
working off WMS servers which can also set the same headers, so looking at
this from another
angle, we could also improve GWC own cache handling... but it would require
a db to store
the extra tile info, its time to leave, etag, and so on).

The c) case is a pain, as we want to expose one tile cache out of GWC, but
also cache a "private"
one, the origin tiles, that we would not show, possibly in the same tile
layer.
Or just give up trying to cache the origin tiles, and do direct requests
every time we need
to produce WMS results or generate a new output tile in the different
gridset.

As a last bit, it could be interesting to extract the cache tiling/disk
quota subsystem from GWC,
and make it into a separate, stand alone library that can be used both by
the GeoTools tile client
module, and GWC.

Many of these ideas are quite a bit of work, so there is a chance none will
get sponsoring
any time soon, but coming up with an ideal plan is still a good exercise
for the future.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

3

I like the way this is going :slight_smile:

I haven’t really looked at the GWC code base, how much work would it be to pull this code out? are there license issues if we pull it down to GeoTools?

Ian

···

Hi,
adding another take on this. Let’s say that instead of a GWC TileLayer we have GeoTools
tile client work as the source https://github.com/geotools/geotools/pull/777
(basically any tile layer could be reworked to have a tile client as its source).
And let’s also say, we create a grid reader wrapping the tile client, making this a natural
data source for GeoServer, which can then be tile-cached.

As a last bit, it could be interesting to extract the cache tiling/disk quota subsystem from GWC,
and make it into a separate, stand alone library that can be used both by the GeoTools tile client
module, and GWC.

4

On Wed, Mar 25, 2015 at 12:13 PM, Ian Turton <ijturton@anonymised.com> wrote:

As a last bit, it could be interesting to extract the cache tiling/disk

quota subsystem from GWC,
and make it into a separate, stand alone library that can be used both by
the GeoTools tile client
module, and GWC.

I haven't really looked at the GWC code base, how much work would it be to
pull this code out? are there license issues if we pull it down to GeoTools?

I'd say, significant enough effort.
Licensing wise, not sure, we might have copyright assignment issues maybe,
the license is the same as GeoTools

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

On 25 March 2015 at 04:59, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, Mar 25, 2015 at 12:13 PM, Ian Turton <ijturton@anonymised.com> wrote:

As a last bit, it could be interesting to extract the cache tiling/disk

quota subsystem from GWC,
and make it into a separate, stand alone library that can be used both
by the GeoTools tile client
module, and GWC.

I haven't really looked at the GWC code base, how much work would it be
to pull this code out? are there license issues if we pull it down to
GeoTools?

I'd say, significant enough effort.
Licensing wise, not sure, we might have copyright assignment issues maybe,
the license is the same as GeoTools

I think the license versions are different: LGPL 2.1 for GT, LGPL 3.0 for
GWC. It looks like all commits in the disk quota module were by OpenPlans,
Boundless, or GeoSolutions except one:
https://github.com/GeoWebCache/geowebcache/commit/0cd9a799f77dbe60e90bbbcded23f63b2062afd1

--

Kevin Smith

Software Engineer | Boundless <http://boundlessgeo.com/&gt;

ksmith@anonymised.com

+1-778-785-7459

@boundlessgeo <http://twitter.com/boundlessgeo/&gt;

<http://twitter.com/boundlessgeo/&gt;

[image: http://boundlessgeo.com/\]
<http://boundlessgeo.com/&gt;

6

On Wed, Mar 25, 2015 at 6:26 PM, Kevin Smith <ksmith@anonymised.com>
wrote:

Licensing wise, not sure, we might have copyright assignment issues maybe,

the license is the same as GeoTools

I think the license versions are different: LGPL 2.1 for GT, LGPL 3.0 for
GWC. It looks like all commits in the disk quota module were by OpenPlans,
Boundless, or GeoSolutions except one:
https://github.com/GeoWebCache/geowebcache/commit/0cd9a799f77dbe60e90bbbcded23f63b2062afd1

Oh interesting, I did not know. Hmm... openplans is dead now, not sure how
we would handle
a licence change in case we would like to go this way...

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

7

On 25 March 2015 at 10:34, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, Mar 25, 2015 at 6:26 PM, Kevin Smith <ksmith@anonymised.com>
wrote:

Licensing wise, not sure, we might have copyright assignment issues

maybe, the license is the same as GeoTools

I think the license versions are different: LGPL 2.1 for GT, LGPL 3.0
for GWC. It looks like all commits in the disk quota module were by
OpenPlans, Boundless, or GeoSolutions except one:
https://github.com/GeoWebCache/geowebcache/commit/0cd9a799f77dbe60e90bbbcded23f63b2062afd1

Oh interesting, I did not know. Hmm... openplans is dead now, not sure how
we would handle
a licence change in case we would like to go this way...

OpenPlans still exists, they just aren't interested in GeoServer or
GeoWebCache development any more. They included the copyrights they held on
GWC when they transferred GeoServer to OSGeo. As usual, GWC was a bit of
an afterthought that didn't get much attention. I still haven't had time
to update the copyright headers.

So it would appear we'd need OSGeo, Boundless, GeoSolutions, and
François-Régis Deboffles to agree if we wanted to make that move.

--

Kevin Smith

Software Engineer | Boundless <http://boundlessgeo.com/&gt;

ksmith@anonymised.com

+1-778-785-7459

@boundlessgeo <http://twitter.com/boundlessgeo/&gt;

<http://twitter.com/boundlessgeo/&gt;

[image: http://boundlessgeo.com/\]
<http://boundlessgeo.com/&gt;

8

I would love to dig up the tile client code for geotools, and teach it about different caches on disk. We could simplify the internals of geowebcache in the process :slight_smile:

···

On 24 March 2015 at 11:58, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
I’m fiddling with a case in which it appears it would make sense to reverse the relationship
between GWC and GS.

Currently we have a WMS layer, that the integrated GWC treats as the source, and can tile cache
in different projections/formats.

But what if… we had the source as a GWC TileLayer instead? Say, a ArcGIS or TMS on disk cache?
Well, we could then create a reader than mosaics and rescales the tiles to run the WMS… but what
about the tile layer that GWC automatically creates out of it? It would be redundant, and it would
double cache the tiles.
So we don’t want it… or do we? What if we want to provide as GWC tiles some projection/format/grid set
that the original tile layer does not support?
It would seem we would want it, but transparently fall back on the original GWC tile layer in case
the projection/format/gridset matches the original tile layer.
And what if the tile layer supports a gridset that we don’t want to expose, but one that we would like
to cache locally, because maybe we are talking to a remote service, like, say we are cascading another
WMTS service, and we don’t want to repeat the requests over and over?
In this case things would also be interesteing in that we should try to play browser, and consider
the caching headers (time to live, etag, and so on).

Any thought on how to integrate things in a clear, linear fashion? :slight_smile:

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

9

On Thu, Mar 26, 2015 at 3:16 AM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

I would love to dig up the tile client code for geotools, and teach it
about different caches on disk. We could simplify the internals of
geowebcache in the process :slight_smile:

Yep, that seems to be the saner solution... but a too expensive one for the
case I'm looking at.
We'll try to keep the door open for such future evolution though

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------