[Geoserver-devel] Time WMS "magic" and caching sometimes at odds, new config to handle the situation

Hi all,
WMS time handling (and in general, dimension handling) can have a fair bit of “magic” going on:

  • When no time is provided, a default time value is set (but the actual value can change over time)
  • When time is provided but it’s not a match, a nearest match behavior can be configured, returning a different time
  • When nearest time is enabled, a max distance can be configured, which might make the lookup fail and return an empty tile regardless
    Abiding to the spec, GeoServer sets a HTTP header, a WARNING, in responses talking about the actual time being used (or the lack of match).
    Which is great for strictly compliant WMS servers that bothered to handle this aspect for the spec… but I don’t know of any that actually do LOL.

If the dataset is strictly static it’s maybe not a big of a deal, as the response to a given URL is going to be mostly static (there might be redundant caching).
But if the time based dataset is dynamic, with continuous ingestion of new data, then we have a problem.

Generally speaking, this is problematic for web clients and anything in between (proxies, edge servers) that might decide to cache the response
in front of the eventual WMS/WMTS (in other words, the issue goes beyond what we might control at the integrated GWC level.

In these cases, the important bit is to give whatever is in front the information about whether the tile is not cacheable or not,
while maybe avoid trying to cache a blank tile locally, or a tile with the wrong content (in dynamic situations, a possible missing time
in a time series could also be filled later).

However… someone could be happy with the status-quo, especially when dealing with static datasets (historical ones, not active).

So… we think a good way to handle this is to add the following settings to caching defaults, and per layer settings, allowing
to disable caching for situations that would cause warnings, letting the admin decide what to do:
image.png
In addition to not caching the tiles, the response headers would have a clear indication that the response is not meant to
be cached, in other words:

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

This should hopefully work across the board.

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Andrea,

Hi all,
WMS time handling (and in general, dimension handling) can have a fair bit of “magic” going on:

  • When no time is provided, a default time value is set (but the actual value can change over time)
  • When time is provided but it’s not a match, a nearest match behavior can be configured, returning a different time
  • When nearest time is enabled, a max distance can be configured, which might make the lookup fail and return an empty tile regardless
    Abiding to the spec, GeoServer sets a HTTP header, a WARNING, in responses talking about the actual time being used (or the lack of match).
    Which is great for strictly compliant WMS servers that bothered to handle this aspect for the spec… but I don’t know of any that actually do LOL.

If the dataset is strictly static it’s maybe not a big of a deal, as the response to a given URL is going to be mostly static (there might be redundant caching).
But if the time based dataset is dynamic, with continuous ingestion of new data, then we have a problem.

Generally speaking, this is problematic for web clients and anything in between (proxies, edge servers) that might decide to cache the response
in front of the eventual WMS/WMTS (in other words, the issue goes beyond what we might control at the integrated GWC level.

In these cases, the important bit is to give whatever is in front the information about whether the tile is not cacheable or not,
while maybe avoid trying to cache a blank tile locally, or a tile with the wrong content (in dynamic situations, a possible missing time
in a time series could also be filled later).

However… someone could be happy with the status-quo, especially when dealing with static datasets (historical ones, not active).

So… we think a good way to handle this is to add the following settings to caching defaults, and per layer settings, allowing
to disable caching for situations that would cause warnings, letting the admin decide what to do:
image.png
In addition to not caching the tiles, the response headers would have a clear indication that the response is not meant to
be cached, in other words:

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

This should hopefully work across the board.

Makes sense to me and it sounds like a good improvement.

Couple of question:

  • so the above options are multiple selectable values? (no radio-button, right?)
  • In the specific example above, caching will be disabled if no nearest match has been found but it will be set if a default value has been used, right?

Regards,
Daniele

···

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Couple of question:

  • so the above options are multiple selectable values? (no radio-button, right?)

Yes, checkboxes, so independent, one can choose the mix of caching they want.
For a fully static set, I guess they might want to allow caching on the default value for example, while disallowing cache creation
on nearest matches and failed matches. Others might decide that disk quota is good enough to manage all that and cache everything,
sparing the database extra queries.
In general I want to leave decisions up to the admin, they know their use case better than I do.

  • In the specific example above, caching will be disabled if no nearest match has been found but it will be set if a default value has been used, right?

Correct

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Andrea,
looks good to me, +1

···

Regards,
Nuno Oliveira

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


Con riferimento alla normativa sul trattamento dei dati
personali (Reg. UE 2016/679 - Regolamento generale sulla
protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo contenuto,
gli eventuali allegati, etc.) è un dato la cui conoscenza
è riservata al/i solo/i destinatario/i indicati dallo
scrivente. Se il messaggio Le è giunto per errore, è
tenuta/o a cancellarlo, ogni altra operazione è illecita.
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to
which it is addressed and may contain information that
is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European
Regulation 2016/679 “GDPR” - copying, dissemination or
use of this e-mail or the information herein by anyone
other than the intended recipient is prohibited. If you
have received this email by mistake, please notify
us immediately by telephone or e-mail.

Sorry I did not have anything to add earlier.

To expand on the meeting discussion, the UI could be presented in terms of desired effect.

(attachments)

image.png

···


Jody Garnett

Hi Jody,
that is pretty good for documentation, but it presumes we can enumerate all possible cases
in advance. I don’t feel like I have a powerful enough crystal ball.

Cheers
Andrea

(attachments)

image.png

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.