Yeah, I'd like to eventually get at full on client side listeners to wfs
updates. But what I'm actually talking about here is much simpler (I
think). If I'm understanding you right, you set a value in the
featureType that has the number of seconds that the cache will be good
for. Correct me if I'm wrong, but presumably on a set of features that
you know wouldn't be updated at all, or would very rarely, you'd set a
very high time. On one with a lots of updates, you wouldn't set it at
all, or maybe you would for a few seconds. What I'm suggesting is you
make one value called 'automatic'. And it resets the cache whenever
there's a transaction request on the geoserver layer. All on the internal
side, and thus pretty darn easy to code. And it still uses your same lazy
mechanism, of resetting the cache, it just does so when there's an update,
instead of an arbitrary number of seconds. Does that make sense? Like
it's a lot less than the full on thing, it's just having the point at
which you do the reset dependent on a transaction. Though I could be
understanding wrong the cache time thing, or maybe it's a thing that
geoserver doesn't have enough control of, telling the client when it
should update instead of use the cache.
Also, it'd be cool to try to contribute your fix to ka-map, maybe once you
have a public demo to show them, so it could work with wms/squid by
default, instead of requiring the hack.
If you're going full ja-map, you may also consider mapbuilder, adding the
functionality there, since we already include it in geoserver, and I think
it has a broader community and is more standards oriented. You can
already do things like WFS-T, for example. But ka-map obviously has some
advantages as well, it just may be worth looking out, depending on how
much needs to change anyways for the port. But a super nice
listening/caching client on J2ee would be nice indeed, especially with the
coming GeoServer 2.0 more modular plug-in architecture.
Chris
On Thu, 17 Nov 2005, Saul Farber wrote:
Chris,
Glad to hear the effort is interesting to some other folks!
As for handling cache-dumping/spatial-feature-caching, etc...that's a
very very interesting question.
First off, let me draw a line between what I've done so far, and what I
understand you to be talking about in terms of more advanced caching.
What I've done so far is borderline trivial. Basically it's a GUI to
control the addition/calculation of a single extra header on the HTTP
response. If that header is present, it instructs other software
(squid, particularly--and I'm a squid super-newbie) to cache getmaps
fetched via KVP/GET (not KVP/POST and not POST/XML). Configuring the
other software to properly cache stuff is actually pretty hard, and the
world of internet caching is extremely advanced. Basically, I've set up
squid, and made a tiny change to geoserver to make squid do it's
squiddy-magic with getmaps.
A much much larger and potentially more interesting question is
something along the lines of "how can we do ka-map style maps
super-efficiently, using all the knowledge that we have about GIS and
WMS to optimize our solution?"
To be honest, I think this is a separate application. Something like a
specialized implementation of squid which is a client to the main
geoserver WMS, and could even be the place where the javascript for the
ka-map port lives.
By the way, your intuition about ka-map was correct. Ka-map is usually
configured by a php script that uses PHP-mapscript to generate map
tiles. But I just changed the format of the request and the php script
to refer to a WMS instead....minimal changes, really. Doing the same
thing with a J2EE based backend (servlet + kamap-configuration-gui +
ka-map source + caching logic) would be a cool port of ka-map to
geoserver. Hmm...I'll start thinking about that...
Maybe the ka-map j2ee port (ja-map?) could be a "listener" for WFS
events involving data updates, and could dump the right portions of its
cache at the right times, as you mentioned.
--saul
On Thu, 2005-11-17 at 09:43 -0500, Chris Holmes wrote:
> Thinking about things a bit more, have you given any thought to tracking
> the transactions against each layer as the sign to kill the cache, instead
> of using a user defined seconds thing? Right now it would have to be sort
> of hacked in, but it wouldn't be too bad a hack. In the future for a
> versioning WFS we're thinking of making a pluggable processing chain, so
> you can define validations, peer reviews, email updates, versioning
> tables, ect., and ideally this could slot right in - notify cache. But I
> think you could just add a bit of code in the current Transaction Response
> (which may be the current candidate for most in need of a refactoring), to
> reset the cache if a transaction on that layer came in. Eventually we
> could maybe even do complex checking of bounds so as to only update the
> cache on areas that were affected by the transaction, but that's probably
> a ways away.
>
> Also, have you given any thought of how mapbuilder could support this? Or
> have you tried with ka-map? It might be nice to see if we could have an
> embedded demo that shows this off in the initial start-up, but I don't
> think we really want to start including squid. I was also under the
> impression that ka-map relied on mapserver specific ways of doing things?
> I would love to be able to see ka-map as a geoserver client.
>
> This also starts to bring up the issue of the featureType editor page
> getting too complicated. This alone won't do it, but there are a number
> of things that could be quite nice at the featureType level, that we just
> set globally right now, citeConformanceHacks, numDecimals, maxFeatures,
> ect. And some, like this, are sort of WMS specific, and some, like
> numDecimals, are sort of WFS specific. Do we have separate pages for
> each? Or perhaps an 'advanced' page?
>
> I'm thinking about this in the context of GeoServer 2.0 thoughts, where we
> want to allow plug-ins, such as WCS, be able to set controls, and they'll
> want to do some of that at the featureType level. An alternative thought
> on doing this could be to flip it around, and have a page for the config
> option, and check boxes, controls for each featureType. So in like the
> WFS config, you'd have 'content', 'description', and 'featureTypes', and
> the 'featureTypes' would list _all_ your featureTypes, and each would have
> the option to check it. Also thinking about this in the context of
> security and user level permissions, at the featureType level.
>
> You're really pushing the bleeding edge of GeoServer Saul, it's great to
> see! We'll do our best to catch up quickly and support your efforts.
>
> Chris
>
> On Wed, 16 Nov 2005, Saul Farber wrote:
>
> > Efren,
> >
> > This is what this patch does (more or less).
> >
> > The WMSCache portion of your diagram is played by software like
> > "squid" (www.squid-cache.org) or other caching proxy/http accelerator.
> >
> > This patch does not have the WMS tier do any actual caching...it's
> > really a very basic header-manipulator. Any request sent DIRECTLY to
> > the tomcat port will be fully rendered exactly as before. It's only for
> > those requesting content via a proxy (generally an intention of the
> > setup architecture) that this would actually do any caching.
> >
> > --saul
> >
> >
> > On Wed, 2005-11-16 at 11:22 -0800, Efren Serra wrote:
> > > Folks,
> > >
> > > I was wondering about caching on the client side instead like is touched on
> > > the following URL: http://www.codeproject.com/csharp/WMSOverviewAll.asp
> > >
> > > I think caching at the WMS tier would tend to convolute the WMS source code
> > > plus complicate the testsuite, etc. Also, if we really wanted to "cache,"
> > > then we should follow the model that CPU's follow (e.g., CPU <--> Cache <-->
> > > Main Memory <--> Disks, etc.) but in our domain:
> > >
> > > Client <--> WMSCache <--> WMS <--> RDBMS, where all WMSCache does is index
> > > GetMap requests to Map's; if there is a cache hit, return the map;
> > > otherwise, forward the request to WMS, cache it, and return it to the
> > > client.
> > >
> > > This would be easily implemented in the Tomcat container by just making sure
> > > that all GetMap requests get sent to the WMSCache Servlet, for instance.
> > >
> > > v/r,
> > > Efren
> > >
> > > -----Original Message-----
> > > From: geoserver-devel-admin@lists.sourceforge.net
> > > [mailto:geoserver-devel-admin@lists.sourceforge.net]On Behalf Of Saul
> > > Farber (JIRA)
> > > Sent: Tuesday, November 15, 2005 1:46 PM
> > > To: geoserver-devel@lists.sourceforge.net
> > > Subject: [Geoserver-devel] [jira] Created: (GEOS-454) Cache-Control
> > > based WMS map caching
> > >
> > >
> > > Cache-Control based WMS map caching
> > > -----------------------------------
> > >
> > > Key: GEOS-454
> > > URL: http://jira.codehaus.org/browse/GEOS-454
> > > Project: GeoServer
> > > Type: New Feature
> > > Components: WMS, Struts, DTO
> > > Environment: All
> > > Reporter: Saul Farber
> > > Assigned to: dblasby
> > > Attachments: Cache-Control.patch
> > >
> > > From looking at http://www.mnot.net/cache_docs/, and based on ideas at
> > > http://sourceforge.net/mailarchive/message.php?msg_id=13467125, I've
> > > implemented a caching-header management system for geoserver.
> > >
> > > Each FeatureType can optionally enable caching headers for maps created from
> > > that FeatureType as well as specifing a number of seconds that such maps
> > > should remain cached.
> > >
> > > On the "Edit FeatureType" page, this patch adds two new fields:
> > > * a checkbox which enables/disables the "Cache-Control: max-age" header
> > > * a text field which allows a user to specify the number of seconds which
> > > will be placed in the "Cache-Control: max-age" header
> > >
> > > For maps which are made up of multiple layers, the lowest "max-age" is used,
> > > and if any layers in a multi-layer map are not cacheable, the entire map is
> > > not cacheable.
> > >
> > > *** SOME IMPORTANT CLARIFICATIONS ***
> > > Caching is a tricky and sometimes sensitive topic, and I want to add some
> > > clarifications about what this patch does and doesn't do.
> > > This patch:
> > > DOES NOT cause geoserver to store images after they've been created
> > > DOES NOT make maps generated by geoserver possibly be out of date
> > > DOES NOT speed up or slow down geoserver in any meaningful way (I guess it
> > > might slow it down by a couple of microseconds as it calculates a tiny bit
> > > of information per-layer)
> > >
> > > This patch:
> > > DOES use well-known, well-defined standards (see
> > > http://www.mnot.net/cache_docs/) to instruct an interenet-wide
> > > content-distribution network to more efficiently cache maps which change
> > > very infrequently
> > > DOES allow applications like ka-map (http://ka-map.maptools.org) and squid
> > > (www.squid-proxy.net) to make awesome googlemaps-like interfaces possible
> > > with geoserver
> > > DOES absolutely nothing by default. It assumes that all layers are
> > > extremely dynamic, and causes none to have appropriate caching headers added
> > > unless explicitly enabled.
> > >
> > > Any comments or suggestions would be very welcome! The outgrowth of this
> > > will be a tutorial explaining how to set up a cluster of geoserver boxes,
> > > interconnected squid proxies and a load balancer to create very fast, very
> > > efficient google-maps-like geoserver-backed web-based mapping applications.
> > >
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > -
> > > If you think it was sent incorrectly contact one of the administrators:
> > > http://jira.codehaus.org/secure/Administrators.jspa
> > > -
> > > For more information on JIRA, see:
> > > http://www.atlassian.com/software/jira
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today
> > > Register for a JBoss Training Course. Free Certification Exam
> > > for All Training Attendees Through End of 2005. For more info visit:
> > > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > > _______________________________________________
> > > Geoserver-devel mailing list
> > > Geoserver-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
> > >
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today
> > > Register for a JBoss Training Course. Free Certification Exam
> > > for All Training Attendees Through End of 2005. For more info visit:
> > > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > > _______________________________________________
> > > Geoserver-devel mailing list
> > > Geoserver-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today
> > Register for a JBoss Training Course. Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > Geoserver-devel mailing list
> > Geoserver-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
> >
>
--
