[Geoserver-devel] Proposing an improvement for the monitoring module

GeoServer GeoServer Developer
1

Hi,
we (GeoSolutions) would like to propose (and code) an improvement for
the monitoring
code.

We want to integrate monitoring in a setup where there already is a centralized
auditing system (that's the proper name for request by request logging)
that expects to be fed with XML files in a specific format.

What we'd like to add is an asynchronous logger that would play in
parallel with the in-memory DAO and would log all requests on
files. The files would be rolled after N requests and every 24 hours
(first of the two limits that is hit).
We don't want to have a specific xml format, so we were thinking of using
a configurable freemarker template that would be used to save the
requests (so if someone wants to do a non xml format, maybe similar to
the apache one, they are free to do so).

This new thing would not be a DAO per se, but something that would
work in parallel to it, especially if one considers the in memory dao.
A straightforward way would be modify the memory dao to talk
to this new auditor block, but I guess it might not be well received.
What about a pluggable extension point for Monitor instead, a RequestListener?

public void requestListener() {
   void requestStarted(RequestData rd);
   void requestUpdated(RequestData rd);
   void requestSaved(RequestData rd);
}

The auditor would find its config properties from the property file
config as well,
something like:

audit=true
template=audit_template.ftl
extension=xml
path=/var/logs/gs_audit
roll_limit=1000

Wondering, can we have the MonitorConfig expose a property getter like:

<T> T getProperty(String name, Class<T> target)

so that eventual additions to the monitoring (thinking also post processors
here) can grab stuff from there in a type safe way?

Cheers
Andrea

--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

-------------------------------------------------------

2

Sounds interesting. At the moment that module is sort of limbo so definitely +1 on any development. Comments inline.

On Mon, Aug 8, 2011 at 7:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
we (GeoSolutions) would like to propose (and code) an improvement for
the monitoring
code.

We want to integrate monitoring in a setup where there already is a centralized
auditing system (that’s the proper name for request by request logging)
that expects to be fed with XML files in a specific format.

What we’d like to add is an asynchronous logger that would play in
parallel with the in-memory DAO and would log all requests on
files. The files would be rolled after N requests and every 24 hours
(first of the two limits that is hit).
We don’t want to have a specific xml format, so we were thinking of using
a configurable freemarker template that would be used to save the
requests (so if someone wants to do a non xml format, maybe similar to
the apache one, they are free to do so).

This new thing would not be a DAO per se, but something that would
work in parallel to it, especially if one considers the in memory dao.
A straightforward way would be modify the memory dao to talk
to this new auditor block, but I guess it might not be well received.
What about a pluggable extension point for Monitor instead, a RequestListener?

public void requestListener() {
void requestStarted(RequestData rd);
void requestUpdated(RequestData rd);
void requestSaved(RequestData rd);
}

Listener interface makes a lot of sense. +1.

The auditor would find its config properties from the property file
config as well,
something like:

audit=true
template=audit_template.ftl
extension=xml
path=/var/logs/gs_audit
roll_limit=1000

Wondering, can we have the MonitorConfig expose a property getter like:

T getProperty(String name, Class target)

so that eventual additions to the monitoring (thinking also post processors
here) can grab stuff from there in a type safe way?

+1, I like that approach… saves extensions work for having to persist there properties. It might be nice to have a version for extensions that is like:

T getProperty(String prefix, String name, Class target)

That would make it more explicit for extensions, so that they have to prefix properties in the file… I guess just a single method and the convention that extensions use “dot notation” works just as well too.

2c.

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
The must-attend event for mobile developers. Connect with experts.
Get tools for creating Super Apps. See the latest technologies.
Sessions, hands-on labs, demos & much more. Register early & save!
http://p.sf.net/sfu/rim-blackberry-1

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

3

On Mon, Aug 8, 2011 at 4:13 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Sounds interesting. At the moment that module is sort of limbo so definitely
+1 on any development. Comments inline.

Hi,
just attached a patch for the work at: http://jira.codehaus.org/browse/GEOS-4712

The patch adds the listeners support, and in particular the allows for
a freemarker
template with default xml dumping configured in the classpath with a header,
content and footer template.

It also adds a couple of bits to the request data to make it more
Freemarker friendly,
and also allows to configure the maximum size of the response body stored in
RequestData (-1 for no limit), and the max size has been updated to 8KB
(otherwise not even the sample requests in demo would be logged fully).

You can test it adding the following to your monitoring.properties file:

audit.enabled=true
audit.path=/tmp
audit.roll_limit=20

The default templates will generate a file like the one attached, which
shows a mix of get and post requests over various protocols.

What do you think? Ok to commit?

Cheers
Andrea

--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

-------------------------------------------------------

(attachments)

geoserver_audit_20110811_1.log (23 KB)

4

Good stuff, just made some comments on the issue.

On Thu, Aug 11, 2011 at 12:17 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Aug 8, 2011 at 4:13 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Sounds interesting. At the moment that module is sort of limbo so definitely
+1 on any development. Comments inline.

Hi,
just attached a patch for the work at: http://jira.codehaus.org/browse/GEOS-4712

The patch adds the listeners support, and in particular the allows for
a freemarker
template with default xml dumping configured in the classpath with a header,
content and footer template.

It also adds a couple of bits to the request data to make it more
Freemarker friendly,
and also allows to configure the maximum size of the response body stored in
RequestData (-1 for no limit), and the max size has been updated to 8KB
(otherwise not even the sample requests in demo would be logged fully).

You can test it adding the following to your monitoring.properties file:

audit.enabled=true
audit.path=/tmp
audit.roll_limit=20

The default templates will generate a file like the one attached, which
shows a mix of get and post requests over various protocols.

What do you think? Ok to commit?

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

On Thu, Aug 11, 2011 at 11:42 PM, Justin Deoliveira
<jdeolive@anonymised.com> wrote:

Good stuff, just made some comments on the issue.

Hi,
committed the changes on trunk along with some docs.

I wanted to backport to 2.1.x but found that the monitoring over there is rather
different... ouch.... especially since that's where I need the patch to work.

Would it be possible to overwrite the 2.1.x monitoring module with the trunk
contents so that I can have the patch working over there too?
As far as I can see the version on trunk has better configuration, more
per request information, and pretty much the same level of testing.

Otherwise I guess I'll have to rewrite the patch for 2.1.x, ugh

Cheers
Andrea

--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

-------------------------------------------------------

6

No objection here… the monitoring module on 2.1.x was stable because of all the development that went on the trunk. I planned to backport after the development slowed down and the module became stable (actually i hoped to backport after promoting it to extension) but never had the time. So yeah +1.

On Fri, Aug 12, 2011 at 11:53 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Thu, Aug 11, 2011 at 11:42 PM, Justin Deoliveira
<jdeolive@anonymised.com> wrote:

Good stuff, just made some comments on the issue.

Hi,
committed the changes on trunk along with some docs.

I wanted to backport to 2.1.x but found that the monitoring over there is rather
different… ouch… especially since that’s where I need the patch to work.

Would it be possible to overwrite the 2.1.x monitoring module with the trunk
contents so that I can have the patch working over there too?
As far as I can see the version on trunk has better configuration, more
per request information, and pretty much the same level of testing.

Otherwise I guess I’ll have to rewrite the patch for 2.1.x, ugh

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.