Hi,
keeping on the WPS hardening topic, another thing that is imho necessary is to exert
some control on how much resources we are eventually reading from the outside,
or from the local filesystem, or from the internal WFS and so on.
Parsing the inputs and turning them into Java object equivalents is a task that is left
to the ProcessParameterIO classes, and is fully managed within this WPS class:
https://github.com/geoserver/geoserver/blob/master/src/extension/wps/wps-core/src/main/java/org/geoserver/wps/executor/SimpleInputProvider.java
I’m thinking about an interface that would allow to control the inputs before they
get parsed (after parsing ProcessFilter is good), wondering if something like this
would be a good fit:
interface WPSInputFilter {
/**
- Allows to alter the input if necessary, or refuse the input
*/
public InputType filter(InputType input, Process process, InputClassifier classifier, Object requestObject) throws WPSException;
/**
- Exert control over the streams, do byte counting
*/
public InputStream filterStream(InputType input, Process process, InputStream stream) throws WPSException;
}
where InputClassifier would be an enumeration:
LITERAL, INLINE_COMPLEX, LOCAL_WFS, LOCAL_WPS, LOCAL_WCS, LOCAL_FILE
and the requestObject owuld be the WFS/WCS/WPS local request in case of local requests
Things I would like to do with this:
- specify a max MB for the download of remote files
- check access to local filesystem (with a different max size, and eventually disallow this altogheter)
- limit the data retrieval time (think of a slow pipe and the wps request stuck there for a long time)
Ideas, opinions?
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
Makes sense and seems pretty simple… don’t have much to add. Have you thought about the output side of things? Or perhaps how to control resources of a process while it is actually executing?
Another question, as these wps extension points accumulate it would be good to document them. Are they currently documented anywhere? It would be nice to have at least an overview doc in the developer guide like we did for security.
On Fri, Aug 10, 2012 at 11:44 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:
Hi,
keeping on the WPS hardening topic, another thing that is imho necessary is to exert
some control on how much resources we are eventually reading from the outside,
or from the local filesystem, or from the internal WFS and so on.
Parsing the inputs and turning them into Java object equivalents is a task that is left
to the ProcessParameterIO classes, and is fully managed within this WPS class:
https://github.com/geoserver/geoserver/blob/master/src/extension/wps/wps-core/src/main/java/org/geoserver/wps/executor/SimpleInputProvider.java
I’m thinking about an interface that would allow to control the inputs before they
get parsed (after parsing ProcessFilter is good), wondering if something like this
would be a good fit:
interface WPSInputFilter {
/**
- Allows to alter the input if necessary, or refuse the input
*/
public InputType filter(InputType input, Process process, InputClassifier classifier, Object requestObject) throws WPSException;
/**
- Exert control over the streams, do byte counting
*/
public InputStream filterStream(InputType input, Process process, InputStream stream) throws WPSException;
}
where InputClassifier would be an enumeration:
LITERAL, INLINE_COMPLEX, LOCAL_WFS, LOCAL_WPS, LOCAL_WCS, LOCAL_FILE
and the requestObject owuld be the WFS/WCS/WPS local request in case of local requests
Things I would like to do with this:
- specify a max MB for the download of remote files
- check access to local filesystem (with a different max size, and eventually disallow this altogheter)
- limit the data retrieval time (think of a slow pipe and the wps request stuck there for a long time)
Ideas, opinions?
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
On Sat, Aug 11, 2012 at 5:12 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:
Makes sense and seems pretty simple… don’t have much to add. Have you thought about the output side of things? Or perhaps how to control resources of a process while it is actually executing?
I did not think much about them, let’s try togheter.
For outputs we could define an interface similar to the inputs I guess, allowing to examine the requested output and
control its stream (not sure we can have the Process instance handy around though, but I believe we can
have the Name of it, we should probably do the same for the input filters)
interface WPSOutputFilter {
/**
- Allows to alter the output if necessary, or refuse the response
*/
public OutputDataType filterOutput(OutputDataType input, Name processName) throws WPSException;
/**
- Exert control over the streams, do byte counting
*/
public InputStream filterStream(InputType input, Name processName, OutputStream stream) throws WPSException;
}
Hmm… the above would not capture handling xml encoder delegates I’m afraid, but it could be used
to handle raw responses and outputs stored as a reference.
Wondering, what kind of control would we make on it? Output size by byte counting?
As for controlling the process while it’s running, these two would not do, we would need that process
lifecycle interface I was talking about some time ago (failed to secure the funding for it).
And even having that, we’d have to fix most of the processes to actually use a progress listener
and pay attention to the cancelling event, otherwise we’d have little chance of being able to
actually stop the processing at a certain point like we do for WMS vector requests
Suggestions welcomed 
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
On Sat, Aug 11, 2012 at 9:45 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:
On Sat, Aug 11, 2012 at 5:12 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:
Makes sense and seems pretty simple… don’t have much to add. Have you thought about the output side of things? Or perhaps how to control resources of a process while it is actually executing?
I did not think much about them, let’s try togheter.
For outputs we could define an interface similar to the inputs I guess, allowing to examine the requested output and
control its stream (not sure we can have the Process instance handy around though, but I believe we can
have the Name of it, we should probably do the same for the input filters)
interface WPSOutputFilter {
/**
- Allows to alter the output if necessary, or refuse the response
*/
public OutputDataType filterOutput(OutputDataType input, Name processName) throws WPSException;
/**
- Exert control over the streams, do byte counting
*/
public InputStream filterStream(InputType input, Name processName, OutputStream stream) throws WPSException;
}
Hmm… the above would not capture handling xml encoder delegates I’m afraid, but it could be used
to handle raw responses and outputs stored as a reference.
Wondering, what kind of control would we make on it? Output size by byte counting?
Yeah… the only practical limits i can think of is amount of output content… and perhaps by content type.
As for controlling the process while it’s running, these two would not do, we would need that process
lifecycle interface I was talking about some time ago (failed to secure the funding for it).
And even having that, we’d have to fix most of the processes to actually use a progress listener
and pay attention to the cancelling event, otherwise we’d have little chance of being able to
actually stop the processing at a certain point like we do for WMS vector requests
Yeah… killing a process that is taking too long to execute makes the most sense to me but agreed would represent a large task in updating all processes to respect its progress listener.
Suggestions welcomed
The only other thing I can think of would be priority and perhaps being able to run a process with lower priority… or perhaps using priority to do a “soft cancel” of a bad process. That said I know that thread priority is fairly os specific with no real guarantees…
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
On Mon, Aug 13, 2012 at 3:41 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:
Hmm… the above would not capture handling xml encoder delegates I’m afraid, but it could be used
to handle raw responses and outputs stored as a reference.
Wondering, what kind of control would we make on it? Output size by byte counting?
Yeah… the only practical limits i can think of is amount of output content… and perhaps by content type.
As for controlling the process while it’s running, these two would not do, we would need that process
lifecycle interface I was talking about some time ago (failed to secure the funding for it).
And even having that, we’d have to fix most of the processes to actually use a progress listener
and pay attention to the cancelling event, otherwise we’d have little chance of being able to
actually stop the processing at a certain point like we do for WMS vector requests
Yeah… killing a process that is taking too long to execute makes the most sense to me but agreed would represent a large task in updating all processes to respect its progress listener.
Right. I guess we can add the mechanism (in fact it’s already there, ProcessManager has a “cancel” method)
and warn in the javadoc that process implementations are not bound to support cancellation.
Suggestions welcomed
The only other thing I can think of would be priority and perhaps being able to run a process with lower priority… or perhaps using priority to do a “soft cancel” of a bad process. That said I know that thread priority is fairly os specific with no real guarantees…
This is actually a problem, the processes are submitted to a thread pool, which has its own uniform priority.
If one wants to have high and low priority processes that would be possible by implementing their own
ProcessManager instances, but I don’t see a nice/clean/simple way to have at the same time thread pooling and random
process priority
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it