[Geoserver-devel] WPS PPIO factory?

GeoServer GeoServer Developer
1

Hi,
I’m looking into the possibility of having WPS PPIOs generated by a pluggable factory.
The idea is that one has some sort of PPIO generator, that can turn a certain java object
into many different representations.

Think for example of just exposing all available WFS output formats as PPIOs, that
would be a job for a pluggable PPIO factory (similarly, we could generate PPIOs
from ogr2ogr just like we do for wfs output formats).

The interface would probably be as simple as:

interface PPIOFactory {

List getProcessParameterIO();
}

There are a few hiccups of course:

  • Some of these PPIOs might be read only, or write only, for this case we
    might actually want to either extend the ProcessParameterIO base class
    with two extra methods, canRead and canWrite (or isReadSupported/isWriteSupported,
    supportsRead/supportsWrite), or have marker interfaces, ReadOnlyPPIO,
    WriteOnlyPPIO
  • We need to tell apart xml/text/binary formats (as they need to be implement different
    base classes) so we either add something to the WFS output formats telling us
    which one is what, or in the case of ogr2ogr, add something in the ogr2ogr configuration
    to state what type of output we are generating (uh, in the case of XML we might
    even have to roll custom xml encoder delegates… so we’d need a factory for those as well…)

Anyways, this is just broad strokes on the idea for the moment. Suggetions/feedback?

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

2

On Mon, Jul 28, 2014 at 7:46 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Anyways, this is just broad strokes on the idea for the moment.
Suggetions/feedback?

Hey people, one at a time please, I'm getting overwhelmed by the feedback!
:-p

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

3

Hi Andrea, I did not have much constructive feedback - all my ideas bash around the edge of your problem in order to determine scope. Sometimes this comes across as making more work for you so I have remained quiet.

So let me try and sort out your email - if I put all the ideas you listed into an interface I get something like:

interface PPIOFactory {
List< ProcessParameterIO > getProcessParameterIO();
}

···

interface ProcessParameterIO {

static ProcessParameterIO find(Parameter<?> p, ApplicationContext context, String mime)

static List findAll(Parameter<?> p, ApplicationContext context)

String getIdentifer()
Class getExternalType()

Class getType()

boolean isReadSupported()
boolean isWriteSupported();
}
interface ComplexPPIO extends ProcessParameterIO {
Object decode( InputStream input );

public Object decode( Object input );
void encode( Object value, OutputStream os);
String getFileExtension();
String getMimeType() ;
}

Quick takes:

  • getProcessParameterIO implies a single item, not a list
  • The base ProcessParameterIO has static methods to look for stuff, may consider making similar “find” methods rather than create a List.
  • Marker Interfaces ProcessParameterInput and ProcessParameterOutput are clear, but a bit annoying for a dynamic data structure. Would prefer methods as you described or an Enum to check
  • PPIO to start a class name? Call that ProcessParameterIOFactory for clarity
  • Telling apart XML/text/Binary is the heart of this problem. Crack that and then build factory after.
  • How to represent combination input/output/mime types? Either list the same format multiple times (once for each combination) or create a MultplexProcessParameterIO
  • I suspect as this grows up input/output flavours we will need to enable/disable much like ProcessFactory

Result of this musing is:

interface ProcessParameterIOFactory {
List find( Parameter<?> p, ApplicationContext context, ProcessParameterIO.Details find )

}

ProcessParameters {

}

static ProcessParameterIO find(Parameter<?> p, ApplicationContext context, String mime)

static List findAll(Parameter<?> p, ApplicationContext context)
class ProcessParameterIO extends PPIO {

/** Simple data object, covering mime type, input, output … grow to accommodate future needs */

class Details {
public String mimetype;

public boolean input;
public boolean output;
public String subformat; // optional: for xml/text/binary (example xml schema URI)
public String version; // optional: for xml/text/binary

ProcessParaemterIODetails( ProcessParaemterIODetails duplicate);
boolean equals( Object other ); // data object
long hashcode(); // data object
}
String getIdentifer()
Class getExternalType()

Class getType(
Details getDetails();
}

Jody Garnett

On Mon, Jul 28, 2014 at 10:46 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
I’m looking into the possibility of having WPS PPIOs generated by a pluggable factory.
The idea is that one has some sort of PPIO generator, that can turn a certain java object
into many different representations.

Think for example of just exposing all available WFS output formats as PPIOs, that
would be a job for a pluggable PPIO factory (similarly, we could generate PPIOs
from ogr2ogr just like we do for wfs output formats).

The interface would probably be as simple as:

interface PPIOFactory {

List getProcessParameterIO();
}

There are a few hiccups of course:

  • Some of these PPIOs might be read only, or write only, for this case we
    might actually want to either extend the ProcessParameterIO base class
    with two extra methods, canRead and canWrite (or isReadSupported/isWriteSupported,
    supportsRead/supportsWrite), or have marker interfaces, ReadOnlyPPIO,
    WriteOnlyPPIO
  • We need to tell apart xml/text/binary formats (as they need to be implement different
    base classes) so we either add something to the WFS output formats telling us
    which one is what, or in the case of ogr2ogr, add something in the ogr2ogr configuration
    to state what type of output we are generating (uh, in the case of XML we might
    even have to roll custom xml encoder delegates… so we’d need a factory for those as well…)

Anyways, this is just broad strokes on the idea for the moment. Suggetions/feedback?

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

4

On Wed, Jul 30, 2014 at 7:35 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Hi Andrea, I did not have much constructive feedback - all my ideas bash
around the edge of your problem in order to determine scope. Sometimes this
comes across as making more work for you so I have remained quiet.

So let me try and sort out your email - if I put all the ideas you listed
into an interface I get something like:

interface PPIOFactory {
   List< ProcessParameterIO > getProcessParameterIO();
}
interface ProcessParameterIO {
* static ProcessParameterIO find(Parameter<?> p, ApplicationContext
context, String mime) *
* static List<ProcessParameterIO> findAll(Parameter<?> p,
ApplicationContext context)*

   String getIdentifer()
  Class getExternalType()
  Class getType()
  boolean isReadSupported()
  boolean isWriteSupported();
}
interface ComplexPPIO extends ProcessParameterIO {
  Object decode( InputStream input );
  public Object decode( Object input );
  void encode( Object value, OutputStream os);
  String getFileExtension();
  String getMimeType() ;
}

Quick takes:
- getProcessParameterIO implies a single item, not a list

getProcessParameterIOs() ?
getAllProcessParameterIO() ?

- The base ProcessParameterIO has static methods to look for stuff, may
consider making similar "find" methods rather than create a List.

Hmm... no, the lookup logic in ProcessParameterIO.find is not simple, we
don't want it duplicated around.
The factory is there to generate PPIOs so that ProcessParameterIO can
register them and use them later for lookups

- Marker Interfaces ProcessParameterInput and ProcessParameterOutput are
clear, but a bit annoying for a dynamic data structure. Would prefer
methods as you described or an Enum to check

Roger that, no problem

- PPIO to start a class name? Call that ProcessParameterIOFactory for
clarity

No problem here

- Telling apart XML/text/Binary is the heart of this problem. Crack that
and then build factory after.

I lost you there. A factory can build all three types at the same time.

- How to represent combination input/output/mime types? Either list the
same format multiple times (once for each combination) or create a
MultplexProcessParameterIO

The above sentence has no meaning to me, don't believe it matches the
current PPIO design.

- I suspect as this grows up input/output flavours we will need to
enable/disable much like ProcessFactory

I disagree here, what we are likely going to want is to enable/disable the
PPIOs on a per process basis, and
we already have the mechanism to do so, we lack GUI and configuration for
it.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

5

Small clarification inline, and then I do not have anything more to add.

···

Telling apart the individual ProcessParameterIO instances (and what they are capable of) is the hear of the problem. Once we have that sorted it may inform what the find methods look like, and what information needs to be available at the Factory level.

Like I do not think we need the FunctionFactory approach as we do not have a FunctionName equivalent that can be listed independently.

Fair enough, I will use more words. Was thinking of a single mime type - say “text/json” - showing up in the list as:

  • GeoJSONPPInput: mimetype=“text/json” version=“1.0” subformat=“geojson”

  • GeoJSONPPOutput: mimetype=“text/json” version=“1.0” subformat=“geojson”

  • ESRIJSONPPInput: mimetype=“text/json” version=“10.2” subformat=“Featureset”

  • ESRIJSONPPOutput: mimetype=“text/json” version=“10.2” subformat=“Featureset”

  • ESRIJSONPPOutput: mimetype=“text/json” version=“10.2.1” subformat=“Featureseti”

  • ESRIJSONPPOutput: mimetype=“text/json” version=“10.2.2” subformat=“Featureset”

This implies [1] GeoJSON 1.0 is available for read/write while [2] ESRI FeatureSet read/write is available for 10.2, while FeatureSet write also supports 10.2.1 and 10.2.2. This is clear, each ProcessParameterIO does does one thing making find easier.

The “multi” idea would be to allow individual ProcessParameterIO implementations to take on more than one format, flavour - resulting in a list such as:

  • GeoJSONPPIO: mimetype=“text/json” version=“1.0” subformat=“geojson” read=true, write=true

  • ESRIJSONPPOutput: mimetype=“text/json” version=“10.2” subformat=“Featureset” read=true, write=true

  • ESRIJSONPPOutput: mimetype=“text/json” version=(“10.2.1”,“10.2.2”) subformat=“Featureset” read=false, write=true

Here are the competing JSON formats:
[1] http://geojson.org
[2] http://resources.arcgis.com/en/help/main/10.2/index.html#//001200000059000000

Jody

I lost you there. A factory can build all three types at the same time.

  • Telling apart XML/text/Binary is the heart of this problem. Crack that and then build factory after.

The above sentence has no meaning to me, don’t believe it matches the current PPIO design.

  • How to represent combination input/output/mime types? Either list the same format multiple times (once for each combination) or create a MultplexProcessParameterIO
6

On Wed, Jul 30, 2014 at 8:42 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Small clarification inline, and then I do not have anything more to add.

- Telling apart XML/text/Binary is the heart of this problem. Crack that

and then build factory after.

I lost you there. A factory can build all three types at the same time.

Telling apart the individual ProcessParameterIO instances (and what they
are capable of) is the hear of the problem. Once we have that sorted it may
inform what the find methods look like, and what information needs to be
available at the Factory level.

Like I do not think we need the FunctionFactory approach as we do not have
a FunctionName equivalent that can be listed independently.

I still don't get it. There are the three base classes for the three cases,
that's all we have needed so far.

- How to represent combination input/output/mime types? Either list the

same format multiple times (once for each combination) or create a
MultplexProcessParameterIO

The above sentence has no meaning to me, don't believe it matches the
current PPIO design.

Fair enough, I will use more words. Was thinking of a single mime type -
say "text/json" - showing up in the list as:
- GeoJSONPPInput: mimetype="text/json" version="1.0" subformat="geojson"
- GeoJSONPPOutput: mimetype="text/json" version="1.0" subformat="geojson"
- ESRIJSONPPInput: mimetype="text/json" version="10.2"
subformat="Featureset"
- ESRIJSONPPOutput: mimetype="text/json" version="10.2"
subformat="Featureset"
- ESRIJSONPPOutput: mimetype="text/json" version="10.2.1"
subformat="Featureseti"
- ESRIJSONPPOutput: mimetype="text/json" version="10.2.2"
subformat="Featureset"

There is no such a such as a version or a subformat, please check the api,
a PPIO binds
a java class to an input/outupt.
To be clear, my objective is to make it possible to add more PPIOs without
having to register
them one by one in the app context (and have to know in advance what they
are), I have
no scope to redesign the way PPIOs are working, are looked up, or the
functionality
they expose, I just want to declare their existence to the current code.

I don't get why to do such a simple task we have to involve a unrelated
large refactoring
of the PPIO code base.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

7

There is no such a such as a version or a subformat, please check the api,
a PPIO binds
a java class to an input/outupt.

I was looking at how the API may grow in the future if you want to untangle
xml/text/binary formats.

To be clear, my objective is to make it possible to add more PPIOs without
having to register
them one by one in the app context (and have to know in advance what they
are), I have
no scope to redesign the way PPIOs are working, are looked up, or the
functionality
they expose, I just want to declare their existence to the current code.

See start of my comments: *"my ideas bash around the edge of your problem
in order to determine scope.*"

In this case I was trying to write down your requirement: "We need to tell
apart xml/text/binary formats (as they need to be implement different base
classes)"

I had two ideas on the table for that - what did you have in mind?

I don't get why to do such a simple task we have to involve a unrelated

large refactoring
of the PPIO code base.

Just trying to make sense of your requirements, design alternatives expand
scope in the hopes of hitting on a small subset. If there is something to
be learned so be it.
--
Jody

8

On Wed, Jul 30, 2014 at 9:56 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

To be clear, my objective is to make it possible to add more PPIOs
without having to register
them one by one in the app context (and have to know in advance what they
are), I have
no scope to redesign the way PPIOs are working, are looked up, or the
functionality
they expose, I just want to declare their existence to the current code.

See start of my comments: *"my ideas bash around the edge of your problem
in order to determine scope.*"

In this case I was trying to write down your requirement: "We need to
tell apart xml/text/binary formats (as they need to be implement different
base classes)"

I had two ideas on the table for that - what did you have in mind?

My sentence had nothing to do with telling apart PPIOs, they can be told
apart already as they
have separate base classes, what we need
to tell apart is what the WFS output format being wrapped is producing,
either binary, text or GML,
so that we can wrap it in the appropriate PPIO wrapper class.
(and/or the ogr generated formats, if those are wrapped instead).

The sentence was:

"so we either add something to the WFS output formats telling us
  which one is what, or in the case of ogr2ogr, add something in the
ogr2ogr configuration
  to state what type of output we are generating (uh, in the case of XML we
might
  even have to roll custom xml encoder delegates... so we'd need a factory
for those as well...)"

In more detailed terms, if we go the wfs output format wrapping
approach, WFSGetFeatureOutputFormat
would need to expose a new method:

getOutputNature(): OutputNature

where OutputNature would be and enumeration with values:
* XML (e.g., GML),
* Text (e.g. csv/json),
* Binary (e.g., shape-zip).

If instead we go for the approach of wrapping specifically the ogr2ogr
generated output formats (instead
of trying to expose each and every wfs output format as a PPIO), it would
be the just the OgrFormat class to have to expose the above method.

Another approach could be to have some sort of "registry" of mime types
that maps the mime type
to its nature (or some set of heuristics), the upside is that the exisiting
wfs output format
API would not have to be touched, the downside is that it would make it
difficult to those
adding output formats via OGR (user generated, instead of added by a
programmer) to control
the exact PPIO type associated.

Makes sense?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

9

Make sense.

···

Jody Garnett

On Wed, Jul 30, 2014 at 1:12 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, Jul 30, 2014 at 9:56 PM, Jody Garnett <jody.garnett@anonymised.com> wrote:

My sentence had nothing to do with telling apart PPIOs, they can be told apart already as they
have separate base classes, what we need
to tell apart is what the WFS output format being wrapped is producing, either binary, text or GML,
so that we can wrap it in the appropriate PPIO wrapper class.
(and/or the ogr generated formats, if those are wrapped instead).

The sentence was:

“so we either add something to the WFS output formats telling us
which one is what, or in the case of ogr2ogr, add something in the ogr2ogr configuration
to state what type of output we are generating (uh, in the case of XML we might
even have to roll custom xml encoder delegates… so we’d need a factory for those as well…)”

In more detailed terms, if we go the wfs output format wrapping approach, WFSGetFeatureOutputFormat
would need to expose a new method:

getOutputNature(): OutputNature

where OutputNature would be and enumeration with values:

  • XML (e.g., GML),
  • Text (e.g. csv/json),
  • Binary (e.g., shape-zip).

If instead we go for the approach of wrapping specifically the ogr2ogr generated output formats (instead
of trying to expose each and every wfs output format as a PPIO), it would
be the just the OgrFormat class to have to expose the above method.

Another approach could be to have some sort of “registry” of mime types that maps the mime type
to its nature (or some set of heuristics), the upside is that the exisiting wfs output format
API would not have to be touched, the downside is that it would make it difficult to those
adding output formats via OGR (user generated, instead of added by a programmer) to control
the exact PPIO type associated.

Makes sense?

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

See start of my comments: "my ideas bash around the edge of your problem in order to determine scope."

In this case I was trying to write down your requirement: “We need to tell apart xml/text/binary formats (as they need to be implement different base classes)”

I had two ideas on the table for that - what did you have in mind?

To be clear, my objective is to make it possible to add more PPIOs without having to register
them one by one in the app context (and have to know in advance what they are), I have
no scope to redesign the way PPIOs are working, are looked up, or the functionality
they expose, I just want to declare their existence to the current code.