[Geoserver-devel] jsr 223 based scripting module(s)

Hi all,

In the past few weeks i have been working on a new set of scripting modules based on jsr 223 and the javax.script api. I know there has been interest around this from many parties so i wanted to share what i have been working on. Eventually I would like to dd it as a community module/extension but for now i think i prefer to just work in git.

The idea is that his new module would eventually take over the scripting modules we currently have , namely:

community/
python
scriptlet

The way I have things structured currently is a core module, and then a module for each language. It breaks down like:

communtity/
script/
core/
bsh/
groovy/
js/
py/
rb/

The new module is designed much like the existing python scripting module that revolves around the idea of providing “scripting hooks” for various extension points. Currently i have only implemented the following hooks:

  • wps process
  • filter functions

I plan to continue to port from the old python module and add additional scripting hooks:

  • output formats (GetMap, GetFeatureInfo, GetFeature, etc…)
  • datastores
  • etc…

The idea is that most of the code to handle a hook will live in the core module leaving nothing to do for a specific language other than adding the language jar (and script engine jar in cases where it is not included in the core language jar).

However the idea is also to support specific customizations for those languages that can support it. For example geoscript. Since python, groovy, and javascript have geoscript bindings they would want to utilize them say for instance when invoking a process hook. Instead of passing in a raw geotools feature the idea would be to wrap it in a geoscript feature before calling the process.

There is also a rest api that allows for various functionality. The first is to invoke a simple script over http so you can do something like this:

curl -G “http://localhost:8080/geoserver/script/foo.py

The interface to the script is a simple one, define a function named “run” and have it accept a request/response object. For example, with beanshell:

foo.bsh:

run(request,response) {
response.setEntity(“Hello World!”)

}

However, this would be the default interface, each language would have the ability to customize this. For instance, with python there is already the WSGI standard for an interface for web applications. So in that module we would want to customize and support that:

foo.py:

def application(environ, start_response):
start_response(‘200 OK’, [(‘Content-Type’, ‘text/plain’)])
return [‘Hello World!’]

Also there is an api for creating and persisting “sessions” on the server, that is script engine instance that will persist state across different calls. The idea here is to be able to interact with scripts on the server via remote clients. For example, for the python extension I have written a client based on this session api that i can invoke from my terminal. The client mimics the regular python shell, but is actually executing on the server side. ( Obviously all of this would be secured by default and only available to the administrator :slight_smile: )

Let me know what you think, the code is in a branch called “scripting” in my github repo:

https://github.com/jdeolive/geoserver/tree/scripting/src/community/script

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

On Fri, Mar 16, 2012 at 10:01 PM, Justin Deoliveira <jdeolive@anonymised.com…1501…> wrote:

Hi all,

In the past few weeks i have been working on a new set of scripting modules based on jsr 223 and the javax.script api. I know there has been interest around this from many parties so i wanted to share what i have been working on. Eventually I would like to dd it as a community module/extension but for now i think i prefer to just work in git.

The idea is that his new module would eventually take over the scripting modules we currently have , namely:

community/
python
scriptlet

The way I have things structured currently is a core module, and then a module for each language. It breaks down like:

communtity/
script/
core/
bsh/
groovy/
js/
py/
rb/

The new module is designed much like the existing python scripting module that revolves around the idea of providing “scripting hooks” for various extension points. Currently i have only implemented the following hooks:

  • wps process
  • filter functions

I plan to continue to port from the old python module and add additional scripting hooks:

  • output formats (GetMap, GetFeatureInfo, GetFeature, etc…)
  • datastores
  • etc…

The idea is that most of the code to handle a hook will live in the core module leaving nothing to do for a specific language other than adding the language jar (and script engine jar in cases where it is not included in the core language jar).

However the idea is also to support specific customizations for those languages that can support it. For example geoscript. Since python, groovy, and javascript have geoscript bindings they would want to utilize them say for instance when invoking a process hook. Instead of passing in a raw geotools feature the idea would be to wrap it in a geoscript feature before calling the process.

There is also a rest api that allows for various functionality. The first is to invoke a simple script over http so you can do something like this:

curl -G “http://localhost:8080/geoserver/script/foo.py

The interface to the script is a simple one, define a function named “run” and have it accept a request/response object. For example, with beanshell:

foo.bsh:

run(request,response) {
response.setEntity(“Hello World!”)

}

However, this would be the default interface, each language would have the ability to customize this. For instance, with python there is already the WSGI standard for an interface for web applications. So in that module we would want to customize and support that:

foo.py:

def application(environ, start_response):
start_response(‘200 OK’, [(‘Content-Type’, ‘text/plain’)])
return [‘Hello World!’]

Also there is an api for creating and persisting “sessions” on the server, that is script engine instance that will persist state across different calls. The idea here is to be able to interact with scripts on the server via remote clients. For example, for the python extension I have written a client based on this session api that i can invoke from my terminal. The client mimics the regular python shell, but is actually executing on the server side. ( Obviously all of this would be secured by default and only available to the administrator :slight_smile: )

Let me know what you think, the code is in a branch called “scripting” in my github repo:

https://github.com/jdeolive/geoserver/tree/scripting/src/community/script

Sounds really interesting.
Considering the specific case of functions and processes, there is the issue of how to describe
the inputs (name and type).
As far as I can see this is done differently on a language by language basis, in the WpsHook
subclass.
I was wondering if we should promote diversity or uniformity in this case, that is, have
all languages roll their own specific way to describe a process or have one uniform way.
Some months ago I’ve tried myself to start a jsr 223 based approach and to work
in a uniform way I was considering using a sidecar json file describing the process/function,
much like the current scripting module describes them.
The pro of this approach is that you have to document and code the metadata reader
just once, the cons is that it’s not integrated and possibly “alien” to the specific language.
Maybe an integrated approach would work where the sidecar is sort of the “uniform baseline”
that every language can use and more integrated approaches can be developed to
allow for a more natural development style on each specific language.

Anyways, just thinking out loud, I like where this is going and hope to see it as part
of GeoServer (ideally, script/core in core, every language as an extension)
sometimes in the future

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


+1 on having a uniform approach across all languages. The sidecar file seems like the most obvious way to provide this metadata.

Is there a move away from using XML towards JSON as a configuration metadata format?

On Sun, Mar 18, 2012 at 3:46 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Mar 16, 2012 at 10:01 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hi all,

In the past few weeks i have been working on a new set of scripting modules based on jsr 223 and the javax.script api. I know there has been interest around this from many parties so i wanted to share what i have been working on. Eventually I would like to dd it as a community module/extension but for now i think i prefer to just work in git.

The idea is that his new module would eventually take over the scripting modules we currently have , namely:

community/
python
scriptlet

The way I have things structured currently is a core module, and then a module for each language. It breaks down like:

communtity/
script/
core/
bsh/
groovy/
js/
py/
rb/

The new module is designed much like the existing python scripting module that revolves around the idea of providing “scripting hooks” for various extension points. Currently i have only implemented the following hooks:

  • wps process
  • filter functions

I plan to continue to port from the old python module and add additional scripting hooks:

  • output formats (GetMap, GetFeatureInfo, GetFeature, etc…)
  • datastores
  • etc…

The idea is that most of the code to handle a hook will live in the core module leaving nothing to do for a specific language other than adding the language jar (and script engine jar in cases where it is not included in the core language jar).

However the idea is also to support specific customizations for those languages that can support it. For example geoscript. Since python, groovy, and javascript have geoscript bindings they would want to utilize them say for instance when invoking a process hook. Instead of passing in a raw geotools feature the idea would be to wrap it in a geoscript feature before calling the process.

There is also a rest api that allows for various functionality. The first is to invoke a simple script over http so you can do something like this:

curl -G “http://localhost:8080/geoserver/script/foo.py

The interface to the script is a simple one, define a function named “run” and have it accept a request/response object. For example, with beanshell:

foo.bsh:

run(request,response) {
response.setEntity(“Hello World!”)

}

However, this would be the default interface, each language would have the ability to customize this. For instance, with python there is already the WSGI standard for an interface for web applications. So in that module we would want to customize and support that:

foo.py:

def application(environ, start_response):
start_response(‘200 OK’, [(‘Content-Type’, ‘text/plain’)])
return [‘Hello World!’]

Also there is an api for creating and persisting “sessions” on the server, that is script engine instance that will persist state across different calls. The idea here is to be able to interact with scripts on the server via remote clients. For example, for the python extension I have written a client based on this session api that i can invoke from my terminal. The client mimics the regular python shell, but is actually executing on the server side. ( Obviously all of this would be secured by default and only available to the administrator :slight_smile: )

Let me know what you think, the code is in a branch called “scripting” in my github repo:

https://github.com/jdeolive/geoserver/tree/scripting/src/community/script

Sounds really interesting.
Considering the specific case of functions and processes, there is the issue of how to describe
the inputs (name and type).
As far as I can see this is done differently on a language by language basis, in the WpsHook
subclass.
I was wondering if we should promote diversity or uniformity in this case, that is, have
all languages roll their own specific way to describe a process or have one uniform way.
Some months ago I’ve tried myself to start a jsr 223 based approach and to work
in a uniform way I was considering using a sidecar json file describing the process/function,
much like the current scripting module describes them.
The pro of this approach is that you have to document and code the metadata reader
just once, the cons is that it’s not integrated and possibly “alien” to the specific language.
Maybe an integrated approach would work where the sidecar is sort of the “uniform baseline”
that every language can use and more integrated approaches can be developed to
allow for a more natural development style on each specific language.

Anyways, just thinking out loud, I like where this is going and hope to see it as part
of GeoServer (ideally, script/core in core, every language as an extension)
sometimes in the future

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf



This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Martin Davis
OpenGeo - http://opengeo.org
Expert service straight from the developers.

On Sun, Mar 18, 2012 at 5:26 PM, Martin Davis <mdavis@anonymised.com> wrote:

+1 on having a uniform approach across all languages. The sidecar file seems like the most obvious way to provide this metadata.

Is there a move away from using XML towards JSON as a configuration metadata format?

Hum… not for me. The idea behind using JSON was the impression that people working
on scripting would have the following preference order when configuring processes:

  1. something native to their scripting language of choice
  2. a json configuration
  3. a xml configuration

Since 1) is desirable but not uniform I went for the second.
But it was not a decision based on a general interest in moving every configuratin
to JSON, more the idea that JSON would have been more palatable than XML
to the scripting crowd.
I personally don’t mind… give me a GUI that allows to edit the script and configure
the parameter visually, then save them in whatever format you like :-p

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


I’m +1 for XML.

There are lots of places where XML hasn’t been a great fit - data formats (GML) and query languages (Filter, SLD) come to mind. But for short, declarative configuation files it still seems like a good fit, and is relatively painless to author in that domain. This especially applies to configuration which will contain a significant proportion of textual data - which I think is the case with Process metadata?

By the way, did you forget one option - Java properties format? 8^)

As for keeping it hidden behind a GUI. it seem likely that as scripting becomes more prevalent a “gray market” of GeoServer-based code will emerge. Users and organizations are going to build up libraries of functionality and share them internally and externally. So the artifacts needed to define that functionality are going to be very visible. It could be worth following the well-trodden Java path of archives and manifest files.

On Sun, Mar 18, 2012 at 2:38 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Mar 18, 2012 at 5:26 PM, Martin Davis <mdavis@anonymised.com> wrote:

+1 on having a uniform approach across all languages. The sidecar file seems like the most obvious way to provide this metadata.

Is there a move away from using XML towards JSON as a configuration metadata format?

Hum… not for me. The idea behind using JSON was the impression that people working
on scripting would have the following preference order when configuring processes:

  1. something native to their scripting language of choice
  2. a json configuration
  3. a xml configuration

Since 1) is desirable but not uniform I went for the second.
But it was not a decision based on a general interest in moving every configuratin
to JSON, more the idea that JSON would have been more palatable than XML
to the scripting crowd.
I personally don’t mind… give me a GUI that allows to edit the script and configure
the parameter visually, then save them in whatever format you like :-p

Cheers

Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Martin Davis
OpenGeo - http://opengeo.org
Expert service straight from the developers.

On Mon, Mar 19, 2012 at 5:11 AM, Martin Davis <mdavis@anonymised.com> wrote:

I’m +1 for XML.

There are lots of places where XML hasn’t been a great fit - data formats (GML) and query languages (Filter, SLD) come to mind. But for short, declarative configuation files it still seems like a good fit, and is relatively painless to author in that domain. This especially applies to configuration which will contain a significant proportion of textual data - which I think is the case with Process metadata?

By the way, did you forget one option - Java properties format? 8^)

As for keeping it hidden behind a GUI. it seem likely that as scripting becomes more prevalent a “gray market” of GeoServer-based code will emerge. Users and organizations are going to build up libraries of functionality and share them internally and externally. So the artifacts needed to define that functionality are going to be very visible. It could be worth following the well-trodden Java path of archives and manifest files.

It is something we need to think about carefully, trying to impose java ways of doing things to people that
already chose to walk away from that language to start with might backfire on us in terms of lower adoption
of the scripting interfaces

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


On metadata: managing metadata externally from the script that is described sounds like a deterrent to scripters regardless of format. I would hope that the scripting module can just define a Java method like:

Metadata describeProcess(String name, String abstract, List<Class<?>> inputs, List> outputs);

And leave it up to scripters to figure out how to call it. Similarly, I am skeptical about having a GUI and suspect most script writers already have an editor that they know and love (and which has spiffy auto-complete and other features that would be a pain to do in Wicket.) A GUI’s probably nice to have, but I would put it low on my (personal) priority list compared to auto-reloading the script when it changes. Just my 2 cents of course.

On compatibility: I am also concerned about opening up scripting to GeoServer users. As recently discussed on another thread on this list, the GeoServer Java API has not historically been a big priority for the project. For example, “extension points” are added as needed and not really enumerated anywhere in the documentation. I think GeoServer is generally pretty good about avoiding backwards-incompatible API changes, but scripting languages can have different issues with what’s required for backward compatibility (for example, calling overloaded Java methods in untyped languages present some unique challenges.) Is there any plans to help support upgrading GeoServer deployments that rely on scripts? Maybe it would be worthwhile investigate sandboxing to limit the “surface area” of GeoServer as exposed to the scripts. More generally, is the GeoServer developer and user community ready to start handling user questions about code in five new languages?

On reliability: From my work on GeoNode and gsconfig.py I know already that it’s quite easy to produce incorrect configurations through the REST API, which GeoServer will happily publish. While I am able (and mostly willing :wink: ) to dig into the datadir and delete some XML when I botch a REST request and have GeoServer telling me that I can’t create a layergroup because it already exists, but can’t delete it because it doesn’t exist, I think that we are asking for trouble by encouraging scripting against the catalog while the catalog’s validation is at its current level of completeness. Remember, by enabling scripting we are basically asking for developers to start developing against GeoServer who have no idea how to use a Java debugger or read a stack trace.


David Winslow
OpenGeo - http://opengeo.org/

On Mon, Mar 19, 2012 at 5:40 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Mar 19, 2012 at 5:11 AM, Martin Davis <mdavis@anonymised.com> wrote:

I’m +1 for XML.

There are lots of places where XML hasn’t been a great fit - data formats (GML) and query languages (Filter, SLD) come to mind. But for short, declarative configuation files it still seems like a good fit, and is relatively painless to author in that domain. This especially applies to configuration which will contain a significant proportion of textual data - which I think is the case with Process metadata?

By the way, did you forget one option - Java properties format? 8^)

As for keeping it hidden behind a GUI. it seem likely that as scripting becomes more prevalent a “gray market” of GeoServer-based code will emerge. Users and organizations are going to build up libraries of functionality and share them internally and externally. So the artifacts needed to define that functionality are going to be very visible. It could be worth following the well-trodden Java path of archives and manifest files.

It is something we need to think about carefully, trying to impose java ways of doing things to people that
already chose to walk away from that language to start with might backfire on us in terms of lower adoption
of the scripting interfaces

Cheers

Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf



This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Indeed, this is a decision i have been fighting with. If you look at the other hooks like function and the “app” hook they do allow for a baseline behavior that the different languages don’t have to customize unless they want to. But as you mention the metadata requirements for a process are much heavier.

I too go back and forth on the idea of an index file describing the process but an weary for the reasons you mentioned. One plus though is that if there was a single metadata file it would make the capabilities document generate much faster. If the metadata lives in the script itself it means a capabilities request has to evaluate every process script on a getcapabilities. Some caching and reloading of the engine only when changes occur will help (which is implemented), but still something to consider.

Davids idea about a second function that would return a java metadata object is an interesting one as well.

On Sun, Mar 18, 2012 at 4:46 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Mar 16, 2012 at 10:01 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hi all,

In the past few weeks i have been working on a new set of scripting modules based on jsr 223 and the javax.script api. I know there has been interest around this from many parties so i wanted to share what i have been working on. Eventually I would like to dd it as a community module/extension but for now i think i prefer to just work in git.

The idea is that his new module would eventually take over the scripting modules we currently have , namely:

community/
python
scriptlet

The way I have things structured currently is a core module, and then a module for each language. It breaks down like:

communtity/
script/
core/
bsh/
groovy/
js/
py/
rb/

The new module is designed much like the existing python scripting module that revolves around the idea of providing “scripting hooks” for various extension points. Currently i have only implemented the following hooks:

  • wps process
  • filter functions

I plan to continue to port from the old python module and add additional scripting hooks:

  • output formats (GetMap, GetFeatureInfo, GetFeature, etc…)
  • datastores
  • etc…

The idea is that most of the code to handle a hook will live in the core module leaving nothing to do for a specific language other than adding the language jar (and script engine jar in cases where it is not included in the core language jar).

However the idea is also to support specific customizations for those languages that can support it. For example geoscript. Since python, groovy, and javascript have geoscript bindings they would want to utilize them say for instance when invoking a process hook. Instead of passing in a raw geotools feature the idea would be to wrap it in a geoscript feature before calling the process.

There is also a rest api that allows for various functionality. The first is to invoke a simple script over http so you can do something like this:

curl -G “http://localhost:8080/geoserver/script/foo.py

The interface to the script is a simple one, define a function named “run” and have it accept a request/response object. For example, with beanshell:

foo.bsh:

run(request,response) {
response.setEntity(“Hello World!”)

}

However, this would be the default interface, each language would have the ability to customize this. For instance, with python there is already the WSGI standard for an interface for web applications. So in that module we would want to customize and support that:

foo.py:

def application(environ, start_response):
start_response(‘200 OK’, [(‘Content-Type’, ‘text/plain’)])
return [‘Hello World!’]

Also there is an api for creating and persisting “sessions” on the server, that is script engine instance that will persist state across different calls. The idea here is to be able to interact with scripts on the server via remote clients. For example, for the python extension I have written a client based on this session api that i can invoke from my terminal. The client mimics the regular python shell, but is actually executing on the server side. ( Obviously all of this would be secured by default and only available to the administrator :slight_smile: )

Let me know what you think, the code is in a branch called “scripting” in my github repo:

https://github.com/jdeolive/geoserver/tree/scripting/src/community/script

Sounds really interesting.
Considering the specific case of functions and processes, there is the issue of how to describe
the inputs (name and type).
As far as I can see this is done differently on a language by language basis, in the WpsHook
subclass.
I was wondering if we should promote diversity or uniformity in this case, that is, have
all languages roll their own specific way to describe a process or have one uniform way.
Some months ago I’ve tried myself to start a jsr 223 based approach and to work
in a uniform way I was considering using a sidecar json file describing the process/function,
much like the current scripting module describes them.
The pro of this approach is that you have to document and code the metadata reader
just once, the cons is that it’s not integrated and possibly “alien” to the specific language.
Maybe an integrated approach would work where the sidecar is sort of the “uniform baseline”
that every language can use and more integrated approaches can be developed to
allow for a more natural development style on each specific language.

Anyways, just thinking out loud, I like where this is going and hope to see it as part
of GeoServer (ideally, script/core in core, every language as an extension)
sometimes in the future

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf



Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

On Mon, Mar 19, 2012 at 7:25 AM, David Winslow <dwinslow@anonymised.com> wrote:

On metadata: managing metadata externally from the script that is described sounds like a deterrent to scripters regardless of format. I would hope that the scripting module can just define a Java method like:

Metadata describeProcess(String name, String abstract, List<Class<?>> inputs, List> outputs);

And leave it up to scripters to figure out how to call it. Similarly, I am skeptical about having a GUI and suspect most script writers already have an editor that they know and love (and which has spiffy auto-complete and other features that would be a pain to do in Wicket.) A GUI’s probably nice to have, but I would put it low on my (personal) priority list compared to auto-reloading the script when it changes. Just my 2 cents of course.

Yeah, I like the idea having a function returning a pre-defined java class, or one that passes all the metadata into the function, beter than the secondary file i think.

As for the GUI I agree that most script writers will already have editors that they prefer and anything we provide in wicket won’t be nearly as powerful. It could still find some utility in cases where the user doesn’t have access to the server file system. There is also a rest api to get/put the contents of scripts, so those users could use that to download locally, edit, and then push back.

On compatibility: I am also concerned about opening up scripting to GeoServer users. As recently discussed on another thread on this list, the GeoServer Java API has not historically been a big priority for the project. For example, “extension points” are added as needed and not really enumerated anywhere in the documentation. I think GeoServer is generally pretty good about avoiding backwards-incompatible API changes, but scripting languages can have different issues with what’s required for backward compatibility (for example, calling overloaded Java methods in untyped languages present some unique challenges.) Is there any plans to help support upgrading GeoServer deployments that rely on scripts? Maybe it would be worthwhile investigate sandboxing to limit the “surface area” of GeoServer as exposed to the scripts. More generally, is the GeoServer developer and user community ready to start handling user questions about code in five new languages?

Part of my view of a scripting extension is it finally is a focus on providing a good api for extension writers. For instance the python extension ships with some python modules for interacting with the geoserver catalog in a more python like way. And geoscript provides a more convenient interface for the underlying geotools stuff.

Regarding supporting new languages I think we simply approach this as we do any extension in that the module must have a maintainer. If we don;t have anyone willing to step up for a language then we leave it unsupported in community.

On reliability: From my work on GeoNode and gsconfig.py I know already that it’s quite easy to produce incorrect configurations through the REST API, which GeoServer will happily publish. While I am able (and mostly willing :wink: ) to dig into the datadir and delete some XML when I botch a REST request and have GeoServer telling me that I can’t create a layergroup because it already exists, but can’t delete it because it doesn’t exist, I think that we are asking for trouble by encouraging scripting against the catalog while the catalog’s validation is at its current level of completeness. Remember, by enabling scripting we are basically asking for developers to start developing against GeoServer who have no idea how to use a Java debugger or read a stack trace.

Unfortunately most of these issues never seem to make it into bug reports or patches? It is certainly a valid concern but at the same time if we wait for the underpinnings of geoserver to be perfect we will never go anywhere. The catalog api will never be perfect unfortunately and there will always be some way to shoot ones self in the foot. The counter argument is that opening it up to a different level of access will hopefully help harden it and flesh out more these configuration issues.


David Winslow
OpenGeo - http://opengeo.org/

On Mon, Mar 19, 2012 at 5:40 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Mar 19, 2012 at 5:11 AM, Martin Davis <mdavis@anonymised.com> wrote:

I’m +1 for XML.

There are lots of places where XML hasn’t been a great fit - data formats (GML) and query languages (Filter, SLD) come to mind. But for short, declarative configuation files it still seems like a good fit, and is relatively painless to author in that domain. This especially applies to configuration which will contain a significant proportion of textual data - which I think is the case with Process metadata?

By the way, did you forget one option - Java properties format? 8^)

As for keeping it hidden behind a GUI. it seem likely that as scripting becomes more prevalent a “gray market” of GeoServer-based code will emerge. Users and organizations are going to build up libraries of functionality and share them internally and externally. So the artifacts needed to define that functionality are going to be very visible. It could be worth following the well-trodden Java path of archives and manifest files.

It is something we need to think about carefully, trying to impose java ways of doing things to people that
already chose to walk away from that language to start with might backfire on us in terms of lower adoption
of the scripting interfaces

Cheers

Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf



This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

On Mon, Mar 19, 2012 at 3:24 PM, Justin Deoliveira <jdeolive@anonymised.com.1501…> wrote:

And leave it up to scripters to figure out how to call it. Similarly, I am skeptical about having a GUI and suspect most script writers already have an editor that they know and love (and which has spiffy auto-complete and other features that would be a pain to do in Wicket.) A GUI’s probably nice to have, but I would put it low on my (personal) priority list compared to auto-reloading the script when it changes. Just my 2 cents of course.

Yeah, I like the idea having a function returning a pre-defined java class, or one that passes all the metadata into the function, beter than the secondary file i think.

As for the GUI I agree that most script writers will already have editors that they prefer and anything we provide in wicket won’t be nearly as powerful. It could still find some utility in cases where the user doesn’t have access to the server file system. There is also a rest api to get/put the contents of scripts, so those users could use that to download locally, edit, and then push back.

Yeah, that’s precisely what I was aiming at with the GUI. I don’t think anybody will code in the editor, in fact a text area would suffice,
but not having access to the server file system is oh so common in enviroments where the OGC service managers are not the
same people as the physical server admin.

On reliability: From my work on GeoNode and gsconfig.py I know already that it’s quite easy to produce incorrect configurations through the REST API, which GeoServer will happily publish. While I am able (and mostly willing :wink: ) to dig into the datadir and delete some XML when I botch a REST request and have GeoServer telling me that I can’t create a layergroup because it already exists, but can’t delete it because it doesn’t exist, I think that we are asking for trouble by encouraging scripting against the catalog while the catalog’s validation is at its current level of completeness. Remember, by enabling scripting we are basically asking for developers to start developing against GeoServer who have no idea how to use a Java debugger or read a stack trace.

Unfortunately most of these issues never seem to make it into bug reports or patches? It is certainly a valid concern but at the same time if we wait for the underpinnings of geoserver to be perfect we will never go anywhere. The catalog api will never be perfect unfortunately and there will always be some way to shoot ones self in the foot. The counter argument is that opening it up to a different level of access will hopefully help harden it and flesh out more these configuration issues.

Yep, agreed on this one as well. In any case, most of the scripting extensions I hear about are not dealing with the configuration.
That said, +1 on harderning the catalog too, but I agree we need bug reports for that.

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


I can add some, but there are already about 60 bugs open against the REST API:

http://jira.codehaus.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+GEOS+AND+resolution+%3D+Unresolved+AND+component+%3D+REST+ORDER+BY+priority+DESC


David Winslow
OpenGeo - http://opengeo.org/

On Mon, Mar 19, 2012 at 10:41 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Mar 19, 2012 at 3:24 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

And leave it up to scripters to figure out how to call it. Similarly, I am skeptical about having a GUI and suspect most script writers already have an editor that they know and love (and which has spiffy auto-complete and other features that would be a pain to do in Wicket.) A GUI’s probably nice to have, but I would put it low on my (personal) priority list compared to auto-reloading the script when it changes. Just my 2 cents of course.

Yeah, I like the idea having a function returning a pre-defined java class, or one that passes all the metadata into the function, beter than the secondary file i think.

As for the GUI I agree that most script writers will already have editors that they prefer and anything we provide in wicket won’t be nearly as powerful. It could still find some utility in cases where the user doesn’t have access to the server file system. There is also a rest api to get/put the contents of scripts, so those users could use that to download locally, edit, and then push back.

Yeah, that’s precisely what I was aiming at with the GUI. I don’t think anybody will code in the editor, in fact a text area would suffice,
but not having access to the server file system is oh so common in enviroments where the OGC service managers are not the
same people as the physical server admin.

On reliability: From my work on GeoNode and gsconfig.py I know already that it’s quite easy to produce incorrect configurations through the REST API, which GeoServer will happily publish. While I am able (and mostly willing :wink: ) to dig into the datadir and delete some XML when I botch a REST request and have GeoServer telling me that I can’t create a layergroup because it already exists, but can’t delete it because it doesn’t exist, I think that we are asking for trouble by encouraging scripting against the catalog while the catalog’s validation is at its current level of completeness. Remember, by enabling scripting we are basically asking for developers to start developing against GeoServer who have no idea how to use a Java debugger or read a stack trace.

Unfortunately most of these issues never seem to make it into bug reports or patches? It is certainly a valid concern but at the same time if we wait for the underpinnings of geoserver to be perfect we will never go anywhere. The catalog api will never be perfect unfortunately and there will always be some way to shoot ones self in the foot. The counter argument is that opening it up to a different level of access will hopefully help harden it and flesh out more these configuration issues.

Yep, agreed on this one as well. In any case, most of the scripting extensions I hear about are not dealing with the configuration.
That said, +1 on harderning the catalog too, but I agree we need bug reports for that.

Cheers

Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Certainly a great start, unfortunately most of them don’t have patches ready to commit. Unfortunately the sad reality is that unless an issue has a patch that can be committed with relatively low effort i can’t justify the time to spend on it unless i have a mandate from another project driving it.

It seems like the geonode project certainly does have this mandate and has some skilled java developers who are familiar with the geoserver code base, so i would hope this would result in more patches coming forth.

Anyways, indeed the backlog of rest api bugs is discouraging which is why i intend to focus solely on that in the hopefully upcoming foss4g bug squashing code sprint.

On Mon, Mar 19, 2012 at 8:46 AM, David Winslow <dwinslow@anonymised.com> wrote:

I can add some, but there are already about 60 bugs open against the REST API:

http://jira.codehaus.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+GEOS+AND+resolution+%3D+Unresolved+AND+component+%3D+REST+ORDER+BY+priority+DESC


David Winslow
OpenGeo - http://opengeo.org/

On Mon, Mar 19, 2012 at 10:41 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Mar 19, 2012 at 3:24 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

And leave it up to scripters to figure out how to call it. Similarly, I am skeptical about having a GUI and suspect most script writers already have an editor that they know and love (and which has spiffy auto-complete and other features that would be a pain to do in Wicket.) A GUI’s probably nice to have, but I would put it low on my (personal) priority list compared to auto-reloading the script when it changes. Just my 2 cents of course.

Yeah, I like the idea having a function returning a pre-defined java class, or one that passes all the metadata into the function, beter than the secondary file i think.

As for the GUI I agree that most script writers will already have editors that they prefer and anything we provide in wicket won’t be nearly as powerful. It could still find some utility in cases where the user doesn’t have access to the server file system. There is also a rest api to get/put the contents of scripts, so those users could use that to download locally, edit, and then push back.

Yeah, that’s precisely what I was aiming at with the GUI. I don’t think anybody will code in the editor, in fact a text area would suffice,
but not having access to the server file system is oh so common in enviroments where the OGC service managers are not the
same people as the physical server admin.

On reliability: From my work on GeoNode and gsconfig.py I know already that it’s quite easy to produce incorrect configurations through the REST API, which GeoServer will happily publish. While I am able (and mostly willing :wink: ) to dig into the datadir and delete some XML when I botch a REST request and have GeoServer telling me that I can’t create a layergroup because it already exists, but can’t delete it because it doesn’t exist, I think that we are asking for trouble by encouraging scripting against the catalog while the catalog’s validation is at its current level of completeness. Remember, by enabling scripting we are basically asking for developers to start developing against GeoServer who have no idea how to use a Java debugger or read a stack trace.

Unfortunately most of these issues never seem to make it into bug reports or patches? It is certainly a valid concern but at the same time if we wait for the underpinnings of geoserver to be perfect we will never go anywhere. The catalog api will never be perfect unfortunately and there will always be some way to shoot ones self in the foot. The counter argument is that opening it up to a different level of access will hopefully help harden it and flesh out more these configuration issues.

Yep, agreed on this one as well. In any case, most of the scripting extensions I hear about are not dealing with the configuration.
That said, +1 on harderning the catalog too, but I agree we need bug reports for that.

Cheers

Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf



Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.