[Geoserver-users] Pre-Proposal: WPS Extension "GetExecutions" Operation

Dear lists,
I would like here to ask your opinion about a possible extension/improvement of the GeoServer WPS Service.

The main goal of this proposal is to allow a client to recognize the list of WPS Executions through a simple request to a WPS Operation. What we would like to achieve would be something similar to this:

image.png

The client makes a simple “GetExecutions” request to the WPS Server, in order to get back an XML document containing the list of current Execution Statuses.

Ideally should be possible also to filter the “GetExecutions” request along with simple parameters, in order to refine the output and get back only the executions status we are looking for.

Adding a bit more to this, if AUTHORIZATION headers are sent along with the “GetExecutions” request, the WPS Server should be able, if a security subsystem is available and enable on the latter, of provide also the list of eventually private/protected resources to the client itself.

Part of this proposal is also to review and improve the “lineage” option of the WPS service, allowing a client to retrieve the Execute Inputs values provided to the process Identifier.

Thoughts?
···

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Alessio,
a few questions to help clarify your proposal:

  • Right now the list of executions is visible only to the administrator via the GUI. How does this request work authorization wise? Anyone can see all executions? Or only the ones he started as an authenticated user?
  • What info gets returned? Same as in the GUI right now? More?
  • How does one filter the list of returned values?
    Cheers

Andrea

image.png

···

On Thu, Jul 12, 2018 at 11:12 AM, Alessio Fabiani <alessio.fabiani@anonymised.com> wrote:

Dear lists,
I would like here to ask your opinion about a possible extension/improvement of the GeoServer WPS Service.

The main goal of this proposal is to allow a client to recognize the list of WPS Executions through a simple request to a WPS Operation. What we would like to achieve would be something similar to this:

image.png

The client makes a simple “GetExecutions” request to the WPS Server, in order to get back an XML document containing the list of current Execution Statuses.

Ideally should be possible also to filter the “GetExecutions” request along with simple parameters, in order to refine the output and get back only the executions status we are looking for.

Adding a bit more to this, if AUTHORIZATION headers are sent along with the “GetExecutions” request, the WPS Server should be able, if a security subsystem is available and enable on the latter, of provide also the list of eventually private/protected resources to the client itself.

Part of this proposal is also to review and improve the “lineage” option of the WPS service, allowing a client to retrieve the Execute Inputs values provided to the process Identifier.

Thoughts?

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.382…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hello Andrea,
thanks for your questions. Let me try to clarify a bit more what we have in mind right now.

Executions List
At a first glance I was thinking to the possibility of having some “public” executions, somehow, but digging a bit more on the proposal this does not make much sense.

The new operation should return only the list of available Executions the logged in user has started, except in the case it is an Administrator. In that case he will be able to get the whole list.

StatusInfo Response
Yes, roughly it should return what we see on the GUI, possibly with a bit more information (when/if available) as detailed in the table below

Names

Definition

Data type and values

Multiplicity and use

JobID

Unambiguously identifier of a execution job within a WPS instance.

Character String a

One (mandatory)

Identifier

Unambiguously identifier of a process within a WPS instance.

ows:Identifier

One (mandatory)

Status

Well-known identifier describing the status of the job.

Character String b

One (mandatory)

ExpirationDate

Date and time by which the job and its results will be no longer accessible. c

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

EstimatedCompletion

Date and time by which the processing job will be finished.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

NextPoll

Date and time for the next suggested status polling.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

PercentCompleted

Percentage of process that has been completed.

Integer{0…100} d

Zero or one (optional) Include if available.

a) Particularly suitable JobIDs are UUIDs or monotonic identifiers such as unique timestamps. If the privacy of a Processing Job is imperative, the JobID should be non-guessable.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

c) This element will usually become available when the execution has finished (Status = “finished”).

d) Zero (0) means the execution has just started, and 100 means the job is complete. This value is informative only without any accuracy guarantees.

GetExecutions Request
In order to filter and refine the Request, I was wondering about a set of simple Request parameters as depicted in the diagram below

image.png

which are better detailed in the table here below

Names

Definition

Data type and values

Multiplicity and use

Owner

Unambiguous identifier of a user within a WPS instance.

Filters out all the jobs not anonymous or not belonging to the specified user.

Character String a

Zero or one (optional).

Identifier

Unambiguous identifier of a process within a WPS instance.

Filters out all the jobs not belonging to the specified process identifier.

ows:Identifier Value shall be one of the process identifiers listed in the ProcessSummary elements in the Capabilities document.

Zero or one (optional).

Status

Well-known identifier describing the status of the job.

Filters out all the jobs with an Execution Status different from the one specified.

String{PENDING, RUNNING, FAILED, CANCELED, FINISHED} b

Zero or one (optional).

OrderBy

One of the OWNER, IDENTIFIER, STATUS.

The Response Document will be ordered accordingly to the value specified. If not value has been specified, the list of Processed will be ordered by Identifier and ExecutionDate.

String{OWNER, IDENTIFIER, STATUS}

Zero or one (optional).

a) It must match one of the usernames available on the Server, if any available. GetStatusesListResponse contents may vary accordingly to security constraints.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

image.png

···

On Thu, Jul 12, 2018 at 11:12 AM, Alessio Fabiani <alessio.fabiani@anonymised.com> wrote:

Dear lists,
I would like here to ask your opinion about a possible extension/improvement of the GeoServer WPS Service.

The main goal of this proposal is to allow a client to recognize the list of WPS Executions through a simple request to a WPS Operation. What we would like to achieve would be something similar to this:

image.png

The client makes a simple “GetExecutions” request to the WPS Server, in order to get back an XML document containing the list of current Execution Statuses.

Ideally should be possible also to filter the “GetExecutions” request along with simple parameters, in order to refine the output and get back only the executions status we are looking for.

Adding a bit more to this, if AUTHORIZATION headers are sent along with the “GetExecutions” request, the WPS Server should be able, if a security subsystem is available and enable on the latter, of provide also the list of eventually private/protected resources to the client itself.

Part of this proposal is also to review and improve the “lineage” option of the WPS service, allowing a client to retrieve the Execute Inputs values provided to the process Identifier.

Thoughts?

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Alessio,
thanks for the useful feedback. Notes inline.

image.png

···

On Thu, Jul 12, 2018 at 12:30 PM, Alessio Fabiani <alessio.fabiani@anonymised.com> wrote:

Executions List
At a first glance I was thinking to the possibility of having some “public” executions, somehow, but digging a bit more on the proposal this does not make much sense.

The new operation should return only the list of available Executions the logged in user has started, except in the case it is an Administrator. In that case he will be able to get the whole list.

Yep, makes sense.

StatusInfo Response
Yes, roughly it should return what we see on the GUI, possibly with a bit more information (when/if available) as detailed in the table below

Names

Definition

Data type and values

Multiplicity and use

JobID

Unambiguously identifier of a execution job within a WPS instance.

Character String a

One (mandatory)

Identifier

Unambiguously identifier of a process within a WPS instance.

ows:Identifier

One (mandatory)

Status

Well-known identifier describing the status of the job.

Character String b

One (mandatory)

ExpirationDate

Date and time by which the job and its results will be no longer accessible. c

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

EstimatedCompletion

Date and time by which the processing job will be finished.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

NextPoll

Date and time for the next suggested status polling.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

PercentCompleted

Percentage of process that has been completed.

Integer{0…100} d

Zero or one (optional) Include if available.

a) Particularly suitable JobIDs are UUIDs or monotonic identifiers such as unique timestamps. If the privacy of a Processing Job is imperative, the JobID should be non-guessable.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

c) This element will usually become available when the execution has finished (Status = “finished”).

d) Zero (0) means the execution has just started, and 100 means the job is complete. This value is informative only without any accuracy guarantees.

How will you guess the estimated completion? Elapsed time vs percentage complete?
In general GeoServer has a hard time with the percentage complete calculation, as the overall process execution
goes through 3 phases:

  • Parsing the inputs and turning them into Java classes suitable for process consumption. This part is given a fixed percentage. If all the inputs are simple (string, number, dates) the % is small, if the inputs are complexes (have to be parsed and may be large or retrieved from remote) I think it’s something like 30%, but you’d have to check in the code.
  • Actual execution, consuming java objects and producing other java objects. This is all over the place, some processes are streaming and give you back a GridCoverage2D backed by a deferred execution or a FeatureCollection that will compute the results when traversed (both to avoid storing a ton of memory), others have no progress support at all, only a subset are actually doing the computation during execute and with reasonable progress support.
  • Result encoding. This is similar to input parsing, if the result is complex the final encoding is given a large-ish percentage of the overall time, also considering that the real execution might happen here as we scroll over the feature collection or encode the grid coverage into a tiff.
    I believe you’re mostly interested in RawInput/RawOutput, not sure how well they are handled by the above “guessing” machinery, I’d suggest to have a look.

GetExecutions Request
In order to filter and refine the Request, I was wondering about a set of simple Request parameters as depicted in the diagram below

image.png

which are better detailed in the table here below

Names

Definition

Data type and values

Multiplicity and use

Owner

Unambiguous identifier of a user within a WPS instance.

Filters out all the jobs not anonymous or not belonging to the specified user.

Character String a

Zero or one (optional).

Identifier

Unambiguous identifier of a process within a WPS instance.

Filters out all the jobs not belonging to the specified process identifier.

ows:Identifier Value shall be one of the process identifiers listed in the ProcessSummary elements in the Capabilities document.

Zero or one (optional).

Status

Well-known identifier describing the status of the job.

Filters out all the jobs with an Execution Status different from the one specified.

String{PENDING, RUNNING, FAILED, CANCELED, FINISHED} b

Zero or one (optional).

OrderBy

One of the OWNER, IDENTIFIER, STATUS.

The Response Document will be ordered accordingly to the value specified. If not value has been specified, the list of Processed will be ordered by Identifier and ExecutionDate.

String{OWNER, IDENTIFIER, STATUS}

Zero or one (optional).

a) It must match one of the usernames available on the Server, if any available. GetStatusesListResponse contents may vary accordingly to security constraints.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

Makes sense yes.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Andrea,
not sure this is useful for our real use case. In our case we can send this estimate from the backend. In particular we are using the “wps-remote” plugin which completely relies on backend processes decoupled by an XMPP layer.

Nevertheless, my intention is to include this parameter on the proposal, even though this will be optional, since it is possible that some Processes instances in the future might use it and provide really useful information to the client.

image.png

···

On Thu, Jul 12, 2018 at 12:30 PM, Alessio Fabiani <alessio.fabiani@anonymised.com> wrote:

Executions List
At a first glance I was thinking to the possibility of having some “public” executions, somehow, but digging a bit more on the proposal this does not make much sense.

The new operation should return only the list of available Executions the logged in user has started, except in the case it is an Administrator. In that case he will be able to get the whole list.

Yep, makes sense.

StatusInfo Response
Yes, roughly it should return what we see on the GUI, possibly with a bit more information (when/if available) as detailed in the table below

Names

Definition

Data type and values

Multiplicity and use

JobID

Unambiguously identifier of a execution job within a WPS instance.

Character String a

One (mandatory)

Identifier

Unambiguously identifier of a process within a WPS instance.

ows:Identifier

One (mandatory)

Status

Well-known identifier describing the status of the job.

Character String b

One (mandatory)

ExpirationDate

Date and time by which the job and its results will be no longer accessible. c

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

EstimatedCompletion

Date and time by which the processing job will be finished.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

NextPoll

Date and time for the next suggested status polling.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

PercentCompleted

Percentage of process that has been completed.

Integer{0…100} d

Zero or one (optional) Include if available.

a) Particularly suitable JobIDs are UUIDs or monotonic identifiers such as unique timestamps. If the privacy of a Processing Job is imperative, the JobID should be non-guessable.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

c) This element will usually become available when the execution has finished (Status = “finished”).

d) Zero (0) means the execution has just started, and 100 means the job is complete. This value is informative only without any accuracy guarantees.

How will you guess the estimated completion? Elapsed time vs percentage complete?
In general GeoServer has a hard time with the percentage complete calculation, as the overall process execution
goes through 3 phases:

  • Parsing the inputs and turning them into Java classes suitable for process consumption. This part is given a fixed percentage. If all the inputs are simple (string, number, dates) the % is small, if the inputs are complexes (have to be parsed and may be large or retrieved from remote) I think it’s something like 30%, but you’d have to check in the code.
  • Actual execution, consuming java objects and producing other java objects. This is all over the place, some processes are streaming and give you back a GridCoverage2D backed by a deferred execution or a FeatureCollection that will compute the results when traversed (both to avoid storing a ton of memory), others have no progress support at all, only a subset are actually doing the computation during execute and with reasonable progress support.
  • Result encoding. This is similar to input parsing, if the result is complex the final encoding is given a large-ish percentage of the overall time, also considering that the real execution might happen here as we scroll over the feature collection or encode the grid coverage into a tiff.
    I believe you’re mostly interested in RawInput/RawOutput, not sure how well they are handled by the above “guessing” machinery, I’d suggest to have a look.

GetExecutions Request
In order to filter and refine the Request, I was wondering about a set of simple Request parameters as depicted in the diagram below

image.png

which are better detailed in the table here below

Names

Definition

Data type and values

Multiplicity and use

Owner

Unambiguous identifier of a user within a WPS instance.

Filters out all the jobs not anonymous or not belonging to the specified user.

Character String a

Zero or one (optional).

Identifier

Unambiguous identifier of a process within a WPS instance.

Filters out all the jobs not belonging to the specified process identifier.

ows:Identifier Value shall be one of the process identifiers listed in the ProcessSummary elements in the Capabilities document.

Zero or one (optional).

Status

Well-known identifier describing the status of the job.

Filters out all the jobs with an Execution Status different from the one specified.

String{PENDING, RUNNING, FAILED, CANCELED, FINISHED} b

Zero or one (optional).

OrderBy

One of the OWNER, IDENTIFIER, STATUS.

The Response Document will be ordered accordingly to the value specified. If not value has been specified, the list of Processed will be ordered by Identifier and ExecutionDate.

String{OWNER, IDENTIFIER, STATUS}

Zero or one (optional).

a) It must match one of the usernames available on the Server, if any available. GetStatusesListResponse contents may vary accordingly to security constraints.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

Makes sense yes.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

I forgot to mention that the processes running on the backend for the use case I was speaking about, are so heavy that the time spent to produce the outputs might be negligible.

image.png

···

On Thu, Jul 12, 2018 at 12:30 PM, Alessio Fabiani <alessio.fabiani@anonymised.com.> wrote:

Executions List
At a first glance I was thinking to the possibility of having some “public” executions, somehow, but digging a bit more on the proposal this does not make much sense.

The new operation should return only the list of available Executions the logged in user has started, except in the case it is an Administrator. In that case he will be able to get the whole list.

Yep, makes sense.

StatusInfo Response
Yes, roughly it should return what we see on the GUI, possibly with a bit more information (when/if available) as detailed in the table below

Names

Definition

Data type and values

Multiplicity and use

JobID

Unambiguously identifier of a execution job within a WPS instance.

Character String a

One (mandatory)

Identifier

Unambiguously identifier of a process within a WPS instance.

ows:Identifier

One (mandatory)

Status

Well-known identifier describing the status of the job.

Character String b

One (mandatory)

ExpirationDate

Date and time by which the job and its results will be no longer accessible. c

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

EstimatedCompletion

Date and time by which the processing job will be finished.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

NextPoll

Date and time for the next suggested status polling.

ISO-8601 date/time string in the form YYYY-MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC)

Zero or one (optional) Include if available.

PercentCompleted

Percentage of process that has been completed.

Integer{0…100} d

Zero or one (optional) Include if available.

a) Particularly suitable JobIDs are UUIDs or monotonic identifiers such as unique timestamps. If the privacy of a Processing Job is imperative, the JobID should be non-guessable.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

c) This element will usually become available when the execution has finished (Status = “finished”).

d) Zero (0) means the execution has just started, and 100 means the job is complete. This value is informative only without any accuracy guarantees.

How will you guess the estimated completion? Elapsed time vs percentage complete?
In general GeoServer has a hard time with the percentage complete calculation, as the overall process execution
goes through 3 phases:

  • Parsing the inputs and turning them into Java classes suitable for process consumption. This part is given a fixed percentage. If all the inputs are simple (string, number, dates) the % is small, if the inputs are complexes (have to be parsed and may be large or retrieved from remote) I think it’s something like 30%, but you’d have to check in the code.
  • Actual execution, consuming java objects and producing other java objects. This is all over the place, some processes are streaming and give you back a GridCoverage2D backed by a deferred execution or a FeatureCollection that will compute the results when traversed (both to avoid storing a ton of memory), others have no progress support at all, only a subset are actually doing the computation during execute and with reasonable progress support.
  • Result encoding. This is similar to input parsing, if the result is complex the final encoding is given a large-ish percentage of the overall time, also considering that the real execution might happen here as we scroll over the feature collection or encode the grid coverage into a tiff.
    I believe you’re mostly interested in RawInput/RawOutput, not sure how well they are handled by the above “guessing” machinery, I’d suggest to have a look.

GetExecutions Request
In order to filter and refine the Request, I was wondering about a set of simple Request parameters as depicted in the diagram below

image.png

which are better detailed in the table here below

Names

Definition

Data type and values

Multiplicity and use

Owner

Unambiguous identifier of a user within a WPS instance.

Filters out all the jobs not anonymous or not belonging to the specified user.

Character String a

Zero or one (optional).

Identifier

Unambiguous identifier of a process within a WPS instance.

Filters out all the jobs not belonging to the specified process identifier.

ows:Identifier Value shall be one of the process identifiers listed in the ProcessSummary elements in the Capabilities document.

Zero or one (optional).

Status

Well-known identifier describing the status of the job.

Filters out all the jobs with an Execution Status different from the one specified.

String{PENDING, RUNNING, FAILED, CANCELED, FINISHED} b

Zero or one (optional).

OrderBy

One of the OWNER, IDENTIFIER, STATUS.

The Response Document will be ordered accordingly to the value specified. If not value has been specified, the list of Processed will be ordered by Identifier and ExecutionDate.

String{OWNER, IDENTIFIER, STATUS}

Zero or one (optional).

a) It must match one of the usernames available on the Server, if any available. GetStatusesListResponse contents may vary accordingly to security constraints.

b) The basic status set is defined in http://docs.opengeospatial.org/is/14-065/14-065.html Table 3. Additional states may be defined by certain operations or extensions of this standard.

Makes sense yes.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

On Fri, Jul 13, 2018 at 3:04 PM, Alessio Fabiani <
alessio.fabiani@anonymised.com> wrote:

Hi Andrea,
not sure this is useful for our real use case. In our case we can send
this estimate from the backend. In particular we are using the "wps-remote"
plugin which completely relies on backend processes decoupled by an XMPP
layer.

Eh, but you're still going through the general WPS machinery, so... you'll
have to find a way to provide that information for each and every possible
process. Looking forward to see how it's done in the proposal.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information. == Ing. Andrea Aime @geowolf Technical Lead
GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39
0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549
http://www.geo-solutions.it http://twitter.com/geosolutions_it
------------------------------------------------------- *Con riferimento
alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 -
Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo contenuto, gli eventuali
allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i
destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per
errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is intended
only for the person or entity to which it is addressed and may contain
information that is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European Regulation 2016/679
“GDPR” - copying, dissemination or use of this e-mail or the information
herein by anyone other than the intended recipient is prohibited. If you
have received this email by mistake, please notify us immediately by
telephone or e-mail.*

On Fri, Jul 13, 2018 at 3:06 PM, Alessio Fabiani <
alessio.fabiani@anonymised.com> wrote:

I forgot to mention that the processes running on the backend for the use
case I was speaking about, are so heavy that the time spent to produce the
outputs might be negligible.

Yep, just remember that you're still going through the WPS general
machinery like local processes, so think about the changes you have in
mind in general terms (how that affects your proposal, I don't know, just
suggesting something to consider).

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information. == Ing. Andrea Aime @geowolf Technical Lead
GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39
0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549
http://www.geo-solutions.it http://twitter.com/geosolutions_it
------------------------------------------------------- *Con riferimento
alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 -
Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo contenuto, gli eventuali
allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i
destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per
errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is intended
only for the person or entity to which it is addressed and may contain
information that is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European Regulation 2016/679
“GDPR” - copying, dissemination or use of this e-mail or the information
herein by anyone other than the intended recipient is prohibited. If you
have received this email by mistake, please notify us immediately by
telephone or e-mail.*