[Geoserver-devel] StreamingRenderer and usage without JAI

Hi folks,

sorry for cross-posting

recently - during the FOSSGIS/OSGeo codesprint in Bonn - we tried to use StreamingRenderer in an Environment without JAI installation (from within uDig)

However, it looks like the Renderer loads classes that reqired JAI and I’m wondering if and how we could refactor it to make JAI optional without breaking renderer for other existing Applications such as GeoServer.

The idea is to make Coverage-rendere/reader support optional to use it for feature-rendering only.

At this stage I’ll collect ideas and hints. Its very important to know which project are using this renderer and how.

Looking forward fro your input and insights

Many Thanks

Frank

Hi Frank,
first, I’m assuming you only want to render vectors? There is no way to read and render rasters without JAI around as far as I know,
making one would likely be a lot of work.

If you’re just looking at vectors, I’m guessing you’re being bit by static initializations… however the rabbit hole might be very very deep,

trying to avoid JAI is unusual for our codebase (although I understand why it might be desirable).

So two words of warning:

  • Might be hard to remove all static init JAI dependencies
  • You should probably look for some way to add a test, or it will easily regress in no time due to people overlooking this need
    Cheers

Andrea

···

On Tue, Mar 27, 2018 at 10:33 PM, Frank Gasdorf via Geoserver-devel <geoserver-devel@lists.sourceforge.net> wrote:

Hi folks,

sorry for cross-posting

recently - during the FOSSGIS/OSGeo codesprint in Bonn - we tried to use StreamingRenderer in an Environment without JAI installation (from within uDig)

However, it looks like the Renderer loads classes that reqired JAI and I’m wondering if and how we could refactor it to make JAI optional without breaking renderer for other existing Applications such as GeoServer.

The idea is to make Coverage-rendere/reader support optional to use it for feature-rendering only.

At this stage I’ll collect ideas and hints. Its very important to know which project are using this renderer and how.

Looking forward fro your input and insights

Many Thanks

Frank


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Comments inline …

···

2018-03-28 12:49 GMT+02:00 Andrea Aime <andrea.aime@anonymised.com>:

Hi Frank,
first, I’m assuming you only want to render vectors? There is no way to read and render rasters without JAI around as far as I know,
making one would likely be a lot of work.

Totally right, I’d like to render features only and make raster support optional because of JAI dependencies (and licencing issues that might be obsolte in near future).

If you’re just looking at vectors, I’m guessing you’re being bit by static initializations… however the rabbit hole might be very very deep,

trying to avoid JAI is unusual for our codebase (although I understand why it might be desirable).

The intention is to get an application running without JAI which implies to have no raster support. This is fine for the first shot and maybe helps to transform StreamingRenderer “god-class” to something
that is easier to maintain and might opens up chances to configure systems (plug-able)

So two words of warning:

  • Might be hard to remove all static init JAI dependencies
  • You should probably look for some way to add a test, or it will easily regress in no time due to people overlooking this need

First finding : the rabbit hole is quite hidden and I’mn still looking for it. Are you saying it would help to add a test-case first that might fail all the time to force refactorings?

I guess its worth to have a test to see if refactor works as expected…

have you heard about similiar requirments or even opposite in any way?

Cheers

Andrea

Many thanks

Frank

On Tue, Mar 27, 2018 at 10:33 PM, Frank Gasdorf via Geoserver-devel <geoserver-devel@anonymised.comsourceforge.net> wrote:

Hi folks,

sorry for cross-posting

recently - during the FOSSGIS/OSGeo codesprint in Bonn - we tried to use StreamingRenderer in an Environment without JAI installation (from within uDig)

However, it looks like the Renderer loads classes that reqired JAI and I’m wondering if and how we could refactor it to make JAI optional without breaking renderer for other existing Applications such as GeoServer.

The idea is to make Coverage-rendere/reader support optional to use it for feature-rendering only.

At this stage I’ll collect ideas and hints. Its very important to know which project are using this renderer and how.

Looking forward fro your input and insights

Many Thanks

Frank


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

On Wed, Mar 28, 2018 at 1:16 PM, Frank Gasdorf <fgdrf@anonymised.com>
wrote:

The intention is to get an application running without JAI which implies
to have no raster support. This is fine for the first shot and maybe helps
to transform StreamingRenderer "god-class" to something
that is easier to maintain and might opens up chances to configure systems
(plug-able)

Hmm... I doubt it will serve your specific purpose, the raster rendering is
already factored out in GridCoverageRenderer,
although, going by memory, there are still lingering bits in the rendering
transformations paths.

About a refactor, please don't unless it's absolutely necessary or you're
going to become the new maintainer of the module,
I'm not trying to pick on you but I had enough of people coming with big
"I'll make it better for you" refactors and then
walking away, leaving me to struggle for months (or years, in the case of
the WMS module refactor) trying to figure out
where functionality is gone and how I'm supposed to extend the new model.

st finding : the rabbit hole is quite hidden and I'mn still looking for

it. Are you saying it would help to add a test-case first that might fail
all the time to force refactorings?

Yes, a module that only contains a test and sets up the classpath in such a
way that JAI is guaranteed not to be there,
with one or more tests checking your use case.
If this is not in the build your work will degrade over time with nobody
noticing.

have you heard about similiar requirments or even opposite in any way?

I heard Jody speaking about it, but no,personally never had the need, nor
had on behalf of a customer.

Cheers
Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

Based on Andrea’s response it may be best to take a copy of streaming renderer, and make a second implementation. There some cool special effects (alpha blending, etc…) that take advantage of interactions between layers that would not translate to the uDig case where we run several streaming renderers in parallel.

Andrea is correct that it is a very difficult class, we would probably need a trial run such as this before being able to propose any kind of refactor that would not make it worse.

···

On 28 March 2018 at 05:16, Andrea Aime <andrea.aime@anonymised.com> wrote:


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


GeoTools-Devel mailing list
GeoTools-Devel@anonymised.com66…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


Jody Garnett

On Wed, Mar 28, 2018 at 1:16 PM, Frank Gasdorf <fgdrf@anonymised.com693…> wrote:

Hmm… I doubt it will serve your specific purpose, the raster rendering is already factored out in GridCoverageRenderer,
although, going by memory, there are still lingering bits in the rendering transformations paths.

About a refactor, please don’t unless it’s absolutely necessary or you’re going to become the new maintainer of the module,
I’m not trying to pick on you but I had enough of people coming with big “I’ll make it better for you” refactors and then
walking away, leaving me to struggle for months (or years, in the case of the WMS module refactor) trying to figure out
where functionality is gone and how I’m supposed to extend the new model.

Yes, a module that only contains a test and sets up the classpath in such a way that JAI is guaranteed not to be there,
with one or more tests checking your use case.
If this is not in the build your work will degrade over time with nobody noticing.

I heard Jody speaking about it, but no,personally never had the need, nor had on behalf of a customer.

Cheers

Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

The intention is to get an application running without JAI which implies to have no raster support. This is fine for the first shot and maybe helps to transform StreamingRenderer “god-class” to something

that is easier to maintain and might opens up chances to configure systems (plug-able)

st finding : the rabbit hole is quite hidden and I’mn still looking for it. Are you saying it would help to add a test-case first that might fail all the time to force refactorings?

have you heard about similiar requirments or even opposite in any way?

On Fri, Mar 30, 2018 at 2:31 AM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Andrea is correct that it is a very difficult class, we would probably
need a trial run such as this before being able to propose any kind of
refactor that would not make it worse.

As usual best to do a proposal (even by mail) before a PR, and limit the
refactorings to automatic ones as much as possible.
Large refactors almost always come with hard to spot regressions, this is
particularly true for a class as poorly tested as the StreamingRenderer
(sorry, when I inherited it, it only had a few tests based on manual visual
inspection, been working hard to add actual ones, but we're not even close
to
good coverage).

So, if a refactor is agreed with and done, I expect those working on the
refactor to be available for fixing the regressions for at least a few
months down the line.
Which is in line with the document I wrote a couple of years ago for
GeoServer:
https://github.com/geoserver/geoserver/wiki/Some-ideas-about-minimum-community-software-responsibility-and-participation

The document fell on deaf ears, but personally I'll give no support
whatsoever to people not respecting it (this work falls under the
"other contributors providing large changes" I believe).

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

Frank how would you feel about a cut down version of feature rendering in the uDig codebase, as a temporary measure. If JAI is available uDig could use streaming renderer like normal?

···

On 29 March 2018 at 17:31, Jody Garnett <jody.garnett@anonymised.com> wrote:

Based on Andrea’s response it may be best to take a copy of streaming renderer, and make a second implementation. There some cool special effects (alpha blending, etc…) that take advantage of interactions between layers that would not translate to the uDig case where we run several streaming renderers in parallel.

Andrea is correct that it is a very difficult class, we would probably need a trial run such as this before being able to propose any kind of refactor that would not make it worse.


Jody Garnett


Jody Garnett

On 28 March 2018 at 05:16, Andrea Aime <andrea.aime@anonymised.com> wrote:


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


GeoTools-Devel mailing list
GeoTools-Devel@anonymised.comge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

On Wed, Mar 28, 2018 at 1:16 PM, Frank Gasdorf <fgdrf@anonymised.com> wrote:

Hmm… I doubt it will serve your specific purpose, the raster rendering is already factored out in GridCoverageRenderer,
although, going by memory, there are still lingering bits in the rendering transformations paths.

About a refactor, please don’t unless it’s absolutely necessary or you’re going to become the new maintainer of the module,
I’m not trying to pick on you but I had enough of people coming with big “I’ll make it better for you” refactors and then
walking away, leaving me to struggle for months (or years, in the case of the WMS module refactor) trying to figure out
where functionality is gone and how I’m supposed to extend the new model.

Yes, a module that only contains a test and sets up the classpath in such a way that JAI is guaranteed not to be there,
with one or more tests checking your use case.
If this is not in the build your work will degrade over time with nobody noticing.

I heard Jody speaking about it, but no,personally never had the need, nor had on behalf of a customer.

Cheers

Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

The intention is to get an application running without JAI which implies to have no raster support. This is fine for the first shot and maybe helps to transform StreamingRenderer “god-class” to something

that is easier to maintain and might opens up chances to configure systems (plug-able)

st finding : the rabbit hole is quite hidden and I’mn still looking for it. Are you saying it would help to add a test-case first that might fail all the time to force refactorings?

have you heard about similiar requirments or even opposite in any way?