[Geoserver-devel] Custom URL protocol handling

Hello Geoserver Devs,

I'm trying to extend GeoTIFF support to be able to read image files directly from a Hadoop Distributed File System. Our idea was to write ImageInputStream and ImageInputStreamSPI classes that would know how to handle the 'hdfs' protocol.
When adding a GeoTIFF datastore through the GeoServer web UI, I received the error that 'hdfs://localhost:9000/sample.tif' is not a valid URL.

How can I expand GeoServer's notion of a valid URL?
Or, is there a better way to implement a hdfs interface?

Optimally, we would like to extend this hdfs access functionality to NetCDF as well, but GeoTIFFs are our primary concern.

Thank you,
Michael Matthews

Hi all,

As a little more context, we are looking for the most suitable way to HDFS-enable the existing raster plugins.

With some of the javax ImageIO SPI interfaces, it looked like we'd be close to being able to have a drop-in solution. Otherwise, it looks like we may have to touch each plugin.

Thoughts?

Cheers,

Jim

On 04/01/2016 11:06 AM, Michael Matthews wrote:

Hello Geoserver Devs,

I'm trying to extend GeoTIFF support to be able to read image files
directly from a Hadoop Distributed File System. Our idea was to write
ImageInputStream and ImageInputStreamSPI classes that would know how to
handle the 'hdfs' protocol.
When adding a GeoTIFF datastore through the GeoServer web UI, I received
the error that 'hdfs://localhost:9000/sample.tif' is not a valid URL.

How can I expand GeoServer's notion of a valid URL?
Or, is there a better way to implement a hdfs interface?

Optimally, we would like to extend this hdfs access functionality to
NetCDF as well, but GeoTIFFs are our primary concern.

Thank you,
Michael Matthews

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Hi Jim and Michael,
Great idea regarding the GeoServer raster data store being more friendly to other URLs such as those using HDFS protocol. I’d like to add that it would be nice to be able to use some other format or perhaps a custom defined protocol for the URL string. We originally looked at encoding data store connection parameters into the URL but ran into this same validation issue in GeoServer. It seemed our best option at the time was to take a look at how PGRaster works, but I’m definitely interested in any evolution of raster configuration going on.

Generally, I like the idea of having more flexibility for configuring the raster formats to include the ability to use ‘hdfs’ protocol in the URL, and how that can be expanded.

Thanks,

Rich

···

On Mon, Apr 4, 2016 at 9:34 AM, Jim Hughes <jnh5y@anonymised.com> wrote:

Hi all,

As a little more context, we are looking for the most suitable way to
HDFS-enable the existing raster plugins.

With some of the javax ImageIO SPI interfaces, it looked like we’d be
close to being able to have a drop-in solution. Otherwise, it looks
like we may have to touch each plugin.

Thoughts?

Cheers,

Jim

On 04/01/2016 11:06 AM, Michael Matthews wrote:

Hello Geoserver Devs,

I’m trying to extend GeoTIFF support to be able to read image files
directly from a Hadoop Distributed File System. Our idea was to write
ImageInputStream and ImageInputStreamSPI classes that would know how to
handle the ‘hdfs’ protocol.
When adding a GeoTIFF datastore through the GeoServer web UI, I received
the error that ‘hdfs://localhost:9000/sample.tif’ is not a valid URL.

How can I expand GeoServer’s notion of a valid URL?
Or, is there a better way to implement a hdfs interface?

Optimally, we would like to extend this hdfs access functionality to
NetCDF as well, but GeoTIFFs are our primary concern.

Thank you,
Michael Matthews


Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel



Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

In general there's really no reason why we can't support a generic handler protocol like the Hadoop InputFormats do. For example you can just specifiy s3a:// or hdfs:// or file:// etc. Java has a setup to deal with handling URL and URI syntax and register handlers for it. We can make that be a runtime dep for a geoserver if it isn't already.

Andrew

On Mon, Apr 04, 2016 at 09:56:37AM -0400, Rich Fecher wrote:

Hi Jim and Michael,
Great idea regarding the GeoServer raster data store being more friendly to
other URLs such as those using HDFS protocol. I'd like to add that it
would be nice to be able to use some other format or perhaps a custom
defined protocol for the URL string. We originally looked at encoding data
store connection parameters into the URL but ran into this same validation
issue in GeoServer. It seemed our best option at the time was to take a
look at how PGRaster <https://github.com/ngageoint/geowave/issues/608&gt;
works, but I'm definitely interested in any evolution of raster
configuration going on.

Generally, I like the idea of having more flexibility for configuring the
raster formats to include the ability to use 'hdfs' protocol in the URL,
and how that can be expanded.

Thanks,
Rich

On Mon, Apr 4, 2016 at 9:34 AM, Jim Hughes <jnh5y@anonymised.com> wrote:

> Hi all,
>
> As a little more context, we are looking for the most suitable way to
> HDFS-enable the existing raster plugins.
>
> With some of the javax ImageIO SPI interfaces, it looked like we'd be
> close to being able to have a drop-in solution. Otherwise, it looks
> like we may have to touch each plugin.
>
> Thoughts?
>
> Cheers,
>
> Jim
>
> On 04/01/2016 11:06 AM, Michael Matthews wrote:
> > Hello Geoserver Devs,
> >
> > I'm trying to extend GeoTIFF support to be able to read image files
> > directly from a Hadoop Distributed File System. Our idea was to write
> > ImageInputStream and ImageInputStreamSPI classes that would know how to
> > handle the 'hdfs' protocol.
> > When adding a GeoTIFF datastore through the GeoServer web UI, I received
> > the error that 'hdfs://localhost:9000/sample.tif' is not a valid URL.
> >
> > How can I expand GeoServer's notion of a valid URL?
> > Or, is there a better way to implement a hdfs interface?
> >
> > Optimally, we would like to extend this hdfs access functionality to
> > NetCDF as well, but GeoTIFFs are our primary concern.
> >
> > Thank you,
> > Michael Matthews
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> > http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140
> > _______________________________________________
> > Geoserver-devel mailing list
> > Geoserver-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>

------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Hi Andrew,
that’s good news, it is exactly what the thread was making me wonder.
As far as I know/remember, all the URL validation and handling revolves against
the JVM native ability to handle a particular protocol.

That said, I’m not sure this has ever been tried, not at any level, but especially with imageio-ext,
so I don’t know what’s going to happen down there. It is something to try and discover I guess.

Cheers
Andrea

···

On Mon, Apr 4, 2016 at 4:22 PM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

In general there’s really no reason why we can’t support a generic handler protocol like the Hadoop InputFormats do. For example you can just specifiy s3a:// or hdfs:// or file:// etc. Java has a setup to deal with handling URL and URI syntax and register handlers for it. We can make that be a runtime dep for a geoserver if it isn’t already.

Andrew

On Mon, Apr 04, 2016 at 09:56:37AM -0400, Rich Fecher wrote:

Hi Jim and Michael,
Great idea regarding the GeoServer raster data store being more friendly to
other URLs such as those using HDFS protocol. I’d like to add that it
would be nice to be able to use some other format or perhaps a custom
defined protocol for the URL string. We originally looked at encoding data
store connection parameters into the URL but ran into this same validation
issue in GeoServer. It seemed our best option at the time was to take a
look at how PGRaster <https://github.com/ngageoint/geowave/issues/608>

works, but I’m definitely interested in any evolution of raster
configuration going on.

Generally, I like the idea of having more flexibility for configuring the
raster formats to include the ability to use ‘hdfs’ protocol in the URL,
and how that can be expanded.

Thanks,
Rich

On Mon, Apr 4, 2016 at 9:34 AM, Jim Hughes <jnh5y@anonymised.com> wrote:

Hi all,

As a little more context, we are looking for the most suitable way to
HDFS-enable the existing raster plugins.

With some of the javax ImageIO SPI interfaces, it looked like we’d be
close to being able to have a drop-in solution. Otherwise, it looks
like we may have to touch each plugin.

Thoughts?

Cheers,

Jim

On 04/01/2016 11:06 AM, Michael Matthews wrote:

Hello Geoserver Devs,

I’m trying to extend GeoTIFF support to be able to read image files
directly from a Hadoop Distributed File System. Our idea was to write
ImageInputStream and ImageInputStreamSPI classes that would know how to
handle the ‘hdfs’ protocol.
When adding a GeoTIFF datastore through the GeoServer web UI, I received
the error that ‘hdfs://localhost:9000/sample.tif’ is not a valid URL.

How can I expand GeoServer’s notion of a valid URL?
Or, is there a better way to implement a hdfs interface?

Optimally, we would like to extend this hdfs access functionality to
NetCDF as well, but GeoTIFFs are our primary concern.

Thank you,
Michael Matthews


Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel



Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel



Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel



Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

That said, I'm not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I'll ask Jim for more info and see if we can mock it up...while we're at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

> That said, I'm not sure this has ever been tried, not at any level, but
> especially with imageio-ext,
>

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

It is definitely possible we missed an extension point to register protocol handlers at the JVM level. I just hadn’t thought of it at the time and didn’t dig in, but I’ll keep it in mind. Also, I’m curious if that works in the case of ‘hdfs’ and ‘s3’ protocol because then I’d think it should work for us too. Thanks!

···

On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I’ll ask Jim for more info and see if we can mock it up…while we’re at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

That said, I’m not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.


Hi Rich,
it should be something along these lines (as said, never tried personally, was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers
Andrea

···

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com> wrote:

It is definitely possible we missed an extension point to register protocol handlers at the JVM level. I just hadn’t thought of it at the time and didn’t dig in, but I’ll keep it in mind. Also, I’m curious if that works in the case of ‘hdfs’ and ‘s3’ protocol because then I’d think it should work for us too. Thanks!

On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I’ll ask Jim for more info and see if we can mock it up…while we’re at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

That said, I’m not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.


==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded users to add URL handlers to their JVM or 2) handle this at the GeoTools/GeoServer level?

For #1, I get the sense that adding URL handlers is more challenging. Also, I doubt all installations would be ok with steps which add things to the JVM setup. On the plus side, it might require little or no code changes on the GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew points out, whenever a Java File is used directly, we might have to tweak code.

Thoughts?

Tentatively, I’m for the second approach.

Jim

···

On 04/04/2016 10:42 AM, Andrea Aime wrote:

Hi Rich,
it should be something along these lines (as said, never tried personally, was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers
Andrea

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com> wrote:

It is definitely possible we missed an extension point to register protocol handlers at the JVM level. I just hadn’t thought of it at the time and didn’t dig in, but I’ll keep it in mind. Also, I’m curious if that works in the case of ‘hdfs’ and ‘s3’ protocol because then I’d think it should work for us too. Thanks!

On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I’ll ask Jim for more info and see if we can mock it up…while we’re at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

That said, I’m not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.


==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)

I thinks HDFS and S3 are stable enough to have some sort of support that ships out of the box. The only worry I would have is the dependency chain that comes with HDFS. If it proves to be larger than we'd like we can just have good doc on installation...

I support #2 as much as possible too. Having all of the code work against InputStreams or URIs or other higher level abstractions is always more flexible and also allows for new datastores and memory-based systems, etc.

Overall, I think having native S3 and HDFS support positions GeoServer well in the "big data" world too. Another option is to have a profile that builds the war file with all the crazy cloud deps installed or something like that? I'd just like to have a "works out of the box" turn-key type solution for HDFS/S3/etc. in the future.

Andrew

On Mon, Apr 04, 2016 at 10:53:30AM -0400, Jim Hughes wrote:

Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded
users to add URL handlers to their JVM or 2) handle this at the
GeoTools/GeoServer level?

For #1, I get the sense that adding URL handlers is more
challenging. Also, I doubt all installations would be ok with steps
which add things to the JVM setup. On the plus side, it might
require little or no code changes on the GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew
points out, whenever a Java File is used directly, we might have to
tweak code.

Thoughts?

Tentatively, I'm for the second approach.

Jim

On 04/04/2016 10:42 AM, Andrea Aime wrote:
>Hi Rich,
>it should be something along these lines (as said, never tried
>personally, was just barely aware it should be possible):
>
>http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol
>
>Cheers
>Andrea
>
>
>On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com
><mailto:rfecher@anonymised.com>> wrote:
>
> It is definitely possible we missed an extension point to register
> protocol handlers at the JVM level. I just hadn't thought of it
> at the time and didn't dig in, but I'll keep it in mind. Also,
> I'm curious if that works in the case of 'hdfs' and 's3' protocol
> because then I'd think it should work for us too. Thanks!
>
> On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com
> <mailto:ahulbert@anonymised.com>> wrote:
>
> Yeah the only thing we will have to worry about is if the code
> is written against java.io.File instead of some other URL lib.
> At some point I guess we could dig in and look. I'll ask Jim
> for more info and see if we can mock it up...while we're at it
> we can do S3 or something too for another test.
>
> On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:
> > On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime
> <andrea.aime@anonymised.com
> <mailto:andrea.aime@anonymised.com>>
> > wrote:
> >
> > > That said, I'm not sure this has ever been tried, not at
> any level, but
> > > especially with imageio-ext,
> > >
> >
> > Sorry, mis-typed, I meant jai-imageio there
> >
> > Cheers
> > Andrea
> >
> > --
> > ==
> > GeoServer Professional Services from the experts! Visit
> > http://goo.gl/it488V for more information.
> > ==
> >
> > Ing. Andrea Aime
> > @geowolf
> > Technical Lead
> >
> > GeoSolutions S.A.S.
> > Via di Montramito 3/A
> > 55054 Massarosa (LU)
> > phone: +39 0584 962313 <tel:%2B39%200584%20962313>
> > fax: +39 0584 1660272 <tel:%2B39%200584%201660272>
> > mob: +39 339 8844549 <tel:%2B39%20%20339%208844549>
> >
> > http://www.geo-solutions.it
> > http://twitter.com/geosolutions_it
> >
> > *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
> >
> > Le informazioni contenute in questo messaggio di posta
> elettronica e/o
> > nel/i file/s allegato/i sono da considerarsi strettamente
> riservate. Il
> > loro utilizzo è consentito esclusivamente al destinatario
> del messaggio,
> > per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo
> > messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di
> > darcene notizia via e-mail e di procedere alla distruzione
> del messaggio
> > stesso, cancellandolo dal Vostro sistema. Conservare il
> messaggio stesso,
> > divulgarlo anche in parte, distribuirlo ad altri soggetti,
> copiarlo, od
> > utilizzarlo per finalità diverse, costituisce comportamento
> contrario ai
> > principi dettati dal D.Lgs. 196/2003.
> >
> >
> >
> > The information in this message and/or attachments, is
> intended solely for
> > the attention and use of the named addressee(s) and may be
> confidential or
> > proprietary in nature or covered by the provisions of
> privacy act
> > (Legislative Decree June, 30 2003, no.196 - Italy's New Data
> Protection
> > Code).Any use not in accord with its purpose, any
> disclosure, reproduction,
> > copying, distribution, or either dissemination, either whole
> or partial, is
> > strictly forbidden except previous formal approval of the named
> > addressee(s). If you are not the intended recipient, please
> contact
> > immediately the sender by telephone, fax or e-mail and
> delete the
> > information in this message that has been received in error.
> The sender
> > does not give any warranty or accept liability as the
> content, accuracy or
> > completeness of sent messages and accepts no
>responsibility for changes
> > made after they were sent or for other risks which arise as
> a result of
> > e-mail transmission, viruses, etc.
> >
> > -------------------------------------------------------
>
>
>
>
>
>--
>==
>GeoServer Professional Services from the experts! Visit
>http://goo.gl/it488V for more information.
>==
>
>Ing. Andrea Aime
>@geowolf
>Technical Lead
>
>GeoSolutions S.A.S.
>Via di Montramito 3/A
>55054 Massarosa (LU)
>phone: +39 0584 962313
>fax: +39 0584 1660272
>mob: +39 339 8844549
>
>http://www.geo-solutions.it
>http://twitter.com/geosolutions_it
>
>*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
>Le informazioni contenute in questo messaggio di posta elettronica
>e/o nel/i file/s allegato/i sono da considerarsi strettamente
>riservate. Il loro utilizzo è consentito esclusivamente al
>destinatario del messaggio, per le finalità indicate nel messaggio
>stesso. Qualora riceviate questo messaggio senza esserne il
>destinatario, Vi preghiamo cortesemente di darcene notizia via
>e-mail e di procedere alla distruzione del messaggio stesso,
>cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>divulgarlo anche in parte, distribuirlo ad altri soggetti,
>copiarlo, od utilizzarlo per finalità diverse, costituisce
>comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
>
>The information in this message and/or attachments, is intended
>solely for the attention and use of the named addressee(s) and may
>be confidential or proprietary in nature or covered by the
>provisions of privacy act (Legislative Decree June, 30 2003,
>no.196 - Italy's New Data Protection Code).Any use not in accord
>with its purpose, any disclosure, reproduction, copying,
>distribution, or either dissemination, either whole or partial, is
>strictly forbidden except previous formal approval of the named
>addressee(s). If you are not the intended recipient, please
>contact immediately the sender by telephone, fax or e-mail and
>delete the information in this message that has been received in
>error. The sender does not give any warranty or accept liability
>as the content, accuracy or completeness of sent messages and
>accepts no responsibility for changes made after they were sent
>or for other risks which arise as a result of e-mail transmission,
>viruses, etc.
>
>
>-------------------------------------------------------
>
>
>------------------------------------------------------------------------------
>
>
>_______________________________________________
>Geoserver-devel mailing list
>Geoserver-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/geoserver-devel

------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

To make this concrete, I think that any HDFS or S3 support would likely be a community module for a bit. Proposing for #2 would involve any API work required, and then we could update existing plugins as needed.

Cheers,

Jim

On 04/04/2016 11:03 AM, Andrew Hulbert wrote:

I thinks HDFS and S3 are stable enough to have some sort of support that ships out of the box. The only worry I would have is the dependency chain that comes with HDFS. If it proves to be larger than we'd like we can just have good doc on installation...

I support #2 as much as possible too. Having all of the code work against InputStreams or URIs or other higher level abstractions is always more flexible and also allows for new datastores and memory-based systems, etc.

Overall, I think having native S3 and HDFS support positions GeoServer well in the "big data" world too. Another option is to have a profile that builds the war file with all the crazy cloud deps installed or something like that? I'd just like to have a "works out of the box" turn-key type solution for HDFS/S3/etc. in the future.

Andrew

On Mon, Apr 04, 2016 at 10:53:30AM -0400, Jim Hughes wrote:

Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded
users to add URL handlers to their JVM or 2) handle this at the
GeoTools/GeoServer level?

For #1, I get the sense that adding URL handlers is more
challenging. Also, I doubt all installations would be ok with steps
which add things to the JVM setup. On the plus side, it might
require little or no code changes on the GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew
points out, whenever a Java File is used directly, we might have to
tweak code.

Thoughts?

Tentatively, I'm for the second approach.

Jim

On 04/04/2016 10:42 AM, Andrea Aime wrote:

Hi Rich,
it should be something along these lines (as said, never tried
personally, was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers
Andrea

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com
<mailto:rfecher@anonymised.com>> wrote:

    It is definitely possible we missed an extension point to register
    protocol handlers at the JVM level. I just hadn't thought of it
    at the time and didn't dig in, but I'll keep it in mind. Also,
    I'm curious if that works in the case of 'hdfs' and 's3' protocol
    because then I'd think it should work for us too. Thanks!

    On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com
    <mailto:ahulbert@anonymised.com>> wrote:

        Yeah the only thing we will have to worry about is if the code
        is written against java.io.File instead of some other URL lib.
        At some point I guess we could dig in and look. I'll ask Jim
        for more info and see if we can mock it up...while we're at it
        we can do S3 or something too for another test.

        On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:
        > On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime
        <andrea.aime@anonymised.com
        <mailto:andrea.aime@anonymised.com>>
        > wrote:
        >
        > > That said, I'm not sure this has ever been tried, not at
        any level, but
        > > especially with imageio-ext,
        > >
        >
        > Sorry, mis-typed, I meant jai-imageio there
        >
        > Cheers
        > Andrea
        >
        > --
        > ==
        > GeoServer Professional Services from the experts! Visit
        > http://goo.gl/it488V for more information.
        > ==
        >
        > Ing. Andrea Aime
        > @geowolf
        > Technical Lead
        >
        > GeoSolutions S.A.S.
        > Via di Montramito 3/A
        > 55054 Massarosa (LU)
        > phone: +39 0584 962313 <tel:%2B39%200584%20962313>
        > fax: +39 0584 1660272 <tel:%2B39%200584%201660272>
        > mob: +39 339 8844549 <tel:%2B39%20%20339%208844549>
        >
        > http://www.geo-solutions.it
        > http://twitter.com/geosolutions_it
        >
        > *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
        >
        > Le informazioni contenute in questo messaggio di posta
        elettronica e/o
        > nel/i file/s allegato/i sono da considerarsi strettamente
        riservate. Il
        > loro utilizzo è consentito esclusivamente al destinatario
        del messaggio,
        > per le finalità indicate nel messaggio stesso. Qualora
        riceviate questo
        > messaggio senza esserne il destinatario, Vi preghiamo
        cortesemente di
        > darcene notizia via e-mail e di procedere alla distruzione
        del messaggio
        > stesso, cancellandolo dal Vostro sistema. Conservare il
        messaggio stesso,
        > divulgarlo anche in parte, distribuirlo ad altri soggetti,
        copiarlo, od
        > utilizzarlo per finalità diverse, costituisce comportamento
        contrario ai
        > principi dettati dal D.Lgs. 196/2003.
        >
        > The information in this message and/or attachments, is
        intended solely for
        > the attention and use of the named addressee(s) and may be
        confidential or
        > proprietary in nature or covered by the provisions of
        privacy act
        > (Legislative Decree June, 30 2003, no.196 - Italy's New Data
        Protection
        > Code).Any use not in accord with its purpose, any
        disclosure, reproduction,
        > copying, distribution, or either dissemination, either whole
        or partial, is
        > strictly forbidden except previous formal approval of the named
        > addressee(s). If you are not the intended recipient, please
        contact
        > immediately the sender by telephone, fax or e-mail and
        delete the
        > information in this message that has been received in error.
        The sender
        > does not give any warranty or accept liability as the
        content, accuracy or
        > completeness of sent messages and accepts no
responsibility for changes
        > made after they were sent or for other risks which arise as
        a result of
        > e-mail transmission, viruses, etc.
        >
        > -------------------------------------------------------

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica
e/o nel/i file/s allegato/i sono da considerarsi strettamente
riservate. Il loro utilizzo è consentito esclusivamente al
destinatario del messaggio, per le finalità indicate nel messaggio
stesso. Qualora riceviate questo messaggio senza esserne il
destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso,
cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti,
copiarlo, od utilizzarlo per finalità diverse, costituisce
comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended
solely for the attention and use of the named addressee(s) and may
be confidential or proprietary in nature or covered by the
provisions of privacy act (Legislative Decree June, 30 2003,
no.196 - Italy's New Data Protection Code).Any use not in accord
with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please
contact immediately the sender by telephone, fax or e-mail and
delete the information in this message that has been received in
error. The sender does not give any warranty or accept liability
as the content, accuracy or completeness of sent messages and
accepts no responsibility for changes made after they were sent
or for other risks which arise as a result of e-mail transmission,
viruses, etc.

-------------------------------------------------------

------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

------------------------------------------------------------------------------
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Hi Jim,
as said in the other mail, I don’t know, but I’m afraid if you don’t go for URL handlers
you’ll have to modify more than just geotools and geoserver (and go down in imageio).

I was actually skeptical about joining this thread because I don’t have any direct experience on this
subject, nor the time to perform research.

That said, if you look at my previous stackoverflow link it seems that URL handlers
can be registered programmatically

Cheers
Andrea

···

On Mon, Apr 4, 2016 at 4:53 PM, Jim Hughes <jnh5y@anonymised.com> wrote:

Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded users to add URL handlers to their JVM or 2) handle this at the GeoTools/GeoServer level?

For #1, I get the sense that adding URL handlers is more challenging. Also, I doubt all installations would be ok with steps which add things to the JVM setup. On the plus side, it might require little or no code changes on the GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew points out, whenever a Java File is used directly, we might have to tweak code.

Thoughts?

Tentatively, I’m for the second approach.

Jim

On 04/04/2016 10:42 AM, Andrea Aime wrote:

Hi Rich,
it should be something along these lines (as said, never tried personally, was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers
Andrea

------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)



Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com> wrote:

It is definitely possible we missed an extension point to register protocol handlers at the JVM level. I just hadn’t thought of it at the time and didn’t dig in, but I’ll keep it in mind. Also, I’m curious if that works in the case of ‘hdfs’ and ‘s3’ protocol because then I’d think it should work for us too. Thanks!

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I’ll ask Jim for more info and see if we can mock it up…while we’re at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

That said, I’m not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.


==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi,

I read the base question and the accepted answer from Andrea’s reference. It seems the para 4 could be an issue requiring a wrapper to be written to handle multiple custom protocols.

Ref: http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

“4 Finally register it during application’s startup via URL#setURLStreamHandlerFactory()

URL.setURLStreamHandlerFactory(new CustomURLStreamHandlerFactory());

Note that the Javadoc explicitly says that you can set it at most once. So if you intend to support multiple custom protocols in the same application, you’d need to generify the custom URLStreamHandlerFactory implementation to cover them all inside the createURLStreamHandler() method.”

So if an installation wants HDFS and not S3, or vice-versa, the wrapper class would have to know about and handle the referencing implementation for specific protocols encountered. Thus we could also add “myprotocol”/”myspecialclass” and what-not to a Spring bean, I suppose, that could return the appropriate URLStreamHandler implemented in the “myspecialclass” or an exception as necessary.

Chris Snider

Senior Software Engineer

Intelligent Software Solutions, Inc.

Description: Description: Description: cid:image001.png@...3926...

(attachments)

image001.png

···

Hi Jim,

as said in the other mail, I don’t know, but I’m afraid if you don’t go for URL handlers

you’ll have to modify more than just geotools and geoserver (and go down in imageio).

I was actually skeptical about joining this thread because I don’t have any direct experience on this

subject, nor the time to perform research.

That said, if you look at my previous stackoverflow link it seems that URL handlers

can be registered programmatically

Cheers

Andrea

On Mon, Apr 4, 2016 at 4:53 PM, Jim Hughes <jnh5y@…1612…> wrote:

Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded users to add URL handlers to their JVM or 2) handle this at the GeoTools/GeoServer level?

For #1, I get the sense that adding URL handlers is more challenging. Also, I doubt all installations would be ok with steps which add things to the JVM setup. On the plus side, it might require little or no code changes on the GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew points out, whenever a Java File is used directly, we might have to tweak code.

Thoughts?

Tentatively, I’m for the second approach.

Jim

On 04/04/2016 10:42 AM, Andrea Aime wrote:

Hi Rich,

it should be something along these lines (as said, never tried personally, was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers

Andrea

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@…403…> wrote:

It is definitely possible we missed an extension point to register protocol handlers at the JVM level. I just hadn’t thought of it at the time and didn’t dig in, but I’ll keep it in mind. Also, I’m curious if that works in the case of ‘hdfs’ and ‘s3’ protocol because then I’d think it should work for us too. Thanks!

On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@…1612…> wrote:

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I’ll ask Jim for more info and see if we can mock it up…while we’re at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@…1268…>
wrote:

That said, I’m not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.


==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi Andrea, Chris,

Andrea, thanks for the input.

Chris, it seems like several handlers could be gathered up for the one-go at registering them. That said, I’m a little worried that if other applications in a container are trying to do the same, then we’ll lose. I don’t know if that concern is a show-stopper or not.

Thanks,

Jim

···

On 04/04/2016 12:44 PM, Chris Snider wrote:

Hi,

I read the base question and the accepted answer from Andrea’s reference. It seems the para 4 could be an issue requiring a wrapper to be written to handle multiple custom protocols.

Ref: http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

“4 Finally register it during application’s startup via URL#setURLStreamHandlerFactory()

URL.setURLStreamHandlerFactory(new CustomURLStreamHandlerFactory());

Note that the Javadoc explicitly says that you can set it at most once. So if you intend to support multiple custom protocols in the same application, you’d need to generify the custom URLStreamHandlerFactory implementation to cover them all inside the createURLStreamHandler() method.”

So if an installation wants HDFS and not S3, or vice-versa, the wrapper class would have to know about and handle the referencing implementation for specific protocols encountered. Thus we could also add “myprotocol”/”myspecialclass” and what-not to a Spring bean, I suppose, that could return the appropriate URLStreamHandler implemented in the “myspecialclass” or an exception as necessary.

Chris Snider

Senior Software Engineer

Intelligent Software Solutions, Inc.

Description: Description: Description: cid:image001.png@anonymised.com

From: Andrea Aime [mailto:andrea.aime@anonymised.com]
Sent: Monday, April 04, 2016 9:23 AM
To: Jim Hughes jnh5y@anonymised.com
Cc: Geoserver-devel geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Custom URL protocol handling

Hi Jim,

as said in the other mail, I don’t know, but I’m afraid if you don’t go for URL handlers

you’ll have to modify more than just geotools and geoserver (and go down in imageio).

I was actually skeptical about joining this thread because I don’t have any direct experience on this

subject, nor the time to perform research.

That said, if you look at my previous stackoverflow link it seems that URL handlers

can be registered programmatically

Cheers

Andrea

On Mon, Apr 4, 2016 at 4:53 PM, Jim Hughes <jnh5y@anonymised.com> wrote:

Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded users to add URL handlers to their JVM or 2) handle this at the GeoTools/GeoServer level?

For #1, I get the sense that adding URL handlers is more challenging. Also, I doubt all installations would be ok with steps which add things to the JVM setup. On the plus side, it might require little or no code changes on the GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew points out, whenever a Java File is used directly, we might have to tweak code.

Thoughts?

Tentatively, I’m for the second approach.

Jim

On 04/04/2016 10:42 AM, Andrea Aime wrote:

Hi Rich,

it should be something along these lines (as said, never tried personally, was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers

Andrea

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com> wrote:

It is definitely possible we missed an extension point to register protocol handlers at the JVM level. I just hadn’t thought of it at the time and didn’t dig in, but I’ll keep it in mind. Also, I’m curious if that works in the case of ‘hdfs’ and ‘s3’ protocol because then I’d think it should work for us too. Thanks!

On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

Yeah the only thing we will have to worry about is if the code is written against java.io.File instead of some other URL lib. At some point I guess we could dig in and look. I’ll ask Jim for more info and see if we can mock it up…while we’re at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

That said, I’m not sure this has ever been tried, not at any level, but
especially with imageio-ext,

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.


==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Ciao,
chiming in late but I'll try to add my experience.

From the top of my head it is hard to ensure that working solely at

the ImageIO level would be enough, in any case most work would go down
there.
If you take the geotiff reader in GeoTools as an example I am pretty
sure you might have to add some more logic in the plugin (waning, i
did not check, just trying to go by memory).

If I had to do this myself I would try to work at the level of
ImageInputStreamSpi as a start.
In a past life we fiddled a bit with a custom URLImageInputStreamSpi
to try and make ImageIO play nicer with URLs
(https://github.com/geosolutions-it/imageio-ext/blob/master/library/streams/src/main/java/it/geosolutions/imageio/stream/input/spi/URLImageInputStreamSpi.java)
so if you manage to register URLs for those protocols we coule make
this class more intelligent and create a specific ImageInputStream if
needed.

As an alternative we could rely on something like
StringImageInputStreamSpi
(https://github.com/geosolutions-it/imageio-ext/blob/master/library/streams/src/main/java/it/geosolutions/imageio/stream/input/spi/StringImageInputStreamSpi.java)
and make that more intellingent.

Another crazy option takes into account that Streams SPIs works with
classes, hence you could use a proper class from the Hadoop API and/or
S3 api that you would use for accessing file and build an SPI for it.
In the SPI you can decide if trying to use an existing
ImageInputStream implementation (like we did for String and URL) or
roll in you own.

I hope this helps, it's been a while since the last time I worked on this...

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

On Mon, Apr 4, 2016 at 9:19 PM, Jim Hughes <jnh5y@anonymised.com> wrote:

Hi Andrea, Chris,

Andrea, thanks for the input.

Chris, it seems like several handlers could be gathered up for the one-go at
registering them. That said, I'm a little worried that if other
applications in a container are trying to do the same, then we'll lose. I
don't know if that concern is a show-stopper or not.

Thanks,

Jim

On 04/04/2016 12:44 PM, Chris Snider wrote:

Hi,

I read the base question and the accepted answer from Andrea’s reference.
It seems the para 4 could be an issue requiring a wrapper to be written to
handle multiple custom protocols.

Ref:
http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

“4 Finally register it during application's startup via
URL#setURLStreamHandlerFactory()

URL.setURLStreamHandlerFactory(new CustomURLStreamHandlerFactory());

Note that the Javadoc explicitly says that you can set it at most once. So
if you intend to support multiple custom protocols in the same application,
you'd need to generify the custom URLStreamHandlerFactory implementation to
cover them all inside the createURLStreamHandler() method.”

So if an installation wants HDFS and not S3, or vice-versa, the wrapper
class would have to know about and handle the referencing implementation for
specific protocols encountered. Thus we could also add
“myprotocol”/”myspecialclass” and what-not to a Spring bean, I suppose, that
could return the appropriate URLStreamHandler implemented in the
“myspecialclass” or an exception as necessary.

Chris Snider

Senior Software Engineer

Intelligent Software Solutions, Inc.

From: Andrea Aime [mailto:andrea.aime@anonymised.com]
Sent: Monday, April 04, 2016 9:23 AM
To: Jim Hughes <jnh5y@anonymised.com>
Cc: Geoserver-devel <geoserver-devel@lists.sourceforge.net>
Subject: Re: [Geoserver-devel] Custom URL protocol handling

Hi Jim,

as said in the other mail, I don't know, but I'm afraid if you don't go for
URL handlers

you'll have to modify more than just geotools and geoserver (and go down in
imageio).

I was actually skeptical about joining this thread because I don't have any
direct experience on this

subject, nor the time to perform research.

That said, if you look at my previous stackoverflow link it seems that URL
handlers

can be registered programmatically

Cheers

Andrea

On Mon, Apr 4, 2016 at 4:53 PM, Jim Hughes <jnh5y@anonymised.com> wrote:

Hi Andrea, all,

In terms of ease, would it be preferable to 1) ask HDFS/S3 minded users to
add URL handlers to their JVM or 2) handle this at the GeoTools/GeoServer
level?

For #1, I get the sense that adding URL handlers is more challenging. Also,
I doubt all installations would be ok with steps which add things to the JVM
setup. On the plus side, it might require little or no code changes on the
GT/GS side of things.

For #2, I imagine there may be more code to update, etc. As Andrew points
out, whenever a Java File is used directly, we might have to tweak code.

Thoughts?

Tentatively, I'm for the second approach.

Jim

On 04/04/2016 10:42 AM, Andrea Aime wrote:

Hi Rich,

it should be something along these lines (as said, never tried personally,
was just barely aware it should be possible):

http://stackoverflow.com/questions/26363573/registering-and-using-a-custom-java-net-url-protocol

Cheers

Andrea

On Mon, Apr 4, 2016 at 4:37 PM, Rich Fecher <rfecher@anonymised.com> wrote:

It is definitely possible we missed an extension point to register protocol
handlers at the JVM level. I just hadn't thought of it at the time and
didn't dig in, but I'll keep it in mind. Also, I'm curious if that works in
the case of 'hdfs' and 's3' protocol because then I'd think it should work
for us too. Thanks!

On Mon, Apr 4, 2016 at 10:34 AM, Andrew Hulbert <ahulbert@anonymised.com> wrote:

Yeah the only thing we will have to worry about is if the code is written
against java.io.File instead of some other URL lib. At some point I guess we
could dig in and look. I'll ask Jim for more info and see if we can mock it
up...while we're at it we can do S3 or something too for another test.

On Mon, Apr 04, 2016 at 04:31:13PM +0200, Andrea Aime wrote:

On Mon, Apr 4, 2016 at 4:30 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

> That said, I'm not sure this has ever been tried, not at any level, but
> especially with imageio-ext,
>

Sorry, mis-typed, I meant jai-imageio there

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure,
reproduction,
copying, distribution, or either dissemination, either whole or partial,
is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro
utilizzo è consentito esclusivamente al destinatario del messaggio, per le
finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio
senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia
via e-mail e di procedere alla distruzione del messaggio stesso,
cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per
finalità diverse, costituisce comportamento contrario ai principi dettati
dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender does
not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

------------------------------------------------------------------------------

_______________________________________________

Geoserver-devel mailing list

Geoserver-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geoserver-devel

------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro
utilizzo è consentito esclusivamente al destinatario del messaggio, per le
finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio
senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia
via e-mail e di procedere alla distruzione del messaggio stesso,
cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per
finalità diverse, costituisce comportamento contrario ai principi dettati
dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender does
not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

------------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

I know for vector data stores there is a great deal of flexibility in what can be used as connection parameters, I am not sure what can be done for the raster formats.

  • One trick that image mosaic and image pyramid use is to point to a directory that contains a configuration file, perhaps the same approach could be used to point to a java property file describing how to connect to your cluster and define additional parameters?
  • Andrew already mentioned making a custom URL handler - do you want to research the topic and run up some test cases on geotools-devel? You could start by double checking it can read content our of a jar or zip?
···

On 4 April 2016 at 06:56, Rich Fecher <rfecher@anonymised.com> wrote:

Hi Jim and Michael,
Great idea regarding the GeoServer raster data store being more friendly to other URLs such as those using HDFS protocol. I’d like to add that it would be nice to be able to use some other format or perhaps a custom defined protocol for the URL string. We originally looked at encoding data store connection parameters into the URL but ran into this same validation issue in GeoServer. It seemed our best option at the time was to take a look at how PGRaster works, but I’m definitely interested in any evolution of raster configuration going on.

Generally, I like the idea of having more flexibility for configuring the raster formats to include the ability to use ‘hdfs’ protocol in the URL, and how that can be expanded.

Thanks,

Rich



Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Jody Garnett

On Mon, Apr 4, 2016 at 9:34 AM, Jim Hughes <jnh5y@anonymised.com2…> wrote:

Hi all,

As a little more context, we are looking for the most suitable way to
HDFS-enable the existing raster plugins.

With some of the javax ImageIO SPI interfaces, it looked like we’d be
close to being able to have a drop-in solution. Otherwise, it looks
like we may have to touch each plugin.

Thoughts?

Cheers,

Jim

On 04/01/2016 11:06 AM, Michael Matthews wrote:

Hello Geoserver Devs,

I’m trying to extend GeoTIFF support to be able to read image files
directly from a Hadoop Distributed File System. Our idea was to write
ImageInputStream and ImageInputStreamSPI classes that would know how to
handle the ‘hdfs’ protocol.
When adding a GeoTIFF datastore through the GeoServer web UI, I received
the error that ‘hdfs://localhost:9000/sample.tif’ is not a valid URL.

How can I expand GeoServer’s notion of a valid URL?
Or, is there a better way to implement a hdfs interface?

Optimally, we would like to extend this hdfs access functionality to
NetCDF as well, but GeoTIFFs are our primary concern.

Thank you,
Michael Matthews


Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel



Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel