[Geoserver-devel] Support for multiple pluggable GWC blob stores and S3

GeoServer GeoServer Developer
1

Hello all,

following GWC’s new support for managing multiple blob stores, including the new S3 backend, I’ve created a PR that allows geoserver to manage the blobstores through the UI [1]

The work on GWC has been performed in two separate steps. First, allowing to configure multiple blobstores in geowebcache.xml, which decouples the tile storage from the data directory and allows to choose different storage locations on a per layer basis. Second, the addition of Amazon S3 support. See the gwc documentation [2] for further details.

Please check and comment on the above mentioned PR if you’re interested, at your earliest convenience.

TIA,
Gabriel

[1] <https://github.com/geoserver/geoserver/pull/1114>
[2] <http://geowebcache.org/docs/1.8.x/configuration/storage.html>

···

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

2

Hi again,

As for the pluggable blobstore support + S3 blobstore PR mentioned before, the obvious question is how much it adds in terms of dependencies.
The dependency on the gwc-aws-s3 jar adds the following deps:

± org.geowebcache:gwc-aws-s3:jar:1.8-SNAPSHOT:compile
[544K ] | - com.amazonaws:aws-java-sdk-s3:jar:1.9.30:compile
[247K ] | - com.amazonaws:aws-java-sdk-kms:jar:1.9.30:compile
[488K ] | - com.amazonaws:aws-java-sdk-core:jar:1.9.30:compile
[577K ] | ± org.apache.httpcomponents:httpclient:jar:4.3.4:compile
[276K ] | | - org.apache.httpcomponents:httpcore:jar:4.3.2:compile
[894K ] | ± com.fasterxml.jackson.core:jackson-databind:jar:2.3.2:compile
[ 35K ] | | ± com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
[195K ] | | - com.fasterxml.jackson.core:jackson-core:jar:2.3.2:compile
[608K ] | - joda-time:joda-time:jar:2.8.1:compile (version selected from constraint [2.2,))

[3864K total]

Question is whether it’d be acceptable to have those on master directly or we need to somehow make it pluggable.
The one complication with making it a separate plugin is it’ll add extra complexity to the UI code which I’m not sure how to solve off the top of my head, so I figured I’d ask whether we can add those deps directly to master before scratching my head thinking how to cleanly separate things.

TIA for any comments.
Gabriel

···

On Fri, Jun 19, 2015 at 2:09 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

Hello all,

following GWC’s new support for managing multiple blob stores, including the new S3 backend, I’ve created a PR that allows geoserver to manage the blobstores through the UI [1]

The work on GWC has been performed in two separate steps. First, allowing to configure multiple blobstores in geowebcache.xml, which decouples the tile storage from the data directory and allows to choose different storage locations on a per layer basis. Second, the addition of Amazon S3 support. See the gwc documentation [2] for further details.

Please check and comment on the above mentioned PR if you’re interested, at your earliest convenience.

TIA,
Gabriel

[1] <https://github.com/geoserver/geoserver/pull/1114>
[2] <http://geowebcache.org/docs/1.8.x/configuration/storage.html>

–

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

–

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

3

Ciao Gabriel,
IMHO it would be better to have this as a separate plugin since GS it’s already big enough.

If needed I can talk to Andrea and see if he has some time to help with this.

···

On Fri, Jun 26, 2015 at 5:35 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

Hi again,

As for the pluggable blobstore support + S3 blobstore PR mentioned before, the obvious question is how much it adds in terms of dependencies.
The dependency on the gwc-aws-s3 jar adds the following deps:

± org.geowebcache:gwc-aws-s3:jar:1.8-SNAPSHOT:compile
[544K ] | - com.amazonaws:aws-java-sdk-s3:jar:1.9.30:compile
[247K ] | - com.amazonaws:aws-java-sdk-kms:jar:1.9.30:compile
[488K ] | - com.amazonaws:aws-java-sdk-core:jar:1.9.30:compile
[577K ] | ± org.apache.httpcomponents:httpclient:jar:4.3.4:compile
[276K ] | | - org.apache.httpcomponents:httpcore:jar:4.3.2:compile
[894K ] | ± com.fasterxml.jackson.core:jackson-databind:jar:2.3.2:compile
[ 35K ] | | ± com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
[195K ] | | - com.fasterxml.jackson.core:jackson-core:jar:2.3.2:compile
[608K ] | - joda-time:joda-time:jar:2.8.1:compile (version selected from constraint [2.2,))

[3864K total]

Question is whether it’d be acceptable to have those on master directly or we need to somehow make it pluggable.
The one complication with making it a separate plugin is it’ll add extra complexity to the UI code which I’m not sure how to solve off the top of my head, so I figured I’d ask whether we can add those deps directly to master before scratching my head thinking how to cleanly separate things.

TIA for any comments.
Gabriel

Monitor 25 network devices or servers for free with OpManager!
OpManager is web-based network management software that monitors
network devices and physical & virtual servers, alerts via email & sms
for fault. Monitor 25 devices for free with no restriction. Download now
http://ad.doubleclick.net/ddm/clk/292181274;119417398;o

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

On Fri, Jun 19, 2015 at 2:09 PM, Gabriel Roldan <groldan@anonymised.com839…> wrote:

Hello all,

following GWC’s new support for managing multiple blob stores, including the new S3 backend, I’ve created a PR that allows geoserver to manage the blobstores through the UI [1]

The work on GWC has been performed in two separate steps. First, allowing to configure multiple blobstores in geowebcache.xml, which decouples the tile storage from the data directory and allows to choose different storage locations on a per layer basis. Second, the addition of Amazon S3 support. See the gwc documentation [2] for further details.

Please check and comment on the above mentioned PR if you’re interested, at your earliest convenience.

TIA,
Gabriel

[1] <https://github.com/geoserver/geoserver/pull/1114>
[2] <http://geowebcache.org/docs/1.8.x/configuration/storage.html>

–

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

–

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

4

On Sat, Jun 27, 2015 at 9:44 AM, Simone Giannecchini <
simone.giannecchini@anonymised.com> wrote:

Ciao Gabriel,
IMHO it would be better to have this as a separate plugin since GS it's
already big enough.

If needed I can talk to Andrea and see if he has some time to help with
this.

Gabriel, could share some description of the extra UI you need to make
pluggable?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Hi Andrea,

sorry for the late reply, been swamped getting vectortiles ready for master (just merged it), and I’m monotasking (or autist).

···

On Tue, Jun 30, 2015 at 5:25 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

there are different wicket components to configure file and s3 blobstores. That’s pretty much it. Guess I’ll have to instrument some sort of extension point just for that, but it’s not that a big deal.

Cheers,
Gabriel

–

On Sat, Jun 27, 2015 at 9:44 AM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Gabriel,
IMHO it would be better to have this as a separate plugin since GS it’s already big enough.

If needed I can talk to Andrea and see if he has some time to help with this.

Gabriel, could share some description of the extra UI you need to make pluggable?

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

6

I can't for the life of me get the geoserver plugin for this to work.

Starting geoserver with the plugin in place sprays catalina with errors.

org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find
class [org.geowebcache.s3.S3BlobStoreConfigProvider] for bean with name
'S3BlobStoreConfigProvider' defined in URL
[jar:file:/C:/Program%20Files%20(x86)/Apache%20Software%20Foundation/Tomcat%208.0/webapps/geoserver/WEB-INF/lib/gs-gwc-s3-2.8.0.jar!/applicationContext.xml];
nested exception is java.lang.ClassNotFoundException:
org.geowebcache.s3.S3BlobStoreConfigProvider
  at
org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1262)

Standalone GWC works fine out of the box.

Seems like its an issue with how gs-gwc is talking to gs-gwc-s3.

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Support-for-multiple-pluggable-GWC-blob-stores-and-S3-tp5211986p5230779.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

7

You appear to be missing at least one dependency. The missing class should be defined in gwc-aws-s3-1.8.0.jar; is this jar present in your WEB-INF/lib?

Where did you get the plugin? Did you build it from source, and if so how? Just the jar or did you make a full build of all main and plugin zip files with:

mvn -DskipTests -Prelease -PcommunityRelease clean install
mvn assembly:attached
mvn -f community/pom.xml assembly:attached

The plugin should be:

community/target/release/geoserver-2.8.0-gwc-s3.zip

On master the plugin contains the following files, all of which should be installed (you should probably have gs-gwc-s3-2.8.0.jar and gwc-aws-s3-1.8.0.jar):

aws-java-sdk-core-1.9.30.jar
aws-java-sdk-kms-1.9.30.jar
aws-java-sdk-s3-1.9.30.jar
gs-gwc-s3-2.9-SNAPSHOT.jar
gwc-aws-s3-1.9-SNAPSHOT.jar
httpclient-4.3.1.jar
httpcore-4.3.jar
jackson-annotations-2.5.0.jar
jackson-core-2.5.3.jar
jackson-databind-2.5.3.jar
joda-time-2.2.jar

Kind regards,
Ben.

On 17/10/15 13:10, dpierce wrote:

I can't for the life of me get the geoserver plugin for this to work.

Starting geoserver with the plugin in place sprays catalina with errors.

  org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find
class [org.geowebcache.s3.S3BlobStoreConfigProvider] for bean with name
'S3BlobStoreConfigProvider' defined in URL
[jar:file:/C:/Program%20Files%20(x86)/Apache%20Software%20Foundation/Tomcat%208.0/webapps/geoserver/WEB-INF/lib/gs-gwc-s3-2.8.0.jar!/applicationContext.xml];
nested exception is java.lang.ClassNotFoundException:
org.geowebcache.s3.S3BlobStoreConfigProvider
  at
org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1262)

Standalone GWC works fine out of the box.

Seems like its an issue with how gs-gwc is talking to gs-gwc-s3.

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Support-for-multiple-pluggable-GWC-blob-stores-and-S3-tp5211986p5230779.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

------------------------------------------------------------------------------
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

8

Yep, that was it. I had compiled it using a few different profiles but
really didn't understand the build process.

Thanks a lot Ben.

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Support-for-multiple-pluggable-GWC-blob-stores-and-S3-tp5211986p5230803.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.