[Geoserver-users] Does GeoServer work with abfs (azure blob filesystem)

GeoServer GeoServer User
1

Dear community,
this question is related to the support of abfs by geoserver.

I am on geoserver 2.17.2.
I connected to a geomesa (3.0.0)-hbase (2.5.5) datastore that is stored on Azure. Hbase stores its data on abfs (azure blob filesytem -the new generation adls).
I have some data that I can read well in hbase, that I ingested through the geomesa-hbase cli tools, and that I can add as a store in geoserver. When I go to the layer preview in geoserver, the gml that contains the geodata, is shown correctly.
However when I try to visualize (using openlayers), I get error message:

23 Oct 16:42:35 WARN [util.DynamicClassLoader] - Failed to identify the fs of dir abfs://storage-fs@anonymised.com/ourhbase/hbase/lib, ignored
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “abfs”
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
…
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.rpc.filter.CqlTransformFilter

I believe this missing class is part of the geomesa-hbase distrbuted runtime jar, which was stored in the folder referenced above.
It looks like he can not access this jar since he does not recognize abfs.

So the question is: is abfs supported in geoserver?

Thanks for feedback!

2

Hi Joris,

Good question. GeoServer does not ship with any Hadoop code by default. Several of the interfaces in GeoTools and GeoServer do properly respect Java URL / URL FactoryHandlers, etc. When those things line up, one can use GeoServer with different URLs for a number of really, really cool deployment options! (That said, there are a few places where assumptions are made, and sometimes one can hit small issues to work through.)

That aside, given that GeoServer does not ship with anything Hadoop and that its core functionality does not presently include any cloud vendors libraries (there are extensions for S3 support), your question comes down to what GeoMesa can do and where an end user can fill in the details.

GeoMesa uses the Hadoop FileSystem interface in a few ways. One of them happens to be in the HBase library code which is checking that the HBase is configured properly for GeoMesa use. Just like GeoServer, GeoMesa does not make an attempt to bundle all the cloud vendors’ libraries, so you’ll need to add them to your deployment. I’d suggest starting here (https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html) to find the jar(s) you need!

Cheers,

Jim

···

On 10/23/2020 1:32 PM, Joris Billen wrote:

Dear community,
this question is related to the support of abfs by geoserver.

I am on geoserver 2.17.2.
I connected to a geomesa (3.0.0)-hbase (2.5.5) datastore that is stored on Azure. Hbase stores its data on abfs (azure blob filesytem -the new generation adls).
I have some data that I can read well in hbase, that I ingested through the geomesa-hbase cli tools, and that I can add as a store in geoserver. When I go to the layer preview in geoserver, the gml that contains the geodata, is shown correctly.
However when I try to visualize (using openlayers), I get error message:

23 Oct 16:42:35 WARN [util.DynamicClassLoader] - Failed to identify the fs of dir abfs://storage-fs@anonymised.com/ourhbase/hbase/lib, ignored
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “abfs”
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
…
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.rpc.filter.CqlTransformFilter

I believe this missing class is part of the geomesa-hbase distrbuted runtime jar, which was stored in the folder referenced above.
It looks like he can not access this jar since he does not recognize abfs.

So the question is: is abfs supported in geoserver?

Thanks for feedback!

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: [http://www.ianturton.com/talks/foss4g.html#/](http://www.ianturton.com/talks/foss4g.html#/)
- The GeoServer user list posting guidelines: [http://geoserver.org/comm/userlist-guidelines.html](http://geoserver.org/comm/userlist-guidelines.html)

If you want to request a feature or an improvement, also see this: [https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer](https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer)

[Geoserver-users@lists.sourceforge.net](mailto:Geoserver-users@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-users](https://lists.sourceforge.net/lists/listinfo/geoserver-users)
3

Hi Jim,
You lead me to the correct solution.
I notice that these 2 jars will fix it when put under $GEOSERVER_HOME/webapps/geoserver/WEB-INF/lib :

azure-storage-8.6.4.jar
hadoop-azure-3.3.0.jar

This will prevent the error:

23 Oct 16:42:35 WARN [util.DynamicClassLoader] - Failed to identify the fs of dir abfs://storage-fs@anonymised.com/hbasecluster/hbase/lib, ignored
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “abfs”
…
at java.security.AccessController.doPrivileged(Native Method)
…
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.rpc.filter.CqlTransformFilter

Best,

···

On 10/23/2020 1:32 PM, Joris Billen wrote:

Dear community,
this question is related to the support of abfs by geoserver.

I am on geoserver 2.17.2.
I connected to a geomesa (3.0.0)-hbase (2.5.5) datastore that is stored on Azure. Hbase stores its data on abfs (azure blob filesytem -the new generation adls).
I have some data that I can read well in hbase, that I ingested through the geomesa-hbase cli tools, and that I can add as a store in geoserver. When I go to the layer preview in geoserver, the gml that contains the geodata, is shown correctly.
However when I try to visualize (using openlayers), I get error message:

23 Oct 16:42:35 WARN [util.DynamicClassLoader] - Failed to identify the fs of dir abfs://storage-fs@anonymised.com/ourhbase/hbase/lib, ignored
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “abfs”
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
…
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.rpc.filter.CqlTransformFilter

I believe this missing class is part of the geomesa-hbase distrbuted runtime jar, which was stored in the folder referenced above.
It looks like he can not access this jar since he does not recognize abfs.

So the question is: is abfs supported in geoserver?

Thanks for feedback!

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: [http://www.ianturton.com/talks/foss4g.html#/](https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ianturton.com%2Ftalks%2Ffoss4g.html%23%2F&data=04%7C01%7Cjoris.billen%40bigindustries.be%7C4f8f8a5bbc914963bff608d879ccafce%7C49c3d703357947bfa8887c913fbdced9%7C0%7C1%7C637393267321852513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=dtm2NCXZKVp33kJvaYX0XCgpITjuholfo6AXNRJy3bc%3D&reserved=0)
- The GeoServer user list posting guidelines: [http://geoserver.org/comm/userlist-guidelines.html](https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgeoserver.org%2Fcomm%2Fuserlist-guidelines.html&data=04%7C01%7Cjoris.billen%40bigindustries.be%7C4f8f8a5bbc914963bff608d879ccafce%7C49c3d703357947bfa8887c913fbdced9%7C0%7C1%7C637393267321852513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=ueIRgjPnAW9sSsWFDw%2FTMVACdJtRCJMbeLSz6vxm5Xk%3D&reserved=0)

If you want to request a feature or an improvement, also see this: [https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer](https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeoserver%2Fgeoserver%2Fwiki%2FSuccessfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer&data=04%7C01%7Cjoris.billen%40bigindustries.be%7C4f8f8a5bbc914963bff608d879ccafce%7C49c3d703357947bfa8887c913fbdced9%7C0%7C1%7C637393267321862506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Glp2nQpTdaYXTJ4l5NPh2Ig%2B5u6sE5wV4dbQpBpU6Vo%3D&reserved=0)

[Geoserver-users@lists.sourceforge.net](mailto:Geoserver-users@anonymised.comsourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-users](https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-users&data=04%7C01%7Cjoris.billen%40bigindustries.be%7C4f8f8a5bbc914963bff608d879ccafce%7C49c3d703357947bfa8887c913fbdced9%7C0%7C1%7C637393267321862506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=VNVcdODRxaP3v%2F6MkflVqUHxZQCKgmNMPopi0PwJZmM%3D&reserved=0)