[Geoserver-devel] GDAL on RHEL7 questions

This started out as a user issue but I think I have gone pretty far down the
rabbit hole and I have a few questions that I wonder if someone on this
mailing list could answer.

Goal: Deploy the GeoServer war including the GDAL plugin on WebLogic 12.1.3
on RHEL7.

Initially, I was able to get this setup working on Windows using the
binaries from ImageIO-ext. The instructions were pretty clear and things
worked after a day or so of figuring things out. It was when I moved to
deploying onto a RHEL7 server that I started having issues. ImageIO-ext also
does not provide binaries for RHEL7 so I was taking a gamble. Once GDAL and
the java bindings installed, booting up GeoServer gave me errors like this
one for all the GDAL types.

java.lang.UnsatisfiedLinkError: org.gdal.gdal.gdalJNI.SWIGDriverUpcast(J)J
        at org.gdal.gdal.gdalJNI.SWIGDriverUpcast(Native Method)
        at org.gdal.gdal.Driver.<init>(Driver.java:15)
        at org.gdal.gdal.gdal.GetDriverByName(gdal.java:264)

In the log, there was also a record of the GDAL version being detected and
it never threw the "no gdaljni in java.library.path" so I could assume that
the JNI detection was working but there was a different link failing. Of
course, there is some hindsight in my email here and I spent way more time
than I wanted to on this issue. After digging and learning more about what
SWIG was doing to wrap GDAL and all the jar dependencies from ImageIO-ext I
finally ran across that SWIG 2 was installed and that there was a major
change between 1.x and 2.x versions, particularly in the class signatures
created by the java.swg template.

swig-1.3.40/Lib/java/java.swg

  PTRCTOR_VISIBILITY $javaclassname(long cPtr, boolean cMemoryOwn) {
    super($imclassname.SWIG$javaclassnameUpcast(cPtr), cMemoryOwn);
    swigCPtr = cPtr;
  }

swig-2.0.0/Lib/java/java.swg

  PTRCTOR_VISIBILITY $javaclassname(long cPtr, boolean cMemoryOwn) {
    super($imclassname.$javaclazznameSWIGUpcast(cPtr), cMemoryOwn);
    swigCPtr = cPtr;
  }

Notice that they changed the order of the SWIG Upcast method which resulted
in the java class have an incorrect signature. (I have no idea why a
underscore got in there.) The error from above is looking for a method that
does not exist.

protected Driver(long cPtr, boolean cMemoryOwn) {
    super(gdalJNI.Driver_SWIGUpcast(cPtr), cMemoryOwn);
    swigCPtr = cPtr;
  }

The end result was that I uninstalled SWIG 2.x and ended up building SWIG
1.3.40 from source. SWIG 1.x couldn't even figure out the RHEL7 host name. I
have no idea what other things might not work. Finally, I was able to get
the GDAL image types detected in the GeoServer stores with SWIG 1.3.40 and
GDAL 2.1.0. Even though I found the source for ImageIO-ext on github, they
include libraries where the source is not available, or at least I couldn't
find it. ImageIO-ext doesn't seem to be upgraded for SWIG 2.x or 3.x. For
example, the dependency imageio-ext-gdal-bindings seems pretty important but
it is stuck at GDAL 1.9.2 and most likely still on the old SWIG api.

https://github.com/geosolutions-it/imageio-ext/search?utf8=✓&q=imageio-ext-gdal-bindings

So, where do I go with my questions? It doesn't seem like I could even patch
this on my own because of hidden libraries. I have no idea if all the
features will even work on RHEL7 because I can't even trust the linking. Is
there a reason why everything is stuck on old libraries and old OS? Is this
the right board for these kind of issues? If not, what would be?

Thanks for reading,
Brian

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-on-RHEL7-questions-tp5270708.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

Hi Brian,
The “GDAL native libs installation on different Linux OSs” is a hot topic.

Summarizing some key points:

  • Right now GeoServer depends on ImageIO-EXT GDAL plugins which still rely on GDAL 1.9.2.
  • Some users did experiments to build and use higher versions of GDAL (1.x, 2.x).
  • Supporting different GDAL versions on a different OS usually requires GDAL manual build on the hosting system itself to make sure there no linking issues or conflicts with the required third party libraries and dependencies.

There is a couple of discussions about this topic:

https://sourceforge.net/p/geoserver/mailman/message/34924220/

https://sourceforge.net/p/geoserver/mailman/message/34982403/

Hope they can provide you some more hints.

Additional note: In the past, we have also noticed that SWIG 1.3.36 seemed a good version to produce the expected signatures with respect to others which were resulting in mismatching methods.

Please, let us know if you still need assistance or clarifications on that.

Cheers,
Daniele

···

On Wed, Jun 8, 2016 at 8:54 PM, brian.blain <brian.blain@anonymised.com> wrote:

This started out as a user issue but I think I have gone pretty far down the
rabbit hole and I have a few questions that I wonder if someone on this
mailing list could answer.

Goal: Deploy the GeoServer war including the GDAL plugin on WebLogic 12.1.3
on RHEL7.

Initially, I was able to get this setup working on Windows using the
binaries from ImageIO-ext. The instructions were pretty clear and things
worked after a day or so of figuring things out. It was when I moved to
deploying onto a RHEL7 server that I started having issues. ImageIO-ext also
does not provide binaries for RHEL7 so I was taking a gamble. Once GDAL and
the java bindings installed, booting up GeoServer gave me errors like this
one for all the GDAL types.

java.lang.UnsatisfiedLinkError: org.gdal.gdal.gdalJNI.SWIGDriverUpcast(J)J
at org.gdal.gdal.gdalJNI.SWIGDriverUpcast(Native Method)
at org.gdal.gdal.Driver.(Driver.java:15)
at org.gdal.gdal.gdal.GetDriverByName(gdal.java:264)

In the log, there was also a record of the GDAL version being detected and
it never threw the “no gdaljni in java.library.path” so I could assume that
the JNI detection was working but there was a different link failing. Of
course, there is some hindsight in my email here and I spent way more time
than I wanted to on this issue. After digging and learning more about what
SWIG was doing to wrap GDAL and all the jar dependencies from ImageIO-ext I
finally ran across that SWIG 2 was installed and that there was a major
change between 1.x and 2.x versions, particularly in the class signatures
created by the java.swg template.

swig-1.3.40/Lib/java/java.swg

PTRCTOR_VISIBILITY $javaclassname(long cPtr, boolean cMemoryOwn) {
super($imclassname.SWIG$javaclassnameUpcast(cPtr), cMemoryOwn);
swigCPtr = cPtr;
}

swig-2.0.0/Lib/java/java.swg

PTRCTOR_VISIBILITY $javaclassname(long cPtr, boolean cMemoryOwn) {
super($imclassname.$javaclazznameSWIGUpcast(cPtr), cMemoryOwn);
swigCPtr = cPtr;
}

Notice that they changed the order of the SWIG Upcast method which resulted
in the java class have an incorrect signature. (I have no idea why a
underscore got in there.) The error from above is looking for a method that
does not exist.

protected Driver(long cPtr, boolean cMemoryOwn) {
super(gdalJNI.Driver_SWIGUpcast(cPtr), cMemoryOwn);
swigCPtr = cPtr;
}

The end result was that I uninstalled SWIG 2.x and ended up building SWIG
1.3.40 from source. SWIG 1.x couldn’t even figure out the RHEL7 host name. I
have no idea what other things might not work. Finally, I was able to get
the GDAL image types detected in the GeoServer stores with SWIG 1.3.40 and
GDAL 2.1.0. Even though I found the source for ImageIO-ext on github, they
include libraries where the source is not available, or at least I couldn’t
find it. ImageIO-ext doesn’t seem to be upgraded for SWIG 2.x or 3.x. For
example, the dependency imageio-ext-gdal-bindings seems pretty important but
it is stuck at GDAL 1.9.2 and most likely still on the old SWIG api.

https://github.com/geosolutions-it/imageio-ext/search?utf8=%E2%9C%93&q=imageio-ext-gdal-bindings

So, where do I go with my questions? It doesn’t seem like I could even patch
this on my own because of hidden libraries. I have no idea if all the
features will even work on RHEL7 because I can’t even trust the linking. Is
there a reason why everything is stuck on old libraries and old OS? Is this
the right board for these kind of issues? If not, what would be?

Thanks for reading,
Brian


View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-on-RHEL7-questions-tp5270708.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.


What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Hello Brian

To add to what Daniele said, note that the e imageio-ext-gdal-bindings-1.9.2.jar included with the GDAL plugin is simply the SWIG bindings for GDAL 1.9.2. If you have generated your own GDAL java bindings using SWIG, you should delete this jar, and instead use the one you generated. Otherwise, it can cause classloader conflicts with the generated jar. See GEOS-7578 for more info on this issue.

Torben

···

On Wed, Jun 15, 2016 at 5:22 AM, Daniele Romagnoli <daniele.romagnoli@anonymised.com> wrote:

Hi Brian,
The “GDAL native libs installation on different Linux OSs” is a hot topic.

Summarizing some key points:

  • Right now GeoServer depends on ImageIO-EXT GDAL plugins which still rely on GDAL 1.9.2.
  • Some users did experiments to build and use higher versions of GDAL (1.x, 2.x).
  • Supporting different GDAL versions on a different OS usually requires GDAL manual build on the hosting system itself to make sure there no linking issues or conflicts with the required third party libraries and dependencies.

There is a couple of discussions about this topic:

https://sourceforge.net/p/geoserver/mailman/message/34924220/

https://sourceforge.net/p/geoserver/mailman/message/34982403/

Hope they can provide you some more hints.

Additional note: In the past, we have also noticed that SWIG 1.3.36 seemed a good version to produce the expected signatures with respect to others which were resulting in mismatching methods.

Please, let us know if you still need assistance or clarifications on that.

Cheers,
Daniele


What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Wed, Jun 8, 2016 at 8:54 PM, brian.blain <brian.blain@anonymised.com> wrote:

This started out as a user issue but I think I have gone pretty far down the
rabbit hole and I have a few questions that I wonder if someone on this
mailing list could answer.

Goal: Deploy the GeoServer war including the GDAL plugin on WebLogic 12.1.3
on RHEL7.

Initially, I was able to get this setup working on Windows using the
binaries from ImageIO-ext. The instructions were pretty clear and things
worked after a day or so of figuring things out. It was when I moved to
deploying onto a RHEL7 server that I started having issues. ImageIO-ext also
does not provide binaries for RHEL7 so I was taking a gamble. Once GDAL and
the java bindings installed, booting up GeoServer gave me errors like this
one for all the GDAL types.

java.lang.UnsatisfiedLinkError: org.gdal.gdal.gdalJNI.SWIGDriverUpcast(J)J
at org.gdal.gdal.gdalJNI.SWIGDriverUpcast(Native Method)
at org.gdal.gdal.Driver.(Driver.java:15)
at org.gdal.gdal.gdal.GetDriverByName(gdal.java:264)

In the log, there was also a record of the GDAL version being detected and
it never threw the “no gdaljni in java.library.path” so I could assume that
the JNI detection was working but there was a different link failing. Of
course, there is some hindsight in my email here and I spent way more time
than I wanted to on this issue. After digging and learning more about what
SWIG was doing to wrap GDAL and all the jar dependencies from ImageIO-ext I
finally ran across that SWIG 2 was installed and that there was a major
change between 1.x and 2.x versions, particularly in the class signatures
created by the java.swg template.

swig-1.3.40/Lib/java/java.swg

PTRCTOR_VISIBILITY $javaclassname(long cPtr, boolean cMemoryOwn) {
super($imclassname.SWIG$javaclassnameUpcast(cPtr), cMemoryOwn);
swigCPtr = cPtr;
}

swig-2.0.0/Lib/java/java.swg

PTRCTOR_VISIBILITY $javaclassname(long cPtr, boolean cMemoryOwn) {
super($imclassname.$javaclazznameSWIGUpcast(cPtr), cMemoryOwn);
swigCPtr = cPtr;
}

Notice that they changed the order of the SWIG Upcast method which resulted
in the java class have an incorrect signature. (I have no idea why a
underscore got in there.) The error from above is looking for a method that
does not exist.

protected Driver(long cPtr, boolean cMemoryOwn) {
super(gdalJNI.Driver_SWIGUpcast(cPtr), cMemoryOwn);
swigCPtr = cPtr;
}

The end result was that I uninstalled SWIG 2.x and ended up building SWIG
1.3.40 from source. SWIG 1.x couldn’t even figure out the RHEL7 host name. I
have no idea what other things might not work. Finally, I was able to get
the GDAL image types detected in the GeoServer stores with SWIG 1.3.40 and
GDAL 2.1.0. Even though I found the source for ImageIO-ext on github, they
include libraries where the source is not available, or at least I couldn’t
find it. ImageIO-ext doesn’t seem to be upgraded for SWIG 2.x or 3.x. For
example, the dependency imageio-ext-gdal-bindings seems pretty important but
it is stuck at GDAL 1.9.2 and most likely still on the old SWIG api.

https://github.com/geosolutions-it/imageio-ext/search?utf8=%E2%9C%93&q=imageio-ext-gdal-bindings

So, where do I go with my questions? It doesn’t seem like I could even patch
this on my own because of hidden libraries. I have no idea if all the
features will even work on RHEL7 because I can’t even trust the linking. Is
there a reason why everything is stuck on old libraries and old OS? Is this
the right board for these kind of issues? If not, what would be?

Thanks for reading,
Brian


View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-on-RHEL7-questions-tp5270708.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.


What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.