[GRASS5] Using external PROJ with 5.3

GRASS GRASS developer
1

Hello
Since Frank has now confirmed that PROJ bug number 368
http://bugzilla.remotesensing.org/show_bug.cgi?id=368
is in fact a feature and will be staying, I have put a workaround into the
5.3 modules that were affected (I still have to do ps.map in 5.7) that
uses the PROJ pj_latlong_from_proj() function to preserve the datum
information when converting to/from lat/long co-ordinates on the same
ellipsoid.

Just as a reminder, the problem was if you had a nadgrids line in your
PROJ_INFO file and were using external PROJ (not the internal GRASS PROJ),
a datum shift would be incorrectly applied and you would get the wrong
results when converting to/from latitude/longitude on the same ellipsoid
in modules such as r.sun, g.region -l, v.mkquads etc.

But now that this has been sorted out there is no reason not to use an
external PROJ (e.g. the latest PROJ.4.4.8) and I will make this
(--with-proj=yes) the default in the configure script for 5.3.0. I also think
the libproj shared library should probably be distributed with GRASS binaries,
as libgdal is now. Should prevent lots of spurious bug reports from people who
haven't installed libproj properly, as it will already be in their
LD_LIBRARY_PATH.

Paul

2

On Thursday 13 May 2004 11:46, Paul Kelly wrote:

I also think the libproj shared library should probably be distributed with GRASS
binaries, as libgdal is now. Should prevent lots of spurious bug reports
from people who haven't installed libproj properly, as it will already be
in their LD_LIBRARY_PATH.

If I understand the situation well, I don't think that any external
library should be distributed with GRASS. Say that somebody
has GDAL 1.9 installed on his system:
/usr/local/lib/libgdal.1.1.so
/usr/local/bin/<gdal programs>
his LD_LIBRARY_PATH is empty and it works well for him, then he installs
GRASS with libgdal.1.1.so from GDAL 1.8.
In GRAS shell, the (old) version of libgdal.1.1.so from GRASS will
be used and all GDAL programs from /usr/local/bin/ will use wrong library
from GRASS.

Radim

3

Hello Radim

On Fri, 14 May 2004, Radim Blazek wrote:

If I understand the situation well, I don't think that any external
library should be distributed with GRASS. Say that somebody
has GDAL 1.9 installed on his system:
/usr/local/lib/libgdal.1.1.so
/usr/local/bin/<gdal programs>
his LD_LIBRARY_PATH is empty and it works well for him, then he installs
GRASS with libgdal.1.1.so from GDAL 1.8.

It won't be 1.8 unless someone prepares a bad binary distribution; the release rules for GRASS say to use the latest GDAL CVS and I think it is a good idea to include a GDAL version that has been tried and tested to work with the version of GRASS Being distributed

In GRAS shell, the (old) version of libgdal.1.1.so from GRASS will
be used and all GDAL programs from /usr/local/bin/ will use wrong library
from GRASS.

I also think it would be (fairly) intuitive to the user there to run the GDAL commands outside GRASS if he was getting bad results. More intuitive than the problem of installing gdal and proj libraries in /usr/local/lib but not updating LD_LIBRARY_PATH or /etc/ld.so.conf. Going by the experience with proj in 5.7 I would expect to get many 'bug' reports about like that and would like to try and cut down the workload on replying to them.

Of course I'm not saying that including GDAL and PROJ Libraries with GRASS is a perfect elegant solution. But I can't see anything really bad about it either. I am assuming of course that due to the way LD_LIBRARY_PATH is set, it is only possible for these libraries to be used if you are inside a GRASS session?

Reducing the amount of bug reports and simple questions about installing GDAL and PROJ is my main motivation in saying they should be included in the binary distributions.

Paul

4

On Friday 14 May 2004 14:38, Paul Kelly wrote:

I also think it would be (fairly) intuitive to the user there to run the
GDAL commands outside GRASS if he was getting bad results. More intuitive
than the problem of installing gdal and proj libraries in /usr/local/lib
but not updating LD_LIBRARY_PATH or /etc/ld.so.conf. Going by the
experience with proj in 5.7 I would expect to get many 'bug' reports about
like that and would like to try and cut down the workload on replying to
them.

Of course I'm not saying that including GDAL and PROJ Libraries with GRASS
is a perfect elegant solution. But I can't see anything really bad about
it either. I am assuming of course that due to the way LD_LIBRARY_PATH is
set, it is only possible for these libraries to be used if you are inside
a GRASS session?

Reducing the amount of bug reports and simple questions about installing
GDAL and PROJ is my main motivation in saying they should be included in
the binary distributions.

OK, I agree with idea, but maybe we could do it in more secure way?
E.g. optionaly copy libgdal.so to $ARCH/libgrass_gdal.so and build
GRASS against this copy? There is already libgrass_shape.so there.

Another question, how should be built GDAL included in GRASS,
and where is the limit for dependencies? GDAL should have at least
xerces and Postgres but probably more, so include also all those libs?

Radim

5

Radim Blazek wrote:

OK, I agree with idea, but maybe we could do it in more secure way?
E.g. optionaly copy libgdal.so to $ARCH/libgrass_gdal.so and build GRASS against this copy? There is already libgrass_shape.so there.

Another question, how should be built GDAL included in GRASS,
and where is the limit for dependencies? GDAL should have at least
xerces and Postgres but probably more, so include also all those libs?

Radim / Paul,

I would agree that renaming libgdal.so to something grass specific
would ensure there is no interference between the GRASS version of GDAL
and any external copy on the system; however, it would make it somewhat
harder for people to install a more recent (or more fully configured) GDAL
and make it work properly. I guess we could just not that if they want to
make r.in.gdal use a different GDAL they can link $ARCH/libgrass_gdal.so to
their GDAL shared library.

I would also note that if you are going to distribute a GDAL shared library
with GRASS you also need to carry along the support data (normally installed
as /usr/local/shared/gdal) with various EPSG tables, and other support data
files used by specific raster and vector drivers. These could live in the
GRASS tree, but r.in.gdal would need to be altered slightly to push an
appropriate file search location on the GDAL data search path.

As for build options I would suggest that PNG, JPEG use external shared
libraries which are assumed to be present on the system. The GIF driver
should be configured --with-gif=internal as should the TIFF and GeoTIFF
driver's unless you want to distribute libtiff and libgeotiff shared
libraries with GRASS. It is often necessary to have fairly recent versions
of these so depending on a system version is risky.

Is ODBC used elsewhere in GRASS? If so, I think GDAL/OGR ODBC support should
also be enabled.

Also, I think HDF support is very desirable to the GRASS user community though
it will tend to add a bunch of additional complexity.

Oracle, ECW, MrSID and JPEG2000 should likely not be included (well this may
change when the ECW source is "opened" at which point ECW and JPEG2000 might
well become practical).

Best regards,

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

6

Hello Frank

On Fri, 14 May 2004, Frank Warmerdam wrote:

[...]

I would also note that if you are going to distribute a GDAL shared library
with GRASS you also need to carry along the support data (normally installed
as /usr/local/shared/gdal) with various EPSG tables, and other support data
files used by specific raster and vector drivers. These could live in the
GRASS tree, but r.in.gdal would need to be altered slightly to push an
appropriate file search location on the GDAL data search path.

Good point. The existing GRASS release rules say only to include libgdal.1.1.so; has this not been a problem for anyone already? In GRASS 5.7 local copies of some of these files are included for co-ordinates system definitions---the function SetCSVFilenameHook() is used in a few places to set the correct location to look for these and it works well. But the point of doing that was that I thought these files weren't guaranteed to be included in a GDAL installation. But if as you say they are necessary for some drivers then we probably could rely on them being installed. Several issues there I suppose with no real answers.

Maybe just make a PROJ and GDAL binary distribution that have been tested with the GRASS binaries, and distribute them all together. Is there an easy equivalent to the GRASS 'make bindist' for making PROJ.4 and GDAL binary packages?

Paul

7

Paul Kelly wrote:

Good point. The existing GRASS release rules say only to include libgdal.1.1.so; has this not been a problem for anyone already? In GRASS 5.7 local copies of some of these files are included for co-ordinates system definitions---the function SetCSVFilenameHook() is used in a few places to set the correct location to look for these and it works well. But the point of doing that was that I thought these files weren't guaranteed to be included in a GDAL installation. But if as you say they are necessary for some drivers then we probably could rely on them being installed. Several issues there I suppose with no real answers.

Paul,

Lots of GDAL distributions get built and sent out without the data
files setup properly, but I don't think having a local copy is a very
good idea unless a GRASS binary distribution make prepares it along with
including a copy of GDAL.

Maybe just make a PROJ and GDAL binary distribution that have been tested with the GRASS binaries, and distribute them all together. Is there an easy equivalent to the GRASS 'make bindist' for making PROJ.4 and GDAL binary packages?

There are or were mkbindist.sh scripts in some of my packages but they
are not in good repair. PROJ and GDAL do have rpm packagings.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

8

I would like to make a suggestion for adding support for vertical
datum into GRASS or at least add the info to the PROJ_INFO file,
so that the conversion can be added later without the need
to change the PROJ_INFO file, e.g.:

name: State Plane
datum: nad83
vdatum: navd88
....

It would be great if this could go into 5.7 before we have a stable
release with the new vector format.
I noticed that PROJ supports only horizontal datum too,
or am I missing something? Are there any plans to add it to PROJ?
It is really important for working with elevation data -
when the vertical datums are mixed up, place that is sinking may appear growing
and the difference can be from few cm to several meters.

I looked for some public domain software - for USA, NOAA/USGS has some
source code and grids available but the licence has some restrictions,
I am not sure how to interpret it. Also, the software is FORTRAN.
The conversion is relatively simple as it involves interpolation on a grid,
but the grid files are rather large.

Below is a link and info about the NOAA programs, in case that somebody
can look at it. I am not sure what would be available for Europe and other parts
of the world,

Helena

http://chartmaker.ncd.noaa.gov/bathytopo/vdatum.htm

VDatum
VDatum is under development as part of the joint NOAA/USGS Bathymetric-Topographic
Demonstration Project, and is a tool for the conversion of elevation data among
28 different vertical datums. Vertical datums can be based on Mean Sea Level
(such as NAVD88), tidally derived surfaces (nautical charts) or three dimensional
space systems (GPS, for example).

VERTCON (Version 2.1)
Computes the modeled difference in orthometric height between the North American
Vertical Datum of 1988 (NAVD 88) and the National Geodetic Vertical Datum of 1929
(NGVD 29) for a given location specified by latitude and longitude.
This conversion is sufficient for many mapping purposes.
An on-line interactive version is also available from the Geodetic Tool Kit.

# GEOID99

Modeled geoid height values with a 1-minute by 1-minute spacing in latitude
and longitude suitable for the conversion of NAD 83 ellipsoid heights into
NAVD 88 orthometric heights. GEOID99 includes software programs XNTG and INTG
for extracting and interpolating values from the model to discrete points.
INTD is also available on line for interactive computations over the web.
GEOID99 replaces the existing GEOID96 product.

                 DISCLAIMER *
* *
* THIS PROGRAM AND SUPPORTING INFORMATION IS FURNISHED BY THE *
* GOVERNMENT OF THE UNITED STATES OF AMERICA, AND IS ACCEPTED AND *
* USED BY THE RECIPIENT WITH THE UNDERSTANDING THAT THE UNITED STATES *
* GOVERNMENT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, CONCERNING THE *
* ACCURACY, COMPLETENESS, RELIABILITY, OR SUITABILITY OF THIS *
* PROGRAM, OF ITS CONSTITUENT PARTS, OR OF ANY SUPPORTING DATA. *
* *
* THE GOVERNMENT OF THE UNITED STATES OF AMERICA SHALL BE UNDER NO *
* LIABILITY WHATSOEVER RESULTING FROM ANY USE OF THIS PROGRAM. THIS *
* PROGRAM SHOULD NOT BE RELIED UPON AS THE SOLE BASIS FOR SOLVING A *
* PROBLEM WHOSE INCORRECT SOLUTION COULD RESULT IN INJURY TO PERSON *
* OR PROPERTY. *
* *
* THIS PROGRAM IS PROPERTY OF THE GOVERNMENT OF THE UNITED STATES *
* OF AMERICA. THEREFORE, THE RECIPIENT FURTHER AGREES NOT TO ASSERT *
* PROPRIETARY RIGHTS THEREIN AND NOT TO REPRESENT THIS PROGRAM TO *
* ANYONE AS BEING OTHER THAN A GOVERNMENT PROGRAM.

9

Hello Helena

Helena wrote:

I would like to make a suggestion for adding support for vertical
datum into GRASS or at least add the info to the PROJ_INFO file,
so that the conversion can be added later without the need
to change the PROJ_INFO file, e.g.:

name: State Plane
datum: nad83
vdatum: navd88
....

First of all I would add that I think this could be awkward as there is
no standard way for GRASS to know that a particular raster or sites
layer in the database actually contains height information, or what the
units would be. As far as I understand it each location could have any
number of raster layers representing a digital elevation model, but to
GRASS each would be just another floating point raster layer with no
information about what the numbers in it represent (that they are
elevation data for the location, or which units they are in). If
elevation data are coming from different sources (GPS, contours, raster
DEMs etc.) there may be raster layers representing lots of different
vertical datums contained in the same location. And there is no way of
marking a particular raster or sites layer to say that this is the
height layer for that location. I know I repeated what I said a bit
there but I hope it makes sense.

It would be great if this could go into 5.7 before we have a stable
release with the new vector format.
I noticed that PROJ supports only horizontal datum too,
or am I missing something? Are there any plans to add it to PROJ?
It is really important for working with elevation data -
when the vertical datums are mixed up, place that is sinking may appear growing
and the difference can be from few cm to several meters.

I don't think supporting vertical datums involves adding any extra
functionality to PROJ---really it only involves adding or substracting
data values and is independent of the 2-D location/projection of those
data values?

I looked for some public domain software - for USA, NOAA/USGS has some
source code and grids available but the licence has some restrictions,
I am not sure how to interpret it. Also, the software is FORTRAN.
The conversion is relatively simple as it involves interpolation on a grid,
but the grid files are rather large.

Below is a link and info about the NOAA programs, in case that somebody
can look at it. I am not sure what would be available for Europe and other parts
of the world,

http://chartmaker.ncd.noaa.gov/bathytopo/vdatum.htm

I had a look at the link and as far as I can see, the software consists
of a selection of grid files (i.e. raster layers) and all it does is
interpolate these grids to find the height offset for a particular 2-D
position and add or subtract this. This is quite simple and the
functionality is in GRASS already (r.mapcalc).

There are lots of localised grids even to cover just a few parts of USA
as the NOAA software does---it uses an ASCII format which is quite large
and would be inefficient to re-distribute. I think any raster format
e.g. GeoTIFF would be appropriate. However there would be too much data
to include with GRASS---it is not something 'compact' that can be stored
in a short text file, like a list of 7 numbers associated with each
horizontal datum.

Something else similar is the OSGM02 Geoid model of the British
Isles---if you register on the Ordnance Survey of Great Britain website
you can download the data for free (see http://www.gps.gov.uk/ )---as I
remember it came in comma-separated variable ASCII text format.

I have used the OSGM02 model in the way I described (creating a GRASS
raster by interpolating the grid data using s.surf.rst, and then
subtracting this from GPS heights using r.mapcalc) with success to
create an orthometric height layer for parts of Northern Ireland.

So to summarise as far as I can see it: the required functionality is
already in GRASS---it is just a data processing/conversion job to import
the ASCII grids (.GTX files in the case of the NOAA software) as a GRASS
raster---or as sites and then interpolate with s.surf.rst.

In the long run perhaps the GRASS database could be extended to have a
facility to mark a specific raster layer (e.g. something with a special
name like is currently done for MASK) as representing a digital
elevation model for that location. Modules that currently have a
command-line parameter for the elevation could then be changed to use
that layer automatically. And the units of the elevation data and name
of the vertical datum could be stored somewhere else in the database.
I'm not sure if this is a good idea though. Has anybody experience of
how other GIS software handles this?

I hope my ideas are of some help and provoke further discussion.

Paul