[GRASS5] Re: OGR-enabled GDAL

On Tue, 26 Aug 2003, Markus Neteler wrote:

Hello Paul,

On Tue, Aug 26, 2003 at 11:20:55AM +0100, Paul Kelly wrote:
> Hello Markus
> I don't understand the new OGR test in GRASS 5.1; it fails for me even
> though my GDAL definitely has OGR included. I think the test for the output
> of 'gdal-config --ogr-enabled' is enough to see if the GDAL installation
> includes OGR support.

Well, it works on Linux and Mac OSX.

Yes, it could be removed, but... :

> I feel gdal-config is supposed to handle all these tests without having to
> do any of the LOC_CHECK_LIBS stuff. Can I change it back or is there an
> important reason for it?

We can change back if we solve the problem that
vector/Makefile
depends on entries in Platform.make where we didn't have any information
about OGR. That's why I was adding the test.
Obviously it must be improved.

I see. This is the general problem that there is no mechanism in the 5.1
build system for optionally compiling programs, like LOC_OPTIONAL in 5.0.
I think it should be a more general solution as there are lots of modules,
like this, e.g. that depend on libtiff, libpng, OpenGL etc. and they
should all be compiled optionally depending on what the configure script
finds.

It looks like the alternative build system (grass/mk) in GRASS 5.0 gives
some ideas of how this could be done. It comes down to maintaining the
old-style module lists (src/CMD/lists/GRASS and src/CMD/lists/optional)
in 5.1 and the top-level makefile compiling all the modules contained in
those lists. Instead of the sub-directory Makefiles containing a list of
all the modules like they do now (why was it changed like this when 5.1
CVS was set up?).

Needs a bit of work.

Paul

On Tuesday 26 August 2003 13:38, Paul Kelly wrote:

> We can change back if we solve the problem that
> vector/Makefile
> depends on entries in Platform.make where we didn't have any information
> about OGR. That's why I was adding the test.
> Obviously it must be improved.

I see. This is the general problem that there is no mechanism in the 5.1
build system for optionally compiling programs, like LOC_OPTIONAL in 5.0.
I think it should be a more general solution as there are lots of modules,
like this, e.g. that depend on libtiff, libpng, OpenGL etc. and they
should all be compiled optionally depending on what the configure script
finds.

Why variables from Platform.make cannot be used as conditions in Makefiles
(like in vector/Makefile and db/drivers/Makefile)?

It looks like the alternative build system (grass/mk) in GRASS 5.0 gives
some ideas of how this could be done. It comes down to maintaining the
old-style module lists (src/CMD/lists/GRASS and src/CMD/lists/optional)
in 5.1 and the top-level makefile compiling all the modules contained in
those lists. Instead of the sub-directory Makefiles containing a list of
all the modules like they do now (why was it changed like this when 5.1
CVS was set up?).

As I wrote already many times, 51 cvs was created to enable to work together
on new vectors (when David was here). I did the best I was able to do,
I used another 'alternative build system' which was available that time.
I thought that some expert can improve/replace it later, unfortunately nobody
did anything better.
So again, I don't insist that current make system should be kept for future.
Don't hesitate to remove it and do something better!

Ra

Hello Radim

On Tue, 26 Aug 2003, Radim Blazek wrote:

> I see. This is the general problem that there is no mechanism in the 5.1
> build system for optionally compiling programs, like LOC_OPTIONAL in 5.0.
> I think it should be a more general solution as there are lots of modules,
> like this, e.g. that depend on libtiff, libpng, OpenGL etc. and they
> should all be compiled optionally depending on what the configure script
> finds.

Why variables from Platform.make cannot be used as conditions in Makefiles
(like in vector/Makefile and db/drivers/Makefile)?

Maybe---I don't know what is best but it just seems a more elegant
solution to me to do it all within the configure script. I don't
understand it well enough to explain; I just have a feeling about what is
the best way to do it :slight_smile:

> It looks like the alternative build system (grass/mk) in GRASS 5.0 gives
> some ideas of how this could be done. It comes down to maintaining the
> old-style module lists (src/CMD/lists/GRASS and src/CMD/lists/optional)
> in 5.1 and the top-level makefile compiling all the modules contained in
> those lists. Instead of the sub-directory Makefiles containing a list of
> all the modules like they do now (why was it changed like this when 5.1
> CVS was set up?).

As I wrote already many times, 51 cvs was created to enable to work together
on new vectors (when David was here). I did the best I was able to do,
I used another 'alternative build system' which was available that time.
I thought that some expert can improve/replace it later, unfortunately nobody
did anything better.
So again, I don't insist that current make system should be kept for future.
Don't hesitate to remove it and do something better!

I always thought it was quite good. I just don't understand it well enough
to know if there was no optional building on purpose. But thinking about
it now I suppose the optional build system was only added to 5.0
when the 5.1 build system was already in operation (this answers the
question). So there would be no harm then in adding the optional build system
to 5.1 as well maybe.

Sometime.

Paul

Paul Kelly wrote:

> > I see. This is the general problem that there is no mechanism in the 5.1
> > build system for optionally compiling programs, like LOC_OPTIONAL in 5.0.
> > I think it should be a more general solution as there are lots of modules,
> > like this, e.g. that depend on libtiff, libpng, OpenGL etc. and they
> > should all be compiled optionally depending on what the configure script
> > finds.
>
> Why variables from Platform.make cannot be used as conditions in Makefiles
> (like in vector/Makefile and db/drivers/Makefile)?

Maybe---I don't know what is best but it just seems a more elegant
solution to me to do it all within the configure script. I don't
understand it well enough to explain; I just have a feeling about what is
the best way to do it :slight_smile:

There are advantages for both approaches.

One advantage of the src/CMD/lists approach is that it's easier to
implement error recovery; failure to build an individual entry in the
list doesn't terminate the rule.

To do this with a conventional "recursive make", each "directory"
Makefile would have to explicitly build its subdirectories in such a
way that failures don't propagate; e.g. by using a loop, so that the
entire process is a single command from make's perspective, or by
forcing a successful exit status from each command ("... ; true").

Similarly, there are advantages to having the test for whether to
build a particular module localised to that module's Makefile. OTOH,
the test would have to be repeated for each rule (program, docs,
install, etc), whereas modifying a global modules list would affect
all rules.

> > It looks like the alternative build system (grass/mk) in GRASS 5.0 gives
> > some ideas of how this could be done. It comes down to maintaining the
> > old-style module lists (src/CMD/lists/GRASS and src/CMD/lists/optional)
> > in 5.1 and the top-level makefile compiling all the modules contained in
> > those lists. Instead of the sub-directory Makefiles containing a list of
> > all the modules like they do now (why was it changed like this when 5.1
> > CVS was set up?).
>
> As I wrote already many times, 51 cvs was created to enable to work together
> on new vectors (when David was here). I did the best I was able to do,
> I used another 'alternative build system' which was available that time.
> I thought that some expert can improve/replace it later, unfortunately nobody
> did anything better.
> So again, I don't insist that current make system should be kept for future.
> Don't hesitate to remove it and do something better!

I always thought it was quite good. I just don't understand it well enough
to know if there was no optional building on purpose. But thinking about
it now I suppose the optional build system was only added to 5.0
when the 5.1 build system was already in operation (this answers the
question).

That's correct. I did quite a lot of work on the 5.0 build system
after the 5.1 system had been derived from it.

So there would be no harm then in adding the optional build system
to 5.1 as well maybe.

However, note that a number of decisions regarding the 5.0 build
system were constrained by compatibility issues. The alternate build
system was constrained by the need to avoid interfering with the
gmake5 mechanism, and other aspects were then constrained by the
desire to maintain compatibility with both build mechanisms
(personally, I only use the original gmake5 mechanism to test that it
still works).

I would like to revisit the build system issue at some point. Key
issues for me are:

1. Out-of-place builds (i.e. not modifying the source tree).

2. Building from a read-only source tree (closely related to point 1).

3. Correct handling of dependencies.

4. Supporting shared libraries, without using libtool (related to
point 3).

5. Tying building of documentation to building the corresponding
module (i.e. no installing manpages for non-existent modules).

Point 1 is already dealt with by the alternate build mechanism. The
main issue with point 2 is the use of "cp" instead of "install" (if
you use "cp" on a read-only file, the copy is also read-only).

Point 3 has been significantly improved in the 5.0.x tree, primarily
through having separate FOOLIB and DEPFOOLIB variables. This also
deals with the main issue for shared libraries (you can't just use the
library's pathname as a linker argument).

Point 5 involves moving the documentation into the module directories,
so it's rather more involved than the others.

--
Glynn Clements <glynn.clements@virgin.net>

On Wednesday 27 August 2003 06:40, Glynn Clements wrote:

To do this with a conventional "recursive make", each "directory"
Makefile would have to explicitly build its subdirectories in such a
way that failures don't propagate; e.g. by using a loop, so that the
entire process is a single command from make's perspective, or by
forcing a successful exit status from each command ("... ; true").

Subdirectories defined in SUBDIRS variable are processed by subdirs rule
in Dir.make:
subdirs:
        @list='$(SUBDIRS)'; \
        for subdir in $$list; do \
            echo $$subdir ; \
            (cd $$subdir && make) || exit 1; \
        done

So this can be easily modified, I think to get over error.

Similarly, there are advantages to having the test for whether to
build a particular module localised to that module's Makefile. OTOH,
the test would have to be repeated for each rule (program, docs,
install, etc), whereas modifying a global modules list would affect
all rules.

I think that there are only few cases, if any, where conditions must
be tested for documents (see below) and installation.

I would like to revisit the build system issue at some point. Key
issues for me are:

1. Out-of-place builds (i.e. not modifying the source tree).

2. Building from a read-only source tree (closely related to point 1).

OK, so I think that because of 1. and 2. the build system must be
completely rewritten.

Point 3 has been significantly improved in the 5.0.x tree, primarily
through having separate FOOLIB and DEPFOOLIB variables. This also
deals with the main issue for shared libraries (you can't just use the
library's pathname as a linker argument).

We have now separate *LIB and *DEP for all libraries.

Point 5 involves moving the documentation into the module directories,
so it's rather more involved than the others.

Documentation (at least manual pages) _is_ moved already to module directories,
thanks to Markus, who has created the system generating man pages from
options' description in the module + additional description.
(Potential problem is cross compilation as you mentioned in other mail.)
It works well. Regularly, Markus also tries to force me to write
'additional description' for new modules, but that does not work
so well :slight_smile:

Ra

On Wed, Aug 27, 2003 at 11:32:46AM +0200, Radim Blazek wrote:
[...]

> Point 5 involves moving the documentation into the module directories,
> so it's rather more involved than the others.

Documentation (at least manual pages) _is_ moved already to module directories,
thanks to Markus, who has created the system generating man pages from
options' description in the module + additional description.
(Potential problem is cross compilation as you mentioned in other mail.)
It works well. Regularly, Markus also tries to force me to write
'additional description' for new modules, but that does not work
so well :slight_smile:

I don't give up hope :slight_smile:

Here you can see all auto-generated manual pages (updated
every Saturday):
http://grass.itc.it/grass51/manuals/html51_user/index.html

inclusive full index and notes (new).
The script building these pages even tells you which modules
are missing the file 'description.html'.

Paul has made it portable, so I don't expect to many problems
(works at least on GNU/Linux, IRIX and Mac OSX).

The docs build process only fails on adding the menu parameters
in case "strange" things are done in the parser part of the
command. I tried to work around that but a few commands are
not updated yet. Probably the approach should be changed,
but at time it almost works.

Even images can be included now in the documentation, e.g.:
http://grass.itc.it/grass51/manuals/html51_user/photo.camera.html

Markus

Radim Blazek wrote:

> I would like to revisit the build system issue at some point. Key
> issues for me are:
>
> 1. Out-of-place builds (i.e. not modifying the source tree).
>
> 2. Building from a read-only source tree (closely related to point 1).

OK, so I think that because of 1. and 2. the build system must be
completely rewritten.

When I did it for the 5.0 alternate build mechanism, most of the
changes boiled down to:

a) the configure script, and

b) bits in individual Gmakefiles which relied upon creating files in
the current directory.

The build system itself only required minor changes. Actually, b) was
only necessary because the generated Makefiles reside in the source
tree, which meant that the current directory was the source directory.

A conventional out-of-place build system creates Makefiles in the
build tree, has the build directory as the current directory and uses
VPATH to locate source files. OTOH, most packages don't have as many
directories as GRASS.

I don't think that the usual approach of having configure generate a
bunch of "Makefile"s from a list of "Makefile.in"s using AC_OUTPUT()
is going to be feasible, as generating each file involves using a
massive "sed" command to substitute the entire autoconf environment.
Assuming that we stick to GNU make (or, at least, makes which have an
"include" directive), the Makefiles can be copied verbatim; they just
need to be copied from the source tree to the build tree.

> Point 3 has been significantly improved in the 5.0.x tree, primarily
> through having separate FOOLIB and DEPFOOLIB variables. This also
> deals with the main issue for shared libraries (you can't just use the
> library's pathname as a linker argument).

We have now separate *LIB and *DEP for all libraries.

OK; that's the main "enabling" requirement. Once that is the case, a
libraries dependencies should be made part of the *LIB variable, e.g.

  GISLIB = -lgis $(DATETIMELIB) $(XDRLIB)

I can see that this is already done to an extent in grass51; it just
remains to do it systematically. In general, if a library requires a
certain depedency, that should be part of a the *LIB variable. The
Makefile for an individual module should only refer to those libraries
which the module uses directly (e.g. most modules shouldn't refer to
$(DATETIMELIB) explicitly).

In order to avoid gratuitous dependencies, certain peripheral
functions probably ought to be removed from the core libraries. E.g.
unix_socks.c probably shouldn't be in libgis, as this results in the
libgis having a dependency upon the socket libraries (on platforms
where socket() etc aren't in libc itself).

> Point 5 involves moving the documentation into the module directories,
> so it's rather more involved than the others.

Documentation (at least manual pages) _is_ moved already to module directories,
thanks to Markus, who has created the system generating man pages from
options' description in the module + additional description.
(Potential problem is cross compilation as you mentioned in other mail.)
It works well. Regularly, Markus also tries to force me to write
'additional description' for new modules, but that does not work
so well :slight_smile:

OK; I should point out that my comments were directed at both 5.0 and
grass51. For 5.3 (or whatever the "intermediate" branch is called) we
need to decide whether to use the 5.0 gmake5 mechanism, the 5.0
alternate mechanism, the grass51 mechanism, or something else.

--
Glynn Clements <glynn.clements@virgin.net>

On Wed, 27 Aug 2003, Glynn Clements wrote:

Paul Kelly wrote:

> > > I see. This is the general problem that there is no mechanism in the 5.1
> > > build system for optionally compiling programs, like LOC_OPTIONAL in 5.0.
> > > I think it should be a more general solution as there are lots of modules,
> > > like this, e.g. that depend on libtiff, libpng, OpenGL etc. and they
> > > should all be compiled optionally depending on what the configure script
> > > finds.
> >
> > Why variables from Platform.make cannot be used as conditions in Makefiles
> > (like in vector/Makefile and db/drivers/Makefile)?
>
> Maybe---I don't know what is best but it just seems a more elegant
> solution to me to do it all within the configure script. I don't
> understand it well enough to explain; I just have a feeling about what is
> the best way to do it :slight_smile:

There are advantages for both approaches.

I have come round to the idea of using the Makefile variables included
from Platform.make in all the Makefiles. This involves not too many
changes, which is always best as it is less likely to break other things.
So for each enabled functionality, there already is a variable within the
configure script set to 1, e.g. USE_OGR=1 or USE_GD=1. So we just need to
add AC_SUBST(USE_OGR) to the configure script and USE_OGR=@USE_OGR@ in
Platform.make.in and the the value of USE_.. will be substituted into
Platform.make. And in the Makefile where the modules are listed, just test
for this variable not being empty and if so add the dependent module's
directory to the SUBDIRS line in the Makefile.

I did it for OGR (v.in.ogr/v.out.ogr) and GD (PNGdriver) in GRASS5.1 and
seems to work fine.

One advantage of the src/CMD/lists approach is that it's easier to
implement error recovery; failure to build an individual entry in the
list doesn't terminate the rule.

To do this with a conventional "recursive make", each "directory"
Makefile would have to explicitly build its subdirectories in such a
way that failures don't propagate; e.g. by using a loop, so that the
entire process is a single command from make's perspective, or by
forcing a successful exit status from each command ("... ; true").

Similarly, there are advantages to having the test for whether to
build a particular module localised to that module's Makefile. OTOH,
the test would have to be repeated for each rule (program, docs,
install, etc), whereas modifying a global modules list would affect
all rules.

The way I did it (that Markus had already started) is in the top-level
directory for each class of modules, i.e. vector/Makefile for all the
vector modules, and involves just compiling or not compiling the whole
directory so I think that works around that potential problem.

But the above change so one module failing doesn't stop the whole build
process and failed modules can be written to an error.log file, still
needs to be done some time.

Paul