[GRASS-dev] installed makefile cleanup

William_Kyngesburye · September 22, 2007, 8:28pm

I started looking at the installed makefile fragments to get them in working order to build addons. I'm mainly interested in building for external installation, so I'm doing a separate INST_XTN setup based on the existing INST_NOW (which builds into the GRASS installation).

This is what I have so far (all tests, nothing in CVS yet):

- first, a problem with INST_NOW. In Grass.make, the first: target has:

first: pre default
  @if test -n "$(INST_NOW)" ; then \
    $(MAKE) inst_now ; \
  fi

then there is an inst_now: target:

inst_now:
INST_NOW= $(MAKE)

That got into a nasty infinite loop for me when used as suggested:

make GRASS_HOME=. MODULE_TOPDIR=/Applications/GRASS-6.3.app/Contents/MacOS INST_NOW=y

So I ignored that for creating my INST_XTN variation.

- installed a copy of demolocation in the installed GRASS. edited .grassrc63 to set GISDBASE to the INST_DIR.

- Grass.make - add below the "ifdef INST_NOW" block:

ifdef INST_XTN
ARCH_DISTDIR = $(INST_DIR)
XTN_DISTDIR = $(GRASS_HOME)/dist.$(ARCH)
ARCH_INC = -I$(ARCH_DISTDIR)/include -I$(XTN_DISTDIR)/include
ARCH_LIBPATH = -L$(ARCH_LIBDIR) -L$(XTN_DISTDIR)/lib
BIN = $(XTN_DISTDIR)/bin
ETC = $(XTN_DISTDIR)/etc
DRIVERDIR = $(XTN_DISTDIR)/driver
DBDRIVERDIR = $(XTN_DISTDIR)/driver/db
endif

I needed to split ARCH_DISTDIR, which points to the installed GRASS, from the distdir for building the extension into. So, that required a few other changes in the other makefile frags:

- platform.make - replace the RUN_GISBASE setting:

ifdef INST_XTN
RUN_GISBASE = $(INST_DIR)
else
RUN_GISBASE = /Users/Shared/src/GRASS/cvs/grass6/dist.i686-apple-darwin8.10.1
endif

I guess, to get INST_NOW to also work, the source RUN_GISBASE should also be changed to the installed GRASS path.

- html.make - conditionally refer to ARCH_DISTDIR or XTN_DISTDIR as needed:

htmlgen:
  @$(MODULE_TOPDIR)/tools/mkhtml.sh $(PGM)
  -if [ "${INST_XTN}" = "y" ] ; then \
    $(MKDIR) $(XTN_DISTDIR)/docs/html ; \
    mv -f $(PGM).tmp.html $(XTN_DISTDIR)/docs/html/$(PGM).html ; \
  else \
    $(MKDIR) $(ARCH_DISTDIR)/docs/html ; \
    mv -f $(PGM).tmp.html $(ARCH_DISTDIR)/docs/html/$(PGM).html ; \
  fi
  -for file in *.png *.jpg ; do \
    head -n 1 $$file | grep '^#!' > /dev/null ; \
    if [ $$? -ne 0 ] ; then \
      if [ "${INST_XTN}" = "y" ] ; then \
        $(INSTALL_DATA) $$file $(XTN_DISTDIR)/docs/html ; \
      else \
        $(INSTALL_DATA) $$file $(ARCH_DISTDIR)/docs/html ; \
      fi \
    fi \
    done 2> /dev/null ; true

- install tools/mkhtml.sh in the installed GRASS

This works mostly. The compilation works. But html.make is having problems - running the module to generate the htmldesc complains about GISDBASE not being set and all I get is the description and no module options/usage.

GISDBASE is set correctly in .grassrc63 in the demolocation I installed. Setting a higher DEBUG level doesn't give me any helpful info.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

Theory of the Universe

There is a theory which states that if ever anyone discovers exactly what the universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarrely inexplicable. There is another theory which states that this has already happened.

-Hitchhiker's Guide to the Galaxy 2nd season intro

William_Kyngesburye · September 22, 2007, 9:53pm

On Sep 22, 2007, at 3:28 PM, William Kyngesburye wrote:

This works mostly. The compilation works. But html.make is having problems - running the module to generate the htmldesc complains about GISDBASE not being set and all I get is the description and no module options/usage.

GISDBASE is set correctly in .grassrc63 in the demolocation I installed. Setting a higher DEBUG level doesn't give me any helpful info.

Oops. When I edited the demolocation .grassrc63, I accidentally replaced the : with an = for the GISDBASE setting. It works now.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"This is a question about the past, is it? ... How can I tell that the past isn't a fiction designed to account for the discrepancy between my immediate physical sensations and my state of mind?"

- The Ruler of the Universe

Glynn_Clements1 · September 23, 2007, 9:48am

William Kyngesburye wrote:

I started looking at the installed makefile fragments to get them in
working order to build addons.

Note that I've just committed a bunch of changed to fix the issues
which I mentioned in my previous email.

I'm mainly interested in building for

external installation, so I'm doing a separate INST_XTN setup based
on the existing INST_NOW (which builds into the GRASS installation).

This is what I have so far (all tests, nothing in CVS yet):

- first, a problem with INST_NOW. In Grass.make, the first: target has:

first: pre default
  @if test -n "$(INST_NOW)" ; then \
    $(MAKE) inst_now ; \
  fi

then there is an inst_now: target:

inst_now:
  INST_NOW= $(MAKE)

That got into a nasty infinite loop for me when used as suggested:

make GRASS_HOME=. MODULE_TOPDIR=/Applications/GRASS-6.3.app/Contents/MacOS INST_NOW=y

Right; AFAICT, it needs to be passed as an environment variable, i.e.
"INST_NOW=y make".

If it is given as an argument, it will override any setting in the
Makefile, which will cause an infinite loop.

So I ignored that for creating my INST_XTN variation.

- installed a copy of demolocation in the installed GRASS.
edited .grassrc63 to set GISDBASE to the INST_DIR.

- Grass.make - add below the "ifdef INST_NOW" block:

ifdef INST_XTN
ARCH_DISTDIR = $(INST_DIR)
XTN_DISTDIR = $(GRASS_HOME)/dist.$(ARCH)
ARCH_INC = -I$(ARCH_DISTDIR)/include -I$(XTN_DISTDIR)/include
ARCH_LIBPATH = -L$(ARCH_LIBDIR) -L$(XTN_DISTDIR)/lib
BIN = $(XTN_DISTDIR)/bin
ETC = $(XTN_DISTDIR)/etc
DRIVERDIR = $(XTN_DISTDIR)/driver
DBDRIVERDIR = $(XTN_DISTDIR)/driver/db
endif

Regarding the last four, this is the wrong approach. It will break
everytime someone adds another variable for a subdirectory.

It appears that GISBASE needs to be split into source and destination
variables. The former would be used for -I/-L switches, the latter for
determining where to place output files.

I needed to split ARCH_DISTDIR, which points to the installed GRASS,
from the distdir for building the extension into. So, that required
a few other changes in the other makefile frags:

- platform.make - replace the RUN_GISBASE setting:

ifdef INST_XTN
RUN_GISBASE = $(INST_DIR)
else
RUN_GISBASE = /Users/Shared/src/GRASS/cvs/grass6/dist.i686-
apple-darwin8.10.1
endif

If GISBASE is split appropriately, it shouldn't be necessary to
conditionalise this.

- html.make - conditionally refer to ARCH_DISTDIR or XTN_DISTDIR as
needed:

Ditto.

htmlgen:
  @$(MODULE_TOPDIR)/tools/mkhtml.sh $(PGM)
  -if [ "${INST_XTN}" = "y" ] ; then \
    $(MKDIR) $(XTN_DISTDIR)/docs/html ; \
    mv -f $(PGM).tmp.html $(XTN_DISTDIR)/docs/html/$(PGM).html ; \
  else \
    $(MKDIR) $(ARCH_DISTDIR)/docs/html ; \
    mv -f $(PGM).tmp.html $(ARCH_DISTDIR)/docs/html/$(PGM).html ; \
  fi
  -for file in *.png *.jpg ; do \
    head -n 1 $$file | grep '^#!' > /dev/null ; \
    if [ $$? -ne 0 ] ; then \
      if [ "${INST_XTN}" = "y" ] ; then \
        $(INSTALL_DATA) $$file $(XTN_DISTDIR)/docs/html ; \
      else \
        $(INSTALL_DATA) $$file $(ARCH_DISTDIR)/docs/html ; \
      fi \
    fi \
    done 2> /dev/null ; true

- install tools/mkhtml.sh in the installed GRASS

This works mostly. The compilation works. But html.make is having
problems - running the module to generate the htmldesc complains
about GISDBASE not being set and all I get is the description and no
module options/usage.

GISDBASE is set correctly in .grassrc63 in the demolocation I
installed.

Does $(GISRC) actually refer to that file, though?

--
Glynn Clements <glynn@gclements.plus.com>

William_Kyngesburye · September 23, 2007, 5:30pm

On Sep 23, 2007, at 4:48 AM, Glynn Clements wrote:

William Kyngesburye wrote:

I started looking at the installed makefile fragments to get them in
working order to build addons.

Note that I've just committed a bunch of changed to fix the issues
which I mentioned in my previous email.

Cool.

- Grass.make - add below the "ifdef INST_NOW" block:

ifdef INST_XTN
ARCH_DISTDIR = $(INST_DIR)
XTN_DISTDIR = $(GRASS_HOME)/dist.$(ARCH)
ARCH_INC = -I$(ARCH_DISTDIR)/include -I$(XTN_DISTDIR)/include
ARCH_LIBPATH = -L$(ARCH_LIBDIR) -L$(XTN_DISTDIR)/lib
BIN = $(XTN_DISTDIR)/bin
ETC = $(XTN_DISTDIR)/etc
DRIVERDIR = $(XTN_DISTDIR)/driver
DBDRIVERDIR = $(XTN_DISTDIR)/driver/db
endif

Regarding the last four, this is the wrong approach. It will break
everytime someone adds another variable for a subdirectory.

It appears that GISBASE needs to be split into source and destination
variables. The former would be used for -I/-L switches, the latter for
determining where to place output files.

...

If GISBASE is split appropriately, it shouldn't be necessary to
conditionalise this.

- html.make - conditionally refer to ARCH_DISTDIR or XTN_DISTDIR as
needed:

Ditto.

(below, just trying to get a handle on the whole thing)

So, I came up with these "bases" for finding/creating files:

- INST_DIR - where make install target copies files to - can be overridden for installing to external location (I worked on this after my initial email).

- GRASS_HOME - source topdir (full path)
- MODULE_TOPDIR - source topdir (usually relative path)

- ARCH_DISTDIR - where targets are built INTO before installation, default based on GRASS_HOME

and these use them (that are of interest to an external build/install):

- RUN_GISBASE -> ARCH_DISTDIR - but right now that's expanded by configure

- RUN_GISRC -> ARCH_DISTDIR

- GISBASE -> ARCH_DISTDIR - usually reset to RUN_GISBASE (which is just ARCH_DISTDIR anyways), but the main makefile uses it directly for GRASS_PDFDIR

- demolocation -> ARCH_DISTDIR

- html.make -> ARCH_DISTDIR - for generating html files into

- makefile includes -> MODULE_TOPDIR
- doxygen commands in doxygen.make -> MODULE_TOPDIR
- FMODE_OBJ in grass.make -> MODULE_TOPDIR
- htmlgen in html.make -> MODULE_TOPDIR - to find tools/mkhtml.sh
- scriptstrings in script.make -> MODULE_TOPDIR - to find translate.c files

- BIN, ETC, DRIVERDIR -> ARCH_DISTDIR

- ARCH_LIBDIR, ARCH_INCDIR -> ARCH_DISTDIR

As used with the INST_NOW/INST_XTN makes, where GRASS_HOME is set to the module source dir and MODULE_TOPDIR is set to the installed GRASS binary, that creates a couple problems:

- ARCH_DISTDIR is local to the module source, but is used to find and use demolocation (RUN_GISBASE, RUN_GISRC).

INST_XTN will work with GRASS_HOME set to the module source dir, but won't find the installed demolocation.

- Also, ARCH_LIBDIR and ARCH_INCDIR are needed to find BOTH installed GRASS libs and headers AND module built libs and headers (for the rare addon module that has its own library). For the INST_NOW build, this works as is.

For the INST_XTN build, the installed GRASS lib/include would have to be added to ARCH_INC and ARCH_LIBPATH (slightly different than I currently have it).

So, as you said (it's making sense to me now), GISBASE somehow needs its own base dir setting, defaulting to ARCH_DISTDIR, and which would only need to be overridden by the INST_XTN build.

How about this: since GISBASE is really only for *running* GRASS, and should only be set as needed (fix that GRASS_PDFDIR usage), don't use it as a makefile var. RUN_GISRC is off the RUN_GISBASE dir, which configures to the same as ARCH_DISTDIR, so base it on that, and default RUN_GISBASE to ARCH_DISTDIR, in platform.make:

RUN_GISBASE = ${ARCH_DISTDIR}
RUN_GISRC = ${RUN_GISBASE}/demolocation/.grassrc${GRASS_VERSION_MAJOR}${GRASS_VERSION_MINOR}

Configure won't need to set anything for RUN_GISBASE then. I wouldn't need those conditionals in html.make. And just nned to add to ARCH_INC and ARCH_LIBPATH for the INST_XTN build.

Then, for modules, in the module source dir:

for INST_NOW:

INST_NOW=y make GRASS_HOME=/path/to/installed/grass MODULE_TOPDIR=/path/to/installed/grass

for INST_XTN:

INST_NOW=y make GRASS_HOME=. MODULE_TOPDIR=/path/to/installed/grass RUN_GISBASE=/path/to/installed/grass

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

Theory of the Universe

There is a theory which states that if ever anyone discovers exactly what the universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarrely inexplicable. There is another theory which states that this has already happened.

-Hitchhiker's Guide to the Galaxy 2nd season intro

William_Kyngesburye · September 23, 2007, 6:07pm

On Sep 23, 2007, at 12:30 PM, William Kyngesburye wrote:

- Also, ARCH_LIBDIR and ARCH_INCDIR are needed to find BOTH installed GRASS libs and headers AND module built libs and headers (for the rare addon module that has its own library). For the INST_NOW build, this works as is.

For the INST_XTN build, the installed GRASS lib/include would have to be added to ARCH_INC and ARCH_LIBPATH (slightly different than I currently have it).

Rats, all the GRASS dependencies refer to ARCH_LIBDIR. So leaving it based on GRASS_HOME for a INST_XTN build won't work.

I got around this in my modbuild by removing $(DEPENDENCIES) from Module.make. The best I could come up with now is (I realize conditionals may not be the best way):

ifdef INST_XTN
$(PROG): $(ARCH_CMD_OBJS)
else
$(PROG): $(ARCH_CMD_OBJS) $(DEPENDENCIES)
endif

and similar for $(ETC)/$(PGM)$(EXE):.

Also, after doing this, that INST_XTN/INST_NOW=y as env var still has problems. After the build, it displays "INST_XTN= make", then tries to look for the libgrass_display dependency. If I remove the inst_xtn target stuff and use INST_XTN as make var ("make INST_XTN=y") it exits cleanly.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

Earth: "Mostly harmless"

- revised entry in the HitchHiker's Guide to the Galaxy

Glynn_Clements1 · September 24, 2007, 1:17am

William Kyngesburye wrote:

So, I came up with these "bases" for finding/creating files:

- INST_DIR - where make install target copies files to - can be
overridden for installing to external location (I worked on this
after my initial email).

- GRASS_HOME - source topdir (full path)

Actually, GRASS_HOME is an alias for DSTDIR (with 5.3, it used to be
possible to have separate source and destination directories, but I
don't think that works at present).

Ideally, this should be used as little as possible. Ideally, just as
the base for the dist.<arch> and bin.<arch> directories.

- MODULE_TOPDIR - source topdir (usually relative path)

This has to be used for finding the Makefile fragments. Everything
else can be found via variables in Platform.make or Grass.make, but we
have to be able to find those files first.

- ARCH_DISTDIR - where targets are built INTO before installation,
default based on GRASS_HOME

and these use them (that are of interest to an external build/install):

- RUN_GISBASE -> ARCH_DISTDIR - but right now that's expanded by
configure

That should be changed; it should be set in the Makefiles from DSTDIR
(or, probably, WINDSTDIR).

- RUN_GISRC -> ARCH_DISTDIR

- GISBASE -> ARCH_DISTDIR - usually reset to RUN_GISBASE (which is
just ARCH_DISTDIR anyways), but the main makefile uses it directly
for GRASS_PDFDIR

Having a variable named GISBASE in the Makefiles is rather confusing,
IMHO. I'd rather see ARCH_DISTDIR and INST_DIR, as it makes it clear
*which* GISBASE we're dealing with.

So, as you said (it's making sense to me now), GISBASE somehow needs
its own base dir setting, defaulting to ARCH_DISTDIR, and which would
only need to be overridden by the INST_XTN build.

How about this: since GISBASE is really only for *running* GRASS, and
should only be set as needed (fix that GRASS_PDFDIR usage), don't use
it as a makefile var.

Agreed.

As for the rest of it, we're probably better off starting from basics.

The following might all be independently variable (with example
locations):

1. Where to find the GRASS source tree (/usr/local/src/grass)
2. Where to find the extension code which is being compiled (/usr/local/src/mygrassext)
3. Where to find output files (/usr/local/src/grass/<module path>/OBJ.<arch>)
4. Where to find "part-installed" files (/usr/local/src/grass/dist.<arch>)
5. Where to find the installed version of GRASS (/usr/local/grass6.3)
6. Where to put output files (/usr/local/src/grass/<module path>/OBJ.<arch>)
7. Where to put "part-installed" files (/usr/local/src/grass/dist.<arch>)
8. Where to install everything (/usr/local/grass6.3)
9. Where to find the Makefile fragments.

AFAICT, there are three main cases:

A. Compiling GRASS from scratch.
B. Compiling additional modules on a system where a compiled GRASS is already installed.
C. Compiling extensions on a system where a compiled GRASS is already installed.

For A, 2 and 5 don't apply.
For B, 1 and 2 don't apply.
For C, 1 doesn't apply.

In each case, the build directory may or may not reside within the
same directory as the source tree.

In all cases, 3 and 4 will share a common base directory, as will 6
and 7.

Also, A will use 4 where B and C use 5.

Where possible, 4 is preferable to 1, i.e. if a file is part-installed
as part of the build process, the part-installed file should be used
instead of the original (e.g. GRASS headers are obtained from
dist.<arch>/include, not from the source tree).

3 should be avoided at all costs. Grass.make[.in] has:

FMODE_OBJ = $(MODULE_TOPDIR)/lib/gis/$(OBJDIR)/fmode.o

This means that, on Windows, you cannot compile additional modules
using an installed version of GRASS, as fmode.o isn't present in the
installed version.

Any file which needs to be used by libraries or modules other than the
one which creates it should go into 7 and be referenced from 4.

9 (MODULE_TOPDIR) may be an alias for either 1 or 5, but it has to
have a fall-back hard-coded into each Makefile, and that fall-back has
to be a relative path to 1.

I'm not entirely sure whether using 9 should be preferred or avoided.

I'm leaning towards the view that it should be preferred, so that 1 is
eliminated. IOW, if you want to find something in the GRASS source
tree, use $(MODULE_TOPDIR). That way, you only have to set one
variable to build against an installed version.

Note that extensions shouldn't set MODULE_TOPDIR, as this needs to
refer to the top of the GRASS source (so that
$(MODULE_TOPDIR)/include/Make/*.make can be found), and extensions
won't know where this is.

--
Glynn Clements <glynn@gclements.plus.com>

William_Kyngesburye · October 2, 2007, 7:52pm

I sortof understand this. Until you can work out the necessary changes (if you decide to), how about some minimal changes to the the extension compilation working? Here are my revised changes to get it working:

- platform.make.in:

RUN_GISBASE = ${ARCH_DISTDIR}
RUN_GISRC = ${RUN_GISBASE}/demolocation/.grassrc${GRASS_VERSION_MAJOR}${GRASS_VERSION_MINOR}

this allows those to be overridden together in the make step, yet still keeps the same configured default for a source build.

- grass.make.in: add after "ifdef INST_NOW":

ifdef INST_XTN
ARCH_INC = -I$(ARCH_DISTDIR)/include -I$(MODULE_TOPDIR)/include
ARCH_LIBPATH = -L$(ARCH_LIBDIR) -L$(MODULE_TOPDIR)/lib
endif

to look for includes and libs in both the module source/dist and the installed GRASS

- grass.make.in: add at end:

ifdef INST_XTN
include $(MODULE_TOPDIR)/include/Make/Install_xtn.make
endif

and have a new Install_xtn.make fragment. This is just a trimmed version of the main makefile with just install targets and any unnecessary bits removed. It expects INST_DIR_XTN to be set in the make command.

- module.make: for the bin/pgm and etc/pgm targets, conditionalize the dependencies:

ifdef INST_XTN
$(BIN)/$(PGM)$(EXE): $(ARCH_CMD_OBJS)
else
$(BIN)/$(PGM)$(EXE): $(ARCH_CMD_OBJS) $(DEPENDENCIES)
endif

...

ifdef INST_XTN
$(ETC)/$(PGM)$(EXE): $(ARCH_CMD_OBJS)
else
$(ETC)/$(PGM)$(EXE): $(ARCH_CMD_OBJS) $(DEPENDENCIES)
endif

since the dependencies are based off ARCH_LIBDIR, which will be in the extension source dir, they won't be found.

Then, to compile an extension, cd to its source and:

make INST_XTN=y GRASS_HOME=. MODULE_TOPDIR=/path/to/installed/grass RUN_GISBASE=/path/to/installed/grass

Note: I still had problems with having an inst_xtn: target (similar to the inst_now target) and putting INST_XTN=y before make. No infinite loop now, but after completing the compilation (successfully) it exits with an error:

/Applications/GRASS-6.3.app/Contents/MacOS/tools/mkhtml.sh v.strahler ; mkdir -p ./dist.i686-apple-darwin8.10.1/docs/html ; /usr/bin/install -c -m 644 v.strahler.tmp.html ./dist.i686-apple-darwin8.10.1/docs/html/v.strahler.html ; for file in *.png *.jpg ; do head -n 1 $file | grep '^#!' > /dev/null ; if [ $? -ne 0 ] ; then /usr/bin/install -c -m 644 $file ./dist.i686-apple-darwin8.10.1/docs/html ; fi done 2> /dev/null ; true
INST_XTN= make
make[2]: *** No rule to make target `dist.i686-apple-darwin8.10.1/lib/libgrass_display.dylib', needed by `dist.i686-apple-darwin8.10.1/bin/v.strahler'. Stop.
make[1]: *** [inst_xtn] Error 2
make: *** [first] Error 2
rm v.strahler.tmp.html

It looks like, when done, INST_XTN gets set to something other than "Y" and it tries to do a normal compile (thus trying to compile dependencies).

Is there a particular reason for doing it this way, instead of having no extra make target and putting INST_XTN after make?

To install the extension in a user-specified location:

make INST_XTN=y GRASS_HOME=. MODULE_TOPDIR=/path/to/installed/grass RUN_GISBASE=/path/to/installed/grass INST_DIR_XTN=/path/to/user/xtn/dir install

On Sep 23, 2007, at 8:17 PM, Glynn Clements wrote:

...

As for the rest of it, we're probably better off starting from basics.

The following might all be independently variable (with example
locations):

1. Where to find the GRASS source tree (/usr/local/src/grass)
2. Where to find the extension code which is being compiled (/usr/local/src/mygrassext)
3. Where to find output files (/usr/local/src/grass/<module path>/OBJ.<arch>)
4. Where to find "part-installed" files (/usr/local/src/grass/dist.<arch>)
5. Where to find the installed version of GRASS (/usr/local/grass6.3)
6. Where to put output files (/usr/local/src/grass/<module path>/OBJ.<arch>)
7. Where to put "part-installed" files (/usr/local/src/grass/dist.<arch>)
8. Where to install everything (/usr/local/grass6.3)
9. Where to find the Makefile fragments.

AFAICT, there are three main cases:

A. Compiling GRASS from scratch.
B. Compiling additional modules on a system where a compiled GRASS is already installed.
C. Compiling extensions on a system where a compiled GRASS is already installed.

...

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"Time is an illusion - lunchtime doubly so."

- Ford Prefect

Glynn_Clements1 · October 2, 2007, 9:52pm

William Kyngesburye wrote:

I sortof understand this. Until you can work out the necessary
changes (if you decide to), how about some minimal changes to the the
extension compilation working?

Unfortunately, use of the "get it working for now" approach is
responsible for much of the current mess. I'd rather take the
opportunity to improve matters than to make them worse.

Here are my revised changes to get it
working:

- platform.make.in:

RUN_GISBASE = ${ARCH_DISTDIR}
RUN_GISRC = ${RUN_GISBASE}/demolocation/.grassrc${GRASS_VERSION_MAJOR}${GRASS_VERSION_MINOR}

this allows those to be overridden together in the make step, yet
still keeps the same configured default for a source build.

Seems okay.

Although, this should be moved into Grass.make.in, as there are no
substitutions. Platform.make.in should just be where configure dumps
its output.

On the same topic, the following substitutions in Grass.make.in should
really go into Platform.make.in:

  LIB_PREFIX=@GRASS_LIB_PREFIX@
  LIB_SUFFIX=@GRASS_LIB_SUFFIX@
  GRASS_LIBRARY_TYPE=@GRASS_LIBRARY_TYPE@

The ones related to the version can stay.

- grass.make.in: add after "ifdef INST_NOW":

ifdef INST_XTN
ARCH_INC = -I$(ARCH_DISTDIR)/include -I$(MODULE_TOPDIR)/include
ARCH_LIBPATH = -L$(ARCH_LIBDIR) -L$(MODULE_TOPDIR)/lib
endif

to look for includes and libs in both the module source/dist and the
installed GRASS

I would rather see additional variables, e.g.:

ARCH_INC = -I$(ARCH_DISTDIR)/include -I$(GISBASE)/include
ARCH_LIBPATH = -L$(ARCH_DISTDIR)/lib -L$(GISBASE)/lib

where $(ARCH_DISTDIR) is where new files are "staged" and $(GISBASE)
is where existing files are found. For normal compilation, the two
would be equal. When extending an existing installation, $(GISBASE)
would refer to the installation while $(ARCH_DISTDIR) refers to the
build location.

That would eliminate the need to conditionalise the settings.

There are quite a few Makefiles which currently use $(GISBASE) instead
of $(ARCH_DISTDIR), but that's relatively easy to fix.

- grass.make.in: add at end:

ifdef INST_XTN
include $(MODULE_TOPDIR)/include/Make/Install_xtn.make
endif

and have a new Install_xtn.make fragment. This is just a trimmed
version of the main makefile with just install targets and any
unnecessary bits removed. It expects INST_DIR_XTN to be set in the
make command.

I'm not sure that this needs to be conditionalised. If there are parts
of the top-level Makefile which might be useful to extensions, they
should be moved into one of the *.make files (or a new file).

In general, the *.make files should define as much as possible
unconditionally, and leave it up to either individual Makefiles or the
user to decide what actually gets used.

- module.make: for the bin/pgm and etc/pgm targets, conditionalize
the dependencies:

ifdef INST_XTN
$(BIN)/$(PGM)$(EXE): $(ARCH_CMD_OBJS)
else
$(BIN)/$(PGM)$(EXE): $(ARCH_CMD_OBJS) $(DEPENDENCIES)
endif

...

ifdef INST_XTN
$(ETC)/$(PGM)$(EXE): $(ARCH_CMD_OBJS)
else
$(ETC)/$(PGM)$(EXE): $(ARCH_CMD_OBJS) $(DEPENDENCIES)
endif

since the dependencies are based off ARCH_LIBDIR, which will be in
the extension source dir, they won't be found.

Actually, extensions are the easy case. Dependencies which are part of
GRASS should use $(GISBASE), which would refer to the installed
version. Dependencies which are part of the extension would use
$(ARCH_DISTDIR).

The case of extending an existing installation is more tricky, as you
could have some of the dependencies in the installation and some in
the staging directory (this assumes that you want to be able to build
additional libraries, not just modules).

But it's not that tricky; we can omit the path from the dependency and
use the "vpath" directive, which allows multiple directories to be
used for locating dependencies. IOW, GISDEP etc would lose the
"$(ARCH_LIBDIR)/" prefix, and we would add e.g.:

vpath %.$(LIB_SUFFIX) $(ARCH_LIBDIR):$(GISBASE)/lib

[Apparently, we need to use a semicolon on (native) Windows.]

In any case, the dependencies aren't critical. We don't actually build
the dependencies if they're missing. It just means that if e.g. libgis
failed to compile, we abandon trying to build modules straight away,
rather than compiling all of the object files then having the linking
stage fail.

Then, to compile an extension, cd to its source and:

make INST_XTN=y GRASS_HOME=. MODULE_TOPDIR=/path/to/installed/grass
RUN_GISBASE=/path/to/installed/grass

Note: I still had problems with having an inst_xtn: target (similar
to the inst_now target) and putting INST_XTN=y before make. No
infinite loop now, but after completing the compilation
(successfully) it exits with an error:

I wouldn't bother with that kind of hack for now. Work on separating
the variables so that it can handle mixing files from the installation
with those from the extension, and getting the extension to compile
into its staging directory.

/Applications/GRASS-6.3.app/Contents/MacOS/tools/mkhtml.sh
v.strahler ; mkdir -p ./dist.i686-apple-darwin8.10.1/docs/html ; /usr/
bin/install -c -m 644 v.strahler.tmp.html ./dist.i686-apple-
darwin8.10.1/docs/html/v.strahler.html ; for file in *.png *.jpg ;
do head -n 1 $file | grep '^#!' > /dev/null ; if [ $? -ne 0 ] ; then /
usr/bin/install -c -m 644 $file ./dist.i686-apple-darwin8.10.1/docs/
html ; fi done 2> /dev/null ; true
INST_XTN= make
make[2]: *** No rule to make target `dist.i686-apple-darwin8.10.1/lib/
libgrass_display.dylib', needed by `dist.i686-apple-darwin8.10.1/bin/
v.strahler'. Stop.
make[1]: *** [inst_xtn] Error 2
make: *** [first] Error 2
rm v.strahler.tmp.html

It looks like, when done, INST_XTN gets set to something other than
"Y" and it tries to do a normal compile (thus trying to compile
dependencies).

Is there a particular reason for doing it this way, instead of having
no extra make target and putting INST_XTN after make?

AFAICT, the INST_NOW mechanism is designed to first build everything
into the installation directory, then re-build (i.e. re-link) into the
staging directory (dist.<arch>).

Running:

INST_NOW=y make

runs make with INST_NOW=y. The first target which is found is:

  # first found target
  first: pre default
    @if test -n "$(INST_NOW)" ; then \
      $(MAKE) inst_now ; \
    fi

This runs the pre and default targets (in parallel; "pre" should
probably come first). Then, it runs "$(MAKE) inst_now", which runs:

inst_now:
INST_NOW= $(MAKE)

IOW, the first pass has INST_NOW=y, while the second has INST_NOW
empty.

If you're using an identical mechanism with INST_XTN, the second pass
is going to fail.

--
Glynn Clements <glynn@gclements.plus.com>

William_Kyngesburye · October 3, 2007, 2:05pm

On Oct 2, 2007, at 4:52 PM, Glynn Clements wrote:

William Kyngesburye wrote:

I sortof understand this. Until you can work out the necessary
changes (if you decide to), how about some minimal changes to the the
extension compilation working?

Unfortunately, use of the "get it working for now" approach is
responsible for much of the current mess. I'd rather take the
opportunity to improve matters than to make them worse.

OK. I guess I'm just being a bit impatient.

So, you're working on this now? Or will get to it? I think this can greatly simplify Benjamin's GEM, as well as allow external "extension" building.

- grass.make.in: add at end:

ifdef INST_XTN
include $(MODULE_TOPDIR)/include/Make/Install_xtn.make
endif

and have a new Install_xtn.make fragment. This is just a trimmed
version of the main makefile with just install targets and any
unnecessary bits removed. It expects INST_DIR_XTN to be set in the
make command.

I'm not sure that this needs to be conditionalised. If there are parts
of the top-level Makefile which might be useful to extensions, they
should be moved into one of the *.make files (or a new file).

In general, the *.make files should define as much as possible
unconditionally, and leave it up to either individual Makefiles or the
user to decide what actually gets used.

Just to make sure: the main makefile's install target will override that makefile frag's install target, in a normal build, since the main makefile defines it after this frag is included?

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"History is an illusion caused by the passage of time, and time is an illusion caused by the passage of history."

- Hitchhiker's Guide to the Galaxy

Glynn_Clements1 · October 3, 2007, 5:28pm

William Kyngesburye wrote:

>> I sortof understand this. Until you can work out the necessary
>> changes (if you decide to), how about some minimal changes to the the
>> extension compilation working?
>
> Unfortunately, use of the "get it working for now" approach is
> responsible for much of the current mess. I'd rather take the
> opportunity to improve matters than to make them worse.

OK. I guess I'm just being a bit impatient.

So, you're working on this now? Or will get to it?

The latter. I have "Makefile fatigue" right now, and I want to allow
people to find any bugs in the most recent bunch of changes.

>> - grass.make.in: add at end:
>>
>> ifdef INST_XTN
>> include $(MODULE_TOPDIR)/include/Make/Install_xtn.make
>> endif
>>
>> and have a new Install_xtn.make fragment. This is just a trimmed
>> version of the main makefile with just install targets and any
>> unnecessary bits removed. It expects INST_DIR_XTN to be set in the
>> make command.
>
> I'm not sure that this needs to be conditionalised. If there are parts
> of the top-level Makefile which might be useful to extensions, they
> should be moved into one of the *.make files (or a new file).
>
> In general, the *.make files should define as much as possible
> unconditionally, and leave it up to either individual Makefiles or the
> user to decide what actually gets used.

Just to make sure: the main makefile's install target will override
that makefile frag's install target, in a normal build, since the
main makefile defines it after this frag is included?

GRASS' top-level Makefile only includes Grass.make and Platform.make.
It doesn't include any of the other *.make fragments.

In general, try to avoid relying upon overriding rules which are
defined in the *.make files. Sometimes, overriding rules is
unavoidable (e.g. some modules which don't understand
--html-description have to override the htmletc target), but it's best
not to rely upon it.

Apart from printing a warning (which adds "noise" to the build
output), if the two versions have both dependencies and commands, the
dependencies get merged (sometimes in ways which make the value of $<
etc unpredictable) but the commands get overridden.

Also, whenever the generic rule is updated, you often end up having to
manually update any local versions.

[E.g. I've just extended the "clean" rule to allow a list of
subdirectories to be specified as CLEAN_SUBDIRS. Previously,
r.terraflow, r.le.setup and v.clean were overriding the clean target
to handle subdirectories, which meant that they weren't deleting the
$(PGM).tmp.html file.]

--
Glynn Clements <glynn@gclements.plus.com>