[GRASS5] 5.7: compilation shared libs on Mac OSX failure

GRASS GRASS developer
1

Today I gave another try to the compilation of GRASS 5.7 with shared libs
on Mac OSX. All libraries compile, but compilation stops at
'driver' lib needed for XDRIVER:

cd display/drivers/lib
make
gcc -L/sw/lib -L/grass57exp/dist.powerpc-apple-darwin6.8/lib -dynamiclib
-Wl,-flat_namespace,-U,_cuserid -L/sw/lib
-L/grass57exp/dist.powerpc-apple-darwin6.8/lib \
        OBJ.powerpc-apple-darwin6.8/Box_abs.o
OBJ.powerpc-apple-darwin6.8/Box_rel.o OBJ.powerpc-apple-darwin6.8/Client.o
OBJ.powerpc-apple-darwin6.8/Clr_table.o OBJ.powerpc-apple-darwin6.8/Color.o
OBJ.powerpc-apple-darwin6.8/Cont_abs.o
OBJ.powerpc-apple-darwin6.8/Cont_rel.o OBJ.powerpc-apple-darwin6.8/Erase.o
OBJ.powerpc-apple-darwin6.8/Font_get.o
OBJ.powerpc-apple-darwin6.8/Get_t_box.o
OBJ.powerpc-apple-darwin6.8/Linemod.o OBJ.powerpc-apple-darwin6.8/Move_abs.o
OBJ.powerpc-apple-darwin6.8/Move_rel.o
OBJ.powerpc-apple-darwin6.8/Num_colors.o
OBJ.powerpc-apple-darwin6.8/Plydts_abs.o
OBJ.powerpc-apple-darwin6.8/Plydts_rel.o
OBJ.powerpc-apple-darwin6.8/Plylne_abs.o
OBJ.powerpc-apple-darwin6.8/Plylne_rel.o
OBJ.powerpc-apple-darwin6.8/Polygn_rel.o
OBJ.powerpc-apple-darwin6.8/Raster_int.o
OBJ.powerpc-apple-darwin6.8/Raster_chr.o
OBJ.powerpc-apple-darwin6.8/Raster_def.o
OBJ.powerpc-apple-darwin6.8/Reset_clrs.o
OBJ.powerpc-apple-darwin6.8/Respond.o OBJ.powerpc-apple-darwin6.8/Returns.o
OBJ.powerpc-apple-darwin6.8/Set_window.o OBJ.powerpc-apple-darwin6.8/Text.o
OBJ.powerpc-apple-darwin6.8/Text_size.o OBJ.powerpc-apple-darwin6.8/Work.o
OBJ.powerpc-apple-darwin6.8/clip.o OBJ.powerpc-apple-darwin6.8/color_supp.o
OBJ.powerpc-apple-darwin6.8/command.o
OBJ.powerpc-apple-darwin6.8/connect_fifo.o
OBJ.powerpc-apple-darwin6.8/connect_sock.o
OBJ.powerpc-apple-darwin6.8/font.o OBJ.powerpc-apple-darwin6.8/main.o
OBJ.powerpc-apple-darwin6.8/pad.o OBJ.powerpc-apple-darwin6.8/utils.o
OBJ.powerpc-apple-darwin6.8/RGB.o OBJ.powerpc-apple-darwin6.8/Text2.o
-lgrass_raster -lgrass_display -lgrass_gis -lgrass_datetime -lintl -o
/grass57exp/dist.powerpc-apple-darwin6.8/lib/libgrass_driver.dylib
ld: Undefined symbols:
_Polygon_abs
_can_do_float
_reset_color
_color
_draw_line
_Get_location_with_box2
_Get_location_with_line2
_Get_location_with_pointer2
_Graph_Close
_Panel_delete
_Panel_restore
_Panel_save
_Graph_Set

The functions are in ../XDRIVER/XDRIVER24/ which is to be compiled later.
Should these functions be moved into the driver lib (how?)?

Without XDRIVER, GRASS 5.7 isn't that funny...

Markus

2

Markus Neteler wrote:

Today I gave another try to the compilation of GRASS 5.7 with shared libs
on Mac OSX. All libraries compile, but compilation stops at
'driver' lib needed for XDRIVER:

cd display/drivers/lib
make
gcc -L/sw/lib -L/grass57exp/dist.powerpc-apple-darwin6.8/lib -dynamiclib
-Wl,-flat_namespace,-U,_cuserid -L/sw/lib
-L/grass57exp/dist.powerpc-apple-darwin6.8/lib \
        OBJ.powerpc-apple-darwin6.8/Box_abs.o

[list of object files snipped]

-lgrass_raster -lgrass_display -lgrass_gis -lgrass_datetime -lintl -o
/grass57exp/dist.powerpc-apple-darwin6.8/lib/libgrass_driver.dylib
ld: Undefined symbols:
_Polygon_abs
_can_do_float
_reset_color
_color
_draw_line
_Get_location_with_box2
_Get_location_with_line2
_Get_location_with_pointer2
_Graph_Close
_Panel_delete
_Panel_restore
_Panel_save
_Graph_Set

The functions are in ../XDRIVER/XDRIVER24/ which is to be compiled later.
Should these functions be moved into the driver lib (how?)?

No. Each driver implements its own version of these functions; you
can't move all of the different implementations into the same library.

This appears to be an issue with the nature of MacOSX shared libraries
(you would have a similar problem trying to build the driver library
as a Windows DLL).

With Linux/Solaris, shared libraries can contain unresolved symbols,
so long as those symbols are resolved whenever the library is actually
used to build an executable. It appears that MacOSX libraries don't
allow unresolved symbols; at least, not with the switches which are
being used here.

It may be that there is a linker switch to allow unresolved symbols in
shared libraries. If not, the simplest solution is to build the driver
library as a static library (I don't know how to do this with the 5.7
build system).

Actually, you may have to use a static library in any case. The driver
library defines default implementations for various functions; with a
shared library, the default implementations may be used instead of
those provided by the driver.

The only other solution would be to re-design the interface between
the driver library and the drivers, so that each driver has to
register the driver-specific functions with the driver library at
run-time (i.e. callbacks). The driver library would then call those
functions via pointers.

--
Glynn Clements <glynn.clements@virgin.net>

3

On Mon, Nov 17, 2003 at 02:42:54PM +0000, Glynn Clements wrote:

Markus Neteler wrote:

> Today I gave another try to the compilation of GRASS 5.7 with shared libs
> on Mac OSX. All libraries compile, but compilation stops at
> 'driver' lib needed for XDRIVER:
>
> cd display/drivers/lib
> make
> gcc -L/sw/lib -L/grass57exp/dist.powerpc-apple-darwin6.8/lib -dynamiclib
> -Wl,-flat_namespace,-U,_cuserid -L/sw/lib
> -L/grass57exp/dist.powerpc-apple-darwin6.8/lib \
> OBJ.powerpc-apple-darwin6.8/Box_abs.o

[list of object files snipped]

> -lgrass_raster -lgrass_display -lgrass_gis -lgrass_datetime -lintl -o
> /grass57exp/dist.powerpc-apple-darwin6.8/lib/libgrass_driver.dylib
> ld: Undefined symbols:
> _Polygon_abs
> _can_do_float
> _reset_color
> _color
> _draw_line
> _Get_location_with_box2
> _Get_location_with_line2
> _Get_location_with_pointer2
> _Graph_Close
> _Panel_delete
> _Panel_restore
> _Panel_save
> _Graph_Set
>
> The functions are in ../XDRIVER/XDRIVER24/ which is to be compiled later.
> Should these functions be moved into the driver lib (how?)?

No. Each driver implements its own version of these functions; you
can't move all of the different implementations into the same library.

This is not clear to me. Eg Graph_Set in 5.3:
src/display/devices/lib/driverlib.h
/* dummies for the driver functions */
int Graph_Close(void);
int Graph_Set(int, char **);

Practically the functions are not there but just "defined" in driverlib.h.

This appears to be an issue with the nature of MacOSX shared libraries
(you would have a similar problem trying to build the driver library
as a Windows DLL).

With Linux/Solaris, shared libraries can contain unresolved symbols,
so long as those symbols are resolved whenever the library is actually
used to build an executable. It appears that MacOSX libraries don't
allow unresolved symbols; at least, not with the switches which are
being used here.

Possibly. Any MacOSX expert here who could recommend the magic flag?

It may be that there is a linker switch to allow unresolved symbols in
shared libraries. If not, the simplest solution is to build the driver
library as a static library (I don't know how to do this with the 5.7
build system).

Globally it's the
--enable-shared

Locally it's not implemented.

Actually, you may have to use a static library in any case. The driver
library defines default implementations for various functions; with a
shared library, the default implementations may be used instead of
those provided by the driver.

OK, but the XDRIVER seems to be the only significant code piece where
shared libs with current flags don't compile. All other stuff is fixed and
compiling, that's why I did't give up already.

The only other solution would be to re-design the interface between
the driver library and the drivers, so that each driver has to
register the driver-specific functions with the driver library at
run-time (i.e. callbacks). The driver library would then call those
functions via pointers.

Sounds like quite some work.

Markus

4

On Mon, Nov 17, 2003 at 03:25:05PM +0000, Paul Kelly wrote:

Hello Markus

On Mon, 17 Nov 2003, Markus Neteler wrote:

> Today I gave another try to the compilation of GRASS 5.7 with shared libs
> on Mac OSX. All libraries compile, but compilation stops at
> 'driver' lib needed for XDRIVER:
>
> cd display/drivers/lib
> make
> gcc -L/sw/lib -L/grass57exp/dist.powerpc-apple-darwin6.8/lib -dynamiclib
> -Wl,-flat_namespace,-U,_cuserid -L/sw/lib

Have you tried the "-undefined suppress" compiler option? I did a quick
search on Google and came across references to this. The name sounds
promising.

More than promising. It simply compiles!
I also tried google, but you got the better keywords.

Only question is if it works (which I cannot test behind the firewall
here, the Mac is remote). I'll get someone to try it there.

Thanks for the hint,

Markus

5

Markus Neteler wrote:

> > The functions are in ../XDRIVER/XDRIVER24/ which is to be compiled later.
> > Should these functions be moved into the driver lib (how?)?
>
> No. Each driver implements its own version of these functions; you
> can't move all of the different implementations into the same library.

This is not clear to me. Eg Graph_Set in 5.3:
src/display/devices/lib/driverlib.h
/* dummies for the driver functions */
int Graph_Close(void);
int Graph_Set(int, char **);

Practically the functions are not there but just "defined" in driverlib.h.

Not all functions have defaults; some of them must be defined by
individual drivers. That includes Graph_Close and Graph_Set.

OTOH, many of the drawing primitives can be defined in terms of other
primitives. E.g. there is a default implementation of Box_abs() which
uses Polygon_abs(), most of the *_rel() functions have implementations
which use the corresponding *_abs() function, etc.

Ultimately, you could implement defaults for everything provided that
the driver provided a function to plot individual pixels. However,
that wouldn't be very efficient. However, the pixel-plotting function
would still be a "reverse" dependency, i.e. the library would have to
depend upon a symbol which couldn't be resolved at the point that the
library was built.

> This appears to be an issue with the nature of MacOSX shared libraries
> (you would have a similar problem trying to build the driver library
> as a Windows DLL).
>
> With Linux/Solaris, shared libraries can contain unresolved symbols,
> so long as those symbols are resolved whenever the library is actually
> used to build an executable. It appears that MacOSX libraries don't
> allow unresolved symbols; at least, not with the switches which are
> being used here.

Possibly. Any MacOSX expert here who could recommend the magic flag?

Paul seems to have answered that for you.

> It may be that there is a linker switch to allow unresolved symbols in
> shared libraries. If not, the simplest solution is to build the driver
> library as a static library (I don't know how to do this with the 5.7
> build system).

Globally it's the
--enable-shared

Locally it's not implemented.

That should be fixed. We shouldn't rely upon it being possible to
build all libraries as shared libraries and, even when it is possible,
it isn't necessarily desirable.

> Actually, you may have to use a static library in any case. The driver
> library defines default implementations for various functions; with a
> shared library, the default implementations may be used instead of
> those provided by the driver.

OK, but the XDRIVER seems to be the only significant code piece where
shared libs with current flags don't compile. All other stuff is fixed and
compiling, that's why I did't give up already.

AFAIK, the driver library is the only one remaining which has weird
dependencies. With the exception of the DBMI libraries, I fixed the
others (ISTR that coorcnv was one) when I added the option to build
shared libraries in 5.0.x (mk/mid.mk.shlib). IIRC, Radim has since
fixed the DBMI libraries (at least for 5.7).

The problem is that there are essentially cyclic dependencies between
the driver library and the drivers themselves. Each driver depends
upon functions in the driver library, but the driver library also
depends upon functions in the driver.

FWIW, if you want to build DLLs on Windows, this issue will probably
be much more problematic than it was for the Mac.

> The only other solution would be to re-design the interface between
> the driver library and the drivers, so that each driver has to
> register the driver-specific functions with the driver library at
> run-time (i.e. callbacks). The driver library would then call those
> functions via pointers.

Sounds like quite some work.

Not really; it's more work than adding another switch to the link
command, but it's mostly boilerplate.

--
Glynn Clements <glynn.clements@virgin.net>

6

On Mon, Nov 17, 2003 at 07:12:18PM +0000, Glynn Clements wrote:

Markus Neteler wrote:

> > > The functions are in ../XDRIVER/XDRIVER24/ which is to be compiled later.
> > > Should these functions be moved into the driver lib (how?)?
> >
> > No. Each driver implements its own version of these functions; you
> > can't move all of the different implementations into the same library.
>
> This is not clear to me. Eg Graph_Set in 5.3:
> src/display/devices/lib/driverlib.h
> /* dummies for the driver functions */
> int Graph_Close(void);
> int Graph_Set(int, char **);
>
> Practically the functions are not there but just "defined" in driverlib.h.

Not all functions have defaults; some of them must be defined by
individual drivers. That includes Graph_Close and Graph_Set.

OTOH, many of the drawing primitives can be defined in terms of other
primitives. E.g. there is a default implementation of Box_abs() which
uses Polygon_abs(), most of the *_rel() functions have implementations
which use the corresponding *_abs() function, etc.

Ultimately, you could implement defaults for everything provided that
the driver provided a function to plot individual pixels. However,
that wouldn't be very efficient. However, the pixel-plotting function
would still be a "reverse" dependency, i.e. the library would have to
depend upon a symbol which couldn't be resolved at the point that the
library was built.

Thanks for your explanations!

> > This appears to be an issue with the nature of MacOSX shared libraries
> > (you would have a similar problem trying to build the driver library
> > as a Windows DLL).
> >
> > With Linux/Solaris, shared libraries can contain unresolved symbols,
> > so long as those symbols are resolved whenever the library is actually
> > used to build an executable. It appears that MacOSX libraries don't
> > allow unresolved symbols; at least, not with the switches which are
> > being used here.
>
> Possibly. Any MacOSX expert here who could recommend the magic flag?

Paul seems to have answered that for you.

Yes, in fact. It compiles and installs and seems to work.
The aclocal.m4 in CVS is updated.
So GRASS 5.7 also works on Mac OSX with shared libs.

> > It may be that there is a linker switch to allow unresolved symbols in
> > shared libraries. If not, the simplest solution is to build the driver
> > library as a static library (I don't know how to do this with the 5.7
> > build system).
>
> Globally it's the
> --enable-shared
>
> Locally it's not implemented.

That should be fixed. We shouldn't rely upon it being possible to
build all libraries as shared libraries and, even when it is possible,
it isn't necessarily desirable.

OK, someone will address that when it's needed (as usual).

> > Actually, you may have to use a static library in any case. The driver
> > library defines default implementations for various functions; with a
> > shared library, the default implementations may be used instead of
> > those provided by the driver.
>
> OK, but the XDRIVER seems to be the only significant code piece where
> shared libs with current flags don't compile. All other stuff is fixed and
> compiling, that's why I did't give up already.

AFAIK, the driver library is the only one remaining which has weird
dependencies. With the exception of the DBMI libraries, I fixed the
others (ISTR that coorcnv was one) when I added the option to build
shared libraries in 5.0.x (mk/mid.mk.shlib). IIRC, Radim has since
fixed the DBMI libraries (at least for 5.7).

The problem is that there are essentially cyclic dependencies between
the driver library and the drivers themselves. Each driver depends
upon functions in the driver library, but the driver library also
depends upon functions in the driver.

FWIW, if you want to build DLLs on Windows, this issue will probably
be much more problematic than it was for the Mac.

> > The only other solution would be to re-design the interface between
> > the driver library and the drivers, so that each driver has to
> > register the driver-specific functions with the driver library at
> > run-time (i.e. callbacks). The driver library would then call those
> > functions via pointers.
>
> Sounds like quite some work.

Not really; it's more work than adding another switch to the link
command, but it's mostly boilerplate.

--
Glynn Clements <glynn.clements@virgin.net>

Markus

7

Just for the record:

While the compilation runs ok now, I faced another
problem with GRASS 5.7/shared libs on MacOSX when using it:

dyld: /usr/local/grass57-17_11_2003/etc/lock can't open library:
/grass57exp/dist.powerpc-apple-darwin6.8/lib/libgrass_gis.dylib (No such file or directory, errno = 2)
/usr/local/grass57-17_11_2003/etc/Init.sh: line 147: 784 Trace/BPT
trap "$ETC/lock" "$lockfile" $$
Unable to properly access /Users/fabrizio/.gislock57
Please notify system personel.

Programs (all?) such as 'etc/lock' have the path to the dynamic
library hardcoded. After installation into a different place
(e.g. /usr/local/something) it still searches the old
path used at compile time.

Solution:

On MacOSX one has to set DYLD_LIBRARY_PATH (instead of LD_LIBRARY_PATH).
After patching /etc/Init.sh GRASS 5.7 with shared libs on MacOSX works now.
The relevant file in CVS is updated (needs still a trivial fix for
"powermac").

Working configuration:

./configure --with-grass50=/grass53exp \
--enable-shared \
--with-includes="/usr/X11R6/include /sw/include" --with-libs=/sw/lib \
--with-tiff-includes=/sw/include --with-tiff-libs=/sw/lib \
--with-tcltk-includes=/sw/include --with-tcltk-libs=/sw/lib/tcl8.4 \
--with-gd-includes=/sw/include \
--with-opengl-libs=/usr/X11R6/lib --with-opengl-includes=/usr/X11R6/include \
--with-fftw-includes=/sw/include \
--with-gdal=/sw/bin/gdal-config \
--without-odbc \
--without-postgres

Cheers

Markus Neteler

8

Markus Neteler wrote:

Just for the record:

While the compilation runs ok now, I faced another
problem with GRASS 5.7/shared libs on MacOSX when using it:

dyld: /usr/local/grass57-17_11_2003/etc/lock can't open library:
/grass57exp/dist.powerpc-apple-darwin6.8/lib/libgrass_gis.dylib (No such file or directory, errno = 2)
/usr/local/grass57-17_11_2003/etc/Init.sh: line 147: 784 Trace/BPT
trap "$ETC/lock" "$lockfile" $$
Unable to properly access /Users/fabrizio/.gislock57
Please notify system personel.

Programs (all?) such as 'etc/lock' have the path to the dynamic
library hardcoded. After installation into a different place
(e.g. /usr/local/something) it still searches the old
path used at compile time.

Solution:

On MacOSX one has to set DYLD_LIBRARY_PATH (instead of LD_LIBRARY_PATH).
After patching /etc/Init.sh GRASS 5.7 with shared libs on MacOSX works now.
The relevant file in CVS is updated (needs still a trivial fix for
"powermac").

Note that SC_CONFIG_CFLAGS in aclocal.m4 sets the variable
LD_LIBRARY_PATH_VAR to the name of the appropriate variable, i.e.:

  Rhapsody-*|Darwin-*)
    ...
      LD_LIBRARY_PATH_VAR="DYLD_LIBRARY_PATH"

Maybe that can be used when creating Init.sh from init.sh.

One thing which stood out when examining SC_CONFIG_CFLAGS was this:

    CC_SEARCH_FLAGS='-Wl,-rpath,${LIB_RUNTIME_DIR}'
    LD_SEARCH_FLAGS=${CC_SEARCH_FLAGS}

I don't know how (or even if) this actually gets used, but it may
cause problems with moving shared libraries (-rpath embeds the library
path into the executable).

Reminder: there is a problem (at least with 5.0/5.3 on Linux) with
shared libraries and tcltkgrass. If tcltkgrass spawns an xterm to run
a particular command, LD_LIBRARY_PATH is reset (as a security measure,
because xterm is setuid root). The GRASS command which is being run
then fails because it can't find its shared libraries.

--
Glynn Clements <glynn.clements@virgin.net>

9

Hy folks,
I got a real strange tk.h problem I never had before.

I run
./configure --prefix=/home/moovida/grass/the_grass/ --bindir=/home/moovida/
grass/bin/ --with-postgres-includes="/usr/include/pgsql/ /usr/include/pgsql/
server/" --with-motif --with-motif-includes=/usr/X11R6/include/ --with-gdal=/
usr/local/bin/gdal-config

and at the point of tcl/tk the following happens:

checking for location of Tcl/Tk includes...
checking for tcl.h... yes
checking for tk.h... no

and it stops.

Considering that tk.h is in /usr/include/ just like tcl.h is, I'm a bit lost
in space. It is just a check if the file exists, so I controlled permissions,
owner and whatever regarding to file integrity.

Did anyone ever encounter this problem?

Thanks, ciao
Andrea

--
____________________________________________________________________________

University of Trento
Department of Civil and Environmental Engineering
Via Mesiano, 77 - Trento (ITALY)

Andrea Antonello
tel: +393288497722
fax: +390461882672
____________________________________________________________________________

10

moovida_lists wrote:

I got a real strange tk.h problem I never had before.

I run
./configure --prefix=/home/moovida/grass/the_grass/ --bindir=/home/moovida/
grass/bin/ --with-postgres-includes="/usr/include/pgsql/ /usr/include/pgsql/
server/" --with-motif --with-motif-includes=/usr/X11R6/include/ --with-gdal=/
usr/local/bin/gdal-config

and at the point of tcl/tk the following happens:

checking for location of Tcl/Tk includes...
checking for tcl.h... yes
checking for tk.h... no

and it stops.

Considering that tk.h is in /usr/include/ just like tcl.h is, I'm a bit lost
in space. It is just a check if the file exists, so I controlled permissions,
owner and whatever regarding to file integrity.

Did anyone ever encounter this problem?

Header checks don't simply check that the file exists; they attempt to
preprocess a test program which consists of a #include statement for
the header in question (in this case "#include <tk.h>"); if the
preprocessor produces any error/warning messages, the check fails.

tk.h depends upon the X headers, as well as tcl.h. So if there are any
problems with the X headers, the check will fail. Note that "problems"
includes anything that results in a warning, not just errors.

Examine config.log for any clues.

--
Glynn Clements <glynn.clements@virgin.net>

11

You are absolutely right.
The config.log was of help.

Thanks & cheers
Andrea

On Thursday 20 November 2003 12:40, Glynn Clements wrote:

moovida_lists wrote:
> I got a real strange tk.h problem I never had before.
>
> I run
> ./configure --prefix=/home/moovida/grass/the_grass/
> --bindir=/home/moovida/ grass/bin/
> --with-postgres-includes="/usr/include/pgsql/ /usr/include/pgsql/
> server/" --with-motif --with-motif-includes=/usr/X11R6/include/
> --with-gdal=/ usr/local/bin/gdal-config
>
> and at the point of tcl/tk the following happens:
>
> checking for location of Tcl/Tk includes...
> checking for tcl.h... yes
> checking for tk.h... no
>
> and it stops.
>
> Considering that tk.h is in /usr/include/ just like tcl.h is, I'm a bit
> lost in space. It is just a check if the file exists, so I controlled
> permissions, owner and whatever regarding to file integrity.
>
> Did anyone ever encounter this problem?

Header checks don't simply check that the file exists; they attempt to
preprocess a test program which consists of a #include statement for
the header in question (in this case "#include <tk.h>"); if the
preprocessor produces any error/warning messages, the check fails.

tk.h depends upon the X headers, as well as tcl.h. So if there are any
problems with the X headers, the check will fail. Note that "problems"
includes anything that results in a warning, not just errors.

Examine config.log for any clues.

--
____________________________________________________________________________

University of Trento
Department of Civil and Environmental Engineering
Via Mesiano, 77 - Trento (ITALY)

Andrea Antonello
tel: +393288497722
fax: +390461882672
____________________________________________________________________________