[GRASS-dev] [GRASS GIS] #909: FTBFS: r.mapcalc in trunk

GDAL · February 5, 2010, 8:48am

#909: FTBFS: r.mapcalc in trunk
-----------------------+----------------------------------------------------
Reporter: hamish | Owner: grass-dev@lists.osgeo.org
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Keywords: r.mapcalc | Platform: Linux
Cpu: x86-64 |
-----------------------+----------------------------------------------------
Hi,

latest trunk:

"make -j8" on debian/stable, 64bit quad-core
{{{
<stdout>:1555: warning: 'yyunput' defined but not used
<stdout>:1598: warning: 'input' defined but not used
gcc -L/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib
-L/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib -Wl,
--export-dynamic -Wl,-rpath-link,/usr/local/src/grass/svn/trunk/dist.
x86_64-unknown-linux-gnu/lib -o /usr/local/src/grass/svn/trunk/dist.
x86_64-unknown-linux-gnu/bin/r.mapcalc OBJ.x86_64-unknown-linux-
gnu/check.o OBJ.x86_64-unknown-linux-gnu/column_shift.o OBJ.x86_64-
unknown-linux-gnu/evaluate.o OBJ.x86_64-unknown-linux-gnu/expression.o
OBJ.x86_64-unknown-linux-gnu/function.o OBJ.x86_64-unknown-linux-
gnu/main.o OBJ.x86_64-unknown-linux-gnu/map.o OBJ.x86_64-unknown-linux-
gnu/mapcalc.tab.o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o OBJ.x86_64-
unknown-linux-gnu/xabs.o OBJ.x86_64-unknown-linux-gnu/xacos.o OBJ.x86_64-
unknown-linux-gnu/xadd.o OBJ.x86_64-unknown-linux-gnu/xand.o OBJ.x86_64-
unknown-linux-gnu/xand2.o OBJ.x86_64-unknown-linux-gnu/xasin.o OBJ.x86_64-
unknown-linux-gnu/xatan.o OBJ.x86_64-unknown-linux-gnu/xbitand.o OBJ.
x86_64-unknown-linux-gnu/xbitnot.o OBJ.x86_64-unknown-linux-gnu/xbitor.o
OBJ.x86_64-unknown-linux-gnu/xbitxor.o OBJ.x86_64-unknown-linux-
gnu/xcoor.o OBJ.x86_64-unknown-linux-gnu/xcos.o OBJ.x86_64-unknown-linux-
gnu/xdiv.o OBJ.x86_64-unknown-linux-gnu/xdouble.o OBJ.x86_64-unknown-
linux-gnu/xeq.o OBJ.x86_64-unknown-linux-gnu/xeval.o OBJ.x86_64-unknown-
linux-gnu/xexp.o OBJ.x86_64-unknown-linux-gnu/xfloat.o OBJ.x86_64-unknown-
linux-gnu/xge.o OBJ.x86_64-unknown-linux-gnu/xgraph.o OBJ.x86_64-unknown-
linux-gnu/xgt.o OBJ.x86_64-unknown-linux-gnu/xif.o OBJ.x86_64-unknown-
linux-gnu/xint.o OBJ.x86_64-unknown-linux-gnu/xisnull.o OBJ.x86_64-
unknown-linux-gnu/xle.o OBJ.x86_64-unknown-linux-gnu/xlog.o OBJ.x86_64-
unknown-linux-gnu/xlt.o OBJ.x86_64-unknown-linux-gnu/xmax.o OBJ.x86_64-
unknown-linux-gnu/xmedian.o OBJ.x86_64-unknown-linux-gnu/xmin.o OBJ.
x86_64-unknown-linux-gnu/xmod.o OBJ.x86_64-unknown-linux-gnu/xmode.o OBJ.
x86_64-unknown-linux-gnu/xmul.o OBJ.x86_64-unknown-linux-gnu/xne.o OBJ.
x86_64-unknown-linux-gnu/xneg.o OBJ.x86_64-unknown-linux-gnu/xnot.o OBJ.
x86_64-unknown-linux-gnu/xnull.o OBJ.x86_64-unknown-linux-gnu/xor.o OBJ.
x86_64-unknown-linux-gnu/xor2.o OBJ.x86_64-unknown-linux-gnu/xpow.o OBJ.
x86_64-unknown-linux-gnu/xrand.o OBJ.x86_64-unknown-linux-gnu/xres.o OBJ.
x86_64-unknown-linux-gnu/xround.o OBJ.x86_64-unknown-linux-gnu/xrowcol.o
OBJ.x86_64-unknown-linux-gnu/xshiftl.o OBJ.x86_64-unknown-linux-
gnu/xshiftr.o OBJ.x86_64-unknown-linux-gnu/xshiftru.o OBJ.x86_64-unknown-
linux-gnu/xsin.o OBJ.x86_64-unknown-linux-gnu/xsqrt.o OBJ.x86_64-unknown-
linux-gnu/xsub.o OBJ.x86_64-unknown-linux-gnu/xtan.o -lgrass_gis
-lgrass_raster -lgrass_btree -lreadline -lhistory -lpthread -lm

/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.tab.c:1442:
undefined reference to `yylex'
OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o: In function `parse_string':
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:270: undefined
reference to `initialize_scanner_string'
OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o: In function `parse_stream':
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:278: undefined
reference to `initialize_scanner_stream'
collect2: ld returned 1 exit status
make[3]: *** [/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-
gnu/bin/r.mapcalc] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory
`/usr/local/src/grass/svn/trunk/raster/r.mapcalc'
}}}

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 5, 2010, 8:53am

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Comment (by hamish):

after `cd r.mapcalc` and `make clean` I can build it successfully. so a
makefile dep problem I guess.

also, I noticed this whiz by:
{{{
mapcalc.l:216: warning, -s option given but default rule can be matched
}}}

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:1>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 5, 2010, 7:19pm

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Comment (by glynn):

Replying to [ticket:909 hamish]:

> latest trunk:
>
> "make -j8" on debian/stable, 64bit quad-core
{{{
gcc ... OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o ...
...
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.tab.c:1442:
undefined reference to `yylex'
...
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:270: undefined
reference to `initialize_scanner_string'
...
}}}
These should be in mapcalc.yy.o, compiled from mapcalc.yy.c, generated by
lex from mapcalc.l.

mapcalc.yy.o is in the list of object files being linked, so make knows
that it needs to build it and believes that it has done so, and gcc isn't
complaining about its absence. Can you provide the make output
corresponding to the generation and compilation of mapcalc.yy.c?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:2>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 5, 2010, 7:49pm

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Comment (by glynn):

Replying to [comment:1 hamish]:

> also, I noticed this whiz by:
{{{
mapcalc.l:216: warning, -s option given but default rule can be matched
}}}

Harmless. It indicates that not all possible inputs are valid, i.e. it's
possible for the expression to have syntax errors. Using the -s flag
causes syntax errors to generate an error rather than silently writing
unmatched text to stdout.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:3>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 8, 2010, 2:42am

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Comment (by hamish):

Replying to [comment:2 glynn]:
> These should be in mapcalc.yy.o, compiled from mapcalc.yy.c,
> generated by lex from mapcalc.l.
>
> mapcalc.yy.o is in the list of object files being linked, so
> make knows that it needs to build it and believes that it has done
> so, and gcc isn't complaining about its absence. Can you provide
> the make output corresponding to the generation and compilation of
> mapcalc.yy.c?

sure. due to the -j8 the output is a bit jumbled up so I'll send you the
entire build log off-list. I just tried again today and it built ok, but I
guess who wins the race is just a whim of timing.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:4>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 9, 2010, 3:21am

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Comment (by glynn):

Replying to [comment:4 hamish]:

> build log with faulty r.mapcalc build attached.

Looking at that...
{{{
3175: make[4]: Entering directory
`/home/hbowman/dev/grass/svn/trunk/raster/r.colors'
3176: make -C ../r.mapcalc
...
3178: make[5]: Entering directory
`/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc'
...
...
3569: make -C r.mapcalc || echo
/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc >>
/home/hbowman/dev/grass/svn/trunk/error.log
...
3577: make[3]: Entering directory
`/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc'
...
3589: gcc ... -o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o -c
mapcalc.yy.c
...
3796: gcc ... -o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o -c
mapcalc.yy.c
}}}

IOW, it's compiling it twice.

Note: r.colors requires r.mapcalc for the thumbnails.

It appears that the raster -> r.mapcalc build starts while
the raster -> r.colors -> r.mapcalc build is still ongoing. The first
build compiles mapcalc.yy.c to mapcalc.yy.o, then later tries to link it
while the second build is compiling it, and ends up using an incomplete
file.

Unrelated parallel make tasks don't communicate with each other (if they
did, the second task would note that the first task was already building
mapcalc.yy.o, and just wait for it to finish).

When raster/Makefile "make"s r.colors and r.mapcalc concurrently, the
tasks are considered unrelated, and that doesn't change when they bump
into each other in r.mapcalc.

One option is to simply remove r.mapcalc from the SUBDIRS list in
raster/Makefile, as r.colors is going to build it. At least, that was my
first thought, but if r.colors.html is up-to-date, it won't happen; and,
if r.mapcalc already exists, it won't be re-made.

Another option is to move the thumbnail generation into e.g. man/Makefile.
But that won't re-build the thumbnails if r.colors is re-made (OTOH, the
thumbnail images are really a derivative of lib/gis/colors/ rather than
r.colors per se).

Another option is to change raster/Makefile thus:
{{{
-htmldir: parsubdirs
+htmldir:
$(MAKE) -C r.mapcalc
$(MAKE) parsubdirs
}}}

BTW, given that there are 200 compilation and 40 linking commands
occurring in the interval in question, I'm quite surprised that you
managed to actually trigger this error.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:5>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 9, 2010, 11:33am

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Comment (by hamish):

> BTW, given that there are 200 compilation and 40 linking
> commands occurring in the interval in question, I'm quite
> surprised that you managed to actually trigger this error.

I'm not, I've always had this natural ability to cause computers to crash
in strange and interesting ways.

... maybe the colorbar thumbnails python script is relatively slow to run
versus compiling a dozen C modules? (for one thing it has to start python
which can take a little while..)
(the entire grass compile is done in less than 2 minutes on that machine
if I give it -j8)

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:6>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 24, 2014, 10:33pm

#909: FTBFS: r.mapcalc in trunk
-----------------------+----------------------------------------------------
Reporter: hamish | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Keywords: r.mapcalc | Platform: Linux
Cpu: x86-64 |
-----------------------+----------------------------------------------------

Comment(by neteler):

Untouched for 4 years, still an issue?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/909#comment:7>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 25, 2014, 1:42am

#909: FTBFS: r.mapcalc in trunk
-------------------------+--------------------------------------------------
  Reporter: hamish | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: worksforme | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
-------------------------+--------------------------------------------------
Changes (by hamish):

* status: new => closed
* resolution: => worksforme

Comment:

In the past I was able to trigger this with regularity. For now it seems
to be working on a similar machine but with an upgraded version of the
linux distro. AFAIK it was never explicitly fixed, although it may have
been incidentally fixed, so closing the ticket but will keep an eye out
for it in case it comes back.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/909#comment:8>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 26, 2014, 4:39pm

#909: FTBFS: r.mapcalc in trunk
------------------------+---------------------------------------------------
  Reporter: hamish | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: normal | Milestone: 7.0.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: r.mapcalc
  Platform: Linux | Cpu: x86-64
------------------------+---------------------------------------------------
Changes (by glynn):

* status: closed => reopened
* resolution: worksforme =>

Comment:

Replying to [comment:7 neteler]:
> Untouched for 4 years, still an issue?

I believe so.

It's caused by a race condition, so it's a matter of chance whether it
actually happens. The probability increases with the number of threads.

FWIW, the use of the "-s" flag was eliminated by r55504.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/909#comment:9>
GRASS GIS <http://grass.osgeo.org>