[GRASS-dev] [GRASS GIS] #718: r.li forgets mask/illegal filename

#718: r.li forgets mask/illegal filename
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 6.4.0
Component: Raster | Version: 6.4.0 RCs
Resolution: | Keywords: r.li, smp
  Platform: All | Cpu: All
------------------------+---------------------------------------------------
Comment (by hamish):

I added some code to strip away the "$HOME/r.li/history/" part of the
config file name, if present.

I notice then if you give the full path to the config file on the command
line it doesn't crash with the glibc error. I suppose giving the long path
name slightly increases the memory allotment allowing the program to just
trash its own memory space and not that of its neighbors? shrug

H

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:20&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 6.4.0
Component: Raster | Version: 6.4.0 RCs
Resolution: | Keywords: r.li, smp
  Platform: All | Cpu: All
------------------------+---------------------------------------------------
Comment (by glynn):

Replying to [comment:19 hamish]:

> I still can't find any joy with gdb so resorting to sticking printf()s
everywhere. With that I can see that it is falling over in
r.li.daemon/list.c's insertNode() on the first malloc() on the second time
that function is called.
> source:grass/trunk/raster/r.li/r.li.daemon/list.c@38777#L32

> unfortunately I'm guessing that just means that the memory is already
corrupt by the time it gets there. :frowning:

That's what it looks like.

Oh, look at insertNode() in list.c:

{{{
void insertNode(list l, msg mess)
{
     node new;

     new = malloc(sizeof(node));
     new->m = malloc(sizeof(msg));
}}}

Even without reading what follows, the above should smell fishy. Don't
malloc() calls usually look like "TYPE *x = malloc(N * sizeof(TYPE))"?
I.e. shouldn't the type of the return value be a pointer to whatever type
appears in the sizeof?

Note: list.h says:

{{{
typedef struct nodoLista *node;
}}}

So "node" is a typedef for a pointer to the structure, so the malloc() is
only allocating enough memory for the pointer, not the structure.

This is just one reason why sane programmers very rarely use typedefs, and
almost never for pointer types.

[They may sometimes be used to hide implementation details in public
interfaces, e.g. "typedef void *HANDLE", but the actual type would
normally be used in the implementation.]

For the sake of anyone who has to clear up any other mistakes in r.li
(given the overall quality, I doubt that this is the only one), I would
ask whoever fixes this to finish the job and eliminate the typedefs
altogether. If something is a pointer to a structure, it should look like
"struct foo *p", not "foo p".

Oh, and bonus points for changing the type names to English, like the
other 99.99% of the GRASS codebase.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:21&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 6.4.0
Component: Raster | Version: 6.4.0 RCs
Resolution: | Keywords: r.li, smp
  Platform: All | Cpu: All
------------------------+---------------------------------------------------
Comment (by neteler):

I guess I found it finally. Try r41560 (trunk), r415599 (6.5), r415598
(6.4).

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/718#comment:22&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.0
Component: Raster | Version: 6.4.0 RCs
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------

Comment(by neteler):

Is this still an issue on MacOSX 64bit? Can it be closed?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/718#comment:23&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.0
Component: Raster | Version: 6.4.0 RCs
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------

Comment(by kyngchaos):

I had trouble remembering what I did for the r.li.setup part. I finally
downloaded the NC data to try hamish's conf's.

The first one works fine, but hamish didn't have a problem with that
either.

The next one gives an error (all jumbled from the multiple workers) and
hangs (must ctl-c to exit):

{{{

> r.li.shannon map=landclass96 conf=test_conf2 out=test_shannon2
ERROR: ERROR: ProgramERROR: error,Program worker()Program
toReceive.type=0 error,
error, worker()worker() toReceive.type=24184648toReceive.type=0
}}}

In 32bit mode, it finishes with no error, but there is also no output.

Patch density using the first conf seems to work in both 64 and 32bit
modes, though there are bands of what looks like noise, in different
places between 32 and 64bit. But I get the same error/hang on 64bit and
no output on 32bit using the other conf.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:24&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.3
Component: Raster | Version: 6.4.2 RCs
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------
Changes (by lucadelu):

  * priority: normal => major
  * version: 6.4.0 RCs => 6.4.2 RCs
  * milestone: 6.4.0 => 6.4.3

Comment:

Replying to [comment:24 kyngchaos]:
> I had trouble remembering what I did for the r.li.setup part. I finally
downloaded the NC data to try hamish's conf's.
>
> The first one works fine, but hamish didn't have a problem with that
either.
>
> The next one gives an error (all jumbled from the multiple workers) and
hangs (must ctl-c to exit):
>
> {{{
>
> > r.li.shannon map=landclass96 conf=test_conf2 out=test_shannon2
> ERROR: ERROR: ProgramERROR: error,Program worker()Program
toReceive.type=0 error,
> error, worker()worker() toReceive.type=24184648toReceive.type=0
> }}}
>

I have the same problem with 6.4.2 to test it I used nc, with the attached
landsat_test file, copy it into ~/.r.li/history and after launch

{{{
r.li.shannon map=lsat7_2002_40 conf=landsat_test out=landsat_shan
}}}

{{{
Segmentation fault
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
ERROR: Program error, worker() toReceive.type=0
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:25&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.3
Component: Raster | Version: 6.4.2 RCs
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------

Comment(by mmetz):

At least the segfaults should be fixed in all branches as of r49886.

Markus M

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:26&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.3
Component: Raster | Version: 6.4.2 RCs
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------

Comment(by neteler):

Thanks for the substantial changes.

On Fedora 16/64bit, I face the strange problem what the workers do not
fork:
{{{
GRASS 6.4.2svn (nc_spm_08):~/grass64 > g.region rast=lsat7_2002_40 -p
projection: 99 (Lambert Conformal Conic)
zone: 0
datum: nad83
ellipsoid: a=6378137 es=0.006694380022900787
north: 228513
south: 214975.5
west: 629992.5
east: 645012
nsres: 28.5
ewres: 28.5
rows: 475
cols: 527
cells: 250325
GRASS 6.4.2svn (nc_spm_08):~/grass64 > time -p r.li.shannon
map=lsat7_2002_40 conf=landsat_test out=landsat_shan
real 0.08
user 0.06
sys 0.01
}}}

In GRASS 7 the same:
{{{
GRASS 7.0.svn (nc_spm_08):~/grass70 > time -p r.li.shannon
in=lsat7_2002_40 conf=landsat_test out=landsat_shan
real 0.04
user 0.00
sys 0.00

GRASS 7.0.svn (nc_spm_08):~/grass70 > uname -a
Linux north.localdomain 3.1.5-6.fc16.x86_64 #1 SMP Thu Dec 15 16:14:44 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux
}}}

I tried with "ddd" and found in daemon.c, line 85, "parsed" is '\001',
hence the
rest fails.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:27&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.3
Component: Raster | Version: 6.4.2 RCs
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------

Comment(by mmetz):

Replying to [comment:27 neteler]:
>
> On Fedora 16/64bit, I face the strange problem what the workers do not
fork:
[snip]
>
{{{
GRASS 7.0.svn (nc_spm_08):~/grass70 > time -p r.li.shannon
in=lsat7_2002_40 conf=landsat_test out=landsat_shan
real 0.04
user 0.00
sys 0.00

GRASS 7.0.svn (nc_spm_08):~/grass70 > uname -a
Linux north.localdomain 3.1.5-6.fc16.x86_64 #1 SMP Thu Dec 15 16:14:44 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux
}}}
>
> I tried with "ddd" and found in daemon.c, line 85, "parsed" is '\001',
hence the
> rest fails.

parsed is 1 and must be interpreted as a number, not as a char.

Can not reproduce, for both 6.4 and 7 (r49887), a result is produced. The
content of the file landsat_shan is
{{{
RESULT 1|4.390990
}}}

Note the different locations of the output file for 6 and 7.

Markus M

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:28&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.4
Component: Raster | Version: svn-releasebranch64
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------
Changes (by neteler):

  * version: 6.4.2 RCs => svn-releasebranch64
  * milestone: 6.4.3 => 6.4.4

Comment:

I retested it with current 6.4.svn:

{{{
g.region rast=lsat7_2002_40 -p

mkdir -p $HOME/.r.li/history/
echo "SAMPLINGFRAME 0|0|1|1
SAMPLEAREA 0.0|0.0|1.0|1.0" > $HOME/.r.li/history/landsat_test

r.li.shannon map=lsat7_2002_40 conf=landsat_test out=landsat_shannon
r.univar landsat_shannon
ERROR: Raster map <landsat_shannon> not found
}}}

... no output is produced by r.li.shannon.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:29&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
-----------------------+----------------------------------------------------
Reporter: kyngchaos | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.4
Component: Raster | Version: svn-releasebranch64
Keywords: r.li, smp | Platform: All
      Cpu: All |
-----------------------+----------------------------------------------------

Comment(by mmetz):

Replying to [comment:29 neteler]:
> I retested it with current 6.4.svn:
>
{{{
g.region rast=lsat7_2002_40 -p

mkdir -p $HOME/.r.li/history/
echo "SAMPLINGFRAME 0|0|1|1
SAMPLEAREA 0.0|0.0|1.0|1.0" > $HOME/.r.li/history/landsat_test

r.li.shannon map=lsat7_2002_40 conf=landsat_test out=landsat_shannon
r.univar landsat_shannon
ERROR: Raster map <landsat_shannon> not found
}}}
>
> ... no output is produced by r.li.shannon.

With this configuration, the output is a text file. The r.li daemon should
have printed a corresponding message with the path to the output file.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/718#comment:30&gt;
GRASS GIS <http://grass.osgeo.org>

#718: r.li forgets mask/illegal filename
------------------------+---------------------------------------------------
  Reporter: kyngchaos | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: major | Milestone: 6.4.4
Component: Raster | Version: svn-releasebranch64
Resolution: fixed | Keywords: r.li, smp
  Platform: All | Cpu: All
------------------------+---------------------------------------------------
Changes (by neteler):

  * status: new => closed
  * resolution: => fixed

Comment:

Replying to [comment:30 mmetz]:
> Replying to [comment:29 neteler]:
> > I retested it with current 6.4.svn:
> >
> {{{
> g.region rast=lsat7_2002_40 -p
>
> mkdir -p $HOME/.r.li/history/
> echo "SAMPLINGFRAME 0|0|1|1
> SAMPLEAREA 0.0|0.0|1.0|1.0" > $HOME/.r.li/history/landsat_test
>
> r.li.shannon map=lsat7_2002_40 conf=landsat_test out=landsat_shannon
> r.univar landsat_shannon
> ERROR: Raster map <landsat_shannon> not found
> }}}
> >
> > ... no output is produced by r.li.shannon.
>
> With this configuration, the output is a text file. The
> r.li daemon should have printed a corresponding message with
> the path to the output file.

It now does in the backported, corrected version:

{{{
r.li.shannon map=lsat7_2002_40 conf=landsat_test out=landsat_shannon
Result written to ASCII file </home/neteler/.r.li/output/landsat_shannon>
}}}

The original issue reported here should also be fixed in the new version.

Closing, feel free top reopen if needed.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/718#comment:31&gt;
GRASS GIS <http://grass.osgeo.org>