[GRASS-user] d.polar strange behavior when using MASK

Stefano_Costa · October 2, 2007, 5:24pm

Hi,
I was trying to use d.polar on an aspect map.

I've been able to produce EPS output w/o any problem, except when using a
raster MASK. My mask leaves out many (about 95%) of the cells in the
whole region. Without MASK it took 2 minutes for d.polar to complete.

If I run d.polar on the same map w/ this MASK, I get no results, even
though I've been waiting (patiently for more than 1 hour.

Any suggestion about this problem? I saw d.polar is a shell script, thus
possibly slow, but I'm confused because on a larger number of cells it
works well.

Best,
Stefano

Glynn_Clements1 · October 3, 2007, 1:11pm

Stefano Costa wrote:

I was trying to use d.polar on an aspect map.

I've been able to produce EPS output w/o any problem, except when using a
raster MASK. My mask leaves out many (about 95%) of the cells in the
whole region. Without MASK it took 2 minutes for d.polar to complete.

If I run d.polar on the same map w/ this MASK, I get no results, even
though I've been waiting (patiently for more than 1 hour.

Any suggestion about this problem? I saw d.polar is a shell script, thus
possibly slow, but I'm confused because on a larger number of cells it
works well.

Try using "sh -x", e.g.

sh -x `which d.polar` map=...

That should provide some clues as to where it's hanging.

--
Glynn Clements <glynn@gclements.plus.com>

Markus_Neteler1 · October 3, 2007, 1:38pm

On Wed, Oct 03, 2007 at 03:11:30PM +0200, Glynn Clements wrote:

Stefano Costa wrote:

> I was trying to use d.polar on an aspect map.
>
> I've been able to produce EPS output w/o any problem, except when using a
> raster MASK. My mask leaves out many (about 95%) of the cells in the
> whole region. Without MASK it took 2 minutes for d.polar to complete.
>
> If I run d.polar on the same map w/ this MASK, I get no results, even
> though I've been waiting (patiently for more than 1 hour.
>
> Any suggestion about this problem? I saw d.polar is a shell script, thus
> possibly slow, but I'm confused because on a larger number of cells it
> works well.

Try using "sh -x", e.g.

sh -x `which d.polar` map=...

That should provide some clues as to where it's hanging.

Due to g.parser, you have to actually edit the d.polar script
and add -x to the first line (in GRASS: edit $GISBASE/scripts/d.polar).

change:
#!/bin/sh

to:
#!/bin/sh -x

Then run the command.

Markus

------------------
ITC -> dall'1 marzo 2007 Fondazione Bruno Kessler
ITC -> since 1 March 2007 Fondazione Bruno Kessler
------------------

Stefano_Costa · October 5, 2007, 9:52am

Il giorno mer, 03/10/2007 alle 14.11 +0100, Glynn Clements ha scritto:

Stefano Costa wrote:
> Any suggestion about this problem? I saw d.polar is a shell script, thus
> possibly slow, but I'm confused because on a larger number of cells it
> works well.

Try using "sh -x", e.g.

sh -x `which d.polar` map=...

That should provide some clues as to where it's hanging.

Hi Glynn,
here's the output:

[Raster MASK present]
GRASS 6.2.1 (donetta):~ > sh -x `which d.polar` map=aspect.5m@PERMANENT
undef=0 eps=built.eps
+ '[' -z /usr/lib/grass ']'
+ '[' map=aspect.5m@PERMANENT '!=' @ARGS_PARSED@ ']'
+ exec g.parser /usr/lib/grass/scripts/d.polar map=aspect.5m@PERMANENT
undef=0 eps=built.eps
r.stats: 100%
Calculating statistics for polar diagram... (be patient)

Nothing more. The only thing I noticed, running top from another
terminal is that there's a "grep" process eating up to 100% of the CPU.

BTW, I'm running GRASS 6.2.1 on Debian Testing

Best,
Stefano

--
Stefano Costa
steko@jabber.linux.it
http://www.iosa.it Archeologia e Software Libero
Io uso Debian GNU/Linux!

Stefano_Costa · October 5, 2007, 10:21am

Due to g.parser, you have to actually edit the d.polar script and add -x
to the first line (in GRASS: edit $GISBASE/scripts/d.polar).

change:
#!/bin/sh

to:
#!/bin/sh -x

Hi Markus,
sorry I didn't notice your message before.

The complete output is:

[Raster MASK present]
GRASS 6.2.1 (donetta):~ > d.polar map=aspect.5m@PERMANENT undef=0
eps=built.eps+ '[' -z /usr/lib/grass ']'
+ '[' map=aspect.5m@PERMANENT '!=' @ARGS_PARSED@ ']'
+ exec g.parser /usr/lib/grass/scripts/d.polar map=aspect.5m@PERMANENT
undef=0 eps=built.eps
+ '[' -z /usr/lib/grass ']'
+ '[' @ARGS_PARSED@ '!=' @ARGS_PARSED@ ']'
++ basename /usr/lib/grass/scripts/d.polar
+ PROG=d.polar
++ which awk
+ '[' '!' -x /usr/bin/awk ']'
+ '[' -n built.eps ']'
+ '[' 0 -eq 1 ']'
+ '[' 0 -eq 1 ']'
+ unset LC_ALL
+ LC_NUMERIC=C
+ export LC_NUMERIC
++ g.tempfile pid=27616
+ TMP=/home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0
+ '[' 0 -ne 0 ']'
+ '[' -z /home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0 ']'
+ r.stats -1 aspect.5m@PERMANENT
r.stats: 100%
++ wc -l /home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0_raw
++ awk '{print $1}'
+ TOTALNUMBER=3354642
+ echo 'Calculating statistics for polar diagram... (be patient)'
Calculating statistics for polar diagram... (be patient)
+ cat /home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0_raw
+ grep -v '^*$'
+ grep -v '^0$'
+ awk '{printf "%d\n", int($1 + .5)}'
+ sed 's+^360+0+g'

... and it hangs at this point.

Best,
Stefano

Glynn_Clements1 · October 5, 2007, 2:26pm

Stefano Costa wrote:

> Due to g.parser, you have to actually edit the d.polar script and add -x
> to the first line (in GRASS: edit $GISBASE/scripts/d.polar).
>
> change:
> #!/bin/sh
>
> to:
> #!/bin/sh -x

Hi Markus,
sorry I didn't notice your message before.

The complete output is:

[Raster MASK present]
GRASS 6.2.1 (donetta):~ > d.polar map=aspect.5m@PERMANENT undef=0
eps=built.eps+ '[' -z /usr/lib/grass ']'
+ '[' map=aspect.5m@PERMANENT '!=' @ARGS_PARSED@ ']'
+ exec g.parser /usr/lib/grass/scripts/d.polar map=aspect.5m@PERMANENT
undef=0 eps=built.eps
+ '[' -z /usr/lib/grass ']'
+ '[' @ARGS_PARSED@ '!=' @ARGS_PARSED@ ']'
++ basename /usr/lib/grass/scripts/d.polar
+ PROG=d.polar
++ which awk
+ '[' '!' -x /usr/bin/awk ']'
+ '[' -n built.eps ']'
+ '[' 0 -eq 1 ']'
+ '[' 0 -eq 1 ']'
+ unset LC_ALL
+ LC_NUMERIC=C
+ export LC_NUMERIC
++ g.tempfile pid=27616
+ TMP=/home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0
+ '[' 0 -ne 0 ']'
+ '[' -z /home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0 ']'
+ r.stats -1 aspect.5m@PERMANENT
r.stats: 100%
++ wc -l /home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0_raw
++ awk '{print $1}'
+ TOTALNUMBER=3354642
+ echo 'Calculating statistics for polar diagram... (be patient)'
Calculating statistics for polar diagram... (be patient)
+ cat /home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0_raw
+ grep -v '^*$'
+ grep -v '^0$'
+ awk '{printf "%d\n", int($1 + .5)}'
+ sed 's+^360+0+g'

... and it hangs at this point.

Strange. How large is the 27616.0_raw file? Does anything get written
to the 27616.0_binned file? What happens if you execute the command
manually, i.e.:

TMP=/home/steko/grassdata/donetta/steko/.tmp/cycnus/27616.0
GIS_OPT_UNDEF=0
cat ${TMP}_raw | grep -v '^*$' | grep -v "^${GIS_OPT_UNDEF}$" | awk '{printf "%d\n", int($1 + .5)}' | sed 's+^360+0+g' > ${TMP}_binned

--
Glynn Clements <glynn@gclements.plus.com>

Stefano_Costa · October 8, 2007, 3:03pm

Il giorno ven, 05/10/2007 alle 15.26 +0100, Glynn Clements ha scritto:

Strange. How large is the 27616.0_raw file? Does anything get written
to the 27616.0_binned file? What happens if you execute the command
manually, i.e.:

steko@cycnus:~/grassdata/donetta/steko/.tmp/cycnus$ ls -l
totale 26340
-rw-rw-rw- 1 steko steko 0 2007-10-05 21:12 15733.0
-rw-rw-rw- 1 steko steko 0 2007-10-05 21:12 15733.1
-rw-rw-rw- 1 steko steko 0 2007-10-05 21:13 15859.0
-rw-r--r-- 1 steko steko 0 2007-10-05 21:13 15859.0_binned
-rw-r--r-- 1 steko steko 6773154 2007-10-05 21:13 15859.0_raw

and the result of the manual command is the same. However:

cat ${TMP}_raw <-- runs fine (14 s on a 6MB file)
cat ${TMP}_raw | grep -v '^*$' <-- very very slow (more than 1h
on the same file)

However, this weekend there was an update to the Debian `grep' package
and now d.polar is working.

I'm not going to file a bug if no one else has this problem.

Thanks, Stefano

--
Stefano Costa
steko@jabber.linux.it
http://www.iosa.it Archeologia e Software Libero
Io uso Debian GNU/Linux!

Glynn_Clements1 · October 8, 2007, 4:59pm

Stefano Costa wrote:

Il giorno ven, 05/10/2007 alle 15.26 +0100, Glynn Clements ha scritto:
> Strange. How large is the 27616.0_raw file? Does anything get written
> to the 27616.0_binned file? What happens if you execute the command
> manually, i.e.:

steko@cycnus:~/grassdata/donetta/steko/.tmp/cycnus$ ls -l
totale 26340
-rw-rw-rw- 1 steko steko 0 2007-10-05 21:12 15733.0
-rw-rw-rw- 1 steko steko 0 2007-10-05 21:12 15733.1
-rw-rw-rw- 1 steko steko 0 2007-10-05 21:13 15859.0
-rw-r--r-- 1 steko steko 0 2007-10-05 21:13 15859.0_binned
-rw-r--r-- 1 steko steko 6773154 2007-10-05 21:13 15859.0_raw

and the result of the manual command is the same. However:

cat ${TMP}_raw <-- runs fine (14 s on a 6MB file)
cat ${TMP}_raw | grep -v '^*$' <-- very very slow (more than 1h
on the same file)

I must be going blind; I missed that '*' the first time around. '^*$'
isn't (AFAICT) a valid regexp, as the '*' must be preceded by an
expression, and I don't think that '^' counts in this context.

I suspect that this is causing grep to get stuck in an infinite loop.
The * operator repeatedly matches the preceding expression until it
fails. But matching against ^ doesn't advance the pointer, so no
matter how many times it matches ^, it's still at the start of the
line.

If it's meant to eliminate lines which consist of nothing but an
asterisk, it should be:

grep -v '^\*$'
or:
grep -v '^[*]$'

to force the * to be treated literally.

Or maybe it's trying to match lines which consist of nothing but
spaces, but the space got "lost", in which case it should be:

grep -v '^ *$'

IIRC, some versions of grep treat operators (?, +, *, etc) as normal
characters if they aren't preceded by an expression, which would
explain why it works on some systems.

However, this weekend there was an update to the Debian `grep' package
and now d.polar is working.

I'm not going to file a bug if no one else has this problem.

It should get fixed as soon as it's determined which pattern the grep
command is supposed to match.

If it isn't fixed within a day, please file a bug report.

--
Glynn Clements <glynn@gclements.plus.com>

Stefano_Costa · October 8, 2007, 5:51pm

Il giorno lun, 08/10/2007 alle 17.59 +0100, Glynn Clements ha scritto:

I must be going blind; I missed that '*' the first time around. '^*$'
isn't (AFAICT) a valid regexp, as the '*' must be preceded by an
expression, and I don't think that '^' counts in this context.

I noticed that, but see below for some more details.

I suspect that this is causing grep to get stuck in an infinite loop.
The * operator repeatedly matches the preceding expression until it
fails. But matching against ^ doesn't advance the pointer, so no
matter how many times it matches ^, it's still at the start of the
line.

If it's meant to eliminate lines which consist of nothing but an
asterisk, it should be:

grep -v '^\*$'

I tried this and it was slow just as its `wrong' counterpart '^*$'.
In fact, I tried both the right and wrong regexps _without_ the -v flag
to see which lines where matching: in both cases ('^*$' and '^\*$') on
my system all NULL cells (represented by *) were matching the regexp,
and the grep was taking hours to complete. My MASK actually leaves about
99% of the cells out.

Given the fact that the Debian package update took away the problem, I
ended up thinking that it was grep badly behaving with the -v flag. But
I have no clues about that.

Best,
Stefano

--
Stefano Costa
steko@jabber.linux.it
http://www.iosa.it Archeologia e Software Libero
Io uso Debian GNU/Linux!

H_B · October 8, 2007, 9:38pm

Stefano wrote:

> cat ${TMP}_raw | grep -v '^*$' <-- very very slow (more than 1h
> on the same file)

Glynn wrote:

I must be going blind; I missed that '*' the first time around. '^*$'
isn't (AFAICT) a valid regexp, as the '*' must be preceded by an
expression, and I don't think that '^' counts in this context.

I suspect that this is causing grep to get stuck in an infinite loop.
The * operator repeatedly matches the preceding expression until it
fails. But matching against ^ doesn't advance the pointer, so no
matter how many times it matches ^, it's still at the start of the
line.

If it's meant to eliminate lines which consist of nothing but an
asterisk, it should be:

grep -v '^\*$'
or:
grep -v '^[*]$'

to force the * to be treated literally.

that's the one, it is getting rid of nulls from r.stats. Updated in 6.3-cvs. If
it all tests out ok I'll backport it to 6.2.x as well.

Hamish

____________________________________________________________________________________
Check out the hottest 2008 models today at Yahoo! Autos.
http://autos.yahoo.com/new_cars.html

Glynn_Clements1 · October 8, 2007, 10:26pm

Stefano Costa wrote:

> If it's meant to eliminate lines which consist of nothing but an
> asterisk, it should be:
>
> grep -v '^\*$'

I tried this and it was slow just as its `wrong' counterpart '^*$'.
In fact, I tried both the right and wrong regexps _without_ the -v flag
to see which lines where matching: in both cases ('^*$' and '^\*$') on
my system all NULL cells (represented by *) were matching the regexp,
and the grep was taking hours to complete.

Do you mean that it does actually complete, it's just very slow?

If you aren't sure whether it's actually processing data or stuck in
an infinite loop, "strace" will show whether it's actually reading the
file.

My MASK actually leaves about
99% of the cells out.

Given the fact that the Debian package update took away the problem, I
ended up thinking that it was grep badly behaving with the -v flag. But
I have no clues about that.

All in all, this does sound like a bug in specific version of grep.

Even so, the regexp should still be changed; * is an operator, and
should be escaped if it is meant to be interpreted literally.

--
Glynn Clements <glynn@gclements.plus.com>