[GRASS-dev] r.random: now with optional cover map parameter

Markus_Neteler1 · October 16, 2007, 12:53pm

Hi,

since v.what.rast is (for me) extremely slow, I have added
cover map support to r.random. You can now do random sampling
(e.g., elevation) and sample at the same time for the random
locations values from a second raster map (e.g., landuse).
The result is only stored when using vector points output.

I use it with -d flag, to sample elevation as 3D vector map
with MODIS Land Surface Temperatures as second data column
in the resulting vector map.

In 6.3-CVS.

Markus

PS: v.what.rast still running on just 300k points while I
implemented above Anyone who could make v.what.rast faster?

------------------
ITC -> dall'1 marzo 2007 Fondazione Bruno Kessler
ITC -> since 1 March 2007 Fondazione Bruno Kessler
------------------

Dylan_Beaudette1 · October 16, 2007, 4:26pm

On Tuesday 16 October 2007, Markus Neteler wrote:

Hi,

since v.what.rast is (for me) extremely slow, I have added
cover map support to r.random. You can now do random sampling
(e.g., elevation) and sample at the same time for the random
locations values from a second raster map (e.g., landuse).
The result is only stored when using vector points output.

I use it with -d flag, to sample elevation as 3D vector map
with MODIS Land Surface Temperatures as second data column
in the resulting vector map.

In 6.3-CVS.

Markus

PS: v.what.rast still running on just 300k points while I
implemented above Anyone who could make v.what.rast faster?

Hmmm.. not sure, but Starspan always works for me, and is very fast!

Dylan

--
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341

H_B · October 17, 2007, 10:15am

Markus Neteler wrote:

since v.what.rast is (for me) extremely slow, I have added
cover map support to r.random.

..

PS: v.what.rast still running on just 300k points while I
implemented above Anyone who could make v.what.rast faster?

Maybe it's the qsort(), maybe it's the i,j loop within a loop.

# spearfish example of the slowness
g.region -d
v.random n=50000 out=rand50k
v.db.addtable rand50k column="elev INT"
v.what.rast rand50k rast=elevation.dem column=elev

Could someone in the know add some profiling how-to tips on the Wiki development page?
http://grass.gdf-hannover.de/wiki/GRASS_Debugging#Using_a_profiling_tool

see also this thread WRT v.rast.stats slowness,
http://thread.gmane.org/gmane.comp.gis.grass.devel/22730

Hamish

Glynn_Clements1 · October 17, 2007, 7:50pm

Hamish wrote:

Markus Neteler wrote:
> since v.what.rast is (for me) extremely slow, I have added
> cover map support to r.random.
..
> PS: v.what.rast still running on just 300k points while I
> implemented above Anyone who could make v.what.rast faster?

Maybe it's the qsort(), maybe it's the i,j loop within a loop.

The loop is certainly inefficient. Rather than shifting the entire
array down every time it finds a duplicate, it should keep separate
source and destination indices, e.g. (untested):

    for (i = j = 0; j < point_cnt; j++)
        if (cache[i].cat != cache[j].cat)
      cache[++i] = cache[j];
        else
            cache[i].count++;
    point_cnt = i + 1;

--
Glynn Clements <glynn@gclements.plus.com>

Markus_Neteler4 · October 21, 2007, 4:23pm

Glynn Clements wrote:

Hamish wrote:

Markus Neteler wrote:
> since v.what.rast is (for me) extremely slow, I have added
> cover map support to r.random.
..
> PS: v.what.rast still running on just 300k points while I
> implemented above Anyone who could make v.what.rast faster?

Maybe it's the qsort(), maybe it's the i,j loop within a loop.

The loop is certainly inefficient. Rather than shifting the entire
array down every time it finds a duplicate, it should keep separate
source and destination indices, e.g. (untested):

    for (i = j = 0; j < point_cnt; j++)
        if (cache[i].cat != cache[j].cat)
      cache[++i] = cache[j];
        else
            cache[i].count++;
    point_cnt = i + 1;

--
Glynn Clements <glynn@gclements.plus.com>

I am afraid to say that I don't know how to implement the suggestion
in v.what.rast...

Markus
--
View this message in context: http://www.nabble.com/r.random%3A-now-with-optional-cover-map-parameter-tf4634057.html#a13330648
Sent from the Grass - Dev mailing list archive at Nabble.com.

Glynn_Clements1 · October 21, 2007, 5:53pm

Markus Neteler wrote:

>> > since v.what.rast is (for me) extremely slow, I have added
>> > cover map support to r.random.
>> ..
>> > PS: v.what.rast still running on just 300k points while I
>> > implemented above Anyone who could make v.what.rast faster?
>>
>> Maybe it's the qsort(), maybe it's the i,j loop within a loop.
>
> The loop is certainly inefficient. Rather than shifting the entire
> array down every time it finds a duplicate, it should keep separate
> source and destination indices, e.g. (untested):
>
> for (i = j = 0; j < point_cnt; j++)
> if (cache[i].cat != cache[j].cat)
> cache[++i] = cache[j];
> else
> cache[i].count++;
> point_cnt = i + 1;

I am afraid to say that I don't know how to implement the suggestion
in v.what.rast...

--- vector/v.what.rast/main.c 17 Oct 2007 14:07:23 -0000 1.26
+++ vector/v.what.rast/main.c 21 Oct 2007 17:52:42 -0000
@@ -218,21 +218,14 @@
qsort (cache, point_cnt, sizeof (struct order), by_cat);

G_debug(1, "Points are sorted, starting duplicate removal loop");
- i = 1;
- while ( i < point_cnt ) {
- if ( cache[i].cat == cache[i-1].cat ) {
- cache[i-1].count++;
- for ( j = i; j < point_cnt - 1; j++ ) {
- cache[j].row = cache[j+1].row;
- cache[j].col = cache[j+1].col;
- cache[j].cat = cache[j+1].cat;
- cache[j].count = cache[j+1].count;
- }
- point_cnt--;
- continue;
- }
- i++;
- }
+
+ for (i = j = 0; j < point_cnt; j++)
+ if (cache[i].cat != cache[j].cat)
+ cache[++i] = cache[j];
+ else
+ cache[i].count++;
+ point_cnt = i + 1;
+
G_debug(1, "%d vector points left after removal of duplicates", point_cnt);

/* Report number of points not used */

--
Glynn Clements <glynn@gclements.plus.com>

Markus_Neteler4 · October 21, 2007, 10:04pm

Glynn Clements wrote:

Markus Neteler wrote:

>> > since v.what.rast is (for me) extremely slow, I have added
>> > cover map support to r.random.
>> ..
>> > PS: v.what.rast still running on just 300k points while I
>> > implemented above Anyone who could make v.what.rast faster?
>>
>> Maybe it's the qsort(), maybe it's the i,j loop within a loop.
>
> The loop is certainly inefficient. Rather than shifting the entire
> array down every time it finds a duplicate, it should keep separate
> source and destination indices, e.g. (untested):
>
> for (i = j = 0; j < point_cnt; j++)
> if (cache[i].cat != cache[j].cat)
> cache[++i] = cache[j];
> else
> cache[i].count++;
> point_cnt = i + 1;

I am afraid to say that I don't know how to implement the suggestion
in v.what.rast...

--- vector/v.what.rast/main.c 17 Oct 2007 14:07:23 -0000 1.26
+++ vector/v.what.rast/main.c 21 Oct 2007 17:52:42 -0000
... patch

Thanks (for your patience): I have made a Spearfish test:

g.region rast=elevation.10m
r.random elevation.10m vect=elevpnts n=5000 --o
v.db.addcol elevpnts col="height double precision"

# old
time v.what.rast elevpnts rast=elevation.10m col=height
real 0m25.253s
user 0m24.756s
sys 0m0.308s

# new
real 0m24.040s
user 0m23.707s
sys 0m0.297s

Interestingly, timing rather identical (also for 1000). Using the DBF
driver here.

Markus
--
View this message in context: http://www.nabble.com/r.random%3A-now-with-optional-cover-map-parameter-tf4634057.html#a13333907
Sent from the Grass - Dev mailing list archive at Nabble.com.

Glynn_Clements1 · October 21, 2007, 10:40pm

Markus Neteler wrote:

>> >> > since v.what.rast is (for me) extremely slow, I have added
>> >> > cover map support to r.random.
>> >> ..
>> >> > PS: v.what.rast still running on just 300k points while I
>> >> > implemented above Anyone who could make v.what.rast faster?
>> >>
>> >> Maybe it's the qsort(), maybe it's the i,j loop within a loop.
>> >
>> > The loop is certainly inefficient. Rather than shifting the entire
>> > array down every time it finds a duplicate, it should keep separate
>> > source and destination indices, e.g. (untested):
>> >
>> > for (i = j = 0; j < point_cnt; j++)
>> > if (cache[i].cat != cache[j].cat)
>> > cache[++i] = cache[j];
>> > else
>> > cache[i].count++;
>> > point_cnt = i + 1;
>
>> I am afraid to say that I don't know how to implement the suggestion
>> in v.what.rast...
>
> --- vector/v.what.rast/main.c 17 Oct 2007 14:07:23 -0000 1.26
> +++ vector/v.what.rast/main.c 21 Oct 2007 17:52:42 -0000
> ... patch
>

Thanks (for your patience): I have made a Spearfish test:

g.region rast=elevation.10m
r.random elevation.10m vect=elevpnts n=5000 --o
v.db.addcol elevpnts col="height double precision"

# old
time v.what.rast elevpnts rast=elevation.10m col=height
real 0m25.253s
user 0m24.756s
sys 0m0.308s

# new
real 0m24.040s
user 0m23.707s
sys 0m0.297s

Interestingly, timing rather identical (also for 1000). Using the DBF
driver here.

It was subsequently suggested that it may be the database access which
is the main performance issue. Personally, I suspect that is probably
the case.

You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
to see if that helps at all.

--
Glynn Clements <glynn@gclements.plus.com>

H_B · October 21, 2007, 10:53pm

>> >> > since v.what.rast is (for me) extremely slow, I have added
>> >> > cover map support to r.random.
>> >> ..
>> >> > PS: v.what.rast still running on just 300k points while I
>> >> > implemented above Anyone who could make v.what.rast faster?
>> >>
>> >> Maybe it's the qsort(), maybe it's the i,j loop within a loop.
>> >
>> > The loop is certainly inefficient. Rather than shifting the entire
>> > array down every time it finds a duplicate, it should keep separate
>> > source and destination indices, e.g. (untested):
>> >
>> > for (i = j = 0; j < point_cnt; j++)

Glynn:

> --- vector/v.what.rast/main.c 17 Oct 2007 14:07:23 -0000 1.26
> +++ vector/v.what.rast/main.c 21 Oct 2007 17:52:42 -0000
> ... patch

Markus:

Thanks (for your patience): I have made a Spearfish test:

g.region rast=elevation.10m
r.random elevation.10m vect=elevpnts n=5000 --o
v.db.addcol elevpnts col="height double precision"

# old
time v.what.rast elevpnts rast=elevation.10m col=height
real 0m25.253s
user 0m24.756s
sys 0m0.308s

# new
real 0m24.040s
user 0m23.707s
sys 0m0.297s

Interestingly, timing rather identical (also for 1000). Using the DBF
driver here.

It is as expected, only a small speed up. While the for i,j loop map may have
been inefficient, it still was fast enough to only take a second to get though.
The bulk of the time was and is being taken up by running db_execute_imediate()
for each point later on in the script. (then db_commit_transaction() is run
after the loop)

To solve this in v.in.garmin and v.in.gpsbabel we wrote all db SET .. to ..
statements to a temporary file, then ran db.execute once for the tmp file
instead of earlier running db.execute for every point. It is not a direct
analogy, but it did made a huge difference there.

Or maybe it is the bsearch() for every point in the db_execute_imediate() loop?

Hamish

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Glynn_Clements1 · October 21, 2007, 11:58pm

Hamish wrote:

> >> >> > since v.what.rast is (for me) extremely slow, I have added
> >> >> > cover map support to r.random.
> >> >> ..
> >> >> > PS: v.what.rast still running on just 300k points while I
> >> >> > implemented above Anyone who could make v.what.rast faster?
> >> >>
> >> >> Maybe it's the qsort(), maybe it's the i,j loop within a loop.
> >> >
> >> > The loop is certainly inefficient. Rather than shifting the entire
> >> > array down every time it finds a duplicate, it should keep separate
> >> > source and destination indices, e.g. (untested):
> >> >
> >> > for (i = j = 0; j < point_cnt; j++)
Glynn:
> > --- vector/v.what.rast/main.c 17 Oct 2007 14:07:23 -0000 1.26
> > +++ vector/v.what.rast/main.c 21 Oct 2007 17:52:42 -0000
> > ... patch
Markus:
> Thanks (for your patience): I have made a Spearfish test:
>
> g.region rast=elevation.10m
> r.random elevation.10m vect=elevpnts n=5000 --o
> v.db.addcol elevpnts col="height double precision"
>
> # old
> time v.what.rast elevpnts rast=elevation.10m col=height
> real 0m25.253s
> user 0m24.756s
> sys 0m0.308s
>
> # new
> real 0m24.040s
> user 0m23.707s
> sys 0m0.297s
>
> Interestingly, timing rather identical (also for 1000). Using the DBF
> driver here.

It is as expected, only a small speed up. While the for i,j loop map may have
been inefficient, it still was fast enough to only take a second to get though.
The bulk of the time was and is being taken up by running db_execute_imediate()
for each point later on in the script. (then db_commit_transaction() is run
after the loop)

To solve this in v.in.garmin and v.in.gpsbabel we wrote all db SET .. to ..
statements to a temporary file, then ran db.execute once for the tmp file
instead of earlier running db.execute for every point. It is not a direct
analogy, but it did made a huge difference there.

That isn't applicable here. The underlying db_execute_immediate()
function can only execute one statement at a time.

Or maybe it is the bsearch() for every point in the db_execute_imediate() loop?

Nope. bsearch() is O(log(n)) per call so it's even less significant
than the duplicate removal (was O(n), is now O(1)).

It's almost certainly the DBMI overhead. Buffered I/O may make some
difference (but won't work on Windows); beyond that, there's nothing
more that can be done with the existing interface.

--
Glynn Clements <glynn@gclements.plus.com>

Markus_Neteler4 · October 22, 2007, 8:59am

Glynn Clements wrote:

Markus Neteler wrote:

>> >> > since v.what.rast is (for me) extremely slow

...

>> >> Maybe it's the qsort(), maybe it's the i,j loop within a loop.

...

>> > The loop is certainly inefficient.

...

> --- vector/v.what.rast/main.c patch

...

I have made a Spearfish test:

g.region rast=elevation.10m
r.random elevation.10m vect=elevpnts n=5000 --o
v.db.addcol elevpnts col="height double precision"

# old
time v.what.rast elevpnts rast=elevation.10m col=height
real 0m25.253s
user 0m24.756s
sys 0m0.308s

# new
real 0m24.040s
user 0m23.707s
sys 0m0.297s

Interestingly, timing rather identical (also for 1000). Using the DBF
driver here.

It was subsequently suggested that it may be the database access which
is the main performance issue. Personally, I suspect that is probably
the case.

You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
to see if that helps at all.

It makes things worse: I have killed the job after 6minutes... (DBF).
Compiling again
without -DUSE_BUFFERED_IO brings me back to 25 seconds.

Running the procedure with SQLite backend as suggested:

time v.what.rast elevpnts rast=elevation.10m col=height
real 0m7.530s
user 0m6.992s
sys 0m0.373s

Running the same procedure connected to PostgreSQL server
on a different machine in intranet:

time v.what.rast elevpnts rast=elevation.10m col=height
real 0m14.085s
user 0m1.419s
sys 0m0.366s

Since SQLite wins, it is not only the DBMI. Still everything is not
extremely fast given that I tested 5000 points but need to work
with 300k points for my real work (ideally for several thousand
maps).

Markus
--
View this message in context: http://www.nabble.com/r.random%3A-now-with-optional-cover-map-parameter-tf4634057.html#a13339117
Sent from the Grass - Dev mailing list archive at Nabble.com.

Glynn_Clements1 · October 22, 2007, 5:59pm

Markus Neteler wrote:

>> >> >> > since v.what.rast is (for me) extremely slow
> ...
>> >> >> Maybe it's the qsort(), maybe it's the i,j loop within a loop.
> ...
>> >> > The loop is certainly inefficient.
> ...
>> > --- vector/v.what.rast/main.c patch
> ...
>> I have made a Spearfish test:
>>
>> g.region rast=elevation.10m
>> r.random elevation.10m vect=elevpnts n=5000 --o
>> v.db.addcol elevpnts col="height double precision"
>>
>> # old
>> time v.what.rast elevpnts rast=elevation.10m col=height
>> real 0m25.253s
>> user 0m24.756s
>> sys 0m0.308s
>>
>> # new
>> real 0m24.040s
>> user 0m23.707s
>> sys 0m0.297s
>>
>> Interestingly, timing rather identical (also for 1000). Using the DBF
>> driver here.
>
> It was subsequently suggested that it may be the database access which
> is the main performance issue. Personally, I suspect that is probably
> the case.
>
> You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
> to see if that helps at all.

It makes things worse: I have killed the job after 6minutes... (DBF).

That implies there's a deadlock, i.e. one side is waiting for data
which is sitting in the other side's buffer.

I'm not sure how that occurs; reading from a stream should flush any
output streams. It would be useful if you could debug this to see
where it's blocking.

Compiling again
without -DUSE_BUFFERED_IO brings me back to 25 seconds.

Running the procedure with SQLite backend as suggested:

time v.what.rast elevpnts rast=elevation.10m col=height
real 0m7.530s
user 0m6.992s
sys 0m0.373s

Running the same procedure connected to PostgreSQL server
on a different machine in intranet:

time v.what.rast elevpnts rast=elevation.10m col=height
real 0m14.085s
user 0m1.419s
sys 0m0.366s

Since SQLite wins, it is not only the DBMI. Still everything is not
extremely fast given that I tested 5000 points but need to work
with 300k points for my real work (ideally for several thousand
maps).

The DBMI may account for a lot of the overhead in the SQLite case. The
difference between SQLite and other drivers has to be due to the
driver, as the DBMI overhead would be the same in each case.

It may be possible to reduce the SQL overhead by storing the
assignments in a separate table then using UPDATE ... FROM (or
CREATE TABLE ... AS). But I don't think that the DBF driver supports
that, and simply populating a table through DBMI may still be
unacceptably slow.

--
Glynn Clements <glynn@gclements.plus.com>

Markus_Neteler4 · October 22, 2007, 8:03pm

Glynn Clements wrote:

Markus Neteler wrote:

Glynn Clements wrote:
> You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
> to see if that helps at all.

It makes things worse: I have killed the job after 6minutes... (DBF).

That implies there's a deadlock, i.e. one side is waiting for data
which is sitting in the other side's buffer.

I'm not sure how that occurs; reading from a stream should flush any
output streams. It would be useful if you could debug this to see
where it's blocking.

I am not sure how to debug such a case, I simply used gdb, waited,
then CTRL-C and bt:

0x0000003d43abfa10 in __read_nocancel () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003d43abfa10 in __read_nocancel () from /lib64/libc.so.6
#1 0x0000003d43a69827 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003d43a6852e in _IO_file_xsgetn_internal () from /lib64/libc.so.6
#3 0x0000003d43a5f0c2 in fread () from /lib64/libc.so.6
#4 0x00002aaaaacfb8b8 in db__recv (buf=0x7fffb4953a5c, size=4) at xdr.c:91
#5 0x00002aaaaacfc678 in db__recv_int (n=0x7fffb4953a5c) at xdrint.c:23
#6 0x00002aaaaacf998e in db__recv_return_code (ret_code=0x7fffb4953a5c) at
ret_codes.c:27
#7 0x00002aaaab3ae0fc in db_start_driver (name=0x114a9d10 "dbf") at
start.c:321
#8 0x00002aaaab3ac91e in db_start_driver_open_database (drvname=0x114a9d10
"dbf",
dbname=0x114a9cd0 "/ssi0/ssi/neteler/grassdata/spearfish60/user1/dbf/")
at db.c:20
#9 0x0000000000401bc0 in main (argc=4, argv=0x7fffb4954bd8) at main.c:125
(gdb)

Using strace:
...
open("/nfsmnt/bartok0/ssi/neteler/software/cvsgrass63/dist.x86_64-unknown-linux-gnu/driver/db/",
O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 15
fstat(15, {st_mode=S_IFDIR|0771, st_size=4096, ...}) = 0
fcntl(15, F_SETFD, FD_CLOEXEC) = 0
getdents(15, /* 8 entries */, 4096) = 208
getdents(15, /* 0 entries */, 4096) = 0
close(15) = 0
pipe([15, 16]) = 0
pipe([17, 18]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaacfa6110) = 22602
close(15) = 0
close(18) = 0
fcntl(16, F_GETFL) = 0x1 (flags O_WRONLY)
fstat(16, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2aaab04ae000
lseek(16, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(17, F_GETFL) = 0 (flags O_RDONLY)
fstat(17, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2aaab04af000
lseek(17, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
read(17, <unfinished ...>

Does this tell you anything?

Markus
--
View this message in context: http://www.nabble.com/r.random%3A-now-with-optional-cover-map-parameter-tf4634057.html#a13351492
Sent from the Grass - Dev mailing list archive at Nabble.com.

Glynn_Clements1 · October 22, 2007, 9:45pm

Markus Neteler wrote:

>> > You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
>> > to see if that helps at all.
>>
>> It makes things worse: I have killed the job after 6minutes... (DBF).
>
> That implies there's a deadlock, i.e. one side is waiting for data
> which is sitting in the other side's buffer.
>
> I'm not sure how that occurs; reading from a stream should flush any
> output streams. It would be useful if you could debug this to see
> where it's blocking.

I am not sure how to debug such a case, I simply used gdb, waited,
then CTRL-C and bt:

In general, you need two terminals, to show both sides, e.g.:

1.
  $ pidof v.what.rast
  $ gdb $GISBASE/bin/v.what.rast
  > attach <pid of v.what.rast process>
  > bt

2.
  $ pidof dbf
  $ gdb $GISBASE/driver/db/dbf
  > attach <pid of dbf process>
  > bt

You may need to set up PATH and LD_LIBRARY_PATH manually so that it
finds the correct binaries.

However, I don't think that it's going to tell me much in this case;
the driver is almost certainly at lib/db/dbmi_driver/driver.c:140 (the
db__recv_procnum() call), with the status code still sitting in the
send buffer.

Try the following patch:

--- lib/db/dbmi_base/xdr.c 15 Oct 2007 05:24:17 -0000 1.6
+++ lib/db/dbmi_base/xdr.c 22 Oct 2007 21:39:28 -0000
@@ -88,6 +88,9 @@
int db__recv(void *buf, size_t size)
{
#if USE_STDIO
+#ifdef USE_BUFFERED_IO
+ fflush(_send);
+#endif
return fread(buf, 1, size, _recv) == size;
#elif USE_READN
return readn(fileno(_recv), buf, size) == size;

--
Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · October 22, 2007, 11:51pm

Glynn Clements wrote:

You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
to see if that helps at all.

I've tried this, and it doesn't appear to help at all. It looks like
the issue is in the driver itself.

It may be more efficient to have an option to populate a new table
rather than updating an existing one. However, AFAICT, none of the
existing drivers actually support db__driver_{update,insert}.

--
Glynn Clements <glynn@gclements.plus.com>

H_B · October 23, 2007, 2:23am

Hi,

I have run valgrind's callgrind profiler for v.what.rast.

results + screen dumps are here:
http://bambi.otago.ac.nz/hamish/grass/dev/

to look at it in detail install kcachegrind and graphviz, then just
kcachegrind <file>

(after un-gzipping the callgrind output files)

I added what I did as an example on the wiki debugging page:
http://grass.gdf-hannover.de/wiki/GRASS_Debugging#Kcachegrind

quick summary of results:
99.5% of the cost was taken by the dbf process, not v.what.rast
(watching 'top' shows you the same)

In the dbf process (#15561) about 1/3rd was taken by G_debug(),
and about 1/2 was taken by G_debug() + G_strcasecmp()

Hamish

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

H_B · October 23, 2007, 3:55am

Markus

Still everything is not extremely fast given that I tested 5000 points but
need to work with 300k points for my real work (ideally for several
thousand maps).

how about using r.what:

v.out.ascii | r.what in=rastmap_to_query | v.in.ascii -z

?

example: (DBF)

G63> time v.out.ascii rand10k fs=space | \
r.what in=elevation.dem null=NULL| grep -v '|NULL$' | \
v.in.ascii -z out=rand10k_elev x=1 y=2 cat=3 z=4

Scanning input for column types ...
Maximum input row length: 44
Maximum number of columns: 4
Minimum number of columns: 4
Importing points ...
Building topology ...
9905 primitives registered
[...]
v.in.ascii complete.

real 0m12.864s
user 0m11.449s
sys 0m1.256s

If you leave off "-z z=4" (thus let v.in.ascii create a table) it only
takes 1 second more.

real 0m13.578s
user 0m11.953s
sys 0m1.448s

Hamish

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Markus_Neteler4 · October 23, 2007, 11:29am

Glynn Clements wrote:

Markus Neteler wrote:

>> > You could try re-compiling the DBMI libraries with -DUSE_BUFFERED_IO
>> > to see if that helps at all.

...

However, I don't think that it's going to tell me much in this case;
the driver is almost certainly at lib/db/dbmi_driver/driver.c:140 (the
db__recv_procnum() call), with the status code still sitting in the
send buffer.

Try the following patch:

--- lib/db/dbmi_base/xdr.c 15 Oct 2007 05:24:17 -0000 1.6
+++ lib/db/dbmi_base/xdr.c 22 Oct 2007 21:39:28 -0000
@@ -88,6 +88,9 @@
int db__recv(void *buf, size_t size)
{
#if USE_STDIO
+#ifdef USE_BUFFERED_IO
+ fflush(_send);
+#endif
return fread(buf, 1, size, _recv) == size;
#elif USE_READN
return readn(fileno(_recv), buf, size) == size;

This actually solves the problem. I have taken liberty to not debug
in details but directly tried (successfully) the patch.
Then we are back to 25 seconds, also with -DUSE_BUFFERED_IO
(and DBF). With SQLite 7 seconds, it seems that the flag has no
relevant effect.

Markus

Markus
--
View this message in context: http://www.nabble.com/r.random%3A-now-with-optional-cover-map-parameter-tf4634057.html#a13362043
Sent from the Grass - Dev mailing list archive at Nabble.com.

Dylan_Beaudette1 · October 23, 2007, 5:23pm

On Monday 22 October 2007, Hamish wrote:

Hi,

I have run valgrind's callgrind profiler for v.what.rast.

results + screen dumps are here:
  http://bambi.otago.ac.nz/hamish/grass/dev/

to look at it in detail install kcachegrind and graphviz, then just
  kcachegrind <file>

(after un-gzipping the callgrind output files)

I added what I did as an example on the wiki debugging page:
  http://grass.gdf-hannover.de/wiki/GRASS_Debugging#Kcachegrind

Very cool. It would be neat to apply this to more GRASS functions to see what
is eating CPU time.

Cheers,

Dylan

quick summary of results:
  99.5% of the cost was taken by the dbf process, not v.what.rast
  (watching 'top' shows you the same)

  In the dbf process (#15561) about 1/3rd was taken by G_debug(),
  and about 1/2 was taken by G_debug() + G_strcasecmp()

Hamish

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

--
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341