[GRASS-dev] v.what and spatial index

[mlennert]

Hello,

Trying to use v.what on a fairly large file (124658 areas) I noticed that it rebuilds the spatial index at every usage, obviously making it extremely inefficient.

This is discussed briefly in bug #3193:

http://www.intevation.de/rt/webrt?serial_num=3193&display=History

And Radim mentions spatial index in his Vector ToDo document:

http://freegis.org/cgi-bin/viewcvs.cgi/grass6/doc/vector/TODO?rev=HEAD&content-type=text/vnd.viewcvs-markup

This is a serious issue when you have to query such large maps frequently.

Radim wrote at the time:

"Writing of the index to disk could be enabled by GRASS variable and/or
v.build option. But that changes vector format which must be supported
in all next version. Before we enable writing of spatial index, it
should be revised if the format can be changed to save some space."

I think that seen the fact that we are quickly moving to new releases and then to further development, it might be the time to take this up again in order to plan ahead.

In the bug report, Hamish proposes to create the spatial index once per session and erase it when leaving the session. I don't know if space is such as problem, but for me even this seems inefficient. Once the index is created it should stay there even when leaving the session. Or at least there should be a configurable option (maybe a GRASS environment variable) to leave the index files in place.

Any reactions ?

Moritz

[mlennert]

Moritz Lennert wrote:

Hello,

Trying to use v.what on a fairly large file (124658 areas) I noticed that it rebuilds the spatial index at every usage, obviously making it extremely inefficient.

This is discussed briefly in bug #3193:

http://www.intevation.de/rt/webrt?serial_num=3193&display=History

And Radim mentions spatial index in his Vector ToDo document:

http://freegis.org/cgi-bin/viewcvs.cgi/grass6/doc/vector/TODO?rev=HEAD&content-type=text/vnd.viewcvs-markup

This is a serious issue when you have to query such large maps frequently.

Radim wrote at the time:

"Writing of the index to disk could be enabled by GRASS variable and/or
v.build option. But that changes vector format which must be supported
in all next version. Before we enable writing of spatial index, it
should be revised if the format can be changed to save some space."

I think that seen the fact that we are quickly moving to new releases and then to further development, it might be the time to take this up again in order to plan ahead.

In the bug report, Hamish proposes to create the spatial index once per session and erase it when leaving the session. I don't know if space is such as problem, but for me even this seems inefficient. Once the index is created it should stay there even when leaving the session. Or at least there should be a configurable option (maybe a GRASS environment variable) to leave the index files in place.

Any reactions ?

I have looked into the issue a bit more, and I see the following:

v.what calls Vect_build_spatial_index()

This function checks whether the spatial index is already available in memory (checking whether Map->plus.Spidx_built is set) and if not, goes on to call Vect_build_sidx_from_topo() which builds the actual index.

IIUC, the memory holding the index is released automatically when v.what calls exit().

This means that for every v.what call the index needs to be rebuilt.

The discussion so far has been about putting the spatial index into a file and then either erasing that file when the GRASS session is closed, or even keeping it across sessions.

But, would it be possible to keep it in memory after v.what exits (or whatever program is first to build the index in a session) and erase it at the end of the GRASS session ? Or this is impossible with GRASS' modular structure ?

Moritz

[Glynn_Clements1]

Moritz Lennert wrote:

v.what calls Vect_build_spatial_index()

This function checks whether the spatial index is already available in
memory (checking whether Map->plus.Spidx_built is set) and if not, goes
on to call Vect_build_sidx_from_topo() which builds the actual index.

IIUC, the memory holding the index is released automatically when v.what
calls exit().

This means that for every v.what call the index needs to be rebuilt.

The discussion so far has been about putting the spatial index into a
file and then either erasing that file when the GRASS session is closed,
or even keeping it across sessions.

But, would it be possible to keep it in memory after v.what exits (or
whatever program is first to build the index in a session) and erase it
at the end of the GRASS session ? Or this is impossible with GRASS'
modular structure ?

Keeping it in memory isn't a practical solution; all of the standard
shared memory mechanisms are relatively complex to use and not
particularly portable. If a file is used often enough, and the system
has enough unused memory, the OS will cache the file in memory.

The main point about storing the spatial index is that the
functionality needs to be integrated into the vector libraries and/or
core modules, so that the index is either updated or deleted if the
map is changed. Any mechanism which requires the user to manually
synchronise the index with the map isn't acceptable, IMHO.

--
Glynn Clements <glynn@gclements.plus.com>

[H_B]

Moritz Lennert wrote:
> v.what calls Vect_build_spatial_index()

note that originally the spatial index *was* written to disk; try running
GRASS 5.7.0 and see (?? not sure exactly when it was removed).

Anyway it was removed after 1-2 years of running with it.

Not a technical answer or argument against it, but be aware that it was
removed with a lot of experience doing it the other way.

further detail can be found in the mailing list archives from the time
it was removed. (maybe same time v.build.all was added?)

Hamish

[mlennert]

Hamish wrote:

Moritz Lennert wrote:

v.what calls Vect_build_spatial_index()

note that originally the spatial index *was* written to disk; try running
GRASS 5.7.0 and see (?? not sure exactly when it was removed).

Anyway it was removed after 1-2 years of running with it.

Not a technical answer or argument against it, but be aware that it was
removed with a lot of experience doing it the other way.

>
> further detail can be found in the mailing list archives from the time
> it was removed. (maybe same time v.build.all was added?)

http://grass.itc.it/pipermail/grass-dev/2004-November/016036.html
and
http://grass.itc.it/pipermail/grass-dev/2004-November/016045.html

and
http://grass.itc.it/pipermail/grass-commit/2004-November/015712.html ff

The main reasoning behind this seems to have been for the needs of large point datasets.

Radim's tests here
http://grass.itc.it/pipermail/grass-dev/2004-November/016036.html

seem to indicate that when the spatial index is reused several times by the module, the overhead of building it on the fly is not so important.

However, v.what does not have an interactive mode (that's the whole idea of v.what so it fits better into the GUI). This means that since it builds the spatial index everytime you call it, everytime you use the query tool you get the building of the spatial index.

Now, I see several options from here:

- Change the query tool in the GUI so that it collects points and then sends the whole series to v.what. This has the big disadvantage that you don't see the results of your clicks immediately (i.e. you "blindly" click on a series of points and then push "go" to see the results for all of these points). It still means that whenever you resuse the query tool spatial index gets rebuilt which can be very annoying if you have to query information out of a map. So, even though this might be a temporary "solution", I don't find it very satisfactory.

- Reimplement an interactive version of v.what, or rather make it possible to use d.what.vect from the GUI. This would probably mean recoding d.what.vect in whatever the GUI language and call GRASS library functions from there.

- Reuse spatial index on disk, possibly making it optional on a per-location or even per-mapset basis, so that those that need can use it and those that would find it annoying could just ignore it.

Moritz

[Glynn_Clements1]

Moritz Lennert wrote:

However, v.what does not have an interactive mode (that's the whole idea
of v.what so it fits better into the GUI). This means that since it
builds the spatial index everytime you call it, everytime you use the
query tool you get the building of the spatial index.

Now, I see several options from here:

- Change the query tool in the GUI so that it collects points and then
sends the whole series to v.what. This has the big disadvantage that you
don't see the results of your clicks immediately (i.e. you "blindly"
click on a series of points and then push "go" to see the results for
all of these points). It still means that whenever you resuse the query
tool spatial index gets rebuilt which can be very annoying if you have
to query information out of a map. So, even though this might be a
temporary "solution", I don't find it very satisfactory.

- Reimplement an interactive version of v.what, or rather make it
possible to use d.what.vect from the GUI. This would probably mean
recoding d.what.vect in whatever the GUI language and call GRASS library
functions from there.

The obvious approach is to extend v.what to allow coordinates to be
read from stdin.

BTW, it appears that v.what currently ignores all coordinate pairs
other than the first.

- Reuse spatial index on disk, possibly making it optional on a
per-location or even per-mapset basis, so that those that need can use
it and those that would find it annoying could just ignore it.

This is a separate issue to making v.what more efficient; other
modules may benefit from not having to regenerate the spatial index
for each invocation.

--
Glynn Clements <glynn@gclements.plus.com>

[mlennert]

Glynn Clements wrote:

Moritz Lennert wrote:

However, v.what does not have an interactive mode (that's the whole idea of v.what so it fits better into the GUI). This means that since it builds the spatial index everytime you call it, everytime you use the query tool you get the building of the spatial index.

Now, I see several options from here:

- Change the query tool in the GUI so that it collects points and then sends the whole series to v.what. This has the big disadvantage that you don't see the results of your clicks immediately (i.e. you "blindly" click on a series of points and then push "go" to see the results for all of these points). It still means that whenever you resuse the query tool spatial index gets rebuilt which can be very annoying if you have to query information out of a map. So, even though this might be a temporary "solution", I don't find it very satisfactory.

- Reimplement an interactive version of v.what, or rather make it possible to use d.what.vect from the GUI. This would probably mean recoding d.what.vect in whatever the GUI language and call GRASS library functions from there.

The obvious approach is to extend v.what to allow coordinates to be
read from stdin.

BTW, it appears that v.what currently ignores all coordinate pairs
other than the first.

Didn't even try that, but yes. Shouldn't be too hard to implement, though. Trevor, any reason you left this out ?

On the other hand, I just noticed that it allows querying multiple maps.

Michael would it be possible to allow some form of multiple map selection in the gui to then call v.what with the names of all of these maps ?

- Reuse spatial index on disk, possibly making it optional on a per-location or even per-mapset basis, so that those that need can use it and those that would find it annoying could just ignore it.

This is a separate issue to making v.what more efficient; other
modules may benefit from not having to regenerate the spatial index
for each invocation.

v.what was just the example I stumbled upon.

Although:

mlennert@geog-pc40:~/SRC/GRASS/CVS/grass6/vector$ grep -R Vect_build_spatial_index *
v.what/main.c: Vect_build_spatial_index ( &Map[i], NULL );

Is there another way of building the spatial index which other models use ?

Moritz

[Radim_Blazek3]

On 9/7/06, Moritz Lennert <mlennert@club.worldonline.be> wrote:

Radim's tests here
http://grass.itc.it/pipermail/grass-dev/2004-November/016036.html

seem to indicate that when the spatial index is reused several times by
the module, the overhead of building it on the fly is not so important.

Yes, I think that the best solution is to rewrite the library to store
spatial index in a file, i.e. never keep the whole structure in memory
and read the tree items directly from the file. That should be
relatively simple IMO.
That is also described in my vector TODO.
It means to add support to read/write Node structures from/to file instead of
memory. Currently the nodes are referenced by pointer to memory which
could be replaced by file offset. You need to add some functions like
   struct Node *RTreeReadNode(long offset)
and use it in other functions, for example:
    struct Node *node = RTreeReadNode ( n->branch[i].child );
    hitCount += RTreeSearch(node, R, shcb, cbarg);
    RTreeFreeNode(node);
instead of
    hitCount += RTreeSearch(n->branch[i].child, R, shcb, cbarg);
But it will be better to keep both options - file and memory
and use it according to vector size.

Most of the general problems and suggested solutions are described in the TODO.

Radim

[Trevor_Wiens2]

On Fri, 08 Sep 2006 09:47:16 +0200
Moritz Lennert wrote:

Glynn Clements wrote:
> Moritz Lennert wrote:
>
>> However, v.what does not have an interactive mode (that's the whole idea
>> of v.what so it fits better into the GUI). This means that since it
>> builds the spatial index everytime you call it, everytime you use the
>> query tool you get the building of the spatial index.
>>
>> Now, I see several options from here:
>
>> - Change the query tool in the GUI so that it collects points and then
>> sends the whole series to v.what. This has the big disadvantage that you
>> don't see the results of your clicks immediately (i.e. you "blindly"
>> click on a series of points and then push "go" to see the results for
>> all of these points). It still means that whenever you resuse the query
>> tool spatial index gets rebuilt which can be very annoying if you have
>> to query information out of a map. So, even though this might be a
>> temporary "solution", I don't find it very satisfactory.
>>
>> - Reimplement an interactive version of v.what, or rather make it
>> possible to use d.what.vect from the GUI. This would probably mean
>> recoding d.what.vect in whatever the GUI language and call GRASS library
>> functions from there.
>
> The obvious approach is to extend v.what to allow coordinates to be
> read from stdin.
>
> BTW, it appears that v.what currently ignores all coordinate pairs
> other than the first.

Didn't even try that, but yes. Shouldn't be too hard to implement,
though. Trevor, any reason you left this out ?

I simply removed the interactive functionality in d.what.vect which
Radim wrote. Other than that I made only minimal changes to the code.
That could be added however fairly easily. I think however the more
pressing issue is some of the re-write stuff that Radim refers to in
his TODO. At this time, I don't have the familiarity with the vector
library to work on that list of items, sorry. However, once the spatial
index is stored in a file and read as needed, this would eliminate the
need for entering multiple point sets wouldn't it? If however we need
multiple point sets, I'm sure that I could add that relatively quickly
if needed.

T
--
Trevor Wiens
twiens@interbaun.com

The significant problems that we face cannot be solved at the same
level of thinking we were at when we created them.
(Albert Einstein)

[mlennert]

Glynn Clements wrote:

Moritz Lennert wrote:

However, v.what does not have an interactive mode (that's the whole idea of v.what so it fits better into the GUI). This means that since it builds the spatial index everytime you call it, everytime you use the query tool you get the building of the spatial index.

Now, I see several options from here:

- Change the query tool in the GUI so that it collects points and then sends the whole series to v.what. This has the big disadvantage that you don't see the results of your clicks immediately (i.e. you "blindly" click on a series of points and then push "go" to see the results for all of these points). It still means that whenever you resuse the query tool spatial index gets rebuilt which can be very annoying if you have to query information out of a map. So, even though this might be a temporary "solution", I don't find it very satisfactory.

- Reimplement an interactive version of v.what, or rather make it possible to use d.what.vect from the GUI. This would probably mean recoding d.what.vect in whatever the GUI language and call GRASS library functions from there.

The obvious approach is to extend v.what to allow coordinates to be
read from stdin.

How would this change from the first approach (if v.what could easily be made to accept multiple coordinates) ? This would again mean 'blindly' collecting points before seeing the results, or having to call v.what multiple times.

BTW, it appears that v.what currently ignores all coordinate pairs
other than the first.

- Reuse spatial index on disk, possibly making it optional on a per-location or even per-mapset basis, so that those that need can use it and those that would find it annoying could just ignore it.

This is a separate issue to making v.what more efficient; other
modules may benefit from not having to regenerate the spatial index
for each invocation.

Well, for me the main inefficiency in v.what is the rebuilding of the spatial index ...

I'll have a look and try to understand the index building code and Radim's suggestions of changing it, but I'm way beyond my knowledge and capacity in C, here...

Moritz

[Glynn_Clements1]

Moritz Lennert wrote:

>> However, v.what does not have an interactive mode (that's the whole idea
>> of v.what so it fits better into the GUI). This means that since it
>> builds the spatial index everytime you call it, everytime you use the
>> query tool you get the building of the spatial index.
>>
>> Now, I see several options from here:
>
>> - Change the query tool in the GUI so that it collects points and then
>> sends the whole series to v.what. This has the big disadvantage that you
>> don't see the results of your clicks immediately (i.e. you "blindly"
>> click on a series of points and then push "go" to see the results for
>> all of these points). It still means that whenever you resuse the query
>> tool spatial index gets rebuilt which can be very annoying if you have
>> to query information out of a map. So, even though this might be a
>> temporary "solution", I don't find it very satisfactory.
>>
>> - Reimplement an interactive version of v.what, or rather make it
>> possible to use d.what.vect from the GUI. This would probably mean
>> recoding d.what.vect in whatever the GUI language and call GRASS library
>> functions from there.
>
> The obvious approach is to extend v.what to allow coordinates to be
> read from stdin.

How would this change from the first approach (if v.what could easily be
made to accept multiple coordinates) ? This would again mean 'blindly'
collecting points before seeing the results, or having to call v.what
multiple times.

v.what would read one coordinate pair, perform the lookup, write out
the result(s), repeat until EOF.

gis.m would start one v.what process; each time you click the mouse
button, it would send the coordinates to the v.what process, read the
result, then display it. It would need to restart v.what if the set of
maps changed, but could use a single process to look up multiple
points.

--
Glynn Clements <glynn@gclements.plus.com>

[mlennert]

Glynn Clements wrote:

Moritz Lennert wrote:

However, v.what does not have an interactive mode (that's the whole idea of v.what so it fits better into the GUI). This means that since it builds the spatial index everytime you call it, everytime you use the query tool you get the building of the spatial index.

Now, I see several options from here:
- Change the query tool in the GUI so that it collects points and then sends the whole series to v.what. This has the big disadvantage that you don't see the results of your clicks immediately (i.e. you "blindly" click on a series of points and then push "go" to see the results for all of these points). It still means that whenever you resuse the query tool spatial index gets rebuilt which can be very annoying if you have to query information out of a map. So, even though this might be a temporary "solution", I don't find it very satisfactory.

- Reimplement an interactive version of v.what, or rather make it possible to use d.what.vect from the GUI. This would probably mean recoding d.what.vect in whatever the GUI language and call GRASS library functions from there.

The obvious approach is to extend v.what to allow coordinates to be
read from stdin.

How would this change from the first approach (if v.what could easily be made to accept multiple coordinates) ? This would again mean 'blindly' collecting points before seeing the results, or having to call v.what multiple times.

v.what would read one coordinate pair, perform the lookup, write out
the result(s), repeat until EOF.

gis.m would start one v.what process; each time you click the mouse
button, it would send the coordinates to the v.what process, read the
result, then display it. It would need to restart v.what if the set of
maps changed, but could use a single process to look up multiple
points.

Here's my attempt at applying what you suggest. It's not very complicated, and seems to work fine here, but I'd appreciate if someone could check that I didn't do anything wrong before committing.

Now we will have to change the query function in the gis manager (from line 1474 of gui/tcltk/gis.m/mapcanvas.tcl) to use v.what with the -s flag and to provide continuous flow of coordinates and an EOF at the end. Michael, can you do this ? Or could someone tell me in rough lines how this can be done ?

Moritz

(attachments)

v.what.stdin.patch (1.87 KB)

[Glynn_Clements1]

Moritz Lennert wrote:

>> How would this change from the first approach (if v.what could easily be
>> made to accept multiple coordinates) ? This would again mean 'blindly'
>> collecting points before seeing the results, or having to call v.what
>> multiple times.
>
> v.what would read one coordinate pair, perform the lookup, write out
> the result(s), repeat until EOF.
>
> gis.m would start one v.what process; each time you click the mouse
> button, it would send the coordinates to the v.what process, read the
> result, then display it. It would need to restart v.what if the set of
> maps changed, but could use a single process to look up multiple
> points.

Here's my attempt at applying what you suggest. It's not very
complicated, and seems to work fine here, but I'd appreciate if someone
could check that I didn't do anything wrong before committing.

One issue which won't show up when running it directly from the shell,
but will if you run it via pipes: you need to explicitly set stdin and
stdout to line-buffered operation with e.g.:

  setvbuf(stdin, NULL, _IOLBF, 0);
  setvbuf(stdout, NULL, _IOLBF, 0);

Now we will have to change the query function in the gis manager (from
line 1474 of gui/tcltk/gis.m/mapcanvas.tcl) to use v.what with the -s
flag and to provide continuous flow of coordinates and an EOF at the
end. Michael, can you do this ? Or could someone tell me in rough lines
how this can be done ?

Ah. Tcl's "open" function doesn't directly support using pipes for
both stdin and stdout; you can choose read (stdout is a pipe, stdin is
inherited) or write (stdin is a pipe, stdout is inherited), but not
both.

You will need to create a named pipe (with mkfifo) then explicitly
redirect one of the ends to it, e.g.

  set fifo $tmpdir/$tmpname
  exec mkfifo $fifo
  set rfh [open "|v.what map=$maplist <$fifo" r]
  fconfigure $rfh -buffering line
  set wfh [open $fifo w]
  fconfigure $wfh -buffering line

--
Glynn Clements <glynn@gclements.plus.com>

[mlennert]

Glynn Clements wrote:

Moritz Lennert wrote:

How would this change from the first approach (if v.what could easily be made to accept multiple coordinates) ? This would again mean 'blindly' collecting points before seeing the results, or having to call v.what multiple times.

v.what would read one coordinate pair, perform the lookup, write out
the result(s), repeat until EOF.

gis.m would start one v.what process; each time you click the mouse
button, it would send the coordinates to the v.what process, read the
result, then display it. It would need to restart v.what if the set of
maps changed, but could use a single process to look up multiple
points.

Here's my attempt at applying what you suggest. It's not very complicated, and seems to work fine here, but I'd appreciate if someone could check that I didn't do anything wrong before committing.

One issue which won't show up when running it directly from the shell,
but will if you run it via pipes: you need to explicitly set stdin and
stdout to line-buffered operation with e.g.:

  setvbuf(stdin, NULL, _IOLBF, 0);
  setvbuf(stdout, NULL, _IOLBF, 0);

I assume this can be done anywhere as long as its before the call to stdin ?

Now we will have to change the query function in the gis manager (from line 1474 of gui/tcltk/gis.m/mapcanvas.tcl) to use v.what with the -s flag and to provide continuous flow of coordinates and an EOF at the end. Michael, can you do this ? Or could someone tell me in rough lines how this can be done ?

Ah. Tcl's "open" function doesn't directly support using pipes for
both stdin and stdout; you can choose read (stdout is a pipe, stdin is
inherited) or write (stdin is a pipe, stdout is inherited), but not
both.

You will need to create a named pipe (with mkfifo) then explicitly
redirect one of the ends to it, e.g.

  set fifo $tmpdir/$tmpname
  exec mkfifo $fifo
  set rfh [open "|v.what map=$maplist <$fifo" r]
  fconfigure $rfh -buffering line
  set wfh [open $fifo w]
  fconfigure $wfh -buffering line

Ok, I understand the principle of this, but don't know how and where to apply it in mapcanvas.tcl. Don't have the time to dig into it right now. Michael ?

Another question is whether it is worth doing all this (don't know how much work it is to apply Glynn's solution) if we are going to change GUI in a foreseeable future...

Thanks, Glynn, for your help.

Moritz

[Trevor_Wiens2]

On Wed, 13 Sep 2006 15:32:00 +0200
Moritz Lennert wrote:

Glynn Clements wrote:
> Moritz Lennert wrote:
>
>>>> How would this change from the first approach (if v.what could easily be
>>>> made to accept multiple coordinates) ? This would again mean 'blindly'
>>>> collecting points before seeing the results, or having to call v.what
>>>> multiple times.
>>> v.what would read one coordinate pair, perform the lookup, write out
>>> the result(s), repeat until EOF.
>>>
>>> gis.m would start one v.what process; each time you click the mouse
>>> button, it would send the coordinates to the v.what process, read the
>>> result, then display it. It would need to restart v.what if the set of
>>> maps changed, but could use a single process to look up multiple
>>> points.
>> Here's my attempt at applying what you suggest. It's not very
>> complicated, and seems to work fine here, but I'd appreciate if someone
>> could check that I didn't do anything wrong before committing.
>
> One issue which won't show up when running it directly from the shell,
> but will if you run it via pipes: you need to explicitly set stdin and
> stdout to line-buffered operation with e.g.:
>
> setvbuf(stdin, NULL, _IOLBF, 0);
> setvbuf(stdout, NULL, _IOLBF, 0);

I assume this can be done anywhere as long as its before the call to stdin ?

>
>> Now we will have to change the query function in the gis manager (from
>> line 1474 of gui/tcltk/gis.m/mapcanvas.tcl) to use v.what with the -s
>> flag and to provide continuous flow of coordinates and an EOF at the
>> end. Michael, can you do this ? Or could someone tell me in rough lines
>> how this can be done ?
>
> Ah. Tcl's "open" function doesn't directly support using pipes for
> both stdin and stdout; you can choose read (stdout is a pipe, stdin is
> inherited) or write (stdin is a pipe, stdout is inherited), but not
> both.
>
> You will need to create a named pipe (with mkfifo) then explicitly
> redirect one of the ends to it, e.g.
>
> set fifo $tmpdir/$tmpname
> exec mkfifo $fifo
> set rfh [open "|v.what map=$maplist <$fifo" r]
> fconfigure $rfh -buffering line
> set wfh [open $fifo w]
> fconfigure $wfh -buffering line

Ok, I understand the principle of this, but don't know how and where to
apply it in mapcanvas.tcl. Don't have the time to dig into it right now.
Michael ?

Another question is whether it is worth doing all this (don't know how
much work it is to apply Glynn's solution) if we are going to change GUI
in a foreseeable future...

Moritz,

Thanks for doing this. This change will also be useful for future GUI's.

T
--
Trevor Wiens
twiens@interbaun.com

The significant problems that we face cannot be solved at the same
level of thinking we were at when we created them.
(Albert Einstein)

[mlennert]

Michael Barton wrote:

Moritz,

I'm tied up all day. I'm not exactly sure how you want to change the behavior of v.what in gism. That is, I don't understand about a stream of coordinates. Could you explain it a bit better. Sorry if I'm dense.

The problem: v.what builds the spatial index everytime it is called. For large files this takes a while making the use of v.what on several points in a row very slow. The query tool of gis.m calls v.what for every query click.

The solution: Allow v.what to accept more than one pair of coordinates by listening on stdin until it reveives an EOF call. The query tool in gis.m opens a connection to v.what in the way described by Glynn below, feeds in the coordinates everytime the user clicks on the map, displays the relevant information and continues to wait for the next click unless it receives some sort of EOF (example: user choses different tool in map display window - or a right click, whatever seems the most convenient). That way a user can display a map, click at different points of the map and immediately receive the information, without having to wait for the spatial index to be built because of separate v.what calls.

Does that make sense ?

If it is not too difficult to implement in gis.m it would be great to have it. Otherwise, it would have to wait for the next gui...

Moritz

Michael
On Sep 13, 2006, at 6:32 AM, Moritz Lennert wrote:

Glynn Clements wrote:

Moritz Lennert wrote:

How would this change from the first approach (if v.what could easily be made to accept multiple coordinates) ? This would again mean 'blindly' collecting points before seeing the results, or having to call v.what multiple times.

v.what would read one coordinate pair, perform the lookup, write out
the result(s), repeat until EOF.

gis.m would start one v.what process; each time you click the mouse
button, it would send the coordinates to the v.what process, read the
result, then display it. It would need to restart v.what if the set of
maps changed, but could use a single process to look up multiple
points.

Here's my attempt at applying what you suggest. It's not very complicated, and seems to work fine here, but I'd appreciate if someone could check that I didn't do anything wrong before committing.

One issue which won't show up when running it directly from the shell,
but will if you run it via pipes: you need to explicitly set stdin and
stdout to line-buffered operation with e.g.:
    setvbuf(stdin, NULL, _IOLBF, 0);
    setvbuf(stdout, NULL, _IOLBF, 0);

I assume this can be done anywhere as long as its before the call to stdin ?

Now we will have to change the query function in the gis manager (from line 1474 of gui/tcltk/gis.m/mapcanvas.tcl) to use v.what with the -s flag and to provide continuous flow of coordinates and an EOF at the end. Michael, can you do this ? Or could someone tell me in rough lines how this can be done ?

Ah. Tcl's "open" function doesn't directly support using pipes for
both stdin and stdout; you can choose read (stdout is a pipe, stdin is
inherited) or write (stdin is a pipe, stdout is inherited), but not
both.
You will need to create a named pipe (with mkfifo) then explicitly
redirect one of the ends to it, e.g.
    set fifo $tmpdir/$tmpname
    exec mkfifo $fifo
    set rfh [open "|v.what map=$maplist <$fifo" r]
    fconfigure $rfh -buffering line
    set wfh [open $fifo w]
    fconfigure $wfh -buffering line

Ok, I understand the principle of this, but don't know how and where to apply it in mapcanvas.tcl. Don't have the time to dig into it right now. Michael ?

Another question is whether it is worth doing all this (don't know how much work it is to apply Glynn's solution) if we are going to change GUI in a foreseeable future...

Thanks, Glynn, for your help.

Moritz

____________________
C. Michael Barton, Professor of Anthropology
School of Human Evolution and Social Change
PO Box 872402
Arizona State University
Tempe, AZ 85287-2402
USA

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

[mlennert]

Glynn Clements wrote:

Moritz Lennert wrote:

How would this change from the first approach (if v.what could easily be made to accept multiple coordinates) ? This would again mean 'blindly' collecting points before seeing the results, or having to call v.what multiple times.

v.what would read one coordinate pair, perform the lookup, write out
the result(s), repeat until EOF.

gis.m would start one v.what process; each time you click the mouse
button, it would send the coordinates to the v.what process, read the
result, then display it. It would need to restart v.what if the set of
maps changed, but could use a single process to look up multiple
points.

Here's my attempt at applying what you suggest. It's not very complicated, and seems to work fine here, but I'd appreciate if someone could check that I didn't do anything wrong before committing.

One issue which won't show up when running it directly from the shell,
but will if you run it via pipes: you need to explicitly set stdin and
stdout to line-buffered operation with e.g.:

  setvbuf(stdin, NULL, _IOLBF, 0);
  setvbuf(stdout, NULL, _IOLBF, 0);

Just to show my ignorance: why stdout ? I'm only using stdin, or ?

Moritz

[Glynn_Clements1]

Moritz Lennert wrote:

> One issue which won't show up when running it directly from the shell,
> but will if you run it via pipes: you need to explicitly set stdin and
> stdout to line-buffered operation with e.g.:
>
> setvbuf(stdin, NULL, _IOLBF, 0);
> setvbuf(stdout, NULL, _IOLBF, 0);

I assume this can be done anywhere as long as its before the call to stdin ?

It needs to be done before anything reads from stdin or writes to
stdout.

--
Glynn Clements <glynn@gclements.plus.com>

[Glynn_Clements1]

Moritz Lennert wrote:

>>>> How would this change from the first approach (if v.what could easily be
>>>> made to accept multiple coordinates) ? This would again mean 'blindly'
>>>> collecting points before seeing the results, or having to call v.what
>>>> multiple times.
>>> v.what would read one coordinate pair, perform the lookup, write out
>>> the result(s), repeat until EOF.
>>>
>>> gis.m would start one v.what process; each time you click the mouse
>>> button, it would send the coordinates to the v.what process, read the
>>> result, then display it. It would need to restart v.what if the set of
>>> maps changed, but could use a single process to look up multiple
>>> points.
>> Here's my attempt at applying what you suggest. It's not very
>> complicated, and seems to work fine here, but I'd appreciate if someone
>> could check that I didn't do anything wrong before committing.
>
> One issue which won't show up when running it directly from the shell,
> but will if you run it via pipes: you need to explicitly set stdin and
> stdout to line-buffered operation with e.g.:
>
> setvbuf(stdin, NULL, _IOLBF, 0);
> setvbuf(stdout, NULL, _IOLBF, 0);

Just to show my ignorance: why stdout ? I'm only using stdin, or ?

v.what would read coordinates from stdin and write information
regarding the point/line/area at (or near) those coordinates to
stdout.

See vector/v.what/what.c; there are 25 occurrences of "stdout" in that
file.

Actually, as what() always calls fflush(stdout), you can omit the
setvbuf() call for stdout.

--
Glynn Clements <glynn@gclements.plus.com>