[GRASS-dev] [GRASS GIS] #224: cache bug in DGLib

#224: cache bug in DGLib
----------------------------------+-----------------------------------------
Reporter: martinl | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: major | Milestone: 6.4.0
Component: default | Version: svn-develbranch6
Keywords: vector, dglib, cache | Platform: All
      Cpu: All |
----------------------------------+-----------------------------------------
Fix outstanding bug DGLib ...

See

http://www.nabble.com/-GRASS5--Grass-5.1's-Directed-Graph-Library-CVS-
snapshot-td8581688.html#a8581693

http://www.nabble.com/v.net.path-td4210952.html#a4217671

http://www.nabble.com/-GRASS5--dgtable-vector-library-
td8582331.html#a8582342

Martin

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by msieczka):

* cc: msieczka (added)

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

A general question about Trac to email:

Is there a way to limit those track changes that are sent to the list ? Currently, any changes, even small ones as below (same for changes in keywords, etc), are sent to the list, leading to quite a lot of traffic...

Moritz

On 09/07/08 18:33, GRASS GIS wrote:

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new Priority: major | Milestone: 6.4.0 Component: default | Version: svn-develbranch6 Resolution: | Keywords: vector, dglib, cache Platform: All | Cpu: All ----------------------+-----------------------------------------------------
Changes (by msieczka):

* cc: msieczka (added)

------------------------------------------------------------------------

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by neteler):

See also
http://trac.osgeo.org/grass/browser/grass/trunk/lib/vector/dglib/BUGS

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by mmetz):

BUG1 fixed in trunk r36109 and devbr6 r36110.

Cache not yet enabled again in Vlib, neither trunk nor devbr6.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by mmetz):

dglib cache back to life in trunk r36113 and devbr6 r36114

testing is very welcome!

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by mmetz):

  * component: default => Vector

Comment:

Leaving ticket open because of BUG2

IMHO BUG2 is nearly impossible to fix without the help of Roberto
Micarelli. At least for me the code is too cryptic.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by mlennert):

Replying to [comment:4 mmetz]:
> dglib cache back to life in trunk r36113 and devbr6 r36114
>
> testing is very welcome!

Wow !! Just tried v.net.alloc and v.net.iso on streets_wake and schools of
the nc_spm_06 demo data and it finishes in seconds, instead of running the
entire night without coming to a result ! Still need to look at the actual
results more in detail (look good at rough glance). Also ran a few
v.net.path's and everything seems fine.

So, already a very loud horray for this fix which completely changes the
usability of network analysis in GRASS !!

Concerning BUG2: Maybe we could think of some test cases ?

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by mmetz):

Replying to [comment:6 mlennert]:
> Replying to [comment:4 mmetz]:
> > dglib cache back to life in trunk r36113 and devbr6 r36114
> >
> > testing is very welcome!
>
> Still need to look at the actual results more in detail (look good at
rough glance). Also ran a few v.net.path's and everything seems fine.

The critical values are the costs returned by Vect_net_shortest_path(). I
did my testing by inserting

fprintf(stdout, "%f\n", cost);

e.g. in v.net.iso/main.c line 254 and v.net.alloc/main.c line 214

then with cache enabled: v.net.iso >costs.cache, similar for cache
disabled

I have added comments in lib/vector/Vlib/net.c explaining how to
disable/enable the cache. The comments and relevant code are in lines 489
- 504. Note that there is no need to comment out line 436.
>
> Concerning BUG2: Maybe we could think of some test cases ?

For BUG1, I created a test vector as described in dglib/BUGS. It should
not be too difficult to create a test vector for BUG2, but unfortunately I
have not the faintest idea where to start poking around in dglib. I really
tried to understand where and how the arcs are inserted, and by now I
think I'm not that bad in understanding source code, but this is beyond
me. You're welcome to get your hands dirty with dglib yourself:-)

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

On Wed, Mar 4, 2009 at 3:09 PM, GRASS GIS <trac@osgeo.org> wrote:

#224: cache bug in DGLib

...

Replying to [comment:4 mmetz]:
> dglib cache back to life in trunk r36113 and devbr6 r36114
>
> testing is very welcome!

Wow !! Just tried v.net.alloc and v.net.iso on streets_wake and schools of
the nc_spm_06 demo data and it finishes in seconds, instead of running the
entire night without coming to a result ! Still need to look at the actual
results more in detail (look good at rough glance). Also ran a few
v.net.path's and everything seems fine.

So, already a very loud horray for this fix which completely changes the
usability of network analysis in GRASS !!

Another WOW from Trento! I added a Spearfish based example to
v.net.alloc (since also the other v.net.* examples are Spearfish based)
and it finished in no time.

Congratulations, MarkusM

Markus

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by neteler):

Since cache is enabled in >= 6.4 (thanks for the important fix), can we
close this ticket?

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: closed
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: fixed | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by mmetz):

  * status: new => closed
  * resolution: => fixed

Comment:

Closing ticket, opening new ticket for BUG2.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/224#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: 6.4.0
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by huhabla):

* cc: huhabla (added)
  * status: closed => reopened
  * version: svn-develbranch6 => 6.4.0
  * resolution: fixed =>

Comment:

It seems to me that this bug still exists in the release version of grass
6.

v.net.path produces wrong results on my 64Bit SUSE 11.2 system in case the
cache is enabled.

You can reproduce the error using the following script within the
spearfish location:

{{{
g.region vect=roads
d.mon start=x0
d.erase
g.copy vect=roads,roads_path --o
v.db.addcol roads_path col="forward double precision, backward double
precision"

# define traveling costs as inverse of speed limit:
v.db.update roads_path col=forward val=1/50
v.db.update roads_path col=backward val=1/50
v.db.update roads_path col=forward val=1/75 where="label='interstate'"
v.db.update roads_path col=backward val=1/75 where="label='interstate'"
v.db.update roads_path col=forward val=1/5 where="label='unimproved road'"
v.db.update roads_path col=backward val=1/5 where="label='unimproved
road'"
v.db.update roads_path col=forward val=1/25 where="label='light-duty road,
improved surface'"
v.db.update roads_path col=backward val=1/25 where="label='light-duty
road, improved surface'"
v.db.select roads_path

# cat x y x y
echo "1 601241.63900415 4914234.39834025 592750.64315353 4925991.16182573"
> /tmp/input.txt
echo "2 601241.63900415 4914234.39834025 595808.58921162 4926703.69294606"
>> /tmp/input.txt
echo "3 601241.63900415 4914234.39834025 604507.406639 4915125.06224066"
>> /tmp/input.txt
echo "4 601241.63900415 4914234.39834025 591206.82572614 4919103.36099585"
>> /tmp/input.txt
echo "5 601241.63900415 4914234.39834025 590791.18257261 4926614.62655602"
>> /tmp/input.txt
echo "6 601241.63900415 4914234.39834025 591088.07053942 4927119.33609959"
>> /tmp/input.txt
echo "7 601241.63900415 4914234.39834025 596194.54356846 4917322.03319502"
>> /tmp/input.txt
echo "8 601241.63900415 4914234.39834025 600172.84232365 4925575.5186722"
>> /tmp/input.txt
echo "9 601241.63900415 4914234.39834025 605041.80497925 4920736.24481328"
>> /tmp/input.txt
echo "10 601241.63900415 4914234.39834025 599074.35684647
4914946.92946058" >> /tmp/input.txt
echo "11 601241.63900415 4914234.39834025 595452.32365145
4920676.86721992" >> /tmp/input.txt
echo "12 601241.63900415 4914234.39834025 608277.88381743
4924269.21161826" >> /tmp/input.txt

v.net.path roads_path afcol=forward abcol=backward out=mypath type=line
dmax=50 file=/tmp/input.txt --o

d.vect map=roads_path@user1 color=255:0:0 lcolor=0:0:0 fcolor=170:170:170
display=shape type=point,line,boundary,area icon=basic/circle size=5
width=4 layer=1 lsize=8 xref=left yref=center llayer=1

d.vect map=mypath@user1 color=0:0:0 lcolor=0:0:0 fcolor=170:170:170
display=shape type=point,line,boundary,area icon=basic/circle size=5
width=2 layer=1 lsize=8 xref=left yref=center llayer=1

}}}

Screenshot is attached.

Disabling the caching mechanism in Vlib/net.c solves the problem:

{{{
grass64_release/lib/vector> svn diff
Index: Vlib/net.c

--- Vlib/net.c (Revision 43825)
+++ Vlib/net.c (Arbeitskopie)
@@ -486,22 +486,22 @@

      pclip = NULL;
      if (List != NULL) {
- nRet =
+ /*nRet =
             dglShortestPath(&(Map->graph), &pSPReport, (dglInt32_t) from,
                             (dglInt32_t) to, clipper, pclip,
&(Map->spCache));
- /* comment out above and uncomment below to debug dglib cache */
- /* nRet =
+ comment out above and uncomment below to debug dglib cache */
+ nRet =
             dglShortestPath(&(Map->graph), &pSPReport, (dglInt32_t) from,
- (dglInt32_t) to, clipper, pclip, NULL); */
+ (dglInt32_t) to, clipper, pclip, NULL);
      }
      else {
- nRet =
+ /*nRet =
             dglShortestDistance(&(Map->graph), &nDistance, (dglInt32_t)
from,
                                 (dglInt32_t) to, clipper, pclip,
&(Map->spCache));
- /* comment out above and uncomment below to debug dglib cache */
- /* nRet =
+ comment out above and uncomment below to debug dglib cache */
+ nRet =
             dglShortestDistance(&(Map->graph), &nDistance, (dglInt32_t)
from,
- (dglInt32_t) to, clipper, pclip, NULL); */
+ (dglInt32_t) to, clipper, pclip, NULL);
      }

      if (nRet == 0) {

}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: vector, dglib, cache
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by hamish):

  * milestone: 6.4.0 => 6.4.1

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: dglib, cache, v.net.path
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by hamish):

  * keywords: vector, dglib, cache => dglib, cache, v.net.path

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: dglib, cache, v.net.path
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------

Comment(by mmetz):

Replying to [comment:10 huhabla]:
> It seems to me that this bug still exists in the release version of
grass 6.
>
> v.net.path produces wrong results on my 64Bit SUSE 11.2 system in case
the cache is enabled.
>
I could reproduce the wrong results and have disabled the dglib cache in
6.4.1. There is a new fix for the dglib cache in trunk and devbr6 (r43857
and r43860, respectively), plus some added documentation about how (I
think that) the cache works.

Markus M

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/224#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: dglib, cache, v.net.path
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------

Comment(by huhabla):

Thanks a lot for this patch and the documentation Martin.
It works for me with trunk and devbr6 using the speafish dataset.

I will test it again with a huge dataset to see if it's stable.

Can the changes be back ported to milestone 6.4.1?

Soeren

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/224#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: dglib, cache, v.net.path
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------

Comment(by mmetz):

Replying to [comment:14 huhabla]:
>
> I will test it again with a huge dataset to see if it's stable.
>
> Can the changes be back ported to milestone 6.4.1?
>
After some more testing, yes. It would be great to have fast vector
network analysis tools in 6.4.1 that produce accurate results.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:15&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: dglib, cache, v.net.path
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------

Comment(by huhabla):

I have tested v.net.path using a dataset with 460000 lines, one start
point and 100 end points. It works quite fast (<5min) and the result seems
to be correct. Screenshot is attached. IMHO back porting to 6.4.1 would be
greatly appreciated.

Soeren

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:16&gt;
GRASS GIS <http://grass.osgeo.org>

#224: cache bug in DGLib
----------------------+-----------------------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: major | Milestone: 6.4.1
Component: Vector | Version: 6.4.0
Resolution: | Keywords: dglib, cache, v.net.path
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------

Comment(by mmetz):

Replying to [comment:16 huhabla]:
> I have tested v.net.path using a dataset with 460000 lines, one start
point and 100 end points. It works quite fast (<5min) and the result seems
to be correct. Screenshot is attached. IMHO back porting to 6.4.1 would be
greatly appreciated.

Thanks for testing!

Would it be asked too much to do the same test with the cache disabled and
compare results? I am aware that this might take some time, e.g. running
overnight.

Also, with cache enabled it would be great if you could design a test
where you recover a path to a given point and in the very next step
recover a path from the same starting point but the end point must be
exactly one line (edge) farther away from the previous end point. First,
this second end point must be reachable, second, the shortest path to this
second end point should go through the previous end point, unless there is
a shorter path avoiding the previous end point. I have done that with a
small test vector that should represent all sorts of special cases
targeting BUG1 and BUG2 but I might have overlooked something.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/224#comment:17&gt;
GRASS GIS <http://grass.osgeo.org>