On Sun, Dec 25, 2011 at 5:04 PM, Pankaj Kr Sharma <pkscwc@gmail.com> wrote:
Dear Developers and users,
On a small test region (11000x11000 cells), with a threshold of 1000sqkm,
r.stream.extract resulted in a stream vector network of 190 points and
lines. The v.net.allpairs generated a table of distances between each point
within 10 seconds on my home machine (only 4gb ram). I thought that I got
my solution for finding sub-basins of a very large area. (Please refer to my
earlier posts in this regard.)On my work setup, (better machine, 16gb RAM , six cores), I followed this
routine:
1. r.grow by one cell on area of interest. (It's an important step. Thanks
to Markus Metz for this tip.) Time taken by the module: approximately 2
hours.2. r.stream.extract on my area of interest.(42000*42000 cells) with 100
sq.km. as threshold (12345 in number of cells.)
Above you said that yourtest region was 11000x11000 cells, not
42000x42000 cells. Both r.grow and r.stream.extract will be
considerably faster if you adjust your computational region to your
test region.
Please note that the r\.stream\.extract module's vector output was anetwork of 1,45,000 (approx) points and lines.
Time taken by module: approximately 24 hours.3. Then I ran "v.net.allpairs" on this network. Now, v.net.allpairs is still
running. Approximately, one week has elapsed.
After three days, a message appeared on the console that says "writing data
into the table". After four days of this message, the progress bar is still
without any colour. ( Which I consider as progress being less than one
percent.)The other thing which I noticed was the memory consumption. The module has
consumed 15.5 gb ram out of 16gb, thereafter it consumed 166gb of swap space
out of 500gb swap space available.( Earlier, I had kept complete one disk of
1 tb for swap.)
This sounds like bad news. v.net.allpairs keeps all <from> categories,
all <to> categories, and all distances from <from> to <to> in memory.
For 145000 network nodes, this would be about 320 GB. Assuming that
half of the features are lines and half of the features are points,
v.net.allpairs would require about 160GB of memory, plus some overhead
to handle the vector if you did not set the environment variable
GRASS_VECTOR_LOWMEM. Granted that you did not run v.net.allpairs with
the -a flag.
Why did you use v.net.allpairs at all on the stream network?
Markus M
Some of the answers for this response of the module, I have myself.
Like, the module has more calculations to do.
(145000*145000)/(190*190)~582410.
Still, as per my calculation of time required 582410*10 seconds/(24*60*60)~
6.75 daysAnd, I should have my postgres table ready for further calculations.
But, still there is no sign of any progress.
Is the time and memory being consumed by the module as per design and ok.I can definitely sub-divide into basins and work further.
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user