Glynn:
> But, if the nature of the algorithm is such that processing time
> becomes a problem long before memory does, then there's no point to
> using any form of segmentation.
as the module gets faster, the size of map you attempt to run it on
gets bigger and memory does start to become an issue...
FWIW the biggest actual-data region I've tried with r.surf.contour was
21000x13000 (this was a long while back, so god knows how long it took to
finish, could have been a week..*). If it stores it in memory in the input
map's CELL_TYPE, that'll take ~ 2.1gb RAM by my numbers (DCELL). If stored
in memory as CELL, half that. I can accept that much, but it is pushing on
needing a 'percent of map to keep in memory' option.
[*] the great inefficiency in the algorithm is that it is Very slow in
areas where it has to search a long way to find non-NULL cells. If you
can minimize NULL cell areas (one way or another) it can be quite quick,
even for big regions. For my big job ISTR that I set ocean to -1 elevation
instead of NULL and had quite rugged terrain so contour density was thick.
Markus Metz:
FWIW, I found that r.watershed.seg is fastest with the equivalent of
cseg_open(200, 200, <variable number of open segments>). It seems
logical to use powers of 2, but 200, 200 is in r.watershed.seg faster
than 128, 128 and 256, 256. 512, 512 is much slower again, there is a
performance peak somewhere around 200, 200 for r.watershed.seg on my
system. The segment library has strange dynamics, no idea what determines
these dynamics, especially this performance peak.
yesterday I added a command line option for segment size & number, and
ran some trials:
SEG_SIZES="16 32 60 64 100 120 128 150 200 250 256 300 432 500 512 1000 1028 2000 2024"
NUM_SEGS="4 8 16 25 32 50 64 100 128 256"
#spearfish
g.region rast=elevation.10m
#rows: 1398
#cols: 1899
r.contour input="elevation.10m" output="elev_contours_10m" step=10
v.to.rast in=elev_contours_10m out=elev_contours.10m use=z
for ... ; do
echo "$segsize x $nsegs :"
time r.surf.contour ...
done
512x512 segments x16 came out best. but basically there were all pretty
similar as long as they were not so small as to be choking the system.
(as original 16x16x8 was). 512x512x16 uses about 19mb of RAM for the proc.
I would guess that the best tuning is somewhat dependent on the size
and shape of the input data.
Glynn:
> I'll remove the segmentation code in 7.0.
MarkusN tests:
[6.5prev: 171.8 sec total usr+sys;
6.5curr: 118.5 sec;
7.0curr: 72.2 sec]
nice!
MarkusN on Linux (old):
147.66 user
24.14 system
Michael on Mac (old):
user 2m00.088s
sys 1m42.810s
Interesting that OSX spends so much more time in the kernel than Linux.
So the improvement will have twice the impact for Macs.
Glynn:
> How does r.univar come into this?
Markus:
Forgot to mention that I used r.mapcalc to calculate the differences
between the 6.5 and 7 results.
I did the same in my tests knowing that it /shouldn't/ have an effect.
If nothing else it's a good fail-safe sanity check. e.g. NULL support
in 6.5 is currently borken, if r.surf.contour outputted maps containing
NULLs it probably would have picked that up.
test-suites are good.
Hamish
