Michael P wrote:
I'm running Mac OS X + SQLite with a 64 bit version of
grass. I ran a whole bunch of tests today and it looks like
there are at least a couple of things going on and SQLite is
one of the issues. I ran a 500m x500m tile with three
different DB back-ends (SQLite, Postgres and DBF) and these
are the results I obtained;
(SQLite)
time v.lidar.edgedetection input=Cal_QTile
output=Cal_QTile_edges
real 38m53.458s
user 1m17.602s
sys 4m6.353s(Postgres)
time v.lidar.edgedetection input=Cal_QTile
output=Cal_QTile_edges
real 6m49.060s
user 0m46.622s
sys 1m20.324s(DBF)
time v.lidar.edgedetection input=Cal_QTile
output=Cal_QTile_edges
real 1m54.065s
user 0m48.530s
sys 1m8.686sThe results with Postgres and SQLite were a real surprise
to me.
I am surprised that all DBs spend so much time in "sys" kernel
land. (a good guess there is the time is spent on I/O with disk
controller) I knew that linux's IO was cleaner than OSX's,
but I didn't know the margin was that much ...
I just ran a test in 64bit linux got the following:
DBF:
real 32m14.192s
user 32m6.880s
sys 0m5.884s
Sqlite:
real 39m45.760s
user 34m25.357s
sys 0m23.225s
I would note that the number of passes it takes to complete
varies between ~ 9 and 20 for my test data depending on
settings, and that affects the time to complete hugely.
(1 pass per subregion, but it skips some number of no-points
subregions)
By watching "top" and gkrellm monitor* I can see that the drop
down to single digit processor use for sqlite happens in the
"Point classification" step, and gkrellm shows that during this
time the disk write is about 22mb/sec, so lets say it is limited
by writing to the disk. After it finishes the point class. step
it jumps back up to 100% for the next pass's interpolations.
(dominated by Tcholetsky decomposition function) Also subsequent
passes often seem to have many fewer points to classify, so
are faster to complete that stage.
[*] something similar to gkrellm can be found in the system
monitor app in the OSX Apps/Utility folder
So DBF is faster to write down a zillion new records to disk.
not all that surprising seeing that it has so little overhead
to worry about.
Hamish:
> if you build from the latest grass 6.5 or 7 svn you
> can use the --verbose flag to follow the action. setting
> 'g.gisenv set="DEBUG=1" gives you even more detail (even
> with existing versions).Does this print out the debugging messages that are in the
code
yes, you just need to switch them on and then you can see what's
happening. already present in all 6.4.0 RC versions.
set it back to "0" to turn debug messages off again.
or do I need to do something else to see those?
in the devel versions (6.5 and 7) I've moved the milepost
debug messages into verbose messages, so you just need to
add --verbose to the command line to see them. percent complete
too. It's a lot cleaner than the debug messages, but you need to
recompile from source to get this.
Hamish