I'm running a few instances of GRASS, and I've hit an extreme bottleneck. The problem is that I can't figure out what it is.
I have a module that I wrote in the GRASS 7.0 environment that should be I/O bound. It normally takes about 10 minutes to finish running. I have seven instances running from one disk, and each one is taking on the order of 10 hours to finish. Why?
NOTE:
- The processes aren't I/O bound. 'iostat' lists a constant 30 MB/s transfer rate. It should be able to handle twice that. The transfers per second are < 100, and I've seen it handle triple that (at 60 MB/s).
- The load average is currently below 1. I have 16 processors, so CPU shouldn't be a problem.
- I have 16 GB of RAM, and it's full, but not thrashing.
If it's not I/O, CPU, or RAM, then what is it slowing GRASS?
Thanks,
Seth
Hello,
the reasons for slow performance can be different. If performance
concerns You, You should perform Your module run profiling.
Take a look at http://valgrind.org/info/tools.html
From Your description is hard to tell where's the problem - as it
might be a disk write - those are more expensive than reads; it could
be inefficient memory allocation, cache misses etc. Also 16 CPU's
don't change a thing, as most of GRASS code is single threaded -
unless You run Your module in parallel (data parallelization), You
will use only one CPU/core. Ensure that You don't hit swap, as it will
bring any fast system down on it's knees.
Not helpful,
Maris.
2012/2/28 Seth Price <seth@pricepages.org>:
I'm running a few instances of GRASS, and I've hit an extreme bottleneck. The problem is that I can't figure out what it is.
I have a module that I wrote in the GRASS 7.0 environment that should be I/O bound. It normally takes about 10 minutes to finish running. I have seven instances running from one disk, and each one is taking on the order of 10 hours to finish. Why?
NOTE:
- The processes aren't I/O bound. 'iostat' lists a constant 30 MB/s transfer rate. It should be able to handle twice that. The transfers per second are < 100, and I've seen it handle triple that (at 60 MB/s).
- The load average is currently below 1. I have 16 processors, so CPU shouldn't be a problem.
- I have 16 GB of RAM, and it's full, but not thrashing.
If it's not I/O, CPU, or RAM, then what is it slowing GRASS?
Thanks,
Seth_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
I'm running a few instances of GRASS, and I've hit an extreme bottleneck. The problem is that I can't figure out what it is.
I have a module that I wrote in the GRASS 7.0 environment that should be I/O bound. It normally takes about 10 minutes to finish running. I have seven instances running from one disk, and each one is taking on the order of 10 hours to finish. Why?
NOTE:
- The processes aren't I/O bound. 'iostat' lists a constant 30 MB/s transfer rate. It should be able to handle twice that. The transfers per second are < 100, and I've seen it handle triple that (at 60 MB/s).
- The load average is currently below 1. I have 16 processors, so CPU shouldn't be a problem.
- I have 16 GB of RAM, and it's full, but not thrashing.
Since you have plenty of RAM, the Linux system might buffer a lot of
the module input/output in the main memory. So the bottleneck may
appear when Linux tries to write the buffered output to the disc.
Does the load value not change in the 10h of processing? Is the
transfer rate constant over the 10h?
My guess is that when your module reads and writes permanently to the
file system it is IO bound and your hard disk is the limiting factor
not the GRASS (raster, vector or voxel) IO implementation.
I assume the seven instances writing to different maps?
For comparison: I have a 24 Core + 64Gig Ram number cruncher, running
24 parallel processes of v.surf.idw for interpolation purpose. HD's
are 4x256GB SSD's in a fast raid configuration. Max write speed i have
measured is about 400MByte/s with large files. This scales very well
without any performance issues with 24 processes which are not IO
bound.
Best regards
Soeren
If it's not I/O, CPU, or RAM, then what is it slowing GRASS?
Thanks,
Seth_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user