Hamish,
On Sun, Feb 24, 2013 at 1:41 AM, Hamish <hamish_b@yahoo.com> wrote:
Doug wrote:
Compiled with openmp and liblas?
fwiw I would not really recommend to compile with OpenMP support
unless you are willing to help develop that. Most of it is fine I
think, but the LU parts used by v.surf.rst and v.surf.bspline are
not very efficient, overhead is like 70%. So you do get a little speedup verus single core, but use a lot of power to do it.
(currently that spawns 10s of thousands of threads when it would be better to spawn nearer to the number of local CPU cores)
I’ve been playing with generating surfaces at home on an 8 core AMD vishera system 3.5 ghz Ubuntu 12.04 system with G7 using this point layer:
±---------------------------------------------------------------------------+
| Name: nc_2a |
| Mapset: PERMANENT |
| Location: nc_stpft |
| Database: /data2/grass70_data |
| Title: |
| Map scale: 1:1 |
| Name of creator: dnewcomb |
| Organization: |
| Source date: Sat Feb 23 08:16:54 2013 |
| Timestamp (first layer): none |
| Map format: native |
| ---------------------------------------------------------------------------- |
| Type of map: vector (level: 1) |
|
| Number of points: 73232338 Number of centroids: 0 |
| Number of lines: 0 Number of boundaries: 0 |
| Number of areas: 0 Number of islands: 0 |
| Number of faces: 0 Number of kernels: 0 |
| Number of volumes: 0 Number of holes: 0 |
|
| Map is 3D: Yes |
| Number of dblinks: 0 |
|
| Projection: Lambert Conformal Conic |
|
| N: 694901.76 S: 482704.72 |
| E: 580721.99 W: 492486.01 |
| B: 1083.51 T: 5558.44 |
|
| Digitization threshold: 0 |
| Comment: |
|
| ±---------------------------------------------------------------------------+ |
| (Sat Feb 23 09:49:39 2013) Command finished (34 sec) |
I get the following 20ft resolution surface using v.usrf.bspline
r.info map=nc_2_bspline_cubic@PERMANENT
±---------------------------------------------------------------------------+
| Layer: nc_2_bspline_cubic@PERMANENT Date: Sat Feb 23 19:13:29 2013 |
| Mapset: PERMANENT Login of Creator: dnewcomb |
| Location: nc_stpft |
| DataBase: /data2/grass70_data |
| Title: bicubic interpolation with Tykhonov regularization ( nc_2_bsplin |
| Timestamp: none |
|
| Type of Map: raster Number of Categories: 0 |
| Data Type: DCELL |
| Rows: 10568 |
| Columns: 4412 |
| Total Cells: 46626016 |
| Projection: Lambert Conformal Conic |
| N: 693610 S: 482260 Res: 19.99905375 |
| E: 580722 W: 492486 Res: 19.99909338 |
| Range of data: min = 92.2649807704968 max = 5558.35850509857 |
|
| Data Description: |
| generated by v.surf.bspline |
|
| Comments: |
| v.surf.bspline -z input=“nc_2a@PERMANENT” layer=“1” raster_output="n\ |
| c_2_bspline_cubic" sie=40.0 sin=40.0 method=“bicubic” lambda_i=0.01 \ |
| solver=“cholesky” maxit=200 error=0.000001 memory=3000 |
|
| ±---------------------------------------------------------------------------+ |
| (Sun Feb 24 19:53:17 2013) Command finished (0 sec) |
subregion 920 of 920…
v.surf.bspline complete.
(Sun Feb 24 08:39:12 2013) Command finished (805 min 43 sec)
As you can see , the surface generation ran in 805 minutes. I’m currently rerunning the analysis with maxit = 100 to see how long it takes with the iterations lowered.
Now with v.surf.rst, I generated the following grid:
---------------------------------------------------------------------------+
| Layer: nc_2_rst_npmin_120@PERMANENT Date: Sat Feb 23 01:14:43 2013 |
| Mapset: PERMANENT Login of Creator: dnewcomb |
| Location: nc_stpft |
| DataBase: /data2/grass70_data |
| Title: ( nc_2_rst_npmin_120 ) |
| Timestamp: none |
|
| Type of Map: raster Number of Categories: 0 |
| Data Type: FCELL |
| Rows: 10568 |
| Columns: 4412 |
| Total Cells: 46626016 |
| Projection: Lambert Conformal Conic |
| N: 693610 S: 482260 Res: 19.99905375 |
| E: 580722 W: 492486 Res: 19.99909338 |
| Range of data: min = -1.409014e+28 max = 7.890661e+27 |
|
| Data Source: |
| vector map nc_2@PERMANENT |
|
|
| Data Description: |
| generated by v.surf.rst |
|
| Comments: |
| tension=40.000000, smoothing=0.100000 |
| dnorm=215.219236, dmin=9.999527, zmult=1.000000 |
| segmax=40, npmin=120, rmsdevi=29786443112213122633433088.000000 |
| zmin_data=1083.510000, zmax_data=9999.000000 |
| zmin_int=-14090138501368785958609092608.000000, zmax_int=78906611338755 |
|
| v.surf.rst input=“nc_2@PERMANENT” layer=“1” elevation="nc_2_rst_npmi\ |
| n_120" tension=40. segmax=40 npmin=120 dmin=9.999527 dmax=49.997634 \ |
| zmult=1.0 |
|
| ±---------------------------------------------------------------------------+ |
| (Sat Feb 23 08:10:21 2013) Command finished (0 sec) |
v.surf.rst complete.
(Sat Feb 23 17:29:55 2013) Command finished (458 min 33 sec)
As you can see, it took only 458 minutes to generate the same resolution surface from the same input points. On the 8 core system, it is using 650% of the cpu to process the data, bouncing up and down 30% or so as it processes.
v.surf.bspline, on the other hand parses the data up into zones and uses 100% cpu up to the time it writes the individual zone out. When it writes the zone out, it spikes up to 750% cpu. For this vector set and grid extent, it generated 920 zones.
V.surf.rst seems to use much more memory than v.surf.bspline
I am of course, speaking as an end user who has not looked at the source code. ( I’m trying to back myself into learning C :-))
For that matter I would not recommend that new users be
encouraged to use development versions at all, unless they are
willing to help develop it, or at least debug it. And if
development nightly builds are made available, it should be made
very clear that it could eat all your data, harm your dog, etc.
Agreed , and I plan on giving the output to other folks to check against the official state grid and the QA points. If this is inefficient, I would really like to try it out after it was made more efficient 
Thanks for all the work!
Doug
best,
Hamish
–
Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb@fws.gov
The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.