[GRASS-user] memory requirement of v.net.allpairs module (Markus Metz)

Pankaj_Kr_Sharma · December 30, 2011, 3:56pm

Dear users and developers,

v.net.allpairs return +ve distance for points in the backward direction of the vector and -ve value for points lying in the forward direction of the vector.
(I have posted a similar script earlier for this purpose. Which was written in bash script format.)
One can use these categories to sum up the areas (r.stream.basins) and calculate the cumulative area without creating individual raster maps ( thus, using hard disk space) for each point.
This module is pretty fast in calculating the distance.
However, from the moment, when it writes the values in database (sequential write , perhaps) , it’s a slow process.
In the reply , it was mentioned that all the information (from cat, to cat, distance ) is held in a temporary file. (reason for large swap space being used 160 gb)
Can’t we have a text file (from this temp file) instead of sequential database write for large vector sets. (for example, as those created by r.stats .)
The text file can be copied to database in few seconds. Whereas, this can take six month (145,000 * 145,000).
I have run it on smaller region and on an vector output of 11000 points (i.e. 11,000 * 11,000) ,it took approximately 12 hours to finish.
Can we have opinion of Dear Daniel Bundala ( the module developer ) on this.

Thanks.

On Wed, Dec 28, 2011 at 5:46 PM, <grass-user-request@lists.osgeo.org> wrote:

Send grass-user mailing list submissions to
grass-user@lists.osgeo.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.osgeo.org/mailman/listinfo/grass-user
or, via email, send a message with subject or body ‘help’ to
grass-user-request@lists.osgeo.org

You can reach the person managing the list at
grass-user-owner@lists.osgeo.org

When replying, please edit your Subject line so it is more specific
than “Re: Contents of grass-user digest…”

Today’s Topics:

Re: memory requirement of v.net.allpairs module (Markus Metz)

Compile and install 6.4.2RC2 (Aldo Clerici)

Re: Compile and install 6.4.2RC2 (Martin Landa)

Message: 1
Date: Tue, 27 Dec 2011 18:53:20 +0100
From: Markus Metz <markus.metz.giswork@googlemail.com>
Subject: Re: [GRASS-user] memory requirement of v.net.allpairs module
To: Pankaj Kr Sharma <pkscwc@gmail.com>
Cc: grass-user@lists.osgeo.org, grass-dev@lists.osgeo.org
Message-ID:
<CAG+h=FGPOU61d45cQVw0UczvGWGJmVLmYYtqaaFPiD7mK-iKhw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Dec 25, 2011 at 5:04 PM, Pankaj Kr Sharma <pkscwc@gmail.com> wrote:

Dear Developers and users,

On a small test region (11000x11000 cells), with a threshold of 1000sqkm,
r.stream.extract resulted in a stream vector network of 190 points and
lines. The v.net.allpairs generated a table of distances between each point
within 10 seconds on my home machine (only 4gb ram). I thought that I got
my solution for finding sub-basins of a very large area. (Please refer to my
earlier posts in this regard.)

On my work setup, (better machine, 16gb RAM , six cores), I followed this
routine:

r.grow by one cell on area of interest. (It’s an important step. Thanks
to Markus Metz for this tip.) Time taken by the module: approximately 2
hours.

r.stream.extract on my area of interest.(42000*42000 cells) with 100
sq.km. as threshold (12345 in number of cells.)

Above you said that yourtest region was 11000x11000 cells, not
42000x42000 cells. Both r.grow and r.stream.extract will be
considerably faster if you adjust your computational region to your
test region.

Please note that the r.stream.extract module’s vector output was a
network of 1,45,000 (approx) points and lines.
Time taken by module: approximately 24 hours.

Then I ran “v.net.allpairs” on this network. Now, v.net.allpairs is still
running. Approximately, one week has elapsed.
After three days, a message appeared on the console that says “writing data
into the table”. After four days of this message, the progress bar is still
without any colour. ( Which I consider as progress being less than one
percent.)

The other thing which I noticed was the memory consumption. The module has
consumed 15.5 gb ram out of 16gb, thereafter it consumed 166gb of swap space
out of 500gb swap space available.( Earlier, I had kept complete one disk of
1 tb for swap.)

This sounds like bad news. v.net.allpairs keeps all categories,
all categories, and all distances from to in memory.
For 145000 network nodes, this would be about 320 GB. Assuming that
half of the features are lines and half of the features are points,
v.net.allpairs would require about 160GB of memory, plus some overhead
to handle the vector if you did not set the environment variable
GRASS_VECTOR_LOWMEM. Granted that you did not run v.net.allpairs with
the -a flag.

Why did you use v.net.allpairs at all on the stream network?

Markus M

Some of the answers for this response of the module, I have myself.

Like, the module has more calculations to do.
(145000145000)/(190190)~582410.
Still, as per my calculation of time required 58241010 seconds/(2460*60)~
6.75 days

And, I should have my postgres table ready for further calculations.
But, still there is no sign of any progress.
Is the time and memory being consumed by the module as per design and ok.

I can definitely sub-divide into basins and work further.

grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Message: 2
Date: Wed, 28 Dec 2011 12:46:17 +0100
From: “Aldo Clerici” <aldo.clerici@unipr.it>
Subject: [GRASS-user] Compile and install 6.4.2RC2
To: <grass-user@lists.osgeo.org>
Message-ID: <045901ccc556$4f6c51e0$ee44f5a0$@clerici@unipr.it>
Content-Type: text/plain; charset=“us-ascii”

Dear Developers and users,

I’m having troubles in compiling and installing grass6.4.2RC2 on Fedora 10.

I installed Python-2.7.2, wxPython-2.9.2.4, numy-1.6.1.

I run: ./configure --with-cxx --with-python=/usr/local/bin/python2.7-config
–with-wxwidgets=/usr/local/bin/wx-config

with the following output:

GRASS is now configured for: i686-pc-linux-gnu

Source directory: /home/clerici/GRASSprograms/grass-6.4.2RC2

Build directory: /home/clerici/GRASSprograms/grass-6.4.2RC2

Installation directory: ${prefix}/grass-6.4.2RC2

Startup script in directory: ${exec_prefix}/bin

C compiler: gcc -g -O2

C++ compiler: c++ -g -O2

Building shared libraries: yes

64bit support: no

OpenGL platform: X11

MacOSX application: no

MacOSX architectures:

MacOSX SDK:

Tcl/Tk NVIZ: yes

BLAS support: no

C++ support: yes

Cairo support: no

DWG support: no

FFMPEG support: no

FFTW support: yes

FreeType support: no

GDAL support: yes

GEOS support: no

GLw support: no

JPEG support: yes

LAPACK support: no

Large File support (LFS): no

Motif support: no

MySQL support: no

NLS support: no

ODBC support: no

OGR support: yes

OpenGL support: yes

PNG support: yes

PostgreSQL support: no

Python support: yes

Readline support: no

SQLite support: no

Tcl/Tk support: yes

wxWidgets support: yes

TIFF support: yes

X11 support: yes

Then: make

GRASS GIS compilation log

Started compilation: Wed Dec 28 12:22:59 CET 2011

–

Errors in:

No errors detected.

–

Finished compilation: Wed Dec 28 12:24:11 CET 2011

And su -c ‘make install’ that seems ok.

However when I run: grass64 -wxpython I have the following error message:

Cleaning up temporary files …

Starting GRASS …

ERROR: wxGUI requires wxPython. No module named wxversion

Error in GUI startup. If necessary, please

report this error to the GRASS developers.

Switching to text mode now.

Hit RETURN to continue…

Where did I go wrong?

Many thanks in advance.

A. Clerici

Parma University

-------------- next part --------------
An HTML attachment was scrubbed…
URL: http://lists.osgeo.org/pipermail/grass-user/attachments/20111228/d8363dca/attachment-0001.html

Message: 3
Date: Wed, 28 Dec 2011 13:16:29 +0100
From: Martin Landa <landa.martin@gmail.com>
Subject: Re: [GRASS-user] Compile and install 6.4.2RC2
To: Aldo Clerici <aldo.clerici@unipr.it>
Cc: grass-user@lists.osgeo.org
Message-ID:
<CA+Ei1Ocvg1hBsjQNt_bPA7ys2pzWiDWDBFRuuo3KD-WBXZg2Aw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

2011/12/28 Aldo Clerici <aldo.clerici@unipr.it>:

ERROR: wxGUI requires wxPython. No module named wxversion

Error in GUI startup. If necessary, please

report this error to the GRASS developers.

Switching to text mode now.

Hit RETURN to continue…

Where did I go wrong?

you need to install package which contains wxversion. On Debian
GNU/Linux it’s eg. python-wxversion.

Martin

–
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

End of grass-user Digest, Vol 68, Issue 48