[GRASS-user] v.surf.idw G_Malloc

Matthew_James · January 14, 2008, 4:48pm

Hi Everyone,

New grass user J I’m using Grass 6.2.3 on windows using cygwin.

I am trying to generate yield maps. I have csv files containing x, y and yield columns.

The yields are calculated at random spots around the paddock.

In general these files can be many thousands of points long.

I’m first importing them into a vector file using v.in.ascii (seems to me that the process of importing such files needs to be documented better – it’s hellish for a new user)

When I try to run v.surf.idw I get a “G_Malloc – out of memory error” – this occurs almost instantaneously.

I’m not at all short on resources (have about 2gb of RAM and a huge pagefile) and I’m wondering if there is some way around this problem.

Thankyou,

Matt

Internal Virus Database is out-of-date.
Checked by AVG Free Edition.
Version: 7.5.503 / Virus Database: 269.15.12/1097 - Release Date: 28/10/2007 1:58 PM

Paul_Kelly · January 14, 2008, 4:59pm

On Tue, 15 Jan 2008, Matthew James wrote:

Hi Everyone,

New grass user J I?m using Grass 6.2.3 on windows using cygwin.

I am trying to generate yield maps. I have csv files containing x, y and
yield columns.

The yields are calculated at random spots around the paddock.

In general these files can be many thousands of points long.

I?m first importing them into a vector file using v.in.ascii (seems to me
that the process of importing such files needs to be documented better ?
it?s hellish for a new user)

When I try to run v.surf.idw I get a ?G_Malloc ? out of memory error? ? this
occurs almost instantaneously.

Please give the exact command lines you used for v.in.ascii and v.surf.idw as it is a bit hard to see what you are doing otherwise (also run v.surf.idw with --verbose in case it generates any useful messages).

Also are you importing the yield value as a z co-ordinate or a database field? For large numbers of points in general performance will be much much better if you avoid the database connection.

Also not building topology during the import (check the v.in.ascii man page; I can't remember the exact flag to use) will give a big performance boost for a large number of points. Unfortunately the version of v.surf.idw in 6.2.3 requires topology to be built to run - if you upgrade to 6.3 this is not required and you have a much better chance of being successfully able to interpret huge datasets with v.surf.idw.

Paul

Matthew_James · January 14, 2008, 5:55pm

Hi Paul - thanks for the fast reply.

Ok - my import command is -

v.in.ascii input=/home/Matthew/XYGrp45.csv output=vectorOut format=point
fs=, skip=0 'columns=x double precision, y double precision, yield double
precision' x=2 y=1 z=3 cat=0

output of that command is -

Maximum input row length: 28
Maximum number of columns: 3
Minimum number of columns: 3
Building topology ...
Registering lines: 1000 2000 3000 4000
5000 6000 7000 8000 9000 10000
11000 12000 13000 14000 15000 16000
17000 18000 19000 20000 21000 22000
23000 24000 25000 26000 27000 28000
29000 30000 31000 32000 33000 34000
35000 36000 37000 38000 39000 40000
41000 42000 43000 44000 45000 46000
47000 48000 49000 50000 51000 52000
53000 54000 55000 56000 57000 58000
59000 60000 61000 62000 63000 64000
65000 66000 67000 68000 69000 70000
71000 72000 73000 74000 75000 76000
77000 78000 79000 80000 81000 82000
83000 84000 85000 86000 87000 88000
89000 90000 91000 92000 93000 94000
95000 96000 97000 98000 99000 100000
101000 102000 103000 104000 105000 106000
107000 108000 109000 110000 111000 112000
113000 114000 115000 116000 117000 118000
119000 120000 121000 122000 123000 124000
125000 126000 127000 128000 129000 130000
131000 132000 133000 134000 135000 136000
137000 138000 139000 140000 141000 142000
143000 144000 145000 146000 147000
147909 primitives registered
Building areas:

0 areas built
0 isles built
Attaching islands:
Attaching centroids:

Topology was built.
Number of nodes : 147872
Number of primitives: 147909
Number of points : 147909
Number of lines : 0
Number of boundaries: 0
Number of centroids : 0
Number of areas : 0
Number of isles : 0

MY SURF.IDW COMMAND -

v.surf.idw input=vectorOut output=rasterOut npoints=12 column=3

Output - G_Malloc - Out of Memory

Cheers,

Matt

-----Original Message-----
From: Paul Kelly [mailto:paul-grass@stjohnspoint.co.uk]
Sent: Tuesday, 15 January 2008 2:59 AM
To: Matthew James
Cc: grass-user@lists.osgeo.org
Subject: Re: [GRASS-user] v.surf.idw G_Malloc

On Tue, 15 Jan 2008, Matthew James wrote:

Hi Everyone,

New grass user J I?m using Grass 6.2.3 on windows using cygwin.

I am trying to generate yield maps. I have csv files containing x, y and
yield columns.

The yields are calculated at random spots around the paddock.

In general these files can be many thousands of points long.

I?m first importing them into a vector file using v.in.ascii (seems to me
that the process of importing such files needs to be documented better ?
it?s hellish for a new user)

When I try to run v.surf.idw I get a ?G_Malloc ? out of memory error? ?

this

occurs almost instantaneously.

Please give the exact command lines you used for v.in.ascii and v.surf.idw
as it is a bit hard to see what you are doing otherwise (also run
v.surf.idw with --verbose in case it generates any useful messages).

Also are you importing the yield value as a z co-ordinate or a database
field? For large numbers of points in general performance will be much
much better if you avoid the database connection.

Also not building topology during the import (check the v.in.ascii man
page; I can't remember the exact flag to use) will give a big performance
boost for a large number of points. Unfortunately the version of
v.surf.idw in 6.2.3 requires topology to be built to run - if you upgrade
to 6.3 this is not required and you have a much better chance of being
successfully able to interpret huge datasets with v.surf.idw.

Paul

Internal Virus Database is out-of-date.
Checked by AVG Free Edition.
Version: 7.5.503 / Virus Database: 269.15.12/1097 - Release Date: 28/10/2007
1:58 PM

Paul_Kelly · January 14, 2008, 6:11pm

On Tue, 15 Jan 2008, Matthew James wrote:

Hi Paul - thanks for the fast reply.

Ok - my import command is -

v.in.ascii input=/home/Matthew/XYGrp45.csv output=vectorOut format=point
fs=, skip=0 'columns=x double precision, y double precision, yield double
precision' x=2 y=1 z=3 cat=0

output of that command is -

Maximum input row length: 28
Maximum number of columns: 3
Minimum number of columns: 3
Building topology ...
Registering lines: 1000 2000 3000 4000
5000 6000 7000 8000 9000 10000
11000 12000 13000 14000 15000 16000
17000 18000 19000 20000 21000 22000
23000 24000 25000 26000 27000 28000
29000 30000 31000 32000 33000 34000
35000 36000 37000 38000 39000 40000
41000 42000 43000 44000 45000 46000
47000 48000 49000 50000 51000 52000
53000 54000 55000 56000 57000 58000
59000 60000 61000 62000 63000 64000
65000 66000 67000 68000 69000 70000
71000 72000 73000 74000 75000 76000
77000 78000 79000 80000 81000 82000
83000 84000 85000 86000 87000 88000
89000 90000 91000 92000 93000 94000
95000 96000 97000 98000 99000 100000
101000 102000 103000 104000 105000 106000
107000 108000 109000 110000 111000 112000
113000 114000 115000 116000 117000 118000
119000 120000 121000 122000 123000 124000
125000 126000 127000 128000 129000 130000
131000 132000 133000 134000 135000 136000
137000 138000 139000 140000 141000 142000
143000 144000 145000 146000 147000
147909 primitives registered
Building areas:

0 areas built
0 isles built
Attaching islands:
Attaching centroids:

Topology was built.
Number of nodes : 147872
Number of primitives: 147909
Number of points : 147909
Number of lines : 0
Number of boundaries: 0
Number of centroids : 0
Number of areas : 0
Number of isles : 0

MY SURF.IDW COMMAND -

v.surf.idw input=vectorOut output=rasterOut npoints=12 column=3

I think you need column=yield rather than column=3. 150000 points isn't a huge amount and should be easy enough to work with. But for much large datasets it would be easier to work with if you create a 3-D vector by using the -z flag with v.in.ascii (and leave out the columns option). Then you could use layer=0 for v.surf.idw to interpolate using the z co-ordinate.

No idea where the G_malloc error is coming from but hopefully it will go away with the correct column parameter...

Paul

Matthew_James · January 14, 2008, 6:50pm

Hi Paul - Alas that does not work either.

Specific errors with verbose -

2593 [main] ? (1904) c:\cycwin\usr\local\grass-6.2.3\bin\v.surf.idw: ***
fatal error - MEM_COMMIT Failed, Win32 error 1455

Cheers,

Matt

-----Original Message-----
From: Paul Kelly [mailto:paul-grass@stjohnspoint.co.uk]
Sent: Tuesday, 15 January 2008 4:12 AM
To: Matthew James
Cc: grass-user@lists.osgeo.org
Subject: RE: [GRASS-user] v.surf.idw G_Malloc

On Tue, 15 Jan 2008, Matthew James wrote:

Hi Paul - thanks for the fast reply.

Ok - my import command is -

v.in.ascii input=/home/Matthew/XYGrp45.csv output=vectorOut format=point
fs=, skip=0 'columns=x double precision, y double precision, yield double
precision' x=2 y=1 z=3 cat=0

output of that command is -

Maximum input row length: 28
Maximum number of columns: 3
Minimum number of columns: 3
Building topology ...
Registering lines: 1000 2000 3000 4000
5000 6000 7000 8000 9000 10000
11000 12000 13000 14000 15000

16000

17000 18000 19000 20000 21000

22000

23000 24000 25000 26000 27000

28000

29000 30000 31000 32000 33000

34000

35000 36000 37000 38000 39000

40000

41000 42000 43000 44000 45000

46000

47000 48000 49000 50000 51000

52000

53000 54000 55000 56000 57000

58000

59000 60000 61000 62000 63000

64000

65000 66000 67000 68000 69000

70000

71000 72000 73000 74000 75000

76000

77000 78000 79000 80000 81000

82000

83000 84000 85000 86000 87000

88000

89000 90000 91000 92000 93000

94000

95000 96000 97000 98000 99000

100000

101000 102000 103000 104000 105000

106000

107000 108000 109000 110000 111000

112000

113000 114000 115000 116000 117000

118000

119000 120000 121000 122000 123000

124000

125000 126000 127000 128000 129000

130000

131000 132000 133000 134000 135000

136000

137000 138000 139000 140000 141000

142000

143000 144000 145000 146000 147000
147909 primitives registered
Building areas:

0 areas built
0 isles built
Attaching islands:
Attaching centroids:

Topology was built.
Number of nodes : 147872
Number of primitives: 147909
Number of points : 147909
Number of lines : 0
Number of boundaries: 0
Number of centroids : 0
Number of areas : 0
Number of isles : 0

MY SURF.IDW COMMAND -

v.surf.idw input=vectorOut output=rasterOut npoints=12 column=3

I think you need column=yield rather than column=3. 150000 points isn't a
huge amount and should be easy enough to work with. But for much large
datasets it would be easier to work with if you create a 3-D vector by
using the -z flag with v.in.ascii (and leave out the columns option). Then
you could use layer=0 for v.surf.idw to interpolate using the z
co-ordinate.

No idea where the G_malloc error is coming from but hopefully it will go
away with the correct column parameter...

Paul

Internal Virus Database is out-of-date.
Checked by AVG Free Edition.
Version: 7.5.503 / Virus Database: 269.15.12/1097 - Release Date: 28/10/2007
1:58 PM

Hamish · January 15, 2008, 4:04am

Matthew James wrote:

I am trying to generate yield maps. I have csv files containing x, y
and yield columns.

The yields are calculated at random spots around the paddock.

In general these files can be many thousands of points long.

they can be up to a few million points long before you have to resort
to "tricks".

When I try to run v.surf.idw I get a “G_Malloc – out of memory error”
– this occurs almost instantaneously.

I’m not at all short on resources (have about 2gb of RAM and a huge
pagefile) and I’m wondering if there is some way around this problem.

I'd guess your region settings are wrong. You have to constrain the IDW
calculation to what you want the raster output map to be like.

try

g.region vect=PointsMap

to zoom to the extents of your points map. Then

g.region -p

and

g.region res=[something] -a

to set an appropriate resolution.

Anything larger than 5000x5000 rows x columns (g.region -p) is
beginning to get big for a raster map. (GRASS can deal with bigger
regions than that, but it starts to take a long time + disk space)

Hamish

____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs