[GRASS-user] Reading file with with space separators

Dear all

presumably a trivial question - but what is the argument for space when importing data?
I have xyz files with the format
..x..y...value
where <.> stands for white space. The spacing is two spaces between x and y and three spaces to data column.

I used
r.in.xyz fs='space', in=dgm10_32292_5548_2_rp.xyz out=test
and obtain:
ERROR: Bad y-coordinate line 1 column 2. <>

I also tried different options for fs: fs=' ', fs = space, fs = \s (whitespace regular expression) - without success

In R it is no problem - the simple read.table with sep="" for any whitespace works. And with rasterFromXYZ (raster package) I can create a raster without problems.

I guess it is also easy in GRASS but I am just too ignorant :slight_smile:

Cheers
Ralf

Am 17.04.2013 um 10:33 schrieb grass-user-request@lists.osgeo.org:

Send grass-user mailing list submissions to
  grass-user@lists.osgeo.org

To subscribe or unsubscribe via the World Wide Web, visit
  http://lists.osgeo.org/mailman/listinfo/grass-user
or, via email, send a message with subject or body 'help' to
  grass-user-request@lists.osgeo.org

You can reach the person managing the list at
  grass-user-owner@lists.osgeo.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of grass-user digest..."

Today's Topics:

  1. Re: installing add-ons for GRASS 6.4.2 on Ubuntu fails
     (G. Allegri)
  2. Re: Using r.quantile result with r.recode (Glynn Clements)
  3. Re: installing add-ons for GRASS 6.4.2 on Ubuntu fails (Hamish)
  4. Re: installing add-ons for GRASS 6.4.2 on Ubuntu fails
     (G. Allegri)
  5. Problem with r.univar and very large values (Johannes Radinger)
  6. Re: Using r.quantile result with r.recode (Pedro Ven?ncio)

----------------------------------------------------------------------

Message: 1
Date: Tue, 16 Apr 2013 21:44:02 +0200
From: "G. Allegri" <giohappy@gmail.com>
To: Markus Neteler <neteler@osgeo.org>
Cc: GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] installing add-ons for GRASS 6.4.2 on Ubuntu
  fails
Message-ID:
  <CAB4g1=y3QQpMakxcbVrekZZRJRzoVbuC4tPgbYeGuKeShchrsg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I've compiled GRASS 6.4.3RC2.
The add-on installation exits with the following error:

/usr/bin/install: impossibile eseguire stat di "/home/giova/
Data/GRASSDB/3003/PERMANENT/.tmp/xubuntu/16152.0/v.mkhexgrid
/man/man1/v.mkhexgrid.1": File o directory non esistente
make: *** [install] Errore 1
WARNING: Installation failed, sorry. Please check above error messages.

In english it's something like

/usr/bin/install: cannot execute stat "/home/giova/
Data/GRASSDB/3003/PERMANENT/.tmp/xubuntu/16152.0/v.mkhexgrid
/man/man1/v.mkhexgrid.1": File or folder not existing
make: *** [install] Error 1
WARNING: Installation failed, sorry. Please check above error messages.

I've installed system wide with sudo, but I launch GRASS without sudoing.
Could the problem be related to this?

giovanni

2013/4/16 G. Allegri <giohappy@gmail.com>

I will go on compiling.
I'm not so expert in sw management on Linux, will it be problematic to
have both a 6.4 and 7 GRASS on the same machine? I would like to test GRASS
7 too...

giovanni

2013/4/16 Markus Neteler <neteler@osgeo.org>

On Mon, Apr 15, 2013 at 12:32 PM, G. Allegri <giohappy@gmail.com> wrote:

Hi,
I've never installed add-ons throught the WxPython GUI.
I need to use the v.mkhexgrid [1], but the process fails because at some
stage (Rules.make) it tries to create/write into /usr/lib/bin folder

(very

strange).

You need (to wait for) current GRASS 6.4.svn or 6.4.3 which hopefully
addresses this problem.

The best would be that you compile 6.4.svn yourself on Ubuntu and
make a test before we release 6.4.3:

http://grasswiki.osgeo.org/wiki/Compile_and_Install_Ubuntu

thanks
Markus

--
Giovanni Allegri
http://about.me/giovanniallegri
blog: http://blog.spaziogis.it
GEO+ geomatica in Italia http://bit.ly/GEOplus

--
Giovanni Allegri
http://about.me/giovanniallegri
blog: http://blog.spaziogis.it
GEO+ geomatica in Italia http://bit.ly/GEOplus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20130416/6a1ad95a/attachment-0001.html&gt;

------------------------------

Message: 2
Date: Tue, 16 Apr 2013 20:50:41 +0100
From: Glynn Clements <glynn@gclements.plus.com>
To: Pedro Ven=?iso-8859-1?B?4g==?=ncio <pedrongvenancio@yahoo.com>
Cc: GRASS users <grass-user@lists.osgeo.org>,
  grass-dev@lists.osgeo.org
Subject: Re: [GRASS-user] Using r.quantile result with r.recode
Message-ID: <20845.43921.602188.954714@cerise.gclements.plus.com>
Content-Type: text/plain; charset=iso-8859-1

[CC to grass-dev for discussion]

Pedro Ven?ncio wrote:

Thank you very much for your answer!

My question lies precisely in the need to know if a quantile value
which falls as the upper limit for one range and the lower limit of
the next, should belong to the class anterior or posterior.

For example, assuming that r.quantile (with -r flag) gives this result:

2:6:1
6:8:2
8:12:3
12:20:4
20:873:5

the value 6 should belong to the first class or second?

r.recode will treat boundary values as belonging to the upper range,
e.g. in the above example, 6.0 will get recoded to 2.

This behaviour stems from Rast_fpreclass_get_cell_value() in
lib/raster/fpreclass.c, and isn't configurable (i.e. there's no way
that r.recode's behaviour could be modified without modifying the
fpreclass functions).

--
Glynn Clements <glynn@gclements.plus.com>

------------------------------

Message: 3
Date: Tue, 16 Apr 2013 17:07:31 -0700 (PDT)
From: Hamish <hamish_b@yahoo.com>
To: "G. Allegri" <giohappy@gmail.com>
Cc: GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] installing add-ons for GRASS 6.4.2 on Ubuntu
  fails
Message-ID:
  <1366157251.11397.YahooMailClassic@web140003.mail.bf1.yahoo.com>
Content-Type: text/plain; charset=us-ascii

G. Allegri wrote:

I've compiled GRASS 6.4.3RC2.The add-on installation
exits with the following error:

...

/usr/bin/install: cannot execute stat "/home/giova/Data/GRASSDB
/3003/PERMANENT/.tmp/xubuntu/16152.0/v.mkhexgrid/man/man1
/v.mkhexgrid.1": File or folder not existing
make: *** [install] Error 1WARNING: Installation failed, sorry.
Please check above error messages.

Hi,

since v.mkhexgrid is just a python script, you can just download
it by hand and put it somewhere in your path.

http://svn.osgeo.org/grass/grass-addons/grass6/vector/v.mkhexgrid/v.mkhexgrid

(~/.grass6/addons/ is a nice place, but ~/bin/ will work too
if you don't have GRASS_ADDON_PATH set up yet)

make sure to run 'chmod +x v.mkhexgrid' on it to make it executable.

I've installed system wide with sudo, but I launch GRASS
without sudoing. Could the problem be related to this?

Probably it is related to packaging work-in-progress issues with
the last version of GRASS on Ubuntu & Debian. Hopefully fixed
with the upcoming release of RC3 and the lastest DebianGIS
packaging rules.

I'm not so expert in sw management on Linux, will it be
problematic to have both a 6.4 and 7 GRASS on the same
machine? I would like to test GRASS 7 too...

In general no problem, as 'make install' will create independent
grass64/ and grass70/ directories for each. From a pre-built
packaging perspective the grass64 deb package contains a symlink
called "grass" for startup which the grass7 package can not
collide with. (only one package can provide a file by the same
name) AFAIR the other common system files like the desktop icons
are all named with the major version in the filename so no
conflict there.

note if you want the v.mkhexgrid script to run in g7 you should
check for a grass7 addons version of it. (or minor changes may
be required to adjust for any changed options)

Hamish

------------------------------

Message: 4
Date: Wed, 17 Apr 2013 08:48:58 +0200
From: "G. Allegri" <giohappy@gmail.com>
To: Hamish <hamish_b@yahoo.com>
Cc: GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] installing add-ons for GRASS 6.4.2 on Ubuntu
  fails
Message-ID:
  <CAB4g1=xFgG8RW7d8odZop2e7V9PGtz-oafQx1_aPY4d3Ti7zPw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi

since v.mkhexgrid is just a python script, you can just download
it by hand and put it somewhere in your path.

that's what I did, even before trying with a self-compiled 6.4.3RC2, but I
cannot find the extension under the GUI menus (I know, I can run it via
CLI).

make sure to run 'chmod +x v.mkhexgrid' on it to make it executable.

Maybe this is thw reason it doesn't load? I will try ASAP

Probably it is related to packaging work-in-progress issues with
the last version of GRASS on Ubuntu & Debian. Hopefully fixed
with the upcoming release of RC3 and the lastest DebianGIS
packaging rules.

This happens with 6.4.3RC2. I've compiled it using the sources from the svn
tags repository. Is trunk the latest 6.4.3? I thought it wad GRASS 7.

In general no problem, as 'make install' will create independent
grass64/ and grass70/ directories for each. From a pre-built
packaging perspective the grass64 deb package contains a symlink
called "grass" for startup which the grass7 package can not
collide with. (only one package can provide a file by the same
name) AFAIR the other common system files like the desktop icons
are all named with the major version in the filename so no
conflict there.

Ok, thx

note if you want the v.mkhexgrid script to run in g7 you should
check for a grass7 addons version of it. (or minor changes may
be required to adjust for any changed options)

I verified that it doesn't exist for g7, but it should be easy to update
it.

giovanni

Hamish

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20130417/9a629e01/attachment-0001.html&gt;

------------------------------

Message: 5
Date: Wed, 17 Apr 2013 10:04:09 +0200
From: Johannes Radinger <johannesradinger@gmail.com>
To: GRASS user list <grass-user@lists.osgeo.org>
Subject: [GRASS-user] Problem with r.univar and very large values
Message-ID:
  <CABsGe_wGMEeXa_=WcDLxx3U7E1U1QeQ8RHsyUBGHt_knJcG51g@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi,

I just faced a problem when trying to run r.univar (both GRASS6.5 and
GRASS7) on a raster which was multiplied by a very large value (1e+200) in
a python script.
The raster is displayed correctly and the cells can be querried and give
the correct large value but r.univar gives: *** stack smashing detected
***: r.univar terminated

As the map is not mutliplied using r.mapcalc but here I used the interface
to numpy arrays. Here a short working example (with the fields raster map
of the small spearfish example dataset). Probably this might work with
other raster input as well.

# With small spearfish dataset
import grass.script.array as garray

large_value = float(1e+200)

x1 = garray.array()
x1.read("lower_distance_tmp_29117")
result = garray.array()
result[...] = x1 * large_value
result.write("result_test")

And then running r.univar on result_test produces this error on my machine.
Can anyone reproduce that?

What might cause that problem? Does anyone have an idea? E.g. do some
summary value get to large (E.g. sum) to be processed? #

/Johannes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20130417/bf7ee688/attachment-0001.html&gt;

------------------------------

Message: 6
Date: Wed, 17 Apr 2013 01:33:28 -0700 (PDT)
From: Pedro Ven?ncio <pedrongvenancio@yahoo.com>
To: Glynn Clements <glynn@gclements.plus.com>
Cc: GRASS users <grass-user@lists.osgeo.org>,
  "grass-dev@lists.osgeo.org" <grass-dev@lists.osgeo.org>
Subject: Re: [GRASS-user] Using r.quantile result with r.recode
Message-ID:
  <1366187608.86300.YahooMailNeo@web122301.mail.ne1.yahoo.com>
Content-Type: text/plain; charset=iso-8859-1

Hi Glynn,

----- Original Message -----
From: Glynn Clements

r.recode will treat boundary values as belonging to the upper range,
e.g. in the above example, 6.0 will get recoded to 2.

So we can not use directly the result of r.quantile with -r flag in r.recode, right?

In the example, 6 should be recoded to 1, right?

Thank you very much!

Best regards,
Pedro

Pedro Ven?ncio wrote:

Thank you very much for your answer!

My question lies precisely in the need to know if a quantile value
which falls as the upper limit for one range and the lower limit of
the next, should belong to the class anterior or posterior.

For example, assuming that r.quantile (with -r flag) gives this result:

2:6:1
6:8:2
8:12:3
12:20:4
20:873:5

the value 6 should belong to the first class or second?

r.recode will treat boundary values as belonging to the upper range,
e.g. in the above example, 6.0 will get recoded to 2.

This behaviour stems from Rast_fpreclass_get_cell_value() in
lib/raster/fpreclass.c, and isn't configurable (i.e. there's no way
that r.recode's behaviour could be modified without modifying the
fpreclass functions).

--
Glynn Clements <glynn@gclements.plus.com>

------------------------------

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

End of grass-user Digest, Vol 84, Issue 35
******************************************

On Wed, Apr 17, 2013 at 10:53 AM, Ralf Schäfer <senator@ecotoxicology.de> wrote:

Dear all

presumably a trivial question - but what is the argument for space when importing data?
I have xyz files with the format
..x..y...value
where <.> stands for white space. The spacing is two spaces between x and y and three spaces to data column.

[ rather difficult to produce a file with different kinds of spacing
between columns ]

You could try

cat dgm10_32292_5548_2_rp.xyz | tr -s ' ' >dgm10_32292_5548_2_rp_one_space.xyz
r.in.xyz fs='space', in=dgm10_32292_5548_2_rp_one_space.xyz out=test

Markus M

I used
r.in.xyz fs='space', in=dgm10_32292_5548_2_rp.xyz out=test
and obtain:
ERROR: Bad y-coordinate line 1 column 2. <>

I also tried different options for fs: fs=' ', fs = space, fs = \s (whitespace regular expression) - without success

In R it is no problem - the simple read.table with sep="" for any whitespace works. And with rasterFromXYZ (raster package) I can create a raster without problems.

I guess it is also easy in GRASS but I am just too ignorant :slight_smile:

Cheers
Ralf

Ralf wrote:

> I have xyz files with the format
> ..x..y...value
> where <.> stands for white space. The spacing is
> two spaces between x and y and three spaces to data column.

Markus Metz wrote:

[ rather difficult to produce a file with different kinds of
spacing between columns ]

fwiw it's pretty standard for fortran output to come with
fixed column widths.

echo "1 2 3" | awk '{printf("%10.3f%10.3f%10.3f\n"), $1, $2, $3}'

You could try

cat dgm10_32292_5548_2_rp.xyz | tr -s ' '
>dgm10_32292_5548_2_rp_one_space.xyz
r.in.xyz fs='space', in=dgm10_32292_5548_2_rp_one_space.xyz
out=test

Yes, I think using 'tr -s' is the best way.

(you can pipe the result directly into r.in.xyz too if you like)

> I used
> r.in.xyz fs='space', in=dgm10_32292_5548_2_rp.xyz out=test
> and obtain:
> ERROR: Bad y-coordinate line 1 column 2. <>

if it really is _always_ two,three spaces between columns,
you could set x=3 y=5 z=8. But maybe a .0 is left off somewhere
and the number of spaces between columns changes?

> In R it is no problem - the simple read.table with
> sep="" for any whitespace works. And with rasterFromXYZ
> (raster package) I can create a raster without problems.

r.in.xyz (and most GRASS modules) split by each literal sep= char,
not until it finds another number the way glibc funtions or awk
might. An offshoot of this (mostly for importing .csv tables
from spreadsheets with v.in.ascii) is that empty data columns
are supported as containing NULL, and not messing up the placement
of columns to the right.

Hamish

Thanks for your help!!!

Am 17.04.2013 um 13:25 schrieb Hamish <hamish_b@yahoo.com>:

Ralf wrote:

I have xyz files with the format
..x..y...value
where <.> stands for white space. The spacing is
two spaces between x and y and three spaces to data column.

Markus Metz wrote:

[ rather difficult to produce a file with different kinds of
spacing between columns ]

yeah, some governmental institutions are really funny

You could try

cat dgm10_32292_5548_2_rp.xyz | tr -s ' '

dgm10_32292_5548_2_rp_one_space.xyz

r.in.xyz fs='space', in=dgm10_32292_5548_2_rp_one_space.xyz
out=test

Yes, I think using 'tr -s' is the best way.

(you can pipe the result directly into r.in.xyz too if you like)

The idea is very well and I should have thought about this myself.
However, I still get an error, although I now have 1-space delimited file.

r.in.xyz fs='space', input=dgm10_32292_5548_2_rp_one_space.xyz out=test

I already achieved the task with the help in R now, but from academic point would really be interested why is not working. I obtain the error message:
ERROR: Not enough data columns. Incorrect delimiter or column number? Found
       the following character(s) in row 1:
       [293220.000 5550000.000 311.432]

if it really is _always_ two,three spaces between columns,
you could set x=3 y=5 z=8. But maybe a .0 is left off somewhere
and the number of spaces between columns changes?

Although it looks strictly regular, this did also not work...

I uploaded sample files here:
http://www.sendspace.com/file/8q3vrx
Perhaps this makes reproduction of the problem easier...

Best regards
Ralf

In R it is no problem - the simple read.table with
sep="" for any whitespace works. And with rasterFromXYZ
(raster package) I can create a raster without problems.

r.in.xyz (and most GRASS modules) split by each literal sep= char,
not until it finds another number the way glibc funtions or awk
might. An offshoot of this (mostly for importing .csv tables
from spreadsheets with v.in.ascii) is that empty data columns
are supported as containing NULL, and not messing up the placement
of columns to the right.

Hamish

Ralf wrote:

However, I still get an error, although I now have 1-space
delimited file.

r.in.xyz fs='space',
input=dgm10_32292_5548_2_rp_one_space.xyz out=test

I already achieved the task with the help in R now, but from
academic point would really be interested why is not
working. I obtain the error message:
ERROR: Not enough data columns. Incorrect delimiter or
column number? Found
the following character(s) in row 1:
[293220.000 5550000.000 311.432]

try fs=space
I think the other (with the comma) is a typo.

I uploaded sample files here:
http://www.sendspace.com/file/8q3vrx
Perhaps this makes reproduction of the problem easier...

works for me:
GRASS65> r.in.xyz -s in=dgm10_32292_5548_2_rp_one_space.xyz out=dummy x=1 y=2 z=3 fs=space
Range: min max
x: 293080.000000 293990.000000
y: 5548130.000000 5549990.000000
z: 256.851000 426.240000

then import worked well using a 10m grid resolution.
(grid resolution confirmed by running r.univar on the method=n
map + 'r.null setnull=0')

Hamish

Thanks this works now!
Very obvious with the comma - but not when you are working with R the whole day where arguments are separated by commas :slight_smile:

best regards
Ralf

Am 17.04.2013 um 23:20 schrieb Hamish <hamish_b@yahoo.com>:

Ralf wrote:

However, I still get an error, although I now have 1-space
delimited file.

r.in.xyz fs='space',
input=dgm10_32292_5548_2_rp_one_space.xyz out=test

I already achieved the task with the help in R now, but from
academic point would really be interested why is not
working. I obtain the error message:
ERROR: Not enough data columns. Incorrect delimiter or
column number? Found
       the following character(s) in row 1:
       [293220.000 5550000.000 311.432]

try fs=space
I think the other (with the comma) is a typo.

I uploaded sample files here:
http://www.sendspace.com/file/8q3vrx
Perhaps this makes reproduction of the problem easier...

works for me:
GRASS65> r.in.xyz -s in=dgm10_32292_5548_2_rp_one_space.xyz out=dummy x=1 y=2 z=3 fs=space
Range: min max
x: 293080.000000 293990.000000
y: 5548130.000000 5549990.000000
z: 256.851000 426.240000

then import worked well using a 10m grid resolution.
(grid resolution confirmed by running r.univar on the method=n
map + 'r.null setnull=0')

Hamish