[GRASS-user] Importing multiple files with r.in.xyz

Hanlie_Pretorius · August 10, 2010, 12:59pm

Hi,

I'm working with WinGRASS 6.4RC6 on Win XP.

I'm following the steps at http://grass.osgeo.org/wiki/Import_XYZ to
import hundreds files containing TRMM rainfall data.

When I try to follow the instruction:
-----
Just amend above procedure to use wildcards. Change in above example
all occurencies of

cat VTL2733.XYZ | ...

to

cat *.XYZ | ...

and use a more reasonable output name of course. That's all to import
even thousands of files (tiled DEM) easily.
-----

Not sure what a 'reasonable output name is', so I run it as shown
below and get the following error:
-----
cat F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt| r.in.xyz
-s in=- fs=, out=test
cat: F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt: No such
file or directory
-----

If I try it with one file only, it works:
-----
cat F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\3B42.000201.0.6.nc.lieb.txt|
r.in.xyz -s in=- fs=, out=test
Range:
x: -28.625000 -27.375000
y: 28.125000 28.625000
z: 0.000000 0.000000
-----

How do I get this to work? And how would I get the output filenames to
be related to the input filenames? For example, if the input filename
is 3B42.000201.0.6.nc.lieb.txt, then I want the output raster to be
named '3B42.000201.0.6'.

Thanks
Hanlie

Saber · August 10, 2010, 2:44pm

Hanile

Try this:

1- Install Paths:
http://www.textpad.com/add-ons/files/utilities/paths.zip
It's a useful tool to copy the path and filenames in windows. It adds a
function to rightclick: Pathcopy. You can select all your rasters and
rightclick to copy the paths:

Paste them in a text file
c:\example\raster1.xyz
c:\example\raster2.xyz
...

2- change them to something like this (use find/replace..macros, etc:)

r.in.xyz -s -g input="c:\example\raster1.xyz" output=raster1 > tmpRegion
SET /p myregion= < tmpRegion
DEL tmpRegion
g.region %myregion%
r.in.xyz --overwrite input="c:\example\raster1.xyz" output=raster1 fs=,

3- Save that as a import.bat
4- From the Start menu, run Grass in Txt mode, change directory to the
folder where import.bat is
5- import.bat

If you google around you should be able to make the script a bit smarter
using DOS and loop (http://www.robvanderwoude.com/ntfor.php).

In linux for example you can shrink it to something like this:

cat list_of_files.txt | while read line; do
echo $line+"Being processed" # or whaterver you want to do with the
$line variable
raster=$line
r.in.xyz -s -g input=$line output=$line > tmpRegion
myregion= ""`head -n 1 tmpRegion`"
g.region $myregion
r.in.xyz --overwrite input=$line output=$line fs=,
done

and your list_of_files.txt is
raster1
raster2
.....

Hope that helps
Saber

On Tue, 2010-08-10 at 14:59 +0200, Hanlie Pretorius wrote:

Hi,

I'm working with WinGRASS 6.4RC6 on Win XP.

I'm following the steps at http://grass.osgeo.org/wiki/Import_XYZ to
import hundreds files containing TRMM rainfall data.

When I try to follow the instruction:
-----
Just amend above procedure to use wildcards. Change in above example
all occurencies of

cat VTL2733.XYZ | ...

to

cat *.XYZ | ...

and use a more reasonable output name of course. That's all to import
even thousands of files (tiled DEM) easily.
-----

Not sure what a 'reasonable output name is', so I run it as shown
below and get the following error:
-----
cat F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt| r.in.xyz
-s in=- fs=, out=test
cat: F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt: No such
file or directory
-----

If I try it with one file only, it works:
-----
cat F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\3B42.000201.0.6.nc.lieb.txt|
r.in.xyz -s in=- fs=, out=test
Range:
x: -28.625000 -27.375000
y: 28.125000 28.625000
z: 0.000000 0.000000
-----

How do I get this to work? And how would I get the output filenames to
be related to the input filenames? For example, if the input filename
is 3B42.000201.0.6.nc.lieb.txt, then I want the output raster to be
named '3B42.000201.0.6'.

Thanks
Hanlie
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Nikos_Alexandris1 · August 10, 2010, 2:48pm

Hanlie Pretorius wrote:

I'm working with WinGRASS 6.4RC6 on Win XP.

[...]

Not sure what a 'reasonable output name is', so I run it as shown
below and get the following error:
-----
cat F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt| r.in.xyz
-s in=- fs=, out=test
cat: F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt: No such
file or directory
-----

[...]

How do I get this to work?

No window$ user here but I think it has to do with the "*.txt" part. How do
wildcard patterns work in these cases (under window$) might be the question to
the answer you are looking for.

And how would I get the output filenames to
be related to the input filenames? For example, if the input filename
is 3B42.000201.0.6.nc.lieb.txt, then I want the output raster to be
named '3B42.000201.0.6'.

Building a for loop to treat one file at a time should do it. In pseudo-code
or in bash this would look like (given all and only .txt files of interest are
in a (the current) directory) :

--%<---
for FILE in *.txt ; do

done
-->%---

How would this work in windows? Look in past threads, there must be something
or wait till the experts chime-in. Sorry for the indirect answers.

Cheers, Nikos

Hanlie_Pretorius · August 10, 2010, 3:09pm

Thanks for the suggestions.

I think I'm going to try the Linux version as soon as I can get to a
Linux machine.

2010/8/10, Saber Razmjooei <razmjooeis@faunalia.co.uk>:

Hanile

Try this:

1- Install Paths:
http://www.textpad.com/add-ons/files/utilities/paths.zip
It's a useful tool to copy the path and filenames in windows. It adds a
function to rightclick: Pathcopy. You can select all your rasters and
rightclick to copy the paths:

Paste them in a text file
c:\example\raster1.xyz
c:\example\raster2.xyz
...

2- change them to something like this (use find/replace..macros, etc:)

r.in.xyz -s -g input="c:\example\raster1.xyz" output=raster1 > tmpRegion
SET /p myregion= < tmpRegion
DEL tmpRegion
g.region %myregion%
r.in.xyz --overwrite input="c:\example\raster1.xyz" output=raster1 fs=,

3- Save that as a import.bat
4- From the Start menu, run Grass in Txt mode, change directory to the
folder where import.bat is
5- import.bat

If you google around you should be able to make the script a bit smarter
using DOS and loop (http://www.robvanderwoude.com/ntfor.php).

In linux for example you can shrink it to something like this:

cat list_of_files.txt | while read line; do
echo $line+"Being processed" # or whaterver you want to do with the
$line variable
raster=$line
r.in.xyz -s -g input=$line output=$line > tmpRegion
myregion= ""`head -n 1 tmpRegion`"
g.region $myregion
r.in.xyz --overwrite input=$line output=$line fs=,
done

and your list_of_files.txt is
raster1
raster2
.....

Hope that helps
Saber

On Tue, 2010-08-10 at 14:59 +0200, Hanlie Pretorius wrote:

Hi,

I'm working with WinGRASS 6.4RC6 on Win XP.

I'm following the steps at http://grass.osgeo.org/wiki/Import_XYZ to
import hundreds files containing TRMM rainfall data.

When I try to follow the instruction:
-----
Just amend above procedure to use wildcards. Change in above example
all occurencies of

cat VTL2733.XYZ | ...

to

cat *.XYZ | ...

and use a more reasonable output name of course. That's all to import
even thousands of files (tiled DEM) easily.
-----

Not sure what a 'reasonable output name is', so I run it as shown
below and get the following error:
-----
cat F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt| r.in.xyz
-s in=- fs=, out=test
cat: F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\*.txt: No such
file or directory
-----

If I try it with one file only, it works:
-----
cat
F:\Hanlie\UCT\M.Sc.\Data\TRMM\2000\02_Februrarie\3B42.000201.0.6.nc.lieb.txt|
r.in.xyz -s in=- fs=, out=test
Range:
x: -28.625000 -27.375000
y: 28.125000 28.625000
z: 0.000000 0.000000
-----

How do I get this to work? And how would I get the output filenames to
be related to the input filenames? For example, if the input filename
is 3B42.000201.0.6.nc.lieb.txt, then I want the output raster to be
named '3B42.000201.0.6'.

Thanks
Hanlie
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Nikos_Alexandris1 · August 10, 2010, 8:04pm

Hanlie Pretorius wrote:

Thanks for the suggestions.

I think I'm going to try the Linux version as soon as I can get to a
Linux machine.

You could also try out osgeo live-cd/dvd's (e.g. <http://live.osgeo.org/>\) or
give (some) Linux(-distro) the chance (it deserves) even via virtualbox for
example.

Nikos

Hanlie_Pretorius · August 11, 2010, 12:10pm

My apologies for my ignorance, but how do I get this to run in Linux?
Am using Ubuntu 10.04 and GRASS 6.4RC6.

I entered the following into the GRASS text window and am testing it
with a list of two files (I removed the region settings because the
files all have the same region, which I have already set):
-----
GRASS 6.4.0RC6 (world_wgs84):~ > cat lieb_files_test.txt
3B42.000201.12.6.nc.lieb.txt
3B42.000201.15.6.nc.lieb.txt
GRASS 6.4.0RC6 (world_wgs84):~ > cat lieb_files_test.txt | while read line; do

echo $line+" being processed"
raster=${$line:(-12)} # cut ".nc.lieb.txt" from raster name
r.in.xyz --overwrite input=$line output=raster method="mean" type="FCELL" fs=, x=2 y=1 z=3

-----

As you can see, when I press enter after the r.in.xyz line, I just get
another prompt.

Am I supposed to save it to a separate file and run this as a script?
If so, where should I put this script?

2010/8/10, Saber Razmjooei <razmjooeis@faunalia.co.uk>:

cat list_of_files.txt | while read line; do
echo $line+"Being processed" # or whaterver you want to do with the
$line variable
raster=$line
r.in.xyz -s -g input=$line output=$line > tmpRegion
myregion= ""`head -n 1 tmpRegion`"
g.region $myregion
r.in.xyz --overwrite input=$line output=$line fs=,
done

and your list_of_files.txt is
raster1
raster2
.....

Hope that helps
Saber

Vincent_Bain · August 11, 2010, 12:27pm

Hi, short intrusion in this current thread, without knowing exactly what
it is related to : I only mean to suggest you to close the while loop...

This prompt ">" means the shell expects the end of your statement ; here
you should type "done" to close the while instruction.

Hope it helps,
Regards.

Vincent

Le mercredi 11 août 2010 à 14:10 +0200, Hanlie Pretorius a écrit :

My apologies for my ignorance, but how do I get this to run in Linux?
Am using Ubuntu 10.04 and GRASS 6.4RC6.

I entered the following into the GRASS text window and am testing it
with a list of two files (I removed the region settings because the
files all have the same region, which I have already set):
-----
GRASS 6.4.0RC6 (world_wgs84):~ > cat lieb_files_test.txt
3B42.000201.12.6.nc.lieb.txt
3B42.000201.15.6.nc.lieb.txt
GRASS 6.4.0RC6 (world_wgs84):~ > cat lieb_files_test.txt | while read line; do
> echo $line+" being processed"
> raster=${$line:(-12)} # cut ".nc.lieb.txt" from raster name
> r.in.xyz --overwrite input=$line output=raster method="mean" type="FCELL" fs=, x=2 y=1 z=3
>
-----

As you can see, when I press enter after the r.in.xyz line, I just get
another prompt.

Am I supposed to save it to a separate file and run this as a script?
If so, where should I put this script?

2010/8/10, Saber Razmjooei <razmjooeis@faunalia.co.uk>:
>
> cat list_of_files.txt | while read line; do
> echo $line+"Being processed" # or whaterver you want to do with the
> $line variable
> raster=$line
> r.in.xyz -s -g input=$line output=$line > tmpRegion
> myregion= ""`head -n 1 tmpRegion`"
> g.region $myregion
> r.in.xyz --overwrite input=$line output=$line fs=,
> done
>
> and your list_of_files.txt is
> raster1
> raster2
> .....
>
> Hope that helps
> Saber
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Micha_Silver · August 11, 2010, 12:32pm

On 08/11/2010 03:10 PM, Hanlie Pretorius wrote:

My apologies for my ignorance, but how do I get this to run in Linux?
Am using Ubuntu 10.04 and GRASS 6.4RC6.

Samber's method will surely work, but you might more simply try as follows:

# start grass in a location which matches the txt file data
# change to the directory where your txt files are, then do
for infile in *.txt; do outrast=`basename ${infile} .txt`; r.in.xyz in=${infile} out=${outrast} fs=,; done

# that's it
--
Micha

I entered the following into the GRASS text window and am testing it
with a list of two files (I removed the region settings because the
files all have the same region, which I have already set):
-----
GRASS 6.4.0RC6 (world_wgs84):~> cat lieb_files_test.txt
3B42.000201.12.6.nc.lieb.txt
3B42.000201.15.6.nc.lieb.txt
GRASS 6.4.0RC6 (world_wgs84):~> cat lieb_files_test.txt | while read line; do

  echo $line+" being processed"
raster=${$line:(-12)} # cut ".nc.lieb.txt" from raster name
r.in.xyz --overwrite input=$line output=raster method="mean" type="FCELL" fs=, x=2 y=1 z=3

-----

As you can see, when I press enter after the r.in.xyz line, I just get
another prompt.

Am I supposed to save it to a separate file and run this as a script?
If so, where should I put this script?

2010/8/10, Saber Razmjooei<razmjooeis@faunalia.co.uk>:

cat list_of_files.txt | while read line; do
     echo $line+"Being processed" # or whaterver you want to do with the
$line variable
  raster=$line
  r.in.xyz -s -g input=$line output=$line> tmpRegion
myregion= ""`head -n 1 tmpRegion`"
g.region $myregion
r.in.xyz --overwrite input=$line output=$line fs=,
done

and your list_of_files.txt is
raster1
raster2
.....

Hope that helps
Saber

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

This mail was received via Mail-SeCure System.

Micha_Silver · August 11, 2010, 12:36pm

On 08/11/2010 03:32 PM, Micha Silver wrote:

On 08/11/2010 03:10 PM, Hanlie Pretorius wrote:

My apologies for my ignorance, but how do I get this to run in Linux?
Am using Ubuntu 10.04 and GRASS 6.4RC6.

Samber's method will surely work, but you might more simply try as follows:

# start grass in a location which matches the txt file data
# change to the directory where your txt files are, then do
for infile in *.txt; do outrast=`basename ${infile} .txt`; r.in.xyz in=${infile} out=${outrast} fs=,; done

Oops- I forgot the x=2 y=1 z=3 parameters. Add those to the grass command also...

# that's it
--
Micha

I entered the following into the GRASS text window and am testing it
with a list of two files (I removed the region settings because the
files all have the same region, which I have already set):
-----
GRASS 6.4.0RC6 (world_wgs84):~> cat lieb_files_test.txt
3B42.000201.12.6.nc.lieb.txt
3B42.000201.15.6.nc.lieb.txt
GRASS 6.4.0RC6 (world_wgs84):~> cat lieb_files_test.txt | while read line; do

  echo $line+" being processed"
raster=${$line:(-12)} # cut ".nc.lieb.txt" from raster name
r.in.xyz --overwrite input=$line output=raster method="mean" type="FCELL" fs=, x=2 y=1 z=3

-----

As you can see, when I press enter after the r.in.xyz line, I just get
another prompt.

Am I supposed to save it to a separate file and run this as a script?
If so, where should I put this script?

2010/8/10, Saber Razmjooei<razmjooeis@faunalia.co.uk>:

cat list_of_files.txt | while read line; do
     echo $line+"Being processed" # or whaterver you want to do with the
$line variable
  raster=$line
  r.in.xyz -s -g input=$line output=$line> tmpRegion
myregion= ""`head -n 1 tmpRegion`"
g.region $myregion
r.in.xyz --overwrite input=$line output=$line fs=,
done

and your list_of_files.txt is
raster1
raster2
.....

Hope that helps
Saber

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

This mail was received via Mail-SeCure System.

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

This mail was received via Mail-SeCure System.

Hanlie_Pretorius · August 11, 2010, 12:36pm

2010/8/11, Vincent Bain <bain@toraval.fr>:

Hi, short intrusion in this current thread, without knowing exactly what
it is related to : I only mean to suggest you to close the while loop...

This prompt ">" means the shell expects the end of your statement ; here
you should type "done" to close the while instruction.

Adding 'done' worked and I got the script running from the command line.

Thanks to everyone who chipped in.

Saber · August 11, 2010, 12:38pm

Open gedit (from Accessories) and copy paste the the commands into it.
Save the file to the same location as your raster and your files.txt file
call it import_xyz.sh

Now, open grass from terminal
Once in grass environment...change directory to your file
sh import_xyz.sh

Cheers
Saber

  On 08/11/2010 03:10 PM, Hanlie Pretorius wrote:

My apologies for my ignorance, but how do I get this to run in Linux?
Am using Ubuntu 10.04 and GRASS 6.4RC6.

Samber's method will surely work, but you might more simply try as
follows:

# start grass in a location which matches the txt file data
# change to the directory where your txt files are, then do
for infile in *.txt; do outrast=`basename ${infile} .txt`; r.in.xyz
in=${infile} out=${outrast} fs=,; done

# that's it
  --
Micha

I entered the following into the GRASS text window and am testing it
with a list of two files (I removed the region settings because the
files all have the same region, which I have already set):
-----
GRASS 6.4.0RC6 (world_wgs84):~> cat lieb_files_test.txt
3B42.000201.12.6.nc.lieb.txt
3B42.000201.15.6.nc.lieb.txt
GRASS 6.4.0RC6 (world_wgs84):~> cat lieb_files_test.txt | while read
line; do

  echo $line+" being processed"
raster=${$line:(-12)} # cut ".nc.lieb.txt" from raster name
r.in.xyz --overwrite input=$line output=raster method="mean"
type="FCELL" fs=, x=2 y=1 z=3

-----

As you can see, when I press enter after the r.in.xyz line, I just get
another prompt.

Am I supposed to save it to a separate file and run this as a script?
If so, where should I put this script?

2010/8/10, Saber Razmjooei<razmjooeis@faunalia.co.uk>:

cat list_of_files.txt | while read line; do
     echo $line+"Being processed" # or whaterver you want to do with
the
$line variable
  raster=$line
  r.in.xyz -s -g input=$line output=$line> tmpRegion
myregion= ""`head -n 1 tmpRegion`"
g.region $myregion
r.in.xyz --overwrite input=$line output=$line fs=,
done

and your list_of_files.txt is
raster1
raster2
.....

Hope that helps
Saber

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

This mail was received via Mail-SeCure System.

Hanlie_Pretorius · August 11, 2010, 1:07pm

2010/8/11, Micha Silver <micha@arava.co.il>:

Samber's method will surely work, but you might more simply try as follows:

# start grass in a location which matches the txt file data
# change to the directory where your txt files are, then do
for infile in *.txt; do outrast=`basename ${infile} .txt`; r.in.xyz
in=${infile} out=${outrast} fs=,; done

This works nicely, except that my input files have names like:
3B42.000201.0.6.nc.lieb.txt
3B42.000201.3.6.nc.lieb.txt

and I want the output rasters to be named
3B42.000201.0.6
3B42.000201.3.6

In fact, it would be best if they could be named:
0201.0.6
0201.3.6

I'm struggling with this string manipulation in the shell scripting language.

# that's it
--
Micha

Nikos_Alexandris1 · August 11, 2010, 1:21pm

Micha Silver <micha@arava.co.il>:

> Samber's method will surely work, but you might more simply try as
> follows:
>
> # start grass in a location which matches the txt file data
> # change to the directory where your txt files are, then do
> for infile in *.txt; do outrast=`basename ${infile} .txt`; r.in.xyz
> in=${infile} out=${outrast} fs=,; done

Hanlie Pretorius wrote:

This works nicely, except that my input files have names like:
3B42.000201.0.6.nc.lieb.txt
3B42.000201.3.6.nc.lieb.txt

and I want the output rasters to be named
3B42.000201.0.6
3B42.000201.3.6

In fact, it would be best if they could be named:
0201.0.6
0201.3.6

I'm struggling with this string manipulation in the shell scripting
language.

Try the following (note the backticks in the beginning before "echo" and in
the end as well):

output=`echo "${infile}" | cut -d"." -f2,3,4 | sed 's/^00//'`

Nikos

Hanlie_Pretorius · August 11, 2010, 1:26pm

2010/8/11, Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de>:

Micha Silver <micha@arava.co.il>:

> Samber's method will surely work, but you might more simply try as
> follows:
>
> # start grass in a location which matches the txt file data
> # change to the directory where your txt files are, then do
> for infile in *.txt; do outrast=`basename ${infile} .txt`; r.in.xyz
> in=${infile} out=${outrast} fs=,; done

Hanlie Pretorius wrote:

This works nicely, except that my input files have names like:
3B42.000201.0.6.nc.lieb.txt
3B42.000201.3.6.nc.lieb.txt

and I want the output rasters to be named
3B42.000201.0.6
3B42.000201.3.6

In fact, it would be best if they could be named:
0201.0.6
0201.3.6

I'm struggling with this string manipulation in the shell scripting
language.

Try the following (note the backticks in the beginning before "echo" and in
the end as well):

output=`echo "${infile}" | cut -d"." -f2,3,4 | sed 's/^00//'`

Yes, this worked Thanks.

Nikos

Hermann_Peifer · August 11, 2010, 1:31pm

On 11/08/2010 15:07, Hanlie Pretorius wrote:

This works nicely, except that my input files have names like:
3B42.000201.0.6.nc.lieb.txt
3B42.000201.3.6.nc.lieb.txt

and I want the output rasters to be named
3B42.000201.0.6
3B42.000201.3.6

In fact, it would be best if they could be named:
0201.0.6
0201.3.6

I'm struggling with this string manipulation in the shell scripting language.

In bash, you could do:

for infile in *.txt ; do r.in.xyz in=${infile} out=${infile:7:8} ... ; done

Hermann