[GRASS-user] using grass on rasters without converting?

GRASS GRASS user
1

Hello Grass World,

As a long time GIS user I've dipped into grass from time to time but never
in depth. One of the things that's deterred me is the fact that I have a
large body of data I work with (tens of gigabytes) and the prospect of
having to convert into grass, process it, and then flip it back out again is
unappealing. I respect the fact that the grass storage format allow it to do
things not possible otherwise. Is there any program available, or a project
in the works, for easy in-and-out conversion? or better yet, a method to use
grass commands on external formats like geotiff without conversion? Is this
a possibility in future or is the structure of grass such that it will just
never work? I'd love to be able to be able to do the likes of
"r.shaded.relief input=mydem.tif output=myshade.tif"

thanks,

-matt
--
View this message in context: http://www.nabble.com/using-grass-on-rasters-without-converting--tp20245591p20245591.html
Sent from the Grass - Users mailing list archive at Nabble.com.

2

Hi Matt,

On Thu, Oct 30, 2008 at 1:07 PM, matt wilkie <maphew@gmail.com> wrote:

Hello Grass World,

As a long time GIS user I've dipped into grass from time to time but never
in depth. One of the things that's deterred me is the fact that I have a
large body of data I work with (tens of gigabytes) and the prospect of
having to convert into grass, process it, and then flip it back out again is
unappealing.

Sure - also for us :slight_smile: But it has been solved recently:

r.external - Link GDAL supported raster file to a binary raster map layer.
http://grass.osgeo.org/grass64/manuals/html64_user/r.external.html

...

I'd love to be able to be able to do the likes of
"r.shaded.relief input=mydem.tif output=myshade.tif"

Yes, you can do that from 6.4.svn onwards:

r.external [-r] in=/path/to/mydem.tif out=mydem
g.region rast=mydem -p
r.shaded.relief input=mydem shadedmap=myshade
r.out.gdal myshade out=myshade.tif

For r.out.gdal, check which format is appropriate to keep
the color table if desired.

Hope this helps,
Markus

3

Markus Neteler wrote:

Hi Matt,

On Thu, Oct 30, 2008 at 1:07 PM, matt wilkie <maphew@gmail.com> wrote:

Hello Grass World,

As a long time GIS user I've dipped into grass from time to time but never
in depth. One of the things that's deterred me is the fact that I have a
large body of data I work with (tens of gigabytes) and the prospect of
having to convert into grass, process it, and then flip it back out again is
unappealing.

Sure - also for us :slight_smile: But it has been solved recently:

r.external - Link GDAL supported raster file to a binary raster map layer.
http://grass.osgeo.org/grass64/manuals/html64_user/r.external.html

This solves a need for me too. Could something like that be done for WMS services, i.e. not downloading them with r.in.wms, but linking them without making a local copy?

Jan

4

As a long time GIS user I've dipped into grass from time to time but never
in depth. One of the things that's deterred me is the fact that I have a
large body of data I work with (tens of gigabytes) and the prospect of
having to convert into grass, process it, and then flip it back out again is
unappealing.

Sure - also for us :slight_smile: But it has been solved recently:

r.external - Link GDAL supported raster file to a binary raster map layer.
http://grass.osgeo.org/grass64/manuals/html64_user/r.external.html

Awesome! Thanks Markus (for letting me know) and everyone who worked
r.external. :slight_smile:

--
-matt

5

On Thu, Oct 30, 2008 at 2:08 PM, Jan Hartmann <j.l.h.hartmann@uva.nl> wrote:

Markus Neteler wrote:

r.external - Link GDAL supported raster file to a binary raster map layer.
http://grass.osgeo.org/grass64/manuals/html64_user/r.external.html

This solves a need for me too. Could something like that be done for WMS
services, i.e. not downloading them with r.in.wms, but linking them without
making a local copy?

In theory, r.external might also accept WMS since GDAL does it:
http://www.gdal.org/frmt_wms.html

cat gdal_wms.xml
<GDAL_WMS>
    <Service name="WMS">
        <Version>1.1.1</Version>
        <ServerUrl>http://onearth.jpl.nasa.gov/wms.cgi?&lt;/ServerUrl&gt;
        <SRS>EPSG:4326</SRS>
        <ImageFormat>image/jpeg</ImageFormat>
        <Layers>modis,global_mosaic</Layers>
        <Styles></Styles>
    </Service>
    <DataWindow>
        <UpperLeftX>-180.0</UpperLeftX>
        <UpperLeftY>90.0</UpperLeftY>
        <LowerRightX>180.0</LowerRightX>
        <LowerRightY>-90.0</LowerRightY>
        <SizeX>2666666</SizeX>
        <SizeY>1333333</SizeY>
    </DataWindow>
    <Projection>EPSG:4326</Projection>
    <BandsCount>3</BandsCount>
</GDAL_WMS>

# this works:
qgis gdal_wms.xml

# this, too:
r.external /tmp/gdal_wms.xml out=gdal_wms

g.region n=90 s=-90 w=-180 e=180 res=1 -p
d.mon x0
d.rast gdal_wms

# this gets stuck:
d.rast gdal_wms
   0%

strace shows
...
select(8, [6 7], , , {0, 100000}) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0
poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 0
select(8, [6 7], , , {0, 100000}) = 0 (Timeout)
...

Maybe some update magic is needed internally (but maybe
I am exaggerating right now...).

Markus

6

Is there any program available, or a project
in the works, for easy in-and-out conversion? or better yet, a method to use
grass commands on external formats like geotiff without conversion?

Have a look at r.external:

http://grass.itc.it/grass64/manuals/html64_user/r.external.html

~ Eric.

7

Markus Neteler wrote:

>> r.external - Link GDAL supported raster file to a binary raster map layer.
>> http://grass.osgeo.org/grass64/manuals/html64_user/r.external.html
>>
>
> This solves a need for me too. Could something like that be done for WMS
> services, i.e. not downloading them with r.in.wms, but linking them without
> making a local copy?

In theory, r.external might also accept WMS since GDAL does it:
http://www.gdal.org/frmt_wms.html

# this gets stuck:
d.rast gdal_wms
   0%

strace shows
...
select(8, [6 7], , , {0, 100000}) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0
poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 0
select(8, [6 7], , , {0, 100000}) = 0 (Timeout)
...

Maybe some update magic is needed internally (but maybe
I am exaggerating right now...).

Maybe GDAL can't handle skipping rows for WMS? Due to the region
mapping, GRASS typically doesn't request rows sequentially (it may
skip rows, or request them in an arbitrary order, even reverse order).

--
Glynn Clements <glynn@gclements.plus.com>

8

Markus Neteler wrote:

> As a long time GIS user I've dipped into grass from time to time but never
> in depth. One of the things that's deterred me is the fact that I have a
> large body of data I work with (tens of gigabytes) and the prospect of
> having to convert into grass, process it, and then flip it back out again is
> unappealing.

Sure - also for us :slight_smile: But it has been solved recently:

r.external - Link GDAL supported raster file to a binary raster map layer.
http://grass.osgeo.org/grass64/manuals/html64_user/r.external.html

However, this is only works for reading. Any maps created by GRASS are
still stored in the GRASS database, and need to be exported explicitly
with e.g. r.out.gdal.

Making it work for writing would require a way to obtain the various
creation options (i.e. file format, data format (UInt8, UInt16 etc),
output directory, probably a lot of other options). Implementation
shouldn't be hard, but specification and design might be.

--
Glynn Clements <glynn@gclements.plus.com>

9

On Thu, Oct 30, 2008 at 6:57 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

In theory, r.external might also accept WMS since GDAL does it:
http://www.gdal.org/frmt_wms.html

# this gets stuck:
d.rast gdal_wms
   0%

strace shows
...
select(8, [6 7], , , {0, 100000}) = 0 (Timeout)
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0
poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 0
select(8, [6 7], , , {0, 100000}) = 0 (Timeout)
...

Maybe some update magic is needed internally (but maybe
I am exaggerating right now...).

Maybe GDAL can't handle skipping rows for WMS? Due to the region
mapping, GRASS typically doesn't request rows sequentially (it may
skip rows, or request them in an arbitrary order, even reverse order).

Here an offlist answer from Frank (fwd with permission):

On Thu, Oct 30, 2008 at 7:34 PM, Frank Warmerdam wrote:

I'm not too familiar with the WMS driver. But if it doesn't support
skipping rows then it is a bug and should be fixed. To file a ticket on
this issue it would be helpful if a demonstration of the problem could
be boiled down to a small C/C++ program or Python script just depending
on GDAL.

Adam Nowaki is the main author of the WMS driver.

Unfortunately I am not skilled enough in Python to write down the
requested test case.

Markus

10

Markus Neteler wrote:

> I'm not too familiar with the WMS driver. But if it doesn't support
> skipping rows then it is a bug and should be fixed. To file a ticket on
> this issue it would be helpful if a demonstration of the problem could
> be boiled down to a small C/C++ program or Python script just depending
> on GDAL.
>
> Adam Nowaki is the main author of the WMS driver.

Unfortunately I am not skilled enough in Python to write down the
requested test case.

Does "gdal_translate -outsize ..." work with WMS?

--
Glynn Clements <glynn@gclements.plus.com>

11

On Thu, Oct 30, 2008 at 10:24 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

> I'm not too familiar with the WMS driver. But if it doesn't support
> skipping rows then it is a bug and should be fixed. To file a ticket on
> this issue it would be helpful if a demonstration of the problem could
> be boiled down to a small C/C++ program or Python script just depending
> on GDAL.
>
> Adam Nowaki is the main author of the WMS driver.

Unfortunately I am not skilled enough in Python to write down the
requested test case.

Does "gdal_translate -outsize ..." work with WMS?

Apparently not:

gdal_translate -outsize 20% 20% wms_jpl_onearth.xml bla.tif
Input file size is 2666666, 1333333
0

... then it goes into poll time-out (according to strace).

Markus

12

Markus Neteler wrote:

>> > I'm not too familiar with the WMS driver. But if it doesn't support
>> > skipping rows then it is a bug and should be fixed. To file a ticket on
>> > this issue it would be helpful if a demonstration of the problem could
>> > be boiled down to a small C/C++ program or Python script just depending
>> > on GDAL.
>> >
>> > Adam Nowaki is the main author of the WMS driver.
>>
>> Unfortunately I am not skilled enough in Python to write down the
>> requested test case.
>
> Does "gdal_translate -outsize ..." work with WMS?

Apparently not:

gdal_translate -outsize 20% 20% wms_jpl_onearth.xml bla.tif
Input file size is 2666666, 1333333
0

... then it goes into poll time-out (according to strace).

In that case, it's a GDAL issue, and that should be a sufficient test
case.

--
Glynn Clements <glynn@gclements.plus.com>

13

On Sat, Nov 1, 2008 at 5:29 AM, Glynn Clements <glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

>> > I'm not too familiar with the WMS driver. But if it doesn't support
>> > skipping rows then it is a bug and should be fixed. To file a ticket on
>> > this issue it would be helpful if a demonstration of the problem could
>> > be boiled down to a small C/C++ program or Python script just depending
>> > on GDAL.
>> >
>> > Adam Nowaki is the main author of the WMS driver.
>>
>> Unfortunately I am not skilled enough in Python to write down the
>> requested test case.
>
> Does "gdal_translate -outsize ..." work with WMS?

Apparently not:

gdal_translate -outsize 20% 20% wms_jpl_onearth.xml bla.tif
Input file size is 2666666, 1333333
0

... then it goes into poll time-out (according to strace).

In that case, it's a GDAL issue, and that should be a sufficient test
case.

Right. Filed as
http://trac.osgeo.org/gdal/ticket/264

Markus