[GRASS-user] Working with a large number of maps

Hi

I have 139 high resolution images, which I need to use as a background map
for some digitizing work. Loading 139x3 layers into gis.m is a real pain and
I can think of three workarounds.

1. Enter Rainer Krug and MapServer.
I don't think my time frames allow for learning another software suite right
now.

2. Use a script to build a .grc file with a group for gis.m.
Before attempting this I would like to know if there are any limits to the
number of layers that can be loaded.

3. Use r.patch to build one mosaic for each band.
I currently have r.patch running. It's been going for about 10.5hrs and has
reached 15%. Does this sound reasonable? I am starting to wonder if I'm
expecting too much. Here are my region settings:

GRASS 6.3.0RC3 (msunduzi_lo31):~ > g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: wgs84
ellipsoid: wgs84
north: -3264858.18753122
south: -3295545.14772988
west: -92005.31394386
east: -43582.78768323
nsres: 0.35000012
ewres: 0.35000019
rows: 87677
cols: 138350
cells: 12130112950

Another question on the usage of r.patch:
I have passed the option --verbose to r.patch and I am expecting the program
to generate VERBOSE output, but all I'm seeing is the following:

GRASS 6.3.0RC3 (msunduzi_lo31):~ > r.patch input=$MAPS
output="aerial_mosaic.red" --overwrite --verbose
r.patch: percent complete:
  15%

Comments and pointers most welcome.

Craig
--
View this message in context: http://www.nabble.com/Working-with-a-large-number-of-maps-tp14384527p14384527.html
Sent from the Grass - Users mailing list archive at Nabble.com.

Hi,

2007/12/18, Craig Leat <Craig@pid.co.za>:

[snip]

Another question on the usage of r.patch:
I have passed the option --verbose to r.patch and I am expecting the program
to generate VERBOSE output, but all I'm seeing is the following:

GRASS 6.3.0RC3 (msunduzi_lo31):~ > r.patch input=$MAPS
output="aerial_mosaic.red" --overwrite --verbose
r.patch: percent complete:
  15%

In SVN-trunk:

$ export GRASS_VERBOSE=0;r.patch in=elevation.10m,elevation.dem out=x
[prints nothing]
$ export GRASS_VERBOSE=1;r.patch in=elevation.10m,elevation.dem out=x
100%
$ export GRASS_VERBOSE=2;r.patch in=elevation.10m,elevation.dem out=x
100%
Creating support files for raster map <x>
$ export GRASS_VERBOSE=3;r.patch in=elevation.10m,elevation.dem out=x
Percent complete...
100%
Creating support files for raster map <x>

Note that GRASS_VERBOSE=0 is the same as --q, GRASS_VERBOSE=1 -> --v.

What kind of verbose output are you expecting?

Martin

--
Martin Landa <landa.martin@gmail.com> * http://gama.fsv.cvut.cz/~landa *

Hi,

2007/12/18, Martin Landa <landa.martin@gmail.com>:

Note that GRASS_VERBOSE=0 is the same as --q, GRASS_VERBOSE=1 -> --v.

Ops, GRASS_VERBOSE=3 -> --v

See

http://grass.gdf-hannover.de/wiki/Development_Specs#Current_abstract_concept

Martin

--
Martin Landa <landa.martin@gmail.com> * http://gama.fsv.cvut.cz/~landa *

Martin Landa wrote:

Hi,

2007/12/18, Craig Leat <Craig@pid.co.za>:

[snip]

What kind of verbose output are you expecting?

Martin
  
Hi

I don't know what to expect, but was hoping for something like:
r.patch progress:
Currently processing map: abc.red, progress: 80%
Total progress: 15%

This may be something to be added to the wishlist?

Craig

Martin Landa wrote:

> What kind of verbose output are you expecting?

Craig:

I don't know what to expect, but was hoping for something like:
r.patch progress:
Currently processing map: abc.red, progress: 80%
Total progress: 15%

This may be something to be added to the wishlist?

--verbose is a no-op for most modules. The module needs to contain
G_verbose_message("some message here") calls in the code for --v to do
anything. That library function is very new (new for GRASS 6.3) so not
much uses it yet. There's an effort to move non-surprising status
messages to only show up in verbose mode, but it is early days for
that.

r.series works by row, then column, then by map. ie it works on all
maps at the same time (sorta) not sequentially. So G_percent() per row
is the best that can be done.

Your region is really really huge, I might suggest making 4 smaller
tiles out of it instead of one master map. I hope you compiled with LFS
(large file support) and your OS is 64bit.

By using 4 or more smaller region tiles to do the patch of non-
overlapping maps it may spend less time looking through empty space so
will be a lot faster. ?

You can use 'g.region -g' + 'r.info -g' to exclude maps which do not at
all fall in the output region. (this optimization might be built
directly into r.series?)

Alternate: let r.series run for x days.

Hamish

      ____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

Hi,

2007/12/18, Craig Leat <craig@pid.co.za>:

[snip]

I don't know what to expect, but was hoping for something like:
r.patch progress:
Currently processing map: abc.red, progress: 80%
Total progress: 15%

Since r.patch works by row (reads x-row of the first, second, ...
given raster map), G_percent() per row seems to be the best solution.

Martin

--
Martin Landa <landa.martin@gmail.com> * http://gama.fsv.cvut.cz/~landa *

Craig wrote:
<snip>

I can think of three workarounds.

1. Enter Rainer Krug and MapServer.
I don't think my time frames allow for learning another software suite right now.

2. Use a script to build a .grc file with a group for gis.m.
Before attempting this I would like to know if there are any limits to the number of layers that can be loaded.

3. Use r.patch to build one mosaic for each band. </snip>

Hamish wrote:

`<snip>
Your region is really really huge, I might suggest making 4 smaller
tiles out of it instead of one master map. I hope you compiled with LFS
(large file support) and your OS is 64bit.

`

I wasn't aware that 32 vs 64 bit architecture affected the maximum file size (RAM size - Yes). I read the following article and since I compiled with LFS on an ext3 filesystem I thought I was good for 2TB.
[https://help.ubuntu.com/community/LinuxFilesystemsExplained](https://help.ubuntu.com/community/LinuxFilesystemsExplained)?

`By using 4 or more smaller region tiles to do the patch of non-
overlapping maps it may spend less time looking through empty space so
will be a lot faster. ?

You can use 'g.region -g' + 'r.info -g' to exclude maps which do not at
all fall in the output region. (this optimization might be built
directly into r.series?)

Alternate: let r.series run for x days. </snip>
`

Thanks for the valuable suggestions. I'm still hoping to get some comment on workaround number two mentioned above. Does anyone know of any pitfalls with trying to load 139x3 layers into a group for gis.m? I'm wondering if there might be a limit on the number of layers that can be loaded or if loading so many layers might seriously affect the performance of the map display?

Craig

Craig Leat wrote:

I have 139 high resolution images, which I need to use as a background map
for some digitizing work. Loading 139x3 layers into gis.m is a real pain and
I can think of three workarounds.

3. Use r.patch to build one mosaic for each band.
I currently have r.patch running. It's been going for about 10.5hrs and has
reached 15%. Does this sound reasonable? I am starting to wonder if I'm
expecting too much. Here are my region settings:

GRASS 6.3.0RC3 (msunduzi_lo31):~ > g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: wgs84
ellipsoid: wgs84
north: -3264858.18753122
south: -3295545.14772988
west: -92005.31394386
east: -43582.78768323
nsres: 0.35000012
ewres: 0.35000019
rows: 87677
cols: 138350
cells: 12130112950

How do the individual maps relate to the overall region?

If they are (approximately) tiles, I would suggest using a
hierarchical approach, patching groups of adjacent tiles into larger
tiles, then patching the larger tiles together, so that each r.patch
operation has relatively few maps, and the earlier stages have
relatively small regions.

The total time taken by r.patch is roughly proportional to the total
number of cells in the region multiplied by the number of input maps,
so the combination of a large region and a large number of maps will
be slow.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements wrote:

Craig Leat wrote:
  

I have 139 high resolution images, which I need to use as a background map
for some digitizing work.
    

How do the individual maps relate to the overall region?

If they are (approximately) tiles, I would suggest using a
hierarchical approach, patching groups of adjacent tiles into larger
tiles, then patching the larger tiles together, so that each r.patch
operation has relatively few maps, and the earlier stages have
relatively small regions.

The total time taken by r.patch is roughly proportional to the total
number of cells in the region multiplied by the number of input maps,
so the combination of a large region and a large number of maps will
be slow.

Thanks for the advice Glynn. The images are tiles and the filenames follow a regular pattern, so I am able to patch blocks together quite easily with a simple script. I included lots of echos in the script and now I even have verbose messages :wink:

The script is running right now and making good progress...

Craig