re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE

GRASS GRASS user
,
1

Dear Markus and other Grass users,

Here comes the latest update about my high resolution topographic map
of Europe - as usual - with some new questions.

Part1 (pasting cgiar-srtm tiles together) was a real pain, since I
needed to recompile GDAL with BigTIFF support in GTiFF driver to be
able to create such large geotiff files. Of course, once new gdal
successfully installed, I had to recompile GRASS against the new
gdal-bigtiff...
As posted on the gdal-dev mailing list, strange things occured during
my various gdalwarp attemps: while pasting all tiles bewteen
{34N;60N;-11W;35E} together in one command takes about 20h, dividing
the job in:
a - first merge "slices" of saying all tiles located in one long. column
b - finally paste all "slices" together
reduces the time needed to complete the command to around 4-5h. (I
posted 1 hour completion-time in the gdal list, but it was on a
smaller region)

Part2 has been updated to import the geotiff file in a lat/long
projection location native to srtm data, and than reproject the
obtained raster into my selected lcc (lambert conical conformal)
projection. This was probably the longest part, since it took r.proj
about 8h to complete this task. It tooks me actually twice as long,
since my region bounds were not properly set to include the distortion
of the lcc projection, so that the first re-projected raster had been
cut off just over Denmark; and I had to launch the reprojection
again...

Part3 is about creating the shadings, still ongoing, and works
surprinsingly well with r.shaded.relief, tough I'm still messing
around with zmult and azimut options to get the desired shadings.
Should somebody have advices here, to get nice shadings for large
srtm-based topographic maps, feel free to post them.
@Markus: I did not try the gdaldem tool to create the shadings, since
I re-projected the DEM in GRASS raster format.

Part4 is were I'm stuck currently, and is about creating coastline
vectors, bathymetry and exporting the results to be re-worked/merged
in GIMP.
a - Importing the vectors from SWBD is no problem, tough It would be
nice to have the 200+ NASA shapefiles merged BEFORE importing a new
layer in GRASS. Is this possible with ogr2ogr ?
b - The big problem are coastlines and waterbodies (+main rivers):
somehow I have to show them on the topographic map, which gdalwarp has
filled out with -32768 values in nodata-waterzones. So either I cut
waterbodies out of the topographic raster along the vectors, or I
somehow have GRASS compute me all waterbodies from the vector layer,
fill them with a nice blue and create a raster which can be pasted
over the topographic raster to get the final map. It seems doable
with mapcalc, but frankly, I do not know at all how to proceed. Any
help here would be greatly appreciated.

Part5 will be about secondary rivers, which will either be taken
directly from VMAP0 hydro layer or extracted with r.watershed (thank
you Markus for pointing this possibility) following the nice little
tutorial provided in the latest r.watershed manpage (bottom of the
page).
I will probably tryout both on a small scale, and keep the one
offering the best output (at my scale). The plan is to create another
vector layer showing only secondary rivers. More on this to follow.

Part6/final: rescaling (optional), exporting and merging in GIMP.
Notice that - at least until now, very little rework is required so
that I might consider finishing the job with GRASS and export one
finished map.

Voila. I sincerely hope this progress report is of some interest,
Thank you for your time and patience when it comes to answer my
numerous questions,

regards,
Felix

2

Very nice!

b - The big problem are coastlines and waterbodies (+main rivers):
somehow I have to show them on the topographic map, which gdalwarp has
filled out with -32768 values in nodata-waterzones. So either I cut
waterbodies out of the topographic raster along the vectors, or I
somehow have GRASS compute me all waterbodies from the vector layer,

When I understand right, its easy. In case of having all vectors as
polygons, try "v.to.rast". Then you have a raster layer of desired
resolution (set with g.region).

fill them with a nice blue and create a raster which can be pasted
over the topographic raster to get the final map. It seems doable
with mapcalc, but frankly, I do not know at all how to proceed. Any
help here would be greatly appreciated.

When you want to overlay the topographic-map with water-information, be
aware what you want to do: when you just want the information, that
there is water, you are done with telling the new raster-layer to be
blue and transparent.
If you want to change the topology for calculation with r.watershed
(e.g. river-burning) or flatten lake-regions or set topography on
coast-lines to zero, it would be a bit more to do.

Cheers,
Achim

3

Hi Felix,

On Sun, Aug 16, 2009 at 7:37 PM, Felix Schalck<felix.schalck@gmail.com> wrote:
...

a - Importing the vectors from SWBD is no problem, tough It would be
nice to have the 200+ NASA shapefiles merged BEFORE importing a new
layer in GRASS. Is this possible with ogr2ogr ?

Merge of two SHAPE files 'file1.shp' and 'file2.shp' into a new file
'file_merged.shp' is performed like this:

# note order "out", then "in":
ogr2ogr file_merged.shp file1.shp
ogr2ogr -update -append file_merged.shp file2.shp -nln file_merged file2

The second command is opening file_merged.shp in update mode, and
trying to find existing layers and append the features being copied.
The -nln option sets the name of the layer to be copied to.

Attached a script to do as many as you want.

b - The big problem are coastlines and waterbodies (+main rivers):
somehow I have to show them on the topographic map, which gdalwarp has
filled out with -32768 values in nodata-waterzones. So either I cut
waterbodies out of the topographic raster along the vectors, or I
somehow have GRASS compute me all waterbodies from the vector layer,
fill them with a nice blue and create a raster which can be pasted
over the topographic raster to get the final map. It seems doable
with mapcalc, but frankly, I do not know at all how to proceed. Any
help here would be greatly appreciated.

Not sure, but set -32768 to NULL (no data) with r.null?
Then use r.colors (nv to set color for NULL):
http://grass.osgeo.org/grass64/manuals/html64_user/r.colors.html

cheers
Markus

(attachments)

ogr_shape_merge.sh (724 Bytes)

4

Dear All-who-may-be-intersted,

First, I'd like to apologize for the delay: lots of stuff prevented me
from working on the map. I've got some spared time now, and the map
isn't finished yet - so I'm back into buisiness!
Secondly, I'd like to thank you Markus, for the great script you sent
me: it worked withouth a hitch, and I have now a single big shapefile
to work on. And sorry for the last message: it was a misclick.

Then comes the map: basically, even though the last replies provided
my with some valuable advices, I'm still stuck with bathymetry -
rivers and coastlines.

1. For the bathymetry, I finally took ETOPO1 data, which gives me
another nice raster, although of much lower resolution (1' compared to
the 3" topographic raster from SRTM DEM). So the plan is to cut off
all the land data from the ETOPO1 raster along the coastlines, to keep
only the sea aeras and than paste this reduced raster onto the main
SRTM topographic raster of greater resolution. Can this step be
automated ? (eg: is there a tool to do the job ?) Of course, should
raster-merge be to complicated, there is always the other option,
which is to export pngs first, and paste the pngs together; but in
both cases, the bathymetric raster has to be cut.

2. Rivers and coastlines are a real pain.
a. I could import a big pasted SWBD shapefile thanks to Markus script,
but the cleaning process of the import command takes days, and gets
somehow stuck within the brigdge removing phase. Result is an
uncomplete vector map, lacking centroids, which can't be further
processed by v.generalize for smoothing (the command dies saying there
is an error in input data), but can be displayed with v.display.
Another problem are all the square borders left around former SWBD
tiles in water aeras; can they be removed ? Or do I have to manually
edit the (huge) vector map ?

b. Then come the rivers: the (somehow corrupted) SWBD vector map shows
only coastlines, smaller closed waterbodies and - although only
partially - larger rivers. I somehow have to complete the river data,
and try following methods:
-r.watershed command in grass, to compute the rivers from DEM data.
This command just dies during the memory allocation process (KILL
signal) because the map is too huge for my system. I tried with
differend -m parameters, but the command always askes for up to 10gb
virtual memory the kernel won't be able to provide. I guess I would
have to work on smaller pieces of the map, set with g.region, and
paste the results together, but this is another time consuming option.
-VMAP0 data import works, but the result is quite disappointing. The
secondary rivers network seems quite good, but this datased is unable
to fix the uncomplete major rivers from the SWBD dataset. For an
instance, by showing both layers (SWBD + VMAP0) on the monitor, I get
all smaller rivers flowing into the rhine, while the rhine itself
remains divided into several un-joined segments.
-OSM(openstreetmap) data is a bit confusing. First it is divided along
countries, so you have to dowload and paste lots of different files.
Than the" Waterway" shapefile is quite poor. And finally the "natural"
shapefile comes with nice river data, but also a lot of confusing
data, like forest aeras. This need some sort of filtering, but I don't
know at all how to do this.

c. Of course, if someone knows better river datasets (scale approching
3", complete, easy to use), I would be happy to try them.

Voila. Even though a lot more problems arise during each step
(resolution has become another one, for an instance), I hope this
project will see an end soon.
Thanks for your help and your patience,

Felix

2009/8/17 Markus Neteler <neteler@osgeo.org>:

Hi Felix,

On Sun, Aug 16, 2009 at 7:37 PM, Felix Schalck<felix.schalck@gmail.com> wrote:
...

a - Importing the vectors from SWBD is no problem, tough It would be
nice to have the 200+ NASA shapefiles merged BEFORE importing a new
layer in GRASS. Is this possible with ogr2ogr ?

Merge of two SHAPE files 'file1.shp' and 'file2.shp' into a new file
'file_merged.shp' is performed like this:

# note order "out", then "in":
ogr2ogr file_merged.shp file1.shp
ogr2ogr -update -append file_merged.shp file2.shp -nln file_merged file2

The second command is opening file_merged.shp in update mode, and
trying to find existing layers and append the features being copied.
The -nln option sets the name of the layer to be copied to.

Attached a script to do as many as you want.

b - The big problem are coastlines and waterbodies (+main rivers):
somehow I have to show them on the topographic map, which gdalwarp has
filled out with -32768 values in nodata-waterzones. So either I cut
waterbodies out of the topographic raster along the vectors, or I
somehow have GRASS compute me all waterbodies from the vector layer,
fill them with a nice blue and create a raster which can be pasted
over the topographic raster to get the final map. It seems doable
with mapcalc, but frankly, I do not know at all how to proceed. Any
help here would be greatly appreciated.

Not sure, but set -32768 to NULL (no data) with r.null?
Then use r.colors (nv to set color for NULL):
http://grass.osgeo.org/grass64/manuals/html64_user/r.colors.html

cheers
Markus