[GRASS-dev] r.in.wms improvements (call for testing)

Hi,

Yesterday I had some trouble connecting to NOAA's Electronic Navigation
Chart(ENC) WMS server[1] using r.in.wms. It turns out that their WMS
server poses some challenges for us. Their ArcIMS system will not handle
POST-data, it will only respond to very long URL strings (GET requests)
[2]. Also their list of layer names include URL problematic characters
like spaces and parser problematic chars like commas[3].

In SVN/trunk I've now adjusted the code to deal with both of those
things, although I haven't done anything for commas in the layer names -
those will still be parsed incorrectly[4]. Also the script is now
slightly less chatty by default and writes metadata to the output file.

Could folks test it please? If it's good it might be worth backporting
for 6.3.0, but it would need a lot of testing before considering that.
Agustin, maybe this solves your problem?

I also tried to fix the tile patching for single-band image. The ArcIMS
server would only send JPEGs or 8bit PNG & GIF images (no GeoTIFF). The
JPGs split into three RGB bands and patch correctly, but the 8bit PNG and
GIF tiles have per-tile unique color indexes; r.patch uses the index
from the first, resulting in funky weirdness. The current solution is a
hack using r.mapcalc's r#,g#,b# operators to split into three bands, then
do the patching, and then r.compositing. Other ideas for solving that:

1- add a -c flag to r.patch to patch by common color not by cat number.

2- use the PNG driver to 'd.rast -o' all the tiles, set GRASS_WIDTH and
  GRASS_HEIGHT to the region dims, then run d.out.file + 'r.in.gdal -o' +
  r.region.

3- Use echo "`seq 0 255`" | r.what.color -i in=tile_0 (or colr/ files
  directly) to get color tables for each tile, then for all the other
  tiles do a search loop for each RGB value and create reclass rules
  for each additional tile then run r.patch and copy the color index
  rules from the first map.

4. (what I did) For each tile extract 3 new 0-255 maps with r.mapcalc's
  r#, g#, b# operators, patch all r, g, b tiles together, then recombine
  into a single map with r.composite.

5. Use gdal_merge.py (but does it have a preserve color mode?)

6. Try NetPBM's pnmcat on raw downloaded tiles before loading into GRASS.
   This is what NVIZ uses to assemble the max. res PPM output.
   (but does it have a preserve color mode?)

"1" would be the best but I expect easier said than done. [beyond me]
"2" is probably the quickest but dirtiest [opens to display lib pains]
"3" has elegance but may be slow [perhaps not so slow?]
"4" slow but functional hack
"5" and "6" on the raw input would probably be very fast, but it is
    unknown to me if they would suffer the same multiple color index
    problem as r.patch.

TODO:
* rewrite the thing in Python (any volunteers? what is the QGIS plugin
   written in? Python or C++? [reuse whatever we can])
* See if GDAL 1.5.0's new WMS function could help simplify the task

[1] NOAA's WMS:
http://ocs-spatial.ncd.noaa.gov/website/encdirect/help/helpfile.htm#Webser
http://ocs-spatial.ncd.noaa.gov/wmsconnector/com.esri.wms.Esrimap/encdirect?

Some nice layers to get started with:
"layers=\
LAND,\
2DEPTH AREA(DEPARE),\
2SEA AREA(SEAARE),\
2SEABED AREA_polygon(SBDARE),\
2SOUNDINGS,\
DEPTH AREA(DEPARE)"

[2] the OGC WMS def ver 1.3.0 says that is allowed, see sec. 6.3.4
[3] the layer names are a mixed bag, most use _ not ' ', only one comma
[4] will GRASS's parser handle escaped \, in a multiple answers list?
    can it be simply done in a shell script? ('s+\\,+%2C+g' is easy
    enough but how to have that ignore a real '\\,' in the layer name?)
    and that means the use will have to munge the layer name manually :confused:

sorry for the long post on a high traffic day,
Hamish

I can't r.in.wms yet (william's svn build, so I'm not absolutely sure if your changes are there)
r.in.wms -l mapserver=http://www.idee.es/wms/PNOA/PNOA srs=EPSG:23030 format=png wmsquery=version=1.1.1 maxcols=1024 maxrows=1024 'curloptions=-C - --retry 5 -s -S' method=nearest --verbose
Using WGET for downloading data.
List of layers for server <http://www.idee.es/wms/PNOA/PNOA&gt;:

18:51:17 URL:http://www.idee.es/wms/PNOA/PNOA [448/448] -> "/Volumes/LaCieDisk/Users/Shared/agus_compartido/grassdata/Projecte/adiez2/.tmp/regadiuet.prearq.uv.es/4333.0capabilities.xml" [1]
wget -c -t 5 -nv --post-data="service=WMS&request=GetCapabilities&version=1.1.1" "http://www.idee.es/wms/PNOA/PNOA&quot; -O "/Volumes/LaCieDisk/Users/Shared/agus_compartido/grassdata/Projecte/adiez2/.tmp/regadiuet.prearq.uv.es/4333.0capabilities.xml";

ERROR: Parsing XML file
------------------------
<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<!DOCTYPE ServiceExceptionReport SYSTEM
  "http://www.idee.es/SgdWms/Server/exception_1_1_1.dtd&quot;&gt;
<ServiceExceptionReport version="1.1.1">
   <ServiceException code="InvalidFormat">
     <![CDATA[
   Par?metros:
     REQUEST INEXISTENTE O INVALIDA
     Verticesdisponibles:
     ]]>
   </ServiceException>
</ServiceExceptionReport>

On Mar 5, 2008, at 9:36 AM, Hamish wrote:

Hi,

Yesterday I had some trouble connecting to NOAA's Electronic Navigation
Chart(ENC) WMS server[1] using r.in.wms. It turns out that their WMS
server poses some challenges for us. Their ArcIMS system will not handle
POST-data, it will only respond to very long URL strings (GET requests)
[2]. Also their list of layer names include URL problematic characters
like spaces and parser problematic chars like commas[3].

In SVN/trunk I've now adjusted the code to deal with both of those
things, although I haven't done anything for commas in the layer names -
those will still be parsed incorrectly[4]. Also the script is now
slightly less chatty by default and writes metadata to the output file.

Could folks test it please? If it's good it might be worth backporting
for 6.3.0, but it would need a lot of testing before considering that.
Agustin, maybe this solves your problem?

I also tried to fix the tile patching for single-band image. The ArcIMS
server would only send JPEGs or 8bit PNG & GIF images (no GeoTIFF). The
JPGs split into three RGB bands and patch correctly, but the 8bit PNG and
GIF tiles have per-tile unique color indexes; r.patch uses the index
from the first, resulting in funky weirdness. The current solution is a
hack using r.mapcalc's r#,g#,b# operators to split into three bands, then
do the patching, and then r.compositing. Other ideas for solving that:

1- add a -c flag to r.patch to patch by common color not by cat number.

2- use the PNG driver to 'd.rast -o' all the tiles, set GRASS_WIDTH and
GRASS_HEIGHT to the region dims, then run d.out.file + 'r.in.gdal -o' +
r.region.

3- Use echo "`seq 0 255`" | r.what.color -i in=tile_0 (or colr/ files
directly) to get color tables for each tile, then for all the other
tiles do a search loop for each RGB value and create reclass rules
for each additional tile then run r.patch and copy the color index
rules from the first map.

4. (what I did) For each tile extract 3 new 0-255 maps with r.mapcalc's
r#, g#, b# operators, patch all r, g, b tiles together, then recombine
into a single map with r.composite.

5. Use gdal_merge.py (but does it have a preserve color mode?)

6. Try NetPBM's pnmcat on raw downloaded tiles before loading into GRASS.
  This is what NVIZ uses to assemble the max. res PPM output.
  (but does it have a preserve color mode?)

"1" would be the best but I expect easier said than done. [beyond me]
"2" is probably the quickest but dirtiest [opens to display lib pains]
"3" has elegance but may be slow [perhaps not so slow?]
"4" slow but functional hack
"5" and "6" on the raw input would probably be very fast, but it is
   unknown to me if they would suffer the same multiple color index
   problem as r.patch.

TODO:
* rewrite the thing in Python (any volunteers? what is the QGIS plugin
  written in? Python or C++? [reuse whatever we can])
* See if GDAL 1.5.0's new WMS function could help simplify the task

[1] NOAA's WMS:
http://ocs-spatial.ncd.noaa.gov/website/encdirect/help/helpfile.htm#Webser
http://ocs-spatial.ncd.noaa.gov/wmsconnector/com.esri.wms.Esrimap/encdirect?

Some nice layers to get started with:
"layers=\
LAND,\
2DEPTH AREA(DEPARE),\
2SEA AREA(SEAARE),\
2SEABED AREA_polygon(SBDARE),\
2SOUNDINGS,\
DEPTH AREA(DEPARE)"

[2] the OGC WMS def ver 1.3.0 says that is allowed, see sec. 6.3.4
[3] the layer names are a mixed bag, most use _ not ' ', only one comma
[4] will GRASS's parser handle escaped \, in a multiple answers list?
   can it be simply done in a shell script? ('s+\\,+%2C+g' is easy
   enough but how to have that ignore a real '\\,' in the layer name?)
   and that means the use will have to munge the layer name manually :confused:

sorry for the long post on a high traffic day,
Hamish
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Agustin Diez Castillo wrote:

I can't r.in.wms yet (william's svn build, so I'm not absolutely sure
if your changes are there)

r.in.wms -l mapserver=http://www.idee.es/wms/PNOA/PNOA srs=EPSG:23030
format=png wmsquery=version=1.1.1 maxcols=1024 maxrows=1024
'curloptions=-C - --retry 5 -s -S' method=nearest --verbose

....

ERROR: Parsing XML file
------------------------
<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<!DOCTYPE ServiceExceptionReport SYSTEM
  "http://www.idee.es/SgdWms/Server/exception_1_1_1.dtd&quot;&gt;
<ServiceExceptionReport version="1.1.1">
   <ServiceException code="InvalidFormat">
     <![CDATA[
   Par?metros:
     REQUEST INEXISTENTE O INVALIDA
     Verticesdisponibles:
     ]]>
   </ServiceException>
</ServiceExceptionReport>

well good news and bad.

The error you see does seem to be due to r.in.wms using the POST-data
method instead of the atomic-GET method and the server not liking that.

By putting the request all on one line I was able to get the
capabilities.xml file:

SERVER="http://www.idee.es/wms/PNOA/PNOA?&quot;
REQUEST="service=WMS&request=GetCapabilities&version=1.1.1"
wget "${SERVER}${REQUEST}" -O pnoa_capabilities.xml

versus
wget --post-data="${REQUEST}" "${SERVER}" -O pnoa_capabilities.xml

After adding an elaborate hack to 'r.in.wms -l' to try the GET method
if the POST-data method XML parsing failed I could get it to download
and parse the layers list.

I then tried doing the same with r.in.wms's wms.download module,
(I added a -g force-GET flag)

r.in.wms mapserver="$SERVER" format=png srs=EPSG:23030 \
  out=test layer=pnoa -p -c -g

It seemed to download something and seemed to process it, but I got an
empty map.

Do you have a useful region for EPSG:23030? I tried:
  n=3741500 s=3649400 w=-554200 e=-427300 res=100

maybe that was why the output map was empty?

It's all getting a bit of a mess and I'm a bit hesitant to commit the
needed fixes as they are somewhere between a kludge and a hack-- the
sooner we get r.in.wms rewritten in Python and/or using GDAL 1.5.0's
fetch method, the better.

Hamish

      ____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping