[GRASS-user] WMS fails on Geosciences Australia downloads

Richard2 · May 12, 2008, 9:57pm

Hi all,

I am trying to access layers on the Geosciences Australia OGC Web Map server:
http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities

Here is the input as generated by the r.in.wms GUI:
r.in.wms output=test mapserver=http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities layers=60 srs=EPSG:18001 format=png wms query=version=1.1.1 maxcols=1024 maxrows=1024 ‘wgetoptions=-c -t 5 --user-agent=MSIE5.5’ method=nearest v=1

This fails with a bunch of errors, partly copied below. The same thing happens running r.in.wms using a single quote as the delimiter.

wms.request ‘folder=/home/richard/grassdata/wms_download’ ‘prefix=test’ ‘mapserver=http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities’ ‘layers=1’ ‘srs=EPSG:18001’ ‘format=geotiff’ ‘wmsquery=version=1.1.1’ ‘maxcols=1024’ ‘maxrows=1024’
###############################
Calculating tiles
###############################
Rel. 4.5.0, 22 Oct 2006
:
projection initialization failure
cause: no options found in ‘init’ file
program abnormally terminated
(standard_in) 1: illegal character: U
Rel. 4.5.0, 22 Oct 2006
:
projection initialization failure
cause: no options found in ‘init’ file
program abnormally terminated
(standard_in) 1: illegal character: U
Rel. 4.5.0, 22 Oct 2006
:
projection initialization failure
cause: no options found in ‘init’ file
program abnormally terminated
(standard_in) 1: illegal character: U
Rel. 4.5.0, 22 Oct 2006
:
projection initialization failure
cause: no options found in ‘init’ file
program abnormally terminated
(standard_in) 1: illegal character: U
(standard_in) 1: parse error

Is it a bug in Grass-GIS or at the other end?

Richard

Hamish · May 13, 2008, 12:43pm

Richard Chirgwin wrote:

I am trying to access layers on the Geosciences Australia
OGC Web Map server:
http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities

....

Is it a bug in Grass-GIS or at the other end?

I'll be kind and say GRASS because /technically/ the other end conforms to the WMS spec. The problem was that the other end does not accept queries in the form of HTTP POST data. Yes, a query server which doesn't accept queries. !@$##$%

Anyway, in the past I tried to auto-detect that and switch to the other method, but the error mode the remote server sends seems to vary for each, so it is hard to do that well. I've now added a new flag to r.in.wms (-g) to force using the GET method. With that I can fetch data from Geosciences Australia's WMS:

r.in.wms -l -g mapserv="http://www.ga.gov.au/wms/getmap?dataset=national&"

# search for Gold:
r.in.wms mapserv="http://www.ga.gov.au/wms/getmap?dataset=national&" \
-g layers=au_occdd output=au_gold format=tiff

I also discovered that Geosciences Australia's WMS server does not ship GeoTiffs, which is r.in.wms's default. So you'll have to use format=<other>.
An issue with format=tiff is that the alpha (transparency) layer that comes from Geosci. Aust. runs from 1-255 instead of 0-255, and so no-data is treated as some-data and the resulting data looks a bit funky. That is easily fixed with r.mapcalc or r.null after import or in the case of the Gold, a simple call to r.colors:
echo "1023 yellow" | r.colors au_gold col=rules

format=gif and png are really slow as the individual tiles use their own custom 8bit color palettes and when merged together the palette from the first tile is used and the rest of the tiles look nuts. So r.in.gdalwarp goes the long way to ensure good results, which is very very slow. gdal_merge.py is probably a much better method for the future.

format=jpeg now works (fixed in latest commit; it has no alpha layer), but the results are subject to lossy JPEG compression/blurring.

yikes.

If you get the contents of the scripts/r.in.wms/ dir from develbranch_6 and put replace 6.3.0's versions and re-make that dir you should be able get it to work (proof attached
http://trac.osgeo.org/grass/browser/grass/branches/develbranch_6/scripts/r.in.wms

calmly,
Hamish

ps- sorry if this email is wrapping at about char 90, yahoo as upgraded their system & removed the "wrap at 72" compose setting, I suppose to make things simpler for us.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

(attachments)

Richard2 · May 14, 2008, 3:50am

Hamish wrote:

Richard Chirgwin wrote:

I am trying to access layers on the Geosciences Australia
OGC Web Map server:
[http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities](http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities)

....

Is it a bug in Grass-GIS or at the other end?


I'll be kind and say GRASS because /technically/ the other end conforms to the WMS spec. The problem was that the other end does not accept queries in the form of HTTP POST data. Yes, a query server which doesn't accept queries. !@$##$%

Anyway, in the past I tried to auto-detect that and switch to the other method, but the error mode the remote server sends seems to vary for each, so it is hard to do that well. I've now added a new flag to r.in.wms (-g) to force using the GET method. With that I can fetch data from Geosciences Australia's WMS:

r.in.wms -l -g mapserv=["http://www.ga.gov.au/wms/getmap?dataset=national&"](http://www.ga.gov.au/wms/getmap?dataset=national&)

# search for Gold:
r.in.wms mapserv=["http://www.ga.gov.au/wms/getmap?dataset=national&"](http://www.ga.gov.au/wms/getmap?dataset=national&) \
   -g layers=au_occdd output=au_gold format=tiff

I also discovered that Geosciences Australia's WMS server does not ship GeoTiffs, which is r.in.wms's default. So you'll have to use format=<other>.
An issue with format=tiff is that the alpha (transparency) layer that comes from Geosci. Aust. runs from 1-255 instead of 0-255, and so no-data is treated as some-data and the resulting data looks a bit funky. That is easily fixed with r.mapcalc or r.null after import or in the case of the Gold, a simple call to r.colors:
  echo "1023 yellow" | r.colors au_gold col=rules

format=gif and png are really slow as the individual tiles use their own custom 8bit color palettes and when merged together the palette from the first tile is used and the rest of the tiles look nuts. So r.in.gdalwarp goes the long way to ensure good results, which is very very slow. gdal_merge.py is probably a much better method for the future.

format=jpeg now works (fixed in latest commit; it has no alpha layer), but the results are subject to lossy JPEG compression/blurring.

yikes.

If you get the contents of the scripts/r.in.wms/ dir from develbranch_6 and put replace 6.3.0's versions and re-make that dir you should be able get it to work (proof attached :)
[http://trac.osgeo.org/grass/browser/grass/branches/develbranch_6/scripts/r.in.wms](http://trac.osgeo.org/grass/browser/grass/branches/develbranch_6/scripts/r.in.wms)

calmly,
Hamish

ps- sorry if this email is wrapping at about char 90, yahoo as upgraded their system & removed the "wrap at 72" compose setting, I suppose to make things simpler for us.

I didn’t notice the mail wrapping issue …

Thanks for all your efforts. There’s still something strange at the Geosciences Australia end; I have repeated your test with the Gold layer (success). But other layers show errors like “wms_roads__0.png not recognised as a supported file format”. So the server would have to be considered “a little flaky” on a good day.

The tiles download okay - the directory shows files with .wget, .png (or tiff or whatever), .proj4 and .tfw formats, so I guess something is breaking / broken in the files GA is sending.

Oh well …

Richard

Hamish · May 14, 2008, 7:56am

Richard Chirgwin wrote:

I am trying to access layers on the Geosciences Australia
OGC Web Map server:

http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities
.....
Hamish:

I'll be kind and say GRASS because /technically/ the other end conforms
to the WMS spec. The problem was that the other end does not accept
queries in the form of HTTP POST data.

[r.in.wms -g flag added to deal with that; this works:]

r.in.wms -l -g mapserv="http://www.ga.gov.au/wms/getmap?dataset=national&"

# search for Gold:
r.in.wms mapserv="http://www.ga.gov.au/wms/getmap?dataset=national&" \
-g layers=au_occdd output=au_gold format=tiff

....

ps- sorry if this email is wrapping at about char 90, yahoo as upgraded
their system & removed the "wrap at 72" compose setting, I suppose to
make things simpler for us.

[I hope to get back to using fetchyahoo.pl+cron and a sane client soon]

Richard:

I didn't notice the mail wrapping issue ...

yours is missing newlines- even worse!
(hint: don't post as HTML/rich formatted)

Thanks for all your efforts. There's still something strange at the
Geosciences Australia end; I have repeated your test with the Gold
layer (success). But other layers show errors like "wms_roads__0.png
not recognised as a supported file format".

if you look in that file you will see text not an image:
<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<!DOCTYPE ServiceExceptionReport SYSTEM "http://schemas.opengeospatial.net/wms/1.1.1/exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1">
<ServiceException code="InvalidSRS">
msWMSLoadGetMapParams(): WMS server error. Invalid SRS given : SRS must be valid for all requested layers.
</ServiceException>
</ServiceExceptionReport>

be sure to delete those old files from $GISDBASE/wms_downloads/ or they will be cached and it'll try to reuse them. ('r.in.wms -c' helps)

'r.in.wms -l' and http://www.ga.gov.au/map/broker/wms_info.php state that the roads layer is in EPSG:3112, so you need to provide that on the command line:

# from a latlon/wgs84 location
g.region n=9S s=46S w=112E e=155E res=0:01 -p
r.in.wms mapserv="http://www.ga.gov.au/wms/getmap?dataset=national&" \
-g layers=roads output=au_roads format=tiff srs=EPSG:3112

So the server would have to be considered "a little flaky" on a good day.

naw, blame r.in.wms. the server is limited in what it will provide and r.in.wms is not handling the error messages well. (the error can come in the form of a web page, XML, or empty output, and success is a real image, so it is hard to pick how to recognize and pass on the error)

devs: One problem is that I don't know how to test for r.in.gdalwarp's exit code. it is called from r.in.wms with eval- the exit code returned is eval's not r.in.gdalwarp's.

The tiles download okay - the directory shows files with .wget, .png
(or tiff or whatever), .proj4 and .tfw formats, so I guess something is
breaking / broken in the files GA is sending.

no, open the tile in a text editor and see

with the extra "srs=EPSG:3112" on the command line I could download the roads layer and it looks good.

Hamish

Richard2 · May 14, 2008, 9:02am

Hamish wrote:

Richard Chirgwin wrote:

I am trying to access layers on the Geosciences Australia
OGC Web Map server:

[http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities](http://www.ga.gov.au/wms/getmap?dataset=national&request=getCapabilities)
.....
Hamish:

I'll be kind and say GRASS because /technically/ the other end conforms
to the WMS spec. The problem was that the other end does not accept
queries in the form of HTTP POST data.

[r.in.wms -g flag added to deal with that; this works:]

r.in.wms -l -g mapserv=["http://www.ga.gov.au/wms/getmap?dataset=national&"](http://www.ga.gov.au/wms/getmap?dataset=national&)

# search for Gold:
r.in.wms mapserv=["http://www.ga.gov.au/wms/getmap?dataset=national&"](http://www.ga.gov.au/wms/getmap?dataset=national&) \
  -g layers=au_occdd output=au_gold format=tiff

....

ps- sorry if this email is wrapping at about char 90, yahoo as upgraded
their system & removed the "wrap at 72" compose setting, I suppose to
make things simpler for us.

[I hope to get back to using fetchyahoo.pl+cron and a sane client soon]

Richard:

I didn't notice the mail wrapping issue ...


yours is missing newlines- even worse!  :)
(hint: don't post as HTML/rich formatted)

Thanks for all your efforts. There's still something strange at the
Geosciences Australia end; I have repeated your test with the Gold
layer (success). But other layers show errors like "wms_roads__0.png
not recognised as a supported file format".


if you look in that file you will see text not an image:
<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<!DOCTYPE ServiceExceptionReport SYSTEM ["http://schemas.opengeospatial.net/wms/1.1.1/exception_1_1_1.dtd"](http://schemas.opengeospatial.net/wms/1.1.1/exception_1_1_1.dtd)>
<ServiceExceptionReport version="1.1.1">
<ServiceException code="InvalidSRS">
msWMSLoadGetMapParams(): WMS server error. Invalid SRS given : SRS must be valid for all requested layers.
</ServiceException>
</ServiceExceptionReport>

be sure to delete those old files from $GISDBASE/wms_downloads/ or they will be cached and it'll try to reuse them. ('r.in.wms -c' helps)

'r.in.wms -l' and  [http://www.ga.gov.au/map/broker/wms_info.php](http://www.ga.gov.au/map/broker/wms_info.php) state that the roads layer is in EPSG:3112, so you need to provide that on the command line:

# from a latlon/wgs84 location
g.region n=9S s=46S w=112E e=155E res=0:01 -p
r.in.wms mapserv=["http://www.ga.gov.au/wms/getmap?dataset=national&"](http://www.ga.gov.au/wms/getmap?dataset=national&) \
  -g layers=roads output=au_roads format=tiff srs=EPSG:3112

Richard:
You’re right; so the trick is to be careful!

I hope this is the last maddening issue; working with:
g.region n=33S s=34S w=150 e=151 nsres=00:00:05 ewres=00:00:05
r.in.wms -c mapserv=“http://www.ga.gov.au/wms/getmap?dataset=national&” -g layers=dem_9sec output=dem_9sec format=png srs=EPSG:3112
(The small region keeps the the tile size down)

I get the 9-second DEM. It looks and displays okay, but the elevation data is wild.
r.what -f input=dem_9sec@PERMANENT east_north=150.8177882,-33.785157971
produces:
150.8177882|-33.785157971||31407|

Since I can reasonably assume that the elevation isn’t 31,407 metres, nor that the elevation in the mountains is lower than near the ocean, something’s awry with the elevation… I will test it later with TIFF to see if that works better on elevation, I’m getting called away at the moment…

Richard

Hamish · May 14, 2008, 9:40am

Richard:

I hope this is the last maddening issue; working with:
g.region n=33S s=34S w=150 e=151 nsres=00:00:05 ewres=00:00:05

r.in.wms -c
mapserv="http://www.ga.gov.au/wms/getmap?dataset=national&"
-g layers=dem_9sec output=dem_9sec format=png srs=EPSG:3112
(The small region keeps the the tile size down)

I get the 9-second DEM. It looks and displays okay, but the
elevation data is wild.

r.what -f input=dem_9sec@PERMANENT
east_north=150.8177882,-33.785157971
produces:
150.8177882|-33.785157971||31407|

Since I can reasonably assume that the elevation isn't
31,407 metres, nor that the elevation in the mountains
is lower than near the ocean, something's awry with the
elevation...

I am afraid you are not looking at the elevation, but rather the color palette entry of the PNG file. ie this is a "visual only" exercise.
WMS servers can dish up real data, but I suspect that to do that they need to support geotiff format, and the GA server only provides tiff/png/jpeg/wbmp pretty imagery. I'm not sure about that though, maybe someone reading knows better about this.

if you look in the capabilities XML file you will see that geotiff is conspicuously missing from the available image formats:
http://www.ga.gov.au/wms/getmap?dataset=national&request=capabilities

If you find no joy, I'd suggest contacting Geosci Aust and asking about it; and also on the Aust-NZ OSGeo mailing list as well. There are some talented people at both places who may be able and keen to help.

Hamish

Richard2 · May 14, 2008, 10:29am

Hamish wrote:

Richard:

I hope this is the last maddening issue; working with:
g.region n=33S s=34S w=150 e=151 nsres=00:00:05 ewres=00:00:05

r.in.wms -c
 mapserv=["http://www.ga.gov.au/wms/getmap?dataset=national&"](http://www.ga.gov.au/wms/getmap?dataset=national&)
 -g layers=dem_9sec output=dem_9sec format=png srs=EPSG:3112
(The small region keeps the the tile size down)

I get the 9-second DEM. It looks and displays okay, but the
elevation data is wild.

r.what -f input=dem_9sec@PERMANENT
east_north=150.8177882,-33.785157971
produces:
150.8177882|-33.785157971||31407|

Since I can reasonably assume that the elevation isn't
31,407 metres, nor that the elevation in the mountains
is lower than near the ocean, something's awry with the
elevation...


I am afraid you are not looking at the elevation, but rather the color palette entry of the PNG file. ie this is a "visual only" exercise.
WMS servers can dish up real data, but I suspect that to do that they need to support geotiff format, and the GA server only provides tiff/png/jpeg/wbmp pretty imagery. I'm not sure about that though, maybe someone reading knows better about this.

Damn, I think you’re right. The TIFF, which I thought included elevation data, seems to suffer a similar problem. Still, all is not lost, because I can get the elevation data from a vector map of the relevant area and create a real DEM …

Richard