[GRASS5] Re: [GRASS] WMS -> CVS

Hi,

> > what about r|v.in.wfs ? Would it be nice in GRASS too?
>
> I didn't know those modules existed or even that WFS existed...

Stephan or Markus wrote it -- I just know, there is some version arround

It would be nice if the wording on both r.in.wms and v.in.wfs for servers, SRS
projections, etc. are in sync before releasing anything.

> > What version of r.in.wms|onearth should go in GRASS cvs must deside
> > someone else :slight_smile: I tryed to write it as simple, as possible - the user
> > should deside/script, if he wants to use tiles. The user *knows* in
> > which projection is he working etc.
> >
> > Current version should be SUBMITTING_SCRIPTS-ready. Avaliable at
> > http://les-ejk.cz/programs/grass/r.in.wms.tgz
>
> Well I have only used r.in.onearth, and found it to be a great tool, but
> I know you were all working on different versions and was hoping you
> could tell me which was the one best suited for general use.
>
> Hamish

I will not tell you, which one is the best, but I try to describe
general properties of these tools and to compare them side by side:

1. r.in.onearth - it was the first version of this tools. it is able to
                  download the maps *only* from one server. There should
                  be some support of Cedric's tiles-workaround.

2. "jachym's" version of r.in.wms - is based on the version of
        r.in.onearth. It can download layers from *any* server. I added
        the flag -l for list of layers. If there is proper instalation
        of python in your box, the output can be nicer.
        In my script, I do not care about tiling and reprojecting - user
        knows, what is is doing (but if you use Lat/Long projectino for
        downloaded data, they will fitt your region corectly)

3. "Cedric's" version of r.in.wms - (I saw it only ones) there is
        support of tiling. r.in.wms combines more scripts together,
        which makes the whole proces IMHO too complicated. I do
        remeber, there was some strange way, how the rasters were
        imported and reprojected to GRASS (creating new teporary
        locations?) - but it is possible, that things are made some other
        way now.

"Cedric's" version is a compilation of "jachym's" and r.in.onearth. There are,
if I recall correctly, five scripts involved in the main event, and one for
r.in.onearth. I did this to separate the code out into pieces. The scripts
are something like:

r.tileset which generates tilings in a general way.

wms.request which generates requests to wms servers. This is heavily lifted
from r.in.onearth and "jachym's" version.

wms.download which downloads requests (a very simple wrapper around wget).

r.in.gdalwarp which imports and patches many files using gdalwarp to reproject
data if necessary (pretty much lifted from r.in.onearth).

and r.in.wms which is a simple wrapper around the first four and what people
would usually use.

Finally it has another r.in.onearth that Sören wrote that wraps up this
r.in.wms.

Most everything here (except for tiling) was lifted straight from jachym's and
Sören's work, which made most of the programming quite easy.

One thing this set is missing (besides html documentation and makefiles) is
some of the interaction with the server that "jachym's" does (listing
layers). I was kind of hoping to parse and comprehend the entire capabilities
request like I did with "raster plaster" but the difficulty and dependencies
of parsing XML in shell scripts was too high. It would be pretty easy to
merge this functionality from jachym's latest scripts.

Separating the scripts clarifies and enforces separations in code. It also
allows more possible scripting by users in terms of downloads, retrying
downloads, etc. It would, however, be nice if wms.request, wms.download, and
r.in.gdalwarp could be slightly less exposed (though no less documented).

--Cedric

It would be nice if the wording on both r.in.wms and v.in.wfs for
servers, SRS projections, etc. are in sync before releasing anything.

Very well, but this is up to the authors of the scripts to understand
and decide... if this doesn't happen very soon (ie now) the scripts will
not be in by the feature freeze for 6.2.

"Cedric's" version is a compilation of "jachym's" and r.in.onearth.

ok, then "Cedric's" is the most featureful one & what we should use.
?

There are, if I recall correctly, five scripts involved in the main
event, and one for r.in.onearth. I did this to separate the code out
into pieces. The scripts are something like:

could non-reusable parts be combined into the main script file as script
function calls? (e.g. cleanup() in r.univar.sh) Less file clutter that
way.

r.tileset which generates tilings in a general way.

wms.request which generates requests to wms servers. This is heavily
lifted from r.in.onearth and "jachym's" version.

wms.download which downloads requests (a very simple wrapper around
wget).

r.in.gdalwarp which imports and patches many files using gdalwarp to
reproject data if necessary (pretty much lifted from r.in.onearth).

and r.in.wms which is a simple wrapper around the first four and what
people would usually use.

Finally it has another r.in.onearth that Sören wrote that wraps up
this r.in.wms.

multiple related scripts should live in the same CVS source directory,
the Makefile can copy the parts to somewhere that $GISBASE/etc/ can find.
e.g. have a look at how the i.oif script creates and uses
$GISBASE/etc/i.oif/ for sub-modules but goes in $GISBASE/scripts/ itself.

Most everything here (except for tiling) was lifted straight from
jachym's and Sören's work, which made most of the programming quite
easy.

A fine example of the collaborative nature of Open Source...

One thing this set is missing (besides html documentation and
makefiles) is some of the interaction with the server that "jachym's"
does (listing layers). I was kind of hoping to parse and comprehend
the entire capabilities request like I did with "raster plaster" but
the difficulty and dependencies of parsing XML in shell scripts was
too high.

A test could be done for an XML parser iff the "-l" flag (or whatever)
was used:

r.in.wms -l ...
if [ -x `which xml2` ] ; then
ERROR: Listing of available maps requires the "xml2" XML parser.
  http://http://dan.egnor.name/xml2

As long as it is just optional functionalilty that will be lost it's not
a huge deal. (& the debian package can just be set to depend on the xml2
package :slight_smile:

It would be pretty easy to merge this functionality from jachym's
latest scripts.

Go for it!

Separating the scripts clarifies and enforces separations in code. It
also allows more possible scripting by users in terms of downloads,
retrying downloads, etc. It would, however, be nice if wms.request,
wms.download, and r.in.gdalwarp could be slightly less exposed
(though no less documented).

A worthy consideration, see r.univar.sh and i.oif examples for two
levels of script integration below full-fledged "GRASS modules".

thanks,
Hamish

ps - some XML->ASCII parsers I found when looking into v.in.gpsbabel:
  v.in.gpsbabel does the parsing by hand, fragile..

* "expat":
  http://expat.sourceforge.net/

* "xml2" might be better choice?:
Description: Convert between XML, HTML, CSV and a line-oriented format
xml2 tools are used to convert XML, HTML and CSV to and from a
line-oriented format more amenable to processing by classic Unix
pipeline processing tools, like grep, sed, awk, cut, shell scripts,
and so forth.
(download to gpx format with gpsbabel and then feed through xml2 to get csv?)
  http://dan.egnor.name/xml2/
  http://dan.egnor.name/xml2/ref

* Also "xmlto" software will do this... but xml2 is more popular.
  http://cyberelk.net/tim/xmlto/

Hi,

I committed the big collection of r.in.wms scripts to CVS. Three of them are
visible publically: r.tileset, r.in.wms, and r.in.onearth.

Sören, can you look over description.html for r.in.onearth?

> It would be nice if the wording on both r.in.wms and v.in.wfs for
> servers, SRS projections, etc. are in sync before releasing anything.

Very well, but this is up to the authors of the scripts to understand
and decide... if this doesn't happen very soon (ie now) the scripts will
not be in by the feature freeze for 6.2.

I used the following options:
mapserver for the base uri of the server
srs for the projection
layer for the layer and
style for the style.
These are all the option types that I imagine might be in common between WMS
and WFS (I've never looked at WFS).

multiple related scripts should live in the same CVS source directory,
the Makefile can copy the parts to somewhere that $GISBASE/etc/ can find.
e.g. have a look at how the i.oif script creates and uses
$GISBASE/etc/i.oif/ for sub-modules but goes in $GISBASE/scripts/ itself.

...

> Separating the scripts clarifies and enforces separations in code. It
> also allows more possible scripting by users in terms of downloads,
> retrying downloads, etc. It would, however, be nice if wms.request,
> wms.download, and r.in.gdalwarp could be slightly less exposed
> (though no less documented).

A worthy consideration, see r.univar.sh and i.oif examples for two
levels of script integration below full-fledged "GRASS modules".

I did it the i.oif way (sort-of). It'll be easy to rip out any of these
pseudo-modules into full fledged modules if they are ever desired since they
already use g.parser.

A test could be done for an XML parser iff the "-l" flag (or whatever)
was used:

r.in.wms -l ...
if [ -x `which xml2` ] ; then
ERROR: Listing of available maps requires the "xml2" XML parser.
  http://http://dan.egnor.name/xml2

As long as it is just optional functionalilty that will be lost it's not
a huge deal. (& the debian package can just be set to depend on the xml2
package :slight_smile:

> It would be pretty easy to merge this functionality from jachym's
> latest scripts.

Go for it!

Done and done.

--Cedric

Hallo, two things:

First:
Started compilation: Út kvě 2 08:20:07 CEST 2006
--
Errors in:
/usr/src/grass6/raster/r.average
/usr/src/grass6/scripts/r.in.onearth
/usr/src/grass6/scripts/r.in.wms
/usr/src/grass6/scripts/r.tileset
--
Finished compilation: Út kvě 2 08:25:28 CEST 2006

bobes:/usr/src/grass6# ls /usr/src/grass6/scripts/r.in.wms
ls: /usr/src/grass6/scripts/r.in.wms: No such file or directory
bobes:/usr/src/grass6#

bobes:/usr/src/grass6# find . -name r.in.wms -print
bobes:/usr/src/grass6#

Something is missing?

Second:
With respect to Soeren, I do not thing, r.in.onearth should be in CVS.
It is too special tool - it is like adding "r.multiply" next to
r.mapcalc

just my two cents

Jachym

On Mon, May 01, 2006 at 08:52:22PM -0700, Cedric Shock wrote:

Hi,

I committed the big collection of r.in.wms scripts to CVS. Three of them are
visible publically: r.tileset, r.in.wms, and r.in.onearth.

Sören, can you look over description.html for r.in.onearth?

> > It would be nice if the wording on both r.in.wms and v.in.wfs for
> > servers, SRS projections, etc. are in sync before releasing anything.
>
> Very well, but this is up to the authors of the scripts to understand
> and decide... if this doesn't happen very soon (ie now) the scripts will
> not be in by the feature freeze for 6.2.

I used the following options:
mapserver for the base uri of the server
srs for the projection
layer for the layer and
style for the style.
These are all the option types that I imagine might be in common between WMS
and WFS (I've never looked at WFS).

>
> multiple related scripts should live in the same CVS source directory,
> the Makefile can copy the parts to somewhere that $GISBASE/etc/ can find.
> e.g. have a look at how the i.oif script creates and uses
> $GISBASE/etc/i.oif/ for sub-modules but goes in $GISBASE/scripts/ itself.
>
...
>
> > Separating the scripts clarifies and enforces separations in code. It
> > also allows more possible scripting by users in terms of downloads,
> > retrying downloads, etc. It would, however, be nice if wms.request,
> > wms.download, and r.in.gdalwarp could be slightly less exposed
> > (though no less documented).
>
> A worthy consideration, see r.univar.sh and i.oif examples for two
> levels of script integration below full-fledged "GRASS modules".

I did it the i.oif way (sort-of). It'll be easy to rip out any of these
pseudo-modules into full fledged modules if they are ever desired since they
already use g.parser.

>
> A test could be done for an XML parser iff the "-l" flag (or whatever)
> was used:
>
> r.in.wms -l ...
> if [ -x `which xml2` ] ; then
> ERROR: Listing of available maps requires the "xml2" XML parser.
> http://http://dan.egnor.name/xml2
>
> As long as it is just optional functionalilty that will be lost it's not
> a huge deal. (& the debian package can just be set to depend on the xml2
> package :slight_smile:
>
> > It would be pretty easy to merge this functionality from jachym's
> > latest scripts.
>
> Go for it!

Done and done.

--Cedric

--
Jachym Cepicky
e-mail: jachym.cepicky@centrum.cz
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/gnupg_public_key/jachym_cepicky-gpg_public_key.asc
-----------------------------------------
OFFICE:
GDF-Hannover
Mengendamm 16d
30177 Hannover
Germany
e-mail: cepicky@gdf-hannover.de
URL: http://gdf-hannover.de
Tel.: +49 511-39088507

Jachym Cepicky wrote:

...

Second:
With respect to Soeren, I do not thing, r.in.onearth should be in CVS.
It is too special tool - it is like adding "r.multiply" next to
r.mapcalc

just my two cents

Yes, please, let's have only a single WMS import script. The name should
be r.in.wms (because
that's generic) and the contents the best of the existing scripts. I am
convinced that the authors
can negotiate a single implementation.

thanks
Markus

r.in.onearth is very nice because it can be used in a straight foreward way. Having generic scripts is nice too, but less user friendly for the homespun users like me.

So I would like to suggest that r.in.onearth should kept alive and made public accessible in some addon-CVS-repository which can be activated by a configure-switch.

Mit freundlichen Grüßen / With kindest regards

Stefan Paulick

http://www.urbeli.com
mailto://stefan.paulick@urbeli.com
/*----------------------*/

Markus Neteler schrieb:

Jachym Cepicky wrote:

...

Second:
With respect to Soeren, I do not thing, r.in.onearth should be in CVS.
It is too special tool - it is like adding "r.multiply" next to
r.mapcalc

just my two cents
  
Yes, please, let's have only a single WMS import script. The name should
be r.in.wms (because
that's generic) and the contents the best of the existing scripts. I am
convinced that the authors
can negotiate a single implementation.

thanks
Markus

_______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5

Jachym,

Hallo, two things:

First:

..

Errors in:

...

/usr/src/grass6/scripts/r.in.wms
/usr/src/grass6/scripts/r.tileset
--
Finished compilation: Út kvě 2 08:25:28 CEST 2006

bobes:/usr/src/grass6# ls /usr/src/grass6/scripts/r.in.wms
ls: /usr/src/grass6/scripts/r.in.wms: No such file or directory
bobes:/usr/src/grass6#

...

Something is missing?

Did you do:
cvs update -d
to also download new directories?

Second:
With respect to Soeren, I do not thing, r.in.onearth should be in CVS.
It is too special tool - it is like adding "r.multiply" next to
r.mapcalc

just my two cents

Sören seemed to be suggesting the same (that it belongs in extension land). It
is gone from the repository.

--Cedric

I committed the big collection of r.in.wms scripts to CVS. Three of
them are visible publically: r.tileset, r.in.wms,

"r.in.wms -l" doesn't work for me.

using the example onearth server on the man page I see wget run, but
then no output and just a prompt. Using terraserver,

G61> r.in.wms mapserver=http://terraserver.microsoft.com/ogcmap6.ashx -l
[...]/16132.0capabilities.xml' saved [421]

error: Opening and ending tag mismatch: br and body
error: Opening and ending tag mismatch: br and html
error: Extra content at the end of the document
G61>

commenting out the rm tempfile line I see the problems:

G61> r.in.wms mapserver=http://wms.jpl.nasa.gov/wms.cgi -l
$ cat 16193.0capabilities.xml
<?xml version='1.0' encoding="UTF-8" standalone="no" ?>
<!DOCTYPE ServiceExceptionReport SYSTEM
"http://www.digitalearth.gov/wmt/xml/exception_1_1_0.dtd ">
<ServiceExceptionReport version="1.1.0">
<ServiceException>
   Service denied due to system overload. Please try again later.
</ServiceException>
</ServiceExceptionReport>

G61> r.in.wms mapserver=http://terraserver.microsoft.com/ogcmap6.ashx -l
$ cat 16210.0capabilities.xml
<html><head><title>OGCMap Test</title></head>
<body>
<h3>Service Exception Report</h3>
<p>Found 5 errors</p>
<br>Invalid Request 'GetCapabilities' - Value 'GetMap' is required
<br>BBOX (bounding box) parameter is missing.
<br>Layers parameter is missing.
<br>Width parameter is missing or is an invalid integer digit string.
<br>Height parameter is missing or is an invalid integer digit string.
</body></html>

Other issues:

I am not a fan of leaving the raw downloads in $GISDBASE.
Perhaps put in $MAPSET/.tmp/$HOST/... ?
i.e. I'd prefer a flag to KEEP the raw download, not to delete it.

I too am a big fan of r.in.onearth's simplicity, even if it is
somewhat redundant. Cedric's version doesn't replicate r.in.wms, it acts
as a frontend to it.

Wish:
g.tempfile needs a -d switch to create+return a dir in
$MAPSET/.tmp/$HOST/ instead of the user creating a file +tricks to use
that name as a tmp dir.

thanks,
Hamish