[GRASS-dev] improving v.in.wms

GRASS GIS GRASS GIS developer
,
1

Luca wrote:

For me +1 to use OWSlib,i already thought to use it for wms/wfs client.
if we add this dependency we could also add others clients (wcs,sos etc)
in a easy way.

Hi,

since it is just a couple characters to += onto the URL to support data
layer names, bounding boxes, etc. I stuck with the current method for now.
Added functionality in r53180:

New flags:
  -l Download server capabilities to 'wms_capabilities.xml' in the current directory and exit
  -r Restrict fetch to features which touch the current region

New/changed parameters:
               url Base URL starting with 'http' and ending in '?'
              name Comma separated names of data layers to download
               srs Specify alternate spatial reference system (example: EPSG:4326)
                      The given code must be supported by the server, consult the capabilities file
  maximum_features Maximum number of features to download
                      (default: unlimited)
       start_index Skip earlier feature IDs and start downloading at this one
                      (default: start with the first feature)

There is still lots of room for improvement and better methods, I just
went with the quick solution. (But it works) Please do improve it further
if you like..

Since the data layer Name is not always very meaningful as the Title, and
the xml capabilities file is often hard to browse, even with xml2 parsing,
I was thinking perhaps to put the OWSlib magic for WFS into an owslib
based interactive GUI browser tool, somewhat similar to QGIS 1.8.0's nice
qbrowser* but with a search/filter function as the grass location wizard
has, allow to limit by current g.region bounds, etc.

[*] qgis 1.8.0's main WFS add layer dialog seems like it is still under
construction; the qbrowser side of things worked a bit better for me.

.. I'm not too worried since WFS is a lot cleaner than WMS data to deal
with, just trying to think of how not to fall into the same traps as we
did for the grass 6 r.in.wms, where the low-level module tried to do too
much itself.

Hamish

ps- the Hannover grass user map wfs in the module's example returns a 404.

2

On Sun, Sep 16, 2012 at 10:16 AM, Hamish <hamish_b@yahoo.com> wrote:
...

ps- the Hannover grass user map wfs in the module's example returns a 404.

Yes, known (the old WMS/WFS cannot be restored due to complete server
loss in a previous crash) - a new, modern user map is in preparation, the
data have been migrated already.

Markus

3

Hello Hamish,

I have some suggestions for improvement of v.in.wfs module.

It would be good when parameters and flags would be same for all OGC modules in GRASS.

I would suggest:
url -> mapserver (original r.in.wms module had mapserver parameter, maybe url is better name but for compatibility reason I would stick to mapserver)

-l -> -c
or in r.in.wms -c -> -l

I would suggest to use similar design of the module as r.in.wms. This allows usage of various drivers for getting data from server. Currently r.in.wms supports GDAL WMS driver and own driver. The structure allows to add other drivers easily (e. g. OWSlib).

Also some parts of source code of r.in.wms can be usable for v.in.wfs (e. g. transformation of bbox). This makes implementation of e. g. transformation of downloaded data into projection of location relatively easy.

I know r.in.wms well so if you would agree with these suggestions I can modify code of r.in.wms to be usable for v.in.wfs module.

Stepan

On 09/16/2012 10:16 AM, Hamish wrote:

Luca wrote:

For me +1 to use OWSlib,i already thought to use it for wms/wfs client.
if we add this dependency we could also add others clients (wcs,sos etc)
in a easy way.

Hi,

since it is just a couple characters to += onto the URL to support data
layer names, bounding boxes, etc. I stuck with the current method for now.
Added functionality in r53180:

New flags:
   -l Download server capabilities to 'wms_capabilities.xml' in the current directory and exit
   -r Restrict fetch to features which touch the current region

New/changed parameters:
                url Base URL starting with 'http' and ending in '?'
               name Comma separated names of data layers to download
                srs Specify alternate spatial reference system (example: EPSG:4326)
                       The given code must be supported by the server, consult the capabilities file
   maximum_features Maximum number of features to download
                       (default: unlimited)
        start_index Skip earlier feature IDs and start downloading at this one
                       (default: start with the first feature)

There is still lots of room for improvement and better methods, I just
went with the quick solution. (But it works) Please do improve it further
if you like..

Since the data layer Name is not always very meaningful as the Title, and
the xml capabilities file is often hard to browse, even with xml2 parsing,
I was thinking perhaps to put the OWSlib magic for WFS into an owslib
based interactive GUI browser tool, somewhat similar to QGIS 1.8.0's nice
qbrowser* but with a search/filter function as the grass location wizard
has, allow to limit by current g.region bounds, etc.

[*] qgis 1.8.0's main WFS add layer dialog seems like it is still under
construction; the qbrowser side of things worked a bit better for me.

.. I'm not too worried since WFS is a lot cleaner than WMS data to deal
with, just trying to think of how not to fall into the same traps as we
did for the grass 6 r.in.wms, where the low-level module tried to do too
much itself.

Hamish

ps- the Hannover grass user map wfs in the module's example returns a 404.
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

4

Štěpán wrote:

I have some suggestions for improvement of v.in.wfs module.

great

It would be good when parameters and flags would be same for
all OGC modules in GRASS.

consistency is good. it means that once the user learns things
once they can re-use that knowledge across all the parts of the
GIS, versus having to learn all the different 400-500 modules
individually. (at the same time though, you don't have to be a
complete slave to that if the fit is wrong somewhere.. there
is strength in flexibility)

I would suggest:
url -> mapserver (original r.in.wms module had mapserver
parameter, maybe url is better name but for compatibility
reason I would stick to mapserver)

don't worry too much about breaking compatibility with the grass
6 names if the change is demonstrably better. That's why it will
be called grass 7 and not grass 6.6.

wrt "mapserver", my main concern is that it shares the same name-
space with the software project, so may be confusing in some
usage contexts. other than that I don't really mind much.

-l -> -c
or in r.in.wms -c -> -l

-l for list of names, -c for full capabilities

I would suggest to use similar design of the module as
r.in.wms. This allows usage of various drivers for getting
data from server.

fwiw originally the idea for the flag was 'list layers' not
show full capabilities. I didn't want to go to the complications
of parsing xml, so for a shortcut I just saved out the full xml
to a local file. (it's a quick hack, feel free to improve...
maybe there is a python tool to reformat the xml in a pretty
way instead of the no-whitespace padding that often the
capabilities download file has?)
I don't know which is better, probably have the back-end module
save the xml file* and the GUI run that in the background and
present the results in a nice gui table for interactive prep.

[*] would need to be able to set the filename to save it to a
temp dir/file.

Currently r.in.wms supports GDAL WMS driver and own driver. The
structure allows to add other drivers easily (e. g. OWSlib).

is that really needed? or would it be better to just support one
method well? (I don't mind a flexible back-end, just wondering)

Also some parts of source code of r.in.wms can be usable for
v.in.wfs (e. g. transformation of bbox). This makes
implementation of e. g. transformation of downloaded data into
projection of location relatively easy.

fwiw I am not much of a fan of on-the-fly reprojections, they
can get a bit muddy. e.g. if using gdal/ogr with +nadgrids datum
transform NTv2 files which are located in places with spaces in
the path name, some of the gdal tools fail as they don't parse
the srs as a single argument and it's impossible to fix that
without a change to the gdal tool and broken compatibility for
all gdal users. (search for the bug report in trac system(s))
Too many headaches maintaining v.in.gpsbabel and r.in.wms make
me prefer to just import into native location and project with
full r.proj/v.proj now ... k.i.s.s. principle.
any module which did try to reproject on the fly should have
a flag to avoid that part of the process completely.

I know r.in.wms well so if you would agree with these
suggestions I can modify code of r.in.wms to be usable for
v.in.wfs module.

I have no further plans to develop v.in.wfs myself; my current
itch is scratched. if someone else would like to do that by all
means take it up and make it great.

cheers,
Hamish

5

On 09/17/2012 04:26 AM, Hamish wrote:

don't worry too much about breaking compatibility with the grass
6 names if the change is demonstrably better. That's why it will
be called grass 7 and not grass 6.6.

wrt "mapserver", my main concern is that it shares the same name-
space with the software project, so may be confusing in some
usage contexts. other than that I don't really mind much.

It is good point I would suggest change mapserver to url in GRASS 7 r.in.wms?

-l -> -c
or in r.in.wms -c -> -l

  -l for list of names, -c for full capabilities
  t

It makes sense to me.

I would suggest to use similar design of the module as
r.in.wms. This allows usage of various drivers for getting
data from server.

fwiw originally the idea for the flag was 'list layers' not
show full capabilities. I didn't want to go to the complications
of parsing xml, so for a shortcut I just saved out the full xml
to a local file. (it's a quick hack, feel free to improve...
maybe there is a python tool to reformat the xml in a pretty
way instead of the no-whitespace padding that often the
capabilities download file has?)
I don't know which is better, probably have the back-end module
save the xml file* and the GUI run that in the background and
present the results in a nice gui table for interactive prep.

[*] would need to be able to set the filename to save it to a
temp dir/file.

I would leave capabilities parsing for WxGUI. I did already some work in this field before GSoC started.

Currently r.in.wms supports GDAL WMS driver and own driver. The
structure allows to add other drivers easily (e. g. OWSlib).

is that really needed? or would it be better to just support one
method well? (I don't mind a flexible back-end, just wondering)

Yes, I mean implement just one method with structure of module which allows to add other methods in future.

Also some parts of source code of r.in.wms can be usable for
v.in.wfs (e. g. transformation of bbox). This makes
implementation of e. g. transformation of downloaded data into
projection of location relatively easy.

fwiw I am not much of a fan of on-the-fly reprojections, they
can get a bit muddy. e.g. if using gdal/ogr with +nadgrids datum
transform NTv2 files which are located in places with spaces in
the path name, some of the gdal tools fail as they don't parse
the srs as a single argument and it's impossible to fix that
without a change to the gdal tool and broken compatibility for
all gdal users. (search for the bug report in trac system(s))
Too many headaches maintaining v.in.gpsbabel and r.in.wms make
me prefer to just import into native location and project with
full r.proj/v.proj now ... k.i.s.s. principle.
any module which did try to reproject on the fly should have
a flag to avoid that part of the process completely.

The r.in.wms module was developed with intend to be used for integration of WMS into WxGUI which should enable on the fly transformation.
Transformation functionality in r.in.wms uses GDAL and transforms raster before it is imported into GRASS. It seemed to be better solution than create some temporary location and transform data already imported into GRASS.

R.in.wms does not have flag for avoiding reprojection but it compares proj.4 definitions of downloaded data and location projections. It is not the ideal solution and it may not work in 100 % possible cases however when I tested it, it worked.

I know r.in.wms well so if you would agree with these
suggestions I can modify code of r.in.wms to be usable for
v.in.wfs module.

I have no further plans to develop v.in.wfs myself; my current
itch is scratched. if someone else would like to do that by all
means take it up and make it great.

After GSoC I was enjoying holidays and avoiding computer as much as possible.
This could force me to start doing something for GRASS again.

I have thought about working on integration of WMS into wxGUI however it would be good to have v.in.wfs module which would be possible to use for integration of WFS into WxGUI and integrate WMS and WFS together.

Stepan

6

2012/9/17 Štěpán Turek <stepan.turek@seznam.cz>:

On 09/17/2012 04:26 AM, Hamish wrote:

don't worry too much about breaking compatibility with the grass
6 names if the change is demonstrably better. That's why it will
be called grass 7 and not grass 6.6.

wrt "mapserver", my main concern is that it shares the same name-
space with the software project, so may be confusing in some
usage contexts. other than that I don't really mind much.

It is good point I would suggest change mapserver to url in GRASS 7
r.in.wms?

+1

Stepan

--
ciao
Luca

http://gis.cri.fmach.it/delucchi/
www.lucadelu.org

7

Hi Vincent,

WMTS support and username/passoword would be useful. I will look at WMTS standard and try to implement it.

It would be very useful for testing to show me where to register for access.

Stepan

On 09/18/2012 07:53 AM, Vincent Bain wrote:

Stepan,

a short reply offlist, just to react to your improvments to r.in.wms.
First, thanks for involving in Grass...
Last spring I told you I was making heavy use of french cartographic
ressources (www.geoportail.gouv.fr). Last month they set up a new wmts
architecture (replacing a wms-c service), which allows to retrieve
raster data such as orthoimagery, elevation data, and so on.
Unfortunately I could not use your python addon (tell me if I'm wrong)
because wmts queries are based on an APIkey/login/password
authentification system.

I wrote a shell script that does the job, partly running gdal tools. Of
course it's designed for private use, very rough, not much optimized,
but it has the ability to build sequential https requests containing the
right login/password header.

I guess it would be easy to implement that "restricted access header
option" to your module; The thing is I ignore nearly everything of
Python ! let me know if you are interested in developing in this
direction. If so, I can show you where to register for access to ign web
data services, so that you can make tests. Attached is my shell script,
fyi.

Bye,
Vincent

8

OK, great !

first you need to register here :
http://api.ign.fr/moncompte/creer
(sorry, IGN web site was redesigned this summer and they dropped the
english version formerly available...)

Then from your account page, you can create a contract (choose the 'Sig'
key type, in french 'Type de clé' set to GIS).
http://api.ign.fr/jsp/site/Portal.jsp?page=myapi&action=createkey&keytype=SIG_MODEL&keylabel=Nom%20d'usage%20de%20la%20clé
Don't forget to pick items you need from the list of available
ressources (e.g. orthoimagery, maps,...)

You're sent a confirmation email, delivering an APIKey.
With the login/password/APIKey triplet you can run r.in.ign

To avoid getting empty response, I suggest you to create a location
matching some covered territories : for example you can make a location
from epsg code 27572 ("Lambert II étendu"), then set up a region with
these params :

north: 2095180
south: 2094980
west: 920650
east: 920850
nsres: 1
ewres: 1

To obtain orthoimagery on the place :
r.in.ign -m --overwrite apikey=$APIKEY user=$USER password=$PASSWORD
wmtslayer=ORTHOIMAGERY.ORTHOPHOTOS output=test

If you try the -c flag to obtain server capabilities don't be surprised
by the very long delay of the server to return the xml file (depends,
but can be more than one minute)

Here it is,
good luck.

Vincent

Le mardi 18 septembre 2012 à 11:19 +0200, Štěpán Turek a écrit :

Hi Vincent,

WMTS support and username/passoword would be useful. I will look at WMTS
standard and try to implement it.

It would be very useful for testing to show me where to register for access.

Stepan

On 09/18/2012 07:53 AM, Vincent Bain wrote:
> Stepan,
>
> a short reply offlist, just to react to your improvments to r.in.wms.
> First, thanks for involving in Grass...
> Last spring I told you I was making heavy use of french cartographic
> ressources (www.geoportail.gouv.fr). Last month they set up a new wmts
> architecture (replacing a wms-c service), which allows to retrieve
> raster data such as orthoimagery, elevation data, and so on.
> Unfortunately I could not use your python addon (tell me if I'm wrong)
> because wmts queries are based on an APIkey/login/password
> authentification system.
>
> I wrote a shell script that does the job, partly running gdal tools. Of
> course it's designed for private use, very rough, not much optimized,
> but it has the ability to build sequential https requests containing the
> right login/password header.
>
> I guess it would be easy to implement that "restricted access header
> option" to your module; The thing is I ignore nearly everything of
> Python ! let me know if you are interested in developing in this
> direction. If so, I can show you where to register for access to ign web
> data services, so that you can make tests. Attached is my shell script,
> fyi.
>
> Bye,
> Vincent
>

9

Štěpán wrote:

WMTS support and username/passoword would be useful. I will
look at WMTS standard and try to implement it.

Hi

see existing gpl2 code (by me anyway..) for it here:
http://gpsdrive.svn.sourceforge.net/viewvc/gpsdrive/trunk/scripts/gpstile_fetch_and_assemble.sh?revision=2680&content-type=text%2Fplain

maybe it is easy to port with just minor modifications?

see also Tiled-WMS (like NASA OnEarth server now is restricted to)
http://gpsdrive.svn.sourceforge.net/viewvc/gpsdrive/trunk/scripts/gpstiled_wms_fetch_and_assemble.sh?revision=2677&content-type=text%2Fplain

The tiled-WMS method is already implemented for GRASS in the r.in.onearth
addon script. maybe much much easier using 'g.region -a' magic.

Hamish

10

Hi Hamish,

I am finishing implementation of WMTS into r.in.wms.
Within a few days I will add this experimental version into GRASS AddOns as r.in.wms2.

Thanks for information about Tiled-WMS.
I will look at it.

Stepan

On 10/01/2012 12:32 PM, Hamish wrote:

Štěpán wrote:

WMTS support and username/passoword would be useful. I will
look at WMTS standard and try to implement it.

Hi

see existing gpl2 code (by me anyway..) for it here:
http://gpsdrive.svn.sourceforge.net/viewvc/gpsdrive/trunk/scripts/gpstile_fetch_and_assemble.sh?revision=2680&content-type=text%2Fplain

maybe it is easy to port with just minor modifications?

see also Tiled-WMS (like NASA OnEarth server now is restricted to)
http://gpsdrive.svn.sourceforge.net/viewvc/gpsdrive/trunk/scripts/gpstiled_wms_fetch_and_assemble.sh?revision=2677&content-type=text%2Fplain

The tiled-WMS method is already implemented for GRASS in the r.in.onearth
addon script. maybe much much easier using 'g.region -a' magic.

Hamish

11

Hi,

I have added r.in.wms2 module with experimental WMTS support into GRASS AddOns.
I have tested it on IGN Geoportal and Czech Cadastral Geoportal. The examples can be found on manual page.

Currently, I work on Tiled-WMS support in r.in.wms2.

Stepan

---------- Původní zpráva ----------
Od: Hamish hamish_b@yahoo.com
Datum: 1. 10. 2012
Předmět: Re: [GRASS-dev] improving v.in.wms [you mean r.in.wms?]

Štěpán wrote:

WMTS support and username/passoword would be useful. I will
look at WMTS standard and try to implement it.

Hi

see existing gpl2 code (by me anyway…) for it here:
http://gpsdrive.svn.sourceforge.net/viewvc/gpsdrive/trunk/scripts/gpstile_fetch_and_assemble.sh?revision=2680&content-type=text%2Fplain

maybe it is easy to port with just minor modifications?

see also Tiled-WMS (like NASA OnEarth server now is restricted to)
http://gpsdrive.svn.sourceforge.net/viewvc/gpsdrive/trunk/scripts/gpstiled_wms_fetch_and_assemble.sh?revision=2677&content-type=text%2Fplain

The tiled-WMS method is already implemented for GRASS in the r.in.onearth
addon script. maybe much much easier using ‘g.region -a’ magic.

Hamish

=