[GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

#2564: r.what: remove 400 maps limit and add some new output options
-------------------------+--------------------------------------------------
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.what | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+--------------------------------------------------
r.what is currently the most efficient tool for sampling (single) STRDS at
point locations.
Usage of r.what for space time data sets however is hampered by the
current limit of 400 maps. Would be nice if that limit could be overcome.
Memory may be an issue in case of thousands of maps?

See discussion on the mailinglist:
http://thread.gmane.org/gmane.comp.gis.grass.devel/62452/focus=62472

In that context also an option/flag to choose between vertical and
horizontal output would be useful (columns for thousands of maps will
probably trouble spread sheet software).

Finally, when using a vector map as input it would be nice if the category
(or content from another column) could be written into the "site name"
column of the output, so a resulting table can be linked more easily to
the original data.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2564&gt;
GRASS GIS <http://grass.osgeo.org>

#2564: r.what: remove 400 maps limit and add some new output options
-------------------------+--------------------------------------------------
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.what | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by huhabla):

I have just committed a new temporal module called t.rast.what in trunk
(r64349 - r64351) that utilized r.what to sample space time raster
datasets using vector points. It provides three output layouts that
transforms the r.what output into row or column layouts. It can run
several r.what processes in parallel using a maximum of 400 raster map
layer in a single r.what process.

Please have a look at it.

Any feedback about its performance, usefulness and handling is highly
welcome.

Improving r.what to allow more than 400 maps is simple, just edit the
fixed value in the source. But, editing this value may cause open file
handler limit problems with your OS. However, these limits can be adjusted
at kernel level configuration for Linux.

Implementing a more intelligent file handling solution requires more
effort.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2564#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

Great news! Thanks.
I shall test immediately.

-----Original Message-----
From: grass-dev-bounces@lists.osgeo.org [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of GRASS GIS
Sent: 29. januar 2015 00:16
Subject: Re: [GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

#2564: r.what: remove 400 maps limit and add some new output options
-------------------------+----------------------------------------------
-------------------------+----
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.what | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+----------------------------------------------
-------------------------+----

Comment(by huhabla):

I have just committed a new temporal module called t.rast.what in trunk
(r64349 - r64351) that utilized r.what to sample space time raster datasets using vector points. It provides three output layouts that transforms the r.what output into row or column layouts. It can run several r.what processes in parallel using a maximum of 400 raster map layer in a single r.what process.

Please have a look at it.

Any feedback about its performance, usefulness and handling is highly welcome.

Improving r.what to allow more than 400 maps is simple, just edit the fixed value in the source. But, editing this value may cause open file handler limit problems with your OS. However, these limits can be adjusted at kernel level configuration for Linux.

Implementing a more intelligent file handling solution requires more effort.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2564#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Hi Sören,

Thanks a lot for that one. Great work as usual. It works really nice and fast.
I compared it to point sampling of (in-db) raster data in PostGIS, where I have exactly the same data (uploaded to PostGIS from TGIS/GRASS).
I sampled 686 raster map (1500x1300 pixels) at 17 points.
PostGIS (with spatial index): 53 seconds
t.rast.what (with 2 workers): 59 seconds
I had some minor issues in my rather old GRASS71 installation.

Please find attached a diff for the things I had to change in order to make it work. However, with "col"-option all coordinates in the header are the same. This I did not manage to fix (yet).

Kind regards,
Stefan

P.S.: BTW, is the structure of the TGIS-database for current GRASS70 the same as for GRASS71?

-----Original Message-----
From: grass-dev-bounces@lists.osgeo.org [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of GRASS GIS
Sent: 29. januar 2015 00:16
Subject: Re: [GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

#2564: r.what: remove 400 maps limit and add some new output options
-------------------------+----------------------------------------------
-------------------------+----
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.what | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+----------------------------------------------
-------------------------+----

Comment(by huhabla):

I have just committed a new temporal module called t.rast.what in trunk
(r64349 - r64351) that utilized r.what to sample space time raster datasets using vector points. It provides three output layouts that transforms the r.what output into row or column layouts. It can run several r.what processes in parallel using a maximum of 400 raster map layer in a single r.what process.

Please have a look at it.

Any feedback about its performance, usefulness and handling is highly welcome.

Improving r.what to allow more than 400 maps is simple, just edit the fixed value in the source. But, editing this value may cause open file handler limit problems with your OS. However, these limits can be adjusted at kernel level configuration for Linux.

Implementing a more intelligent file handling solution requires more effort.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2564#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

(attachments)

t,rast.what.diff (3.41 KB)

Forgot to mention:

t.rast.what does not seem to write to stdout (as the description suggests), only to file. Output to stdout could be an improvement for people who want to read the output directly into R for example (for that purpose the " one_point_per_row_output"-layout is splendid).

Carrying also the coordinates-option from r.what to t.rast.what would be nice for cases where one only has a few coordinates to process and where copying them from a text file or web and pasting them into the module is more convenient than creating a vector map for a single use case in advance (and probably deleting the vector map afterwards)...

Finally, as written in the r.what ticket, being able to make use of the site_name column also in "vector-map-mode" would be useful, because it is more demanding (both for user and computer) to join additional information to the output (or the other way around) using two columns in double precision compared to e.g. one cat column(or whatever a user might like to have in order to carachterize the sites), esp. because coordinates may be given with different level of precision in r.what and v.to.db for example. But this applies to r.what first of course...
An intermediate solution could be to pipe something like:
v.to.db map=INPUT_VECTOR_MAP option=coor qcolumn=cat separator=comma -p | awk -v FS=',' -v OFS=',' '{print $2,$3,$1}' | tail -n +2
to the coordinates option in r.what... That way the cat column would show up in the r.what output...

Thanks again for the excellent tool!

Cheers,
Stefan

-----Original Message-----
From: grass-dev-bounces@lists.osgeo.org [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of Blumentrath, Stefan
Sent: 29. januar 2015 07:30
To: grass-dev@lists.osgeo.org
Subject: Re: [GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

Great news! Thanks.
I shall test immediately.

-----Original Message-----
From: grass-dev-bounces@lists.osgeo.org [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of GRASS GIS
Sent: 29. januar 2015 00:16
Subject: Re: [GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

#2564: r.what: remove 400 maps limit and add some new output options
-------------------------+----------------------------------------------
-------------------------+----
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.what | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+----------------------------------------------
-------------------------+----

Comment(by huhabla):

I have just committed a new temporal module called t.rast.what in trunk
(r64349 - r64351) that utilized r.what to sample space time raster datasets using vector points. It provides three output layouts that transforms the r.what output into row or column layouts. It can run several r.what processes in parallel using a maximum of 400 raster map layer in a single r.what process.

Please have a look at it.

Any feedback about its performance, usefulness and handling is highly welcome.

Improving r.what to allow more than 400 maps is simple, just edit the fixed value in the source. But, editing this value may cause open file handler limit problems with your OS. However, these limits can be adjusted at kernel level configuration for Linux.

Implementing a more intelligent file handling solution requires more effort.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2564#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Dear Stefan,
thank you very much for your feedback, the patch and suggestions.

I have integrated some of your suggestions in t.rast.what. There is
now stdout and coordinates support in the latest trunk version
(r64368). The column layout bug is now fixed and the documentation was
updated according to the new features. Everything is now covered with
tests. Hence, if the modules does not run as you expect, please make a
test run (in a grass session make python test_what.py in the testsuite
folder of t.rast.what) and send me the result of the test, so i can
investigate the problem.

I did not modified the options in t.rast.what as your patch suggested,
since the options should run by default in your suggested
configuration.

The temporal database layout and its handling (distributed, mapset
specific temporal databases) should be identical in the grass7.0
release version and grass7 trunk.

Best regards
Soeren

2015-01-30 8:49 GMT+01:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:

Forgot to mention:

t.rast.what does not seem to write to stdout (as the description suggests), only to file. Output to stdout could be an improvement for people who want to read the output directly into R for example (for that purpose the " one_point_per_row_output"-layout is splendid).

Carrying also the coordinates-option from r.what to t.rast.what would be nice for cases where one only has a few coordinates to process and where copying them from a text file or web and pasting them into the module is more convenient than creating a vector map for a single use case in advance (and probably deleting the vector map afterwards)...

Finally, as written in the r.what ticket, being able to make use of the site_name column also in "vector-map-mode" would be useful, because it is more demanding (both for user and computer) to join additional information to the output (or the other way around) using two columns in double precision compared to e.g. one cat column(or whatever a user might like to have in order to carachterize the sites), esp. because coordinates may be given with different level of precision in r.what and v.to.db for example. But this applies to r.what first of course...
An intermediate solution could be to pipe something like:
v.to.db map=INPUT_VECTOR_MAP option=coor qcolumn=cat separator=comma -p | awk -v FS=',' -v OFS=',' '{print $2,$3,$1}' | tail -n +2
to the coordinates option in r.what... That way the cat column would show up in the r.what output...

Thanks again for the excellent tool!

Cheers,
Stefan

-----Original Message-----
From: grass-dev-bounces@lists.osgeo.org [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of Blumentrath, Stefan
Sent: 29. januar 2015 07:30
To: grass-dev@lists.osgeo.org
Subject: Re: [GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

Great news! Thanks.
I shall test immediately.

-----Original Message-----
From: grass-dev-bounces@lists.osgeo.org [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of GRASS GIS
Sent: 29. januar 2015 00:16
Subject: Re: [GRASS-dev] [GRASS GIS] #2564: r.what: remove 400 maps limit and add some new output options

#2564: r.what: remove 400 maps limit and add some new output options
-------------------------+----------------------------------------------
-------------------------+----
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.what | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+----------------------------------------------
-------------------------+----

Comment(by huhabla):

I have just committed a new temporal module called t.rast.what in trunk
(r64349 - r64351) that utilized r.what to sample space time raster datasets using vector points. It provides three output layouts that transforms the r.what output into row or column layouts. It can run several r.what processes in parallel using a maximum of 400 raster map layer in a single r.what process.

Please have a look at it.

Any feedback about its performance, usefulness and handling is highly welcome.

Improving r.what to allow more than 400 maps is simple, just edit the fixed value in the source. But, editing this value may cause open file handler limit problems with your OS. However, these limits can be adjusted at kernel level configuration for Linux.

Implementing a more intelligent file handling solution requires more effort.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2564#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev