[GRASS-dev] [GRASS GIS] #2892: t.rast.aggregate: error when input space time raster dataset store many rasters

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-------------------------------------------------+-------------------------
Reporter: tmsz | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Keywords: wingrass, t.rast.aggregate, | CPU: x86-64
  g.message |
Platform: MSWindows 7 |
-------------------------------------------------+-------------------------
Steps to reproduce (follow example from Soren Gebbert presentation: The
temporal GRASS GIS framework, page 53):[[BR]]

* download the ECAD climate dataset from
[http://eca.knmi.nl/download/ensembles/data/Grid_0.25deg_reg/\] as several
netCDF files (tg_0.25deg_reg_1950-1964_v12.0.nc.gz,
tg_0.25deg_reg_1965-1979_v12.0.nc.gz,
tg_0.25deg_reg_1965-1979_v12.0.nc.gz,
tg_0.25deg_reg_1995-2015_v12.0.nc.gz) and import them with r.in.gdal in a
Lat/Lon location.[[BR]]

* import netCDF files
  * {{{r.in.gdal -oe input=tg_0.25deg_reg_1950-1964_v5.0.nc
output=temperature_mean offset=0}}}[[BR]]
  * {{{r.in.gdal -oe input=tg_0.25deg_reg_1965-1979_v12.0.nc
output=temperature_mean offset=5480}}}[[BR]]
  * {{{r.in.gdal -oe input=tg_0.25deg_reg_1980-1994_v12.0.nc
output=temperature_mean offset=10959}}}[[BR]]
  * {{{r.in.gdal -oe input=tg_0.25deg_reg_1995-2015_v12.0.nc
output=temperature_mean offset=16439}}}[[BR]]

   total count of temperature_mean rasters = 23925[[BR]]

  * {{{t.create type=strds output=temperature_mean_1950_2015_daily
temporal=absolute title="European mean temperature 1950-2015"
description="The European daily mean temperature"}}}[[BR]]

  * create text file with all temperature_mean rasters[[BR]]

{{{
#!div style="font-size: 100%"
    python code:
    {{{#!python
import os
map_list = os.path.join(os.path.expanduser('~'), 'map_list.txt')
with open(map_list, 'w') as f:
     for i in range(23925):
         f.write('temperature_mean.{0}\n'.format(i + 1))
    }}}
}}}

* {{{t.register -i type=rast input=temperature_mean_1950_2015_daily
file="path to the map_list.txt file" start=1950-01-01 increment="1
days"}}}[[BR]]

* {{{g.region -p rast=temperature_mean.1}}}[[BR]]

* {{{t.rast.aggregate input=temperature_mean_1950_2015_daily
method=average output=temperature_mean_1950_2015_monthly
base=temperature_mean_monthly granularity="1 months"}}}[[BR]]

Error message:
{{{
Traceback (most recent call last):
   File "C:\Program Files (x86)\GRASS GIS
7.0.2/scripts/t.rast.aggregate.py", line 215, in <module>
     main()
   File "C:\Program Files (x86)\GRASS GIS
7.0.2/scripts/t.rast.aggregate.py", line 185, in main
     nprocs))
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 580, in info
     message(msg, flag='i')
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 551, in message
     run_command("g.message", flags=flag, message=msg,
errors='ignore')
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 396, in
run_command
     ps = start_command(*args, **kwargs)
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 364, in
start_command
     return Popen(args, **popts)
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 67, in __init__
     subprocess.Popen.__init__(self, args, **kwargs)
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\Python27\lib\subprocess.py", line 711, in __init__
     errread, errwrite)
   File "C:\Program Files (x86)\GRASS GIS
7.0.2\Python27\lib\subprocess.py", line 953, in
_execute_child
     startupinfo)
WindowsError: [Error 206] The filename or extension is too
long
}}}

Error appear in the core.py file, class Popen (row 67). I try debug
subprocess.py file. Error realated with creating process (subprocess.py
file, method _subprocess.CreateProcess, row 946) when g.message.exe
command is callled (created process) with long message argument as args
parameter in method _subprocess.CreateProcess (attached file with args
argument). [[BR]]

On Windows OS platform is command-line string limitation for total length
of the following command line [https://support.microsoft.com/en-
us/kb/830473].

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2892&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------
Changes (by tmsz):

* Attachment "args.tar.xz" added.

args argument text file

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2892&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------

Comment (by veroandreo):

Hi,

I followed your workflow but I wasn't able to reproduce the error you get.
I'm using grass71 (r67703) under Fedora 23. Have you tried with the new
grass gis stable version 7.0.3?

Btw, your offsets in r.in.gdal didn't work for me, they left one map out
and then it was not possible to register maps. So, here is how I changed
them (and the total count is now 23922):

{{{
r.in.gdal -oe input=tg_0.25deg_reg_1950-1964_v12.0.nc
output=temperature_mean offset=0
r.in.gdal -oe input=tg_0.25deg_reg_1965-1979_v12.0.nc
output=temperature_mean offset=5479 --o
r.in.gdal -oe input=tg_0.25deg_reg_1980-1994_v12.0.nc
output=temperature_mean offset=10957 --o
r.in.gdal -oe input=tg_0.25deg_reg_1995-2015_v12.0.nc
output=temperature_mean offset=16436 --o
}}}

For the rest, the monthly aggregation works just fine here and no error
regarding filename or extension. Have you tried shrinking names? Something
like: basename=month_tmean?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2892#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------

Comment (by tmsz):

Hi,

This error appear on OS Windows platform only. Offset for import rasters
can be different on OS GNU/Linux. I think problem is not with raster name
length.[[BR]]

When I run t.rast.aggregate, which inner create process (in the source
code for create new process is called python method
_subprocess.CreateProcess() on OS Windows platform) with argument which is
command-line calling g.message command with argument message which is very
long. On Windows OS platform is command-line string limitation for total
length of the following command line (last previous link).

{{{
#!div style="font-size: 100%"
short preview, how is g.message called on OS Windows:
    {{{
  "C:\Program Files (x86)\GRASS GIS 7.0.2\bin\g.message.EXE" -i
"message=granularity_list:
[<grass.temporal.space_time_datasets.RasterDataset object at 0x040968F0>,
...
    }}}
}}}

Check the text file (complete g.message command calling) in the attached
previous archive.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2892#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------

Comment (by tmsz):

Hi again,

I appologize for my bad offset value. Your offset value are correct.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2892#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------

Comment (by veroandreo):

Hi

No problem... But that was not the issue anyway, right? So, I really
recommend to update to the last stable release and try again (perhaps
there has been some change with regards to the version you are using), and
maybe also try with shorter names.

I'm not a programmer, so unfortunately I cannot further help here... we
need a specialist :slight_smile:

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2892#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------

Comment (by tmsz):

Hi again,

I tried last stable grass gis (wingrass) 7.0.3, code revision 67691 and
t.rast.aggregate work correctly now. Thank you for your help and time :).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2892#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#2892: t.rast.aggregate: error when input space time raster dataset store many
rasters
-----------------------+---------------------------------------------------
  Reporter: tmsz | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.4
Component: Temporal | Version: 7.0.2
Resolution: fixed | Keywords: wingrass, t.rast.aggregate, g.message
       CPU: x86-64 | Platform: MSWindows 7
-----------------------+---------------------------------------------------
Changes (by tmsz):

* status: new => closed
* resolution: => fixed

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2892#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>