[GRASS-user] Aggregating daily maps in relative strds per month

GRASS GRASS user
1

Is there a better way to aggregate a *relative* spatio-temporal raster
data set composed by daily maps (output from r.sun.daily) in monthly
average strds-es?

At the moment I use the following csv and script:

--%<---
# month_start_end_day.csv
January|1|31
February|32|59
March|60|90
April|91|120
May|121|151
June|152|181
July|182|212
August|213|243
September|244|273
October|274|304
November|305|334
December|335|365
--->%--

and

--%<---
# loop over spatio-temporal raster data sets of interest
for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do
    # loop over months (start day, end day, month name) in csv
    for LINE in `cat month_start_end_day.csv`

    do
        # set positional parameters
        set -- $(echo ${LINE} |tr "|" " ")

        # aggregate
        t.rast.aggregate input=${STRDS} output=${STRDS}_$1_average basename=${STRDS}_$1_average where="start_time >= $2 AND start_time <= $3" granularity=1 --o
    done

done
--->%--

Any tip highly appreciated, Nikos

2

Hi Nikos!

If the data in fact represent daily data, why don’t you register it as absolute time?? In that way, you would just loop over strds…

for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do
t.rast.aggregate input=${STRDS} output=${STRDS}_monthly_average basename=${STRDS}_average granularity=“1 month” method=average
done

Cheers,
Vero

Is there a better way to aggregate a relative spatio-temporal raster
data set composed by daily maps (output from r.sun.daily) in monthly
average strds-es?

At the moment I use the following csv and script:

–%<—

month_start_end_day.csv

January|1|31
February|32|59
March|60|90
April|91|120
May|121|151
June|152|181
July|182|212
August|213|243
September|244|273
October|274|304
November|305|334
December|335|365
—>%–

and

–%<—

loop over spatio-temporal raster data sets of interest

for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do

loop over months (start day, end day, month name) in csv

for LINE in cat month_start_end_day.csv

do

set positional parameters

set – $(echo ${LINE} |tr “|” " ")

aggregate

t.rast.aggregate input=${STRDS} output=${STRDS}$1_average basename=${STRDS}$1_average where=“start_time >= $2 AND start_time <= $3” granularity=1 --o
done

done
—>%–

Any tip highly appreciated, Nikos

grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

3

* Veronica Andreo <veroandreo@gmail.com> [2015-05-17 17:07:30 -0300]:

Hi Nikos!

Ciao Vero :-).

If the data in fact represent daily data, why don't you register it as
absolute time?? In that way, you would just loop over strds...

It is output from r.sun.daily. Below, looping over monthly Linke
Turbidity values.

--%<---
for LINE in `cat monthly_linke_map_start_day_end_day.csv`
do

    set -- $(echo ${LINE} |tr "|" " ")
    
    r.sun.daily elevation=srtm4@PERMANENT aspect_value=0 slope_value=0.0 \
    albedo=albedo@albedo linke=$1@linke start_day=$2 end_day=$3 \
    glob_rad_basename=global_rad_zero insol_time_basename=insol_time \
    beam_rad_basename=beam_rad_zero diff_rad_basename=diff_rad_zero --o

done
--->%--

The script creates automatically
an "relative" strds, and for a good reason I guess. I can't think which
of the input's, for r.sun.daily, would fit for an absolute timestamping.

For sure it's not going to be the elevation model. Linke Turbidity and
albeo also not. Any help is highly appreciated if there is something
for which an absolute timestamping can be justified.

for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do
t.rast.aggregate input=${STRDS} output=${STRDS}_monthly_average
basename=${STRDS}_average granularity="1 month" method=average
done

So, the above won't work :-(. I don't know how bad the idea of having
preset months (start and end days for such cases) to operate on relative
s-t data sets.

Plus, an idea I already expressed elsewhere in the list: will it be
useful to have support for multiple LinkeT maps? Not sure how the
implementation should look like yet.

Cheers,
Vero

Thanks for caring, Nikos

--%<---

Nikos:

Is there a better way to aggregate a *relative* spatio-temporal raster
data set composed by daily maps (output from r.sun.daily) in monthly
average strds-es?

At the moment I use the following csv and script:

--%<---
# month_start_end_day.csv
January|1|31
February|32|59
March|60|90
April|91|120
May|121|151
June|152|181
July|182|212
August|213|243
September|244|273
October|274|304
November|305|334
December|335|365
--->%--

and

--%<---
# loop over spatio-temporal raster data sets of interest
for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do
    # loop over months (start day, end day, month name) in csv
    for LINE in `cat month_start_end_day.csv`

    do
        # set positional parameters
        set -- $(echo ${LINE} |tr "|" " ")

        # aggregate
        t.rast.aggregate input=${STRDS} output=${STRDS}_$1_average
basename=${STRDS}_$1_average where="start_time >= $2 AND start_time <= $3"
granularity=1 --o
    done

done
--->%--

Any tip highly appreciated, Nikos
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

--
Nikos Alexandris | Remote Sensing Scientist, Dr
Themidos 3, 42100, Trikala, Greece
GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

4

Ciao Nikos, again :slight_smile:

In case of relative time the temporal unit of the input space time raster dataset is used. The granularity must be specified with an integer value.

···

If the data in fact represent daily data, why don’t you register it as
absolute time?? In that way, you would just loop over strds…

It is output from r.sun.daily. Below, looping over monthly Linke
Turbidity values.

–%<—
for LINE in cat monthly_linke_map_start_day_end_day.csv
do

set – $(echo ${LINE} |tr “|” " ")

r.sun.daily elevation=srtm4@PERMANENT aspect_value=0 slope_value=0.0
albedo=albedo@albedo linke=$1@linke start_day=$2 end_day=$3
glob_rad_basename=global_rad_zero insol_time_basename=insol_time
beam_rad_basename=beam_rad_zero diff_rad_basename=diff_rad_zero --o

done
—>%–

The script creates automatically
an “relative” strds, and for a good reason I guess. I can’t think which
of the input’s, for r.sun.daily, would fit for an absolute timestamping.

For sure it’s not going to be the elevation model. Linke Turbidity and
albeo also not. Any help is highly appreciated if there is something
for which an absolute timestamping can be justified.

Never used this add-on but I see now it creates a strds with relative time. it seems you cannot change that, yo can’t specify anywhere an absolute time, that’s right. But as then you don’t have a constant granularity to aggregate data, cos in fact you want to aggregate it as monthly absolute data (otherwise you’d just use an integer number, i.e.: 10, 20, 30, whatever)… I would take maps in the output of r.sun daily and register them in another strds (with absolute time) and then aggregate monthly…

for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time

t.create type=strds temporaltype=absolute output=${STRDS} title=“bla” description=“bla” --o

t.register -i type=rast input=${STRDS} maps=g.list rast pat=${STRDS}* sep=, start=“2003-01-01” increment=“1 day”

or whatever pattern you use, or a list of ordered maps

then the aggregation

t.rast.aggregate input=${STRDS} output=${STRDS}_monthly_average basename=${STRDS}_average granularity=“1 month” method=average

done

just an idea…
or use your script as it was :slight_smile:

for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do
t.rast.aggregate input=${STRDS} output=${STRDS}_monthly_average
basename=${STRDS}_average granularity=“1 month” method=average
done

So, the above won’t work :-(. I don’t know how bad the idea of having
preset months (start and end days for such cases) to operate on relative
s-t data sets.

dunno… thing is, in relative time, how do you know for every possible case at which point does a month start… i mean, for that, you have absolute time, right? Maybe, r.sun.daily could output an absolute time strds… don’t know if it’s feasible though

Plus, an idea I already expressed elsewhere in the list: will it be
useful to have support for multiple LinkeT maps? Not sure how the
implementation should look like yet.

Cheers,
Vero

Thanks for caring, Nikos

Niente! Not so useful though :slight_smile:
Cheers,
Vero

5

* Veronica Andreo <veroandreo@gmail.com> [2015-05-18 06:07:24 -0300]:

Ciao Nikos, again :slight_smile:

In case of relative time the temporal unit of the input space time raster
dataset is used. The granularity must be specified with an integer value.

> If the data in fact represent daily data, why don't you register it as
> > absolute time?? In that way, you would just loop over strds...

I have to decide for which year. And, with global irradiation, I am not
sure it makes sense. An expert opinion would help here.

> It is output from r.sun.daily. Below, looping over monthly Linke
> Turbidity values.
>
> --%<---
> for LINE in `cat monthly_linke_map_start_day_end_day.csv`
> do
>
> set -- $(echo ${LINE} |tr "|" " ")
>
> r.sun.daily elevation=srtm4@PERMANENT aspect_value=0 slope_value=0.0 \
> albedo=albedo@albedo linke=$1@linke start_day=$2 end_day=$3 \
> glob_rad_basename=global_rad_zero insol_time_basename=insol_time \
> beam_rad_basename=beam_rad_zero diff_rad_basename=diff_rad_zero --o
>
> done
> --->%--
>
> The script creates automatically
> an "relative" strds, and for a good reason I guess. I can't think which
> of the input's, for r.sun.daily, would fit for an absolute timestamping.
>
> For sure it's not going to be the elevation model. Linke Turbidity and
> albeo also not. Any help is highly appreciated if there is something
> for which an absolute timestamping can be justified.
>

Never used this add-on but I see now it creates a strds with relative time.
it seems you cannot change that, yo can't specify anywhere an absolute
time, that's right.

Well, I can. In fact I have added missing options (and will update the
add-on once I have done tests and someone else confirms it's ok). But, I
think it doesn't make sense.

But as then you don't have a constant granularity to
aggregate data, cos in fact you want to aggregate it as monthly absolute
data (otherwise you'd just use an integer number, i.e.: 10, 20, 30,
whatever)... I would take maps in the output of r.sun daily and register
them in another strds (with absolute time) and then aggregate monthly...

for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time

t.create type=strds temporaltype=absolute output=${STRDS} title="bla"
description="bla" --o

t.register -i type=rast input=${STRDS} maps=`g.list rast pat=${STRDS}*
sep=,` start="2003-01-01" increment="1 day"
# or whatever pattern you use, or a list of ordered maps

Right, this "start" is the "missing" key.

# then the aggregation
t.rast.aggregate input=${STRDS} output=${STRDS}_monthly_average
basename=${STRDS}_average granularity="1 month" method=average

done

just an idea...
or use your script as it was :slight_smile:

Right. Thanks for your tutorial in the Wiki. It got me going in
no-time. Amazing work.

> for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
> > do
> > t.rast.aggregate input=${STRDS} output=${STRDS}_monthly_average
> > basename=${STRDS}_average granularity="1 month" method=average
> > done
>
> So, the above won't work :-(. I don't know how bad the idea of having
> preset months (start and end days for such cases) to operate on relative
> s-t data sets.
>

dunno... thing is, in relative time, how do you know for every possible
case at which point does a month start... i mean, for that, you have
absolute time, right? Maybe, r.sun.daily could output an absolute time
strds... don't know if it's feasible though

Correct. I'll just keep on working manually for the time being.

> Plus, an idea I already expressed elsewhere in the list: will it be
> useful to have support for multiple LinkeT maps? Not sure how the
> implementation should look like yet.
>
> > Cheers,
> > Vero
>
> Thanks for caring, Nikos
>

Niente! Not so useful though :slight_smile:
Cheers,
Vero

The discussion forces me to clear my understanding. So, very useful
actually.

Ciao-ciao!

6

Hi Nikos,
sorry for the late response.

The aggregation of daily relative time STRDS with Gregorian monthly
granularity is not possible, since in relative time mode, there is no
connection between daily and monthly calendar hierarchy. The reason is
that in relative time mode there is no starting point in calendar time
at which day 1,2 or n starts. Hence you never know to which month or
year a day is associated, because of the lack of a calendar hierarchy.
The number of days per month and per year are different in the
Gregorian calendar, hence the output of r.sun.daily should be in my
humble opinion yearly and monthly specific.

However, you can create a second STDS with relative time, that
contains of 12 maps with the according time intervals (31 days, 28/29
days, ...) and use t.rast.aggregate.ds to aggregate the first STRDS
based on the time intervals of the second STDS.

Best regards
Soeren

2015-05-16 13:41 GMT+02:00 Nikos Alexandris <nik@nikosalexandris.net>:

Is there a better way to aggregate a *relative* spatio-temporal raster
data set composed by daily maps (output from r.sun.daily) in monthly
average strds-es?

At the moment I use the following csv and script:

--%<---
# month_start_end_day.csv
January|1|31
February|32|59
March|60|90
April|91|120
May|121|151
June|152|181
July|182|212
August|213|243
September|244|273
October|274|304
November|305|334
December|335|365
--->%--

and

--%<---
# loop over spatio-temporal raster data sets of interest
for STRDS in global_rad_zero beam_rad_zero diff_rad_zero insol_time
do
    # loop over months (start day, end day, month name) in csv
    for LINE in `cat month_start_end_day.csv`

    do
        # set positional parameters
        set -- $(echo ${LINE} |tr "|" " ")

        # aggregate
        t.rast.aggregate input=${STRDS} output=${STRDS}_$1_average basename=${STRDS}_$1_average where="start_time >= $2 AND start_time <= $3" granularity=1 --o
    done

done
--->%--

Any tip highly appreciated, Nikos
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

7

* Sören Gebbert <soerengebbert@googlemail.com> [2015-05-18 13:08:08 +0200]:

Hi Nikos,
sorry for the late response.

The aggregation of daily relative time STRDS with Gregorian monthly
granularity is not possible, since in relative time mode, there is no
connection between daily and monthly calendar hierarchy. The reason is
that in relative time mode there is no starting point in calendar time
at which day 1,2 or n starts. Hence you never know to which month or
year a day is associated, because of the lack of a calendar hierarchy.

Right.

The number of days per month and per year are different in the
Gregorian calendar, hence the output of r.sun.daily should be in my
humble opinion yearly and monthly specific.

The differences are so subtle (for example for Linke Turbdity) that
maybe it doesn't make sense to have different data sets for different
years. See also, for example the monthly sums of global irradiation on a
horizontal surface (kWh/m2) period 1998-2011 from
<http://re.jrc.ec.europa.eu/pvgis/download/data/hor_rad_PVGIS_cmsaf_monthly_latlon.zip&gt;\.

They are just 12 maps. To get the average values, one needs to divide
with... ? 30 for all months?

Having absolute timestamps would make life easier with the
fantastic temporal framework. I'll re-do again my workflow and decide
upon a year. After all, I did extract the following for a specific year
(don't remember which one it was now :-?):

January|1|31|01
February|32|59|02
March|60|90|03
April|91|120|04
May|121|151|05
June|152|181|06
July|182|212|07
August|213|243|08
September|244|273|09
October|274|304|10
November|305|334|11
December|335|365|12

However, you can create a second STDS with relative time, that
contains of 12 maps with the according time intervals (31 days, 28/29
days, ...) and use t.rast.aggregate.ds to aggregate the first STRDS
based on the time intervals of the second STDS.

That's cool! Danke vielmals Soeren.

Nikos

8

Nikos Alexandris:

Having absolute timestamps would make life easier with the
fantastic temporal framework. I'll re-do again my workflow and decide
upon a year. After all, I did extract the following for a specific year
(don't remember which one it was now :-?):

I think it was 2009. So, I am trying to go 'absolute':

# list maps
g.list raster pattern=global_rad_zero_[0-9][0-9][0-9]

global_rad_zero_001
..
global_rad_zero_365

# erase timestamps
for Map in `g.list raster pattern=global_rad_zero_[0-9][0-9][0-9]` ;do
r.timestamp map=${Map} date=none ;done

# create an absolute strds
t.create out=global_rad_zero title="Clearsky global irradiation"
description="Clearsky global irradiation on a horizontal surface raster
maps
[Wh.m-2.day-1]" --o

# register maps (which have no timestamp!)
t.register -i input=global_rad_zero maps=`g.list raster pattern=global_rad_zero_[0-9][0-9][0-9] sep=,` start="2009-01-01" increment='1 days' --o

0..ERROR: invalid literal for int() with base 10: '2009-01-01'

Why? Nikos

9

Hi Nikos,
You need to use t.remove to delete the time stamps from the temporal database. The *.timestamp modules should not be used at all. They are not connected with the temporal framework.
The error appears because the maps still have relative time stamps in the temporal database which can not be overwritten with absolute time stamps.

Ciao
Sören

Am 18.05.2015 19:47 schrieb “Nikos Alexandris” <nik@nikosalexandris.net>:

Nikos Alexandris:

Having absolute timestamps would make life easier with the
fantastic temporal framework. I’ll re-do again my workflow and decide
upon a year. After all, I did extract the following for a specific year
(don’t remember which one it was now :-?):

I think it was 2009. So, I am trying to go ‘absolute’:

list maps

g.list raster pattern=global_rad_zero_[0-9][0-9][0-9]

global_rad_zero_001
…
global_rad_zero_365

erase timestamps

for Map in g.list raster pattern=global_rad_zero_[0-9][0-9][0-9] ;do
r.timestamp map=${Map} date=none ;done

create an absolute strds

t.create out=global_rad_zero title=“Clearsky global irradiation”
description=“Clearsky global irradiation on a horizontal surface raster
maps
[Wh.m-2.day-1]” --o

register maps (which have no timestamp!)

t.register -i input=global_rad_zero maps=g.list raster pattern=global_rad_zero_[0-9][0-9][0-9] sep=, start=“2009-01-01” increment=‘1 days’ --o

0…ERROR: invalid literal for int() with base 10: ‘2009-01-01’

Why? Nikos

10

* Sören Gebbert <soerengebbert@googlemail.com> [2015-05-18 20:34:19 +0200]:

Hi Nikos,
You need to use t.remove to delete the time stamps from the temporal
database. The *.timestamp modules should not be used at all. They are not
connected with the temporal framework.
The error appears because the maps still have relative time stamps in the
temporal database which can not be overwritten with absolute time stamps.

Thanks for spotting. I am not sure about the following:

1) how does t.connect/t.create work if I erase the "tgis"
directory manually? Is it not possible to create a new tgis from
scratch?

2) although I remove strds-es via t.remove, the command `t.list rast` still
shows maps which do not exist anymore anywhere. How come?

Danke, Nikos

11

Hi Nikos,

2015-05-18 23:07 GMT+02:00 Nikos Alexandris <nik@nikosalexandris.net>:

* Sören Gebbert <soerengebbert@googlemail.com> [2015-05-18 20:34:19 +0200]:

Hi Nikos,
You need to use t.remove to delete the time stamps from the temporal
database. The *.timestamp modules should not be used at all. They are not
connected with the temporal framework.
The error appears because the maps still have relative time stamps in the
temporal database which can not be overwritten with absolute time stamps.

Thanks for spotting. I am not sure about the following:

1) how does t.connect/t.create work if I erase the "tgis"
directory manually? Is it not possible to create a new tgis from
scratch?

The tgis directory can be removed, all temporal commands will check if
the mapset specific temporal database exists and will create one if
needed.

2) although I remove strds-es via t.remove, the command `t.list rast` still
shows maps which do not exist anymore anywhere. How come?

Maps can be registered in several different STDS. Hence removing a
STDS without explicitly removing the registered map layers (forced
with -rf) will only un-register the maps from the specific STDS but
not from the temporal database.
Use t.unregister to simply un-register map layers from the temporal
database (and all associated STDS) or use the -rf option in t.remove
to fully remove map layers from the temporal and spatial database. The
module g.remove will not delete the time stamps from the temporal
database, since only temporal commands are aware of it. The module
t.support allows to check STDS for removed or overwritten map layers
to update the temporal database accordingly.

Best regards
Soeren

Danke, Nikos

12

* Sören Gebbert <soerengebbert@googlemail.com> [2015-05-18 23:17:55 +0200]:

Hi Nikos,

2015-05-18 23:07 GMT+02:00 Nikos Alexandris <nik@nikosalexandris.net>:
> * Sören Gebbert <soerengebbert@googlemail.com> [2015-05-18 20:34:19 +0200]:
>
>> Hi Nikos,
>> You need to use t.remove to delete the time stamps from the temporal
>> database. The *.timestamp modules should not be used at all. They are not
>> connected with the temporal framework.
>> The error appears because the maps still have relative time stamps in the
>> temporal database which can not be overwritten with absolute time stamps.
>
> Thanks for spotting. I am not sure about the following:
>
> 1) how does t.connect/t.create work if I erase the "tgis"
> directory manually? Is it not possible to create a new tgis from
> scratch?

The tgis directory can be removed, all temporal commands will check if
the mapset specific temporal database exists and will create one if
needed.

I removed the tgis directory manually (rm -rf). Then issued `t.connect -d` or/and
`t.connect -c`. Though this sets the connection driver/path in the VAR
file, the '-p' reports as if all is ok, I only receive the error

ERROR: Unable to connect to sqlite3 database:
/geo/grassdb/ellas/oiti/wgs84_old/PERMANENT/tgis/sqlite.db
Exception: "unable to open database file"
Please use t.connect to set a read- and writable temporal database
backend

from any of the t.connect or t.create commands I tried.

> 2) although I remove strds-es via t.remove, the command `t.list rast` still
> shows maps which do not exist anymore anywhere. How come?

Oh my, I think I've asked that again in the past, apologies.

Maps can be registered in several different STDS. Hence removing a
STDS without explicitly removing the registered map layers (forced
with -rf) will only un-register the maps from the specific STDS but
not from the temporal database.
Use t.unregister to simply un-register map layers from the temporal
database (and all associated STDS) or use the -rf option in t.remove
to fully remove map layers from the temporal and spatial database. The
module g.remove will not delete the time stamps from the temporal
database, since only temporal commands are aware of it. The module
t.support allows to check STDS for removed or overwritten map layers
to update the temporal database accordingly.

Yet, what if g.remove is used anyhow for a series of registered maps.
How will unregister work it the maps don't exist anymore? t.remove -rf
doesn't do it. I mean, t.remove expects some strds, for example, to
operate, right? And t.support the same.

Maybe this has been also answered by you in this previous mail in the
list (I can hardly recall -- will search).

Thanks for your patience, Nikos

13

I couldn't find my way out to clean the existing mess I created. I am
re-doing all from scratch. It was all worth the time spent though.

It there is interest for open questions (of mine), below an error
message and a question.

Thanks, Nikos

Soeren:

> The tgis directory can be removed, all temporal commands will check if
> the mapset specific temporal database exists and will create one if
> needed.

Nikos:

I removed the tgis directory manually (rm -rf). Then issued `t.connect -d` or/and
`t.connect -c`. Though this sets the connection driver/path in the VAR
file, the '-p' reports as if all is ok, I only receive the error

ERROR: Unable to connect to sqlite3 database:
/geo/grassdb/ellas/oiti/wgs84_old/PERMANENT/tgis/sqlite.db
Exception: "unable to open database file"
Please use t.connect to set a read- and writable temporal database
backend

from any of the t.connect or t.create commands I tried.

+ a more detailed message from withing a GRASS session and ipython,
trying to force tgis.init():

--%<---
In [3]: %tb
---------------------------------------------------------------------------
SystemExit Traceback (most recent call last)
<ipython-input-2-9e2cb3184ba5> in <module>()
----> 1 tgis.init()

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in init(raise_fatal_error)
    658 return
    659
--> 660 create_temporal_database(dbif)
    661
    662 ###############################################################################

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in create_temporal_database(dbif)
    769 # Connect now to the database
    770 if not dbif.connected:
--> 771 dbif.connect()
    772
    773 # Execute the SQL statements for sqlite

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in connect(self)
    878 conn = self.connections[mapset]
    879 if conn.is_connected() is False:
--> 880 conn .connect(dbstring)
    881
    882 self.connected = True

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in connect(self, dbstring)
   1077 "temporal database backend") % (
   1078 {"db": self.dbmi.__name__,
-> 1079 "string": tgis_database_string, "ex": e, }))
   1080
   1081 def close(self):

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/pygrass/messages/__init__.pyc in fatal(self, message)
    269 raise FatalError(message)
    270 else:
--> 271 sys.exit(1)
    272
    273 def debug(self, level, message):

SystemExit: 1
--->%--

Nikos:

> > 2) although I remove strds-es via t.remove, the command `t.list rast` still
> > shows maps which do not exist anymore anywhere. How come?

Soeren:

> Maps can be registered in several different STDS. Hence removing a
> STDS without explicitly removing the registered map layers (forced
> with -rf) will only un-register the maps from the specific STDS but
> not from the temporal database.
> Use t.unregister to simply un-register map layers from the temporal
> database (and all associated STDS) or use the -rf option in t.remove
> to fully remove map layers from the temporal and spatial database. The
> module g.remove will not delete the time stamps from the temporal
> database, since only temporal commands are aware of it. The module
> t.support allows to check STDS for removed or overwritten map layers
> to update the temporal database accordingly.

What happens if ones removes the strds using t.remove without '-rf' or/and no
t.support -m was executed. And, additionally, removes a series of (previously)
registered maps as well using g.remove? It seems that this results in ghost entries
in the t-db (?).

t.unregister will not work if the maps and the strds don't exist, right?
t.remove -rf

Nikos

14

Hi Nikos,
being not able to recreate the temporal database in a mapset after
deleting it is definitely a bug in the temporal framework. I have to
investigate this.

Regarding ghost entries:

You can use "t.list type=rast" to list all raster map layer registered
in the temporal database, with or without STRDS association.
Use the "maps" option of module t.unregister to un-register any kind
of map layers (existing, ghost, non-STRDS association, ...).

Best regards
Soeren

2015-05-19 9:54 GMT+02:00 Nikos Alexandris <nik@nikosalexandris.net>:

I couldn't find my way out to clean the existing mess I created. I am
re-doing all from scratch. It was all worth the time spent though.

It there is interest for open questions (of mine), below an error
message and a question.

Thanks, Nikos

Soeren:

> The tgis directory can be removed, all temporal commands will check if
> the mapset specific temporal database exists and will create one if
> needed.

Nikos:

I removed the tgis directory manually (rm -rf). Then issued `t.connect -d` or/and
`t.connect -c`. Though this sets the connection driver/path in the VAR
file, the '-p' reports as if all is ok, I only receive the error

ERROR: Unable to connect to sqlite3 database:
/geo/grassdb/ellas/oiti/wgs84_old/PERMANENT/tgis/sqlite.db
Exception: "unable to open database file"
Please use t.connect to set a read- and writable temporal database
backend

from any of the t.connect or t.create commands I tried.

+ a more detailed message from withing a GRASS session and ipython,
trying to force tgis.init():

--%<---
In [3]: %tb
---------------------------------------------------------------------------
SystemExit Traceback (most recent call last)
<ipython-input-2-9e2cb3184ba5> in <module>()
----> 1 tgis.init()

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in init(raise_fatal_error)
    658 return
    659
--> 660 create_temporal_database(dbif)
    661
    662 ###############################################################################

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in create_temporal_database(dbif)
    769 # Connect now to the database
    770 if not dbif.connected:
--> 771 dbif.connect()
    772
    773 # Execute the SQL statements for sqlite

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in connect(self)
    878 conn = self.connections[mapset]
    879 if conn.is_connected() is False:
--> 880 conn .connect(dbstring)
    881
    882 self.connected = True

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.pyc in connect(self, dbstring)
   1077 "temporal database backend") % (
   1078 {"db": self.dbmi.__name__,
-> 1079 "string": tgis_database_string, "ex": e, }))
   1080
   1081 def close(self):

/osgeo/grass70/dist.x86_64-unknown-linux-gnu/etc/python/grass/pygrass/messages/__init__.pyc in fatal(self, message)
    269 raise FatalError(message)
    270 else:
--> 271 sys.exit(1)
    272
    273 def debug(self, level, message):

SystemExit: 1
--->%--

Nikos:

> > 2) although I remove strds-es via t.remove, the command `t.list rast` still
> > shows maps which do not exist anymore anywhere. How come?

Soeren:

> Maps can be registered in several different STDS. Hence removing a
> STDS without explicitly removing the registered map layers (forced
> with -rf) will only un-register the maps from the specific STDS but
> not from the temporal database.
> Use t.unregister to simply un-register map layers from the temporal
> database (and all associated STDS) or use the -rf option in t.remove
> to fully remove map layers from the temporal and spatial database. The
> module g.remove will not delete the time stamps from the temporal
> database, since only temporal commands are aware of it. The module
> t.support allows to check STDS for removed or overwritten map layers
> to update the temporal database accordingly.

What happens if ones removes the strds using t.remove without '-rf' or/and no
t.support -m was executed. And, additionally, removes a series of (previously)
registered maps as well using g.remove? It seems that this results in ghost entries
in the t-db (?).

t.unregister will not work if the maps and the strds don't exist, right?
t.remove -rf

Nikos

15

* Sören Gebbert:

being not able to recreate the temporal database in a mapset after
deleting it is definitely a bug in the temporal framework. I have to
investigate this.

Need a ticket for this?

Regarding ghost entries:

You can use "t.list type=rast" to list all raster map layer registered
in the temporal database, with or without STRDS association.
Use the "maps" option of module t.unregister to un-register any kind
of map layers (existing, ghost, non-STRDS association, ...).

It worked as follows:

# create a list with the maps in question
t.list rast |grep global_rad_clearsky > global_rad_clearsky_maps

# unregister using "it"
t.unregister file=global_rad_clearsky_maps

Question:

Is this eligible as an enhancement request? e.g. do:
t.unregister type=rast pattern=global_rad_clearsky*

?

Also, the maps derived from r.sun.daily have a relative timestamp
(irrelevant if the user requests for the '-t' flag to register
automatically in a strds or not).

With t.list at this point, there is no entry of the maps in question in
the temporal data base. Naturally. As well, the following won't work directly:

# create a strds
t.create out=global_rad_clearsky title="Clearsky global irradiation"
description="Clearsky global irradiation on a horizontal surface raster
maps [Wh.m-2.day-1]"

# (try to) register
t.register input=global_rad_clearsky maps=`g.list raster
pattern=global_rad_clearsky_* sep=,` start="2009-01-01" increment='1 days'

0..ERROR: invalid literal for int() with base 10: '2009-01-01'

Expected, right? To override this, without using the r.timestamp module
at all, I did:

# intentionally issuing a command that is expected to fail, yet...
t.register input=beam_rad_clearsky maps=`g.list raster
pattern=beam_rad_clearsky_* sep=,`
0..13..27..41..54..68..82..95..100

0..ERROR: Temporal type of space time dataset
<beam_rad_clearsky@global_rad_clearsky> and map
<beam_rad_clearsky_001@global_rad_clearsky> are different

...the above created entries with the requested map names in the
temporal data base as "maps with relative time available in mapset...".

So, unregister (as above, using a list with the map names in question).
This will clean the relative timestamps from the maps.

The, recalling the first t.register command, as is, to register the
maps in the absolute strds will work, eg:

t.register input=global_rad_clearsky maps=`g.list raster
pattern=global_rad_clearsky_* sep=,` start="2009-01-01" increment='1 days'

Question:

Would it make sense to create a flag assisting in working this around?
Or would it be a "faulty design" ?

Thanks, Nikos