[GRASS-user] detrending time series maps

GRASS GRASS user
1

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3 times per month), in the period November-February, for 7 years. They are not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that probably there is a descending trend when considering the maps ordered by id.

I would like to fit a proper time depending fitting curve for each pixel and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or some simple bash/python scripting?

thank you

Ivan

2

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within t.rast.algebra to detrend the values of the maps within the strds, something like “detrended_strds = trend_strds - (trend_strds*map(slope) + map(offset))”. Others suggest, to detrend by subtracting the previous value, i.e. that would imply using the temporal algebra with the temporal index, something like “detrended_strds = trend_strds[1] - trend_strds[0]”.

I haven’t tested any of these, just a couple of ideas :wink: However, I do not know how this might interact with seasonality within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<grass-user@lists.osgeo.org>) escribió:

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps ordered
by id.

I would like to fit a proper time depending fitting curve for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or
some simple bash/python scripting?

thank you

Ivan

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

3

Hi Veronica

Thank you. It goes in the direction of my idea evn if my problem is exactly trying to take into account the correct gaps between that data

I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan

···

On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within t.rast.algebra to detrend the values of the maps within the strds, something like “detrended_strds = trend_strds - (trend_strds*map(slope) + map(offset))”. Others suggest, to detrend by subtracting the previous value, i.e. that would imply using the temporal algebra with the temporal index, something like “detrended_strds = trend_strds[1] - trend_strds[0]”.

I haven’t tested any of these, just a couple of ideas :wink: However, I do not know how this might interact with seasonality within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<grass-user@lists.osgeo.org>) escribió:

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps ordered
by id.

I would like to fit a proper time depending fitting curve for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or
some simple bash/python scripting?

thank you

Ivan

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

4

Dear Veronica

I think I found a simple solution using temporal raster modules. Here is an example:

#evaluating info of the strds
eval t.info mystrds -g

#getting the starting day (of the year, 0-365) of my strds
startday=$(date -d “$start_time” “+%j”)

#Creating a new strds where each pixel has the value of the count of the days starting from the start_day of my strds (the start day in my dataset is in 2016)
t.rast.mapcalc inputs=mystrds expression=“(start_year()-2016)*365-${startday} +start_doy()” output=days basename=days nprocs=xxx --o

#fitting the trend equation
r.regression.series xseries=“t.rast.list in=days columns=name sep=, format=line” yseries=" t.rast.list in=mystrds columns=name sep=, format=line" out=regression_offset,regression_slope,regression_rsq,regression_t meth=offset,slope,rsq,t

#detrending
t.rast.mapcalc input=mystrds,days expression=“mystrds_detrend = mystrds-(regression_offset+regression_slope*days)” output=mystrds_detrend basename=mystrds_detrend nprocs=xxx method=start --o

Best

Ivan

···

On 23/12/23 14:53, Ivan Marchesini wrote:

Hi Veronica

Thank you. It goes in the direction of my idea evn if my problem is exactly trying to take into account the correct gaps between that data

I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan

On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within t.rast.algebra to detrend the values of the maps within the strds, something like “detrended_strds = trend_strds - (trend_strds*map(slope) + map(offset))”. Others suggest, to detrend by subtracting the previous value, i.e. that would imply using the temporal algebra with the temporal index, something like “detrended_strds = trend_strds[1] - trend_strds[0]”.

I haven’t tested any of these, just a couple of ideas :wink: However, I do not know how this might interact with seasonality within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<grass-user@lists.osgeo.org>) escribió:

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps ordered
by id.

I would like to fit a proper time depending fitting curve for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or
some simple bash/python scripting?

thank you

Ivan

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

5

Hello Ivan,

Thanks for coming back to this :slight_smile:

I see what you did with creating the days time series. In that way you acknowledge irregular gaps, right? Otherwise, as t.rast.series method=slope,offset uses r.series in the background, it will use index as independent variable and therefore maps are considered equally separated in time.

However, why do you multiply by days strds? From my understanding, detrending by subtracting the results of a model obbeys this rule: value(t) = observed(t) - predicted(t). Then, this mystrds-(regression_offset+regression_slopedays) should be mystrds-(regression_offset+regression_slopemystrds).

Best,
Vero

El jue, 28 dic 2023 a las 8:14, Ivan Marchesini (<ivan.marchesini@gmail.com>) escribió:

Dear Veronica

I think I found a simple solution using temporal raster modules. Here is an example:

#evaluating info of the strds
eval [t.info](http://t.info) mystrds -g

#getting the starting day (of the year, 0-365) of my strds
startday=$(date -d “$start_time” “+%j”)

#Creating a new strds where each pixel has the value of the count of the days starting from the start_day of my strds (the start day in my dataset is in 2016)
t.rast.mapcalc inputs=mystrds expression=“(start_year()-2016)*365-${startday} +start_doy()” output=days basename=days nprocs=xxx --o

#fitting the trend equation
r.regression.series xseries=“t.rast.list in=days columns=name sep=, format=line” yseries=" t.rast.list in=mystrds columns=name sep=, format=line" out=regression_offset,regression_slope,regression_rsq,regression_t meth=offset,slope,rsq,t

#detrending
t.rast.mapcalc input=mystrds,days expression=“mystrds_detrend = mystrds-(regression_offset+regression_slope*days)” output=mystrds_detrend basename=mystrds_detrend nprocs=xxx method=start --o

Best

Ivan

On 23/12/23 14:53, Ivan Marchesini wrote:

Hi Veronica

Thank you. It goes in the direction of my idea evn if my problem is exactly trying to take into account the correct gaps between that data

I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan

On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within t.rast.algebra to detrend the values of the maps within the strds, something like “detrended_strds = trend_strds - (trend_strds*map(slope) + map(offset))”. Others suggest, to detrend by subtracting the previous value, i.e. that would imply using the temporal algebra with the temporal index, something like “detrended_strds = trend_strds[1] - trend_strds[0]”.

I haven’t tested any of these, just a couple of ideas :wink: However, I do not know how this might interact with seasonality within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<grass-user@lists.osgeo.org>) escribió:

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps ordered
by id.

I would like to fit a proper time depending fitting curve for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or
some simple bash/python scripting?

thank you

Ivan

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

6

Hi veronica

I see what you did with creating the days time series. In that way you acknowledge irregular gaps, right?

yes this is the reason

However, why do you multiply by days strds? From my understanding, detrending by subtracting the results of a model obbeys this rule: value(t) = observed(t) - predicted(t). Then, this mystrds-(regression_offset+regression_slopedays) should be mystrds-(regression_offset+regression_slopemystrds).

hmm

may be I’m wrong but I used r.regression.series to assess a relationship between mystrds values and time (days)

As a consequence the offset and slope maps I obtain are b and a in the following linear equation

y=ax+b

i.e.

predicted_mystrds=slope*days+offset

This is why I suppose that for detrending I need to do:

mystrds-(predicted_mystrds)

i.e.

mystrds-(slope*days+offset)

I’m I wrong?

thank you

Ivan

7

Hi Ivan:

···

On 29/12/2023 20:34, Ivan Marchesini via grass-user wrote:

Hi veronica

I see what you did with creating the days time series. In that way you acknowledge irregular gaps, right?

yes this is the reason

However, why do you multiply by days strds? From my understanding, detrending by subtracting the results of a model obbeys this rule: value(t) = observed(t) - predicted(t). Then, this mystrds-(regression_offset+regression_slopedays) should be mystrds-(regression_offset+regression_slopemystrds).

hmm

may be I’m wrong but I used r.regression.series to assess a relationship between mystrds values and time (days)

As a consequence the offset and slope maps I obtain are b and a in the following linear equation

y=ax+b

i.e.

predicted_mystrds=slope*days+offset

This is why I suppose that for detrending I need to do:

mystrds-(predicted_mystrds)

i.e.

mystrds-(slope*days+offset)

I’m not sure about your procedure of regressing days from start of the time series vs the time series. But I would point out:

The “elephant in the room” for any time series analysis is temporal autocorrelation. If you have some seasonal variation in your data, then I don’t think that such a regression could work. You mentioned data from Nov-Feb over several years, so I assume there is some seasonal effect going on? Normally you have to first decompose the time series to remove the seasonal variation, and then examine the trend.

I’m I wrong?

thank you

Ivan

Best,
Vero

El jue, 28 dic 2023 a las 8:14, Ivan Marchesini (<ivan.marchesini@gmail.com>) escribió:

Dear Veronica

I think I found a simple solution using temporal raster modules. Here is an example:

#evaluating info of the strds
eval [t.info](http://t.info) mystrds -g

#getting the starting day (of the year, 0-365) of my strds
startday=$(date -d “$start_time” “+%j”)

#Creating a new strds where each pixel has the value of the count of the days starting from the start_day of my strds (the start day in my dataset is in 2016)
t.rast.mapcalc inputs=mystrds expression=“(start_year()-2016)*365-${startday} +start_doy()” output=days basename=days nprocs=xxx --o

#fitting the trend equation
r.regression.series xseries=“t.rast.list in=days columns=name sep=, format=line” yseries=" t.rast.list in=mystrds columns=name sep=, format=line" out=regression_offset,regression_slope,regression_rsq,regression_t meth=offset,slope,rsq,t

#detrending
t.rast.mapcalc input=mystrds,days expression=“mystrds_detrend = mystrds-(regression_offset+regression_slope*days)” output=mystrds_detrend basename=mystrds_detrend nprocs=xxx method=start --o

Best

Ivan

On 23/12/23 14:53, Ivan Marchesini wrote:

Hi Veronica

Thank you. It goes in the direction of my idea evn if my problem is exactly trying to take into account the correct gaps between that data

I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan

On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within t.rast.algebra to detrend the values of the maps within the strds, something like “detrended_strds = trend_strds - (trend_strds*map(slope) + map(offset))”. Others suggest, to detrend by subtracting the previous value, i.e. that would imply using the temporal algebra with the temporal index, something like “detrended_strds = trend_strds[1] - trend_strds[0]”.

I haven’t tested any of these, just a couple of ideas :wink: However, I do not know how this might interact with seasonality within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<grass-user@lists.osgeo.org>) escribió:

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps ordered
by id.

I would like to fit a proper time depending fitting curve for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or
some simple bash/python scripting?

thank you

Ivan

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

_______________________________________________
grass-user mailing list
[grass-user@lists.osgeo.org](mailto:grass-user@lists.osgeo.org)
[https://lists.osgeo.org/mailman/listinfo/grass-user](https://lists.osgeo.org/mailman/listinfo/grass-user)


-- 
Micha Silver
Ben Gurion Univ.
Sde Boker, Remote Sensing Lab
cell: +972-523-665918
8

That’s right, Ivan!
End of year mental death, sorry for the noise…

Vero

El vie, 29 dic 2023 a las 15:34, Ivan Marchesini (<ivan.marchesini@gmail.com>) escribió:

Hi veronica

I see what you did with creating the days time series. In that way you acknowledge irregular gaps, right?

yes this is the reason

However, why do you multiply by days strds? From my understanding, detrending by subtracting the results of a model obbeys this rule: value(t) = observed(t) - predicted(t). Then, this mystrds-(regression_offset+regression_slopedays) should be mystrds-(regression_offset+regression_slopemystrds).

hmm

may be I’m wrong but I used r.regression.series to assess a relationship between mystrds values and time (days)

As a consequence the offset and slope maps I obtain are b and a in the following linear equation

y=ax+b

i.e.

predicted_mystrds=slope*days+offset

This is why I suppose that for detrending I need to do:

mystrds-(predicted_mystrds)

i.e.

mystrds-(slope*days+offset)

I’m I wrong?

thank you

Ivan

Best,
Vero

El jue, 28 dic 2023 a las 8:14, Ivan Marchesini (<ivan.marchesini@gmail.com>) escribió:

Dear Veronica

I think I found a simple solution using temporal raster modules. Here is an example:

#evaluating info of the strds
eval [t.info](http://t.info) mystrds -g

#getting the starting day (of the year, 0-365) of my strds
startday=$(date -d “$start_time” “+%j”)

#Creating a new strds where each pixel has the value of the count of the days starting from the start_day of my strds (the start day in my dataset is in 2016)
t.rast.mapcalc inputs=mystrds expression=“(start_year()-2016)*365-${startday} +start_doy()” output=days basename=days nprocs=xxx --o

#fitting the trend equation
r.regression.series xseries=“t.rast.list in=days columns=name sep=, format=line” yseries=" t.rast.list in=mystrds columns=name sep=, format=line" out=regression_offset,regression_slope,regression_rsq,regression_t meth=offset,slope,rsq,t

#detrending
t.rast.mapcalc input=mystrds,days expression=“mystrds_detrend = mystrds-(regression_offset+regression_slope*days)” output=mystrds_detrend basename=mystrds_detrend nprocs=xxx method=start --o

Best

Ivan

On 23/12/23 14:53, Ivan Marchesini wrote:

Hi Veronica

Thank you. It goes in the direction of my idea evn if my problem is exactly trying to take into account the correct gaps between that data

I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan

On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within t.rast.algebra to detrend the values of the maps within the strds, something like “detrended_strds = trend_strds - (trend_strds*map(slope) + map(offset))”. Others suggest, to detrend by subtracting the previous value, i.e. that would imply using the temporal algebra with the temporal index, something like “detrended_strds = trend_strds[1] - trend_strds[0]”.

I haven’t tested any of these, just a couple of ideas :wink: However, I do not know how this might interact with seasonality within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<grass-user@lists.osgeo.org>) escribió:

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps ordered
by id.

I would like to fit a proper time depending fitting curve for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS modules or
some simple bash/python scripting?

thank you

Ivan

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user