[GRASS-user] New AddOns for downloading and importing Sentinel products

Hi,

today I have uploaded a new toolset for downloading and importing
Sentinel products, namely two modules:

r.sentinel.download [1] and r.sentinel.import [2].

The first module requires Sentinelsat and Pandas library, the second
GDAL with OpenJPEG.

Testing and feedback welcome! Enjoy, Ma

[1] https://grass.osgeo.org/grass72/manuals/addons/r.sentinel.download.html
[2] https://grass.osgeo.org/grass72/manuals/addons/r.sentinel.import.html

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

On Fri, Jan 26, 2018 at 11:36 PM, Martin Landa <landa.martin@gmail.com> wrote:

Hi,

today I have uploaded a new toolset for downloading and importing
Sentinel products, namely two modules:

r.sentinel.download [1] and r.sentinel.import [2].

The first module requires Sentinelsat and Pandas library, the second
GDAL with OpenJPEG.

Testing and feedback welcome! Enjoy, Ma

[1] https://grass.osgeo.org/grass72/manuals/addons/r.sentinel.download.html
[2] https://grass.osgeo.org/grass72/manuals/addons/r.sentinel.import.html

Cool!
I was able to successfully select and download S2 scenes for the North
Carolina area.

Do you plan to add reprojection on the fly through r.import?

thanks, Markus

BTW: Last year we created a portal for ESA which let's you filter
Sentinel-2 scenes by extra metadata derived from open data:

  Earth Observation metadata enhancer (EO-me)
  https://sentinel.esa.int/web/sentinel/-/filter-sentinel-2a-scenes-with-the-new-release-of-eo-me
  https://eome.mundialis.de

It meanwhile also supports Sentinel-2B and comes with an OpenSearch interface:
https://eome.mundialis.de/eomedocs/opensearch.html

Hi Markus,

2018-01-28 20:45 GMT+01:00 Markus Neteler <neteler@osgeo.org>:

Do you plan to add reprojection on the fly through r.import?

r.sentinel.import -o

BTW: Last year we created a portal for ESA which let's you filter
Sentinel-2 scenes by extra metadata derived from open data:

  Earth Observation metadata enhancer (EO-me)
  https://sentinel.esa.int/web/sentinel/-/filter-sentinel-2a-scenes-with-the-new-release-of-eo-me
  https://eome.mundialis.de

It meanwhile also supports Sentinel-2B and comes with an OpenSearch interface:
https://eome.mundialis.de/eomedocs/opensearch.html

Interesting, Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Hi Martin,

On Sun, Jan 28, 2018 at 9:01 PM, Martin Landa <landa.martin@gmail.com> wrote:

Hi Markus,

2018-01-28 20:45 GMT+01:00 Markus Neteler <neteler@osgeo.org>:

Do you plan to add reprojection on the fly through r.import?

r.sentinel.import -o

I see. Perhaps I got confused because without -o it tells me:

...
       In case of no significant differences in the projection definitions,
       use the -o flag to ignore them and use current location definition.
       Consider generating a new location from the input dataset using the
       'location' parameter.

So it would be good to suppress this message which likely comes from a
r.in.gdal call and replace it with one related to
  -o Reproject raster data using r.import if needed

Would that be possible?

thanks
Markus

Hi Martin,

the reprojections is done when using -o (as advertised in the flag description).

A related idea, if you don't mind:

GRASS 7.4.1svn (nc_spm_08_grass7):~ > r.sentinel.import
input=/home/mneteler/tmp/ pattern='B0[4|8]' -o
...
Estimated target resolution for input band <T17SQV_20180124T160541_B04>:
9.79587588067

It would probably be best to fetch the nominal resolution from the S2
band description and apply that rather than estimating it (at least,
most users will expect to get 10m in this example). Or, optionally
make it a flag like e.g. "-k keep nominal band resolution when in
metric CRS" or similar which could then also imply bilinear resampling
rather than the standard nearest neighbor resampling.

Just an idea,
Markus

Hi Martin,

Very nice work! This will simplify some of my workflows significantly!
One parameter I was missing was however "area_relation". Please find attached a diff, in case you agree that it would be a useful addition (the python script is successfully tested, though I did not check the html changes).

Two things I was wondering are:
1. it might make sense to also provide the "url" as an option, cause there are some national mirrors of ESAs scihub around (at least this is the case for Norway) that expose the same API but are faster / cause have less traffic. What do you think?
2. the tools currently only support Sentinel-2, right. Do you plan to extend them to Sentinel-1/3... in future?

Cheers
Stefan

-----Original Message-----
From: grass-user [mailto:grass-user-bounces@lists.osgeo.org] On Behalf Of Martin Landa
Sent: fredag 26. januar 2018 23.36
To: GRASS users list <grass-user@lists.osgeo.org>
Subject: [GRASS-user] New AddOns for downloading and importing Sentinel products

Hi,

today I have uploaded a new toolset for downloading and importing Sentinel products, namely two modules:

r.sentinel.download [1] and r.sentinel.import [2].

The first module requires Sentinelsat and Pandas library, the second GDAL with OpenJPEG.

Testing and feedback welcome! Enjoy, Ma

[1] https://grass.osgeo.org/grass72/manuals/addons/r.sentinel.download.html
[2] https://grass.osgeo.org/grass72/manuals/addons/r.sentinel.import.html

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

(attachments)

r.sentinel.diff (2.99 KB)

Hi Markus,

2018-01-28 21:15 GMT+01:00 Markus Neteler <neteler@osgeo.org>:

So it would be good to suppress this message which likely comes from a
r.in.gdal call and replace it with one related to
  -o Reproject raster data using r.import if needed

done, projection check is done by `r.in.gdal -j`. I have also rename
-o flag to -r. Flag -o is used for overriding projection check which
is this case. Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Hi,

2018-01-28 21:36 GMT+01:00 Markus Neteler <neteler@osgeo.org>:

It would probably be best to fetch the nominal resolution from the S2
band description and apply that rather than estimating it (at least,
most users will expect to get 10m in this example). Or, optionally

implemented in r72198 by reading input file using GDAL.

make it a flag like e.g. "-k keep nominal band resolution when in
metric CRS" or similar which could then also imply bilinear resampling
rather than the standard nearest neighbor resampling.

I would prefer avoid adding new flags if not necessary :slight_smile: Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Hi Stefan,

2018-01-29 9:11 GMT+01:00 Stefan Blumentrath <Stefan.Blumentrath@nina.no>:

Very nice work! This will simplify some of my workflows significantly!

nice to hear.

One parameter I was missing was however "area_relation". Please find attached a diff, in case you agree that it would be a useful addition (the python script is successfully tested, though I did not check the html changes).

Patch applied (r72199)

Two things I was wondering are:
1. it might make sense to also provide the "url" as an option, cause there are some national mirrors of ESAs scihub around (at least this is the case for Norway) that expose the same API but are faster / cause have less traffic. What do you think?

Would make sense.

2. the tools currently only support Sentinel-2, right. Do you plan to extend them to Sentinel-1/3... in future?

In future :slight_smile: Ma

PS: Patches welcome :slight_smile:

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

On 26/01/18 23:36, Martin Landa wrote:

Hi,

today I have uploaded a new toolset for downloading and importing
Sentinel products, namely two modules:

r.sentinel.download [1] and r.sentinel.import [2].

The first module requires Sentinelsat and Pandas library, the second
GDAL with OpenJPEG.

Testing and feedback welcome!

Shouldn't these be better called i.sentinel.* ? They are clearly imagery related.

Moritz

Hi,

2018-02-16 16:17 GMT+01:00 Moritz Lennert <mlennert@club.worldonline.be>:

Shouldn't these be better called i.sentinel.* ? They are clearly imagery
related.

you are probably right. Than we should also rename r.modis and
probably also other modules. What do you think? Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

On 16/02/18 19:12, Martin Landa wrote:

Hi,

2018-02-16 16:17 GMT+01:00 Moritz Lennert <mlennert@club.worldonline.be>:

Shouldn't these be better called i.sentinel.* ? They are clearly imagery
related.

you are probably right. Than we should also rename r.modis and
probably also other modules. What do you think? Ma

+1

Concerning r.sentinel.download: A colleague has been trying to use it under windows, but:

- installing GRASS using osgeo4w gives you sentinelsat, but pandas doesn't seem to be installed in the correct version as with

import pandas

you get an error message about a missing natype module (something like that, am not at the windows computer, now).

- installing GRASS using the standalone installer, my colleague has difficulty installing sentinelsat as 'pip install sentinelsat' fails

Has anyone else tried this on Windows ?

- installing GRASS using osgeo4w gives you sentinelsat, but pandas
doesn't seem to be installed in the correct version as with

import pandas

you get an error message about a missing natype module (something like
that, am not at the windows computer, now).

quick test here on my OSGeo4W 64bit:

run o-help for a list of available commands
C:\>python
Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

import pandas

no problem here to import pandas

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Users-f3884509.html

On 19/02/18 14:12, Moritz Lennert wrote:

On 16/02/18 19:12, Martin Landa wrote:

Hi,

2018-02-16 16:17 GMT+01:00 Moritz Lennert <mlennert@club.worldonline.be>:

Shouldn't these be better called i.sentinel.* ? They are clearly imagery
related.

you are probably right. Than we should also rename r.modis and
probably also other modules. What do you think? Ma

+1

Concerning r.sentinel.download: A colleague has been trying to use it
under windows, but:

- installing GRASS using osgeo4w gives you sentinelsat, but pandas
doesn't seem to be installed in the correct version as with

import pandas

you get an error message about a missing natype module (something like
that, am not at the windows computer, now).

- installing GRASS using the standalone installer, my colleague has
difficulty installing sentinelsat as 'pip install sentinelsat' fails

Has anyone else tried this on Windows ?

P.S.

I consider it unideal practice to write passwords in clear on the command line. I don't know if it would be very complicated to work with a sentinel scihub password file...

Moritz

Has anyone else tried this on Windows ?

tested here with

---------
GRASS version: 7.4.0
GRASS SVN revision: r72154
Build date: 2018-01-27
Build platform: x86_64-w64-mingw32
GDAL: 2.2.3
PROJ.4: 4.9.3
GEOS: 3.5.0
SQLite: 3.17.0
Python: 2.7.5
wxPython: 2.8.12.1
Platform: Windows-8-6.2.9200 (OSGeo4W)
---------

r.sentinel.download -l user=myusesr password=mypwd
1a51afeb-9d18-410a-aec3-2366cfe22d81 2018-01-24T10:13:09Z 10% S2MSI1C
e6cda378-d2e4-4c9c-809f-524b3a354ec2 2018-01-29T10:12:51Z 12% S2MSI1C
84b5b211-69a8-4f96-beae-02ff05e87d36 2017-12-22T10:04:19Z 19% S2MSI1C
6fc7a685-971c-49ef-ad86-b993ef644b2a 2018-02-10T10:01:39Z 19% S2MSI1C
e64633ea-d302-49c7-a7ed-fdbf143b3b2f 2017-12-25T10:14:19Z 24% S2MSI1C
5 Sentinel product(s) found

no time to test the download itself at the moment

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Users-f3884509.html

Has anyone else tried this on Windows ?

ok, tested also the dowload itself:

r.sentinel.download user=myuser password=mypwd output=D:\dl\sentinel
map=myriver@data start=2018-01-20 end=2018-01-27 limit=1
1 Sentinel product(s) found
Downloading data into <D:\dl\sentinel>...
1a51afeb-9d18-410a-aec3-2366cfe22d81 ->
D:\dl\sentinel\S2B_MSIL1C_20180124T101309_N0206_R022_T33TUM_20180124T190119.SAFE
Downloading: 0%| | 0.00/885M [00:00<?, ?B/s]
Downloading: 0%| | 1.05M/885M [00:02<30:27,
484kB/s]
Downloading: 0%| | 2.10M/885M [00:04<29:58,
491kB/s]
Downloading: 0%| | 3.15M/885M [00:08<37:55,
388kB/s]
Downloading: 0%| | 4.19M/885M [00:11<38:13,
384kB/s]
Downloading: 1%| | 5.24M/885M [00:13<36:07,
406kB/s]
Downloading: 1%| | 6.29M/885M [00:15<33:18,
440kB/s]
Downloading: 1%| | 7.34M/885M [00:18<37:31,
390kB/s]
Downloading: 1%| | 8.39M/885M [00:21<37:45,
387kB/s]

seems also to work so far.

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Users-f3884509.html

Hi,

2018-02-19 14:22 GMT+01:00 Moritz Lennert <mlennert@club.worldonline.be>:

I consider it unideal practice to write passwords in clear on the command
line. I don't know if it would be very complicated to work with a sentinel
scihub password file...

what is a structure of such file? Thanks for pointers, Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Le Mon, 19 Feb 2018 20:13:51 +0100,
Martin Landa <landa.martin@gmail.com> a écrit :

Hi,

2018-02-19 14:22 GMT+01:00 Moritz Lennert
<mlennert@club.worldonline.be>:
> I consider it unideal practice to write passwords in clear on the
> command line. I don't know if it would be very complicated to work
> with a sentinel scihub password file...

what is a structure of such file? Thanks for pointers, Ma

Following the example of .pgpass for PostgreSQL, it could be a simple
text file with:

login:password

and the module would only use it if it is chmod 600, i.e. only readable
by the owner.

Moritz

On Mon, Feb 19, 2018 at 4:49 PM, Moritz Lennert <
mlennert@club.worldonline.be> wrote:

and the module would only use it if it is chmod 600, i.e. only readable
by the owner.

...and the code to test that (user behaves safely) should be somewhere in
g.cloud and/or g.remote:

https://trac.osgeo.org/grass/browser/grass-addons/grass7/general/g.cloud
https://github.com/wenzeslaus/g.remote

On 20/02/18 01:24, Vaclav Petras wrote:

On Mon, Feb 19, 2018 at 4:49 PM, Moritz Lennert <mlennert@club.worldonline.be <mailto:mlennert@club.worldonline.be>> wrote:

    and the module would only use it if it is chmod 600, i.e. only readable
    by the owner.

...and the code to test that (user behaves safely) should be somewhere in g.cloud and/or g.remote:

https://trac.osgeo.org/grass/browser/grass-addons/grass7/general/g.cloud
https://github.com/wenzeslaus/g.remote

I suppose you mean should _also_ be ?

Moritz