[GRASS-dev] i.sentinel.import does not list downloaded data

neteler · August 14, 2018, 12:58pm

Hi,

I have downloaded some selected Sentinel-2 scenes using "sentinelsat" [1]:

sentinelsat -u myuser -p secret --uuid
98f00e12-3b62-4c54-8554-c4439bb1ebef --download
...

which downloaded as desired (since i.sentinel.download does not have a
uuid parameter I used sentinelsat):

ls -1 data/
S2A_MSIL2A_20180628T115401_N0208_R023_T29ULT_20180628T125228.zip
S2A_MSIL2A_20180628T115401_N0208_R023_T29ULU_20180628T125228.zip
S2A_MSIL2A_20180628T115401_N0208_R023_T29UMT_20180628T125228.zip
S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.zip

Now trying to import these scenes, I don't get anything with
i.sentinel.import [2]

# print
i.sentinel.import -p input=data
--> nothing, should list the channels?

i.sentinel.import -p input=data pattern="B0[4|8]_10m"
--> nothing, should list the selected channels?

Does anyone know why this fails? What am I missing?

thanks,
Markus

[1] http://sentinelsat.readthedocs.io/en/stable/cli.html
[2] https://grass.osgeo.org/grass7/manuals/addons/i.sentinel.import.html

martinl · August 15, 2018, 9:49am

Hi,

út 14. 8. 2018 v 14:58 odesílatel Markus Neteler <neteler@osgeo.org> napsal:

which downloaded as desired (since i.sentinel.download does not have a
uuid parameter I used sentinelsat):

now possible

$ i.sentinel.download settings=sentinel.txt
uuid=98f00e12-3b62-4c54-8554-c4439bb1ebef output=data

# print
i.sentinel.import -p input=data
--> nothing, should list the channels?

This seems to work for me.

$ i.sentinel.import -p input=/tmp/sentinel
/tmp/sentinel/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.SAFE/GRANULE/L2A_T29UMU_A015750_20180628T115400/IMG_DATA/R10m/T29UMU_20180628T115401_B03_10m.jp2
0 (EPSG: 32629)
/tmp/sentinel/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.SAFE/GRANULE/L2A_T29UMU_A015750_20180628T115400/IMG_DATA/R10m/T29UMU_20180628T115401_B08_10m.jp2
0 (EPSG: 32629)

Can you try on empty data directory with fresh addons?

Martin

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

neteler · August 15, 2018, 8:28pm

On Wed, Aug 15, 2018 at 11:50 AM Martin Landa <landa.martin@gmail.com> wrote:

Hi,

út 14. 8. 2018 v 14:58 odesílatel Markus Neteler <neteler@osgeo.org> napsal:
> which downloaded as desired (since i.sentinel.download does not have a
> uuid parameter I used sentinelsat):

now possible

Testing:

GRASS 7.5.svn (ireland):~ > i.sentinel.download -l
settings=~/.esa_sentinelhub uuid=98f00e12-3b62-4c54-8554-c4439bb1ebef
98f00e12-3b62-4c54-8554-c4439bb1ebef 2018-06-28T11:54:01Z 0% S2MSI2A

Works - great, thank you!!

$ i.sentinel.download settings=sentinel.txt
uuid=98f00e12-3b62-4c54-8554-c4439bb1ebef output=data

> # print
> i.sentinel.import -p input=data
> --> nothing, should list the channels?

This seems to work for me.

$ i.sentinel.import -p input=/tmp/sentinel
/tmp/sentinel/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.SAFE/GRANULE/L2A_T29UMU_A015750_20180628T115400/IMG_DATA/R10m/T29UMU_20180628T115401_B03_10m.jp2
0 (EPSG: 32629)
/tmp/sentinel/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.SAFE/GRANULE/L2A_T29UMU_A015750_20180628T115400/IMG_DATA/R10m/T29UMU_20180628T115401_B08_10m.jp2
0 (EPSG: 32629)

Can you try on empty data directory with fresh addons?

# relative path
GRASS 7.5.svn (ireland):/scratch/ireland > i.sentinel.import -p input=data

# absolute path
GRASS 7.5.svn (ireland):/scratch/ireland > i.sentinel.import -p
input=/scratch/ireland/data

## ... nothing reported while the data are there:
GRASS 7.5.svn (ireland):/scratch/ireland > ls -lart data/
total 3267768
-rw-rw-r-- 1 mundialis mundialis 784858224 Aug 14 12:32
S2A_MSIL2A_20180628T115401_N0208_R023_T29ULT_20180628T125228.zip
-rw-rw-r-- 1 mundialis mundialis 1037024033 Aug 14 12:35
S2A_MSIL2A_20180628T115401_N0208_R023_T29UMT_20180628T125228.zip
-rw-rw-r-- 1 mundialis mundialis 955002429 Aug 14 12:51
S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.zip
-rw-rw-r-- 1 mundialis mundialis 569305492 Aug 14 13:33
S2A_MSIL2A_20180628T115401_N0208_R023_T29ULU_20180628T125228.zip

# test with empty one:
GRASS 7.5.svn (ireland):/scratch/ireland > mkdir data2
GRASS 7.5.svn (ireland):/scratch/ireland > i.sentinel.import -p
input=/scratch/ireland/data2/
GRASS 7.5.svn (ireland):/scratch/ireland >

Unfortunately nothing yet...

Markus

martinl · August 15, 2018, 9:30pm

Hi,

st 15. 8. 2018 v 22:28 odesílatel Markus Neteler <neteler@osgeo.org> napsal:

## ... nothing reported while the data are there:
GRASS 7.5.svn (ireland):/scratch/ireland > ls -lart data/
total 3267768
-rw-rw-r-- 1 mundialis mundialis 784858224 Aug 14 12:32
S2A_MSIL2A_20180628T115401_N0208_R023_T29ULT_20180628T125228.zip

right, that's because you downloaded Sentinel products by sentinelsat
directly and not by i.sentinel.download.

i.sentinel.download unzips downloaded archives, and i.sentinel.import
expects uncompressed data. I wanted to change this behaviour some time
ago. i.sentinel.import should read zipped archives directly. Will
change it in the next days.

Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

martinl · August 16, 2018, 6:47am

Hi,

st 15. 8. 2018 v 23:30 odesílatel Martin Landa <landa.martin@gmail.com> napsal:

right, that's because you downloaded Sentinel products by sentinelsat
directly and not by i.sentinel.download.

i.sentinel.download unzips downloaded archives, and i.sentinel.import
expects uncompressed data. I wanted to change this behaviour some time
ago. i.sentinel.import should read zipped archives directly. Will
change it in the next days.

done in r73107. Testing welcome

Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

neteler · August 16, 2018, 3:53pm

Hi Martin,

On Thu, Aug 16, 2018 at 8:47 AM Martin Landa <landa.martin@gmail.com> wrote:

Hi,

st 15. 8. 2018 v 23:30 odesílatel Martin Landa <landa.martin@gmail.com> napsal:
> right, that's because you downloaded Sentinel products by sentinelsat
> directly and not by i.sentinel.download.
>
> i.sentinel.download unzips downloaded archives, and i.sentinel.import
> expects uncompressed data. I wanted to change this behaviour some time
> ago. i.sentinel.import should read zipped archives directly. Will
> change it in the next days.

done in r73107. Testing welcome

Truely great!

Works also with filtering in multiple (zip'ed) Sentinel-2 scenes!

i.sentinel.import -p input=data pattern="B0[4|8]_10m"
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29ULT_20180628T125228.SAFE/GRANULE/L2A_T29ULT_A015750_20180628T115400/IMG_DATA/R10m/T29ULT_20180628T115401_B04_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29ULT_20180628T125228.SAFE/GRANULE/L2A_T29ULT_A015750_20180628T115400/IMG_DATA/R10m/T29ULT_20180628T115401_B08_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29ULU_20180628T125228.SAFE/GRANULE/L2A_T29ULU_A015750_20180628T115400/IMG_DATA/R10m/T29ULU_20180628T115401_B04_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29ULU_20180628T125228.SAFE/GRANULE/L2A_T29ULU_A015750_20180628T115400/IMG_DATA/R10m/T29ULU_20180628T115401_B08_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMT_20180628T125228.SAFE/GRANULE/L2A_T29UMT_A015750_20180628T115400/IMG_DATA/R10m/T29UMT_20180628T115401_B04_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMT_20180628T125228.SAFE/GRANULE/L2A_T29UMT_A015750_20180628T115400/IMG_DATA/R10m/T29UMT_20180628T115401_B08_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.SAFE/GRANULE/L2A_T29UMU_A015750_20180628T115400/IMG_DATA/R10m/T29UMU_20180628T115401_B08_10m.jp2
0 (EPSG: 32629)
data/S2A_MSIL2A_20180628T115401_N0208_R023_T29UMU_20180628T125228.SAFE/GRANULE/L2A_T29UMU_A015750_20180628T115400/IMG_DATA/R10m/T29UMU_20180628T115401_B04_10m.jp2
0 (EPSG: 32629)

Thanks a lot, looks now like the perfect tool!

Markus

PS: geobeers in Prague then!

neteler · August 21, 2018, 10:29am

Hi Martin,

found a small glitch when a user enters a non-sensical UUID (low priority):

GRASS 7.5.svn (utm32N):~ > i.sentinel.download
settings=~/.esa_sentinelhub
uuid=b8d040e9-9466-453c-8792-5a451f0943a0_1_8A1104_20170314T104411 -l
Traceback (most recent call last):
  File "/home/mundialis/.grass7/addons/scripts/i.sentinel.download",
line 386, in <module>
    sys.exit(main())
  File "/home/mundialis/.grass7/addons/scripts/i.sentinel.download",
line 358, in main
    downloader.set_uuid(options['uuid'].split(','))
  File "/home/mundialis/.grass7/addons/scripts/i.sentinel.download",
line 303, in set_uuid
    for k, v in self._api.get_product_odata(uuid, full=True).items():
  File "/usr/lib/python2.7/site-packages/sentinelsat/sentinel.py",
line 383, in get_product_odata
    _check_scihub_response(response)
  File "/usr/lib/python2.7/site-packages/sentinelsat/sentinel.py",
line 892, in _check_scihub_response
    raise api_error
sentinelsat.sentinel.SentinelAPIError: HTTP status 500 Internal Server
Error: InvalidKeyException : Invalid key
(b8d040e9-9466-453c-8792-5a451f0943a0_1_8A1104_20170314T104411) to
access Products

Could you please catch this error message?

Thanks
Markus

martinl · August 21, 2018, 3:57pm

Hi Markus,

út 21. 8. 2018 v 12:29 odesílatel Markus Neteler <neteler@osgeo.org> napsal:

sentinelsat.sentinel.SentinelAPIError: HTTP status 500 Internal Server
Error: InvalidKeyException : Invalid key
(b8d040e9-9466-453c-8792-5a451f0943a0_1_8A1104_20170314T104411) to
access Products

Could you please catch this error message?

fixed in r73129. Martin

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa