GSoC 2024 - Final Report | Add EODAG Support - GRASS GIS

Initiatives Google Summer of Code
1

Final Report

Title: Add EODAG Support to GRASS GIS
Community: GRASS GIS - OSGeo
GrassWiki: GRASS GSoC 2024 EODAG Support - GRASS-Wiki

Abstract

This project introduces a new extension for GRASS GIS, i.eodag, which integrates the EODAG API with GRASS GIS, serving as a central portal for downloading Earth Observation data and products. Before this project, downloading such data required three separate modules: i.landsat.download, i.sentinel.download, and i.modis.download. Each module relied on different APIs, leading to significant maintenance efforts. The i.eodag extension simplifies this process by providing a unified source for all geospatial products, reducing the maintenance burden to a single API—the EODAG API [1]. Additionally, i.landsat.download and i.sentinel.download have been reimplemented to internally use i.eodag, with their logic abstracted to improve modularity and maintainability.

The state of integration BEFORE the start of GSoC

Before this project, all of MODIS, Landsat and Sentinel download tools relied on different APIs, necessitating significant maintenance to accommodate for updates from both the APIs themselves and their respective providers. Specifically:

  • i.landsat.download [2]:
    • Utilized the landsatxplore [3] Python library to download Landsat data from USGS. This library is no longer maintained by the original author.
  • i.sentinel.download [4] utilized three distinct data sources:
    • sentinelsat [5] for downloading Sentinel data from the Copernicus Open Access Hub [6]. However, after the data migration to the Copernicus Data Space Ecosystem [7], Sentinel products were no longer available through this source, rendering portions of the previous implementation non-functional.
    • landsatxplore [8] to access Sentinel data from USGS [9].
    • Manual data requests via theGoogle Cloud Storage API [10].

The state of integration AFTER GSoC

  1. I implemented a new module, i.eodag [12], which serves as a unified portal for downloading a variety of Earth Observation data products, extending beyond just Sentinel and Landsat. This integration allows GRASS GIS to rely on a single API, EODAG, instead of managing multiple APIs like landsatxplore [8], sentinelsat [5], and Google Cloud Storage [10]. Some of the features implemented in i.eodag:
    1. Supports generic searching without specifying a provider.
    2. Supports limiting searching to a single provider.
    3. Supports simplified search result listing.
    4. Supports extended search result listing, printing scene metadata in JSON.
    5. Downloads scenes to a specified directory.
    6. Enables searching for scenes by their IDs.
    7. Saves search results in a GeoJSON file, including all scenes’ metadata.
    8. Reads previously generated GeoJSON files for further filtering.
    9. Downloads scenes saved in a GeoJSON file.
    10. Reads a text file with a list of IDs and searches for them.
    11. Supports TOTP Authentication for Creodias [13] provider.
    12. Offers the possibility to use either a pre-existing Vector Map or the current computational region as the Area of Interest.
    13. Includes an option to skip fully downloaded scenes by using the MD5 Hash.
    14. Provides an option to filter scenes using RegEx.
    15. Provides an option to filter scenes using their metadata (e.g., orbitDirection), with filters varying depending on product type and provider.
    16. Supports querying/filtering with both OR and AND relations, as well as relational operators (e.g., ‘lt’, ‘gt’, etc.).
    17. Supports printing of relevant EODAG metadata e.g. available providers, available products and their corresponding queryables (aka. products’ metadata).
    18. For the full list of features and options, as well as usage examples, please refer to the i.eodag manual. [12]
  2. I added a comprehensive test suite for i.eodag, featuring resilient and robust test cases. These tests are designed to be flexible, accommodating different EODAG versions and future developments.
  3. I reimplemented i.landsat.download using i.eodag, while maintaining backward compatibility. The reimplementation abstracted the logic, making the tool more modular and extensible for future updates.
  4. I reimplemented i.sentinel.download using i.eodag, reducing the need for maintaining multiple APIs. This reimplementation simplified and abstracted the logic effectively.

Conclusion

To wrap it up, I’m glad to have successfully achieved the three main targets of the project, with the invaluable support from my mentors!

  • Integrated EODAG API to GRASS GIS through a new tool i.eodag, serving as a unified portal for downloading a variety of Earth Observation data products.
  • Reimplemented i.landsat.download using the new i.eodag tool.
  • Reimplemented i.sentinel.download using the new i.eodag tool.

I am incredibly grateful for the support I received this summer and for the opportunity to contribute to GRASS GIS through GSoC 2024. This experience has been rewarding, and I am excited to continue enhancing i.eodag. While we’ve made significant progress, there are still exciting opportunities for further improvement, as outlined in the Future Work section below.

Future Work

  • Investigate the possibility of the integration of the i.eodag tool into i.modis.download too. This could be achieved by either adding i.eodag as an option or reimplementing i.modis.download to rely on i.eodag as the primary source for MODIS data.

  • Investigate how EODAG product types can be matched to GRASS GIS image band libraries. [11]

  • Consider adding an option for adding labels for downloaded scenes with the r.support tool.

  • There is room for improvement in data querying and filtering, particularly for i.sentinel.download, which supports over 20 products, each with its own set of auxiliary metadata. A useful enhancement would be to provide users with an option to retrieve a list of available filters for each product type, enabling more refined searches. Although the filtering method is already implemented and functional, adding a method to access the metadata (queryables / filters) list for each product type would be beneficial. Since i.eodag currently offers metadata lists for each product type, this feature can be implemented in i.sentinel.download by leveraging the i.eodag tool.

  • The process of listing queryable options in i.eodag could be enhanced by extracting them directly from a data product instance rather than relying solely on the EODAG API, which may sometimes be outdated.

Links

Pull Requests Log

Pull Request Description Date Status
#1090 i.eodag: add i.eodag module May 27, 2024 Merged
#1097 i.eodag: add files support June 9, 2024 Merged
#1104 i.eodag: remove pandas dependency and save results in json/geojson files June 16, 2024 Merged
#1136 i.eodag: add query, print, and footprints options July 2, 2024 Merged
#1142 i.eodag: add operators to queryables and allow multiple values July 11, 2024 Merged
#1144 i.landsat.download: change API from landsatxplore to EODAG using i.eodag July 17, 2024 Merged
#1145 i.eodag: add timeout & wait options July 18, 2024 Merged
#1150 i.sentinel.download: use EODAG as a unified API for downloading sentinel products July 31, 2024 Merged
#1152 i.eodag: additional options and modifications for i.sentinel.download August 1, 2024 Merged
#1156 i.landsat.download: put back settings option for credentials August 4, 2024 Merged
#1159 i.eodag: add dynamic setting for Creodias OTP August 6, 2024 Merged
#1163 i.eodag: add testsuite August 8, 2024 Merged
#1165 i.eodag: update i.eodag to be compatible with EODAG 3 August 8, 2024 Merged
#1167 i.eodag: add -p flag to print query summary August 9, 2024 Merged
#1170 i.eodag: handle cases when sorting parameters are not available August 14, 2024 Merged

Documentation & GitHub Fork

Ttile Link
OSGeo Wiki Page Add EODAG Support to GRASS GIS
Github Fork GitHub Fork
EODAG API EODAG
i.eodag i.eodag manual
i.sentinel.download i.sentinel.download manual
i.landsat.download i.landsat.download manual

[1] Earth Observation Data Access Gateway (2.12.1)
[2] i.landsat.download - GRASS GIS manual
[3] GitHub - yannforget/landsatxplore: Search and download Landsat scenes from EarthExplorer.
[4] i.sentinel.download - GRASS GIS manual
[5] GitHub - sentinelsat/sentinelsat: Search and download Copernicus Sentinel satellite images
[6] https://scihub.copernicus.eu/
[7] https://dataspace.copernicus.eu/
[8] GitHub - yannforget/landsatxplore: Search and download Landsat scenes from EarthExplorer.
[9] https://earthexplorer.usgs.gov/
[10] https://cloud.google.com/solutions/geospatial?hl=en
[11] Investigate how eodag product_types can be matched to GRASS GIS image band libraries · Issue #7 · HamedElgizery/grass-addons · GitHub
[12] i.eodag - GRASS GIS manual
[13] https://creodias.eu/

1 Like