[GRASS-dev] Landsat 8 Collection-1 QA bits

Stefan,

in processing Collection-1 Landsat 8 products, `i.landsat8.qc` deserves
an update [0][1]. Please note that Pre-Collection products are removed
from EarthExplorer [2][3].

See:

[0] https://landsat.usgs.gov/sites/default/files/documents/landsat_QA_tools_userguide.pdf
[1] https://landsat.usgs.gov/collectionqualityband
[2] https://landsat.usgs.gov/september-1-2017-countdown-landsat-pre-collection-datasets-removal
[3] https://landsat.usgs.gov/oct-3-2017-pre-collection-dataset-removal-prompts-ee-changes

Below, bits and descriptions transferred in python dictionaries,
blindly following the script's current structure (kept "old" and "new"
for comparison). I would like your view on this, or anyone else's who is
interested.

What do you think of:

- introducing a `collection=` option in the module and make it treat
  both, Pre-Collection and Collection-1 products?

- removing Pre-collection QA bits altogether?

Nikos
--

Hier Pre-Collection und Collection-1 zusammen:

    """
    Pre-Collection populated bits:

    0 Designated Fill
    1 Dropped Frame
    2 Terrain Occlusion
    3 Reserved (not currently used)
    4-5 Water Confidence
    6-7 Reserved for cloud shaddow
    8-9 Vegetation confidence
    10-11 Snow/Ice confidence
    12-13 Cirrus confidence
    14-15 Cloud confidence
    """

    # Define bit length (single or double bits) -- Pre-Collection
    bl = {'designated_fill': 1,
          'dropped_frame': 1,
          'terrain_occlusion': 1,
          # 'reserved': 1,
          'water': 2,
          'cloud_shadow': 2,
          'vegetation': 2,
          'snow_ice': 2,
          'cirrus': 2,
          'cloud': 2}

    # Define bit position start -- Pre-Collection
    bps = {
        'designated_fill': 0,
        'dropped_frame': 1,
        'terrain_occlusion': 2,
        # 'reserved': 3,
        'water': 4,
        'cloud_shadow': 6,
        'vegetation': 8,
        'snow_ice': 10,
        'cirrus': 12,
        'cloud': 14}

    """
    Collection-1 populated bits

    0 Designated Fill
    1 Terrain Occlusion
    2-3 Radiometric Saturation
    4 Cloud
    5-6 Cloud Confidence
    7-8 Cloud Shadow Confidence
    9-10 Snow/Ice confidence
    11-12 Cirrus confidence
    13
    14
    15
    """

    # Define bit length (single or double bits) -- Collection-1
    bit_length = {'designated_fill': 1,
          'terrain_occlusion': 1,
          'radiometric_saturation': 2,
          'cloud': 1,
          'cloud_confidence': 2,
          'cloud_shadow_confidence': 2,
          'snow_ice_confidence': 2,
          'cirrus_confidence': 2,
          # 'reserved': 1,
          # 'reserved': 1}

    # Define bit position start -- Collection-1
    bit_position_start = {
        'designated_fill': 0,
        'terrain_occlusion': 1,
        'radiometric_saturation': 2,
        'cloud': 4,
        'cloud_confidence': 5,
        'cloud_shadow_confidence': 7,
        'snow_ice_confidence': 9,
        'cirrus_confidence': 11,
        # 'reserved': 13,
        # 'reserved': 14,
        # 'reserved': 15}

    """
    Pre-Collection

    For the single bits (0, 1, 2, and 3):
        0 = No, this condition does not exist
        1 = Yes, this condition exists.
    """

    """
    Collection-1

    For the single bits (0, 1, and 4):
        0 = "No" = This condition does not exist
        1 = "Yes" = This condition exists
    """

    # Define single bits dictionary -- Identical for Pre- and C1
    single_bits = {'No': '0',
                   'Yes': '1'}

    """
    Collection-1

    For radiometric saturation bits (2-3), read from left to right, represent how many bands contain saturation:

    00 - No bands contain saturation
    01 - 1-2 bands contain saturation
    10 - 3-4 bands contain saturation
    11 - 5 or more bands contain saturation
    """

    radiometric_saturation = {'No bands contain saturation': '00',
                              '1-2 bands contain saturation': '01',
                              '3-4 bands contain saturation': '10',
                              '5 or more bands contain saturation': '11'}

    """
    Pre-Collection

    The double bits (4-5, 6-7, 8-9, 10-11, 12-13, and 14-15), read from left to
    right, represent levels of confidence that a condition exists:
    00 = 'Not Determined' = Algorithm did not determine the status
                            of this condition
    01 = 'No' = Algorithm has low to no confidence that this condition exists
                (0-33 percent confidence)
    10 = 'Maybe' = Algorithm has medium confidence that this condition exists
                (34-66 percent confidence)
    11 = 'Yes' = Algorithm has high confidence that this condition exists
                (67-100 percent confidence).
    """

    # Define double bits dictionary -- Pre-Collection
    double_bits = {'Not Determined': '00',
                   'No': '01',
                   'Maybe': '10',
                   'Yes': '11'}

    """
    Collection-1

    For the remaining double bits (5-6, 7-8, 9-10, 11-12), read from left to
    right, represent levels of confidence that a condition exists:

    00 = "Not Determined" = Algorithm did not determine the status of this
    condition / "No" = This condition does not exist

    01 = "Low" = Algorithm has low to no confidence that this condition exists
    (0-33 percent confidence)

    10 = "Medium" = Algorithm has medium confidence that this condition exists
    (34-66 percent confidence)

    11 = "High" = Algorithm has high confidence that this condition exists
    (67-100 percent confidence
    """

    # Define double bits dictionary -- Colletion-1
    double_bits_c1 = {'Not Determined': '00',
                   'Low': '01',
                   'Medium': '10',
                   'High': '11'}

Hi Nikos,

Thanks for the heads-up and updates on this matter.

Since GUI cannot be changed dynamically based on selected options, I guess removing Pre-collection QA bits altogether is the most sensible solution? Alternatively, the module could be split into two: one for Pre-collection and one collection-1 version...

If Pre-Collection products are removed, it might be more convenient to change the rules to the new collection products?

Feel free to implement changes you prefer...

Cheers
Stefan

-----Original Message-----
From: Nikos Alexandris <nik@nikosalexandris.net>
Sent: fredag 1. juni 2018 19:10
To: Stefan Blumentrath <Stefan.Blumentrath@nina.no>
Cc: GRASS-GIS development mailing list <grass-dev@lists.osgeo.org>
Subject: Landsat 8 Collection-1 QA bits

Stefan,

in processing Collection-1 Landsat 8 products, `i.landsat8.qc` deserves an update [0][1]. Please note that Pre-Collection products are removed from EarthExplorer [2][3].

See:

[0] https://landsat.usgs.gov/sites/default/files/documents/landsat_QA_tools_userguide.pdf
[1] https://landsat.usgs.gov/collectionqualityband
[2] https://landsat.usgs.gov/september-1-2017-countdown-landsat-pre-collection-datasets-removal
[3] https://landsat.usgs.gov/oct-3-2017-pre-collection-dataset-removal-prompts-ee-changes

Below, bits and descriptions transferred in python dictionaries, blindly following the script's current structure (kept "old" and "new"
for comparison). I would like your view on this, or anyone else's who is interested.

What do you think of:

- introducing a `collection=` option in the module and make it treat
  both, Pre-Collection and Collection-1 products?

- removing Pre-collection QA bits altogether?

Nikos
--

Hier Pre-Collection und Collection-1 zusammen:

    """
    Pre-Collection populated bits:

    0 Designated Fill
    1 Dropped Frame
    2 Terrain Occlusion
    3 Reserved (not currently used)
    4-5 Water Confidence
    6-7 Reserved for cloud shaddow
    8-9 Vegetation confidence
    10-11 Snow/Ice confidence
    12-13 Cirrus confidence
    14-15 Cloud confidence
    """

    # Define bit length (single or double bits) -- Pre-Collection
    bl = {'designated_fill': 1,
          'dropped_frame': 1,
          'terrain_occlusion': 1,
          # 'reserved': 1,
          'water': 2,
          'cloud_shadow': 2,
          'vegetation': 2,
          'snow_ice': 2,
          'cirrus': 2,
          'cloud': 2}

    # Define bit position start -- Pre-Collection
    bps = {
        'designated_fill': 0,
        'dropped_frame': 1,
        'terrain_occlusion': 2,
        # 'reserved': 3,
        'water': 4,
        'cloud_shadow': 6,
        'vegetation': 8,
        'snow_ice': 10,
        'cirrus': 12,
        'cloud': 14}

    """
    Collection-1 populated bits

    0 Designated Fill
    1 Terrain Occlusion
    2-3 Radiometric Saturation
    4 Cloud
    5-6 Cloud Confidence
    7-8 Cloud Shadow Confidence
    9-10 Snow/Ice confidence
    11-12 Cirrus confidence
    13
    14
    15
    """

    # Define bit length (single or double bits) -- Collection-1
    bit_length = {'designated_fill': 1,
          'terrain_occlusion': 1,
          'radiometric_saturation': 2,
          'cloud': 1,
          'cloud_confidence': 2,
          'cloud_shadow_confidence': 2,
          'snow_ice_confidence': 2,
          'cirrus_confidence': 2,
          # 'reserved': 1,
          # 'reserved': 1,
          # 'reserved': 1}

    # Define bit position start -- Collection-1
    bit_position_start = {
        'designated_fill': 0,
        'terrain_occlusion': 1,
        'radiometric_saturation': 2,
        'cloud': 4,
        'cloud_confidence': 5,
        'cloud_shadow_confidence': 7,
        'snow_ice_confidence': 9,
        'cirrus_confidence': 11,
        # 'reserved': 13,
        # 'reserved': 14,
        # 'reserved': 15}

    """
    Pre-Collection

    For the single bits (0, 1, 2, and 3):
        0 = No, this condition does not exist
        1 = Yes, this condition exists.
    """

    """
    Collection-1

    For the single bits (0, 1, and 4):
        0 = "No" = This condition does not exist
        1 = "Yes" = This condition exists
    """

    # Define single bits dictionary -- Identical for Pre- and C1
    single_bits = {'No': '0',
                   'Yes': '1'}

    """
    Collection-1

    For radiometric saturation bits (2-3), read from left to right, represent how many bands contain saturation:

    00 - No bands contain saturation
    01 - 1-2 bands contain saturation
    10 - 3-4 bands contain saturation
    11 - 5 or more bands contain saturation
    """

    radiometric_saturation = {'No bands contain saturation': '00',
                              '1-2 bands contain saturation': '01',
                              '3-4 bands contain saturation': '10',
                              '5 or more bands contain saturation': '11'}

    """
    Pre-Collection

    The double bits (4-5, 6-7, 8-9, 10-11, 12-13, and 14-15), read from left to
    right, represent levels of confidence that a condition exists:
    00 = 'Not Determined' = Algorithm did not determine the status
                            of this condition
    01 = 'No' = Algorithm has low to no confidence that this condition exists
                (0-33 percent confidence)
    10 = 'Maybe' = Algorithm has medium confidence that this condition exists
                (34-66 percent confidence)
    11 = 'Yes' = Algorithm has high confidence that this condition exists
                (67-100 percent confidence).
    """

    # Define double bits dictionary -- Pre-Collection
    double_bits = {'Not Determined': '00',
                   'No': '01',
                   'Maybe': '10',
                   'Yes': '11'}

    """
    Collection-1

    For the remaining double bits (5-6, 7-8, 9-10, 11-12), read from left to
    right, represent levels of confidence that a condition exists:

    00 = "Not Determined" = Algorithm did not determine the status of this
    condition / "No" = This condition does not exist

    01 = "Low" = Algorithm has low to no confidence that this condition exists
    (0-33 percent confidence)

    10 = "Medium" = Algorithm has medium confidence that this condition exists
    (34-66 percent confidence)

    11 = "High" = Algorithm has high confidence that this condition exists
    (67-100 percent confidence
    """

    # Define double bits dictionary -- Colletion-1
    double_bits_c1 = {'Not Determined': '00',
                   'Low': '01',
                   'Medium': '10',
                   'High': '11'}

Stefan:

Hi Nikos,

Thanks for the heads-up and updates on this matter.

Since GUI cannot be changed dynamically based on selected options, I
guess removing Pre-collection QA bits altogether is the most sensible
solution? Alternatively, the module could be split into two: one for
Pre-collection and one collection-1 version...

This option, here https://gitlab.com/NikosAlexandris/i.landsat8.qa. Let
me know what you think (sorry for rewording names and comments).

Manual not updated.

Nikos

If Pre-Collection products are removed, it might be more convenient to
change the rules to the new collection products?

Feel free to implement changes you prefer...

Cheers Stefan

Stefan, svn committers,

in syncing the i.landsat8.swlst module, I just unintentionally committed
local changes I had for `i.landsat8.qc`. I am sorry for that. What is
the best way to revert only this module in this case?

Thank you, Nikos

* Nikos Alexandris <nik@nikosalexandris.net> [2018-06-05 12:07:52 +0200]:

Stefan:

Hi Nikos,

Thanks for the heads-up and updates on this matter.

Since GUI cannot be changed dynamically based on selected options, I
guess removing Pre-collection QA bits altogether is the most sensible
solution? Alternatively, the module could be split into two: one for
Pre-collection and one collection-1 version...

This option, here https://gitlab.com/NikosAlexandris/i.landsat8.qa. Let
me know what you think (sorry for rewording names and comments).

Manual not updated.

Nikos

If Pre-Collection products are removed, it might be more convenient to
change the rules to the new collection products?

Feel free to implement changes you prefer...

Cheers Stefan

--
Nikos Alexandris | Remote Sensing & Geomatics
GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

Hei Nikos,

No worries. Did not have the time to test, but if your changes make Collection 1 work, why reverting?
Happy to see you following up!
And if something by accident is not working as expected, let us rather fix your updated version than reverting your commit.

Cheers
Stefan

-----Original Message-----
From: Nikos Alexandris <nik@nikosalexandris.net>
Sent: lørdag 9. juni 2018 08:00
To: Stefan Blumentrath <Stefan.Blumentrath@nina.no>
Cc: GRASS-GIS development mailing list <grass-dev@lists.osgeo.org>
Subject: Re: Landsat 8 Collection-1 QA bits

Stefan, svn committers,

in syncing the i.landsat8.swlst module, I just unintentionally committed local changes I had for `i.landsat8.qc`. I am sorry for that. What is the best way to revert only this module in this case?

Thank you, Nikos

* Nikos Alexandris <nik@nikosalexandris.net> [2018-06-05 12:07:52 +0200]:

Stefan:

Hi Nikos,

Thanks for the heads-up and updates on this matter.

Since GUI cannot be changed dynamically based on selected options, I
guess removing Pre-collection QA bits altogether is the most sensible
solution? Alternatively, the module could be split into two: one for
Pre-collection and one collection-1 version...

This option, here https://gitlab.com/NikosAlexandris/i.landsat8.qa. Let
me know what you think (sorry for rewording names and comments).

Manual not updated.

Nikos

If Pre-Collection products are removed, it might be more convenient to
change the rules to the new collection products?

Feel free to implement changes you prefer...

Cheers Stefan

--
Nikos Alexandris | Remote Sensing & Geomatics GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

Hi,

the commit was about i.landsat8.swlt. I was, however, in the parent
directory when actually executing the commit command. This, included
minor changes in i.landsat8.qc. Nothing serious, as I see now.

Still, I would like to learn how I could revert partially a commit, if
possible?

For the QA module, https://gitlab.com/NikosAlexandris/i.landsat8.qa is a
separate module. Note s/qc/qa/. We can keep both for the time being?

Thanks, Nikos

* Stefan Blumentrath <Stefan.Blumentrath@nina.no> [2018-06-09 10:08:57 +0000]:

Hei Nikos,

No worries. Did not have the time to test, but if your changes make Collection 1 work, why reverting?
Happy to see you following up!
And if something by accident is not working as expected, let us rather fix your updated version than reverting your commit.

Cheers
Stefan

-----Original Message-----
From: Nikos Alexandris <nik@nikosalexandris.net>
Sent: lørdag 9. juni 2018 08:00
To: Stefan Blumentrath <Stefan.Blumentrath@nina.no>
Cc: GRASS-GIS development mailing list <grass-dev@lists.osgeo.org>
Subject: Re: Landsat 8 Collection-1 QA bits

Stefan, svn committers,

in syncing the i.landsat8.swlst module, I just unintentionally committed local changes I had for `i.landsat8.qc`. I am sorry for that. What is the best way to revert only this module in this case?

Thank you, Nikos

* Nikos Alexandris <nik@nikosalexandris.net> [2018-06-05 12:07:52 +0200]:

Stefan:

Hi Nikos,

Thanks for the heads-up and updates on this matter.

Since GUI cannot be changed dynamically based on selected options, I
guess removing Pre-collection QA bits altogether is the most sensible
solution? Alternatively, the module could be split into two: one for
Pre-collection and one collection-1 version...

This option, here https://gitlab.com/NikosAlexandris/i.landsat8.qa. Let
me know what you think (sorry for rewording names and comments).

Manual not updated.

Nikos

If Pre-Collection products are removed, it might be more convenient to
change the rules to the new collection products?

Feel free to implement changes you prefer...

Cheers Stefan

--
Nikos Alexandris | Remote Sensing & Geomatics GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

--
Nikos Alexandris | Remote Sensing & Geomatics
GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

Hi Nikos,

On Sun, Jun 10, 2018 at 8:25 PM, Nikos Alexandris
<nik@nikosalexandris.net> wrote:
...

Still, I would like to learn how I could revert partially a commit, if
possible?

Maybe this helps:
https://trac.osgeo.org/grass/wiki/HowToSVN#Revertingsubmittedchanges

Best
Markus