[GRASS-user] WV-2 top of atmosphere reflectance calculation

Hi All,

I'm working on a python script that extracts the variables necessary
to convert WorldView-2 DNs to top of atmosphere reflectance values.
The equations come from
http://www.digitalglobe.com/sites/default/files/Radiometric_Use_of_WorldView-2_Imagery%20(1).pdf.

the mapcalc function is a combination of two of the formulas from the
pdf. The first formula converts DNs to band averaged spectral
radiance:

Radiance = (Absolute Calibration Factor * DN) / Effective Bandwidth (page 9)

The second formula (page 16) incorporates some other variables to
convert to top of atmosphere reflectance:

TOA Reflectance = (Radiance * Earth Sun Distance ^2 * pi) / (Band
Averaged Solar Spectral Irradiance * cos(solar zenith angle))

The band averaged solar spectral Irradiance is eSun in the mapcalc formula.

My mapcalc function:

   grass.mapcalc("$output = ((($absCal * $input_rast) / $efb) * $esd^2
* $pi) / ($eSun * cos($solarGeom))",
       output=output, absCal=absCalFactor, input_rast=raster,
       efb=effectiveBandwidth, esd=earthSunDist, pi=math.pi, eSun=eSun,
       solarGeom=solarGeom)

As far as I can tell my formula looks correct, but some bands have max
values exceeding 1.0, which I don't think should happen. I was hoping
some additional eyes might be able to shed some light on where I'm
going wrong.

Thanks,
Eric

On Thursday 11 of July 2013 22:38:10 Eric Goddard wrote:

Hi All,

I'm working on a python script that extracts the variables necessary
to convert WorldView-2 DNs to top of atmosphere reflectance values.

[..]

Dear Eric,

are you interested in working together to get WorldView "band filters" in
i.atcorr? See:
<http://trac.osgeo.org/grass/browser/grass/trunk/imagery/i.atcorr/README&gt;

I am working these days with a few WV scenes.

Best, Nikos

Hi Nikos,
yes, I’m definitely interested. I haven’t even considered atmospheric corrections yet, I’ve been trying to get the DN to reflectance part working correctly first. I will post my complete Python script for that soon. Right now it is a one off script to process all of my scenes by once I have it working correctly I would like to turn it into a more generic DN to reflectance tool.

Eric

On Jul 12, 2013 4:15 AM, “Nikos Alexandris” <nik@nikosalexandris.net> wrote:

On Thursday 11 of July 2013 22:38:10 Eric Goddard wrote:

Hi All,

I’m working on a python script that extracts the variables necessary
to convert WorldView-2 DNs to top of atmosphere reflectance values.

[…]

Dear Eric,

are you interested in working together to get WorldView “band filters” in
i.atcorr? See:
<http://trac.osgeo.org/grass/browser/grass/trunk/imagery/i.atcorr/README>

I am working these days with a few WV scenes.

Best, Nikos

Hi Nikos,

I looked at the i.atcorr readme you linked. I just want to be sure I
understand this correctly: when you say band filters, are you talking
about part F: pre-defined sensor bands of the i.atcorr man page?
(http://grass.osgeo.org/grass70/manuals/i.atcorr.html)

I've also posted my python script to convert WV-2 multispectral DNs
to top of atmosphere reflectance at
https://gist.github.com/egoddard/5989382. It is kind of messy because
it was just meant to be a one-off script at this point :slight_smile:

Thanks,
Eric

On Fri, Jul 12, 2013 at 7:14 AM, Eric Goddard <egoddard1010@gmail.com> wrote:

Hi Nikos,
yes, I'm definitely interested. I haven't even considered atmospheric
corrections yet, I've been trying to get the DN to reflectance part working
correctly first. I will post my complete Python script for that soon. Right
now it is a one off script to process all of my scenes by once I have it
working correctly I would like to turn it into a more generic DN to
reflectance tool.

Eric

On Jul 12, 2013 4:15 AM, "Nikos Alexandris" <nik@nikosalexandris.net> wrote:

On Thursday 11 of July 2013 22:38:10 Eric Goddard wrote:
> Hi All,
>
> I'm working on a python script that extracts the variables necessary
> to convert WorldView-2 DNs to top of atmosphere reflectance values.

[..]

Dear Eric,

are you interested in working together to get WorldView "band filters"
in
i.atcorr? See:
<http://trac.osgeo.org/grass/browser/grass/trunk/imagery/i.atcorr/README&gt;

I am working these days with a few WV scenes.

Best, Nikos

Eric Goddard wrote:

Hi Nikos,

Hi Eric :slight_smile:

yes, I'm definitely interested.

Cool!

I haven't even considered atmospheric corrections yet, I've been trying to
get the DN to reflectance part working correctly first. I will post my
complete Python script for that soon. Right now it is a one off script to
process all of my scenes by once I have it working correctly I would like to
turn it into a more generic DN to reflectance tool.

Nice, I'll check it out soon.

Nikos

Eric Goddard wrote:

I looked at the i.atcorr readme you linked. I just want to be sure I
understand this correctly: when you say band filters, are you talking
about part F: pre-defined sensor bands of the i.atcorr man page?
(http://grass.osgeo.org/grass70/manuals/i.atcorr.html)

Right, part F. If I got it right, we _only_ need to create a .csv file that
will contain the spectral radiance responses for each band within the sensor's
"spectral range" capabilities.

The csv file has to be structured as follows:

1) the 1st line must be a header with the Wavelength (in nm), followed by band
names

2) the following lines will be the actual (estimated?) spectral responses

3) if a spectral radiance response is NULL at a given wavelength, the
(line,column) should be left empty

Examples are to be found in
<http://trac.osgeo.org/grass/browser/grass/trunk/imagery/i.atcorr/sensors_csv&gt;\.

After preparing the .csv file, we can use the create_iwave.py script to
interpolate the "filter functions" to the "correct" step size of 2.5 nm (in
case the filter function data are delivered with a different step size).

In the PDF document you linked in your first post,

<http://www.digitalglobe.com/sites/default/files/Radiometric_Use_of_WorldView-2_Imagery%20(1).pdf&gt;,

there is

"Figure 2: WorldView-2 Relative Spectral Radiance Response (nm)" in page 3,
some "averaged band passes" in "Table 2: WORLDVIEW-2 Band Passes [nm]" in page
4 and "Table 3. WorldView-2 Effective Bandwidths" again in page 4.

We need to do some math to approximate the spectral responses at a 2.5nm step.
However, there is a note:

"Note that the actual data values used to create these plots are
available as digital files, upon request from DigitalGlobe".

I think we need to know the(se) functions/numbers from which "Figure 2" has
been derived. Then we can do some math in R or so to get what is needed.

Well, a bit more work after all than I though -- as usual :-).

I've also posted my python script to convert WV-2 multispectral DNs
to top of atmosphere reflectance at
https://gist.github.com/egoddard/5989382. It is kind of messy because
it was just meant to be a one-off script at this point :slight_smile:

Nice, will check...

Nikos

On Monday 15 of July 2013 02:10:56 Nikos Alexandris wrote:

In the PDF document you linked in your first post,

<http://www.digitalglobe.com/sites/default/files/Radiometric_Use_of_WorldVie
w-2_Imagery%20%281%29.pdf>

Also, a "response curves" only document for QB, WV-1 and WV-2.

<https://www.digitalglobe.com/sites/default/files/DigitalGlobe_Spectral_Response_0.pdf&gt;

Nikos

Eric Goddard wrote:

I've also posted my python script to convert WV-2 multispectral DNs
to top of atmosphere reflectance at
https://gist.github.com/egoddard/5989382. It is kind of messy because
it was just meant to be a one-off script at this point :slight_smile:

Eric, I am on it -- didn't read the whole script yet. Do you execute the
python script inside a Location/Mapset with the bands in question? (I guess
so). Where do you store the metadata files? How does the script find 'em?

Any quick example?

Thanks, Nikos

I completely agree with keeping the discussion on list–I was in a rush this morning and forgot to hit reply all. My out-of-context replies are below, followed by the discussion that occurred off-list.

Using an ‘r’ before a string in python just denotes that it is a raw string, so special characters such as backslashes will be ignored. This is just a habit because until recently I had to use Windows at work.

I followed your other posting about your NITF woes…that sounds rough. I’m happy my data came as geotiffs!

My reasoning behind making it more generic is so that it could eventually become a WV2 variant of the i.landsat.toar/i.aster.toar modules. Once the generic tool is done it could still be scripted to process a large number of individual scenes.

Could you send me a copy of your metadata file? It looks like our imagery differs in the amount of processing applied, but I haven’t been able to find any references for the WV-2 processing levels so far. It’s definitely more difficult to find the necessary information for privately operated satellites. It would be great to see the other potential metadata forms for the generic toar tool.

Thanks again,

Eric

On Thu, Jul 18, 2013 at 6:56 AM, Nikos Alexandris <nik@nikosalexandris.net> wrote:

It seems it’s complicated. In the XML file for the image I am working with,

13APR10075059-M1BS-500060446050_05_P003.XML

there is no “EARLIESTACQTIME” string (referring to line 119 in your script).

I will have to postpone it a bit with testing. It’s a common secret all
these constanctly changing metadata strings… A bit frustrating some times.

Best, Nikos

Hi!

On Thursday 18 of July 2013 06:04:40 Eric Goddard wrote:

On line 211 in the main() function you have to specify the path to the

WorldView-2 source directory that contains all of the imagery, IMD

files, XML files, etc.

In the “version” I downloaded from gist, there is probably a typo/leftover:

imageryPath = r"/path/to/imagery/folder"

If I am wrnog, I can’t see how “r” could be useful now – I am just about to try.

In my GRASS setup I used r.external to link all

of the files in that path. If the original TIF file in the imagery

path was 12OCT19171330-M3DS_R1C1-052823926030_01_P001.TIF, for

example, the raster name in grass would be

x2OCT19171330-M3DS_R1C1-052823926030_01_P001.; the

metadata file that is associated with that tif is

12OCT19171330-M3DS-052823926030_01_P001.XML.

I guess I will need to adjust various stuff. I have only .ntf files. Of course, I converted 'em in GeoTiffs but I’d like to see how it plays with NITF containers and SUBDATASETs.

The AssociateMetaDataWithRaster() function takes the list of grass

rasters (acquired in line 213), and creates a dictionary that has the

grass raster name as the key and the XML file as the value.

Ok.

The loop starting at line 219 goes over each raster, calls the

ExtractVariablesFromMetadata() function (in which the first argument

is the path to the xml file for the raster, and the second is the band

number) and uses the lxml library to search for and extract the

variables from the XML metadata document.

Sound nice.

Thanks for going over it, Nikos. A coworker and I spent hours combing

through it and we couldn’t find any mistakes in the math (not to say

there aren’t any of course!); I’m all out of ideas for what could be

going wrong.

Myself, I am just a DIY scripter. If I bomb into anything obvious, of course I’ll point it out.

I’m currently working on a more generic version in which

you specify the image and the path to the xml file to make it easier

to test.

Yes, it’ll be much better. I too have a lot of hardcoded stuff in my scripts. But, it doens’t make too much sense in the end/long run.

Sorry for the strange python spacing too…Vim configuration

gone wrong :).

:smiley:

Hey, thanks a lot.

N

ps- just my humble opinion, better to keep discussion on-list.

Nikos Alexandris wrote:

...get WorldView "band filters" in i.atcorr? See:
<http://trac.osgeo.org/grass/browser/grass/trunk/imagery/i.atcorr/README&gt;

See: <http://trac.osgeo.org/grass/ticket/2059&gt;\.

Can anyone, please, review and confirm? If it is ok, can we have that "on-
line"?

The plot (via R) looks (to me) "identical" with the "official" one in
<http://www.digitalglobe.com/downloads/Radiometric_Use_of_WorldView-2_Imagery.pdf&gt;\.

Thank you, Nikos