[GRASS-dev] ascii export and import, large file problem

GRASS GIS GRASS GIS developer
1

I’m using grass6.3 updated today so the large file support for the ascii commands is included. I export a file using r.out.arc and then read it back in using r.in.arc. The attached jpg shows the original raster on the right. The screen on the left is the original raster minus the exported and imported version. The bottom two thirds or so of the left raster is zero, as it should be, but the top 1/3 has a bunch of small values (range is – to +2.9). The header info on the ascii file is

ncols 17067
nrows 21048
xllcorner -163395.4953122
yllcorner -516147.35893555
cellsize 119.047796
NODATA_value -9999

Is there a bug in r.out or r.in that only shows up when the number of pixels is really big?

Any help appreciated.

Regards, Jerry

(attachments)

diffMapSmall.jpg

2

Jerry Nelson wrote:

I'm using grass6.3 updated today so the large file support for the ascii
commands is included. I export a file using r.out.arc and then read it back
in using r.in.arc. The attached jpg shows the original raster on the right.
The screen on the left is the original raster minus the exported and
imported version. The bottom two thirds or so of the left raster is zero, as
it should be, but the top 1/3 has a bunch of small values (range is - to
+2.9).

My first guess is that the export->import process is changing the
vertical extent of the map slightly, so the calculation in the upper
portion of the map is using cells which are off by one row.

What does r.info say about the bounds of the two maps?

--
Glynn Clements <glynn@gclements.plus.com>

3

The regions were in fact different before and after the export/import. We
used the default decimal setting. We're trying 16 decimals to see if that
makes a difference.

Jerry

-----Original Message-----
From: Glynn Clements [mailto:glynn@gclements.plus.com]
Sent: Thursday, April 12, 2007 9:43 AM
To: Jerry Nelson
Cc: 'grass-dev'
Subject: Re: [GRASS-dev] ascii export and import, large file problem

Jerry Nelson wrote:

I'm using grass6.3 updated today so the large file support for the ascii
commands is included. I export a file using r.out.arc and then read it

back

in using r.in.arc. The attached jpg shows the original raster on the

right.

The screen on the left is the original raster minus the exported and
imported version. The bottom two thirds or so of the left raster is zero,

as

it should be, but the top 1/3 has a bunch of small values (range is - to
+2.9).

My first guess is that the export->import process is changing the
vertical extent of the map slightly, so the calculation in the upper
portion of the map is using cells which are off by one row.

What does r.info say about the bounds of the two maps?

--
Glynn Clements <glynn@gclements.plus.com>

4

To provide more info,

The 'after' info
| Rows: 21048

| Columns: 17067

| Total Cells: 359226216

| Projection: UTM (zone 37)

| N: 1989570.65127245 S: -516147.35893555 Res: 119.047796

| E: 1868393.2390198 W: -163395.4953122 Res: 119.047796

| Range of data: min = 0.000000 max = 557.001038

The 'before' info
| Rows: 21048
| Columns: 17067

| Total Cells: 359226216

| Projection: UTM (zone 37)

| N: 1989664.48241023 S: -516147.35893555 Res: 119.05225396

| E: 1868393.2316229 W: -163395.4953122 Res: 119.04779557

| Range of data: min = 0.000000 max = 557.001038

-----Original Message-----
From: Glynn Clements [mailto:glynn@gclements.plus.com]
Sent: Thursday, April 12, 2007 9:43 AM
To: Jerry Nelson
Cc: 'grass-dev'
Subject: Re: [GRASS-dev] ascii export and import, large file problem

Jerry Nelson wrote:

I'm using grass6.3 updated today so the large file support for the ascii
commands is included. I export a file using r.out.arc and then read it

back

in using r.in.arc. The attached jpg shows the original raster on the

right.

The screen on the left is the original raster minus the exported and
imported version. The bottom two thirds or so of the left raster is zero,

as

it should be, but the top 1/3 has a bunch of small values (range is - to
+2.9).

My first guess is that the export->import process is changing the
vertical extent of the map slightly, so the calculation in the upper
portion of the map is using cells which are off by one row.

What does r.info say about the bounds of the two maps?

--
Glynn Clements <glynn@gclements.plus.com>

5

Jerry Nelson wrote:

> > I'm using grass6.3 updated today so the large file support for the ascii
> > commands is included. I export a file using r.out.arc and then read it back
> > in using r.in.arc. The attached jpg shows the original raster on the right.
> > The screen on the left is the original raster minus the exported and
> > imported version. The bottom two thirds or so of the left raster is zero, as
> > it should be, but the top 1/3 has a bunch of small values (range is - to
> > +2.9).
>
> My first guess is that the export->import process is changing the
> vertical extent of the map slightly, so the calculation in the upper
> portion of the map is using cells which are off by one row.
>
> What does r.info say about the bounds of the two maps?

To provide more info,

The 'after' info

Rows: 21048

Res: 119.047796

The 'before' info

Rows: 21048

Res: 119.05225396

119.05225396 - 119.047796 = 0.00445796
0.00445796 * 21048 = 93.8311421

So, the imported map has shrunk by almost a whole cell. That would
certainly explain the results.

Ah, I see where the problem lies:

The 'before' info

Res: 119.05225396
Res: 119.04779557

Your cells aren't square, but the ArcGrid format doesn't appear to
allow for non-square cells (single "cellsize" value rather than
separate x/y values). r.out.arc uses the horizontal resolution for the
cellsize value; if the vertical resolution is different, you lose.

This specific issue can't be fixed. However, if the original data had
square cells, something is going wrong on the initial import.

We might want to add a check for this to r.out.arc. We can't actually
do anything beyond warn you that exporting will lose information,
although that's better than nothing.

--
Glynn Clements <glynn@gclements.plus.com>

6

An update on ascii export and import. When we set decimals to 16 in the
export and import commands, the difference worked as expected; that is,
there is no difference between the original and the exported/imported/ new
version. Is this something that needs to be put in the documentation? Jerry

-----Original Message-----
From: Glynn Clements [mailto:glynn@gclements.plus.com]
Sent: Thursday, April 12, 2007 5:58 PM
To: Jerry Nelson
Cc: 'grass-dev'
Subject: RE: [GRASS-dev] ascii export and import, large file problem

Jerry Nelson wrote:

> > I'm using grass6.3 updated today so the large file support for the

ascii

> > commands is included. I export a file using r.out.arc and then read it

back

> > in using r.in.arc. The attached jpg shows the original raster on the

right.

> > The screen on the left is the original raster minus the exported and
> > imported version. The bottom two thirds or so of the left raster is

zero, as

> > it should be, but the top 1/3 has a bunch of small values (range is -

to

> > +2.9).
>
> My first guess is that the export->import process is changing the
> vertical extent of the map slightly, so the calculation in the upper
> portion of the map is using cells which are off by one row.
>
> What does r.info say about the bounds of the two maps?

To provide more info,

The 'after' info

Rows: 21048

Res: 119.047796

The 'before' info

Rows: 21048

Res: 119.05225396

119.05225396 - 119.047796 = 0.00445796
0.00445796 * 21048 = 93.8311421

So, the imported map has shrunk by almost a whole cell. That would
certainly explain the results.

Ah, I see where the problem lies:

The 'before' info

Res: 119.05225396
Res: 119.04779557

Your cells aren't square, but the ArcGrid format doesn't appear to
allow for non-square cells (single "cellsize" value rather than
separate x/y values). r.out.arc uses the horizontal resolution for the
cellsize value; if the vertical resolution is different, you lose.

This specific issue can't be fixed. However, if the original data had
square cells, something is going wrong on the initial import.

We might want to add a check for this to r.out.arc. We can't actually
do anything beyond warn you that exporting will lose information,
although that's better than nothing.

--
Glynn Clements <glynn@gclements.plus.com>

7

Jerry Nelson wrote:

An update on ascii export and import. When we set decimals to 16 in the
export and import commands, the difference worked as expected; that is,
there is no difference between the original and the exported/imported/ new
version. Is this something that needs to be put in the documentation? Jerry

You're omitting something important: the details of the projection
step.

r.proj projects into an existing region; it doesn't create it itself.
Something created that region, and that's where everything went wrong.

Beyond that, if you export an ArcGrid file from a region with
non-square pixels, then re-import that file, the result will be wrong
however many decimal places you use.

--
Glynn Clements <glynn@gclements.plus.com>