[GRASS-user] Spatial data exchange formats

GRASS GRASS user
,
1

On Wed, 30 May 2018, Helmut Kudrnovsky wrote:

So you have to look by ogrinfo/gdalinfo --formats; you can get further
information on the GDAL Website.

   I have LiDAR files in two ESRI formats and don't know which would be the
better one to use: .shp and .adf.

   My research taught me that .adf is Amiga Data File, and I thought that
Amiga died several decades ago. Seems that ESRI is using that format for
some reason and it does not appear to be supported by gdal.

   The directories with the adf files also contain .dat, .nit, and .dir
files and come in three versions: bare earth, highest hits, and intensity.

   My use of these DEMs is for hydrological modeling, including storm runoff
with times of concentration for specified 24-hour storm hydrographs. So, I
want to become informed about my choices. Recommended readings also
appreciated.

Regards,

Rich

2

Hi Rich,

Actually, your ADF file is part of a vector format from ESRI named Coverage, that includes some files in a folder with the name of your data and some other data in a folder called INFO. Here is some more info on the Coverage format [1]

GDAL OGR can handle both shape and coverage formats so you could import any of the two. I believe it does not matter which one you choose. Just fire up GRASS and import your vector file.

To import the Coverage, I believe in GRASS you have to select the ADF file inside the coverage directory. To import the shapefile, just select the file with the shp extension

Cheers
Daniel

[1] - https://support.esri.com/en/technical-article/000007209

On Fri, Jun 8, 2018 at 3:15 PM Rich Shepard <rshepard@appl-ecosys.com> wrote:

On Wed, 30 May 2018, Helmut Kudrnovsky wrote:

So you have to look by ogrinfo/gdalinfo --formats; you can get further
information on the GDAL Website.

I have LiDAR files in two ESRI formats and don’t know which would be the
better one to use: .shp and .adf.

My research taught me that .adf is Amiga Data File, and I thought that
Amiga died several decades ago. Seems that ESRI is using that format for
some reason and it does not appear to be supported by gdal.

The directories with the adf files also contain .dat, .nit, and .dir
files and come in three versions: bare earth, highest hits, and intensity.

My use of these DEMs is for hydrological modeling, including storm runoff
with times of concentration for specified 24-hour storm hydrographs. So, I
want to become informed about my choices. Recommended readings also
appreciated.

Regards,

Rich

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

3

On Mon, 18 Jun 2018, Daniel Victoria wrote:

Actually, your ADF file is part of a vector format from ESRI named
Coverage, that includes some files in a folder with the name of your data
and some other data in a folder called INFO. Here is some more info on the
Coverage format [1]

Daniel,

   I thought a grid meant raster, not vector. I wonder why LiDAR data is
presented in a vector format rather than a raster DEM. Off-hand, I don't
know how I'll use vector maps for hydrological modeling, particularly storm
water runoff.

   In the meantime, I've switched focus to munging 30 years of river stage
height data. Still have 7 years to go. Tedious, even with emacs macros and
awk scripts, but I'll finish this effort before returning to the topography.

Many thanks,

Rich

4

Hi Rich,

You are correct about GRID being a raster format in ESRI world. But from your description, I believe what you have is a coverage, which is ESRI vector format for ArcINFO (shapefile beeing another vector format).

LiDAR data is obtained as XYZ point cloud. That’s why you probably have vector data. Further processing is needed to transform it to a raster format.

Cheers
Daniel

On Mon, Jun 18, 2018 at 11:25 AM Rich Shepard <rshepard@appl-ecosys.com> wrote:

On Mon, 18 Jun 2018, Daniel Victoria wrote:

Actually, your ADF file is part of a vector format from ESRI named
Coverage, that includes some files in a folder with the name of your data
and some other data in a folder called INFO. Here is some more info on the
Coverage format [1]

Daniel,

I thought a grid meant raster, not vector. I wonder why LiDAR data is
presented in a vector format rather than a raster DEM. Off-hand, I don’t
know how I’ll use vector maps for hydrological modeling, particularly storm
water runoff.

In the meantime, I’ve switched focus to munging 30 years of river stage
height data. Still have 7 years to go. Tedious, even with emacs macros and
awk scripts, but I’ll finish this effort before returning to the topography.

Many thanks,

Rich

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

5

Hi Rich,

Looks like I confused things a bit.

ESRI ArcINFO used two formats that spreded files into a directory with the layer name and another directory called INFO. In the directory with the layer name, the files had the .ADF extension you mentioned in the first email. But their content is different.

From the list you sent, you have a file named hdr.adf, which is a giveaway that we are dealing with an ESRI GRID (raster format). The presence of a .OVR file also indicates that it’s a grid, OVR being the raster pyramid overviews.

So, if you have a Raster product, it means that the person providing the data took care of the processing to generate both the bare earth model and the highest hit.

Anyway, you can use r.in.gdal to import those GRIDS into GRASS. Just point to the hdr.adf file and all should be OK.

Cheers and sorry for the confusion.
Daniel

PS - Some references explaining both ESRI formats
http://desktop.arcgis.com/en/arcmap/10.3/manage-data/raster-and-images/esri-grid-format.htm

http://desktop.arcgis.com/en/arcmap/10.3/manage-data/coverages/contents-of-a-coverage-workspace.htm

On Thu, Jun 21, 2018 at 11:50 AM Rich Shepard <rshepard@appl-ecosys.com> wrote:

On Mon, 18 Jun 2018, Daniel Victoria wrote:

You are correct about GRID being a raster format in ESRI world. But from
your description, I believe what you have is a coverage, which is ESRI
vector format for ArcINFO (shapefile being another vector format).

Daniel,

What information can I provide that clarifies the format in which these
data were delivered so I know how to import them?

A couple of years ago I downloaded LiDAR data from the same state agency
and that was provided in .gdb format. When I imported the files I had clouds
of data points for bare earth and highest hits. Why the newer data are in a
different format I’ve no idea. The agency tells me they’ll sell the point
cloud data, but my project budget has no provision for buying data.

Attached is a list of directories and files for the 2014 LiDAR data
flights. I need to learn how to import these and use them to create bare
earth and highest hit DEMs useful for hydrological modeling. Help from
those of you who focus on spatial analyses with grass is needed. If I can
provide more information, such as specific files, tell me and I’ll provide
them. (The entire directory compressed using xz is about 165M.)

Regards,

Rich

6

On Thu, 21 Jun 2018, Daniel Victoria wrote:

ESRI ArcINFO used two formats that spreded files into a directory with the
layer name and another directory called INFO. In the directory with the
layer name, the files had the .ADF extension you mentioned in the first
email. But their content is different.

Daniel,

   In the late 1980s I used Arc/INFO on a Prime and pc-arc/info on a pc. Info
was the attribute database.

From the list you sent, you have a file named hdr.adf, which is a giveaway
that we are dealing with an ESRI GRID (raster format). The presence of a
.OVR file also indicates that it's a grid, OVR being the raster pyramid
overviews.

   Thought so.

So, if you have a Raster product, it means that the person providing the
data took care of the processing to generate both the bare earth model and
the highest hit.

Anyway, you can use r.in.gdal to import those GRIDS into GRASS. Just point
to the hdr.adf file and all should be OK.

   My problems are that pointing r.in.gdal to the hdr.adf files display only
the strange wedge/rectangle I attached to messages earlier in this thread.
They should be retangles for a full 1 degree x 1 degree topographic quad map
with colors varying by elevation.

   My need is to learn why I'm not getting clean imports of either bare_earth
or highest_hits for any of the three years of data. This is why this thread
has continued so long. The only option to r.in.gdal I've used is '-o' to
use the source's projection for the map.

Thanks,

Rich

7

OK.

I tried to download the data you placed on file dropper (other email tread). But both links you sent gave me corrupt files.

If you can manage to place the files in some shared folder or point to the original data source, I can take a look.

Have you tried opening in some other software, like QGis?

Cheers
Daniel

On Thu, Jun 21, 2018 at 2:24 PM Rich Shepard <rshepard@appl-ecosys.com> wrote:

On Thu, 21 Jun 2018, Daniel Victoria wrote:

ESRI ArcINFO used two formats that spreded files into a directory with the
layer name and another directory called INFO. In the directory with the
layer name, the files had the .ADF extension you mentioned in the first
email. But their content is different.

Daniel,

In the late 1980s I used Arc/INFO on a Prime and pc-arc/info on a pc. Info
was the attribute database.

From the list you sent, you have a file named hdr.adf, which is a giveaway
that we are dealing with an ESRI GRID (raster format). The presence of a
.OVR file also indicates that it’s a grid, OVR being the raster pyramid
overviews.

Thought so.

So, if you have a Raster product, it means that the person providing the
data took care of the processing to generate both the bare earth model and
the highest hit.

Anyway, you can use r.in.gdal to import those GRIDS into GRASS. Just point
to the hdr.adf file and all should be OK.

My problems are that pointing r.in.gdal to the hdr.adf files display only
the strange wedge/rectangle I attached to messages earlier in this thread.
They should be retangles for a full 1 degree x 1 degree topographic quad map
with colors varying by elevation.

My need is to learn why I’m not getting clean imports of either bare_earth
or highest_hits for any of the three years of data. This is why this thread
has continued so long. The only option to r.in.gdal I’ve used is ‘-o’ to
use the source’s projection for the map.

Thanks,

Rich

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

8

On Thu, 21 Jun 2018, Daniel Victoria wrote:

I tried to download the data you placed on file dropper (other email
tread). But both links you sent gave me corrupt files.

Daniel,

   There was something funky going on with file dropper. But, I just
successfully uploaded bare_earth.tar.xz there:

<http://www.fileconvoy.com/dfl.php?id=g1042def7af51cbea10000921895d47f19134d21070&gt;

File size is 7.6M (uncompressed: 669M).

   I just downloaded it using that URL and the file size is the same as that
of the uploaded file. It will remain there for 3 days.

Have you tried opening in some other software, like QGis?

   No. I have qgis-2.18.18 installed but have not used it. I'm getting closer
to replacing this ancient 32-bit system with a more capable 74-bit system.
The new system will accommodate the latest qgis and I'll spend time learning
it then for map publication.

Thanks,

Rich

9

On Thu, 21 Jun 2018, Rich Shepard wrote:

to replacing this ancient 32-bit system with a more capable 74-bit system.

   Er, ... that should be 64-bit system.

Rich

10

On Thu, Jun 21, 2018 at 7:24 PM, Rich Shepard <rshepard@appl-ecosys.com> wrote:

My problems are that pointing r.in.gdal to the hdr.adf files display only
the strange wedge/rectangle I attached to messages earlier in this thread.
They should be retangles for a full 1 degree x 1 degree topographic quad map
with colors varying by elevation.

No, they should not. They cover a part of that rectangle. You need to patch coverages from different years together in order to get a more complete coverage.

My need is to learn why I’m not getting clean imports of either bare_earth
or highest_hits for any of the three years of data. This is why this thread
has continued so long. The only option to r.in.gdal I’ve used is ‘-o’ to
use the source’s projection for the map.

The -o flag of r.in.gdal overrides the source’s projection for the map instead of using it.

Markus M

Thanks,

Rich

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

11

Rich,

I was able to open your GRID in QGis. And if it works in QGis, it should work in GRASS too.

What I’ve seen is that most of you data has values -3.4028e+38 which appears to be the NODATA value. So, are you sure this part of the data is not a corner of the mapped area? Because there is valid data for the upper and left borders. Also, this data is not a 1 x 1 degree rectangle, as you said it should be. So this makes me believe that we are looking at just one corner of the data, where it happens to have a lot of NODATA cells.

Cheers
Daniel

On Thu, Jun 21, 2018 at 3:17 PM Markus Metz <markus.metz.giswork@gmail.com> wrote:

On Thu, Jun 21, 2018 at 7:24 PM, Rich Shepard <rshepard@appl-ecosys.com> wrote:

My problems are that pointing r.in.gdal to the hdr.adf files display only
the strange wedge/rectangle I attached to messages earlier in this thread.
They should be retangles for a full 1 degree x 1 degree topographic quad map
with colors varying by elevation.

No, they should not. They cover a part of that rectangle. You need to patch coverages from different years together in order to get a more complete coverage.

My need is to learn why I’m not getting clean imports of either bare_earth
or highest_hits for any of the three years of data. This is why this thread
has continued so long. The only option to r.in.gdal I’ve used is ‘-o’ to
use the source’s projection for the map.

The -o flag of r.in.gdal overrides the source’s projection for the map instead of using it.

Markus M

Thanks,

Rich

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

12

Trying to help those trying to help Rich:

the tiling scheme is available in a web viewer here:

https://gis.dogami.oregon.gov/maps/lidarviewer/

the archive containing the data in question is available here:

http://www.oregongeology.org/pubs/ldq/LDQ-45122C3.zip (1.8 GB)

you need to patch together all bare earth coverages from all years in order to get a more complete coverage

Markus M

···

On Thu, Jun 21, 2018 at 8:27 PM, Daniel Victoria <daniel.victoria@gmail.com> wrote:

Rich,

I was able to open your GRID in QGis. And if it works in QGis, it should work in GRASS too.

What I’ve seen is that most of you data has values -3.4028e+38 which appears to be the NODATA value. So, are you sure this part of the data is not a corner of the mapped area? Because there is valid data for the upper and left borders. Also, this data is not a 1 x 1 degree rectangle, as you said it should be. So this makes me believe that we are looking at just one corner of the data, where it happens to have a lot of NODATA cells.

Cheers

Daniel

On Thu, Jun 21, 2018 at 3:17 PM Markus Metz <markus.metz.giswork@gmail.com> wrote:

On Thu, Jun 21, 2018 at 7:24 PM, Rich Shepard <rshepard@appl-ecosys.com> wrote:

My problems are that pointing r.in.gdal to the hdr.adf files display only
the strange wedge/rectangle I attached to messages earlier in this thread.
They should be retangles for a full 1 degree x 1 degree topographic quad map
with colors varying by elevation.

No, they should not. They cover a part of that rectangle. You need to patch coverages from different years together in order to get a more complete coverage.

My need is to learn why I’m not getting clean imports of either bare_earth
or highest_hits for any of the three years of data. This is why this thread
has continued so long. The only option to r.in.gdal I’ve used is ‘-o’ to
use the source’s projection for the map.

The -o flag of r.in.gdal overrides the source’s projection for the map instead of using it.

Markus M

Thanks,

Rich

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

13

On Thu, 21 Jun 2018, Markus Metz wrote:

No, they should not. They cover a part of that rectangle. You need to
patch coverages from different years together in order to get a more
complete coverage.

Markus,

   Ah, I didn't realize this. That explains the why there are three files
with different names for three years.

The -o flag of r.in.gdal overrides the source's projection for the map
instead of using it.

   Mea culpa! I had that backwards.

Thanks very much,

Rich

14

On Thu, 21 Jun 2018, Daniel Victoria wrote:

Daniel,

What I've seen is that most of you data has values -3.4028e+38 which
appears to be the NODATA value. So, are you sure this part of the data is
not a corner of the mapped area? Because there is valid data for the upper
and left borders. Also, this data is not a 1 x 1 degree rectangle, as you
said it should be. So this makes me believe that we are looking at just
one corner of the data, where it happens to have a lot of NODATA cells.

   Yes, Markus pointed this out to me. I'll import all three years' worth of
data and patch them together.

   I did not recognize this from reading the descriptive data that
accompanied the spatial data.

Thanks again,

Rich

15

On Thu, 21 Jun 2018, Markus Metz wrote:

you need to patch together all bare earth coverages from all years in
order to get a more complete coverage

Markus,

   I missed this point when I read the descriptive data. It certainly
explains why there's 2009_OLC_Hood-to-Coast, 2013_OLC_Clackamol, and
2014_OLC_Metro subdirectories.

   Now I'm on the right track. My thanks to all of you.

Regards,

Rich