[GRASS-user] Importing data in .atx/.gdb format

On Wed, 14 Sep 2016, Even Rouault wrote:

Rename this directory as /~/data/grassdata/or-trans.gdb, since the
FileGDB/OpenFileGDB drivers are strict about the directory name ending
with .gdb, and use that as the name provided to v.in.ogr / ogrinfo.

Even,

   I did not pick up on the need to put a .gdb extenstion on the
subdirectory. That certainly makes all the difference, resolves the issue,
and has taught me valuable knowledge.

Thank you very much,

Rich

On Wed, 14 Sep 2016, Even Rouault wrote:

Cool I've just updated the doc of the FileGDB and OpenFileGDB driver to be
very clear on that :

"""The OpenFileGDB driver provides read access to File Geodatabases
(.gdb directories) created by ArcGIS 9 and above. The dataset name must be
the directory/folder name, and it must end with the .gdb extension."""

(will be refreshed online in a few hours)

Even,

   That will help folks like me who are new to importing those data.

   I do need to move the source directory and re-import because the
data are not long-lat coordinates and also not the projection of my project
data. When grass finished importing it told me the map(s) could not be
displayed because the north edge was undefined, and the source projection
was not the target projection. Then the grass shell froze. Killing the
process solved that.

I appreciate all the help and insight all of you provided me,

Rich

On Sep 14, 2016 7:36 PM, “Rich Shepard” <rshepard@appl-ecosys.com> wrote:

Even,

That will help folks like me who are new to importing those data.

Glad to see all these improvements!

I do need to move the source directory and re-import because the
data are not long-lat coordinates and also not the projection of my project
data.

Rich, could you try also v.import here?

Markus

On Wed, Sep 14, 2016 at 4:59 PM, Rich Shepard <rshepard@appl-ecosys.com> wrote:

On Wed, 14 Sep 2016, Helmut Kudrnovsky wrote:

http://www.gdal.org/drv_openfilegdb.html

  Based on this line I get potentially useful information from grass:
"A specific .gdbtable file (including "system" tables) can also be opened
directly."

  Still cannot use the directory as the input target:

v.in.ogr input=~/data/grassdata/or-trans/

ERROR: Unable to open data source </home/rshepard/data/grassdata/or-trans/>

However, a single .gdbtable file produces this output:

v.in.ogr input=~/data/grassdata/or-trans/a00000001.gdbtable

WARNING: All available OGR layers will be imported into vector map
         <GDB_SystemCatalog>

Using CLI, what is the output of
v.in.ogr input=~/data/grassdata/or-trans/a00000001.gdbtable -l
?

ERROR: Projection of dataset does not appear to match current location.

       GRASS LOCATION PROJ_INFO is:
       name: NAD_1983_HARN_StatePlane_Oregon_North_FIPS_3601_Feet_Intl
       datum: nad83harn
       ellps: grs80
       proj: lcc
       lat_1: 44.33333333333334
       lat_2: 46
       lat_0: 43.66666666666666
       lon_0: -120.5
       x_0: 2500000
       y_0: 0
       no_defs: defined

       Import dataset PROJ_INFO is:
       Dataset proj = 0 (unreferenced/unknown)

       In case of no significant differences in the projection definitions,
       use the -o flag to ignore them and use current location definition.
       Consider generating a new location with 'location' parameter from
       input data set.

  This tells me that the source data might be in long-lat and not projected.

This tells you only that the coordinate reference system of the input
data is unknown. It might be Lat/Lon or something else. You can try to
1) import the data into a Lat/Lon location, reproject to the target
location and check if the geolocation is ok
2) ask the data provider about the coordinate reference system for
this data source

Markus M

Since grass will not display the source files' location name I will move
them to the WGS84/ long-lat location.

  This still does not import the files. Using the GUI and selecting
Directory and OpenFileGDB format all files are greyed out and grass won't
open anything because no file has been selected.

  Changing to File with OpenFileGDB format and selecting a single .gdbtable
file produces these results:

(Wed Sep 14 07:50:52 2016) v.import
input=/home/rshepard/data/grassdata/WGS84/odot2014/a00000001.gdbtable
layer=GDB_SystemCatalog output=GDB_SystemCatalog --overwrite
WARNING: All available OGR layers will be imported into vector map
<GDB_SystemCatalog>
ERROR: Coordinate reference system not available for input
</home/rshepard/data/grassdata/WGS84/odot2014/a00000001.gdbtable>
(Wed Sep 14 07:50:52 2016) Command finished (0 sec)

  I'm not ignoring what Stefan, Helmut, and Markus are writing, and I have
again read the gdal.org web page on ESRI OpenFileGDB, but I'm not getting
the expected results.

  If there's no available reference system there must be something broken or
incomplete in the source tarball. I'll contact the repository this morning
and let you know what I learn.

Thanks,

Rich

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Rich Shepard wrote

On Wed, 14 Sep 2016, Even Rouault wrote:

Cool I've just updated the doc of the FileGDB and OpenFileGDB driver to
be
very clear on that :

"""The OpenFileGDB driver provides read access to File Geodatabases
(.gdb directories) created by ArcGIS 9 and above. The dataset name must
be
the directory/folder name, and it must end with the .gdb extension."""

(will be refreshed online in a few hours)

Even,

   That will help folks like me who are new to importing those data.

   I do need to move the source directory and re-import because the
data are not long-lat coordinates and also not the projection of my
project
data. When grass finished importing it told me the map(s) could not be
displayed because the north edge was undefined, and the source projection
was not the target projection. Then the grass shell froze. Killing the
process solved that.

IMHO it is a data provider issue.

Data without any information about srs (epsg code, projection, latlon or
whatever) should go back to the data Providers.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Importing-data-in-atx-gdb-format-tp5285623p5285954.html
Sent from the Grass - Users mailing list archive at Nabble.com.

On Wed, 14 Sep 2016, Markus Metz wrote:

Using CLI, what is the output of v.in.ogr
input=~/data/grassdata/or-trans/a00000001.gdbtable -l ?

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000001.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000001.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_SystemCatalog

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000002.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000002.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_DBTune

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000003.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000003.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_SpatialRefs

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000004.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000004.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_Items

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000005.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000005.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_ItemRelationships

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000006.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000006.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_ItemRelationshipTypes

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000007.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000007.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_ItemTypes

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000009.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000009.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
OR_Trans_for_Public_Distribution_8_2014

This tells you only that the coordinate reference system of the input
data is unknown. It might be Lat/Lon or something else. You can try to
1) import the data into a Lat/Lon location, reproject to the target
location and check if the geolocation is ok

   Tried this without success.

2) ask the data provider about the coordinate reference system for
this data source

   According to the metadata.xml file: the projection is Lambert Conformal Conic stdparll-1 43.000000
stdparll-2 45.500000
longitude cm -120.500000
latprj 41.750000
   ...
cell res. 0.004096m (1 International feet)
North American Datum of 1983

   Which is very close to, but not exactly the same standard paralles as the
project location.

Rich

On Wed, 14 Sep 2016, Helmut Kudrnovsky wrote:

IMHO it is a data provider issue.
Data without any information about srs (epsg code, projection, latlon or
whatever) should go back to the data Providers.

Helmut,

   The metadata.xml file has projection information but that's apprently not
seen by ogr when importing the data. See my response to Markus for details.

Rich

Rich Shepard wrote

On Wed, 14 Sep 2016, Helmut Kudrnovsky wrote:

IMHO it is a data provider issue.
Data without any information about srs (epsg code, projection, latlon or
whatever) should go back to the data Providers.

Helmut,

   The metadata.xml file has projection information but that's apprently
not
seen by ogr when importing the data. See my response to Markus for
details.

and why don't you try

ogrinfo -al your_directory_holds_your_atx_stuff

to see what ogrinfo tells you about srs (without guessing what a xml file
tells) and post the relevant parts about the srs here?

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Importing-data-in-atx-gdb-format-tp5285623p5285961.html
Sent from the Grass - Users mailing list archive at Nabble.com.

On Wed, 14 Sep 2016, Helmut Kudrnovsky wrote:

and why don't you try

ogrinfo -al your_directory_holds_your_atx_stuff

to see what ogrinfo tells you about srs (without guessing what a xml file
tells) and post the relevant parts about the srs here?

Helmut,

   Now that produces really interesting output:

$ ogrinfo -al .
FAILURE:
Unable to open datasource .' with the following drivers.
  ...
   -> OpenFileGDB
  ...

which suggests to my naive mind the souce tarball is missing a projection
file.

Great suggestion,

Rich

On Wed, 14 Sep 2016, Rich Shepard wrote:

$ ogrinfo -al .
FAILURE:
Unable to open datasource .' with the following drivers.

   OK. It doesn't like the period. The real output requires the fully
qualified path name. Here's the real output:

Layer name: OR_Trans_for_Public_Distribution_8_2014
Geometry: Multi Line String
Feature Count: 734092
Extent: (222969.013780, 88605.329396) - (2302604.888780, 1652481.930118)
Layer SRS WKT:
PROJCS["NAD83 / Oregon Lambert (ft)",
     GEOGCS["NAD83",
         DATUM["North_American_Datum_1983",
             SPHEROID["GRS 1980",6378137,298.257222101,
                 AUTHORITY["EPSG","7019"]],
             TOWGS84[0,0,0,0,0,0,0],
             AUTHORITY["EPSG","6269"]],
         PRIMEM["Greenwich",0,
             AUTHORITY["EPSG","8901"]],
         UNIT["degree",0.0174532925199433,
             AUTHORITY["EPSG","9122"]],
         AUTHORITY["EPSG","4269"]],
     PROJECTION["Lambert_Conformal_Conic_2SP"],
     PARAMETER["standard_parallel_1",43],
     PARAMETER["standard_parallel_2",45.5],
     PARAMETER["latitude_of_origin",41.75],
     PARAMETER["central_meridian",-120.5],
     PARAMETER["false_easting",1312335.958],
     PARAMETER["false_northing",0],
     UNIT["foot",0.3048,
         AUTHORITY["EPSG","9002"]],
     AXIS["X",EAST],
     AXIS["Y",NORTH],
     AUTHORITY["EPSG","2992"]]

   So, I'll create the location with that EPSG number.

Thanks,

Rich

$ ogrinfo -al .
FAILURE:
Unable to open datasource .' with the following drivers.

"." is this really your directory which holds the filegdb data?

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Importing-data-in-atx-gdb-format-tp5285623p5285969.html
Sent from the Grass - Users mailing list archive at Nabble.com.

On Wed, Sep 14, 2016 at 10:39 PM, Rich Shepard <rshepard@appl-ecosys.com> wrote:

On Wed, 14 Sep 2016, Markus Metz wrote:

Using CLI, what is the output of v.in.ogr
input=~/data/grassdata/or-trans/a00000001.gdbtable -l ?

Getting closer...

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000001.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000001.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_SystemCatalog

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000002.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000002.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_DBTune

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000003.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000003.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_SpatialRefs

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000004.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000004.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_Items

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000005.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000005.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_ItemRelationships

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000006.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000006.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_ItemRelationshipTypes

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000007.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000007.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
GDB_ItemTypes

v.in.ogr input=~/data/grassdata/ODOT2014.gdb/a00000009.gdbtable -l

Data source </home/rshepard/data/grassdata/ODOT2014.gdb/a00000009.gdbtable>
(format 'OpenFileGDB') contains 1 layers:
OR_Trans_for_Public_Distribution_8_2014

The output of v.in.ogr input=~/data/grassdata/ODOT2014.gdb -l would be
GDB_SystemCatalog
GDB_DBTune
GDB_SpatialRefs <--- interesting
GDB_Items
GDB_ItemRelationships
GDB_ItemRelationshipTypes
GDB_ItemTypes
OR_Trans_for_Public_Distribution_8_2014

can you confirm?

This tells you only that the coordinate reference system of the input
data is unknown. It might be Lat/Lon or something else. You can try to
1) import the data into a Lat/Lon location, reproject to the target
location and check if the geolocation is ok

  Tried this without success.

2) ask the data provider about the coordinate reference system for
this data source

  According to the metadata.xml file: the projection is Lambert Conformal
Conic stdparll-1 43.000000
stdparll-2 45.500000
longitude cm -120.500000
latprj 41.750000
  ...
cell res. 0.004096m (1 International feet)
North American Datum of 1983

  Which is very close to, but not exactly the same standard paralles as the
project location.

Try to create a location with the metadata.xml info, or better with
the WKT description you obtained, then import the needed layers to
this location, finally reproject the data to the target location,
check geolocation.

Markus M

On Wed, 14 Sep 2016, Helmut Kudrnovsky wrote:

"." is this really your directory which holds the filegdb data?

Helmut,

   Bash recognizes this as cwd (current working directory). However, ...

   Here's a summary of what I did so that others can gain from my learning
and import a OFGDB data file on the first try.

   0. Run ogrinfo -al on the source directory and direct output to a text
file. In my case it produced a 2.4G output file. Read from the top (less
output.txt | less) the projection information, including the EPSG projection
code.

   1. With the data source in a temporary directory I started grass73 and
created a new location: ODOT2014.gdb using the proper EPSG projection code
(and sub-variety number) and a new mapset, data_source.gdb. Exited grass

   2. Copied (could have moved) all data files from their temporary
directory to ~/data/grassdata/ODOT2014.gdb/data_source.gdb. Restarted grass.

   3. Using the GUI: File -> Import vector (v.in.ogr). Select source as
Directory, format as OpenFileGDB, point to
~/data/grassdata/ODOT2014.gdb/data_source.gdb, and click the "Import"
button.

   4. Make a pot of coffee. Return to computer and see all the roads
(highways, roads, streets) in the state displayed in the Map Display window.

Many thanks again to everyone for your patience and helpful responses,

Rich

PROJCS["NAD83 / Oregon Lambert (ft)",
   GEOGCS["NAD83",

> DATUM["North_American_Datum_1983",

            SPHEROID["GRS 1980",

you have to find the correct EPSG code related to this infomations.

the simpliest task is to ask your data provider for the correct EPSG code of
their data.

the less simple task is to guess the related informations from hints to find
e.g. in www.epsg.io

HTH

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Importing-data-in-atx-gdb-format-tp5285623p5285989.html
Sent from the Grass - Users mailing list archive at Nabble.com.

Le mercredi 14 septembre 2016 23:56:40, Helmut Kudrnovsky a écrit :

>PROJCS["NAD83 / Oregon Lambert (ft)",
>
> GEOGCS["NAD83",
>
> DATUM["North_American_Datum_1983",
>
> SPHEROID["GRS 1980",

you have to find the correct EPSG code related to this infomations.

the simpliest task is to ask your data provider for the correct EPSG code
of their data.

No need for that. The EPSG code (there are several ones in a WKT strig, but
the main one, the one that applies to the top-level "node") is always at the
end of the WKT string :

[...]
    UNIT["foot",0.3048,
        AUTHORITY["EPSG","9002"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    AUTHORITY["EPSG","2992"]] <----- here it is !

--
Spatialys - Geospatial professional services
http://www.spatialys.com

Le mercredi 14 septembre 2016 23:47:57, Rich Shepard a écrit :

On Wed, 14 Sep 2016, Helmut Kudrnovsky wrote:
> "." is this really your directory which holds the filegdb data?

Helmut,

   Bash recognizes this as cwd (current working directory). However, ...

I've just added support in the OGR FileGDB & OpenFileGDB drivers for '.' as a
valid dataset name (the resolved directory of '.' must still end up with .gdb)

   Here's a summary of what I did so that others can gain from my learning
and import a OFGDB data file on the first try.

   0. Run ogrinfo -al on the source directory and direct output to a text
file. In my case it produced a 2.4G output file. Read from the top (less
output.txt | less) the projection information, including the EPSG
projection code.

You can use ogrinfo -al -so (-so means Summary Only) to avoid listing the
features and get only projection info, feature count and field definitions. Will
save you a few gigabytes.

   1. With the data source in a temporary directory I started grass73 and
created a new location: ODOT2014.gdb using the proper EPSG projection code
(and sub-variety number) and a new mapset, data_source.gdb. Exited grass

   2. Copied (could have moved) all data files from their temporary
directory to ~/data/grassdata/ODOT2014.gdb/data_source.gdb. Restarted
grass.

   3. Using the GUI: File -> Import vector (v.in.ogr). Select source as
Directory, format as OpenFileGDB, point to
~/data/grassdata/ODOT2014.gdb/data_source.gdb, and click the "Import"
button.

   4. Make a pot of coffee. Return to computer and see all the roads
(highways, roads, streets) in the state displayed in the Map Display
window.

Many thanks again to everyone for your patience and helpful responses,

Rich

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

--
Spatialys - Geospatial professional services
http://www.spatialys.com

On Thu, 15 Sep 2016, Even Rouault wrote:

I've just added support in the OGR FileGDB & OpenFileGDB drivers for '.' as a
valid dataset name (the resolved directory of '.' must still end up with .gdb)

Even,

   Thank you. For some of us ... me, at least ... using '.' to represent the
pwd is automatic.

You can use ogrinfo -al -so (-so means Summary Only) to avoid listing the
features and get only projection info, feature count and field definitions. Will
save you a few gigabytes.

   Now this is really good to know. Of course I then lose the coffee break
wating for the output. :slight_smile:

Rich

On Thu, 15 Sep 2016, Even Rouault wrote:

You can use ogrinfo -al -so (-so means Summary Only) to avoid listing the
features and get only projection info, feature count and field
definitions. Will save you a few gigabytes.

   I just tried to import a different transportation map in OpenFileGDB
without success. The location is the same as the previous map so I let grass
create a new mapset, moved the source files there, restarted grass, and was
told there was no layers selected. That's because none were displayed from
the source .gdb directory.

   From the command line 'ogrinfo -al -so' fails; it cannot open any file
regardless of type. Yet, the directory contains the same file names as the
previous transportation map: *.atx, *.gdb*. The metadata.xml file has no
EPSG number listed so I used the same 2992/mod 6 as with the other map.

   Can someone suggest a diagnostic procedure to identify what's different
with this data set?

TIA,

Rich

   I just tried to import a different transportation map in OpenFileGDB
without success. The location is the same as the previous map so I let
grass create a new mapset, moved the source files there, restarted grass,
and was told there was no layers selected. That's because none were
displayed from the source .gdb directory.

   From the command line 'ogrinfo -al -so' fails;

What do you exactly mean by "fail" ? Does it report something like

FAILURE:
Unable to open datasource `foo.gdb' with the following drivers.
  -> PCIDSK
  -> netCDF
  -> JP2KAK

or

INFO: Open of `foo.gdb'
      using driver `OpenFileGDB' successful.

but no layer are reported ?

   it cannot open any file
regardless of type. Yet, the directory contains the same file names as the
previous transportation map: *.atx, *.gdb*. The metadata.xml file has no
EPSG number listed so I used the same 2992/mod 6 as with the other map.

(Side note: metadata.xml is not something required in a file geodatabase, so
this is completely ignored by the driver. Must be something specific to your
data provider. The projection info used by the driver is stored in one of the
"system" .gdb tables)

   Can someone suggest a diagnostic procedure to identify what's different
with this data set?

Do you get error messages ? Otherwise adding "--debug on" (double dash before
debug) to the ogrinfo command line might give extra messages. Otherwise you
could share the dataset or a link to it, and could have a look.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

The location is the same as the previous map so I let grass
create a new mapset, moved the source files there, restarted grass, and was
told there was no layers selected.

what do you mean by "let grass create a new mapset, moved the source files
there"?

moving the filegdb directory to a mapset directory?
or just v.in.ogr input=your_filegdb_data within a separate mapset?

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Importing-data-in-atx-gdb-format-tp5285623p5286172.html
Sent from the Grass - Users mailing list archive at Nabble.com.