[GRASSLIST:4237] Cannot open driver pg

David_Orme · August 24, 2004, 10:00am

Hi,

I need to convert about 700 raster datasets into shapefiles - I've written a shell script that does this nicely, except that after processing about 80 of the files through to shapefile I get the following error. The script keeps running, it just fails to write out the shapefiles from then on:

Converting Zosterops_lateralis_7454
CREATING SUPPORT FILES FOR Zosterops_lateralis_7454
25 primitives registered
13 areas built %
5 isles built
Number of nodes : 17
Number of primitives: 25
Number of points : 0
Number of lines : 0
Number of boundaries: 18
Number of centroids : 7
Number of areas : 13
Number of isles : 5
Number of areas without centroid : 6
WARNING: The table <Zosterops_lateralis_7454> is now part of vector map
<Zosterops_lateralis_7454> and may be deleted or overwritten by GRASS modules.
ERROR 4: Failed to open Shapefile `./Shapefiles/Zosterops_lateralis_7454.shp'.

Too many open files: /usr/local/grass57/driver/db/
ERROR: Cannot open driver pg
REMOVE [Zosterops_lateralis_7454]
raster
header
category
color MISSING
history
misc
fcell MISSING
g3dcell MISSING

I've appended the shell script - there is a separate wrapper script which just contains a 'while read' loop to pass in the variables that I need to be added in to the attribute table and also a separate SQL template that defines the attribute table structure and then updates the entries using the provided variables. I'm using a home build of GRASS 5.7 (grass57_exp_2004_01_10) on Mac OS X 10.3.5 and using PostgreSQL 7.4.1.

I can just restart GRASS and run batches of 80 or so but it would be great to know where I'm going wrong - I obviously need to release the files in some way.

Thanks,
David

#!/bin/bash

# variables are: infile, smname, smnumber, presence, origin, source tag, sourcename, data_type
# generalize with pg_db, out_dir?

# requires a PostgreSQL dtabase called tmpgrass

if /usr/local/pgsql/bin/psql -l | grep -q "tmpgrass"
then

  # get the file basename for export
  fname=`basename $1`

  echo Converting $fname

  v.database driver=pg database=tmpgrass

  #check for output folder
  if ! ls | grep -q Shapefiles
  then
    mkdir ./Shapefiles
  fi

  r.in.arc $1 output=$fname type=CELL

  r.to.vect $fname output=$fname feature=area

  # get correct attribute table:
  # substitute species name into the sql commands into a temporary file
  tmpname=tmp.asc2grd.$$.sql
  sed -e "s/SMNAME/$2/g" -e "s/SMNUM/$3/g" -e "s/FNAME/$fname/g" -e "s/PRCODE/$4/g" -e "s/ORCODE/$5/g" -e "s/SRCTAG/$6/g" -e "s/SRCNAME/$7/g" -e "s/DATATYPE/$8/g" shpSqlTemplate.sql > $tmpname

  # and then execute the sql file
  db.execute driver=pg database=tmpgrass input=$tmpname

  # reconnect the vector data to the modified table
  v.db.connect -o map=$fname driver=pg database=tmpgrass table=$fname key=id

  # dump the shapefile version
  v.out.ogr -c $fname dsn=./Shapefiles layer=$fname type=area

  # delete the temporary raster and sql file
  g.remove rast=$fname vect=$fname
  rm $tmpname

else
  echo "PostgreSQL database tmpgrass does not exist:"
  echo " - create using createdb tmpgrass"
fi

David_Orme · August 25, 2004, 11:51am

Hi again,

Found a solution - when I used v.out.ogr in a loop and wrote directly to the final destination directory then I got the "too many files" error after the folder got to around 350 files (approx. 80 shapefiles). If I send the output of v.out.ogr to /tmp and then move the files to the final directory, the whole list of files is processed. I don't understand why this should be so if anyone can enlighten me, that'd be great.

i.e. within a loop of 700 files:

v.out.ogr -c $fname dsn=./FinalDestDir layer=$fname type=area

fails after about 80 but...

v.out.ogr -c $fname dsn=/tmp layer=$fname type=area
mv /tmp/$fname* ./FinalDestDir

...works without hitch.

Cheers,
David

On 24 Aug 2004, at 11:00, David Orme wrote:

Hi,

I need to convert about 700 raster datasets into shapefiles - I've written a shell script that does this nicely, except that after processing about 80 of the files through to shapefile I get the following error. The script keeps running, it just fails to write out the shapefiles from then on:

Converting Zosterops_lateralis_7454
CREATING SUPPORT FILES FOR Zosterops_lateralis_7454
25 primitives registered
13 areas built %
5 isles built
Number of nodes : 17
Number of primitives: 25
Number of points : 0
Number of lines : 0
Number of boundaries: 18
Number of centroids : 7
Number of areas : 13
Number of isles : 5
Number of areas without centroid : 6
WARNING: The table <Zosterops_lateralis_7454> is now part of vector map
         <Zosterops_lateralis_7454> and may be deleted or overwritten by GRASS modules.
ERROR 4: Failed to open Shapefile `./Shapefiles/Zosterops_lateralis_7454.shp'.

Too many open files: /usr/local/grass57/driver/db/
ERROR: Cannot open driver pg
REMOVE [Zosterops_lateralis_7454]
raster
header
category
color MISSING
history
misc
fcell MISSING
g3dcell MISSING

I've appended the shell script - there is a separate wrapper script which just contains a 'while read' loop to pass in the variables that I need to be added in to the attribute table and also a separate SQL template that defines the attribute table structure and then updates the entries using the provided variables. I'm using a home build of GRASS 5.7 (grass57_exp_2004_01_10) on Mac OS X 10.3.5 and using PostgreSQL 7.4.1.

I can just restart GRASS and run batches of 80 or so but it would be great to know where I'm going wrong - I obviously need to release the files in some way.

Thanks,
David

#!/bin/bash

# variables are: infile, smname, smnumber, presence, origin, source tag, sourcename, data_type
# generalize with pg_db, out_dir?

# requires a PostgreSQL dtabase called tmpgrass

if /usr/local/pgsql/bin/psql -l | grep -q "tmpgrass"
then

  # get the file basename for export
  fname=`basename $1`

  echo Converting $fname

  v.database driver=pg database=tmpgrass

  #check for output folder
  if ! ls | grep -q Shapefiles
  then
    mkdir ./Shapefiles
  fi

  r.in.arc $1 output=$fname type=CELL

  r.to.vect $fname output=$fname feature=area

  # get correct attribute table:
  # substitute species name into the sql commands into a temporary file
  tmpname=tmp.asc2grd.$$.sql
  sed -e "s/SMNAME/$2/g" -e "s/SMNUM/$3/g" -e "s/FNAME/$fname/g" -e "s/PRCODE/$4/g" -e "s/ORCODE/$5/g" -e "s/SRCTAG/$6/g" -e "s/SRCNAME/$7/g" -e "s/DATATYPE/$8/g" shpSqlTemplate.sql > $tmpname

  # and then execute the sql file
  db.execute driver=pg database=tmpgrass input=$tmpname

  # reconnect the vector data to the modified table
  v.db.connect -o map=$fname driver=pg database=tmpgrass table=$fname key=id

  # dump the shapefile version
  v.out.ogr -c $fname dsn=./Shapefiles layer=$fname type=area

  # delete the temporary raster and sql file
  g.remove rast=$fname vect=$fname
  rm $tmpname

else
  echo "PostgreSQL database tmpgrass does not exist:"
  echo " - create using createdb tmpgrass"
fi