[GRASS5] shape file limit with v.in.org.all

GRASS GIS GRASS GIS developer
1

I have an odd script question. I am using a script v.in.ogr.all

===============================================
#!/bin/bash

#test to see if script is being run from GRASS

if test "$GISBASE" = ""; then
         echo "You must be in GRASS to run this program"
         exit
fi

for MAP in `ls ~/projects/mn_dnr/shape_files/forties/county/*.shp`; do
     v.in.ogr -o dsn=~/projects/mn_dnr/shape_files/forties/county/ output=`ba\
sename $MAP .shp` layer=`basename $MAP .shp`;

done

to import a bunch (>100) shape files. When I run the script on the whole directory of shape files, I get the error:

ERROR 4: Unable to open /Network/Servers/truffula.fr.umn.edu/Volumes/disk1/home1/kwythers/projects/mn_dnr/shape_files/forties/county/wrig_forties.shx or /Network/Servers/truffula.fr.umn.edu/Volumes/disk1/home1/kwythers/projects/mn_dnr/shape_files/forties/county/wrig_forties.SHX.

repeated for every shape file. However, if I run the same script on just a few files, the world is cool. So far I have run it on 1 file, 3 files, 6 files, and 20 files. I am about to try 50 or so files. Has anyone see this kind of behavior before?

Kirk

2

On Wed, Jan 11, 2006 at 02:02:52PM -0600, Kirk R. Wythers wrote:

I have an odd script question. I am using a script v.in.ogr.all

===============================================
#!/bin/bash

#test to see if script is being run from GRASS

if test "$GISBASE" = ""; then
        echo "You must be in GRASS to run this program"
        exit
fi

for MAP in `ls ~/projects/mn_dnr/shape_files/forties/county/*.shp`; do
    v.in.ogr -o dsn=~/projects/mn_dnr/shape_files/forties/county/
output=`ba\
sename $MAP .shp` layer=`basename $MAP .shp`;

done

to import a bunch (>100) shape files. When I run the script on the
whole directory of shape files, I get the error:

ERROR 4: Unable to open /Network/Servers/truffula.fr.umn.edu/Volumes/
disk1/home1/kwythers/projects/mn_dnr/shape_files/forties/county/
wrig_forties.shx or /Network/Servers/truffula.fr.umn.edu/Volumes/
disk1/home1/kwythers/projects/mn_dnr/shape_files/forties/county/
wrig_forties.SHX.

repeated for every shape file. However, if I run the same script on
just a few files, the world is cool. So far I have run it on 1 file,
3 files, 6 files, and 20 files. I am about to try 50 or so files. Has
anyone see this kind of behavior before?

Kirk,

I guess that the parameter length created by `ls *...` is
exceeding a Un*x limit (for the shell or whatever).
You may try to rewrite it to first go into the
directory, then run the loop in that directory.

Markus

3

On Jan 12, 2006, at 1:15 AM, Markus Neteler wrote:

On Wed, Jan 11, 2006 at 02:02:52PM -0600, Kirk R. Wythers wrote:

I have an odd script question. I am using a script v.in.ogr.all

===============================================
#!/bin/bash

#test to see if script is being run from GRASS

if test "$GISBASE" = ""; then
        echo "You must be in GRASS to run this program"
        exit
fi

for MAP in `ls ~/projects/mn_dnr/shape_files/forties/county/*.shp`; do
    v.in.ogr -o dsn=~/projects/mn_dnr/shape_files/forties/county/
output=`ba\
sename $MAP .shp` layer=`basename $MAP .shp`;

done

to import a bunch (>100) shape files. When I run the script on the
whole directory of shape files, I get the error:

ERROR 4: Unable to open /Network/Servers/truffula.fr.umn.edu/Volumes/
disk1/home1/kwythers/projects/mn_dnr/shape_files/forties/county/
wrig_forties.shx or /Network/Servers/truffula.fr.umn.edu/Volumes/
disk1/home1/kwythers/projects/mn_dnr/shape_files/forties/county/
wrig_forties.SHX.

repeated for every shape file. However, if I run the same script on
just a few files, the world is cool. So far I have run it on 1 file,
3 files, 6 files, and 20 files. I am about to try 50 or so files. Has
anyone see this kind of behavior before?

Kirk,

I guess that the parameter length created by `ls *...` is
exceeding a Un*x limit (for the shell or whatever).
You may try to rewrite it to first go into the
directory, then run the loop in that directory.

Thanks Markus. I have another large collection of shape files to read in today. I'll give it a go. I never figured out exactly what the limit was (something between 25 and 100 files, I would guess).

Markus

4

Kirk:

>> for MAP in `ls ~/projects/mn_dnr/shape_files/forties/county/
>> *.shp`; do

..

>> to import a bunch (>100) shape files. When I run the script on the
>> whole directory of shape files, I get the error:

..

>> repeated for every shape file. However, if I run the same script on
>> just a few files, the world is cool. So far I have run it on 1
>file, > 3 files, 6 files, and 20 files. I am about to try 50 or so
>files. Has > anyone see this kind of behavior before?

Markus:

> I guess that the parameter length created by `ls *...` is
> exceeding a Un*x limit (for the shell or whatever).
> You may try to rewrite it to first go into the
> directory, then run the loop in that directory.

Kirk again:

Thanks Markus. I have another large collection of shape files to read
in today. I'll give it a go. I never figured out exactly what the
limit was (something between 25 and 100 files, I would guess).

You are probably past this now, but for the greater understanding &
future scripts..

I think what Markus was getting at is that it wasn't a file limit, but a
string character limit. cd'ing to the directory will remove the
directory from each file from the `ls` string (as you do with basename
later), which means you can fit a lot more names in the 4096? chars you
have available for the shell to work with.

"~/projects/mn_dnr/shape_files/forties/county/" * 100 maps is an extra
4500 chars. That would put you over the 4096 limit. Alternatively to
"cd" first, running 'basename .shp' in the `ls` step would mean a
savings of an extra 4 chars per map, which may help for a huge number of
maps.

hope it all makes sense now,

Hamish