[GRASS-dev] erratic behavior of v.in.ascii with WinGRASS

GRASS GIS GRASS GIS developer
1

I’ve run into a problem running v.in.ascii for WinGRASS importing an ASCII points file. Sometimes it works OK. Other times, it causes a Windows dbf.exe error but still gets the job done. In most cases, it causes a dbf.exe error and only imports the first point–with a coor error.

Here are a couple of other pieces of information.

It worked flawlessly on the Windows machine of a student in my lab, when she imported points (with decimal degrees coordinates) into a latlon region.

If the cat column is specified, it causes a Windows dbf.exe error on all machines importing into a UTM/State Plane location. Sometimes only the first point is imported and other cases all points are imported. When the points are correctly imported, the dbf file is correctly built.

If the cat column is NOT specified, it works fine on some machines. On others it causes a wish error and crashes the GUI (and hence GRASS). But all the points are imported correctly. [In fact it crashes the TclTk GUI on my MacBook when I upload points, but doesn’t crash it on at least some other Macs]. The associated dbf file is correctly built.

I’ve tested this in a classroom lab setting and on 2 different student machines, using data from the North Carolina demo set (schools_lu.txt), data from Spearfish, and data from elsewhere. I initially thought that this might be a result of an installation error, but given these results, it seems more like a problem with v.in.ascii. The fact that it crashes the GUI on my MacBook is especially suspicious in this regard.

Michael

Michael Barton, Professor

Professor of Anthropology
Director of Graduate Studies
School of Human Diversity & Social Change
Center for Social Dynamics & Complexity

Arizona State University

Tempe, AZ 85287-2402

USA

voice: 480-965-6262; fax: 480-965-7671

www: http://www.public.asu.edu/~cmbarton

2

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

3

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

4

On Feb 4, 2008, at 9:45 PM, Michael Barton wrote:

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

that file was actually output from GRASS - this is the command:

r.what -f elevation,landuse96_28m null=-9999 < schools.txt > schools_el_lu.txt

the original file is here (I assume that one works):
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools.txt

Apparently I haven't tried to import it. It was probably generated on Mac.
I can try it out and report back,

Helena

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

5

Yet additional information.

The file is messy because not all records have the same number of fields. Here is an example of 2 records.

628658.81149001|226880.93032087|9|-9999|-9999
636555.15570527|224580.64987627|10|125.6878585815||4|Managed Herbaceous Cover

In fact these *should* be

628658.81149001|226880.93032087|9|-9999|-9999|
636555.15570527|224580.64987627|10|125.6878585815|4|Managed Herbaceous Cover

My Mac will import the file in its original form, but crashes the TclTk when I use v.in.ascii from the GUI dialog. Another student's Mac imported the file with no problem.

If I clean it up so that it is formatted like the 2nd pair of records above, it imports on my Mac fine.

That said...

1) v.in.ascii should not crash any system if it encounters a bad file. It should exit with an error message
2) v.in.ascii in WinGRASS also crashed with a clean file on another student's computer today. It was a small one of 5 points. So there is still a problem under Windows.

I tried running it from the command line with the messy original file. It did NOT import the points and output an error message to the terminal...

Scanning input for column types...
Maximum input row length: 89
Maximum number of columns: 1
Minimum number of columns: 1
ERROR: y column number > minimum last column number
        (incorrect field separator?)

So... why does v.in.ascii go ahead with importing the points when run through the TclTk GUI when it does not import the points when run from the command line? The TclTk GUI has pretty robust error trapping now. So why does v.in.ascii bring it down? Could it be the 'formatting' of the error message? Something else going on internally so that it's not getting a message back to itself to NOT continue with importing?

Michael
____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

On Feb 4, 2008, at 8:23 PM, Helena Mitasova wrote:

On Feb 4, 2008, at 9:45 PM, Michael Barton wrote:

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

that file was actually output from GRASS - this is the command:

r.what -f elevation,landuse96_28m null=-9999 < schools.txt > schools_el_lu.txt

the original file is here (I assume that one works):
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools.txt

Apparently I haven't tried to import it. It was probably generated on Mac.
I can try it out and report back,

Helena

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
grass-dev Info Page

6

Michael,

my Mac definitely does not like fs=|
it imports it correctly without it (as it i the default separator)
or when it is "|"

Helena

g.region rast=elevation
  v.in.ascii schools_el_lu.txt output=schools_lutest format=point fs=| skip=0 x=1 y=2 z=0 cat=3
Scanning input for column types ...
Maximum input row length: 89
Maximum number of columns: 1
Minimum number of columns: 1
ERROR: y column number > minimum last column number
        (incorrect field separator?)

v.in.ascii schools_el_lu.txt output=schools_lutest format=point fs="|" skip=0 x=1 y=2 z=0 cat=3
Scanning input for column types ...
Maximum input row length: 89
Maximum number of columns: 7
Minimum number of columns: 5
Importing points ...
Populating table...
Building topology ...
167 primitives registered
Building areas: 100%
0 areas built
0 isles built
Attaching islands:
Attaching centroids: 100%
Topology was built.
Number of nodes : 167
Number of primitives: 167
Number of points : 167
Number of lines : 0
Number of boundaries: 0
Number of centroids : 0
Number of areas : 0
Number of isles : 0
v.in.ascii complete.

On Feb 4, 2008, at 10:23 PM, Helena Mitasova wrote:

On Feb 4, 2008, at 9:45 PM, Michael Barton wrote:

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

that file was actually output from GRASS - this is the command:

r.what -f elevation,landuse96_28m null=-9999 < schools.txt > schools_el_lu.txt

the original file is here (I assume that one works):
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools.txt

Apparently I haven't tried to import it. It was probably generated on Mac.
I can try it out and report back,

Helena

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

7

I just emailed you the response - it is the fs=|
try to skip it to see what happens. It does not seem to have anything with the data in the file

Helena

Helena Mitasova
Associate Professor
Department of Marine, Earth
and Atmospheric Sciences
1125 Jordan Hall, Campus Box 8208
North Carolina State University
Raleigh NC 27695-8208
http://skagit.meas.ncsu.edu/~helena/

On Feb 4, 2008, at 10:46 PM, Michael Barton wrote:

Yet additional information.

The file is messy because not all records have the same number of fields. Here is an example of 2 records.

628658.81149001|226880.93032087|9|-9999|-9999
636555.15570527|224580.64987627|10|125.6878585815||4|Managed Herbaceous Cover

In fact these *should* be

628658.81149001|226880.93032087|9|-9999|-9999|
636555.15570527|224580.64987627|10|125.6878585815|4|Managed Herbaceous Cover

My Mac will import the file in its original form, but crashes the TclTk when I use v.in.ascii from the GUI dialog. Another student's Mac imported the file with no problem.

If I clean it up so that it is formatted like the 2nd pair of records above, it imports on my Mac fine.

That said...

1) v.in.ascii should not crash any system if it encounters a bad file. It should exit with an error message
2) v.in.ascii in WinGRASS also crashed with a clean file on another student's computer today. It was a small one of 5 points. So there is still a problem under Windows.

I tried running it from the command line with the messy original file. It did NOT import the points and output an error message to the terminal...

Scanning input for column types...
Maximum input row length: 89
Maximum number of columns: 1
Minimum number of columns: 1
ERROR: y column number > minimum last column number
       (incorrect field separator?)

So... why does v.in.ascii go ahead with importing the points when run through the TclTk GUI when it does not import the points when run from the command line? The TclTk GUI has pretty robust error trapping now. So why does v.in.ascii bring it down? Could it be the 'formatting' of the error message? Something else going on internally so that it's not getting a message back to itself to NOT continue with importing?

Michael
____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

On Feb 4, 2008, at 8:23 PM, Helena Mitasova wrote:

On Feb 4, 2008, at 9:45 PM, Michael Barton wrote:

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

that file was actually output from GRASS - this is the command:

r.what -f elevation,landuse96_28m null=-9999 < schools.txt > schools_el_lu.txt

the original file is here (I assume that one works):
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools.txt

Apparently I haven't tried to import it. It was probably generated on Mac.
I can try it out and report back,

Helena

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

8

I did few more experiments (with a different file, but still
generated by GRASS - this time using v.out.ascii)
and it indeed looks like the fs=| is responsible
for the truly erratic behavior - I think that without quotes it is represented as
pipe(?) so depending on where you put it you get different response:

at the end of the command:
GRASS 6.3.cvs (nc_spm_sed):~/grassdata/indata/ncdata > v.in.ascii ftest.txt out=test2 fs=|
>
and nothing happens

or in the middle, before output (it never gets to it)

GRASS 6.3.cvs (nc_spm_sed):~/grassdata/indata/ncdata > v.in.ascii ftest.txt fs=| out=test2

ERROR: Required parameter <output> not set:
     (Name for output vector map).

I am wondering whether there are other commands that would need to have this fixed
(at least on man page).
fs=| is default so it is probably rarely used (at least on command line), but apparently
in needs quotes

Helena

On Feb 4, 2008, at 10:46 PM, Michael Barton wrote:

Yet additional information.

The file is messy because not all records have the same number of fields. Here is an example of 2 records.

628658.81149001|226880.93032087|9|-9999|-9999
636555.15570527|224580.64987627|10|125.6878585815||4|Managed Herbaceous Cover

In fact these *should* be

628658.81149001|226880.93032087|9|-9999|-9999|
636555.15570527|224580.64987627|10|125.6878585815|4|Managed Herbaceous Cover

My Mac will import the file in its original form, but crashes the TclTk when I use v.in.ascii from the GUI dialog. Another student's Mac imported the file with no problem.

If I clean it up so that it is formatted like the 2nd pair of records above, it imports on my Mac fine.

That said...

1) v.in.ascii should not crash any system if it encounters a bad file. It should exit with an error message
2) v.in.ascii in WinGRASS also crashed with a clean file on another student's computer today. It was a small one of 5 points. So there is still a problem under Windows.

I tried running it from the command line with the messy original file. It did NOT import the points and output an error message to the terminal...

Scanning input for column types...
Maximum input row length: 89
Maximum number of columns: 1
Minimum number of columns: 1
ERROR: y column number > minimum last column number
       (incorrect field separator?)

So... why does v.in.ascii go ahead with importing the points when run through the TclTk GUI when it does not import the points when run from the command line? The TclTk GUI has pretty robust error trapping now. So why does v.in.ascii bring it down? Could it be the 'formatting' of the error message? Something else going on internally so that it's not getting a message back to itself to NOT continue with importing?

Michael
____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

On Feb 4, 2008, at 8:23 PM, Helena Mitasova wrote:

On Feb 4, 2008, at 9:45 PM, Michael Barton wrote:

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

that file was actually output from GRASS - this is the command:

r.what -f elevation,landuse96_28m null=-9999 < schools.txt > schools_el_lu.txt

the original file is here (I assume that one works):
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools.txt

Apparently I haven't tried to import it. It was probably generated on Mac.
I can try it out and report back,

Helena

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

9

Hi Helena,

It looks like the '|' is partly to blame, but not entirely.

If I use the original file with missmatched numbers of fields between records AND use the '|' separator, I have trouble on my Mac.

However, if I fix the file so that all records have the same number of fields (keeping '|' as separator) OR I leave the file with messed up fields but replace '|' with ',' it works fine.

So it seems more a problem of reading the '|' separator in the text file in some circumstances than it does having it in the command string.

I'll have to try this out with Windows and see if switching to commas fixes things.

Michael
____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

On Feb 4, 2008, at 9:22 PM, Helena Mitasova wrote:

I did few more experiments (with a different file, but still
generated by GRASS - this time using v.out.ascii)
and it indeed looks like the fs=| is responsible
for the truly erratic behavior - I think that without quotes it is represented as
pipe(?) so depending on where you put it you get different response:

at the end of the command:
GRASS 6.3.cvs (nc_spm_sed):~/grassdata/indata/ncdata > v.in.ascii ftest.txt out=test2 fs=|
>
and nothing happens

or in the middle, before output (it never gets to it)

GRASS 6.3.cvs (nc_spm_sed):~/grassdata/indata/ncdata > v.in.ascii ftest.txt fs=| out=test2

ERROR: Required parameter <output> not set:
    (Name for output vector map).

I am wondering whether there are other commands that would need to have this fixed
(at least on man page).
fs=| is default so it is probably rarely used (at least on command line), but apparently
in needs quotes

Helena

On Feb 4, 2008, at 10:46 PM, Michael Barton wrote:

Yet additional information.

The file is messy because not all records have the same number of fields. Here is an example of 2 records.

628658.81149001|226880.93032087|9|-9999|-9999
636555.15570527|224580.64987627|10|125.6878585815||4|Managed Herbaceous Cover

In fact these *should* be

628658.81149001|226880.93032087|9|-9999|-9999|
636555.15570527|224580.64987627|10|125.6878585815|4|Managed Herbaceous Cover

My Mac will import the file in its original form, but crashes the TclTk when I use v.in.ascii from the GUI dialog. Another student's Mac imported the file with no problem.

If I clean it up so that it is formatted like the 2nd pair of records above, it imports on my Mac fine.

That said...

1) v.in.ascii should not crash any system if it encounters a bad file. It should exit with an error message
2) v.in.ascii in WinGRASS also crashed with a clean file on another student's computer today. It was a small one of 5 points. So there is still a problem under Windows.

I tried running it from the command line with the messy original file. It did NOT import the points and output an error message to the terminal...

Scanning input for column types...
Maximum input row length: 89
Maximum number of columns: 1
Minimum number of columns: 1
ERROR: y column number > minimum last column number
       (incorrect field separator?)

So... why does v.in.ascii go ahead with importing the points when run through the TclTk GUI when it does not import the points when run from the command line? The TclTk GUI has pretty robust error trapping now. So why does v.in.ascii bring it down? Could it be the 'formatting' of the error message? Something else going on internally so that it's not getting a message back to itself to NOT continue with importing?

Michael
____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

On Feb 4, 2008, at 8:23 PM, Helena Mitasova wrote:

On Feb 4, 2008, at 9:45 PM, Michael Barton wrote:

On Feb 4, 2008, at 7:23 PM, Hamish wrote:

Michael Barton wrote:

I've run into a problem running v.in.ascii for WinGRASS importing an
ASCII points file. Sometimes it works OK. Other times, it causes a
Windows dbf.exe error but still gets the job done. In most cases, it
causes a dbf.exe error and only imports the first point--with a coor
error.

Here are a couple of other pieces of information.

can you 1) run it with 'g.gisenv DEBUG=5' and 2) make the input file
and exact command line used available.

how about if you skip DB creation with '-t'?

Hamish

We've used several files to test. But the main one is from the N. Carolina demo data set, external files for import <http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools_el_lu.txt&gt;\.

that file was actually output from GRASS - this is the command:

r.what -f elevation,landuse96_28m null=-9999 < schools.txt > schools_el_lu.txt

the original file is here (I assume that one works):
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/schools.txt

Apparently I haven't tried to import it. It was probably generated on Mac.
I can try it out and report back,

Helena

This is a messy dataset, which may be part of the cause. But OTOH, it should behave the same way on all platforms and not cause a Windows dbf.exe error.

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

I just tried it and saw it flash a malloc and dbmi error before it crashed the GUI

I'll try the debug on my Mac. I don't think that will work on Windows.

Michael

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
grass-dev Info Page

10

On 05/02/08 06:00, Michael Barton wrote:

Hi Helena,

It looks like the '|' is partly to blame, but not entirely.

If I use the original file with missmatched numbers of fields between records AND use the '|' separator, I have trouble on my Mac.

However, if I fix the file so that all records have the same number of fields (keeping '|' as separator) OR I leave the file with messed up fields but replace '|' with ',' it works fine.

So it seems more a problem of reading the '|' separator in the text file in some circumstances than it does having it in the command string.

Just to make sure: I don't think Helena meant that the '|' as separator causes any problems. It's the fact that you use

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

with fs=| where | is unquoted and thus interpreted as pipe command.

Moritz

11

Helena Mitasova wrote:

I did few more experiments (with a different file, but still
generated by GRASS - this time using v.out.ascii)
and it indeed looks like the fs=| is responsible
for the truly erratic behavior - I think that without quotes it is
represented as
pipe(?) so depending on where you put it you get different response:

at the end of the command:
GRASS 6.3.cvs (nc_spm_sed):~/grassdata/indata/ncdata > v.in.ascii
ftest.txt out=test2 fs=|
>
and nothing happens

or in the middle, before output (it never gets to it)

GRASS 6.3.cvs (nc_spm_sed):~/grassdata/indata/ncdata > v.in.ascii
ftest.txt fs=| out=test2

ERROR: Required parameter <output> not set:
     (Name for output vector map).

I am wondering whether there are other commands that would need to
have this fixed
(at least on man page).
fs=| is default so it is probably rarely used (at least on command
line), but apparently
in needs quotes

The "|" is a shell metacharacter (for both bash and cmd.exe), and
needs to be escaped or quoted if you want to use it in an argument,
e.g.:

  v.in.ascii ftest.txt fs=\| out=test2
or:
  v.in.ascii ftest.txt fs='|' out=test2
or:
  v.in.ascii ftest.txt fs="|" out=test2

[Use the last one on Windows.]

Without the backslash or quotes, the command will be parsed as:

  v.in.ascii ftest.txt fs= | out=test2

This runs v.in.ascii with the separator set to the empty string, and
its stdout redirected to the shell command "out=test2" (i.e. an
assignment).

The Tcl/Tk GUI doesn't use the shell to execute commands, so it
shouldn't be an issue there.

--
Glynn Clements <glynn@gclements.plus.com>

12

On Feb 5, 2008, at 5:44 AM, Moritz Lennert wrote:

On 05/02/08 06:00, Michael Barton wrote:

Hi Helena,
It looks like the '|' is partly to blame, but not entirely.
If I use the original file with missmatched numbers of fields between records AND use the '|' separator, I have trouble on my Mac.
However, if I fix the file so that all records have the same number of fields (keeping '|' as separator) OR I leave the file with messed up fields but replace '|' with ',' it works fine.
So it seems more a problem of reading the '|' separator in the text file in some circumstances than it does having it in the command string.

Just to make sure: I don't think Helena meant that the '|' as separator causes any problems. It's the fact that you use

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

with fs=| where | is unquoted and thus interpreted as pipe command.

Yes,

I understood. But it also looks like it's more complicated. In some circumstances fs=| works OK (i.e., without quotes) and in other circumstances it doesn't--suggesting that there also may be a problem when v.in.ascii reads the | character in the ascii file in some cases.

Given these tests and what Glynn said (subsequent post), it seems safest to always quote or escape the | character in v.in.ascii. In this regard, is there a problem with making the default separator for the g.parser section of v.in.ascii "|" instead of the current | ? Seems like this would avoid problems when v.in.ascii is run in the GUI (i.e., instead of having to change the default entry in the separator field in each case).

As an aside, the | character seems like an odd default separator for data fields in GRASS. I suppose it's a holdover from the ancient past. But the fact that this is also has meaning as a pipe seems to make it a risky separator in general. What about switching this to something else in GRASS 7?

Michael

13

On 05/02/08 16:08, Michael Barton wrote:

On Feb 5, 2008, at 5:44 AM, Moritz Lennert wrote:

On 05/02/08 06:00, Michael Barton wrote:

Hi Helena,
It looks like the '|' is partly to blame, but not entirely.
If I use the original file with missmatched numbers of fields between records AND use the '|' separator, I have trouble on my Mac.
However, if I fix the file so that all records have the same number of fields (keeping '|' as separator) OR I leave the file with messed up fields but replace '|' with ',' it works fine.
So it seems more a problem of reading the '|' separator in the text file in some circumstances than it does having it in the command string.

Just to make sure: I don't think Helena meant that the '|' as separator causes any problems. It's the fact that you use

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

with fs=| where | is unquoted and thus interpreted as pipe command.

Yes,

I understood. But it also looks like it's more complicated. In some circumstances fs=| works OK (i.e., without quotes) and in other circumstances it doesn't--suggesting that there also may be a problem when v.in.ascii reads the | character in the ascii file in some cases.

I would rather guess that depending on how the command line was formulated, i.e. where the fs=| is placed, the command runs correctly with the default values (i.e. '|' as seperator) or doesn't.

Given these tests and what Glynn said (subsequent post), it seems safest to always quote or escape the | character in v.in.ascii. In this regard, is there a problem with making the default separator for the g.parser section of v.in.ascii "|" instead of the current | ? Seems like this would avoid problems when v.in.ascii is run in the GUI (i.e., instead of having to change the default entry in the separator field in each case).

if g.parser accepts "|" as entry, why not ?

As an aside, the | character seems like an odd default separator for data fields in GRASS. I suppose it's a holdover from the ancient past. But the fact that this is also has meaning as a pipe seems to make it a risky separator in general. What about switching this to something else in GRASS 7?

I've always found '|' quite useful as it is a symbol you are pretty sure not to find in any text fields you might import. All others (e.g. ',' ';', etc) are more likely to be used in a text field.

Moritz

14

On Feb 5, 2008, at 8:48 AM, Moritz Lennert wrote:

I would rather guess that depending on how the command line was formulated, i.e. where the fs=| is placed, the command runs correctly with the default values (i.e. '|' as seperator) or doesn't.

Not in the case of my Mac gui crash. Exact same command argument order (in gui):

fs=| + different number of fields in different records (i.e., data file format bad) = crash TclTk
fs=| + same number of fields in all records (i.e., data file format good) = OK
fs="|" + different number of fields in different records (i.e., data file format bad) = OK
fs=, + different number of fields in different records (i.e., data file format bad) = OK

Windows may be a different story, however. I haven't had a chance to check that.

Michael

____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

15

On Feb 5, 2008 4:08 PM, Michael Barton <michael.barton@asu.edu> wrote:

On Feb 5, 2008, at 5:44 AM, Moritz Lennert wrote:

> On 05/02/08 06:00, Michael Barton wrote:
>> Hi Helena,
>> It looks like the '|' is partly to blame, but not entirely.
>> If I use the original file with missmatched numbers of fields
>> between records AND use the '|' separator, I have trouble on my Mac.
>> However, if I fix the file so that all records have the same
>> number of fields (keeping '|' as separator) OR I leave the file
>> with messed up fields but replace '|' with ',' it works fine.
>> So it seems more a problem of reading the '|' separator in the
>> text file in some circumstances than it does having it in the
>> command string.
>
> Just to make sure: I don't think Helena meant that the '|' as
> separator causes any problems. It's the fact that you use
>
> v.in.ascii input=/Users/cmbarton/schools_el_lu.txt
> output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3
>
> with fs=| where | is unquoted and thus interpreted as pipe command.

Yes,

I understood. But it also looks like it's more complicated. In some
circumstances fs=| works OK (i.e., without quotes) and in other
circumstances it doesn't--suggesting that there also may be a problem
when v.in.ascii reads the | character in the ascii file in some cases.

Why not adding "pipe" here:

vector/v.in.ascii/in.c
...
    fs = delim_opt->answer;
    if (strcmp(fs, "\\t") == 0)
        fs = "\t";
    if (strcmp(fs, "tab") == 0)
        fs = "\t";
    if (strcmp(fs, "space") == 0)
        fs = " ";

?

Markus

16

On 05/02/08 17:17, Michael Barton wrote:

On Feb 5, 2008, at 8:48 AM, Moritz Lennert wrote:

I would rather guess that depending on how the command line was formulated, i.e. where the fs=| is placed, the command runs correctly with the default values (i.e. '|' as seperator) or doesn't.

Not in the case of my Mac gui crash. Exact same command argument order (in gui):

fs=| + different number of fields in different records (i.e., data file format bad) = crash TclTk
fs=| + same number of fields in all records (i.e., data file format good) = OK
fs="|" + different number of fields in different records (i.e., data file format bad) = OK
fs=, + different number of fields in different records (i.e., data file format bad) = OK

As Glynn said:

The Tcl/Tk GUI doesn't use the shell to execute commands, so it
shouldn't be an issue there.

So, since all the above are done via the gui my guess seems to be totally off anyhow...

IIRC, you said that you cannot reproduce this behaviour on the command line, i.e. even the first case works on the command line ?

Moritz

17

Michael Barton wrote:

>> It looks like the '|' is partly to blame, but not entirely.
>> If I use the original file with missmatched numbers of fields
>> between records AND use the '|' separator, I have trouble on my Mac.
>> However, if I fix the file so that all records have the same
>> number of fields (keeping '|' as separator) OR I leave the file
>> with messed up fields but replace '|' with ',' it works fine.
>> So it seems more a problem of reading the '|' separator in the
>> text file in some circumstances than it does having it in the
>> command string.
>
> Just to make sure: I don't think Helena meant that the '|' as
> separator causes any problems. It's the fact that you use
>
> v.in.ascii input=/Users/cmbarton/schools_el_lu.txt
> output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3
>
> with fs=| where | is unquoted and thus interpreted as pipe command.

Yes,

I understood. But it also looks like it's more complicated. In some
circumstances fs=| works OK (i.e., without quotes) and in other
circumstances it doesn't--suggesting that there also may be a problem
when v.in.ascii reads the | character in the ascii file in some cases.

If the string is parsed by the shell, it needs to be quoted, otherwise
it doesn't.

Given these tests and what Glynn said (subsequent post), it seems
safest to always quote or escape the | character in v.in.ascii. In
this regard, is there a problem with making the default separator for
the g.parser section of v.in.ascii "|" instead of the current | ?

Yes. That would cause the quotes to be treated as part of the argument
to the fs= option.

Seems like this would avoid problems when v.in.ascii is run in the
GUI (i.e., instead of having to change the default entry in the
separator field in each case).

There shouldn't be any need for quotes when using the GUI. The GUI
should not be using the shell at all.

As an aside, the | character seems like an odd default separator for
data fields in GRASS. I suppose it's a holdover from the ancient
past. But the fact that this is also has meaning as a pipe seems to
make it a risky separator in general. What about switching this to
something else in GRASS 7?

There's only thing that's "risky" about using "|" is that certain bugs
in code which executes commands will show up rather than getting
overlooked.

If such bugs exist, they should be fixed, rather than having
individual cases worked around.

--
Glynn Clements <glynn@gclements.plus.com>

18

On Feb 6, 2008, at 10:41 AM, Glynn Clements wrote:

Michael Barton wrote:

It looks like the '|' is partly to blame, but not entirely.
If I use the original file with missmatched numbers of fields
between records AND use the '|' separator, I have trouble on my Mac.
However, if I fix the file so that all records have the same
number of fields (keeping '|' as separator) OR I leave the file
with messed up fields but replace '|' with ',' it works fine.
So it seems more a problem of reading the '|' separator in the
text file in some circumstances than it does having it in the
command string.

Just to make sure: I don't think Helena meant that the '|' as
separator causes any problems. It's the fact that you use

v.in.ascii input=/Users/cmbarton/schools_el_lu.txt
output=schools_lu format=point fs=| skip=0 x=1 y=2 z=0 cat=3

with fs=| where | is unquoted and thus interpreted as pipe command.

Yes,

I understood. But it also looks like it's more complicated. In some
circumstances fs=| works OK (i.e., without quotes) and in other
circumstances it doesn't--suggesting that there also may be a problem
when v.in.ascii reads the | character in the ascii file in some cases.

If the string is parsed by the shell, it needs to be quoted, otherwise
it doesn't.

Given these tests and what Glynn said (subsequent post), it seems
safest to always quote or escape the | character in v.in.ascii. In
this regard, is there a problem with making the default separator for
the g.parser section of v.in.ascii "|" instead of the current | ?

Yes. That would cause the quotes to be treated as part of the argument
to the fs= option.

I don't understand. Helena suggested that I put quotes around "|", which I did and which worked--from the GUI. So why not make this the default

Seems like this would avoid problems when v.in.ascii is run in the
GUI (i.e., instead of having to change the default entry in the
separator field in each case).

There shouldn't be any need for quotes when using the GUI. The GUI
should not be using the shell at all.

This is where it seems to be needed nonetheless. If I run from the GUI and don't use quotes (and the file structure is bad), it crashes TclTk. If I run from the command line, it's not running in TclTk and so TclTk doesn't crash.

As an aside, the | character seems like an odd default separator for
data fields in GRASS. I suppose it's a holdover from the ancient
past. But the fact that this is also has meaning as a pipe seems to
make it a risky separator in general. What about switching this to
something else in GRASS 7?

There's only thing that's "risky" about using "|" is that certain bugs
in code which executes commands will show up rather than getting
overlooked.

If such bugs exist, they should be fixed, rather than having
individual cases worked around.

OK. But I don't understand what needs to be fixed here (though I'd be happy to have whatever needs to be fixed, fixed). I'm hearing quote the | character and don't quote the | character. So I'm confused.

Michael

19

Moritz Lennert wrote:

>> I would rather guess that depending on how the command line was
>> formulated, i.e. where the fs=| is placed, the command runs correctly
>> with the default values (i.e. '|' as seperator) or doesn't.
>>
>
> Not in the case of my Mac gui crash. Exact same command argument order
> (in gui):
>
> fs=| + different number of fields in different records (i.e., data
> file format bad) = crash TclTk
> fs=| + same number of fields in all records (i.e., data file format
> good) = OK
> fs="|" + different number of fields in different records (i.e., data
> file format bad) = OK
> fs=, + different number of fields in different records (i.e., data
> file format bad) = OK
>

As Glynn said:
> The Tcl/Tk GUI doesn't use the shell to execute commands, so it
> shouldn't be an issue there.

So, since all the above are done via the gui my guess seems to be
totally off anyhow...

If you use quotes in the GUI, the quotes will get passed onto the
command, i.e.

  strcmp(argv[1], "fs=\"|\"") == 0

This will cause both " and | to be treated as field separators.

Note that Tcl "exec" and "open |..." only recognise | as a pipeline
command separator if it is a separate argument. They won't split fs=|
into fs= and | the way that the shell will.

IIRC, you said that you cannot reproduce this behaviour on the command
line, i.e. even the first case works on the command line ?

fs=| will never work on the command line; the shell will always parse
the | as the pipeline command separator.

--
Glynn Clements <glynn@gclements.plus.com>

20

On 06/02/08 19:03, Glynn Clements wrote:

Moritz Lennert wrote:

I would rather guess that depending on how the command line was formulated, i.e. where the fs=| is placed, the command runs correctly with the default values (i.e. '|' as seperator) or doesn't.

Not in the case of my Mac gui crash. Exact same command argument order (in gui):

fs=| + different number of fields in different records (i.e., data file format bad) = crash TclTk
fs=| + same number of fields in all records (i.e., data file format good) = OK
fs="|" + different number of fields in different records (i.e., data file format bad) = OK
fs=, + different number of fields in different records (i.e., data file format bad) = OK

As Glynn said:

The Tcl/Tk GUI doesn't use the shell to execute commands, so it
shouldn't be an issue there.

So, since all the above are done via the gui my guess seems to be totally off anyhow...

If you use quotes in the GUI, the quotes will get passed onto the
command, i.e.

  strcmp(argv[1], "fs=\"|\"") == 0

This will cause both " and | to be treated as field separators.

Note that Tcl "exec" and "open |..." only recognise | as a pipeline
command separator if it is a separate argument. They won't split fs=|
into fs= and | the way that the shell will.

IIRC, you said that you cannot reproduce this behaviour on the command line, i.e. even the first case works on the command line ?

fs=| will never work on the command line; the shell will always parse
the | as the pipeline command separator.

Sorry, yes, obviously. I meant the second part of that case, i.e. "different number of fields in different records".

Moritz