[GRASS-user] v.in.ascii : ERROR: write sidx

GRASS GRASS user
1

Hello,

I have just installed GRASS 7.0 svn 57844 on Linux Mint 15.

v.in.ascii stopped during the build process : “ERROR: write sidx: wrong node position in file”

Like this :

Scanning input for column types…
Number of columns: 3
Importing points…
100%
Building topology for vector map PTS_menhir2_pc_88M@menhir2…
Registering primitives…
88509169 primitives registered
88509169 vertices registered
Building areas…
100%
0 areas built
0 isles built
Attaching islands…
Attaching centroids…
100%
Number of nodes: 0
Number of primitives: 88509169
Number of points: 88509169
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
ERROR: write sidx: wrong node position in file

The command was :
v.in.ascii -z input=${FILE} output=${NAME_OUT} separator=“${FS}” z=3 --overwrite

From a shell script.

Is this a known issue ?

++

yann

2

Le Jeune Yann wrote:

Hello,

I have just installed GRASS 7.0 svn 57844 on Linux Mint 15.

v.in.ascii stopped during the build process : "ERROR: write sidx: wrong node
position in file"

Like this :

Scanning input for column types...
Number of columns: 3
Importing points...
100%
Building topology for vector map <PTS_menhir2_pc_88M@menhir2>...
Registering primitives...
88509169 primitives registered
88509169 vertices registered
Building areas...
100%
0 areas built
0 isles built
Attaching islands...
Attaching centroids...
100%
Number of nodes: 0
Number of primitives: 88509169
Number of points: 88509169
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
ERROR: write sidx: wrong node position in file

If you are using a 32bit OS, you might need large file support (LFS)
which is by default enabled. LFS can but should not be disabled with
--disable-largefile.

Do you have enough free disk space?

If you do not need topology for the points, you can try v.in.ascii -b.

The command was :
v.in.ascii -z input=${FILE} output=${NAME_OUT} separator="${FS}" z=3
--overwrite

From a shell script.

Is this a known issue ?

No.

Markus M

3

If you are using a 32bit OS, you might need large file support (LFS)
which is by default enabled. LFS can but should not be disabled with
--disable-largefile.

i use a 64 bit system, with LFS

Do you have enough free disk space?

Yes

If you do not need topology for the points, you can try v.in.ascii -b.

that works for me, Thanks a lot !!

++

yann

4

Sorry, in Fact i still need topology for further processes,

Any ideas ?

Thanks

yann

···

2013/9/26 Le Jeune Yann <lj.yann@gmail.com>

–

Yann Le Jeune
Web : http://www.paleosystem.fr/ylj_pro/

If you are using a 32bit OS, you might need large file support (LFS)
which is by default enabled. LFS can but should not be disabled with
–disable-largefile.

i use a 64 bit system, with LFS

Do you have enough free disk space?

Yes

If you do not need topology for the points, you can try v.in.ascii -b.

that works for me, Thanks a lot !!

++

yann

5

On Thu, Sep 26, 2013 at 7:05 PM, Le Jeune Yann <lj.yann@gmail.com> wrote:

Sorry, in Fact i still need topology for further processes,
Any ideas ?

May be now you are able to build the topology using "v.build" module,
or do you have still a error?

Pietro

6

I tried v.build after v.in.ascii -b and there’s the same error at the same step… :frowning:

···

2013/9/26 Pietro <peter.zamb@gmail.com>

On Thu, Sep 26, 2013 at 7:05 PM, Le Jeune Yann <lj.yann@gmail.com> wrote:

Sorry, in Fact i still need topology for further processes,
Any ideas ?

May be now you are able to build the topology using “v.build” module,
or do you have still a error?

Pietro

–

Yann Le Jeune
Web : http://www.paleosystem.fr/ylj_pro/

7

Yann wrote:

The command was :
v.in.ascii -z input=${FILE} output=${NAME_OUT} separator="${FS}" z=3 --overwrite

Hi,

just to note -- the {curly} brackets do nothing to
protect from spaces in filenames in this situation,
"double" quotes should be used for that.

the curly brackets protect the variable name, not its
contents, so are mostly useful for when a letter,
number, or "_" follows the variable name.

I actually think it is dangerous to use the curly
brackets because it tricks people into thinking they
are protected when they are not.

regards,
Hamish

8

Hi,

Thanks for the {curly} thing !

I tried v.in.ascii from the Layer manager (i use sqlite driver) :

(Thu Sep 26 23:58:29 2013)
v.in.ascii -z --overwrite --verbose input=/media/yann/DATA_SIG/3D_photo/menhir2_pc_88M.csv output=PTS_menhir2_pc_88M separator= z=3
WARNING: Vector map <PTS_menhir2_pc_88M> already exists and will be overwritten
Using native format
Scanning input for column types…
Maximum input row length: 30
Number of columns: 3
Importing points…
Building topology for vector map PTS_menhir2_pc_88M@menhir2…
Registering primitives…
88509169 primitives registered
88509169 vertices registered
Building areas…
0 areas built
0 isles built
Attaching islands…
Attaching centroids…
Topology was built
Number of nodes: 0
Number of primitives: 88509169
Number of points: 88509169
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
ERROR: write sidx: wrong node position in file
(Fri Sep 27 00:44:28 2013) Command finished (45 min 59 sec)

I have some disk space, but i looked at my system monitor and my RAM went to SWAP (i have only 16 Go RAM, 15 Go SWAP) The swap don’t seems to have been totally used (but at least half) and i have not in front of my PC when the error occurred.

I will try this again with more RAM.

Thanks for your answers.

yann

···

2013/9/26 Hamish <hamish_b@yahoo.com>

Yann wrote:

The command was :
v.in.ascii -z input=${FILE} output=${NAME_OUT} separator=“${FS}” z=3 --overwrite

Hi,

just to note – the {curly} brackets do nothing to
protect from spaces in filenames in this situation,
“double” quotes should be used for that.

the curly brackets protect the variable name, not its
contents, so are mostly useful for when a letter,
number, or “_” follows the variable name.

I actually think it is dangerous to use the curly
brackets because it tricks people into thinking they
are protected when they are not.

regards,
Hamish

9

Yann wrote:

I tried v.in.ascii from the Layer manager (i use sqlite driver) :

(Thu Sep 26 23:58:29 2013)
v.in.ascii

-z --overwrite --verbose
input=/media/yann/DATA_SIG/3D_photo/menhir2_pc_88M.csv
output=PTS_menhir2_pc_88M separator= z=3

WARNING: Vector map <PTS_menhir2_pc_88M> already exists and will be overwritten
Using native format
Scanning input for column types...
Maximum input row length: 30
Number of columns: 3
Importing points...
Building topology for vector map <PTS_menhir2_pc_88M@menhir2>...
Registering primitives...
88509169 primitives registered
88509169 vertices registered
Building areas...
0 areas built
0 isles built
Attaching islands...
Attaching centroids...
Topology was built
Number of nodes: 0
Number of primitives: 88509169
Number of points: 88509169
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
ERROR: write sidx: wrong node position in file
(Fri Sep 27 00:44:28 2013) Command finished (45 min 59 sec)

I have some disk space, but i looked at my system monitor and my RAM
went to SWAP (i have only 16 Go RAM, 15 Go SWAP) The swap don't seems
to have been totally used (but at least half) and i have not in front
of my PC when the error occurred.

I will try this again with more RAM.

Ok, 88 million points and running 64bit Linux. Since 3D points
and no additional data columns columns I think it will default
to not creating a database, so you don't have to worry about the
-t flag or the memory needed for that, just the finite amount of
RAM per point for topology. Maybe 64GB RAM will be enough for 88M,
perhaps more, but it will need a lot to build topology for all those
points.

May I ask what process you'd like to do with the data? Perhaps
r.in.xyz + r.to.vect helps reduce the number of points to something
more manageable. The v.in.ascii '-r' flag can be helpful for importing
only those points within the current computational region of interest,
and if importing without building topology (the '-b' flag you tried
before) many of the LiDAR modules and v.surf.rst have been modified
to work without needing it.
If it is some other vector module maybe we can modify it to load data
without topology too.
Is it sparse x,y data or gridded?

regards,
Hamish

10

Le Jeune Yann wrote:

[...]

ERROR: write sidx: wrong node position in file
(Fri Sep 27 00:44:28 2013) Command finished (45 min 59 sec)

I have some disk space, but i looked at my system monitor and my RAM went to
SWAP (i have only 16 Go RAM, 15 Go SWAP) The swap don't seems to have been
totally used (but at least half) and i have not in front of my PC when the
error occurred.

You can try to set the environment variable GRASS_VECTOR_LOWMEM with

export GRASS_VECTOR_LOWMEM=1

This will substantially reduce memory requirements when building
topology, particularly for points. You will need a bit of free disk
space where your grassdata are located.

HTH,

Markus M

I will try this again with more RAM.

Thanks for your answers.

yann

2013/9/26 Hamish <hamish_b@yahoo.com>

Yann wrote:
> The command was :
> v.in.ascii -z input=${FILE} output=${NAME_OUT} separator="${FS}" z=3
> --overwrite

Hi,

just to note -- the {curly} brackets do nothing to
protect from spaces in filenames in this situation,
"double" quotes should be used for that.

the curly brackets protect the variable name, not its
contents, so are mostly useful for when a letter,
number, or "_" follows the variable name.

I actually think it is dangerous to use the curly
brackets because it tricks people into thinking they
are protected when they are not.

regards,
Hamish

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

11

Ok, 88 million points and running 64bit Linux. Since 3D points
and no additional data columns columns I think it will default
to not creating a database, so you don't have to worry about the
-t flag or the memory needed for that, just the finite amount of
RAM per point for topology. Maybe 64GB RAM will be enough for 88M,
perhaps more, but it will need a lot to build topology for all those
points.

Ok

May I ask what process you'd like to do with the data?

I am processing point clouds from photogrammetry (here an engraved
megalith, 6.5 m²) and LIDAR data in forested areas (archaeology at another
scale).
For this i am working on a script that perform all classical processes (for
achaeologist, including qualimetry, animations, PDF maps etc.) for all
*.csv file in a given directory. It's a work in progress, shellscript, but
i hope to do this in C and/ or in a contribution to grass if it worth it.

Perhaps
r.in.xyz + r.to.vect helps reduce the number of points to something
more manageable.

I will try this.

The v.in.ascii '-r' flag can be helpful for importing
only those points within the current computational region of interest,
and if importing without building topology (the '-b' flag you tried
before) many of the LiDAR modules and v.surf.rst have been modified
to work without needing it.

ok, i will looke at that too.

If it is some other vector module maybe we can modify it to load data
without topology too.

i use also v.neighbors

Is it sparse x,y data or gridded?

It's sparse data.

Thanks à lot !!!

yann

12

2013/9/27 Markus Metz <markus.metz.giswork@gmail.com>

You can try to set the environment variable GRASS_VECTOR_LOWMEM with

export GRASS_VECTOR_LOWMEM=1

This will substantially reduce memory requirements when building
topology, particularly for points. You will need a bit of free disk
space where your grassdata are located.

Ok, thanks !

i will try this too.

yann

13

I tried r.in.xyz and it’s perfect for me.

Thanks a lot !

yann

···

2013/9/27 Le Jeune Yann <lj.yann@gmail.com>

–

Yann Le Jeune
Web : http://www.paleosystem.fr/ylj_pro/

2013/9/27 Markus Metz <markus.metz.giswork@gmail.com>

You can try to set the environment variable GRASS_VECTOR_LOWMEM with

export GRASS_VECTOR_LOWMEM=1

This will substantially reduce memory requirements when building
topology, particularly for points. You will need a bit of free disk
space where your grassdata are located.

Ok, thanks !

i will try this too.

yann

14

Yann wrote:

> The command was :
> v.in.ascii -z input=${FILE} output=${NAME_OUT} separator="${FS}" z=3
> --overwrite

Hamish wrote:

Hi,

just to note -- the {curly} brackets do nothing to
protect from spaces in filenames in this situation,
"double" quotes should be used for that.

the curly brackets protect the variable name, not its
contents, so are mostly useful for when a letter,
number, or "_" follows the variable name.

I actually think it is dangerous to use the curly
brackets because it tricks people into thinking they
are protected when they are not.

Maybe "${FILE}" then? I remember this as being Glynn's recommendation, some
time ago...

Best, Nikos

15

On Thu, Oct 3, 2013 at 8:44 PM, Nikos Alexandris <nik@nikosalexandris.net> wrote:

Yann wrote:

The command was :
v.in.ascii -z input=${FILE} output=${NAME_OUT} separator=“${FS}” z=3
–overwrite

Hamish wrote:

Hi,

just to note – the {curly} brackets do nothing to
protect from spaces in filenames in this situation,
“double” quotes should be used for that.

the curly brackets protect the variable name, not its
contents, so are mostly useful for when a letter,
number, or “_” follows the variable name.

I actually think it is dangerous to use the curly
brackets because it tricks people into thinking they
are protected when they are not.

Maybe “${FILE}” then? I remember this as being Glynn’s recommendation, some
time ago…

Yes “${FILE}” should protect spaces and variable name.

$ MY_VARIABLE=“GRASS is GIS”
$ for A in $MY_VARIABLE; do echo $A; done;

GRASS
is
GIS
$ for A in “${MY_VARIABLE}”; do echo $A; done;
GRASS is GIS

But if I want to make thinks robust I use Python. Although module call itself is more complicated

input=file_name

is less complicated than

input=“${FILE}”

Best,
Vaclav

Best, Nikos

grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user