[GRASS-user] v.rast.stats error: "Unable to seek"

Dear all,

I am getting the error “Unable to seek” with v.stats.error. There is not much information that could point the cause, just a warning saying that some data base files are not found. I checked the database connection and everything looks in order (see below). Any hints on what may be causing this error?

Thank you.

v.rast.stats map=PSU raster=MAL_Mode_5x5 method=number column_prefix=mal

ERROR: Unable to seek: Invalid argument

ERROR: An error occurred while converting vector to raster

WARNING: No data base element files found

db.connect -p

driver: pg

database: s4a

schema: mal

group:

psql -d s4a -p 5434

psql (12.5 (Ubuntu 12.5-0ubuntu0.20.04.1))

Type “help” for help.

s4a=# \dt mal.*

List of relations

Schema | Name | Type | Owner

···

Luís

Hi Micha, thank you for replying.

I recreated the PSU layer in another mapset with the default SQLite back-end, v.rast.stats fails with the exact same error. So this is not related to the back-end.

···

Below are the outputs you requested. Thank you.

v.info PSU

±---------------------------------------------------------------------------+

| Name: PSU |

| Mapset: MAL |

| Location: S4AHomolosine |

| Database: /home/duque004/Work/GRASSDATA |

| Title: |

| Map scale: 1:1 |

| Name of creator: duque004 |

| Organization: |

| Source date: Fri Jan 29 08:13:08 2021 |

| Timestamp (first layer): none |

|----------------------------------------------------------------------------|

| Map format: native |

|----------------------------------------------------------------------------|

| Type of map: vector (level: 2) |

| |

| Number of points: 0 Number of centroids: 19468516 |

| Number of lines: 0 Number of boundaries: 38945853 |

| Number of areas: 19468516 Number of islands: 1 |

| |

| Map is 3D: No |

| Number of dblinks: 1 |

| |

| Projection: unknown |

| |

| N: 4289569.27224353 S: -4452930.72775647 |

| E: 6679312.25515029 W: -2226387.74484971 |

| |

| Digitization threshold: 0 |

| Comment: |

| |

±---------------------------------------------------------------------------+

r.info MAL_Mode_5x5

±---------------------------------------------------------------------------+

| Map: MAL_Mode_5x5 Date: Thu Jan 28 09:08:14 2021 |

| Mapset: MAL Login of Creator: duque004 |

| Location: S4AHomolosine |

| DataBase: /home/duque004/Work/GRASSDATA |

| Title: 5x5 neighborhood: mode of MAL_AFRICA |

| Timestamp: none |

|----------------------------------------------------------------------------|

| |

| Type of Map: raster Number of Categories: 0 |

| Data Type: CELL |

| Rows: 87425 |

| Columns: 89057 |

| Total Cells: 7785808225 |

| Projection: unknown |

| N: 4289569.27224353 S: -4452930.72775647 Res: 100 |

| E: 6679312.25515029 W: -2226387.74484971 Res: 100 |

| Range of data: min = 0 max = 1 |

| |

| Data Description: |

| generated by r.neighbors |

| |

| Comments: |

| r.neighbors input=“MAL_AFRICA” output=“MAL_Mode_5x5” method=“mode” s\ |

| ize=5 |

| |

±---------------------------------------------------------------------------+

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Friday, January 29, 2021 3:31 PM, Micha Silver tsvibar@gmail.com wrote:

On Fri, Jan 29, 2021 at 3:53 PM Luí­s Moreira de Sousa via grass-user <grass-user@lists.osgeo.org> wrote:

Dear all,

I am getting the error “Unable to seek” with v.stats.error. There is not much information that could point the cause, just a warning saying that some data base files are not found. I checked the database connection and everything looks in order (see below). Any hints on what may be causing this error?

Thank you.

v.rast.stats map=PSU raster=MAL_Mode_5x5 method=number column_prefix=mal

ERROR: Unable to seek: Invalid argument

ERROR: An error occurred while converting vector to raster

WARNING: No data base element files found

db.connect -p

driver: pg

database: s4a

schema: mal

group:

psql -d s4a -p 5434

psql (12.5 (Ubuntu 12.5-0ubuntu0.20.04.1))

Type “help” for help.

s4a=# \dt mal.*

List of relations

Schema | Name | Type | Owner

--------±-------------------------±------±---------

mal | psu | table | duque004

(1 rows)

Could it be simply CAPS in the vector name vs small letters in the Postgres table?

Can you post the outputs from:

v.info PSU

r.info MAL_Mode_5x5

s4a=# select count(*) from mal.psu;

count


19468516

(1 row)

Luís


grass-user mailing list

grass-user@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/grass-user

Micha Silver

Ben Gurion Univ

Sde-Boker Remote Sensing Lab

cell: +972 (52) 3665918

Luís

Sent with ProtonMail Secure Email.

Does the example in v.rast.stats manual [1] work for you, Luis? Is your polygon vector map “healthy”? The error you get seems to be while converting the vector to raster, hence I wonder if boundaries of all areas in your vector are right.

my 0.02 cents
Vero

[1] https://grass.osgeo.org/grass78/manuals/v.rast.stats.html

El lun, 1 feb 2021 a las 16:31, Luí­s Moreira de Sousa via grass-user (<grass-user@lists.osgeo.org>) escribió:

Hi Micha, thank you for replying.

I recreated the PSU layer in another mapset with the default SQLite back-end, v.rast.stats fails with the exact same error. So this is not related to the back-end.

Below are the outputs you requested. Thank you.

v.info PSU

±---------------------------------------------------------------------------+

| Name: PSU |

| Mapset: MAL |

| Location: S4AHomolosine |

| Database: /home/duque004/Work/GRASSDATA |

| Title: |

| Map scale: 1:1 |

| Name of creator: duque004 |

| Organization: |

| Source date: Fri Jan 29 08:13:08 2021 |

| Timestamp (first layer): none |

|----------------------------------------------------------------------------|

| Map format: native |

|----------------------------------------------------------------------------|

| Type of map: vector (level: 2) |

| |

| Number of points: 0 Number of centroids: 19468516 |

| Number of lines: 0 Number of boundaries: 38945853 |

| Number of areas: 19468516 Number of islands: 1 |

| |

| Map is 3D: No |

| Number of dblinks: 1 |

| |

| Projection: unknown |

| |

| N: 4289569.27224353 S: -4452930.72775647 |

| E: 6679312.25515029 W: -2226387.74484971 |

| |

| Digitization threshold: 0 |

| Comment: |

| |

±---------------------------------------------------------------------------+

r.info MAL_Mode_5x5

±---------------------------------------------------------------------------+

| Map: MAL_Mode_5x5 Date: Thu Jan 28 09:08:14 2021 |

| Mapset: MAL Login of Creator: duque004 |

| Location: S4AHomolosine |

| DataBase: /home/duque004/Work/GRASSDATA |

| Title: 5x5 neighborhood: mode of MAL_AFRICA |

| Timestamp: none |

|----------------------------------------------------------------------------|

| |

| Type of Map: raster Number of Categories: 0 |

| Data Type: CELL |

| Rows: 87425 |

| Columns: 89057 |

| Total Cells: 7785808225 |

| Projection: unknown |

| N: 4289569.27224353 S: -4452930.72775647 Res: 100 |

| E: 6679312.25515029 W: -2226387.74484971 Res: 100 |

| Range of data: min = 0 max = 1 |

| |

| Data Description: |

| generated by r.neighbors |

| |

| Comments: |

| r.neighbors input=“MAL_AFRICA” output=“MAL_Mode_5x5” method=“mode” s\ |

| ize=5 |

| |

±---------------------------------------------------------------------------+

Luís

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Friday, January 29, 2021 3:31 PM, Micha Silver <tsvibar@gmail.com> wrote:

On Fri, Jan 29, 2021 at 3:53 PM Luí­s Moreira de Sousa via grass-user <grass-user@lists.osgeo.org> wrote:

Dear all,

I am getting the error “Unable to seek” with v.stats.error. There is not much information that could point the cause, just a warning saying that some data base files are not found. I checked the database connection and everything looks in order (see below). Any hints on what may be causing this error?

Thank you.

v.rast.stats map=PSU raster=MAL_Mode_5x5 method=number column_prefix=mal

ERROR: Unable to seek: Invalid argument

ERROR: An error occurred while converting vector to raster

WARNING: No data base element files found

db.connect -p

driver: pg

database: s4a

schema: mal

group:

psql -d s4a -p 5434

psql (12.5 (Ubuntu 12.5-0ubuntu0.20.04.1))

Type “help” for help.

s4a=# \dt mal.*

List of relations

Schema | Name | Type | Owner

--------±-------------------------±------±---------

mal | psu | table | duque004

(1 rows)

Could it be simply CAPS in the vector name vs small letters in the Postgres table?

Can you post the outputs from:

v.info PSU

r.info MAL_Mode_5x5

s4a=# select count(*) from mal.psu;

count


19468516

(1 row)

Luís


grass-user mailing list

grass-user@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/grass-user

Micha Silver

Ben Gurion Univ

Sde-Boker Remote Sensing Lab

cell: +972 (52) 3665918


grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Hi Luis:

On 2/1/21 5:30 PM, Luí s Moreira de Sousa wrote:

Hi Micha, thank you for replying.

I recreated the PSU layer in another mapset with the default SQLite back-end, v.rast.stats fails with the exact same error. So this is not related to the back-end.

Below are the outputs you requested. Thank you.

> v.info PSU
+----------------------------------------------------------------------------+
| Name: PSU |
| Mapset: MAL |
| Location: S4AHomolosine |
| Database: /home/duque004/Work/GRASSDATA |
| Title: |
| Map scale: 1:1 |
| Name of creator: duque004 |
| Organization: |
| Source date: Fri Jan 29 08:13:08 2021 |
| Timestamp (first layer): none |
|----------------------------------------------------------------------------|
| Map format: native |
|----------------------------------------------------------------------------|
| Type of map: vector (level: 2) |
| |
| Number of points: 0 Number of centroids: 19468516 |
| Number of lines: 0 Number of boundaries: 38945853 |
| Number of areas: 19468516 Number of islands: 1 |
| |
| Map is 3D: No |
| Number of dblinks: 1 |
| |
| Projection: unknown |
| |
| N: 4289569.27224353 S: -4452930.72775647 |
| E: 6679312.25515029 W: -2226387.74484971 |
| |
| Digitization threshold: 0 |
| Comment: |
| |
+----------------------------------------------------------------------------+

> r.info MAL_Mode_5x5
+----------------------------------------------------------------------------+
| Map: MAL_Mode_5x5 Date: Thu Jan 28 09:08:14 2021 |
| Mapset: MAL Login of Creator: duque004 |
| Location: S4AHomolosine |
| DataBase: /home/duque004/Work/GRASSDATA |
| Title: 5x5 neighborhood: mode of MAL_AFRICA |
| Timestamp: none |
|----------------------------------------------------------------------------|
| |
| Type of Map: raster Number of Categories: 0 |
| Data Type: CELL |
| Rows: 87425 |
| Columns: 89057 |
| Total Cells: 7785808225 |
| Projection: unknown |
| N: 4289569.27224353 S: -4452930.72775647 Res: 100 |
| E: 6679312.25515029 W: -2226387.74484971 Res: 100 |
| Range of data: min = 0 max = 1 |
| |
| Data Description: |
| generated by r.neighbors |
| |
| Comments: |
| r.neighbors input="MAL_AFRICA" output="MAL_Mode_5x5" method="mode" s\ |
| ize=5 |
| |
+----------------------------------------------------------------------------+

So the vector contains about 20 million polygons, and the raster about 8 billion cells. (~=32GB ?)

Does you computer have enough muscle for this?

--
Luís

Sent with ProtonMail <https://protonmail.com> Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, January 29, 2021 3:31 PM, Micha Silver <tsvibar@gmail.com> wrote:

On Fri, Jan 29, 2021 at 3:53 PM Luí­s Moreira de Sousa via grass-user <grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>> wrote:

    Dear all,

    I am getting the error "Unable to seek" with v.stats.error. There
    is not much information that could point the cause, just a
    warning saying that some data base files are not found. I checked
    the database connection and everything looks in order (see
    below). Any hints on what may be causing this error?

    Thank you.

    > v.rast.stats map=PSU raster=MAL_Mode_5x5 method=number
    column_prefix=mal
    ERROR: Unable to seek: Invalid argument
    ERROR: An error occurred while converting vector to raster
    WARNING: No data base element files found

    > db.connect -p
    driver: pg
    database: s4a
    schema: mal
    group:

    > psql -d s4a -p 5434
    psql (12.5 (Ubuntu 12.5-0ubuntu0.20.04.1))
    Type "help" for help.

    s4a=# \dt mal.*
     List of relations
    Schema | Name | Type | Owner
    --------+--------------------------+-------+----------
    mal | psu | table | duque004
    (1 rows)

Could it be simply CAPS in the vector name vs small letters in the Postgres table?
Can you post the outputs from:
v.info <http://v.info> PSU
r.info <http://r.info> MAL_Mode_5x5

    s4a=# select count(*) from mal.psu;
     count
    ----------
    19468516
    (1 row)

    -- Luís

    _______________________________________________
    grass-user mailing list
    grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>
    https://lists.osgeo.org/mailman/listinfo/grass-user
    <https://lists.osgeo.org/mailman/listinfo/grass-user&gt;

--
Micha Silver
Ben Gurion Univ
Sde-Boker Remote Sensing Lab
cell: +972 (52) 3665918

--
Micha Silver
Ben Gurion Univ.
Sde Boker, Remote Sensing Lab
cell: +972-523-665918
https://orcid.org/0000-0002-1128-1325

Hi Luis,

From the error message, it seems v.to.rast is failing, which may be due to a lack of resources - as Micha suggests.

Can you try
a) v.to.rast input=PSU output=test use=cat
if that does not work, we at least know it is not v.rast.stats and we might get a more specific error message... Not sure if the memory option in v.to.rast would have an effect here...

b) try to subdivide your data into chunks / tiles (polygons within tiles)?

Cheers
Stefan

-----Original Message-----
From: grass-user <grass-user-bounces@lists.osgeo.org> On Behalf Of Micha Silver
Sent: mandag 1. februar 2021 21:32
To: Luí s Moreira de Sousa <luis.de.sousa@protonmail.ch>; GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] v.rast.stats error: "Unable to seek"

Hi Luis:

On 2/1/21 5:30 PM, Luí s Moreira de Sousa wrote:

Hi Micha, thank you for replying.

I recreated the PSU layer in another mapset with the default SQLite
back-end, v.rast.stats fails with the exact same error. So this is not
related to the back-end.

Below are the outputs you requested. Thank you.

> v.info PSU
+----------------------------------------------------------------------------+
| Name: PSU |
| Mapset: MAL |
| Location: S4AHomolosine
| |
| Database: /home/duque004/Work/GRASSDATA
| |
| Title: |
| Map scale: 1:1
| | Name of creator:
duque004 |
| Organization: |
| Source date: Fri Jan 29 08:13:08
2021 |
| Timestamp (first layer):
none |
|----------------------------------------------------------------------------|
| Map format: native
| |
|----------------------------------------------------------------------------|
| Type of map: vector (level:
2) |
| |
| Number of points: 0 Number of centroids:
19468516 |
| Number of lines: 0 Number of boundaries:
38945853 |
| Number of areas: 19468516 Number of islands:
1 |
| |
| Map is 3D: No |
| Number of dblinks: 1
||
| |
| Projection:
unknown |
| |
| N: 4289569.27224353 S:
-4452930.72775647 |
| E: 6679312.25515029 W:
-2226387.74484971 |
| |
| Digitization threshold:
0 |
| Comment: |
| |
+----------------------------------------------------------------------------+

> r.info MAL_Mode_5x5
+----------------------------------------------------------------------------+
| Map: MAL_Mode_5x5 Date: Thu Jan 28 09:08:14
2021 |
| Mapset: MAL Login of Creator:
duque004 |
| Location: S4AHomolosine |
| DataBase: /home/duque004/Work/GRASSDATA |
| Title: 5x5 neighborhood: mode of
MAL_AFRICA |
| Timestamp: none |
|----------------------------------------------------------------------------|
| |
| Type of Map: raster Number of Categories:
0 |
| Data Type:
CELL |
| Rows: 87425 |
| Columns: 89057
||
| Total Cells:
7785808225 |
| Projection:
unknown |
| N: 4289569.27224353 S: -4452930.72775647 Res:
100 |
| E: 6679312.25515029 W: -2226387.74484971 Res:
100 |
| Range of data: min = 0 max =
1 |
| |
| Data Description: |
| generated by
r.neighbors |
| |
| Comments: |
| r.neighbors input="MAL_AFRICA" output="MAL_Mode_5x5"
method="mode" s\ |
| ize=5 |
| |
+----------------------------------------------------------------------------+

So the vector contains about 20 million polygons, and the raster about 8 billion cells. (~=32GB ?)

Does you computer have enough muscle for this?

--
Luís

Sent with ProtonMail <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotonmail.com%2F&amp;data=04|01||3a73488d58cd411cbc0f08d8c6f07342|6cef373021314901831055b3abf02c73|0|0|637478083673619175|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&amp;sdata=4t7uF0b7BR9kcAqAReOPlaM6z8L7fwAJ94zMC66lzTc%3D&amp;reserved=0&gt; Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, January 29, 2021 3:31 PM, Micha Silver <tsvibar@gmail.com>
wrote:

On Fri, Jan 29, 2021 at 3:53 PM Luí­s Moreira de Sousa via grass-user
<grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>> wrote:

    Dear all,

    I am getting the error "Unable to seek" with v.stats.error. There
    is not much information that could point the cause, just a
    warning saying that some data base files are not found. I checked
    the database connection and everything looks in order (see
    below). Any hints on what may be causing this error?

    Thank you.

    > v.rast.stats map=PSU raster=MAL_Mode_5x5 method=number
    column_prefix=mal
    ERROR: Unable to seek: Invalid argument
    ERROR: An error occurred while converting vector to raster
    WARNING: No data base element files found

    > db.connect -p
    driver: pg
    database: s4a
    schema: mal
    group:

    > psql -d s4a -p 5434
    psql (12.5 (Ubuntu 12.5-0ubuntu0.20.04.1))
    Type "help" for help.

    s4a=# \dt mal.*
     List of relations
    Schema | Name | Type | Owner
    --------+--------------------------+-------+----------
    mal | psu | table | duque004
    (1 rows)

Could it be simply CAPS in the vector name vs small letters in the
Postgres table?
Can you post the outputs from:
v.info
<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fv.i
nfo%2F&amp;data=04%7C01%7C%7C3a73488d58cd411cbc0f08d8c6f07342%7C6cef3
73021314901831055b3abf02c73%7C0%7C0%7C637478083673619175%7CUnknown%7C
TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
CI6Mn0%3D%7C3000&amp;sdata=Lc8A9SfmNwQKAMt3TTXYh%2FYBNurG6kUYAubGcxKh
3hw%3D&amp;reserved=0> PSU r.info
<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fr.i
nfo%2F&amp;data=04%7C01%7C%7C3a73488d58cd411cbc0f08d8c6f07342%7C6cef3
73021314901831055b3abf02c73%7C0%7C0%7C637478083673619175%7CUnknown%7C
TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
CI6Mn0%3D%7C3000&amp;sdata=GyWpg6cTSQfOc0LOFHKvEzLa4lxJXbinsTJlPm0lbt
Y%3D&amp;reserved=0> MAL_Mode_5x5

    s4a=# select count(*) from mal.psu;
     count
    ----------
    19468516
    (1 row)

    --
    Luís

    _______________________________________________
    grass-user mailing list
    grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>
    https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgrass-user&amp;data=04|01||3a73488d58cd411cbc0f08d8c6f07342|6cef373021314901831055b3abf02c73|0|0|637478083673629172|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&amp;sdata=2P01wK1Ak3X0QzZsMNLQa%2FLrQoVWrd8C7FgNezSRYRM%3D&amp;reserved=0
    
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
sts.osgeo.org%2Fmailman%2Flistinfo%2Fgrass-user&amp;data=04%7C01%7C%7
C3a73488d58cd411cbc0f08d8c6f07342%7C6cef373021314901831055b3abf02c73%
7C0%7C0%7C637478083673629172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=2
P01wK1Ak3X0QzZsMNLQa%2FLrQoVWrd8C7FgNezSRYRM%3D&amp;reserved=0>

--
Micha Silver
Ben Gurion Univ
Sde-Boker Remote Sensing Lab
cell: +972 (52) 3665918

--
Micha Silver
Ben Gurion Univ.
Sde Boker, Remote Sensing Lab
cell: +972-523-665918
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Forcid.org%2F0000-0002-1128-1325&amp;data=04|01||3a73488d58cd411cbc0f08d8c6f07342|6cef373021314901831055b3abf02c73|0|0|637478083673629172|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&amp;sdata=PuJmTChNnqdqy%2F3CrSb3FbfoIMprgxLPwf%2B0j86VS8s%3D&amp;reserved=0

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgrass-user&amp;data=04|01||3a73488d58cd411cbc0f08d8c6f07342|6cef373021314901831055b3abf02c73|0|0|637478083673629172|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&amp;sdata=2P01wK1Ak3X0QzZsMNLQa%2FLrQoVWrd8C7FgNezSRYRM%3D&amp;reserved=0

Thank you all for various replies. Some reactions:

1. The workstation has 32 GB of RAM and 12 CPU. What is the expected RAM requirements for v.rast.stats?

2. v.to.rast also fails, see below.

3. What would be the way to check the health of the vector? The table in Postgres at least is fully usable.

v.to.rast input=PSU output=test use=cat

ERROR: Unable to seek: Invalid argument

--
Luís

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, February 1, 2021 11:56 PM, Stefan Blumentrath <Stefan.Blumentrath@nina.no> wrote:

So the vector contains about 20 million polygons, and the raster about 8 billion cells. (~=32GB ?)

Does you computer have enough muscle for this?

Hi again Luis,

The error comes from v.to.rast (which is used inside v.rast.stats).

From the error message (which suggests that the G_fseek() function gets an invalid argument), it occurs most likely here: https://github.com/OSGeo/grass/blob/master/lib/gis/seek.c
However, the root cause may be at a different place...

Could you try:

a) if assigning more memory to v.to.rast has any effect
v.to.rast input=PSU output=test use=cat memory=10000
b) if assigning category values to the raster causes problems
v.to.rast input=PSU output=test use=val value=1
in order to narrow down a bit?

Please also report v.info PSU -c (to see data types of the columns)...

Anyway, I guess here are people with C-knowledge required...

Cheers
Stefan

-----Original Message-----
From: Luís Moreira de Sousa <luis.de.sousa@protonmail.ch>
Sent: tirsdag 2. februar 2021 15:56
To: Stefan Blumentrath <Stefan.Blumentrath@nina.no>
Cc: Micha Silver <tsvibar@gmail.com>; GRASS user list <grass-user@lists.osgeo.org>
Subject: RE: [GRASS-user] v.rast.stats error: "Unable to seek"

Thank you all for various replies. Some reactions:

1. The workstation has 32 GB of RAM and 12 CPU. What is the expected RAM requirements for v.rast.stats?

2. v.to.rast also fails, see below.

3. What would be the way to check the health of the vector? The table in Postgres at least is fully usable.

v.to.rast input=PSU output=test use=cat

ERROR: Unable to seek: Invalid argument

--
Luís

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, February 1, 2021 11:56 PM, Stefan Blumentrath <Stefan.Blumentrath@nina.no> wrote:

So the vector contains about 20 million polygons, and the raster about 8 billion cells. (~=32GB ?)

Does you computer have enough muscle for this?

Hi Stefan, thank you for the reply.

The outputs you request are below. v.rast.stats takes about 8 GB of RAM before failing, only 1/4 of what is available in the workstation. I also tried increasing the memory parameter but it never goes above 8 GB and fails all the same.

Let me know if there is something else I can check. Thank you.

v.info PSU -c

Displaying column types/names for database connection of layer <1>:
INTEGER|cat
INTEGER|value
CHARACTER|label

v.to.rast input=PSU output=test use=cat memory=10000

ERROR: Unable to seek: Invalid argument

v.to.rast input=PSU output=test use=val value=1

ERROR: Unable to seek: Invalid argument

--
Luís

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, February 2, 2021 11:24 PM, Stefan Blumentrath <Stefan.Blumentrath@nina.no> wrote:

Hi again Luis,

The error comes from v.to.rast (which is used inside v.rast.stats).

From the error message (which suggests that the G_fseek() function gets an invalid argument), it occurs most likely here: https://github.com/OSGeo/grass/blob/master/lib/gis/seek.c
However, the root cause may be at a different place...

Could you try:

a) if assigning more memory to v.to.rast has any effect
v.to.rast input=PSU output=test use=cat memory=10000
b) if assigning category values to the raster causes problems
v.to.rast input=PSU output=test use=val value=1
in order to narrow down a bit?

Please also report v.info PSU -c (to see data types of the columns)...

Anyway, I guess here are people with C-knowledge required...

Cheers
Stefan

Hi Luí­s,

On Wed, Feb 3, 2021 at 5:51 PM Luí­s Moreira de Sousa via grass-user
<grass-user@lists.osgeo.org> wrote:

Hi Stefan, thank you for the reply.

The outputs you request are below. v.rast.stats takes about 8 GB of RAM before failing, only 1/4 of what is available in the workstation. I also tried increasing the memory parameter but it never goes above 8 GB and fails all the same.

If I am not mistaken, the memory= parameter isn't used at all in
v.rast.stats, hence not passed to v.to.rast:
https://github.com/OSGeo/grass/blob/e5afa5a8a0f8e39376f4ddfbda1a62abdef99a21/scripts/v.rast.stats/v.rast.stats.py#L167

For a test, you could use g.gisenv and set the "cache" memory to a
higher value (v.to.rast will respect it then):
https://grass.osgeo.org/grass78/manuals/variables.html#list-of-selected-internal-grass-environment-variables
--> MEMORYMB

Eg.

# set to 12 GB (default: 300 MB)
g.gisenv set="MEMORYMB=12000"

Not sure if that would help anyway.

However:

Let me know if there is something else I can check. Thank you.

yes, you may try with "valgrind":
https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind

CMD="v.to.rast input=PSU output=test use=cat memory=10000"
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes $CMD --o

The output may give some hints (it will be long).

Best
Markus

Don't want to bring bad news, but it looks more like an offset
overflow. You will not catch it with valgrind. Although it might catch
a bug leading to offset value explosion, most likely the main cause is
just code written for handling of small/medium datasets and not
large/huge ones (=imperfect logic => can't catch that with valgrind).

My suggestion: recompile GRASS with -ftrapv. If it is an integer
overflow, at least it will become clearly obvious.
https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html

The bad thing is G_fseek calling G_fatal_error without knowing actual
file name (lets put pipes aside) and thus it is impossible to tell
where exactly the error originated from the error message alone.
Better would be to bubble up error to main program and let it deal
with it in a clean way. Of course, as GRASS is designed around current
idioms (handling failure is responsibility of a library making module
development really easy), this will not happen in a foreseeable
future.

One thing you could do – drasticly reduce size of computational region.
Māris.

Hi Markus,

I started by setting up the memory limit setting with g.gisenv, but v.to.rast fails with the same error.

I then tried valgrind. It is not really saying much, I fiddled a bit with it but can't more outputs that what you see below.

valgrind --tool=memcheck --leak-check=yes --show-reachable=yes $CMD

==60598== Memcheck, a memory error detector
==60598== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==60598== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==60598== Command: v.to.rast input=PSU output=test use=cat memory=10000
==60598==
==60598== Warning: set address range perms: large range [0x154c9d040, 0x170a47950) (undefined)
Killed

I think I can create a workflow to replicate this error, I will work on that next week.

Cheers.

--
Luís Moreira de Sousa
Pastoor Bruggemanlaan 21
6861 GR Oosterbeek
The Netherlands
Phone: +31 628 544 755
Email: luis.de.sousa@protonmail.ch
Mastodon: @luis_de_sousa@mastodon.social
URL: https://ldesousa.codeberg.page

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, February 3, 2021 6:57 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi Luí­s,

On Wed, Feb 3, 2021 at 5:51 PM Luí­s Moreira de Sousa via grass-user
grass-user@lists.osgeo.org wrote:

> Hi Stefan, thank you for the reply.
> The outputs you request are below. v.rast.stats takes about 8 GB of RAM before failing, only 1/4 of what is available in the workstation. I also tried increasing the memory parameter but it never goes above 8 GB and fails all the same.

If I am not mistaken, the memory= parameter isn't used at all in
v.rast.stats, hence not passed to v.to.rast:
https://github.com/OSGeo/grass/blob/e5afa5a8a0f8e39376f4ddfbda1a62abdef99a21/scripts/v.rast.stats/v.rast.stats.py#L167

For a test, you could use g.gisenv and set the "cache" memory to a
higher value (v.to.rast will respect it then):
https://grass.osgeo.org/grass78/manuals/variables.html#list-of-selected-internal-grass-environment-variables
--> MEMORYMB

Eg.

set to 12 GB (default: 300 MB)

===============================

g.gisenv set="MEMORYMB=12000"

Not sure if that would help anyway.

However:

> Let me know if there is something else I can check. Thank you.

yes, you may try with "valgrind":
https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind

CMD="v.to.rast input=PSU output=test use=cat memory=10000"
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes $CMD --o

The output may give some hints (it will be long).

Best
Markus

Hi Maris,

thank you for the details. I can try compiling with the flag you suggest, but I need a bit more time. Will let you know if I succeed.

Regards.

--
Luís Moreira de Sousa
Pastoor Bruggemanlaan 21
6861 GR Oosterbeek
The Netherlands
Phone: +31 628 544 755
Email: luis.de.sousa@protonmail.ch
Mastodon: @luis_de_sousa@mastodon.social
URL: https://ldesousa.codeberg.page

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, February 4, 2021 10:21 AM, Maris Nartiss <maris.gis@gmail.com> wrote:

Don't want to bring bad news, but it looks more like an offset
overflow. You will not catch it with valgrind. Although it might catch
a bug leading to offset value explosion, most likely the main cause is
just code written for handling of small/medium datasets and not
large/huge ones (=imperfect logic => can't catch that with valgrind).

My suggestion: recompile GRASS with -ftrapv. If it is an integer
overflow, at least it will become clearly obvious.
https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html

The bad thing is G_fseek calling G_fatal_error without knowing actual
file name (lets put pipes aside) and thus it is impossible to tell
where exactly the error originated from the error message alone.
Better would be to bubble up error to main program and let it deal
with it in a clean way. Of course, as GRASS is designed around current
idioms (handling failure is responsibility of a library making module
development really easy), this will not happen in a foreseeable
future.

One thing you could do – drasticly reduce size of computational region.
Māris.

Hi Luís,

Did you get any insights?

Best,
Markus

On Fri, Feb 5, 2021 at 4:17 PM Luí­s Moreira de Sousa
<luis.de.sousa@protonmail.ch> wrote:

Hi Maris,

thank you for the details. I can try compiling with the flag you suggest, but I need a bit more time. Will let you know if I succeed.

Regards.

--
Luís Moreira de Sousa
Pastoor Bruggemanlaan 21
6861 GR Oosterbeek
The Netherlands
Phone: +31 628 544 755
Email: luis.de.sousa@protonmail.ch
Mastodon: @luis_de_sousa@mastodon.social
URL: https://ldesousa.codeberg.page

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, February 4, 2021 10:21 AM, Maris Nartiss <maris.gis@gmail.com> wrote:

> Don't want to bring bad news, but it looks more like an offset
> overflow. You will not catch it with valgrind. Although it might catch
> a bug leading to offset value explosion, most likely the main cause is
> just code written for handling of small/medium datasets and not
> large/huge ones (=imperfect logic => can't catch that with valgrind).
>
> My suggestion: recompile GRASS with -ftrapv. If it is an integer
> overflow, at least it will become clearly obvious.
> https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html
>
> The bad thing is G_fseek calling G_fatal_error without knowing actual
> file name (lets put pipes aside) and thus it is impossible to tell
> where exactly the error originated from the error message alone.
> Better would be to bubble up error to main program and let it deal
> with it in a clean way. Of course, as GRASS is designed around current
> idioms (handling failure is responsibility of a library making module
> development really easy), this will not happen in a foreseeable
> future.
>
> One thing you could do – drasticly reduce size of computational region.
> Māris.

--
Markus Neteler, PhD
https://www.mundialis.de - free data with free software
https://grass.osgeo.org
https://courses.neteler.org/blog

Hi Markus,

I tried compiling GRASS with the -ftrapv flag, but it was failing (can't remember now why). I was also supposed to create a replicating procedure, but other things came up in the meantime. However, it looks like for rasters larger than a certain size none of the modules depending on v.to.rast function correctly.

I will try to have a look again at an agnostic replication procedure.

Cheers.

--
Luís

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, May 18, 2021 10:56 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi Luís,

Did you get any insights?

Best,
Markus

On Fri, Feb 5, 2021 at 4:17 PM Luí­s Moreira de Sousa
luis.de.sousa@protonmail.ch wrote:

> Hi Maris,
> thank you for the details. I can try compiling with the flag you suggest, but I need a bit more time. Will let you know if I succeed.
> Regards.
> --
> Luís Moreira de Sousa
> Pastoor Bruggemanlaan 21
> 6861 GR Oosterbeek
> The Netherlands
> Phone: +31 628 544 755
> Email: luis.de.sousa@protonmail.ch
> Mastodon: @luis_de_sousa@mastodon.social
> URL: https://ldesousa.codeberg.page
> Sent with ProtonMail Secure Email.
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, February 4, 2021 10:21 AM, Maris Nartiss maris.gis@gmail.com wrote:
>
> > Don't want to bring bad news, but it looks more like an offset
> > overflow. You will not catch it with valgrind. Although it might catch
> > a bug leading to offset value explosion, most likely the main cause is
> > just code written for handling of small/medium datasets and not
> > large/huge ones (=imperfect logic => can't catch that with valgrind).
> > My suggestion: recompile GRASS with -ftrapv. If it is an integer
> > overflow, at least it will become clearly obvious.
> > https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html
> > The bad thing is G_fseek calling G_fatal_error without knowing actual
> > file name (lets put pipes aside) and thus it is impossible to tell
> > where exactly the error originated from the error message alone.
> > Better would be to bubble up error to main program and let it deal
> > with it in a clean way. Of course, as GRASS is designed around current
> > idioms (handling failure is responsibility of a library making module
> > development really easy), this will not happen in a foreseeable
> > future.
> > One thing you could do – drasticly reduce size of computational region.
> > Māris.

--

Markus Neteler, PhD
https://www.mundialis.de - free data with free software
https://grass.osgeo.org
https://courses.neteler.org/blog

Hi

On Wed, May 19, 2021 at 10:16 AM Luí­s Moreira de Sousa
<luis.de.sousa@protonmail.ch> wrote:

Hi Markus,

I tried compiling GRASS with the -ftrapv flag, but it was failing (can't remember now why).

Just post the error message when you come back to this.

I was also supposed to create a replicating procedure, but other things came up in the meantime. However, it looks like for rasters larger than a certain size none of the modules depending on v.to.rast function correctly.

GRASS GIS 7 supports the off_t type, hence it can address an enormous
amount of raster data (like 1e+18 pixels or more), see
https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_data_processing

Bugs and limitations are always possible but there is interest in
fixing them wherever possible.

I will try to have a look again at an agnostic replication procedure.

Alright.

Best
Markus

Hi again Markus,

below is a small example with an extent of 10^12 cells. It does not get to v.rast.stats, as the r.random.surface command fails before that (without error message). I will do more experiments in the following days.

Regards.

# Set region for zones
g.region n=10000000 s=0 w=0 e=10000000 nsres=2000 ewres=2000

# Create zones map
r.mapcalc expr="zones=row()*10000+col()"
r.to.vect -b input=zones output=zones column=zone type=area

# Set computation region
g.region n=10000000 s=0 w=0 e=10000000 nsres=100 ewres=100

# Generate random raster : this command fails without error message
r.random.surface output=random seed=420 distance=1000 --overwrite

# Stats
v.rast.stats map=zones raster=random method=number column_prefix=stats

--
Luís

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 19, 2021 5:09 PM, Markus Neteler <neteler@osgeo.org> wrote:

GRASS GIS 7 supports the off_t type, hence it can address an enormous
amount of raster data (like 1e+18 pixels or more), see
https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_data_processing

Bugs and limitations are always possible but there is interest in
fixing them wherever possible.

> I will try to have a look again at an agnostic replication procedure.

Alright.

Best
Markus