[GRASS-user] error in remove mask

Giuseppe_Amatulli · May 24, 2017, 6:59pm

Hi all,
I have a strange error when I’m try to remove the mask

GRASS 7.0.2 (loc_river_fill_GLOBE):~ > r.mask -r
Traceback (most recent call last):
File “/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/scripts/r.mask”, line 184, in
main()
File “/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/scripts/r.mask”, line 104, in main
type = ‘raster’, name = ‘MASK’)
File “/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/etc/python/grass/script/core.py”, line 394, in run_command
return handle_errors(returncode, returncode, args, kwargs)
File “/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/etc/python/grass/script/core.py”, line 312, in handle_errors
returncode=returncode)
grass.exceptions.CalledModuleError: Module run None [‘g.remove’, ‘–q’, ‘-f’, ‘type=raster’, ‘name=MASK’] ended with error
Process ended with non-zero return code -11. See errors in the (error) output.
[Raster MASK present]

even if i try to remove
the MASK directly
g.remove -f type=raster name=MASK

or the file associate to it
g.remove -f type=raster name=UNIT3753

I get Segmentation fault

I try also to r.in.gdal the file again, the file is imported successfully, but I still not able to remove the MASK.

In the past I have used the UNIT3753-raster several time without problem than one week ago I used to create another data set
r.patch input=“UNIT3753,UNIT4000” output=“UNIT3753_4000”

Could be that the UNIT3753 raster is “link” to the UNIT3753_4000 and lock any kind of operation associate to it?

Thanks
Giuseppe

···

Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511

Teaching: http://spatial-ecology.org
Work: https://environment.yale.edu/profile/giuseppe-amatulli/

NikosAlexandris · May 24, 2017, 9:21pm

* Giuseppe Amatulli <giuseppe.amatulli@gmail.com> [2017-05-24 14:59:02 -0400]:

Hi all,
I have a strange error when I'm try to remove the mask

GRASS 7.0.2 (loc_river_fill_GLOBE):~ > r.mask -r

Dear Giuseppe,

Are you bound to 7.0.2? 7.2 is the latest released version.

Traceback (most recent call last):
File "/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/scripts/r.mask",
line 184, in <module>
   main()
File "/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/scripts/r.mask",
line 104, in main
   type = 'raster', name = 'MASK')
File
"/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/etc/python/grass/script/core.py",
line 394, in run_command
   return handle_errors(returncode, returncode, args, kwargs)
File
"/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/etc/python/grass/script/core.py",
line 312, in handle_errors
   returncode=returncode)
grass.exceptions.CalledModuleError: Module run None ['g.remove', '--q',
'-f', 'type=raster', 'name=MASK'] ended with error
Process ended with non-zero return code -11. See errors in the (error)
output.
[Raster MASK present]

What does `g.list -m raster` say?

even if i try to remove
the MASK directly
g.remove -f type=raster name=MASK
or the file associate to it
g.remove -f type=raster name=UNIT3753
I get Segmentation fault

would you have time to debug this one? See
https://grasswiki.osgeo.org/wiki/GRASS_Debugging.

As a last resort, did you try to hard-remove manually (as per rm -rf) all files
related to MASK and UNIT3735 (inside the directories cell, fcell, cell_misc and
cell_hd)?

I try also to r.in.gdal the file again, the file is imported successfully,
but I still not able to remove the MASK.

In the past I have used the UNIT3753-raster several time without problem
than one week ago I used to create another data set
r.patch input="UNIT3753,UNIT4000" output="UNIT3753_4000"

Could be that the UNIT3753 raster is "link" to the UNIT3753_4000 and lock
any kind of operation associate to it?

Unless I miss something, the answer is no. `r.patch` produces a new and
independent map.

Nikos

Giuseppe_Amatulli · May 24, 2017, 9:40pm

Hi Nikos
I just sort out.
the file /PERMANENT/cell_misc/UNIT3753/reclassed_to was very big (2.5g) and it was not be able to be removed with the r.remove.

Thank you
Giuseppe

···

On 24 May 2017 at 17:21, Nikos Alexandris <nik@nikosalexandris.net> wrote:

Giuseppe Amatulli <giuseppe.amatulli@gmail.com> [2017-05-24 14:59:02 -0400]:

Hi all,
I have a strange error when I’m try to remove the mask

GRASS 7.0.2 (loc_river_fill_GLOBE):~ > r.mask -r

Dear Giuseppe,

Are you bound to 7.0.2? 7.2 is the latest released version.

Traceback (most recent call last):
File “/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/scripts/r.mask”,
line 184, in
main()
File “/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/scripts/r.mask”,
line 104, in main
type = ‘raster’, name = ‘MASK’)
File
“/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/etc/python/grass/script/core.py”,
line 394, in run_command
return handle_errors(returncode, returncode, args, kwargs)
File
“/gpfs/apps/hpc.rhel7/Apps/GRASS/7.0.2/grass-7.0.2/etc/python/grass/script/core.py”,
line 312, in handle_errors
returncode=returncode)
grass.exceptions.CalledModuleError: Module run None [‘g.remove’, ‘–q’,
‘-f’, ‘type=raster’, ‘name=MASK’] ended with error
Process ended with non-zero return code -11. See errors in the (error)
output.
[Raster MASK present]

What does g.list -m raster say?

even if i try to remove
the MASK directly
g.remove -f type=raster name=MASK
or the file associate to it
g.remove -f type=raster name=UNIT3753
I get Segmentation fault

would you have time to debug this one? See
https://grasswiki.osgeo.org/wiki/GRASS_Debugging.

As a last resort, did you try to hard-remove manually (as per rm -rf) all files
related to MASK and UNIT3735 (inside the directories cell, fcell, cell_misc and
cell_hd)?

I try also to r.in.gdal the file again, the file is imported successfully,
but I still not able to remove the MASK.

In the past I have used the UNIT3753-raster several time without problem
than one week ago I used to create another data set
r.patch input=“UNIT3753,UNIT4000” output=“UNIT3753_4000”

Could be that the UNIT3753 raster is “link” to the UNIT3753_4000 and lock
any kind of operation associate to it?

Unless I miss something, the answer is no. r.patch produces a new and
independent map.

Nikos

–

Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511

Teaching: http://spatial-ecology.org
Work: https://environment.yale.edu/profile/giuseppe-amatulli/

neteler · May 24, 2017, 10:29pm

On Wed, May 24, 2017 at 11:40 PM, Giuseppe Amatulli
<giuseppe.amatulli@gmail.com> wrote:

Hi Nikos
I just sort out.
the file /PERMANENT/cell_misc/UNIT3753/reclassed_to was very big (2.5g) and
it was not be able to be removed with the r.remove.

That size is not an issue at all (the EU DEM is 25GB or more for
example)... the problem must be elsewhere. Happy to help if possible.
BTW: I have compiled GRASS 7.2 also for RHEL via EPEL, see the download page.

Markus

--
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

Giuseppe_Amatulli · May 25, 2017, 3:24pm

Hi Markus,
in reality the computation beyond the mask problem is very complex, I will try to summarize.
I have the UNIT3753 mask sit on the PERMANENT and I create 1000 location in a multicore environment (HPC).
Each location use the UNIT3753-mask from the PERMANENT. This action produce that the cell_misc/UNIT3753/reclassed_to is written simultaneously from different cores.
I think that this action creates a reclassed_to file very large and probably also with different permission which was not be able to be removed.

I solved by manually remove the reclassed_to file, so now the process is running smoothly.

A potential solution would be:

for each location g.copy raster=PERMANENT,UNIT3753 , and than apply the r.mask using the UNIT3753 in the location and not in the PERMANENT.
This would work but data duplication will be there.
a second solution would be if the reclassed_to can have a unique name (e.g. reclassed_to$$) associate to the location.

Any thought?
Thank you
Best Regards
Giuseppe
p.s. in few weeks I will be in Berlin so we may chat in person.

···

On 24 May 2017 at 18:29, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 24, 2017 at 11:40 PM, Giuseppe Amatulli
<giuseppe.amatulli@gmail.com> wrote:

Hi Nikos
I just sort out.
the file /PERMANENT/cell_misc/UNIT3753/reclassed_to was very big (2.5g) and
it was not be able to be removed with the r.remove.

That size is not an issue at all (the EU DEM is 25GB or more for
example)… the problem must be elsewhere. Happy to help if possible.
BTW: I have compiled GRASS 7.2 also for RHEL via EPEL, see the download page.

Markus

–
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

–

Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511

Teaching: http://spatial-ecology.org
Work: https://environment.yale.edu/profile/giuseppe-amatulli/

Markus_Metz · May 28, 2017, 9:18pm

On Thu, May 25, 2017 at 5:24 PM, Giuseppe Amatulli <giuseppe.amatulli@gmail.com> wrote:

Hi Markus,
in reality the computation beyond the mask problem is very complex, I will try to summarize.
I have the UNIT3753 mask sit on the PERMANENT and I create 1000 location in a multicore environment (HPC).
Each location use the UNIT3753-mask from the PERMANENT. This action produce that the cell_misc/UNIT3753/reclassed_to is written simultaneously from different cores.

This is the problem. If several processes try to edit the same file at once, this can cause data corruption. The reclassed_to file holds the names of the reclassed raster maps based on that raster map. Instead of r.mask, you can use r.mapcalc to create specific MASKs for each process on each compute node. This should avoid these problems, as long as each process on each compute node uses its own unique mapset.

Markus M (running GRASS on an HPC system every day)

I think that this action creates a reclassed_to file very large and probably also with different permission which was not be able to be removed.

I solved by manually remove the reclassed_to file, so now the process is running smoothly.

A potential solution would be:

for each location g.copy raster=PERMANENT,UNIT3753 , and than apply the r.mask using the UNIT3753 in the location and not in the PERMANENT.
This would work but data duplication will be there.

a second solution would be if the reclassed_to can have a unique name (e.g. reclassed_to$$) associate to the location.

Any thought?
Thank you
Best Regards
Giuseppe
p.s. in few weeks I will be in Berlin so we may chat in person.

On 24 May 2017 at 18:29, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 24, 2017 at 11:40 PM, Giuseppe Amatulli
<giuseppe.amatulli@gmail.com> wrote:

Hi Nikos
I just sort out.
the file /PERMANENT/cell_misc/UNIT3753/reclassed_to was very big (2.5g) and
it was not be able to be removed with the r.remove.

That size is not an issue at all (the EU DEM is 25GB or more for
example)… the problem must be elsewhere. Happy to help if possible.
BTW: I have compiled GRASS 7.2 also for RHEL via EPEL, see the download page.

Markus

–
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

–
Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511
Teaching: http://spatial-ecology.org
Work: https://environment.yale.edu/profile/giuseppe-amatulli/

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Giuseppe_Amatulli · June 1, 2017, 8:25am

Thanks,
I implemented accordingly and seams to work fine.
Thank you.
Best
Giuseppe

···

On 28 May 2017 at 17:18, Markus Metz <markus.metz.giswork@gmail.com> wrote:

On Thu, May 25, 2017 at 5:24 PM, Giuseppe Amatulli <giuseppe.amatulli@gmail.com> wrote:

Hi Markus,
in reality the computation beyond the mask problem is very complex, I will try to summarize.
I have the UNIT3753 mask sit on the PERMANENT and I create 1000 location in a multicore environment (HPC).
Each location use the UNIT3753-mask from the PERMANENT. This action produce that the cell_misc/UNIT3753/reclassed_to is written simultaneously from different cores.

This is the problem. If several processes try to edit the same file at once, this can cause data corruption. The reclassed_to file holds the names of the reclassed raster maps based on that raster map. Instead of r.mask, you can use r.mapcalc to create specific MASKs for each process on each compute node. This should avoid these problems, as long as each process on each compute node uses its own unique mapset.

Markus M (running GRASS on an HPC system every day)

I think that this action creates a reclassed_to file very large and probably also with different permission which was not be able to be removed.

I solved by manually remove the reclassed_to file, so now the process is running smoothly.

A potential solution would be:

for each location g.copy raster=PERMANENT,UNIT3753 , and than apply the r.mask using the UNIT3753 in the location and not in the PERMANENT.
This would work but data duplication will be there.

a second solution would be if the reclassed_to can have a unique name (e.g. reclassed_to$$) associate to the location.

Any thought?
Thank you
Best Regards
Giuseppe
p.s. in few weeks I will be in Berlin so we may chat in person.

On 24 May 2017 at 18:29, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 24, 2017 at 11:40 PM, Giuseppe Amatulli
<giuseppe.amatulli@gmail.com> wrote:

Hi Nikos
I just sort out.
the file /PERMANENT/cell_misc/UNIT3753/reclassed_to was very big (2.5g) and
it was not be able to be removed with the r.remove.

That size is not an issue at all (the EU DEM is 25GB or more for
example)… the problem must be elsewhere. Happy to help if possible.
BTW: I have compiled GRASS 7.2 also for RHEL via EPEL, see the download page.

Markus

–
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

–
Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511
Teaching: http://spatial-ecology.org
Work: https://environment.yale.edu/profile/giuseppe-amatulli/

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

–

Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511

Teaching: http://spatial-ecology.org
Work: https://environment.yale.edu/profile/giuseppe-amatulli/