[GRASS-dev] g.gui.animation freezes when loading strds

GRASS GRASS developer
1

Hi,

recently I was playing with tgrass having 32 maps registered in strds.
I am not even able to load this dataset to g.gui.animation.

$ g.region -p
projection: 99 (ETRS89 / LAEA Europe)
north: 3096980
south: 3062980
west: 4690940
east: 4747240
nsres: 20
ewres: 20
rows: 1700
cols: 2815
cells: 4785500

$ g.gui.animation strds=clearcut

After dozen of seconds I have to kill g.gui.animation because it
freezes and consumes most of RAM. Debug messages do not show anything
useful.

Is it a know issues or should be reported?

Martin

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

2

Does it help to reduce the region?

···

On Aug 8, 2017 8:44 AM, “Martin Landa” <landa.martin@gmail.com> wrote:

Hi,

recently I was playing with tgrass having 32 maps registered in strds.
I am not even able to load this dataset to g.gui.animation.

$ g.region -p
projection: 99 (ETRS89 / LAEA Europe)
north: 3096980
south: 3062980
west: 4690940
east: 4747240
nsres: 20
ewres: 20
rows: 1700
cols: 2815
cells: 4785500

$ g.gui.animation strds=clearcut

After dozen of seconds I have to kill g.gui.animation because it
freezes and consumes most of RAM. Debug messages do not show anything
useful.

Is it a know issues or should be reported?

Martin

–
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

3

2017-08-08 15:03 GMT+02:00 Anna Petrášová <kratochanna@gmail.com>:

Does it help to reduce the region?

it doesn't seem to help (i tried both changing res and bounds). Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

4

Weird, please report it. Does it freeze with just series of tested maps?

···

On Aug 8, 2017 9:06 AM, “Martin Landa” <landa.martin@gmail.com> wrote:

2017-08-08 15:03 GMT+02:00 Anna Petrášová <kratochanna@gmail.com>:

Does it help to reduce the region?

it doesn’t seem to help (i tried both changing res and bounds). Ma

–
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

5

Hi,

2017-08-08 15:18 GMT+02:00 Anna Petrášová <kratochanna@gmail.com>:

Weird, please report it. Does it freeze with just series of tested maps?

yes, I remember that few days ago I was able to load datasets on same
machine without any problem. Application started to freeze and consume
RAM when exporting animation. Strangely few days later I am not even
able to load dataset. Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

6

Hello,

I’ve also experienced the freezing issue (dunno if it the same thing or not). Here, the rendering freezes if strds has more than 500 maps. I do not even get to the export part. Cancel does not close it, so I have to kill it (trunk r71319).

Vero

···

2017-08-08 15:23 GMT+02:00 Martin Landa <landa.martin@gmail.com>:

Hi,

2017-08-08 15:18 GMT+02:00 Anna Petrášová <kratochanna@gmail.com>:

Weird, please report it. Does it freeze with just series of tested maps?

yes, I remember that few days ago I was able to load datasets on same
machine without any problem. Application started to freeze and consume
RAM when exporting animation. Strangely few days later I am not even
able to load dataset. Ma

–
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

7

2017-08-08 15:18 GMT+02:00 Anna Petrášová <kratochanna@gmail.com>:

Weird, please report it. Does it freeze with just series of tested maps?

I did quick test with nc_climate_spm_2000_2012 datasets.
g.gui.animation loaded data, I was able to export also animation as
avi file. I will check my dataset again. Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

8

Hi Veronica,

On Tue, Aug 8, 2017 at 3:46 PM, Veronica Andreo <veroandreo@gmail.com> wrote:

I’ve also experienced the freezing issue (dunno if it the same thing or not). Here, the rendering freezes if strds has more than 500 maps. I do not even get to the export part. Cancel does not close it, so I have to kill it (trunk r71319).

Same problem here, when I reach 502 maps raise the following exception:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 16762, in <lambda>
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/frame.py", line 194, in SetAnimations
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py", line 305, in SetAnimations
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py", line 328, in _setAnimations
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py", line 407, in _updateBitmapData
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py", line 234, in Load
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py", line 366, in Render
  File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 218, in Queue
  File "/usr/lib64/python2.7/multiprocessing/queues.py", line 63, in __init__
  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 147, in __init__
  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__
OSError: [Errno 24] Too many open files

But checking the number of opened file on my system, it seems two order lower than the maximum:

$ cat /proc/sys/fs/file-nr
23422   0       1628292
$ cat /proc/sys/fs/file-max 
1628292

I’ve attached also the complete log of the command in DEBUG mode, but I was not able to understand where the problem could be. I’ve also rebooted just to be sure that the system was fresh, but again after 502 maps it breaks with the same error.

Any ideas?

Pietro

​

(attachments)

log_gui_animation.7z (2.36 KB)

9

On Thu, Aug 10, 2017 at 11:16 AM, Pietro <peter.zamb@gmail.com> wrote:

Hi Veronica,

On Tue, Aug 8, 2017 at 3:46 PM, Veronica Andreo <veroandreo@gmail.com>
wrote:

I've also experienced the freezing issue (dunno if it the same thing or
not). Here, the rendering freezes if strds has more than 500 maps. I do not
even get to the export part. Cancel does not close it, so I have to kill it
(trunk r71319).

Same problem here, when I reach 502 maps raise the following exception:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line
16762, in <lambda>
  File "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/frame.py",
line 194, in SetAnimations
  File
"/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py",
line 305, in SetAnimations
  File
"/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py",
line 328, in _setAnimations
  File
"/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py",
line 407, in _updateBitmapData
  File
"/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py", line
234, in Load
  File
"/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py", line
366, in Render
  File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 218, in
Queue
  File "/usr/lib64/python2.7/multiprocessing/queues.py", line 63, in
__init__
  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 147, in
__init__
  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in
__init__
OSError: [Errno 24] Too many open files

But checking the number of opened file on my system, it seems two order
lower than the maximum:

$ cat /proc/sys/fs/file-nr
23422 0 1628292
$ cat /proc/sys/fs/file-max
1628292

I’ve attached also the complete log of the command in DEBUG mode, but I was
not able to understand where the problem could be. I’ve also rebooted just
to be sure that the system was fresh, but again after 502 maps it breaks
with the same error.

I just tested 1100 maps without problem. In case this is relevant:

ulimit -n
1024

Any ideas?

Pietro

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

10

On Thu, Aug 10, 2017 at 8:15 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

On Thu, Aug 10, 2017 at 11:16 AM, Pietro <peter.zamb@gmail.com> wrote:

Hi Veronica,

On Tue, Aug 8, 2017 at 3:46 PM, Veronica Andreo <veroandreo@gmail.com>
wrote:

I’ve also experienced the freezing issue (dunno if it the same thing or
not). Here, the rendering freezes if strds has more than 500 maps. I do not
even get to the export part. Cancel does not close it, so I have to kill it
(trunk r71319).

Same problem here, when I reach 502 maps raise the following exception:

Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py”, line
16762, in
File “/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/frame.py”,
line 194, in SetAnimations
File
“/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py”,
line 305, in SetAnimations
File
“/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py”,
line 328, in _setAnimations
File
“/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py”,
line 407, in _updateBitmapData
File
“/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py”, line
234, in Load
File
“/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py”, line
366, in Render
File “/usr/lib64/python2.7/multiprocessing/init.py”, line 218, in
Queue
File “/usr/lib64/python2.7/multiprocessing/queues.py”, line 63, in
init
File “/usr/lib64/python2.7/multiprocessing/synchronize.py”, line 147, in
init
File “/usr/lib64/python2.7/multiprocessing/synchronize.py”, line 75, in
init
OSError: [Errno 24] Too many open files

But checking the number of opened file on my system, it seems two order
lower than the maximum:

$ cat /proc/sys/fs/file-nr
23422 0 1628292
$ cat /proc/sys/fs/file-max

the OS limit of the number of open files is given in
/proc/sys/fs/file-max

but that is irrelevant, what matters is the maximum allowed number of open files for the current user

1628292

I’ve attached also the complete log of the command in DEBUG mode, but I was
not able to understand where the problem could be. I’ve also rebooted just
to be sure that the system was fresh, but again after 502 maps it breaks
with the same error.

I just tested 1100 maps without problem. In case this is relevant:

ulimit -n
1024

ulimit -n reports the maximum allowed number of open files for the current user, this is the relevant limit

A solution to the error “Too many open files” would be to open the files/maps one at a time and closing them again, similar to the -z flag of r.series.

Just an idea,

Markus M

Any ideas?

Pietro

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

11

On Thu, Aug 10, 2017 at 2:34 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Thu, Aug 10, 2017 at 8:15 PM, Anna Petrášová <kratochanna@gmail.com>
wrote:

On Thu, Aug 10, 2017 at 11:16 AM, Pietro <peter.zamb@gmail.com> wrote:
> Hi Veronica,
>
> On Tue, Aug 8, 2017 at 3:46 PM, Veronica Andreo <veroandreo@gmail.com>
> wrote:
>>
>> I've also experienced the freezing issue (dunno if it the same thing or
>> not). Here, the rendering freezes if strds has more than 500 maps. I do
>> not
>> even get to the export part. Cancel does not close it, so I have to
>> kill it
>> (trunk r71319).
>
> Same problem here, when I reach 502 maps raise the following exception:
>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line
> 16762, in <lambda>
> File
> "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/frame.py",
> line 194, in SetAnimations
> File
>
> "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py",
> line 305, in SetAnimations
> File
>
> "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py",
> line 328, in _setAnimations
> File
>
> "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/controller.py",
> line 407, in _updateBitmapData
> File
> "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py",
> line
> 234, in Load
> File
> "/home/pietro/.local/grass-7.3.svn/gui/wxpython/animation/provider.py",
> line
> 366, in Render
> File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 218, in
> Queue
> File "/usr/lib64/python2.7/multiprocessing/queues.py", line 63, in
> __init__
> File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 147,
> in
> __init__
> File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75,
> in
> __init__
> OSError: [Errno 24] Too many open files
>
> But checking the number of opened file on my system, it seems two order
> lower than the maximum:
>
> $ cat /proc/sys/fs/file-nr
> 23422 0 1628292
> $ cat /proc/sys/fs/file-max

the OS limit of the number of open files is given in
/proc/sys/fs/file-max
but that is irrelevant, what matters is the maximum allowed number of open
files for the current user

> 1628292
>
> I’ve attached also the complete log of the command in DEBUG mode, but I
> was
> not able to understand where the problem could be. I’ve also rebooted
> just
> to be sure that the system was fresh, but again after 502 maps it breaks
> with the same error.

I just tested 1100 maps without problem. In case this is relevant:

ulimit -n
1024

ulimit -n reports the maximum allowed number of open files for the current
user, this is the relevant limit

good to know, that's why I used 1100, because it's higher than the
limit, but it still worked.

A solution to the error "Too many open files" would be to open the
files/maps one at a time and closing them again, similar to the -z flag of
r.series.

I was trying to figure out why they would be opened at all. Not sure yet.

Just an idea,

Markus M

>
> Any ideas?
>
> Pietro
>
>
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

12

Hi Markus,

On Thu, Aug 10, 2017 at 8:34 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

the OS limit of the number of open files is given in
/proc/sys/fs/file-max
but that is irrelevant, what matters is the maximum allowed number of open files for the current user

[cut]

ulimit -n reports the maximum allowed number of open files for the current user, this is the relevant limit

Thank you for the clarification. In my system the user limit is 1024.

I can confirm that raise the current limit with:

$ ulimit -n 4096

fixed the problem.

A solution to the error "Too many open files" would be to open the files/maps one at a time and closing them again, similar to the -z flag of r.series.

I was trying to understand where these file are opened, and it seems
to me that the python code is always opening and closing all the
files, It is not clear to me which ctypes function is just opening the
files without closing them.

Best regards

Pietro

13

On Thu, Aug 10, 2017 at 5:29 PM, Pietro <peter.zamb@gmail.com> wrote:

Hi Markus,

On Thu, Aug 10, 2017 at 8:34 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

the OS limit of the number of open files is given in
/proc/sys/fs/file-max
but that is irrelevant, what matters is the maximum allowed number of open files for the current user

[cut]

ulimit -n reports the maximum allowed number of open files for the current user, this is the relevant limit

Thank you for the clarification. In my system the user limit is 1024.

I can confirm that raise the current limit with:

$ ulimit -n 4096

fixed the problem.

A solution to the error "Too many open files" would be to open the files/maps one at a time and closing them again, similar to the -z flag of r.series.

I was trying to understand where these file are opened, and it seems
to me that the python code is always opening and closing all the
files, It is not clear to me which ctypes function is just opening the
files without closing them.

Does the problem appear when you just use series of raster maps, or is
it related to the temporal stuff? I would expect the problem would be
in both and in that case, I believe it does not depend on any ctypes
functions, all should be module calls.

Best regards

Pietro

14

Hi,

2017-08-08 16:41 GMT+02:00 Martin Landa <landa.martin@gmail.com>:

I did quick test with nc_climate_spm_2000_2012 datasets.
g.gui.animation loaded data, I was able to export also animation as
avi file. I will check my dataset again. Ma

I had time to dig into `g.gui.animation` source code and discover
where the problem is. It is caused by wrong time granularity in my
dataset. I have 32 maps valid for each year. My dataset was wrongly
created with granularity of 1 sec.

$ t.rast.list clearcut | head -n3
name|mapset|start_time|end_time
CZ0524011_clearcut_PX20_1984_2016___1984|PERMANENT|1984-01-01
00:00:00|1984-12-31 23:59:59
CZ0524011_clearcut_PX20_1984_2016___1985|PERMANENT|1985-01-01
00:00:00|1985-12-31 23:59:59

$ t.info clearcut | grep Granularity
| Granularity:................ 1 second

It causes problem since `g.gui.animation` tries to sample maps into
given granularity [1]. In my case it nicely fills RAM in few seconds.
It would be probably meaningful to check how many maps are sampled and
at least to print a fat big warning if it exceeds reasonable limit.
What do you think?

After fixing my dataset to granularity to 1 year, `g.gui.animation`
starts to work.

$ t.info clearcut | grep Granularity
| Granularity:................ 1 year

$ t.rast.list clearcut | head -n3
name|mapset|start_time|end_time
CZ0524011_clearcut_PX20_1984_2016___1984|PERMANENT|1984-01-01
00:00:00|1985-01-01 00:00:00
CZ0524011_clearcut_PX20_1984_2016___1985|PERMANENT|1985-01-01
00:00:00|1986-01-01 00:00:00

Ma

[1] https://trac.osgeo.org/grass/browser/grass/trunk/gui/wxpython/animation/temporal_manager.py#L265

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa