[GRASS-dev] [GRASS GIS] #1800: v.kernel crashes after long execution

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------
A v.kernel run on a dataset with over 700,000 points abruptly crashes
after several hours of processing:
'''v.kernel input=geocoding_after_highway_name_fix@PERMANENT
output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766 ROWS: 4710 COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file
C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)'''

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

Sorry, the formatting got messed up. Trying again:

{{{
v.kernel input=geocoding_after_highway_name_fix@PERMANENT
output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766 ROWS: 4710 COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file
C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by mmetz):

Replying to [comment:1 arencambre]:
>
{{{
v.kernel input=geocoding_after_highway_name_fix@PERMANENT
output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766 ROWS: 4710 COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file
C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)
}}}
>

Is there enough free disk space? You could test with creating a dummy
raster with the same region settings using e.g

{{{
r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)"
}}}

This should tell you if the problem is specific to v.kernel or more
general.

BTW, v.kernel is about twice as fast in trunk r53896.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

I have over 100 GB disk space. I didn't detect much disk utilization
during the run.

Not clear what {{{r.mapcalc "mockingbird_and_central_high_resolution =
rand(0.0, 1.0)"}}} is going to do?
'''mockingbird_and_central_high_resolution''' is the bad output from the
v.kernel run that crashed.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by mmetz):

Replying to [comment:3 arencambre]:
> I have over 100 GB disk space. I didn't detect much disk utilization
during the run.
Over 100 GB of free disk space?
>
> Not clear what {{{r.mapcalc "mockingbird_and_central_high_resolution =
rand(0.0, 1.0)"}}} is going to do?
'''mockingbird_and_central_high_resolution''' is the bad output from the
v.kernel run that crashed.

This should tell you if the problem is specific to v.kernel or more
general, because the crash was caused by the GRASS library handling raster
IO, not by v.kernel itself. Therefore the idea is to create a raster map
with the same type and name for the same region settings, but with a
different module in order to find out if the crash in the GRASS library is
specific to v.kernel or more general.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

Sorry, meant that I have over 100GB free.

Running {{{r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0,
1.0)"}}} appeared to be successful:

{{{
C:\Program Files (x86)\Quantum GIS Lisboa>r.mapcalc
"mockingbird_and_central_hig
h_resolution = rand(0.0, 1.0)"
  100%

C:\Program Files (x86)\Quantum GIS Lisboa>
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by mmetz):

Replying to [comment:5 arencambre]:
> Sorry, meant that I have over 100GB free.
>
> Running
>
{{{
r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)"
}}}
> appeared to be successful

Strange. I tried to reproduce this error with the current release branch
(6.4.svn) and with trunk on a laptop with Fedora Linux 16 64bit, Intel
core i5 at 2.4 GHz and 4 GB RAM.
I followed this tutorial:

http://qgis.spatialthoughts.com/2012/07/tutorial-making-heatmaps-using-
qgis-and.html

and used this dataset:

http://data.london.gov.uk/datafiles/crime-community-safety/police-uk-
crime-data-surrey.zip

with region settings:

g.region n=318266 s=117316 e=593618 w=363772 res=100 -p

No success, i.e. v.kernel finishes successfully in a reasonable amount of
time:

{{{
GRASS 6.4.svn

42 MB RAM used by v.kernel

uniform kernel
real 1m8.270s
user 0m28.876s
sys 0m37.474s

gaussian kernel
real 19m37.177s
user 9m4.927s
sys 9m59.545s

GRASS 7

19 MB RAM used by v.kernel

uniform kernel
real 1m8.286s
user 0m43.967s
sys 0m22.599s

gaussian kernel
real 5m24.772s
user 4m0.915s
sys 1m15.982s
}}}

I suggest to update to GRASS 6.4.3RC1.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

I just emailed you offlist with a scenario to reproduce with 6.4.2. I will
install 6.4.3RC1 here and see if that makes a difference.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

CORRECTION: I am reproducing the very, very long execution with the
scenario I emailed you. I am not clear if it still crashes. Regardless, it
appears that the very, very long execution may be the key problem here.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

Even with 6.4.3RC1, it's still very, very slow. On the progress bar's 2nd
pass, it's been a few minutes since it has budged.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel crashes after long execution
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

On 6.4.3RC1, the status bar (% complete indicator at bottom of v.kernel
window) actually hasn't budged at all since the 2nd pass started. So we
either have a bug--the status bar doesn't update--or v.kernel is taking
even longer to run than with 6.4.2.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel is very slow
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

Refocusing the issue on the core problem. Crashing could simply be because
something else is wrong that causes it to run for too long.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel is very slow
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by arencambre):

r53983 of 6.4.3 is MUCH faster. An almost all day v.kernel run is now only
25 minutes.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel is very slow
------------------------+---------------------------------------------------
Reporter: arencambre | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Keywords: | Platform: MSWindows 7
      Cpu: x86-64 |
------------------------+---------------------------------------------------

Comment(by neteler):

Replying to [comment:12 arencambre]:
> r53983 of 6.4.3 is MUCH faster. An almost all day v.kernel run is now
only 25 minutes.

Can we consider this as fixed?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

#1800: v.kernel is very slow
--------------------------+-------------------------------------------------
  Reporter: arencambre | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 6.4.3
Component: Vector | Version: 6.4.2
Resolution: fixed | Keywords:
  Platform: MSWindows 7 | Cpu: x86-64
--------------------------+-------------------------------------------------
Changes (by arencambre):

  * status: new => closed
  * resolution: => fixed

Comment:

Yes.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1800#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>