#1694: r.in.lidar tries to allocate way too much memory
------------------------+---------------------------------------------------
Reporter: torsti | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Raster | Version: svn-trunk
Keywords: r.in.lidar | Platform: Linux
Cpu: x86-64 |
------------------------+---------------------------------------------------
Comment(by torsti):
So memory allocation is based on the extent of the region and not the
bounding box of the LAS data, that explains a lot. That was my mistake
there! Still, to be on the safe side I've included the more detailed
information that was asked for.
g.region -p
{{{
D2/2: G__read_Cell_head
D2/2: G__read_Cell_head_array
D2/2: G__read_Cell_head
D2/2: G__read_Cell_head_array
projection: 1 (UTM)
zone: 35
datum: etrs89
ellipsoid: grs80
north: 7776450.217
south: 6605838.902
west: 61686.152
east: 732907.723
nsres: 1.00000027
ewres: 0.99999936
rows: 1170611
cols: 671222
cells: 785739856642
}}}
lasinfo R4133C4.laz
{{{
---------------------------------------------------------
Header Summary
---------------------------------------------------------
Version: 1.2
Source ID: 0
Reserved: 0
Project ID/GUID: '00000000-0000-0000-0000-000000000000'
System ID: ''
Generating Software: 'EspaEngine'
File Creation Day/Year: 0/0
Header Byte Size 227
Data Offset: 329
Header Padding: 2
Number Var. Length Records: 1
Point Data Format: 1
Number of Point Records: 11064863
Compressed: True
Compression Info: LASzip Version 2.1r0 c2 50000: POINT10 2
GPSTIME11 2
Number of Points by Return: 0 0 0 0 0
Scale Factor X Y Z: 0.01 0.01 0.01
Offset X Y Z: -0.00 -0.00 -0.00
Min X Y Z: 389000.00 7149000.00 91.36
Max X Y Z: 391999.99 7151999.99 139.61
Spatial Reference:
None
None
...
}}}
I updated r.in.lidar to revision
[https://trac.osgeo.org/grass/changeset/52593 52593].
both r.in.lidar and r.in.xyz complain about the amount of memory, because
the region is too big, but the amount of memory they ask for is not in the
exabyte range.
{{{
> r.in.lidar -o --overwrite input=R4133C4.laz output=R4133C4.las
method=mean
D2/2: G__read_Cell_head
D2/2: G__read_Cell_head_array
Over-riding projection check
D2/2: region.n=7776450.217000 region.s=6605838.902000
region.ns_res=1.000000
D2/2: region.rows=1170611 [box_rows=1170611] region.cols=671222
Current region rows: 1170611, cols: 671222
ERROR: G_calloc: unable to allocate 785741027253 * 4 bytes of memory at
main.c:534
}}}
{{{
> las2txt --keep-classes 2 --parse xyz --delimiter="|" --input R4133C4.las
--output=/tmp/las.tmp
> r.in.xyz input=/tmp/las.tmp output=R4133C4_ground_points
D2/2: G__read_Cell_head
D2/2: G__read_Cell_head_array
D2/2: region.n=7776450.217000 region.s=6605838.902000
region.ns_res=1.000000
D2/2: region.rows=1170611 [box_rows=1170611] region.cols=671222
Current region rows: 1170611, cols: 671222
ERROR: G_calloc: unable to allocate 785741027253 * 4 bytes of memory at
main.c:491
}}}
After adjusting the extent to the BBOX of the LAS data:
{{{
g.region -a n=7152000 s=7149000 e=392000 w=389000 res=1
}}}
r.in.lidar
{{{
D2/2: G__read_Cell_head
D2/2: G__read_Cell_head_array
Over-riding projection check
D2/2: region.n=7152000.000000 region.s=7149000.000000
region.ns_res=1.000000
D2/2: region.rows=3000 [box_rows=3000] region.cols=3000
Reading data ...
D2/2: pass=1/1 pass_n=7152000.000000 pass_s=7149000.000000 rows=3000
D2/2: allocating n_array
D2/2: allocating sum_array
100%
D2/2: pass 1 finished, 11064827 coordinates in box
Writing to map ...
100%
D1/2: close R4133C4.las compressed
D1/2: G_find_raster2(): name=R4133C4.las mapset=PERMANENT
D1/2: G_find_raster2(): name=R4133C4.las mapset=PERMANENT
D1/2: G_find_raster2(): name=R4133C4.las mapset=PERMANENT
r.in.lidar complete. 11064827 points found in region.
D1/2: Processed 11064863 points
}}}
r.in.xyz
{{{
D2/2: G__read_Cell_head
D2/2: G__read_Cell_head_array
D2/2: region.n=7152000.000000 region.s=7149000.000000
region.ns_res=1.000000
D2/2: region.rows=3000 [box_rows=3000] region.cols=3000
D2/2: estimated number of lines in file: 3224929
Reading data ...
D2/2: pass=1/1 pass_n=7152000.000000 pass_s=7149000.000000 rows=3000
D2/2: allocating n_array
D2/2: allocating sum_array
100%
D2/2: pass 1 finished, 3135351 coordinates in box
Writing to map ...
100%
D1/2: close R4133C4_ground_points compressed
D1/2: G_find_raster2(): name=R4133C4_ground_points mapset=PERMANENT
D1/2: G_find_raster2(): name=R4133C4_ground_points mapset=PERMANENT
D1/2: G_find_raster2(): name=R4133C4_ground_points mapset=PERMANENT
r.in.xyz complete. 3135351 points found in region.
D1/2: Processed 3135367 lines.
}}}
So everything seems to work now.
Sorry for the most likely unfounded & inaccurate bug report.
Cheers,
Torsti
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1694#comment:8>
GRASS GIS <http://grass.osgeo.org>
