[GRASS-dev] r.in.gdal slow with input of type=UInt16

GRASS GIS GRASS GIS developer
,
1

Hi,

when I try to import map, which was previously exported using r.out.gdal
type=UInt16, it takes too long:

GRASS 6.3.cvs (spearfish60):~ > g.region -dp
projection: 1 (UTM)
zone: 13
datum: nad27
ellipsoid: clark66
north: 4928010
south: 4913700
west: 589980
east: 609000
nsres: 30
ewres: 30
rows: 477
cols: 634
cells: 302418

GRASS 6.3.cvs (spearfish60):~ > r.out.gdal in=soils out=/tmp/soils.tif
type=UInt16
100%
r.out.gdal hotovo.

GRASS 6.3.cvs (spearfish60):~ > time r.in.gdal in=/tmp/soils.tif -o
out=tmp --o
100%

real 0m42.631s
user 0m41.239s
sys 0m0.120s

GRASS 6.3.cvs (spearfish60):~ > r.out.gdal in=soils out=/tmp/soils.tif
type=UInt32
ERROR 6: SetColorTable() only supported for Byte or UInt16 bands in
TIFF format.
100%
r.out.gdal hotovo.

GRASS 6.3.cvs (spearfish60):~ > time r.in.gdal in=/tmp/soils.tif -o
out=tmp --o

100%

real 0m0.327s
user 0m0.192s
sys 0m0.064s

strace r.in.gdal in=/tmp/soils.tif out=tmp --o
[....]
access("/home/jachym/grassdata/spearfish60/jachym/cell/tmp", F_OK) = 0
access("/home/jachym/grassdata/spearfish60/jachym/fcell/tmp", F_OK) =
-1 ENOENT (No such file or directory)
access("/home/jachym/grassdata/spearfish60/jachym/g3dcell/tmp", F_OK)
= -1 ENOENT (No such file or directory)
write(10, "# 54 categories\n\n\n0.00 0.00 0.00"..., 38) = 38
close(10) = 0
munmap(0xb5b31000, 4096) = 0
access("/home/jachym/grassdata/spearfish60/jachym/cell_misc/tmp/histo
gram", F_OK) = -1 ENOENT (No such file or directory) time(NULL)
                      = 1180021981
access("/home/jachym/grassdata/spearfish60/jachym/hist", F_OK) = 0
access("/home/jachym/grassdata/spearfish60/jachym/hist", F_OK) = 0
creat("/home/jachym/grassdata/spearfish60/jachym/hist/tmp", 0666) = 10
close(10) = 0
open("/home/jachym/grassdata/spearfish60/jachym/hist/tmp", O_WRONLY) =
10
fcntl64(10, F_GETFL) = 0x1 (flags O_WRONLY)
fstat64(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb5b31000
_llseek(10, 0, [0], SEEK_CUR) = 0
write(10, "Thu May 24 17:53:01 2007\ntmp\njac"..., 121) = 121
close(10) = 0
munmap(0xb5b31000, 4096) = 0
brk(0x81f3000) = 0x81f3000
brk(0x8214000) = 0x8214000
brk(0x8235000) = 0x8235000
brk(0x8256000) = 0x8256000
brk(0x8277000) = 0x8277000
brk(0x8298000) = 0x8298000
brk(0x82b9000) = 0x82b9000
brk(0x82da000) = 0x82da000
brk(0x82fb000) = 0x82fb000
brk(0x831c000) = 0x831c000
brk(0x833d000) = 0x833d000
brk(0x835e000) = 0x835e000
brk(0x837f000) = 0x837f000
brk(0x83a0000) = 0x83a0000
brk(0x83c1000) = 0x83c1000
brk(0x83e2000) = 0x83e2000
brk(0x8403000) = 0x8403000
brk(0x8424000) = 0x8424000
brk(0x8445000) = 0x8445000
access("/home/jachym/grassdata/spearfish60/jachym/colr2/jachym/tmp",
F_OK) = -1 ENOENT (No such file or directory)
access("/home/jachym/grassdata/spearfish60/jachym/colr", F_OK) = 0
access("/home/jachym/grassdata/spearfish60/jachym/colr", F_OK) = 0
creat("/home/jachym/grassdata/spearfish60/jachym/colr/tmp", 0666) = 10
close(10) = 0
open("/home/jachym/grassdata/spearfish60/jachym/colr/tmp", O_WRONLY) =
10
fcntl64(10, F_GETFL) = 0x1 (flags O_WRONLY)
fstat64(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb5b31000
_llseek(10, 0, [0], SEEK_CUR) = 0
write(10, "% 0 65535\n0:122:43:136\n1:33:70:7"..., 4096) = 4096
write(10, "9:0\n620:0\n621:0\n622:0\n623:0\n624:"..., 4096) = 4096
write(10, "\n1259:0\n1260:0\n1261:0\n1262:0\n126"..., 4096) = 4096
write(10, "1844:0\n1845:0\n1846:0\n1847:0\n1848"..., 4096) = 4096
[....]

While "brk(brk(0x8445000) = 0x8445000)",
processor is on 100% and this is the place, where it takes so long.

Anybody could give me some advice?

Thanks

Jachym

--
Jachym Cepicky
e-mail: jachym.cepicky@gmail.com
URL: http://les-ejk.cz
GPG: http://www.les-ejk.cz/pgp/jachym_cepicky-gpg.pub

2

Jachym Cepicky wrote:

Hi,

when I try to import map, which was previously exported using r.out.gdal
type=UInt16, it takes too long:

GRASS 6.3.cvs (spearfish60):~ > g.region -dp
projection: 1 (UTM)
zone: 13
datum: nad27
ellipsoid: clark66
north: 4928010
south: 4913700
west: 589980
east: 609000
nsres: 30
ewres: 30
rows: 477
cols: 634
cells: 302418

GRASS 6.3.cvs (spearfish60):~ > r.out.gdal in=soils out=/tmp/soils.tif
type=UInt16
100%
r.out.gdal hotovo.

GRASS 6.3.cvs (spearfish60):~ > time r.in.gdal in=/tmp/soils.tif -o
out=tmp --o
100%

real 0m42.631s
user 0m41.239s
sys 0m0.120s

...

While "brk(brk(0x8445000) = 0x8445000)",
processor is on 100% and this is the place, where it takes so long.

Anybody could give me some advice?

Jachym,

What is the size of soils.tif? brk() is the memory allocated which suggests
to me that the process may be taking an unreasonable amount of memory and that
this is causing problems.

If you hadn't just created it with r.out.gdal, I'd suspect that soils.tif
was a dangerous sort of file, such as a 1bit fax compressed file all in
one strip. But that clearly can't be the case.

The fact that things stall just after creating the histogram file makes
me wonder if it has something to do with histogram creation. But I can't
see any obvious reason creating a histogram of a 16bit file should be slow.

You might want to actually run r.in.gdal in gdb, and break and get
tracebacks a few times. I find this a useful technique to identify
what a program is doing when it appears unexpectedly stalled.

I presume a "gdal_translate soils.tif soils2.tif" is fast? If so, that
suggests it isn't the actual GDAL operation that is slow.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

3

hi Frank,
Frank Warmerdam píše v Pá 25. 05. 2007 v 10:07 -0400:

Jachym,

What is the size of soils.tif? brk() is the memory allocated which suggests
to me that the process may be taking an unreasonable amount of memory and that
this is causing problems.

-rw-r--r-- 1 jachym jachym 1,4M 2007-05-24 17:50 /tmp/soils.tif

If you hadn't just created it with r.out.gdal, I'd suspect that soils.tif
was a dangerous sort of file, such as a 1bit fax compressed file all in
one strip. But that clearly can't be the case.

if I export soils to soils.tif with type=UInit32 , back import works
fast

The fact that things stall just after creating the histogram file makes
me wonder if it has something to do with histogram creation. But I can't
see any obvious reason creating a histogram of a 16bit file should be slow.

You might want to actually run r.in.gdal in gdb, and break and get
tracebacks a few times. I find this a useful technique to identify
what a program is doing when it appears unexpectedly stalled.

I presume a "gdal_translate soils.tif soils2.tif" is fast? If so, that
suggests it isn't the actual GDAL operation that is slow.

yes, it is fast,

thanks for this hints

Jachym
--
Jachym Cepicky
e-mail: jachym.cepicky@gmail.com
URL: http://les-ejk.cz
GPG: http://www.les-ejk.cz/pgp/jachym_cepicky-gpg.pub