[GRASS-user] importing Mars DEM takes too long

annakrat · November 18, 2016, 9:08pm

Hi,

I was trying to import Mars DEM with r.in.gdal, and it takes forever
to finish it (It showed 3% after 10 minutes). This is what gdalinfo
says:

gdalinfo /tmp/ESP_035583_2060_RED.JP2

Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
Files: /tmp/ESP_035583_2060_RED.JP2
       /tmp/ESP_035583_2060_RED.JP2.aux.xml
Size is 13094, 26340
Coordinate System is:
PROJCS["Equirectangular MARS",
    GEOGCS["GCS_MARS",
        DATUM["unknown",
            SPHEROID["unnamed",3392593.6110435,0]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Equirectangular"],
    PARAMETER["latitude_of_origin",25],
    PARAMETER["central_meridian",180],
    PARAMETER["standard_parallel_1",0],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]
Origin = (3889293.250000000000000,1519463.750000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Corner Coordinates:
Upper Left ( 3889293.250, 1519463.750) (114d18'56.57"W, 50d39'41.21"N)
Lower Left ( 3889293.250, 1506293.750) (114d18'56.57"W, 50d26'20.49"N)
Upper Right ( 3895840.250, 1519463.750) (114d12'18.52"W, 50d39'41.21"N)
Lower Right ( 3895840.250, 1506293.750) (114d12'18.52"W, 50d26'20.49"N)
Center ( 3892566.750, 1512878.750) (114d15'37.55"W, 50d33' 0.85"N)
Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Undefined
  Overviews: 6547x13170, 3273x6585, 1636x3292, 818x1646, 409x823,
204x411, 102x205
  Overviews: arbitrary
  Image Structure Metadata:
    NBITS=10

GRASS location was created from this file:

projection: 99 (Equirectangular MARS)
zone: 0
datum: ** unknown (default: WGS84) **
ellipsoid: a=3392593.6110435 es=0
north: 1519463.75
south: 1506293.75
west: 3889293.25
east: 3895840.25
nsres: 0.5
ewres: 0.5
rows: 26340
cols: 13094
cells: 344895960

I tried to open it in QGIS and it appeared instantly. Then, I tried to
export it from QGIS as Geotiff, but that took again very long (I
didn't wait for it to finish). Running gdal_translate in command line
can't finish either. I tried to debug r.in.gdal and it seems to be
stuck on line 1103 in GDALRasterIO call. Any advice? I can send the
data privately.

Thank you,

Anna

Sajid_Pareeth1 · November 19, 2016, 11:06am

Hi Anna

I am not sure if it is related, but I had similar issues with Sentinel JP2 files.

The solution was to use the ECW library to read the JP2 files instead of JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.

I had to recompile gdal to add the ECW support.

Add ECW support to GDAL

./configure --with-ecw=/usr/local
…

To read a jp2 file using ecw plugin, switch off the other two drivers

export GDAL_SKIP=JP2OpenJPEG,JPEG2000

Ofcourse you have to install libecw before recompiling gdal.

Hope it helps.

Sajid

···

On Fri, Nov 18, 2016 at 10:08 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

Hi,

I was trying to import Mars DEM with r.in.gdal, and it takes forever
to finish it (It showed 3% after 10 minutes). This is what gdalinfo
says:

gdalinfo /tmp/ESP_035583_2060_RED.JP2
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
Files: /tmp/ESP_035583_2060_RED.JP2
/tmp/ESP_035583_2060_RED.JP2.aux.xml
Size is 13094, 26340
Coordinate System is:
PROJCS[“Equirectangular MARS”,
GEOGCS[“GCS_MARS”,
DATUM[“unknown”,
SPHEROID[“unnamed”,3392593.6110435,0]],
PRIMEM[“Greenwich”,0],
UNIT[“degree”,0.0174532925199433]],
PROJECTION[“Equirectangular”],
PARAMETER[“latitude_of_origin”,25],
PARAMETER[“central_meridian”,180],
PARAMETER[“standard_parallel_1”,0],
PARAMETER[“false_easting”,0],
PARAMETER[“false_northing”,0],
UNIT[“metre”,1,
AUTHORITY[“EPSG”,“9001”]]]
Origin = (3889293.250000000000000,1519463.750000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Corner Coordinates:
Upper Left ( 3889293.250, 1519463.750) (114d18’56.57"W, 50d39’41.21"N)
Lower Left ( 3889293.250, 1506293.750) (114d18’56.57"W, 50d26’20.49"N)
Upper Right ( 3895840.250, 1519463.750) (114d12’18.52"W, 50d39’41.21"N)
Lower Right ( 3895840.250, 1506293.750) (114d12’18.52"W, 50d26’20.49"N)
Center ( 3892566.750, 1512878.750) (114d15’37.55"W, 50d33’ 0.85"N)
Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Undefined
Overviews: 6547x13170, 3273x6585, 1636x3292, 818x1646, 409x823,
204x411, 102x205
Overviews: arbitrary
Image Structure Metadata:
NBITS=10

GRASS location was created from this file:

projection: 99 (Equirectangular MARS)
zone: 0
datum: ** unknown (default: WGS84) **
ellipsoid: a=3392593.6110435 es=0
north: 1519463.75
south: 1506293.75
west: 3889293.25
east: 3895840.25
nsres: 0.5
ewres: 0.5
rows: 26340
cols: 13094
cells: 344895960

I tried to open it in QGIS and it appeared instantly. Then, I tried to
export it from QGIS as Geotiff, but that took again very long (I
didn’t wait for it to finish). Running gdal_translate in command line
can’t finish either. I tried to debug r.in.gdal and it seems to be
stuck on line 1103 in GDALRasterIO call. Any advice? I can send the
data privately.

Thank you,

Anna

grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

neteler · November 19, 2016, 12:07pm

On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth <spareeth@gmail.com> wrote:

Hi Anna

I am not sure if it is related, but I had similar issues with Sentinel JP2
files.

.. me too, but...:

The solution was to use the ECW library to read the JP2 files instead of
JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.

I had to recompile gdal to add the ECW support.

I did that for some weeks, then dropped it (also because of the
license situation and the packaging pain).
Recent GDAL seems to be faster due to driver improvements.

Anna: which GAL version do you use?

Markus

annakrat · November 19, 2016, 1:08pm

On Sat, Nov 19, 2016 at 7:07 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth <spareeth@gmail.com> wrote:

Hi Anna

I am not sure if it is related, but I had similar issues with Sentinel JP2
files.

.. me too, but...:

The solution was to use the ECW library to read the JP2 files instead of
JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.

I had to recompile gdal to add the ECW support.

I did that for some weeks, then dropped it (also because of the
license situation and the packaging pain).
Recent GDAL seems to be faster due to driver improvements.

Anna: which GAL version do you use?

Markus

GDAL 1.11.3, released 2015/09/16, from Ubuntu 16.04 packages

I wonder why QGIS was able to read it without problems? I would think
they use GDAL too?

I'll try to get the latest GDAL and test again. Thanks both of you.

Anna

neteler · November 19, 2016, 3:55pm

On Nov 19, 2016 2:08 PM, “Anna Petrášová” <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 7:07 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth <spareeth@gmail.com> wrote:

Hi Anna

I am not sure if it is related, but I had similar issues with Sentinel JP2
files.

… me too, but…:

The solution was to use the ECW library to read the JP2 files instead of
JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.

I had to recompile gdal to add the ECW support.

I did that for some weeks, then dropped it (also because of the
license situation and the packaging pain).
Recent GDAL seems to be faster due to driver improvements.

Anna: which GAL version do you use?

Markus

GDAL 1.11.3, released 2015/09/16, from Ubuntu 16.04 packages

That’s a bit oldish for JP2000.

I wonder why QGIS was able to read it without problems? I would think
they use GDAL too?

Did you set the cache? Eg

export GDAL_CACHEMAX=2000

Markus

annakrat · November 19, 2016, 5:17pm

On Sat, Nov 19, 2016 at 10:55 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 2:08 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 7:07 AM, Markus Neteler <neteler@osgeo.org> wrote:
> On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth <spareeth@gmail.com>
> wrote:
>> Hi Anna
>>
>> I am not sure if it is related, but I had similar issues with Sentinel
>> JP2
>> files.
>
> .. me too, but...:
>
>> The solution was to use the ECW library to read the JP2 files instead
>> of
>> JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.
>>
>> I had to recompile gdal to add the ECW support.
>
> I did that for some weeks, then dropped it (also because of the
> license situation and the packaging pain).
> Recent GDAL seems to be faster due to driver improvements.
>
> Anna: which GAL version do you use?
>
> Markus

GDAL 1.11.3, released 2015/09/16, from Ubuntu 16.04 packages

That's a bit oldish for JP2000.

I wonder why QGIS was able to read it without problems? I would think
they use GDAL too?

Did you set the cache? Eg

export GDAL_CACHEMAX=2000

I installed GDAL 2.1.2 and set the cachemax, but it doesn't help, it's
still slow. I might try the ECW then.

Markus

neteler · November 19, 2016, 7:20pm

On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 10:55 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 2:08 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

...

Did you set the cache? Eg

export GDAL_CACHEMAX=2000

I installed GDAL 2.1.2 and set the cachemax, but it doesn't help, it's
still slow. I might try the ECW then.

(got the file offlist)

So, the file size is 154M. My test:

gdal-config --version
2.0.2

gdalinfo ESP_035583_2060_RED.JP2 | grep driver
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library

time -p grass72 -c ESP_035583_2060_RED.JP2 ~/grassdata/mars_dem --exec
r.import input=ESP_035583_2060_RED.JP2 output=ESP_035583_2060_RED
Cleaning up temporary files...
Creating new GRASS GIS location/mapset...
Executing <r.import input=ESP_035583_2060_RED.JP2
output=ESP_035583_2060_RED> ...
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_035583_2060_RED>...

.... it takes forever.

Note:
The difference with QGIS is here that we import while QGIS is simply
viewing it using the image pyramids which are useless when it comes to
import.

Can you make a test with GDAL 2.1?

Markus

annakrat · November 19, 2016, 10:55pm

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 10:55 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 2:08 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

...

Did you set the cache? Eg

export GDAL_CACHEMAX=2000

I installed GDAL 2.1.2 and set the cachemax, but it doesn't help, it's
still slow. I might try the ECW then.

(got the file offlist)

So, the file size is 154M. My test:

gdal-config --version
2.0.2

gdalinfo ESP_035583_2060_RED.JP2 | grep driver
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library

time -p grass72 -c ESP_035583_2060_RED.JP2 ~/grassdata/mars_dem --exec
r.import input=ESP_035583_2060_RED.JP2 output=ESP_035583_2060_RED
Cleaning up temporary files...
Creating new GRASS GIS location/mapset...
Executing <r.import input=ESP_035583_2060_RED.JP2
output=ESP_035583_2060_RED> ...
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_035583_2060_RED>...

.... it takes forever.

Note:
The difference with QGIS is here that we import while QGIS is simply
viewing it using the image pyramids which are useless when it comes to
import.

Can you make a test with GDAL 2.1?

See above, I already tested with 2.1.2 (released in October).

Markus

neteler · November 20, 2016, 8:02am

On Nov 19, 2016 11:55 PM, “Anna Petrášová” <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
…
See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to r.in.gdal?

Markus

annakrat · November 20, 2016, 8:17pm

On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 11:55 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <neteler@osgeo.org> wrote:
> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová

...

See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to
r.in.gdal?

I haven't measured it, but it seems very slow as well. Do you think I
should make a ticket for that for GDAL?

Markus

neteler · November 20, 2016, 8:37pm

On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 11:55 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <neteler@osgeo.org> wrote:
> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová

...

See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to
r.in.gdal?

I haven't measured it, but it seems very slow as well. Do you think I
should make a ticket for that for GDAL?

This would be good, to shed some light here.

If needed, here a Mars DEM for the ticket to simplify things for them:
http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

Markus

annakrat · November 20, 2016, 8:47pm

On Sun, Nov 20, 2016 at 3:37 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 11:55 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <neteler@osgeo.org> wrote:
> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová

...

See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to
r.in.gdal?

I haven't measured it, but it seems very slow as well. Do you think I
should make a ticket for that for GDAL?

This would be good, to shed some light here.

If needed, here a Mars DEM for the ticket to simplify things for them:
http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

Yes, I got that file from somebody, but I just found these too. Thank you

Markus

hellik · November 20, 2016, 9:13pm

Markus Neteler wrote

On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová <

kratochanna@

> wrote:

On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler <

neteler@

> wrote:

On Nov 19, 2016 11:55 PM, "Anna Petrášová" <

kratochanna@

> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <

neteler@

> wrote:

> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová

...

See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to
r.in.gdal?

I haven't measured it, but it seems very slow as well. Do you think I
should make a ticket for that for GDAL?

This would be good, to shed some light here.

If needed, here a Mars DEM for the ticket to simplify things for them:
http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

just tested one of the smaller ones:

ESP_020673_1750_RED_C_01_ORTHO.JP2

r.import input=D:\dl\mars\ESP_020673_1750_RED_C_01_ORTHO.JP2
output=ESP_020673_1750_RED_C_01_ORTHO
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_020673_1750_RED_C_01_ORTHO>...
(Sun Nov 20 22:09:07 2016) Command finished (11 sec)

by

System Info
GRASS version: 7.3.svn
GRASS SVN revision: r69851
Build date: 2016-11-20
Build platform: x86_64-w64-mingw32
GDAL: 2.1.2
PROJ.4: 4.9.3
GEOS: 3.5.0
SQLite: 3.14.1
Python: 2.7.5
wxPython: 2.8.12.1
Platform: Windows-8-6.2.9200 (OSGeo4W)

no issues (slow or other) here.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296680.html
Sent from the Grass - Users mailing list archive at Nabble.com.

annakrat · November 20, 2016, 9:49pm

On Sun, Nov 20, 2016 at 4:13 PM, Helmut Kudrnovsky <hellik@web.de> wrote:

Markus Neteler wrote

On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová <

kratochanna@

> wrote:

On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler <

neteler@

> wrote:

On Nov 19, 2016 11:55 PM, "Anna Petrášová" <

kratochanna@

> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <

neteler@

> wrote:

> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová

...

See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to
r.in.gdal?

I haven't measured it, but it seems very slow as well. Do you think I
should make a ticket for that for GDAL?

This would be good, to shed some light here.

If needed, here a Mars DEM for the ticket to simplify things for them:
http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

just tested one of the smaller ones:

ESP_020673_1750_RED_C_01_ORTHO.JP2

r.import input=D:\dl\mars\ESP_020673_1750_RED_C_01_ORTHO.JP2
output=ESP_020673_1750_RED_C_01_ORTHO
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_020673_1750_RED_C_01_ORTHO>...
(Sun Nov 20 22:09:07 2016) Command finished (11 sec)

by

System Info
GRASS version: 7.3.svn
GRASS SVN revision: r69851
Build date: 2016-11-20
Build platform: x86_64-w64-mingw32
GDAL: 2.1.2
PROJ.4: 4.9.3
GEOS: 3.5.0
SQLite: 3.14.1
Python: 2.7.5
wxPython: 2.8.12.1
Platform: Windows-8-6.2.9200 (OSGeo4W)

no issues (slow or other) here.

11 seconds can be pretty long depending on the size of the file. Try
to convert it to tif and load again, it will be probably much faster.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296680.html
Sent from the Grass - Users mailing list archive at Nabble.com.
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

neteler · November 20, 2016, 9:54pm

On Sun, Nov 20, 2016 at 9:37 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová <kratochanna@gmail.com> wrote:

On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Nov 19, 2016 11:55 PM, "Anna Petrášová" <kratochanna@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler <neteler@osgeo.org> wrote:
> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová

...

See above, I already tested with 2.1.2 (released in October).

FWIW: I have recompiled GDAL 2.1 for Fedora (extra repo):
https://copr.fedorainfracloud.org/coprs/neteler/GDAL/

At least Sentinel-2 is easier to read now

The Mars DEM remains slow also then (as you observed, too) with the
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library

Markus

hellik · November 20, 2016, 10:02pm

11 seconds can be pretty long depending on the size of the file. Try
to convert it to tif and load again, it will be probably much faster.

the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).

the import takes about 7 seconds.

there is some difference in importing time ; maybe cause of compressed vs.
uncompressed.

it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
driver.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296685.html
Sent from the Grass - Users mailing list archive at Nabble.com.

annakrat · November 20, 2016, 10:38pm

On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky <hellik@web.de> wrote:

11 seconds can be pretty long depending on the size of the file. Try
to convert it to tif and load again, it will be probably much faster.

the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).

the import takes about 7 seconds.

there is some difference in importing time ; maybe cause of compressed vs.
uncompressed.

it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
driver.

I just tested it on in wingrass both standalone and from OSGeo4W and
it's very slow with ~50MB JP2 file. I tested this one:
http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_020528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2

How could you import it in 11 s? Maybe the problem appears only with
certain files?

Anna

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296685.html
Sent from the Grass - Users mailing list archive at Nabble.com.
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

hellik · November 20, 2016, 11:24pm

Anna Petrášová wrote

On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky <

hellik@

> wrote:

11 seconds can be pretty long depending on the size of the file. Try
to convert it to tif and load again, it will be probably much faster.

the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).

the import takes about 7 seconds.

there is some difference in importing time ; maybe cause of compressed
vs.
uncompressed.

it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
driver.

I just tested it on in wingrass both standalone and from OSGeo4W and
it's very slow with ~50MB JP2 file. I tested this one:
http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_020528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2

How could you import it in 11 s? Maybe the problem appears only with
certain files?

tested your file with
------------------------------------------
System Info
GRASS version: 7.3.svn
GRASS SVN revision: r69851
Build date: 2016-11-20
Build platform: x86_64-w64-mingw32
GDAL: 2.1.2
PROJ.4: 4.9.3
GEOS: 3.5.0
SQLite: 3.14.1
Python: 2.7.5
wxPython: 2.8.12.1
Platform: Windows-8-6.2.9200 (OSGeo4W)
------------------------------------------
gdalinfo --formats | grep JPEG2000
JP2ECW -raster,vector- (rov): ERDAS JPEG2000 (SDK 5.0)
JP2MrSID -raster- (rov): MrSID JPEG2000
------------------------------------------
gdal_translate -co COMPRESS=LZW -co PREDICTOR=2
ESP_020528_1750_RED_C_01_ORTHO.JP2
ESP_020528_1750_RED_C_01_ORTHO_compressed.tif
Input file size is 7552, 19113
0...10...20...30...40...50...60...70...80...90...100 - done.
------------------------------------------
gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2
ESP_020528_1750_RED_C_01_ORTHO_uncompressed.tif
Input file size is 7552, 19113
0...10...20...30...40...50...60...70...80...90...100 - done.
------------------------------------------
ls -l
total 268597
-rw-r--r-- 1 49609275 Nov 20 22:58 ESP_020528_1750_RED_C_01_ORTHO.JP2
-rw-r--r-- 1 80937523 Nov 20 23:00
ESP_020528_1750_RED_C_01_ORTHO_compressed.tif
-rw-r--r-- 1 144495106 Nov 20 23:01
ESP_020528_1750_RED_C_01_ORTHO_uncompressed.tif
------------------------------------------
r.import input=D:\dl\mars\ESP_020528_1750_RED_C_01_ORTHO.JP2
output=ESP_020528_1750_RED_C_01_ORTHO
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_020528_1750_RED_C_01_ORTHO>...
(Sun Nov 20 23:59:03 2016) Command finished (11 sec)
------------------------------------------
r.import input=D:\dl\mars\ESP_020528_1750_RED_C_01_ORTHO_compressed.tif
output=ESP_020528_1750_RED_C_01_ORTHO_compressed
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_020528_1750_RED_C_01_ORTHO_compressed>...
(Mon Nov 21 00:04:39 2016) Command finished (6 sec)
------------------------------------------
r.import input=D:\dl\mars\ESP_020528_1750_RED_C_01_ORTHO_uncompressed.tif
output=ESP_020528_1750_RED_C_01_ORTHO_uncompressed
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
WARNING: Datum <unknown> not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map <ESP_020528_1750_RED_C_01_ORTHO_uncompressed>...
(Mon Nov 21 00:05:41 2016) Command finished (5 sec)
------------------------------------------

no special tuning here on my side, just used the OSGeo4W settings on a 8GB
RAM win 10 box.

there are differences in file size (compressed vs. uncompresssed) and
importing time (also compressed vs. uncompressed).

just have a look at GDAL website regarding jpeg2000 driver implementations
(http://www.gdal.org/formats_list.html):

JPEG2000 (.jp2, .j2k) JPEG2000 Yes Yes 2GiB No, needs libjasper
JPEG2000 (.jp2, .j2k) JP2ECW Yes Yes 500MB No, needs ECW SDK
JPEG2000 (.jp2, .j2k) JP2KAK Yes Yes No limits No, needs Kakadu library
JPEG2000 (.jp2, .j2k) JP2MrSID Yes Yes No, needs MrSID SDK
JPEG2000 (.jp2, .j2k) JP2OpenJPEG Yes Yes No, needs OpenJPEG library
(v2)

OSGeo4W here on my side uses ERDAS JPEG2000 (SDK 5.0)/MrSID JPEG2000; thus
there may be perfomance issues between these two and JP2OpenJPEG?

still worth to ask on the GDAL ML.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296687.html
Sent from the Grass - Users mailing list archive at Nabble.com.

rouault · November 20, 2016, 11:32pm

On dimanche 20 novembre 2016 17:38:19 CET Anna Petrášová wrote:

On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky <hellik@web.de> wrote:
>>11 seconds can be pretty long depending on the size of the file. Try
>>to convert it to tif and load again, it will be probably much faster.
>>
> the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
>
> the import takes about 7 seconds.
>
> there is some difference in importing time ; maybe cause of compressed vs.
> uncompressed.
>
> it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
> driver.

I just tested it on in wingrass both standalone and from OSGeo4W and
it's very slow with ~50MB JP2 file. I tested this one:
http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_02
0528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2

How could you import it in 11 s? Maybe the problem appears only with
certain files?

When enabling GDAL debug messages (CPL_DEBUG=ON), one can see:

OPENJPEG: nX0 = 0
OPENJPEG: nY0 = 0
OPENJPEG: nTileW = 7552
OPENJPEG: nTileH = 19113
OPENJPEG: nTilesX = 1
OPENJPEG: nTilesY = 1
OPENJPEG: mct = 0
OPENJPEG: psImage->x0 = 0
OPENJPEG: psImage->y0 = 0
OPENJPEG: psImage->x1 = 7552
OPENJPEG: psImage->y1 = 19113
OPENJPEG: psImage->numcomps = 1

The tile dimensions are the same as the image, meaning the image is single
tiled.
The OpenJPEG library must thus decompress the whole time into memory since it
doesn't support decompressing a region of interest within a tile. To avoid
exposing a block size so large, GDAL exposes a 1024x1024 artificial blocking,
which will in fact the image to be decompressed each time one of those blocks
is read.

I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES
open option that can help with whole image conversion"
thats add the open option :

USE_TILE_AS_BLOCK=YES/NO</b>: (GDAL > 2.2) Whether to always use the
JPEG-2000 block size as the GDAL block size
Defaults to NO. Setting this option can be useful when doing whole image
decompression and the image is single-tiled. Note however that the tile size
must not exceed 2 GB since that's the limit supported by GDAL.</li>

(note: this limitation of 2GB is reached unfortunately with other images as
ESP_020528_1750_IRB_A_01_ORTHO.JP2)

$ time GDAL_CACHEMAX=150 gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2
out.tif -co TILED=YES -oo USE_TILE_AS_BLOCK=YES -co COMPRESS=DEFLATE
real 0m22.981s
user 0m22.756s
sys 0m0.220s

GDAL_CACHEMAX must be larger than width * height * data_type_size *
number_of_bands = 7552 * 19113 * 1 * 1 = 144 MB, so as to avoid the block
corresponding to the JPEG-2000 image to be evicted from the GDAL block cache.

Even

Anna

> -----
> best regards
> Helmut
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5
> 296530p5296685.html Sent from the Grass - Users mailing list archive at
> Nabble.com.
> _______________________________________________
> grass-user mailing list
> grass-user@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

--
Spatialys - Geospatial professional services
http://www.spatialys.com

hellik · November 20, 2016, 11:42pm

Even Rouault-2 wrote

On dimanche 20 novembre 2016 17:38:19 CET Anna Petrášová wrote:

On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky <

hellik@

> wrote:

>>11 seconds can be pretty long depending on the size of the file. Try
>>to convert it to tif and load again, it will be probably much faster.
>>
> the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
>
> the import takes about 7 seconds.
>
> there is some difference in importing time ; maybe cause of compressed
vs.
> uncompressed.
>
> it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
> driver.

I just tested it on in wingrass both standalone and from OSGeo4W and
it's very slow with ~50MB JP2 file. I tested this one:
http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_02
0528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2

How could you import it in 11 s? Maybe the problem appears only with
certain files?

When enabling GDAL debug messages (CPL_DEBUG=ON), one can see:

OPENJPEG: nX0 = 0
OPENJPEG: nY0 = 0
OPENJPEG: nTileW = 7552
OPENJPEG: nTileH = 19113
OPENJPEG: nTilesX = 1
OPENJPEG: nTilesY = 1
OPENJPEG: mct = 0
OPENJPEG: psImage->x0 = 0
OPENJPEG: psImage->y0 = 0
OPENJPEG: psImage->x1 = 7552
OPENJPEG: psImage->y1 = 19113
OPENJPEG: psImage->numcomps = 1

The tile dimensions are the same as the image, meaning the image is single
tiled.
The OpenJPEG library must thus decompress the whole time into memory since
it
doesn't support decompressing a region of interest within a tile. To avoid
exposing a block size so large, GDAL exposes a 1024x1024 artificial
blocking,
which will in fact the image to be decompressed each time one of those
blocks
is read.

I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES
open option that can help with whole image conversion"
thats add the open option :

USE_TILE_AS_BLOCK=YES/NO

*

: (GDAL > 2.2) Whether to always use the
JPEG-2000 block size as the GDAL block size
Defaults to NO. Setting this option can be useful when doing whole image
decompression and the image is single-tiled. Note however that the tile
size
must not exceed 2 GB since that's the limit supported by GDAL.
</li>
(note: this limitation of 2GB is reached unfortunately with other images
as
ESP_020528_1750_IRB_A_01_ORTHO.JP2)

$ time GDAL_CACHEMAX=150 gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2
out.tif -co TILED=YES -oo USE_TILE_AS_BLOCK=YES -co COMPRESS=DEFLATE
real 0m22.981s
user 0m22.756s
sys 0m0.220s

GDAL_CACHEMAX must be larger than width * height * data_type_size *
number_of_bands = 7552 * 19113 * 1 * 1 = 144 MB, so as to avoid the block
corresponding to the JPEG-2000 image to be evicted from the GDAL block
cache.

Even, thanks for the insight and improvements on GDAL's level!

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296689.html
Sent from the Grass - Users mailing list archive at Nabble.com.