[GRASS-user] Viewing ecw files

GRASS GRASS user
1

Hello all
I have tried to import some ecw files into grass and with some exceptions (problems reading some lines from input) I can do it. But main problem I have is that I use ecw files over 700 MB each one and they become more than 7 GB once imported. You can imagine the amount of disk space it takes to work with 10-12 maps of such size and how slow everything goes. Is there any way to see these maps without importing them or with a smaller output size?

It's absolutely necessary for me to use these maps, so I thank any solution.

Carlos

2

Carlos Dávila wrote:

Hello all
I have tried to import some ecw files into grass and with some exceptions (problems reading some lines from input) I can do it. But main problem I have is that I use ecw files over 700 MB each one and they become more than 7 GB once imported. You can imagine the amount of disk space it takes to work with 10-12 maps of such size and how slow everything goes. Is there any way to see these maps without importing them or with a smaller output size?

It's absolutely necessary for me to use these maps, so I thank any solution.

Carlos,

Well, for simple visualization you might want to use a viewer package
like QGIS or OpenEV. The linux and win32 OpenEV in FWTools includes ECW
support.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

3

Carlos Dávila wrote:
> I have tried to import some ecw files into grass and with some
> exceptions (problems reading some lines from input) I can do it. But
> main problem I have is that I use ecw files over 700 MB each one and
> they become more than 7 GB once imported. You can imagine the amount
> of disk space it takes to work with 10-12 maps of such size and how
> slow everything goes. Is there any way to see these maps without
> importing them or with a smaller output size?
>
> It's absolutely necessary for me to use these maps, so I thank any
> solution.

Frank Warmerdam wrote:

Well, for simple visualization you might want to use a viewer package
like QGIS or OpenEV. The linux and win32 OpenEV in FWTools includes
ECW support.

I would add that if you like you can use "gdal_translate -srcwin" to
extract a small section into a GeoTIFF.

Hamish

4

On Tue, 2007-02-13 at 17:42 +0100, Carlos Dávila wrote:

Hello all
I have tried to import some ecw files into grass and with some
exceptions (problems reading some lines from input) I can do it. But
main problem I have is that I use ecw files over 700 MB each one and
they become more than 7 GB once imported. You can imagine the amount of
disk space it takes to work with 10-12 maps of such size and how slow
everything goes. Is there any way to see these maps without importing
them or with a smaller output size?

It's absolutely necessary for me to use these maps, so I thank any solution.

It would be really nice if GRASS had a wavelet based scheme for imagery,
but that would require significant development and many of us don't have
the spare time available. At the moment, Hamish's suggestion is about
the best that can be done. GRASS cannot handle such large datasets with
efficiency.

ECW is a wonderful format. It is hard to beat the compression and speed
of scale change.

--
Brad Douglas <rez touchofmadness com> KB8UYR/6
Address: 37.493,-121.924 / WGS84 National Map Corps #TNMC-3785

5

It would be really nice if GRASS had a wavelet based scheme for imagery,
but that would require significant development and many of us don't have
the spare time available. At the moment, Hamish's suggestion is about
the best that can be done. GRASS cannot handle such large datasets with
efficiency.

ECW is a wonderful format. It is hard to beat the compression and speed
of scale change.

Isn't Ogg Theora working on wavelet compression for imagery?

nick

************************************************************
Opinions contained in this e-mail do not necessarily reflect
the opinions of the Queensland Department of Main Roads,
Queensland Transport or Maritime Safety Queensland, or
endorsed organisations utilising the same infrastructure.
If you have received this electronic mail message in error,
please immediately notify the sender and delete the message
from your computer.
************************************************************

6

nicholas.g.lawrence@mainroads.qld.gov.au wrote:

It would be really nice if GRASS had a wavelet based scheme for imagery,
but that would require significant development and many of us don't have
the spare time available. At the moment, Hamish's suggestion is about
the best that can be done. GRASS cannot handle such large datasets with
efficiency.

ECW is a wonderful format. It is hard to beat the compression and speed
of scale change.

Isn't Ogg Theora working on wavelet compression for imagery?

Do you mean Tarkin? That's wavelet compression for video. Is it usable
for still images too?

Maciek

7

On Thu, 2007-02-15 at 20:55 +0100, Maciej Sieczka wrote:

nicholas.g.lawrence@mainroads.qld.gov.au wrote:
>> It would be really nice if GRASS had a wavelet based scheme for imagery,
>> but that would require significant development and many of us don't have
>> the spare time available. At the moment, Hamish's suggestion is about
>> the best that can be done. GRASS cannot handle such large datasets with
>> efficiency.
>
>> ECW is a wonderful format. It is hard to beat the compression and speed
>> of scale change.
>
> Isn't Ogg Theora working on wavelet compression for imagery?

Do you mean Tarkin? That's wavelet compression for video. Is it usable
for still images too?

No, it's not appropriate for GIS. Wavelets can be highly tunable to the
application at hand. We want more of what wavelets have to offer than
just compression.

I did notice that there is an add-on r.wavelet module, but I have not
used it. The backend library seems to be license-free and GPL, IIRC.

Wavelets for GRASS as an imagery format is "pie in the sky". :-\ There
are too many other issues that need attention at the moment.

I'd be interested in what Glynn's opinion is. It may be more trouble
that its worth (we'd need someone dedicated to implementing and
supporting it). However, I do see it becoming increasingly popular at
municipalities who regularly deal with very large imagery sets (10G-1T
sizes) and not being able to support a comparable format is a major
disadvantage.

Beyond getting Glynn's input, it's probably a waste of time discussing
it further unless someone stands up and offers their expertise and time
to implement it as an optional format (it is not appropriate for
non-contiguous rasters). Even so, I'd still recommend that person work
on other pressing issues before adding a new imagery format.

--
Brad Douglas <rez touchofmadness com> KB8UYR/6
Address: 37.493,-121.924 / WGS84 National Map Corps #TNMC-3785

8

> Nicholas wrote:
> >> It would be really nice if GRASS had a wavelet based scheme for
> >> imagery, but that would require significant development and many
> >> of us don't have the spare time available. At the moment,
> >> Hamish's suggestion is about the best that can be done. GRASS
> >> cannot handle such large datasets with efficiency.
> >>
> >> ECW is a wonderful format. It is hard to beat the compression
> >> and speed of scale change.

Brad:

I did notice that there is an add-on r.wavelet module, but I have not
used it. The backend library seems to be license-free and GPL, IIRC.

Wavelets for GRASS as an imagery format is "pie in the sky". :-\
There are too many other issues that need attention at the moment.

maybe a r.external module could read/write GDAL formats (JPEG2000,
ECW, MrSID..) as v.external does for OGR formats?

A simple script using gdal_translate to extract and import the current
region (read-only) could be written in a few minutes. (r.in.gdal flag?)

FWIW, ER Mapper's ECW SDK page states "Free unlimited compression use in
GPL style software". http://www.ermapper.com/ProductView.aspx?t=28

Also that page claims that there is a lossless version available- which
is a requirement for us.

Hamish

9

On Fri, 2007-02-16 at 12:03 +1300, Hamish wrote:

> > Nicholas wrote:
> > >> It would be really nice if GRASS had a wavelet based scheme for
> > >> imagery, but that would require significant development and many
> > >> of us don't have the spare time available. At the moment,
> > >> Hamish's suggestion is about the best that can be done. GRASS
> > >> cannot handle such large datasets with efficiency.
> > >>
> > >> ECW is a wonderful format. It is hard to beat the compression
> > >> and speed of scale change.
Brad:
> I did notice that there is an add-on r.wavelet module, but I have not
> used it. The backend library seems to be license-free and GPL, IIRC.
>
> Wavelets for GRASS as an imagery format is "pie in the sky". :-\
> There are too many other issues that need attention at the moment.

Hamish:
maybe a r.external module could read/write GDAL formats (JPEG2000,
ECW, MrSID..) as v.external does for OGR formats?

A simple script using gdal_translate to extract and import the current
region (read-only) could be written in a few minutes. (r.in.gdal flag?)

The biggest problem is that when we import ECW in, it gets exploded to
very large sizes. This is especially true for lossy compression where
ratios can reach > 50:1.

FWIW, ER Mapper's ECW SDK page states "Free unlimited compression use in
GPL style software". http://www.ermapper.com/ProductView.aspx?t=28

Also that page claims that there is a lossless version available- which
is a requirement for us.

What they offer, we cannot use reliably as an internal format. They
have competing products and we would be subject to their whim (they
reserve the right to modify the license as they see fit). There are
some great people in the company who recognize the OSS community, but
there are also those who do not and are combative about the topic.

ECW complies with the JPEG2000 format, which is patented. This is the
crux of the various JP2K codecs. I don't know what current status of
the patent is. Maybe it has been re-licensed to public domain or
similar or if there are cases that require a license. Maybe someone can
research into that. It does appear that there is some way around it.

I also think GeoJasPer[1] would be preferable to ECW, given our needs.
I'd much rather use a format that was built with OSS in mind, not as an
afterthought.

[1] http://www.dimin.net/software/geojasper/

--
Brad Douglas <rez touchofmadness com> KB8UYR/6
Address: 37.493,-121.924 / WGS84 National Map Corps #TNMC-3785

10

Brad Douglas wrote:

> >> It would be really nice if GRASS had a wavelet based scheme for imagery,
> >> but that would require significant development and many of us don't have
> >> the spare time available. At the moment, Hamish's suggestion is about
> >> the best that can be done. GRASS cannot handle such large datasets with
> >> efficiency.
> >
> >> ECW is a wonderful format. It is hard to beat the compression and speed
> >> of scale change.
> >
> > Isn't Ogg Theora working on wavelet compression for imagery?
>
> Do you mean Tarkin? That's wavelet compression for video. Is it usable
> for still images too?

No, it's not appropriate for GIS. Wavelets can be highly tunable to the
application at hand. We want more of what wavelets have to offer than
just compression.

I did notice that there is an add-on r.wavelet module, but I have not
used it. The backend library seems to be license-free and GPL, IIRC.

Wavelets for GRASS as an imagery format is "pie in the sky". :-\ There
are too many other issues that need attention at the moment.

I'd be interested in what Glynn's opinion is. It may be more trouble
that its worth (we'd need someone dedicated to implementing and
supporting it). However, I do see it becoming increasingly popular at
municipalities who regularly deal with very large imagery sets (10G-1T
sizes) and not being able to support a comparable format is a major
disadvantage.

Beyond getting Glynn's input, it's probably a waste of time discussing
it further unless someone stands up and offers their expertise and time
to implement it as an optional format (it is not appropriate for
non-contiguous rasters). Even so, I'd still recommend that person work
on other pressing issues before adding a new imagery format.

The main requirement for the existing raster I/O code is that you need
to be able to read rows in an arbitrary order without needing to store
the entire uncompressed data in memory. It wouldn't be particularly
hard to generalise the lower levels of the raster input code to allow
reading from any format which supports seeking (i.e. any uncompressed
format, or any compressed format which uses independently-compressed
segments of a reasonable size).

Writing isn't an issue; if you want to be able to write rows in random
order, you have to assert this when you create the map, by using
G_open_cell_new_random() (in 6.x, nothing actually uses random
output).

However, when reading maps, there's no way for the library to tell
whether the module will read sequentially or randomly, so it has to be
able to cope with random access.

[Assuming sequential I/O then failing the first time that the client
seeks backwards is a possibility, but somewhat less than ideal.]

For the future, any new raster API should require the client to
provide some indictation of the access strategy at open time. Formats
which use a single compressed data stream wouldn't allow arbitrary
seeking, although it would be possible to skip forwards (just discard
unneeded data) and to skip backwards up to a pre-selected limit
(allocate a sufficient cache).

--
Glynn Clements <glynn@gclements.plus.com>