[Geoserver-devel] Compression format support for Geotools/UDig

Ciao Cameron,
I remember talking to Shaun about issues related to ECW support.

Long story short.
The work on interfacing GDAL Java bindings to ImageIO/JAI and then straight
into GeoTools, started as an experiment during last summer under the hat of
the GSOC 2006. The student working on it under having me as a mentor (poor
guy! :slight_smile: ) was Daniele Romagnoli. The outcome at the end of the summer was
that we managed to have a decent ECW ImageReader and an experimental HDF4
reader (well HDF format is always a pain).

After a couple of months when both me and Daniele for various reasons had no
time to work on the project we brought it back to life at the beginning of
November 2006, starting to think about adding writing capabilities.

Since then we have spent like 20% of our time working on the library to
extend it to other formats and to add other capabilities beyond what you can
do using only gdal.

The library.
The structure of the library right now is as follows:

+library
  +gdalframework
    General framework to communicate with gdal swig bindings
  +customstreams
      A large set of high performance java streams
+plugin
  +arcgrid
    An ImageIO reader for ascii grids based on gdal bindings, it
should be already pretty usable.
  +directkakadu
    A simple ImageIO reader for jp2 files based on kakadu
library directly, needs s ome work
  +ecw
    An imageio reader for ECW file format through gdal, it
should be already pretty usable.
  +hdf
    An imageio reader for HDF4 file format through gdal, need
some tuning
  +javaarcgrid
    A pure java ImageIO reader for ascii grids, ready.
  +jhdf
    An experimental plugin for reading and writing hdf4/5 files
using drectly the jhdf lib
  +jp2kakadu
    An ImageIO reader and writer for jp2 files using kakadu
through gdal, very good state. I have used it to compress
1.5 gb geotiff with no problems.
  +jpeg
    An ImageIO reader for jpeg files using gdal, it should be
already pretty usable.
  +mrsid
    A smple Imageio erader for mrsid files using gdal, it should
be already pretty usable.
  +geotiff
    An ImageIO reader and writer for geotiff using gdal, it
should be already pretty usable.

NOTE that for formats like ECW, jp2 and mrsid, once you have an ImageIO
reader, writing geotools plugin is a matter of 2-3 days.

Now the status.
There is a lot of code already to share. There a few flaws that I wish I get
the time to work on soon:

-better handling of metadata (gdal itself is not really focused on metadata
management though there is a RFC about that right now)
-better management of writing
Writing can be tricky in gdal when it comes to driver that only support the
create copy. Right now we are relying on an ugly hack to get it going but I
am trying to think about a way to fix it for good.
-more multidimensional formats
Actually we are are going to integrate some work we did on netcdf with the
great work that other geotools developers have been doing.

Conclusion.
We are thinking about releasing as soon as possible as a fully fledged Open
Source project under LGPL in order to get some money and/or help from people
to to support this work. Maybe we didn't talk much about what we were doing
but we were pretty busy :-).

As far as talking about timing, are we talking about read-only support or
r-w? Which formats would you be interested with? Just ECW/JP2?

The best thing (at least from my standpoint) would be put together more than
one party that is interested in getting this kind of capabilities into
udig/geoserver/geotools so that would be easier to come up with some sort of
support. Note that what we do can be "easily" integrated into udig and/or
geoserver.

Regards,
Simone.

PS, sorry for cross posting!!

-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it

-------------------------------------------------------

-----Original Message-----
From: Cameron Shorter [mailto:cameron.shorter@anonymised.com]
Sent: martedì 29 maggio 2007 0.51
To: Simone Giannecchini; Christopher Schmidt; Shaun Kolomeitz
Cc: udig-devel@anonymised.com
Subject: Compression format support for Geotools/UDig

Simone,
Shaun is looking to replace Arc View 3.1 in his organisation with UDig.
One of his requirements is support for data in a compressed format such
as JPEG2000 or ECW/JPGE2000.

What is the state of development in this area?
How much development is left to be done? Ie, what level of effort is
required to get a compressed format?

Shaun is crunching numbers at the moment to see if he can put forward a
business case for switching to UDig.

Chris Holmes wrote:
<snip>
>> The other "nice" thing would be the ability to extract
>> (cookie-cut) imagery related to the project area of interest (for
taking
>> out into the field). Having this data in a compressed format such as
>> JPEG2000 or ECW/JPGE2000 would also be a plus.
> I notice there are some discussions of JPEG2000 in the java lists
which suggests that someone is
> working on it. Anyone able to give an update?

Simone is the one who's been working on it. He can likely give you more
information.

--
Cameron Shorter
Systems Architect, http://lisasoft.com.au
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

What a great email - quick put it on the wiki :slight_smile:

Seriously thanks for the review Simone / Cameron etc...

Jody

Ciao Cameron,
I remember talking to Shaun about issues related to ECW support.

Long story short.
The work on interfacing GDAL Java bindings to ImageIO/JAI and then straight
into GeoTools, started as an experiment during last summer under the hat of
the GSOC 2006. The student working on it under having me as a mentor (poor
guy! :slight_smile: ) was Daniele Romagnoli. The outcome at the end of the summer was
that we managed to have a decent ECW ImageReader and an experimental HDF4
reader (well HDF format is always a pain).

After a couple of months when both me and Daniele for various reasons had no
time to work on the project we brought it back to life at the beginning of
November 2006, starting to think about adding writing capabilities.

Since then we have spent like 20% of our time working on the library to
extend it to other formats and to add other capabilities beyond what you can
do using only gdal.

The library.
The structure of the library right now is as follows:

+library
  +gdalframework
    General framework to communicate with gdal swig bindings
  +customstreams
      A large set of high performance java streams +plugin
  +arcgrid
    An ImageIO reader for ascii grids based on gdal bindings, it
should be already pretty usable.
  +directkakadu
    A simple ImageIO reader for jp2 files based on kakadu
library directly, needs s ome work
  +ecw
    An imageio reader for ECW file format through gdal, it
should be already pretty usable.
  +hdf
    An imageio reader for HDF4 file format through gdal, need
some tuning
  +javaarcgrid
    A pure java ImageIO reader for ascii grids, ready.
  +jhdf
    An experimental plugin for reading and writing hdf4/5 files
using drectly the jhdf lib
  +jp2kakadu
    An ImageIO reader and writer for jp2 files using kakadu
through gdal, very good state. I have used it to compress
1.5 gb geotiff with no problems.
  +jpeg
    An ImageIO reader for jpeg files using gdal, it should be
already pretty usable.
  +mrsid
    A smple Imageio erader for mrsid files using gdal, it should
be already pretty usable.
  +geotiff
    An ImageIO reader and writer for geotiff using gdal, it
should be already pretty usable.

NOTE that for formats like ECW, jp2 and mrsid, once you have an ImageIO
reader, writing geotools plugin is a matter of 2-3 days.

Now the status.
There is a lot of code already to share. There a few flaws that I wish I get
the time to work on soon:

-better handling of metadata (gdal itself is not really focused on metadata
management though there is a RFC about that right now)
-better management of writing
Writing can be tricky in gdal when it comes to driver that only support the
create copy. Right now we are relying on an ugly hack to get it going but I
am trying to think about a way to fix it for good.
-more multidimensional formats
Actually we are are going to integrate some work we did on netcdf with the
great work that other geotools developers have been doing.

Conclusion.
We are thinking about releasing as soon as possible as a fully fledged Open
Source project under LGPL in order to get some money and/or help from people
to to support this work. Maybe we didn't talk much about what we were doing
but we were pretty busy :-).

As far as talking about timing, are we talking about read-only support or
r-w? Which formats would you be interested with? Just ECW/JP2?

The best thing (at least from my standpoint) would be put together more than
one party that is interested in getting this kind of capabilities into
udig/geoserver/geotools so that would be easier to come up with some sort of
support. Note that what we do can be "easily" integrated into udig and/or
geoserver.

Regards,
Simone.

PS, sorry for cross posting!!

-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions S.A.S.
Via Carignoni 51 55041 Camaiore (LU) Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it

-------------------------------------------------------

-----Original Message-----
From: Cameron Shorter [mailto:cameron.shorter@anonymised.com] Sent: martedì 29 maggio 2007 0.51
To: Simone Giannecchini; Christopher Schmidt; Shaun Kolomeitz
Cc: udig-devel@anonymised.com
Subject: Compression format support for Geotools/UDig

Simone,
Shaun is looking to replace Arc View 3.1 in his organisation with UDig. One of his requirements is support for data in a compressed format such as JPEG2000 or ECW/JPGE2000.

What is the state of development in this area?
How much development is left to be done? Ie, what level of effort is required to get a compressed format?

Shaun is crunching numbers at the moment to see if he can put forward a business case for switching to UDig.

Chris Holmes wrote:
<snip>
>> The other "nice" thing would be the ability to extract
>> (cookie-cut) imagery related to the project area of interest (for taking
>> out into the field). Having this data in a compressed format such as
>> JPEG2000 or ECW/JPGE2000 would also be a plus.
> I notice there are some discussions of JPEG2000 in the java lists which suggests that someone is
> working on it. Anyone able to give an update?
Simone is the one who's been working on it. He can likely give you more information.

Ciao Simone,
Thankyou very much for the heads up.

I believe that our essential requirement is to READ one compressed format from UDig.
Anything above that is a bonus.

Our aim is to get all this working within the OGC's OWS5 testbed framework. Ie, this year.

Sorry for being brief - I'm currently sitting in the Google Developer Day in Sydney.

Simone Gannecchini wrote:

Ciao Cameron,
I remember talking to Shaun about issues related to ECW support.

Long story short.
The work on interfacing GDAL Java bindings to ImageIO/JAI and then straight
into GeoTools, started as an experiment during last summer under the hat of
the GSOC 2006. The student working on it under having me as a mentor (poor
guy! :slight_smile: ) was Daniele Romagnoli. The outcome at the end of the summer was
that we managed to have a decent ECW ImageReader and an experimental HDF4
reader (well HDF format is always a pain).

After a couple of months when both me and Daniele for various reasons had no
time to work on the project we brought it back to life at the beginning of
November 2006, starting to think about adding writing capabilities.

Since then we have spent like 20% of our time working on the library to
extend it to other formats and to add other capabilities beyond what you can
do using only gdal.

The library.
The structure of the library right now is as follows:

+library
  +gdalframework
    General framework to communicate with gdal swig bindings
  +customstreams
      A large set of high performance java streams +plugin
  +arcgrid
    An ImageIO reader for ascii grids based on gdal bindings, it
should be already pretty usable.
  +directkakadu
    A simple ImageIO reader for jp2 files based on kakadu
library directly, needs s ome work
  +ecw
    An imageio reader for ECW file format through gdal, it
should be already pretty usable.
  +hdf
    An imageio reader for HDF4 file format through gdal, need
some tuning
  +javaarcgrid
    A pure java ImageIO reader for ascii grids, ready.
  +jhdf
    An experimental plugin for reading and writing hdf4/5 files
using drectly the jhdf lib
  +jp2kakadu
    An ImageIO reader and writer for jp2 files using kakadu
through gdal, very good state. I have used it to compress
1.5 gb geotiff with no problems.
  +jpeg
    An ImageIO reader for jpeg files using gdal, it should be
already pretty usable.
  +mrsid
    A smple Imageio erader for mrsid files using gdal, it should
be already pretty usable.
  +geotiff
    An ImageIO reader and writer for geotiff using gdal, it
should be already pretty usable.

NOTE that for formats like ECW, jp2 and mrsid, once you have an ImageIO
reader, writing geotools plugin is a matter of 2-3 days.

Now the status.
There is a lot of code already to share. There a few flaws that I wish I get
the time to work on soon:

-better handling of metadata (gdal itself is not really focused on metadata
management though there is a RFC about that right now)
-better management of writing
Writing can be tricky in gdal when it comes to driver that only support the
create copy. Right now we are relying on an ugly hack to get it going but I
am trying to think about a way to fix it for good.
-more multidimensional formats
Actually we are are going to integrate some work we did on netcdf with the
great work that other geotools developers have been doing.

Conclusion.
We are thinking about releasing as soon as possible as a fully fledged Open
Source project under LGPL in order to get some money and/or help from people
to to support this work. Maybe we didn't talk much about what we were doing
but we were pretty busy :-).

As far as talking about timing, are we talking about read-only support or
r-w? Which formats would you be interested with? Just ECW/JP2?

The best thing (at least from my standpoint) would be put together more than
one party that is interested in getting this kind of capabilities into
udig/geoserver/geotools so that would be easier to come up with some sort of
support. Note that what we do can be "easily" integrated into udig and/or
geoserver.

Regards,
Simone.

PS, sorry for cross posting!!

-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions S.A.S.
Via Carignoni 51 55041 Camaiore (LU) Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it

-------------------------------------------------------

-----Original Message-----
From: Cameron Shorter [mailto:cameron.shorter@anonymised.com] Sent: martedì 29 maggio 2007 0.51
To: Simone Giannecchini; Christopher Schmidt; Shaun Kolomeitz
Cc: udig-devel@anonymised.com
Subject: Compression format support for Geotools/UDig

Simone,
Shaun is looking to replace Arc View 3.1 in his organisation with UDig. One of his requirements is support for data in a compressed format such as JPEG2000 or ECW/JPGE2000.

What is the state of development in this area?
How much development is left to be done? Ie, what level of effort is required to get a compressed format?

Shaun is crunching numbers at the moment to see if he can put forward a business case for switching to UDig.

Chris Holmes wrote:
<snip>
>> The other "nice" thing would be the ability to extract
>> (cookie-cut) imagery related to the project area of interest (for taking
>> out into the field). Having this data in a compressed format such as
>> JPEG2000 or ECW/JPGE2000 would also be a plus.
> I notice there are some discussions of JPEG2000 in the java lists which suggests that someone is
> working on it. Anyone able to give an update?
Simone is the one who's been working on it. He can likely give you more information.

--
Cameron Shorter
Systems Architect, http://lisasoft.com.au
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

Ciao Cameron,
with a not-such-a-big effort we could easily integrate reading
capabilities for mrsid-ecw-jp2kakadu into udig by exploiting gdal,
guaranteed. The real cool thing would be trying to avoid having to
pass through gdal and concentrate on interfacing directly one of the
above mentioned format to have more freedom and exploit the full scale
of capabilities.
As an instance we could play directly with kakadu and extend what we
have already available (which was not designed for geospatial data so
it does not parse any geo info).

Let me know your opinion.
Regards,
Simone.

On 5/31/07, Cameron Shorter <cameron.shorter@anonymised.com> wrote:

Ciao Simone,
Thankyou very much for the heads up.

I believe that our essential requirement is to READ one compressed
format from UDig.
Anything above that is a bonus.

Our aim is to get all this working within the OGC's OWS5 testbed
framework. Ie, this year.

Sorry for being brief - I'm currently sitting in the Google Developer
Day in Sydney.

Simone Gannecchini wrote:
> Ciao Cameron,
> I remember talking to Shaun about issues related to ECW support.
>
> Long story short.
> The work on interfacing GDAL Java bindings to ImageIO/JAI and then straight
> into GeoTools, started as an experiment during last summer under the hat of
> the GSOC 2006. The student working on it under having me as a mentor (poor
> guy! :slight_smile: ) was Daniele Romagnoli. The outcome at the end of the summer was
> that we managed to have a decent ECW ImageReader and an experimental HDF4
> reader (well HDF format is always a pain).
>
> After a couple of months when both me and Daniele for various reasons had no
> time to work on the project we brought it back to life at the beginning of
> November 2006, starting to think about adding writing capabilities.
>
> Since then we have spent like 20% of our time working on the library to
> extend it to other formats and to add other capabilities beyond what you can
> do using only gdal.
>
> The library.
> The structure of the library right now is as follows:
>
> +library
> +gdalframework
> General framework to communicate with gdal swig bindings
> +customstreams
> A large set of high performance java streams
> +plugin
> +arcgrid
> An ImageIO reader for ascii grids based on gdal bindings, it
> should be already pretty usable.
> +directkakadu
> A simple ImageIO reader for jp2 files based on kakadu
> library directly, needs s ome work
> +ecw
> An imageio reader for ECW file format through gdal, it
> should be already pretty usable.
> +hdf
> An imageio reader for HDF4 file format through gdal, need
> some tuning
> +javaarcgrid
> A pure java ImageIO reader for ascii grids, ready.
> +jhdf
> An experimental plugin for reading and writing hdf4/5 files
> using drectly the jhdf lib
> +jp2kakadu
> An ImageIO reader and writer for jp2 files using kakadu
> through gdal, very good state. I have used it to compress
> 1.5 gb geotiff with no problems.
> +jpeg
> An ImageIO reader for jpeg files using gdal, it should be
> already pretty usable.
> +mrsid
> A smple Imageio erader for mrsid files using gdal, it should
> be already pretty usable.
> +geotiff
> An ImageIO reader and writer for geotiff using gdal, it
> should be already pretty usable.
>
> NOTE that for formats like ECW, jp2 and mrsid, once you have an ImageIO
> reader, writing geotools plugin is a matter of 2-3 days.
>
> Now the status.
> There is a lot of code already to share. There a few flaws that I wish I get
> the time to work on soon:
>
> -better handling of metadata (gdal itself is not really focused on metadata
> management though there is a RFC about that right now)
> -better management of writing
> Writing can be tricky in gdal when it comes to driver that only support the
> create copy. Right now we are relying on an ugly hack to get it going but I
> am trying to think about a way to fix it for good.
> -more multidimensional formats
> Actually we are are going to integrate some work we did on netcdf with the
> great work that other geotools developers have been doing.
>
> Conclusion.
> We are thinking about releasing as soon as possible as a fully fledged Open
> Source project under LGPL in order to get some money and/or help from people
> to to support this work. Maybe we didn't talk much about what we were doing
> but we were pretty busy :-).
>
> As far as talking about timing, are we talking about read-only support or
> r-w? Which formats would you be interested with? Just ECW/JP2?
>
> The best thing (at least from my standpoint) would be put together more than
> one party that is interested in getting this kind of capabilities into
> udig/geoserver/geotools so that would be easier to come up with some sort of
> support. Note that what we do can be "easily" integrated into udig and/or
> geoserver.
>
> Regards,
> Simone.
>
> PS, sorry for cross posting!!
>
> -------------------------------------------------------
> Eng. Simone Giannecchini
> President /CEO GeoSolutions S.A.S.
> Via Carignoni 51
> 55041 Camaiore (LU)
> Italy
>
> phone: +39 0584983027
> fax: +39 0584983027
> mob: +39 333 8128928
>
> http://www.geo-solutions.it
>
> -------------------------------------------------------
>
> -----Original Message-----
> From: Cameron Shorter [mailto:cameron.shorter@anonymised.com]
> Sent: martedì 29 maggio 2007 0.51
> To: Simone Giannecchini; Christopher Schmidt; Shaun Kolomeitz
> Cc: udig-devel@anonymised.com
> Subject: Compression format support for Geotools/UDig
>
> Simone,
> Shaun is looking to replace Arc View 3.1 in his organisation with UDig.
> One of his requirements is support for data in a compressed format such
> as JPEG2000 or ECW/JPGE2000.
>
> What is the state of development in this area?
> How much development is left to be done? Ie, what level of effort is
> required to get a compressed format?
>
> Shaun is crunching numbers at the moment to see if he can put forward a
> business case for switching to UDig.
>
> Chris Holmes wrote:
> <snip>
> >> The other "nice" thing would be the ability to extract
> >> (cookie-cut) imagery related to the project area of interest (for
> taking
> >> out into the field). Having this data in a compressed format such as
> >> JPEG2000 or ECW/JPGE2000 would also be a plus.
> > I notice there are some discussions of JPEG2000 in the java lists
> which suggests that someone is
> > working on it. Anyone able to give an update?
>
> Simone is the one who's been working on it. He can likely give you more
> information.
>

--
Cameron Shorter
Systems Architect, http://lisasoft.com.au
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

--
-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions

http://www.geo-solutions.it

-------------------------------------------------------