[Geoserver-devel] BXML data format (from Ivan Price)

GeoServer GeoServer Developer
1

Ivan has been having trouble getting his mail through, so for now I'm just
forwarding it on:

Hi there,

Back on the 24/9/2003 Cameron Shorter posted a message to this group as=
king
about BXML and whether any thought had been put into implementing BXML =
has
an output / input format to Geoserver.. was there a response to that ma=
il,
has there been any discussions here ?

I'll paste the text discussing BXML for reference below.

After (successfully) testing our Geoserver looking at Esri SDE with
Integraph Geomedia Viewer (free, but need to order the CD), and
Professional and the new version of their (free) WFS dataserver the
bulkiness of GML is really evident as you can directly compare the vari=
ous
formats to GML.. As WFS/Geoserver becomes a core business tool for us
spatial folk the XML bloat will need to be addressed.

Keep up the great work geoserver peoples !

-ivan

Ivan Price
Government Information Solutions
Department of Land Information
Western Australia
Ivan.Price@anonymised.com
+61 08 9273 7520

------ Forwarded Message
> From: Craig Bruce <csbruce@anonymised.com>
> Date: Tue, 23 Sep 2003 01:39:05 -0400
> To: wms-dev@anonymised.com
> Subject: [wms-dev] CWXML-BXML library release
>
> For anyone who may be interested, CubeWerx has made a public beta
release
> of its "cwxml" library. The home page is at:
>
> http://www.cubewerx.com/cwxml/
>
> and a Binary-XML design & performance report is available at:
>
> http://www.cubewerx.com/main/HTML/Binary_XML_Encoding.html
>
> What is CWXML?
>
> CWXML is an high-performance, open-source C-language library for par=
sing
> and generating XML and BXML (below) formats with a straightforward A=
PI.
> Initial testing indicates that it is 3 or more times as fast as othe=
r
> popular libraries such as expat and libxml2 at parsing XML and much
> faster again with BXML. The library is being developed by CubeWerx =
as
> the reference implementation for the BXML format. The parser accept=
s
and
> automatically recognizes the following formats: XML, GZIPped XML, BX=
ML,
> BXML with internal GZIP, and BXML with external GZIP. It is license=
d
> under the GNU LGPL.
>
> What is BXML?
>
> BXML (Binary XML) is an straightforward, open, patent-unencumbered
> binary-encoding format for XML data that is a stand-alone work-alike=

> drop-in replacement for an XML file that mirrors the XML markup
structures
> in a way that is similar to the in-memory representations of many
> parser libraries. BXML was developed by CubeWerx Inc. for the OpenG=
IS=AE
> Consortium and it makes all XML documents more compact and efficient=
to
> parse and generate by using a symbol table for element/attribute nam=
es
> and length-prefix encoding all arbitrary-length structures (strings,=

> blobs, arrays). But it especially makes dense-numeric XML documents=

much
> more efficient by allowing raw arrays of different common types of
numbers.
> For example, imagery data, the butt of many XML-bloat jokes, can be
handled
> in BXML just as well if not better than it is handled in PNG format.=

> A numeric array can pass from end-to-end in a client/server environm=
ent
as
> a raw chunk of data without ever being recoded. Dense numeric data =
also
> compresses faster and more compactly when encoded in binary rather t=
han
> text. BXML also has features for random access. The BXML specifica=
tion
> is available from:
>
> http://www.opengis.org/techno/discussions/03-002r8.pdf
>
> It was originally designed in part to address GML-bulk/slowness
problems.

------ End of Forwarded Message

    **
Responsibility for the administration and management of Crown land has
transferred from the Department of Land Information (DLI), formerly DOL=
A,
to the Department for Planning and Infrastructure (DPI) as of July 1, 2=
003.

Although continuing to operate from DLI's Midland office, Land
Administration Services and associated support staff (who will be known=
as
Land Asset Management Services) can now be contacted on (08) 9347 5000 =
and
post: PO Box 1575 Midland, WA, 6936.

    **

This e-mail and any files transmitted with it are intended only for the=
use
of the addressee(s). It may contain information that is confidential a=
nd
privileged. If you are not an intended recipient, any use, interferenc=
e
with, disclosure, distribution or copying of this material is unauthori=
sed
and prohibited. If you receive this in error, please notify the author =
by
return e-mail or telephone +61-8-9273 7533. Information in this messag=
e
not relating to the official business of DLI shall be understood as nei=
ther
given nor endorsed by it. While every care is taken, it is recommended=

that you scan any attachments for viruses. DLI liability is limited to
re-supplying affected attachments.

    **
Responsibility for the administration and management of Crown land has
transferred from the Department of Land Information (DLI), formerly DOLA,
to the Department for Planning and Infrastructure (DPI) as of July 1, 2003.

Although continuing to operate from DLI's Midland office, Land
Administration Services and associated support staff (who will be known as
Land Asset Management Services) can now be contacted on (08) 9347 5000 and
post: PO Box 1575 Midland, WA, 6936.

    **

This e-mail and any files transmitted with it are intended only for the use
of the addressee(s). It may contain information that is confidential and
privileged. If you are not an intended recipient, any use, interference
with, disclosure, distribution or copying of this material is unauthorised
and prohibited. If you receive this in error, please notify the author by
return e-mail or telephone +61-8-9273 7533. Information in this message
not relating to the official business of DLI shall be understood as neither
given nor endorsed by it. While every care is taken, it is recommended
that you scan any attachments for viruses. DLI liability is limited to
re-supplying affected attachments.

2

Yes, we definitely do need to address the bloat of xml. The first thing
that I'm planning on doing, hopefully within a month or two (or earlier if
anyone wants to roll it in, it should actually be pretty easy to do) is to
make GML2.gz an available format. You don't get a few of the speed
increases in parsing, but it _definitely_ cuts substantially down
the size of the responses - especially since xml is so structured with
lots of blank space often - it zips up really well. This shouldn't be too
hard - I think gabriel might have started to implement it - he was
attempting to automatically see if the clients could handle it - but I
think the way to do it is to just make a format that clients can request.
I can do that part, I'm just not positive how to do the actual gzip,
though I think there's still some code in AbstractService. So I think
that'll be a decent improvement.

As for bxml - I think it's just done in c for now, so we'd have to do all
the porting efforts. If it is available for java I'd look into
implementing it. But recently galdos was talking about BinXML (which is
something different, done by expway instead of cubewerx) - basically the
situation is not the best because there is no accepted standard, and it
should be done by w3c, not the ogc. So we may just wait it out, unless of
course someone is interested in doing the work and contributing. But I
think gzipping it should help a bit.

Chris

On Tue, 9 Mar 2004, Chris Holmes wrote:

Ivan has been having trouble getting his mail through, so for now I'm just
forwarding it on:

Hi there,

Back on the 24/9/2003 Cameron Shorter posted a message to this group as=
king
about BXML and whether any thought had been put into implementing BXML =
has
an output / input format to Geoserver.. was there a response to that ma=
il,
has there been any discussions here ?

I'll paste the text discussing BXML for reference below.

After (successfully) testing our Geoserver looking at Esri SDE with
Integraph Geomedia Viewer (free, but need to order the CD), and
Professional and the new version of their (free) WFS dataserver the
bulkiness of GML is really evident as you can directly compare the vari=
ous
formats to GML.. As WFS/Geoserver becomes a core business tool for us
spatial folk the XML bloat will need to be addressed.

Keep up the great work geoserver peoples !

-ivan

Ivan Price
Government Information Solutions
Department of Land Information
Western Australia
Ivan.Price@anonymised.com
+61 08 9273 7520

------ Forwarded Message
> From: Craig Bruce <csbruce@anonymised.com>
> Date: Tue, 23 Sep 2003 01:39:05 -0400
> To: wms-dev@anonymised.com
> Subject: [wms-dev] CWXML-BXML library release
>
> For anyone who may be interested, CubeWerx has made a public beta
release
> of its "cwxml" library. The home page is at:
>
> http://www.cubewerx.com/cwxml/
>
> and a Binary-XML design & performance report is available at:
>
> http://www.cubewerx.com/main/HTML/Binary_XML_Encoding.html
>
> What is CWXML?
>
> CWXML is an high-performance, open-source C-language library for par=
sing
> and generating XML and BXML (below) formats with a straightforward A=
PI.
> Initial testing indicates that it is 3 or more times as fast as othe=
r
> popular libraries such as expat and libxml2 at parsing XML and much
> faster again with BXML. The library is being developed by CubeWerx =
as
> the reference implementation for the BXML format. The parser accept=
s
and
> automatically recognizes the following formats: XML, GZIPped XML, BX=
ML,
> BXML with internal GZIP, and BXML with external GZIP. It is license=
d
> under the GNU LGPL.
>
> What is BXML?
>
> BXML (Binary XML) is an straightforward, open, patent-unencumbered
> binary-encoding format for XML data that is a stand-alone work-alike=

> drop-in replacement for an XML file that mirrors the XML markup
structures
> in a way that is similar to the in-memory representations of many
> parser libraries. BXML was developed by CubeWerx Inc. for the OpenG=
IS=AE
> Consortium and it makes all XML documents more compact and efficient=
to
> parse and generate by using a symbol table for element/attribute nam=
es
> and length-prefix encoding all arbitrary-length structures (strings,=

> blobs, arrays). But it especially makes dense-numeric XML documents=

much
> more efficient by allowing raw arrays of different common types of
numbers.
> For example, imagery data, the butt of many XML-bloat jokes, can be
handled
> in BXML just as well if not better than it is handled in PNG format.=

> A numeric array can pass from end-to-end in a client/server environm=
ent
as
> a raw chunk of data without ever being recoded. Dense numeric data =
also
> compresses faster and more compactly when encoded in binary rather t=
han
> text. BXML also has features for random access. The BXML specifica=
tion
> is available from:
>
> http://www.opengis.org/techno/discussions/03-002r8.pdf
>
> It was originally designed in part to address GML-bulk/slowness
problems.

------ End of Forwarded Message

    **
Responsibility for the administration and management of Crown land has
transferred from the Department of Land Information (DLI), formerly DOL=
A,
to the Department for Planning and Infrastructure (DPI) as of July 1, 2=
003.

Although continuing to operate from DLI's Midland office, Land
Administration Services and associated support staff (who will be known=
as
Land Asset Management Services) can now be contacted on (08) 9347 5000 =
and
post: PO Box 1575 Midland, WA, 6936.

    **

This e-mail and any files transmitted with it are intended only for the=
use
of the addressee(s). It may contain information that is confidential a=
nd
privileged. If you are not an intended recipient, any use, interferenc=
e
with, disclosure, distribution or copying of this material is unauthori=
sed
and prohibited. If you receive this in error, please notify the author =
by
return e-mail or telephone +61-8-9273 7533. Information in this messag=
e
not relating to the official business of DLI shall be understood as nei=
ther
given nor endorsed by it. While every care is taken, it is recommended=

that you scan any attachments for viruses. DLI liability is limited to
re-supplying affected attachments.

    **
Responsibility for the administration and management of Crown land has
transferred from the Department of Land Information (DLI), formerly DOLA,
to the Department for Planning and Infrastructure (DPI) as of July 1, 2003.

Although continuing to operate from DLI's Midland office, Land
Administration Services and associated support staff (who will be known as
Land Asset Management Services) can now be contacted on (08) 9347 5000 and
post: PO Box 1575 Midland, WA, 6936.

    **

This e-mail and any files transmitted with it are intended only for the use
of the addressee(s). It may contain information that is confidential and
privileged. If you are not an intended recipient, any use, interference
with, disclosure, distribution or copying of this material is unauthorised
and prohibited. If you receive this in error, please notify the author by
return e-mail or telephone +61-8-9273 7533. Information in this message
not relating to the official business of DLI shall be understood as neither
given nor endorsed by it. While every care is taken, it is recommended
that you scan any attachments for viruses. DLI liability is limited to
re-supplying affected attachments.

-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--

3

About gzipping responses:

I think gzipping the response automatically if the user agent supports it
should be better and can benefit other responses/services.
Of cuorse we should implement it as a request format if it is in the WFS
spec.
The problem here is that, in AbstractService, we're giving the HttpResponse
object to the choosen output stratagy. Due to that, we can't assume
very much what the stratagy will do with the response object, if it will
write directly to the response output stream, or uf it will write to
somewhere else and just open the response output stream at the final.

By the other way, looking if the client user agent supports gzip is very
trivial, since its just a header in the http request.

So, what I thought was to implement a HttpResponse decorator who do the
gzipping and puts the encoding http header in the response header,
and pass it to the stratagy instead of passing it the original request. This
sounds good and pretty easy to me, and avoids adding extra complexity to
AbstractService.doService, wich allready is difficult to read.

Now, sayd that, the problem is that not all responses will really benefit
from being gzipped. For example, all XML responses will, as any WFS response
as well as SVG images, but JPEG, PNG, etc will not.

To solve that, we should may be ask the content responsible, by adding a
(canBeCompressed()?) method to the Response interface. Or may be having a
lookup of compressable content types, since we allready have a
getContentType() method. But this last options sounds less optimal.

Tell me waht you think, I was planning implement the gzipping today anyway.

best regards,

gabriel.

PS: yeserday, I've commited a patch for FeatureTypeInfo that was preventing
WMS to work, as well as WFS, when no info.xml was present for a feature
type, or info.xml do not declared the exposed attributes. It seems Jira do
not manages concurreny very well, because at the same time, it seems that
dzwiers commited another patch, so when I pressed the Resolve button, his
comment appeared and mine disappeared.
It's a lot of casualllity, but may be it worst to know it.

----- Original Message -----
From: "Chris Holmes" <cholmes@anonymised.com>
To: <geoserver-devel@lists.sourceforge.net>
Cc: <Ivan.Price@anonymised.com>
Sent: Tuesday, March 09, 2004 9:09 AM
Subject: Re: [Geoserver-devel] BXML data format (from Ivan Price)

Yes, we definitely do need to address the bloat of xml. The first thing
that I'm planning on doing, hopefully within a month or two (or earlier if
anyone wants to roll it in, it should actually be pretty easy to do) is to
make GML2.gz an available format. You don't get a few of the speed
increases in parsing, but it _definitely_ cuts substantially down
the size of the responses - especially since xml is so structured with
lots of blank space often - it zips up really well. This shouldn't be too
hard - I think gabriel might have started to implement it - he was
attempting to automatically see if the clients could handle it - but I
think the way to do it is to just make a format that clients can request.
I can do that part, I'm just not positive how to do the actual gzip,
though I think there's still some code in AbstractService. So I think
that'll be a decent improvement.

As for bxml - I think it's just done in c for now, so we'd have to do all
the porting efforts. If it is available for java I'd look into
implementing it. But recently galdos was talking about BinXML (which is
something different, done by expway instead of cubewerx) - basically the
situation is not the best because there is no accepted standard, and it
should be done by w3c, not the ogc. So we may just wait it out, unless of
course someone is interested in doing the work and contributing. But I
think gzipping it should help a bit.

Chris

On Tue, 9 Mar 2004, Chris Holmes wrote:

> Ivan has been having trouble getting his mail through, so for now I'm

just

> forwarding it on:
>
> Hi there,
>
> Back on the 24/9/2003 Cameron Shorter posted a message to this group as=
> king
> about BXML and whether any thought had been put into implementing BXML =
> has
> an output / input format to Geoserver.. was there a response to that ma=
> il,
> has there been any discussions here ?
>
> I'll paste the text discussing BXML for reference below.
>
> After (successfully) testing our Geoserver looking at Esri SDE with
> Integraph Geomedia Viewer (free, but need to order the CD), and
> Professional and the new version of their (free) WFS dataserver the
> bulkiness of GML is really evident as you can directly compare the vari=
> ous
> formats to GML.. As WFS/Geoserver becomes a core business tool for us
> spatial folk the XML bloat will need to be addressed.
>
> Keep up the great work geoserver peoples !
>
> -ivan
>
>
> Ivan Price
> Government Information Solutions
> Department of Land Information
> Western Australia
> Ivan.Price@anonymised.com
> +61 08 9273 7520
>
>
> ------ Forwarded Message
> > From: Craig Bruce <csbruce@anonymised.com>
> > Date: Tue, 23 Sep 2003 01:39:05 -0400
> > To: wms-dev@anonymised.com
> > Subject: [wms-dev] CWXML-BXML library release
> >
> > For anyone who may be interested, CubeWerx has made a public beta
> release
> > of its "cwxml" library. The home page is at:
> >
> > http://www.cubewerx.com/cwxml/
> >
> > and a Binary-XML design & performance report is available at:
> >
> > http://www.cubewerx.com/main/HTML/Binary_XML_Encoding.html
> >
> > What is CWXML?
> >
> > CWXML is an high-performance, open-source C-language library for par=
> sing
> > and generating XML and BXML (below) formats with a straightforward A=
> PI.
> > Initial testing indicates that it is 3 or more times as fast as othe=
> r
> > popular libraries such as expat and libxml2 at parsing XML and much
> > faster again with BXML. The library is being developed by CubeWerx =
> as
> > the reference implementation for the BXML format. The parser accept=
> s
> and
> > automatically recognizes the following formats: XML, GZIPped XML, BX=
> ML,
> > BXML with internal GZIP, and BXML with external GZIP. It is license=
> d
> > under the GNU LGPL.
> >
> > What is BXML?
> >
> > BXML (Binary XML) is an straightforward, open, patent-unencumbered
> > binary-encoding format for XML data that is a stand-alone work-alike=
>
> > drop-in replacement for an XML file that mirrors the XML markup
> structures
> > in a way that is similar to the in-memory representations of many
> > parser libraries. BXML was developed by CubeWerx Inc. for the OpenG=
> IS=AE
> > Consortium and it makes all XML documents more compact and efficient=
> to
> > parse and generate by using a symbol table for element/attribute nam=
> es
> > and length-prefix encoding all arbitrary-length structures (strings,=
>
> > blobs, arrays). But it especially makes dense-numeric XML documents=
>
> much
> > more efficient by allowing raw arrays of different common types of
> numbers.
> > For example, imagery data, the butt of many XML-bloat jokes, can be
> handled
> > in BXML just as well if not better than it is handled in PNG format.=
>
> > A numeric array can pass from end-to-end in a client/server environm=
> ent
> as
> > a raw chunk of data without ever being recoded. Dense numeric data =
> also
> > compresses faster and more compactly when encoded in binary rather t=
> han
> > text. BXML also has features for random access. The BXML specifica=
> tion
> > is available from:
> >
> > http://www.opengis.org/techno/discussions/03-002r8.pdf
> >
> > It was originally designed in part to address GML-bulk/slowness
> problems.
>
> ------ End of Forwarded Message
>
>
>
>
> **
> Responsibility for the administration and management of Crown land has
> transferred from the Department of Land Information (DLI), formerly DOL=
> A,
> to the Department for Planning and Infrastructure (DPI) as of July 1, 2=
> 003.
>
> Although continuing to operate from DLI's Midland office, Land
> Administration Services and associated support staff (who will be known=
> as
> Land Asset Management Services) can now be contacted on (08) 9347 5000 =
> and
> post: PO Box 1575 Midland, WA, 6936.
>
> **
>
> This e-mail and any files transmitted with it are intended only for the=
> use
> of the addressee(s). It may contain information that is confidential a=
> nd
> privileged. If you are not an intended recipient, any use, interferenc=
> e
> with, disclosure, distribution or copying of this material is unauthori=
> sed
> and prohibited. If you receive this in error, please notify the author =
> by
> return e-mail or telephone +61-8-9273 7533. Information in this messag=
> e
> not relating to the official business of DLI shall be understood as nei=
> ther
> given nor endorsed by it. While every care is taken, it is recommended=
>
> that you scan any attachments for viruses. DLI liability is limited to
> re-supplying affected attachments.
> =
>
>
>
>
> **
> Responsibility for the administration and management of Crown land has
> transferred from the Department of Land Information (DLI), formerly

DOLA,

> to the Department for Planning and Infrastructure (DPI) as of July 1,

2003.

>
> Although continuing to operate from DLI's Midland office, Land
> Administration Services and associated support staff (who will be known

as

> Land Asset Management Services) can now be contacted on (08) 9347 5000

and

> post: PO Box 1575 Midland, WA, 6936.
>
> **
>
> This e-mail and any files transmitted with it are intended only for the

use

> of the addressee(s). It may contain information that is confidential

and

> privileged. If you are not an intended recipient, any use, interference
> with, disclosure, distribution or copying of this material is

unauthorised

> and prohibited. If you receive this in error, please notify the author

by

> return e-mail or telephone +61-8-9273 7533. Information in this message
> not relating to the official business of DLI shall be understood as

neither

> given nor endorsed by it. While every care is taken, it is recommended
> that you scan any attachments for viruses. DLI liability is limited to
> re-supplying affected attachments.
>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>

--

-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

4

I think gzipping the response automatically if the user agent supports it
should be better and can benefit other responses/services.
Of cuorse we should implement it as a request format if it is in the WFS
spec.

Do you have any idea how many user agents will support it? Is that common
to do? The one nice thing about it as a request format is that you can
advertise it in the capabilities document, so people know what they are
getting.

The problem here is that, in AbstractService, we're giving the HttpResponse
object to the choosen output stratagy. Due to that, we can't assume
very much what the stratagy will do with the response object, if it will
write directly to the response output stream, or uf it will write to
somewhere else and just open the response output stream at the final.

Hrm. Perhaps just the speed response should support gzipping. As long as
we advertise right in the caps document it should be fine. All
geoserver's now go out set with the SPEED strategy - I feel like the other
two could almost be deprecated. In my mind they are much more for
debugging purposes, and as such I don't think we need to add gzip support
for all of them.

By the other way, looking if the client user agent supports gzip is very
trivial, since its just a header in the http request.

Yeah, again I'm just curious how many clients are actually going to do
this, and how seamlessly it will be handled by them.

So, what I thought was to implement a HttpResponse decorator who do the
gzipping and puts the encoding http header in the response header,
and pass it to the stratagy instead of passing it the original request. This
sounds good and pretty easy to me, and avoids adding extra complexity to
AbstractService.doService, wich allready is difficult to read.

Now, sayd that, the problem is that not all responses will really benefit
from being gzipped. For example, all XML responses will, as any WFS response
as well as SVG images, but JPEG, PNG, etc will not.

To solve that, we should may be ask the content responsible, by adding a
(canBeCompressed()?) method to the Response interface. Or may be having a
lookup of compressable content types, since we allready have a
getContentType() method. But this last options sounds less optimal.

I'm fine with either (I must admit I still am wfs centric, and it does
not naturally occur to me that everything can't be gzipped ;).

Tell me waht you think, I was planning implement the gzipping today anyway.

Yeah, I'd say do it in speed strategey and advertise in the caps document
if the strategy is speed, and read in the response. Another thought could
be a gzip strategy - similar to speed but also does the fancy gzipping
stuff.

PS: yeserday, I've commited a patch for FeatureTypeInfo that was preventing
WMS to work, as well as WFS, when no info.xml was present for a feature
type, or info.xml do not declared the exposed attributes.

Huh? info.xml no longer should have any knowledge of exposed attributes,
that information is now all contained in the schema.xml file - reading if
it's minOccurs=0 or 1. And I'm not sure I understand info.xml not being
present - if it's not present then there should be no featureType, no?
That's how you define which featureTypes to expose - and if there is no
info.xml in a folder it doesn't break the wms/wfs, at least not that I
could see with how the code works.

It seems Jira do
not manages concurreny very well, because at the same time, it seems that
dzwiers commited another patch, so when I pressed the Resolve button, his
comment appeared and mine disappeared.
It's a lot of casualllity, but may be it worst to know it.

That's not good.

Chris

----- Original Message -----
From: "Chris Holmes" <cholmes@anonymised.com>
To: <geoserver-devel@lists.sourceforge.net>
Cc: <Ivan.Price@anonymised.com>
Sent: Tuesday, March 09, 2004 9:09 AM
Subject: Re: [Geoserver-devel] BXML data format (from Ivan Price)

> Yes, we definitely do need to address the bloat of xml. The first thing
> that I'm planning on doing, hopefully within a month or two (or earlier if
> anyone wants to roll it in, it should actually be pretty easy to do) is to
> make GML2.gz an available format. You don't get a few of the speed
> increases in parsing, but it _definitely_ cuts substantially down
> the size of the responses - especially since xml is so structured with
> lots of blank space often - it zips up really well. This shouldn't be too
> hard - I think gabriel might have started to implement it - he was
> attempting to automatically see if the clients could handle it - but I
> think the way to do it is to just make a format that clients can request.
> I can do that part, I'm just not positive how to do the actual gzip,
> though I think there's still some code in AbstractService. So I think
> that'll be a decent improvement.
>
> As for bxml - I think it's just done in c for now, so we'd have to do all
> the porting efforts. If it is available for java I'd look into
> implementing it. But recently galdos was talking about BinXML (which is
> something different, done by expway instead of cubewerx) - basically the
> situation is not the best because there is no accepted standard, and it
> should be done by w3c, not the ogc. So we may just wait it out, unless of
> course someone is interested in doing the work and contributing. But I
> think gzipping it should help a bit.
>
> Chris
>
> On Tue, 9 Mar 2004, Chris Holmes wrote:
>
> > Ivan has been having trouble getting his mail through, so for now I'm
just
> > forwarding it on:
> >
> > Hi there,
> >
> > Back on the 24/9/2003 Cameron Shorter posted a message to this group as=
> > king
> > about BXML and whether any thought had been put into implementing BXML =
> > has
> > an output / input format to Geoserver.. was there a response to that ma=
> > il,
> > has there been any discussions here ?
> >
> > I'll paste the text discussing BXML for reference below.
> >
> > After (successfully) testing our Geoserver looking at Esri SDE with
> > Integraph Geomedia Viewer (free, but need to order the CD), and
> > Professional and the new version of their (free) WFS dataserver the
> > bulkiness of GML is really evident as you can directly compare the vari=
> > ous
> > formats to GML.. As WFS/Geoserver becomes a core business tool for us
> > spatial folk the XML bloat will need to be addressed.
> >
> > Keep up the great work geoserver peoples !
> >
> > -ivan
> >
> >
> > Ivan Price
> > Government Information Solutions
> > Department of Land Information
> > Western Australia
> > Ivan.Price@anonymised.com
> > +61 08 9273 7520
> >
> >
> > ------ Forwarded Message
> > > From: Craig Bruce <csbruce@anonymised.com>
> > > Date: Tue, 23 Sep 2003 01:39:05 -0400
> > > To: wms-dev@anonymised.com
> > > Subject: [wms-dev] CWXML-BXML library release
> > >
> > > For anyone who may be interested, CubeWerx has made a public beta
> > release
> > > of its "cwxml" library. The home page is at:
> > >
> > > http://www.cubewerx.com/cwxml/
> > >
> > > and a Binary-XML design & performance report is available at:
> > >
> > > http://www.cubewerx.com/main/HTML/Binary_XML_Encoding.html
> > >
> > > What is CWXML?
> > >
> > > CWXML is an high-performance, open-source C-language library for par=
> > sing
> > > and generating XML and BXML (below) formats with a straightforward A=
> > PI.
> > > Initial testing indicates that it is 3 or more times as fast as othe=
> > r
> > > popular libraries such as expat and libxml2 at parsing XML and much
> > > faster again with BXML. The library is being developed by CubeWerx =
> > as
> > > the reference implementation for the BXML format. The parser accept=
> > s
> > and
> > > automatically recognizes the following formats: XML, GZIPped XML, BX=
> > ML,
> > > BXML with internal GZIP, and BXML with external GZIP. It is license=
> > d
> > > under the GNU LGPL.
> > >
> > > What is BXML?
> > >
> > > BXML (Binary XML) is an straightforward, open, patent-unencumbered
> > > binary-encoding format for XML data that is a stand-alone work-alike=
> >
> > > drop-in replacement for an XML file that mirrors the XML markup
> > structures
> > > in a way that is similar to the in-memory representations of many
> > > parser libraries. BXML was developed by CubeWerx Inc. for the OpenG=
> > IS=AE
> > > Consortium and it makes all XML documents more compact and efficient=
> > to
> > > parse and generate by using a symbol table for element/attribute nam=
> > es
> > > and length-prefix encoding all arbitrary-length structures (strings,=
> >
> > > blobs, arrays). But it especially makes dense-numeric XML documents=
> >
> > much
> > > more efficient by allowing raw arrays of different common types of
> > numbers.
> > > For example, imagery data, the butt of many XML-bloat jokes, can be
> > handled
> > > in BXML just as well if not better than it is handled in PNG format.=
> >
> > > A numeric array can pass from end-to-end in a client/server environm=
> > ent
> > as
> > > a raw chunk of data without ever being recoded. Dense numeric data =
> > also
> > > compresses faster and more compactly when encoded in binary rather t=
> > han
> > > text. BXML also has features for random access. The BXML specifica=
> > tion
> > > is available from:
> > >
> > > http://www.opengis.org/techno/discussions/03-002r8.pdf
> > >
> > > It was originally designed in part to address GML-bulk/slowness
> > problems.
> >
> > ------ End of Forwarded Message
> >
> >
> >
> >
> > **
> > Responsibility for the administration and management of Crown land has
> > transferred from the Department of Land Information (DLI), formerly DOL=
> > A,
> > to the Department for Planning and Infrastructure (DPI) as of July 1, 2=
> > 003.
> >
> > Although continuing to operate from DLI's Midland office, Land
> > Administration Services and associated support staff (who will be known=
> > as
> > Land Asset Management Services) can now be contacted on (08) 9347 5000 =
> > and
> > post: PO Box 1575 Midland, WA, 6936.
> >
> > **
> >
> > This e-mail and any files transmitted with it are intended only for the=
> > use
> > of the addressee(s). It may contain information that is confidential a=
> > nd
> > privileged. If you are not an intended recipient, any use, interferenc=
> > e
> > with, disclosure, distribution or copying of this material is unauthori=
> > sed
> > and prohibited. If you receive this in error, please notify the author =
> > by
> > return e-mail or telephone +61-8-9273 7533. Information in this messag=
> > e
> > not relating to the official business of DLI shall be understood as nei=
> > ther
> > given nor endorsed by it. While every care is taken, it is recommended=
> >
> > that you scan any attachments for viruses. DLI liability is limited to
> > re-supplying affected attachments.
> > =
> >
> >
> >
> >
> > **
> > Responsibility for the administration and management of Crown land has
> > transferred from the Department of Land Information (DLI), formerly
DOLA,
> > to the Department for Planning and Infrastructure (DPI) as of July 1,
2003.
> >
> > Although continuing to operate from DLI's Midland office, Land
> > Administration Services and associated support staff (who will be known
as
> > Land Asset Management Services) can now be contacted on (08) 9347 5000
and
> > post: PO Box 1575 Midland, WA, 6936.
> >
> > **
> >
> > This e-mail and any files transmitted with it are intended only for the
use
> > of the addressee(s). It may contain information that is confidential
and
> > privileged. If you are not an intended recipient, any use, interference
> > with, disclosure, distribution or copying of this material is
unauthorised
> > and prohibited. If you receive this in error, please notify the author
by
> > return e-mail or telephone +61-8-9273 7533. Information in this message
> > not relating to the official business of DLI shall be understood as
neither
> > given nor endorsed by it. While every care is taken, it is recommended
> > that you scan any attachments for viruses. DLI liability is limited to
> > re-supplying affected attachments.
> >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: IBM Linux Tutorials
> > Free Linux tutorial presented by Daniel Robbins, President and CEO of
> > GenToo technologies. Learn everything from fundamentals to system
> > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> > _______________________________________________
> > Geoserver-devel mailing list
> > Geoserver-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
> >
>
> --
>
>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--

5

> I think gzipping the response automatically if the user agent supports

it

> should be better and can benefit other responses/services.
> Of cuorse we should implement it as a request format if it is in the WFS
> spec.
Do you have any idea how many user agents will support it? Is that common
to do?

Yeah, definately it is so common. Some web servers can be configured to
manage it automatically in deed, compressing html output on the fly if the
user agent supports it.

The one nice thing about it as a request format is that you can
advertise it in the capabilities document, so people know what they are
getting.

All the web browsers supports it, it's just transparent for the user, on
each request, they send the http header saying that they supports gzip and
deflate (zip) encoding formats. But our users will not be just web browsers,
off cuorse. There're no problem for me in adding it in capabilities, I just
need to look at the spec to see how it must be addressed.

What I was thinking is, that since this technique is so common, managing it
automatically in abstractservice should be ok, because the idea of
AbstractService.doService is to call Response.writeTo giving it an already
optimized outputstream, so every response specialization should not take
care of wrapping the outputstream either with a BufferedOutputStream or a
GzipOutputStream or whatever, they just would need to take care of writing
the response content without care of optimization, since the decision was
taken at a higer level, based on stratagy and, in this case, in compression
capabilities.

In this scenario, if you're ok with it, I'll do just the two things. Well,
may be if the specific gzipped WFS format is specified, it's response object
should return false in supportsCompression()? and take care itself, for
clarity...
I don't know, don't you think it should make sense to compress all the
'compressable' responses?

> PS: yeserday, I've commited a patch for FeatureTypeInfo that was

preventing

> WMS to work, as well as WFS, when no info.xml was present for a feature
> type, or info.xml do not declared the exposed attributes.
Huh? info.xml no longer should have any knowledge of exposed attributes,
that information is now all contained in the schema.xml file - reading if
it's minOccurs=0 or 1. And I'm not sure I understand info.xml not being
present - if it's not present then there should be no featureType, no?
That's how you define which featureTypes to expose - and if there is no
info.xml in a folder it doesn't break the wms/wfs, at least not that I
could see with how the code works.

Oh! sorry, my fault. I mean schema.xml. The point was that in the ausence of
__schema.xml__ FeatureTypeInfo was returning an empty FeatureType. Excuse
the spell error.

Gabriel

> It seems Jira do
> not manages concurreny very well, because at the same time, it seems

that

> dzwiers commited another patch, so when I pressed the Resolve button,

his

> comment appeared and mine disappeared.
> It's a lot of casualllity, but may be it worst to know it.
That's not good.

Chris

>
> ----- Original Message -----
> From: "Chris Holmes" <cholmes@anonymised.com>
> To: <geoserver-devel@lists.sourceforge.net>
> Cc: <Ivan.Price@anonymised.com>
> Sent: Tuesday, March 09, 2004 9:09 AM
> Subject: Re: [Geoserver-devel] BXML data format (from Ivan Price)
>
>
> > Yes, we definitely do need to address the bloat of xml. The first

thing

> > that I'm planning on doing, hopefully within a month or two (or

earlier if

> > anyone wants to roll it in, it should actually be pretty easy to do)

is to

> > make GML2.gz an available format. You don't get a few of the speed
> > increases in parsing, but it _definitely_ cuts substantially down
> > the size of the responses - especially since xml is so structured with
> > lots of blank space often - it zips up really well. This shouldn't be

too

> > hard - I think gabriel might have started to implement it - he was
> > attempting to automatically see if the clients could handle it - but I
> > think the way to do it is to just make a format that clients can

request.

> > I can do that part, I'm just not positive how to do the actual gzip,
> > though I think there's still some code in AbstractService. So I think
> > that'll be a decent improvement.
> >
> > As for bxml - I think it's just done in c for now, so we'd have to do

all

> > the porting efforts. If it is available for java I'd look into
> > implementing it. But recently galdos was talking about BinXML (which

is

> > something different, done by expway instead of cubewerx) - basically

the

> > situation is not the best because there is no accepted standard, and

it

> > should be done by w3c, not the ogc. So we may just wait it out,

unless of

> > course someone is interested in doing the work and contributing. But

I

> > think gzipping it should help a bit.
> >
> > Chris
> >
> > On Tue, 9 Mar 2004, Chris Holmes wrote:
> >
> > > Ivan has been having trouble getting his mail through, so for now

I'm

> just
> > > forwarding it on:
> > >
> > > Hi there,
> > >
> > > Back on the 24/9/2003 Cameron Shorter posted a message to this group

as=

> > > king
> > > about BXML and whether any thought had been put into implementing

BXML =

> > > has
> > > an output / input format to Geoserver.. was there a response to that

ma=

> > > il,
> > > has there been any discussions here ?
> > >
> > > I'll paste the text discussing BXML for reference below.
> > >
> > > After (successfully) testing our Geoserver looking at Esri SDE with
> > > Integraph Geomedia Viewer (free, but need to order the CD), and
> > > Professional and the new version of their (free) WFS dataserver the
> > > bulkiness of GML is really evident as you can directly compare the

vari=

> > > ous
> > > formats to GML.. As WFS/Geoserver becomes a core business tool for

us

> > > spatial folk the XML bloat will need to be addressed.
> > >
> > > Keep up the great work geoserver peoples !
> > >
> > > -ivan
> > >
> > >
> > > Ivan Price
> > > Government Information Solutions
> > > Department of Land Information
> > > Western Australia
> > > Ivan.Price@anonymised.com
> > > +61 08 9273 7520
> > >
> > >
> > > ------ Forwarded Message
> > > > From: Craig Bruce <csbruce@anonymised.com>
> > > > Date: Tue, 23 Sep 2003 01:39:05 -0400
> > > > To: wms-dev@anonymised.com
> > > > Subject: [wms-dev] CWXML-BXML library release
> > > >
> > > > For anyone who may be interested, CubeWerx has made a public beta
> > > release
> > > > of its "cwxml" library. The home page is at:
> > > >
> > > > http://www.cubewerx.com/cwxml/
> > > >
> > > > and a Binary-XML design & performance report is available at:
> > > >
> > > > http://www.cubewerx.com/main/HTML/Binary_XML_Encoding.html
> > > >
> > > > What is CWXML?
> > > >
> > > > CWXML is an high-performance, open-source C-language library for

par=

> > > sing
> > > > and generating XML and BXML (below) formats with a

straightforward A=

> > > PI.
> > > > Initial testing indicates that it is 3 or more times as fast as

othe=

> > > r
> > > > popular libraries such as expat and libxml2 at parsing XML and

much

> > > > faster again with BXML. The library is being developed by

CubeWerx =

> > > as
> > > > the reference implementation for the BXML format. The parser

accept=

> > > s
> > > and
> > > > automatically recognizes the following formats: XML, GZIPped XML,

BX=

> > > ML,
> > > > BXML with internal GZIP, and BXML with external GZIP. It is

license=

> > > d
> > > > under the GNU LGPL.
> > > >
> > > > What is BXML?
> > > >
> > > > BXML (Binary XML) is an straightforward, open,

patent-unencumbered

> > > > binary-encoding format for XML data that is a stand-alone

work-alike=

> > >
> > > > drop-in replacement for an XML file that mirrors the XML markup
> > > structures
> > > > in a way that is similar to the in-memory representations of many
> > > > parser libraries. BXML was developed by CubeWerx Inc. for the

OpenG=

> > > IS=AE
> > > > Consortium and it makes all XML documents more compact and

efficient=

> > > to
> > > > parse and generate by using a symbol table for element/attribute

nam=

> > > es
> > > > and length-prefix encoding all arbitrary-length structures

(strings,=

> > >
> > > > blobs, arrays). But it especially makes dense-numeric XML

documents=

> > >
> > > much
> > > > more efficient by allowing raw arrays of different common types

of

> > > numbers.
> > > > For example, imagery data, the butt of many XML-bloat jokes, can

be

> > > handled
> > > > in BXML just as well if not better than it is handled in PNG

format.=

> > >
> > > > A numeric array can pass from end-to-end in a client/server

environm=

> > > ent
> > > as
> > > > a raw chunk of data without ever being recoded. Dense numeric

data =

> > > also
> > > > compresses faster and more compactly when encoded in binary

rather t=

> > > han
> > > > text. BXML also has features for random access. The BXML

specifica=

> > > tion
> > > > is available from:
> > > >
> > > > http://www.opengis.org/techno/discussions/03-002r8.pdf
> > > >
> > > > It was originally designed in part to address GML-bulk/slowness
> > > problems.
> > >
> > > ------ End of Forwarded Message
> > >
> > >
> > >
> > >
> > > **
> > > Responsibility for the administration and management of Crown land

has

> > > transferred from the Department of Land Information (DLI), formerly

DOL=

> > > A,
> > > to the Department for Planning and Infrastructure (DPI) as of July

1, 2=

> > > 003.
> > >
> > > Although continuing to operate from DLI's Midland office, Land
> > > Administration Services and associated support staff (who will be

known=

> > > as
> > > Land Asset Management Services) can now be contacted on (08) 9347

5000 =

> > > and
> > > post: PO Box 1575 Midland, WA, 6936.
> > >
> > > **
> > >
> > > This e-mail and any files transmitted with it are intended only for

the=

> > > use
> > > of the addressee(s). It may contain information that is

confidential a=

> > > nd
> > > privileged. If you are not an intended recipient, any use,

interferenc=

> > > e
> > > with, disclosure, distribution or copying of this material is

unauthori=

> > > sed
> > > and prohibited. If you receive this in error, please notify the

author =

> > > by
> > > return e-mail or telephone +61-8-9273 7533. Information in this

messag=

> > > e
> > > not relating to the official business of DLI shall be understood as

nei=

> > > ther
> > > given nor endorsed by it. While every care is taken, it is

recommended=

> > >
> > > that you scan any attachments for viruses. DLI liability is limited

to

> > > re-supplying affected attachments.
> > > =
> > >
> > >
> > >
> > >
> > > **
> > > Responsibility for the administration and management of Crown land

has

> > > transferred from the Department of Land Information (DLI), formerly
> DOLA,
> > > to the Department for Planning and Infrastructure (DPI) as of July

1,

> 2003.
> > >
> > > Although continuing to operate from DLI's Midland office, Land
> > > Administration Services and associated support staff (who will be

known

> as
> > > Land Asset Management Services) can now be contacted on (08) 9347

5000

> and
> > > post: PO Box 1575 Midland, WA, 6936.
> > >
> > > **
> > >
> > > This e-mail and any files transmitted with it are intended only for

the

> use
> > > of the addressee(s). It may contain information that is

confidential

> and
> > > privileged. If you are not an intended recipient, any use,

interference

> > > with, disclosure, distribution or copying of this material is
> unauthorised
> > > and prohibited. If you receive this in error, please notify the

author

> by
> > > return e-mail or telephone +61-8-9273 7533. Information in this

message

> > > not relating to the official business of DLI shall be understood as
> neither
> > > given nor endorsed by it. While every care is taken, it is

recommended

> > > that you scan any attachments for viruses. DLI liability is limited

to

> > > re-supplying affected attachments.
> > >
> > >
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by: IBM Linux Tutorials
> > > Free Linux tutorial presented by Daniel Robbins, President and CEO

of

> > > GenToo technologies. Learn everything from fundamentals to system
> > >

administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

> > > _______________________________________________
> > > Geoserver-devel mailing list
> > > Geoserver-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
> > >
> >
> > --
> >
> >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: IBM Linux Tutorials
> > Free Linux tutorial presented by Daniel Robbins, President and CEO of
> > GenToo technologies. Learn everything from fundamentals to system
> > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> > _______________________________________________
> > Geoserver-devel mailing list
> > Geoserver-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>

--

6

Regarding gzipped GML, should GeoServer do this at all? This is a
protocol level thing, the client requests compression if possible and
the server compresses if able. You could probably make GeoServer do
compression just be running it behind a web server that supported
compression, like Apache with mod_deflate enabled. Why add any
infrastructure at all to GeoServer, it is an HTTP thing.

P.

--
       __
      /
      | Paul Ramsey
      | Refractions Research
      | Email: pramsey@anonymised.com
      | Phone: (250) 885-0632
      \_

7

Regarding gzipped GML, should GeoServer do this at all? This is a
protocol level thing, the client requests compression if possible and
the server compresses if able. You could probably make GeoServer do
compression just be running it behind a web server that supported
compression, like Apache with mod_deflate enabled. Why add any
infrastructure at all to GeoServer, it is an HTTP thing.

Well, I think GeoServer should definitely send gzipped gml if the client
specifically requests it (in the getFeature request), as this is
completely in the scope of the spec, to request different formats. This
way clients who don't know how to ask for the gzipped stuff in the http
header can also get it.

As for doing it automatically, I'm a bit ambivalent. On the one hand I do
sorta agree that it's a lower level compression that geoserver shouldn't
do. But on the other hand geoserver is a server, and one of the main
design goals is to make it really easy to set up while still being very
powerful. I imagine that a number of users who have some data to share
aren't sys admins who are super experienced at compiling apache with
mod_deflate, let alone setting up apache with a connector to a servlet
container (something I've done once or twice and definitely found it to be
a bitch). Most all installs of geoserver I've seen are run directly from
the servlet container, so from that perspective I think it makes sense to
do it for them if we can, as I don't believe (I could be wrong here) that
tomcat or resin or any of them do that sort of compression in their built
in web servers. So yeah, the one thing I'm going to insist with Gabriel
rolling it in is that it's absolutely seamless. Gabriel, could you start
it on a branch or give it a config option similar to the SPEED strategy,
so we can test it out before it just shows up in geoserver code? I think
it should maybe wait until 1.2.1, at least as a default. Make the option
available in 1.2.0 so we can all sanity check it if we want to. And even
in 1.2.1 I think we should put an option in the services.xml file (and in
the ui) for users to turn it on or off.

Chris

P.

--