[GeoNetwork-users] ArcSDE harvester problems

Walter_Simonazzi · July 23, 2009, 11:30am

Dear all,

We are testing the ArcSDE harvester functionality using GN 2.4.0 (final)
/jetty/McKoi.

ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).

dbtune: SDEbinary storage,

Metadata: ISO 19115 and 19139.

This is what we get:

----------------------------------------------------------------------------------------------------------------------------

Connected to ArcSDE

start retrieve metadata

ISO metadata found

cool

Start of alignment for : REFDATABASE

Recoverable error

XTRE0540: Ambiguous rule match for /

Matches both "document-node()" on line 5 of

file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913

9.xsl

and "document-node()" on line 30 of

file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.

xsl

ArcSDE harvest finished
-------------------------------------------------------------------------------------------------------------------------------------
An no metadata is harvested, any clue/help can be rewarded with a beer in
Barcelona ;-).

Many thanks,

Walter & Maria Jose

--
--

Walter Simonazzi
European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª
planta
Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
08193 Bellaterra (Barcelona)
Spain

E-mail: walter.simonazzi (at) uab.es
Direct phone: +34 93 581 3868
Fax: +34 93 581 3545
Secretary: +34 93 581 3518
http://terrestrial.eionet.europa.eu

PatB · July 25, 2009, 4:17pm

I got it working with 9.3 - but it was a struggle. First thing I'd do is
verify the metadata you're getting out of sde 9.2 is the format you're
expecting (ISO), by default 9.3 returns esri's flavor of fgdc.

Next take a look at
...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java

around line 175 it has the following:

// create JDOM element from String-XML
Element metadataElement = Xml.loadString(metadata, false);

// transform ESRI output to ISO19115
Element iso19115 = Xml.transform(metadataElement,
FGDC_ISO_TRANSFORMER_LOCATION);

// transform ISO19115 to ISO19139
Element iso19139 = Xml.transform(iso19115,
ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);

So, if you track down the transformers and see what they're doing, it goes
from ESRI Catalog 8 (which doesn't match anything I saw coming out of 9.3)
to ISO19115 to ISO19139. I got around it by using esri's metaid as the uuid
and commenting out the transformers above so the metadata is stored as
fgdc-std. I'm still surprised that it worked.

Good luck! It sounds like you're close, you may also need to look at:
...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java

there's a line in there that sets ISO_METADATA_IDENTIFIER to look for
MD_Metadata, which is what identifies the type of incoming metadata. I had
to change mine to "metadata" for fgdc.

Hope this helps. (oh, and you have to rebuild the jars using ant if you
mess with the .java files)

WSimonazzi wrote:

Dear all,

We are testing the ArcSDE harvester functionality using GN 2.4.0 (final)
/jetty/McKoi.

ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).

dbtune: SDEbinary storage,

Metadata: ISO 19115 and 19139.

This is what we get:

----------------------------------------------------------------------------------------------------------------------------

Connected to ArcSDE

start retrieve metadata

ISO metadata found

cool

Start of alignment for : REFDATABASE

Recoverable error

XTRE0540: Ambiguous rule match for /

Matches both "document-node()" on line 5 of

file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913

9.xsl

and "document-node()" on line 30 of

file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.

xsl

ArcSDE harvest finished
-------------------------------------------------------------------------------------------------------------------------------------
An no metadata is harvested, any clue/help can be rewarded with a beer in
Barcelona ;-).

Many thanks,

Walter & Maria Jose

--
--

Walter Simonazzi
European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª
planta
Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
08193 Bellaterra (Barcelona)
Spain

E-mail: walter.simonazzi (at) uab.es
Direct phone: +34 93 581 3868
Fax: +34 93 581 3545
Secretary: +34 93 581 3518
http://terrestrial.eionet.europa.eu
------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--
View this message in context: http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

Heikki_Doeleman · July 25, 2009, 8:27pm

hi Pat,

glad to hear to hear that you got it working, and sorry to hear that you had
to struggle to get there.

The reason for this chain of transformations in the ArcSDE harvester is that
we try to turn things into ISO19139 ("preferred" format) for storage in
GeoNetwork, as you correctly deduced.

I think it would be in everybody's best interest to get this harvester
working more smoothly, also across different versions of ESRI's catalog.

From what you write I gather that version 9.3 returns a FGDC flavour that's

not correctly handled by the transformation in FGDC_ISO_TRANSFORMER_LOCATION
in the harvester code ?

If you could provide me with a spec on what it does actually return, I could
add a different transformation for that one to make the code work. If it's
different from ESRI catalog version 8, we could make the ESRI version a
parameter in the harvesting settings, to get things right for both 8 and 9.3
(and what about 9.2?)

Thanks very much for sharing your analysis,

kind regards
Heikki Doeleman

On Sat, Jul 25, 2009 at 6:17 PM, PatB <pbarnhart@anonymised.com> wrote:

I got it working with 9.3 - but it was a struggle. First thing I'd do is
verify the metadata you're getting out of sde 9.2 is the format you're
expecting (ISO), by default 9.3 returns esri's flavor of fgdc.

Next take a look at

...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java

around line 175 it has the following:

// create JDOM element from String-XML
Element metadataElement = Xml.loadString(metadata, false);

// transform ESRI output to ISO19115
Element iso19115 = Xml.transform(metadataElement,
FGDC_ISO_TRANSFORMER_LOCATION);

// transform ISO19115 to ISO19139
Element iso19139 = Xml.transform(iso19115,
ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);

So, if you track down the transformers and see what they're doing, it goes
from ESRI Catalog 8 (which doesn't match anything I saw coming out of 9.3)
to ISO19115 to ISO19139. I got around it by using esri's metaid as the
uuid
and commenting out the transformers above so the metadata is stored as
fgdc-std. I'm still surprised that it worked.

Good luck! It sounds like you're close, you may also need to look at:
...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java

there's a line in there that sets ISO_METADATA_IDENTIFIER to look for
MD_Metadata, which is what identifies the type of incoming metadata. I had
to change mine to "metadata" for fgdc.

Hope this helps. (oh, and you have to rebuild the jars using ant if you
mess with the .java files)

WSimonazzi wrote:
>
> Dear all,
>
> We are testing the ArcSDE harvester functionality using GN 2.4.0 (final)
> /jetty/McKoi.
>
> ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).
>
> dbtune: SDEbinary storage,
>
> Metadata: ISO 19115 and 19139.
>
> This is what we get:
>
>
----------------------------------------------------------------------------------------------------------------------------
>
> Connected to ArcSDE
>
> start retrieve metadata
>
> ISO metadata found
>
> cool
>
> Start of alignment for : REFDATABASE
>
> Recoverable error
>
> XTRE0540: Ambiguous rule match for /
>
> Matches both "document-node()" on line 5 of
>
>
>
file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913
>
> 9.xsl
>
> and "document-node()" on line 30 of
>
>
>
file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.
>
> xsl
>
> ArcSDE harvest finished
>
-------------------------------------------------------------------------------------------------------------------------------------
> An no metadata is harvested, any clue/help can be rewarded with a beer in
> Barcelona ;-).
>
> Many thanks,
>
> Walter & Maria Jose
>
> --
> --
> ================================================
> Walter Simonazzi
> European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª
> planta
> Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
> 08193 Bellaterra (Barcelona)
> Spain
> ================================================
> E-mail: walter.simonazzi (at) uab.es
> Direct phone: +34 93 581 3868
> Fax: +34 93 581 3545
> Secretary: +34 93 581 3518
> http://terrestrial.eionet.europa.eu
>
------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
>

--
View this message in context:
http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

PatB · July 27, 2009, 9:21pm

Hi Heikki,

I agree completely! It’d be awesome to have geonetwork harvest metadata
from arcsde regardless of the version or format.

I uploaded (or at least tried to, let me know if it didn't work) a dummy
metadata record exported from ArcCatalog 9.3. The esri documentation
(http://www.esri.com/metadata/) mentions several times that ‘by default’,
catalog stores metadata as fgdc with the additional tags added by esri – it
doesn’t mention how you would do it ‘not by default’. There are also a
number of stylesheets within catalog so you can display/export metadata in
whatever format you prefer (they’re on the page referenced above) but the
records are still stored as fgdc esri. I haven’t done much with metadata in
9.2 but from the original post I answered it sounds like ISO metadata was
being detected (it found ‘MD_Metadata’ in the xml blob and said cool so
maybe it depends on what format it was in when created. Not sure. They’ve
also started talking about a more detailed ESRI ISO format but I don’t think
it’s being widely used (at least not yet).

So anyway, the only other thing I ran into was that the ArcSDE query returns
all records regardless of which tables the user has rights to. I’m not sure
if this is an across the board problem or specific to my setup but I solved
it by adding a ‘Where Owner=…’ clause in the arcsdemetadataadapter.java
file.

Oh, and I replaced intermap with openlayers but I’ve already seen some posts
about that. In 2.4 RC1 the aoi search didn’t work, in RC2 it did but then
the search by category was broken, I haven’t checked it since the release of
2.4 but it might be worth looking into.

Thanks!

Heikki Doeleman wrote:

hi Pat,

glad to hear to hear that you got it working, and sorry to hear that you
had
to struggle to get there.

The reason for this chain of transformations in the ArcSDE harvester is
that
we try to turn things into ISO19139 ("preferred" format) for storage in
GeoNetwork, as you correctly deduced.

I think it would be in everybody's best interest to get this harvester
working more smoothly, also across different versions of ESRI's catalog.

From what you write I gather that version 9.3 returns a FGDC flavour
that's
not correctly handled by the transformation in
FGDC_ISO_TRANSFORMER_LOCATION
in the harvester code ?

If you could provide me with a spec on what it does actually return, I
could
add a different transformation for that one to make the code work. If it's
different from ESRI catalog version 8, we could make the ESRI version a
parameter in the harvesting settings, to get things right for both 8 and
9.3
(and what about 9.2?)

Thanks very much for sharing your analysis,

kind regards
Heikki Doeleman

On Sat, Jul 25, 2009 at 6:17 PM, PatB <pbarnhart@anonymised.com> wrote:

I got it working with 9.3 - but it was a struggle. First thing I'd do is
verify the metadata you're getting out of sde 9.2 is the format you're
expecting (ISO), by default 9.3 returns esri's flavor of fgdc.

Next take a look at

...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java

around line 175 it has the following:

// create JDOM element from String-XML
Element metadataElement = Xml.loadString(metadata, false);

// transform ESRI output to ISO19115
Element iso19115 = Xml.transform(metadataElement,
FGDC_ISO_TRANSFORMER_LOCATION);

// transform ISO19115 to ISO19139
Element iso19139 = Xml.transform(iso19115,
ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);

So, if you track down the transformers and see what they're doing, it
goes
from ESRI Catalog 8 (which doesn't match anything I saw coming out of
9.3)
to ISO19115 to ISO19139. I got around it by using esri's metaid as the
uuid
and commenting out the transformers above so the metadata is stored as
fgdc-std. I'm still surprised that it worked.

Good luck! It sounds like you're close, you may also need to look at:
...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java

there's a line in there that sets ISO_METADATA_IDENTIFIER to look for
MD_Metadata, which is what identifies the type of incoming metadata. I
had
to change mine to "metadata" for fgdc.

Hope this helps. (oh, and you have to rebuild the jars using ant if you
mess with the .java files)

WSimonazzi wrote:
>
> Dear all,
>
> We are testing the ArcSDE harvester functionality using GN 2.4.0
(final)
> /jetty/McKoi.
>
> ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).
>
> dbtune: SDEbinary storage,
>
> Metadata: ISO 19115 and 19139.
>
> This is what we get:
>
>
----------------------------------------------------------------------------------------------------------------------------
>
> Connected to ArcSDE
>
> start retrieve metadata
>
> ISO metadata found
>
> cool
>
> Start of alignment for : REFDATABASE
>
> Recoverable error
>
> XTRE0540: Ambiguous rule match for /
>
> Matches both "document-node()" on line 5 of
>
>
>
file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913
>
> 9.xsl
>
> and "document-node()" on line 30 of
>
>
>
file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.
>
> xsl
>
> ArcSDE harvest finished
>
-------------------------------------------------------------------------------------------------------------------------------------
> An no metadata is harvested, any clue/help can be rewarded with a beer
in
> Barcelona ;-).
>
> Many thanks,
>
> Walter & Maria Jose
>
> --
> --
> ================================================
> Walter Simonazzi
> European Topic Centre on Land Use and Spatial Information Torre C5-S,
4ª
> planta
> Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
> 08193 Bellaterra (Barcelona)
> Spain
> ================================================
> E-mail: walter.simonazzi (at) uab.es
> Direct phone: +34 93 581 3868
> Fax: +34 93 581 3545
> Secretary: +34 93 581 3518
> http://terrestrial.eionet.europa.eu
>
------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
>

--
View this message in context:
http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

http://n2.nabble.com/file/n3336435/utilities_storm.xml utilities_storm.xml
--
View this message in context: http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3336435.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

andrew_walsh · July 28, 2009, 5:50am

Hi PatB,

I would be interested to know in more detail about how you got the ARCSDE harvester working to harvest
fgdc metadata from ESRI as a we have a lot of geodatabases here (esri9.3 with fgdc(esri modified) metadata)
and I am the metadata guy here. I would like get my GN2.4.0 to harvest to fgdc schema in the GN catalog as
a starting point and maybe later I could convert this to ISO19139.

I have downloaded GN 2.4.0 source code, installed esri's ARCSDE jar files in /lib
and made a change to ArcSDEMetadataAdapter.java as you suggested (I changed 'MD_Metadata'
to 'metadata' and re-compiled). The harvestor is pulling out fgdc which I verified with print debug statement with Java
but goes no further as its still trying to convert version8 to ISO19115 to ISO19139.

Could let me know what lines you changed in your Java files or better still send a copy
of the changed files.

I noticed that my harvest extracted metadata for all the features (geodatabases/tables/layers).
I specified one database i.e 'RAN.secchi' in my harvest request but it harvested all the geodatabases.
How could I restrict my query to harvest just one layer rather than the whole collection?

Thanks and Regards
Andrew Walsh-DOM

----- Original Message ----- From: "PatB" <pbarnhart@anonymised.com>
To: <geonetwork-users@lists.sourceforge.net>
Sent: Tuesday, July 28, 2009 7:21 AM
Subject: Re: [GeoNetwork-users] ArcSDE harvester problems

Hi Heikki,
I agree completely! It’d be awesome to have geonetwork harvest metadatafrom arcsde regardless of the version or format.
I uploaded (or at least tried to, let me know if it didn't work) a dummymetadata record exported from ArcCatalog 9.3. The esri documentation(http://www.esri.com/metadata/) mentions several times that ‘by default’,catalog stores metadata as fgdc with the additional tags added by esri – itdoesn’t mention how you would do it ‘not by default’. There are also anumber of stylesheets within catalog so you can display/export metadata inwhatever format you prefer (they’re on the page referenced above) but therecords are still stored as fgdc esri. I haven’t done much with metadata in9.2 but from the original post I answered it sounds like ISO metadata wasbeing detected (it found ‘MD_Metadata’ in the xml blob and said cool somaybe it depends on what format it was in when created. Not sure. They’vealso started talking about a more detailed ESRI ISO format but I don’t thinkit’s being widely used (at least not yet).
So anyway, the only other thing I ran into was that the ArcSDE query returnsall records regardless of which tables the user has rights to. I’m not sureif this is an across the board problem or specific to my setup but I solvedit by adding a ‘Where Owner=…’ clause in the arcsdemetadataadapter.javafile.
Oh, and I replaced intermap with openlayers but I’ve already seen some postsabout that. In 2.4 RC1 the aoi search didn’t work, in RC2 it did but thenthe search by category was broken, I haven’t checked it since the release of2.4 but it might be worth looking into.
Thanks!

Heikki Doeleman wrote:> > hi Pat,> > glad to hear to hear that you got it working, and sorry to hear that you> had> to struggle to get there.> > The reason for this chain of transformations in the ArcSDE harvester is> that> we try to turn things into ISO19139 ("preferred" format) for storage in> GeoNetwork, as you correctly deduced.> > I think it would be in everybody's best interest to get this harvester> working more smoothly, also across different versions of ESRI's catalog.> > From what you write I gather that version 9.3 returns a FGDC flavour> that's> not correctly handled by the transformation in> FGDC_ISO_TRANSFORMER_LOCATION> in the harvester code ?> > If you could provide me with a spec on what it does actually return, I> could> add a different transformation for that one to make the code work. If it's> different from ESRI catalog version 8, we could make the ESRI version a> parameter in the harvesting settings, to get things right for both 8 and> 9.3> (and what about 9.2?)> > Thanks very much for sharing your analysis,> > kind regards> Heikki Doeleman> > > On Sat, Jul 25, 2009 at 6:17 PM, PatB <pbarnhart@anonymised.com> wrote:> >>>> I got it working with 9.3 - but it was a struggle. First thing I'd do is>> verify the metadata you're getting out of sde 9.2 is the format you're>> expecting (ISO), by default 9.3 returns esri's flavor of fgdc.>>>> Next take a look at>>>> ...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java>>>> around line 175 it has the following:>>>> // create JDOM element from String-XML>> Element metadataElement = Xml.loadString(metadata, false);>>>> // transform ESRI output to ISO19115>> Element iso19115 = Xml.transform(metadataElement,>> FGDC_ISO_TRANSFORMER_LOCATION);>>>> // transform ISO19115 to ISO19139>> Element iso19139 = Xml.transform(iso19115,>> ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);>>>> So, if you track down the transformers and see what they're doing, it>> goes>> from ESRI Catalog 8 (which doesn't match anything I saw coming out of>> 9.3)>> to ISO19115 to ISO19139. I got around it by using esri's metaid as the>> uuid>> and commenting out the transformers above so the metadata is stored as>> fgdc-std. I'm still surprised that it worked.>>>> Good luck! It sounds like you're close, you may also need to look at:>> ...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java>>>> there's a line in there that sets ISO_METADATA_IDENTIFIER to look for>> MD_Metadata, which is what identifies the type of incoming metadata. I>> had>> to change mine to "metadata" for fgdc.>>>> Hope this helps. (oh, and you have to rebuild the jars using ant if you>> mess with the .java files)>>>>>> WSimonazzi wrote:>> >>> > Dear all,>> >>> > We are testing the ArcSDE harvester functionality using GN 2.4.0>> (final)>> > /jetty/McKoi.>> >>> > ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).>> >>> > dbtune: SDEbinary storage,>> >>> > Metadata: ISO 19115 and 19139.>> >>> > This is what we get:>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------->>> >>> > Connected to ArcSDE>> >>> > start retrieve metadata>> >>> > ISO metadata found>> >>> > cool>> >>> > Start of alignment for : REFDATABASE>> >>> > Recoverable error>> >>> > XTRE0540: Ambiguous rule match for />> >>> > Matches both "document-node()" on line 5 of>> >>> >>> >>> file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913>> >>> > 9.xsl>> >>> > and "document-node()" on line 30 of>> >>> >>> >>> file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.>> >>> > xsl>> >>> > ArcSDE harvest finished>> >>> -------------------------------------------------------------------------------------------------------------------------------------->> > An no metadata is harvested, any clue/help can be rewarded with a beer>> in>> > Barcelona ;-).>> >>> > Many thanks,>> >>> > Walter & Maria Jose>> >>> > -->> > -->> > ================================================>> > Walter Simonazzi>> > European Topic Centre on Land Use and Spatial Information Torre C5-S,>> 4ª>> > planta>> > Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona>> > 08193 Bellaterra (Barcelona)>> > Spain>> > ================================================>> > E-mail: walter.simonazzi (at) uab.es>> > Direct phone: +34 93 581 3868>> > Fax: +34 93 581 3545>> > Secretary: +34 93 581 3518>> > http://terrestrial.eionet.europa.eu>> >>> ------------------------------------------------------------------------------>> > _______________________________________________>> > GeoNetwork-users mailing list>> > GeoNetwork-users@lists.sourceforge.net>> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users>> > GeoNetwork OpenSource is maintained at>> > http://sourceforge.net/projects/geonetwork>> >>> >>>>> -->> View this message in context:>> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html>> Sent from the GeoNetwork users mailing list archive at Nabble.com.>>>>>> ------------------------------------------------------------------------------>> _______________________________________________>> GeoNetwork-users mailing list>> GeoNetwork-users@lists.sourceforge.net>> https://lists.sourceforge.net/lists/listinfo/geonetwork-users>> GeoNetwork OpenSource is maintained at>> http://sourceforge.net/projects/geonetwork>>> ------------------------------------------------------------------------------> _______________________________________________> GeoNetwork-users mailing list> GeoNetwork-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/geonetwork-users> GeoNetwork OpenSource is maintained at> http://sourceforge.net/projects/geonetwork> > http://n2.nabble.com/file/n3336435/utilities_storm.xml utilities_storm.xml -- View this message in context: http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3336435.htmlSent from the GeoNetwork users mailing list archive at Nabble.com.
------------------------------------------------------------------------------Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july_______________________________________________GeoNetwork-users mailing listGeoNetwork-users@anonymised.com://lists.sourceforge.net/lists/listinfo/geonetwork-usersGeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Heikki_Doeleman · July 28, 2009, 11:29am

hi Pat and all other (potential) ArcSDE harvesters,

thanks for posting the ESRI output, yes that worked

I've run this through the transformations in ArcSDEHarvester, with the
following results :

(1) ARC_TO_ISO19115_TRANSFORMER_LOCATION :

produces well-formed XML, however it's not completely valid w.r.t. the
ISO19115 XSD in GeoNetwork's
<install-dir>/web/geonetwork/xml/schemas/iso19115/schema.xsd.

Now I'm not too sure what ISO19115 XSD that actually is, because as far as I
understood the ISO19139 XSD is the schema for the ISO19115 spec. I think it
was some ad-hoc XSD made for ISO19115 from before the existence of the
ISO19139 XSD, but as I said I'm not sure about this.

In any case, I repaired the output to make it valid for that ISO19115 XSD
(had to add an <mdContact> element, an <idCitation> element, and a <tpCat>
element).

Using this modified ISO19115, I ran it through the second transformation :

(2) ISO19115_TO_ISO19139_TRANSFORMER_LOCATION

Again this resulted in a well-formed XML document, but not totally valid
w.r.t. to the ISO19139 XSD (I used GeoNetwork's copy in
<install-dir>/web/geonetwork/xml/schemas/iso19139/schema.xsd).

To make it valid ISO19139 I had to change the format of
MD_Metadata/dateStamp/gco:DateTime to an actual date-time format (the value
produced in the transformation was something like "20040901"), and I had to
add a value to
MD_Metadata/identificationInfo/MD_DataIdentification/citation/CI_Citation/date/CI_Date/date/gco:DateTime
(it did not have a value).

So I think we can make these empirical observations:

* ArcSDE appears to put out FGDC-ish metadata, but there are also cases
where it appears to put out ISO19139 ;

* The FGDC output from version 9.3 is not transformed correctly by
ARC_TO_ISO19115_TRANSFORMER_LOCATION. This can be due to incomplete FGDC
output from ArcSDE, or due to some missing steps in that transformation. Or
it can be that the ISO19115 XSD in GeoNetwork is not 100% appropriate ;

* The ISO19139 output from the transformation in
ISO19115_TO_ISO19139_TRANSFORMER_LOCATION does not transform to a correct
ISO19139 format. This seems due to wrong assumptions about the input XML.

Now to make the ArcSDEHarvester work like a charm for all cases, I propose
the following :

(I) it should scan for "MD_Metadata" in the output, as it does now; these
metadata should not be subjected to the chain of transformations to turn
FGDC into ISO19139, obviously. It should be saved in GeoNetwork as it is,
assuming that this really is valid (or at least not too badly misformed)
ISO19139 ;

(II) it should also scan for "metadata" in the output. These metadata should
go through the chain of transformations to turn it into ISO19139, as that is
GN's preferred format. The transformations need to be made a bit more robust
to avoid putting out invalid formats -- even if the input XML may be
incomplete or a bit malformed.

(III) if the community really insists on it I could make it optional to
transform things to ISO19139, if you must save it in FGDC format

(IV) we should find out what exactly is that ISO19115 XSD in GeoNetwork. Who
made this, and for what purpose ? And it would surely perform better (and be
less error-prone) if we transformed the FGDC directly to ISO19139, in just a
single step. Should we aim for that ?

(V) we should find out if there are differences between various versions of
ArcSDE, for both the FGDC and ISO output, that we should take into account.
So I'd like to ask everybody who has some version of ArcSDE, could you
please share some example metadata output from your catalog ? Or is there an
ESRI employee in the audience who can shed light on this ?

(VI) as for accessing only your targeted database schema, I'll take a look
again at the documentation of ESRI's plug-in, but I do not have any ArcSDE
available for testing. Anyone who does, willing to help me test this ?

Looking forward to your reactions,

kind regards
Heikki Doeleman

On Mon, Jul 27, 2009 at 11:21 PM, PatB <pbarnhart@anonymised.com> wrote:

Hi Heikki,

I agree completely! It’d be awesome to have geonetwork harvest metadata
from arcsde regardless of the version or format.

I uploaded (or at least tried to, let me know if it didn't work) a dummy
metadata record exported from ArcCatalog 9.3. The esri documentation
(http://www.esri.com/metadata/) mentions several times that ‘by default’,
catalog stores metadata as fgdc with the additional tags added by esri – it
doesn’t mention how you would do it ‘not by default’. There are also a
number of stylesheets within catalog so you can display/export metadata in
whatever format you prefer (they’re on the page referenced above) but the
records are still stored as fgdc esri. I haven’t done much with metadata
in
9.2 but from the original post I answered it sounds like ISO metadata was
being detected (it found ‘MD_Metadata’ in the xml blob and said cool so
maybe it depends on what format it was in when created. Not sure. They’ve
also started talking about a more detailed ESRI ISO format but I don’t
think
it’s being widely used (at least not yet).

So anyway, the only other thing I ran into was that the ArcSDE query
returns
all records regardless of which tables the user has rights to. I’m not
sure
if this is an across the board problem or specific to my setup but I solved
it by adding a ‘Where Owner=…’ clause in the arcsdemetadataadapter.java
file.

Oh, and I replaced intermap with openlayers but I’ve already seen some
posts
about that. In 2.4 RC1 the aoi search didn’t work, in RC2 it did but then
the search by category was broken, I haven’t checked it since the release
of
2.4 but it might be worth looking into.

Thanks!

Heikki Doeleman wrote:
>
> hi Pat,
>
> glad to hear to hear that you got it working, and sorry to hear that you
> had
> to struggle to get there.
>
> The reason for this chain of transformations in the ArcSDE harvester is
> that
> we try to turn things into ISO19139 ("preferred" format) for storage in
> GeoNetwork, as you correctly deduced.
>
> I think it would be in everybody's best interest to get this harvester
> working more smoothly, also across different versions of ESRI's catalog.
>
> From what you write I gather that version 9.3 returns a FGDC flavour
> that's
> not correctly handled by the transformation in
> FGDC_ISO_TRANSFORMER_LOCATION
> in the harvester code ?
>
> If you could provide me with a spec on what it does actually return, I
> could
> add a different transformation for that one to make the code work. If
it's
> different from ESRI catalog version 8, we could make the ESRI version a
> parameter in the harvesting settings, to get things right for both 8 and
> 9.3
> (and what about 9.2?)
>
> Thanks very much for sharing your analysis,
>
> kind regards
> Heikki Doeleman
>
>
> On Sat, Jul 25, 2009 at 6:17 PM, PatB <pbarnhart@anonymised.com>
wrote:
>
>>
>> I got it working with 9.3 - but it was a struggle. First thing I'd do
is
>> verify the metadata you're getting out of sde 9.2 is the format you're
>> expecting (ISO), by default 9.3 returns esri's flavor of fgdc.
>>
>> Next take a look at
>>
>>
...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java
>>
>> around line 175 it has the following:
>>
>> // create JDOM element from String-XML
>> Element metadataElement = Xml.loadString(metadata, false);
>>
>> // transform ESRI output to ISO19115
>> Element iso19115 = Xml.transform(metadataElement,
>> FGDC_ISO_TRANSFORMER_LOCATION);
>>
>> // transform ISO19115 to ISO19139
>> Element iso19139 = Xml.transform(iso19115,
>> ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);
>>
>> So, if you track down the transformers and see what they're doing, it
>> goes
>> from ESRI Catalog 8 (which doesn't match anything I saw coming out of
>> 9.3)
>> to ISO19115 to ISO19139. I got around it by using esri's metaid as the
>> uuid
>> and commenting out the transformers above so the metadata is stored as
>> fgdc-std. I'm still surprised that it worked.
>>
>> Good luck! It sounds like you're close, you may also need to look at:
>> ...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java
>>
>> there's a line in there that sets ISO_METADATA_IDENTIFIER to look for
>> MD_Metadata, which is what identifies the type of incoming metadata. I
>> had
>> to change mine to "metadata" for fgdc.
>>
>> Hope this helps. (oh, and you have to rebuild the jars using ant if you
>> mess with the .java files)
>>
>>
>> WSimonazzi wrote:
>> >
>> > Dear all,
>> >
>> > We are testing the ArcSDE harvester functionality using GN 2.4.0
>> (final)
>> > /jetty/McKoi.
>> >
>> > ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).
>> >
>> > dbtune: SDEbinary storage,
>> >
>> > Metadata: ISO 19115 and 19139.
>> >
>> > This is what we get:
>> >
>> >
>>
----------------------------------------------------------------------------------------------------------------------------
>> >
>> > Connected to ArcSDE
>> >
>> > start retrieve metadata
>> >
>> > ISO metadata found
>> >
>> > cool
>> >
>> > Start of alignment for : REFDATABASE
>> >
>> > Recoverable error
>> >
>> > XTRE0540: Ambiguous rule match for /
>> >
>> > Matches both "document-node()" on line 5 of
>> >
>> >
>> >
>>
file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913
>> >
>> > 9.xsl
>> >
>> > and "document-node()" on line 30 of
>> >
>> >
>> >
>>
file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.
>> >
>> > xsl
>> >
>> > ArcSDE harvest finished
>> >
>>
-------------------------------------------------------------------------------------------------------------------------------------
>> > An no metadata is harvested, any clue/help can be rewarded with a beer
>> in
>> > Barcelona ;-).
>> >
>> > Many thanks,
>> >
>> > Walter & Maria Jose
>> >
>> > --
>> > --
>> > ================================================
>> > Walter Simonazzi
>> > European Topic Centre on Land Use and Spatial Information Torre C5-S,
>> 4ª
>> > planta
>> > Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
>> > 08193 Bellaterra (Barcelona)
>> > Spain
>> > ================================================
>> > E-mail: walter.simonazzi (at) uab.es
>> > Direct phone: +34 93 581 3868
>> > Fax: +34 93 581 3545
>> > Secretary: +34 93 581 3518
>> > http://terrestrial.eionet.europa.eu
>> >
>>
------------------------------------------------------------------------------
>> > _______________________________________________
>> > GeoNetwork-users mailing list
>> > GeoNetwork-users@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>> > GeoNetwork OpenSource is maintained at
>> > http://sourceforge.net/projects/geonetwork
>> >
>> >
>>
>> --
>> View this message in context:
>> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html
>> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>>
>>
>>
------------------------------------------------------------------------------
>> _______________________________________________
>> GeoNetwork-users mailing list
>> GeoNetwork-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>>
>
------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
>
http://n2.nabble.com/file/n3336435/utilities_storm.xml utilities_storm.xml
--
View this message in context:
http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3336435.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus
on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

Francois_Prunayre · July 28, 2009, 7:08pm

Hello Heikki, thanks for this proposal, Some comments on your questions.

2009/7/28 heikki <tropicano@anonymised.com>:

(I) it should scan for "MD_Metadata" in the output, as it does now; these
metadata should not be subjected to the chain of transformations to turn
FGDC into ISO19139, obviously. It should be saved in GeoNetwork as it is,
assuming that this really is valid (or at least not too badly misformed)
ISO19139 ;

Sounds ok.

(II) it should also scan for "metadata" in the output. These metadata should
go through the chain of transformations to turn it into ISO19139, as that is
GN's preferred format. The transformations need to be made a bit more robust
to avoid putting out invalid formats -- even if the input XML may be
incomplete or a bit malformed.

Even if a record if not XSD valid when harvested it should not cause
so many troubles in GeoNetwork but it could be nice to fix has many
issues as we can during the XSL processing. Missing contact or topic
category should be ok, maybe the use of Date instead of DateTime could
be a problem for temporal search. When you edit a new record,
sometimes it could be not XSD valid but useful and could work well in
the catalogue.

(III) if the community really insists on it I could make it optional to
transform things to ISO19139, if you must save it in FGDC format

(IV) we should find out what exactly is that ISO19115 XSD in GeoNetwork. Who
made this, and for what purpose ?

This schema has been used when available (not sure who made it)
because iso 19139 was not available when iso 19115 FDIS was released,
so lots of people used it. The ESRI profil was also based on that
schema (ArcCatalogue 8.X I think, the one used when the stylsheet was
made in GeoNetwork).

And it would surely perform better (and be

less error-prone) if we transformed the FGDC directly to ISO19139, in just a
single step. Should we aim for that ?

I heard about some ongoing work to migrate from FGDC to ISO, maybe a
XSL transformation is also available to make a one step transformation
?

Cheers.
Francois

(V) we should find out if there are differences between various versions of
ArcSDE, for both the FGDC and ISO output, that we should take into account.
So I'd like to ask everybody who has some version of ArcSDE, could you
please share some example metadata output from your catalog ? Or is there an
ESRI employee in the audience who can shed light on this ?

(VI) as for accessing only your targeted database schema, I'll take a look
again at the documentation of ESRI's plug-in, but I do not have any ArcSDE
available for testing. Anyone who does, willing to help me test this ?

Looking forward to your reactions,

kind regards
Heikki Doeleman

On Mon, Jul 27, 2009 at 11:21 PM, PatB <pbarnhart@anonymised.com> wrote:

Hi Heikki,

I agree completely! It’d be awesome to have geonetwork harvest metadata
from arcsde regardless of the version or format.

I uploaded (or at least tried to, let me know if it didn't work) a dummy
metadata record exported from ArcCatalog 9.3. The esri documentation
(http://www.esri.com/metadata/) mentions several times that ‘by default’,
catalog stores metadata as fgdc with the additional tags added by esri – it
doesn’t mention how you would do it ‘not by default’. There are also a
number of stylesheets within catalog so you can display/export metadata in
whatever format you prefer (they’re on the page referenced above) but the
records are still stored as fgdc esri. I haven’t done much with metadata
in
9.2 but from the original post I answered it sounds like ISO metadata was
being detected (it found ‘MD_Metadata’ in the xml blob and said cool so
maybe it depends on what format it was in when created. Not sure. They’ve
also started talking about a more detailed ESRI ISO format but I don’t
think
it’s being widely used (at least not yet).

So anyway, the only other thing I ran into was that the ArcSDE query
returns
all records regardless of which tables the user has rights to. I’m not
sure
if this is an across the board problem or specific to my setup but I solved
it by adding a ‘Where Owner=…’ clause in the arcsdemetadataadapter.java
file.

Oh, and I replaced intermap with openlayers but I’ve already seen some
posts
about that. In 2.4 RC1 the aoi search didn’t work, in RC2 it did but then
the search by category was broken, I haven’t checked it since the release
of
2.4 but it might be worth looking into.

Thanks!

Heikki Doeleman wrote:
>
> hi Pat,
>
> glad to hear to hear that you got it working, and sorry to hear that you
> had
> to struggle to get there.
>
> The reason for this chain of transformations in the ArcSDE harvester is
> that
> we try to turn things into ISO19139 ("preferred" format) for storage in
> GeoNetwork, as you correctly deduced.
>
> I think it would be in everybody's best interest to get this harvester
> working more smoothly, also across different versions of ESRI's catalog.
>
> From what you write I gather that version 9.3 returns a FGDC flavour
> that's
> not correctly handled by the transformation in
> FGDC_ISO_TRANSFORMER_LOCATION
> in the harvester code ?
>
> If you could provide me with a spec on what it does actually return, I
> could
> add a different transformation for that one to make the code work. If
it's
> different from ESRI catalog version 8, we could make the ESRI version a
> parameter in the harvesting settings, to get things right for both 8 and
> 9.3
> (and what about 9.2?)
>
> Thanks very much for sharing your analysis,
>
> kind regards
> Heikki Doeleman
>
>
> On Sat, Jul 25, 2009 at 6:17 PM, PatB <pbarnhart@anonymised.com>
wrote:
>
>>
>> I got it working with 9.3 - but it was a struggle. First thing I'd do
is
>> verify the metadata you're getting out of sde 9.2 is the format you're
>> expecting (ISO), by default 9.3 returns esri's flavor of fgdc.
>>
>> Next take a look at
>>
>>
...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java
>>
>> around line 175 it has the following:
>>
>> // create JDOM element from String-XML
>> Element metadataElement = Xml.loadString(metadata, false);
>>
>> // transform ESRI output to ISO19115
>> Element iso19115 = Xml.transform(metadataElement,
>> FGDC_ISO_TRANSFORMER_LOCATION);
>>
>> // transform ISO19115 to ISO19139
>> Element iso19139 = Xml.transform(iso19115,
>> ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);
>>
>> So, if you track down the transformers and see what they're doing, it
>> goes
>> from ESRI Catalog 8 (which doesn't match anything I saw coming out of
>> 9.3)
>> to ISO19115 to ISO19139. I got around it by using esri's metaid as the
>> uuid
>> and commenting out the transformers above so the metadata is stored as
>> fgdc-std. I'm still surprised that it worked.
>>
>> Good luck! It sounds like you're close, you may also need to look at:
>> ...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java
>>
>> there's a line in there that sets ISO_METADATA_IDENTIFIER to look for
>> MD_Metadata, which is what identifies the type of incoming metadata. I
>> had
>> to change mine to "metadata" for fgdc.
>>
>> Hope this helps. (oh, and you have to rebuild the jars using ant if you
>> mess with the .java files)
>>
>>
>> WSimonazzi wrote:
>> >
>> > Dear all,
>> >
>> > We are testing the ArcSDE harvester functionality using GN 2.4.0
>> (final)
>> > /jetty/McKoi.
>> >
>> > ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).
>> >
>> > dbtune: SDEbinary storage,
>> >
>> > Metadata: ISO 19115 and 19139.
>> >
>> > This is what we get:
>> >
>> >
>>
----------------------------------------------------------------------------------------------------------------------------
>> >
>> > Connected to ArcSDE
>> >
>> > start retrieve metadata
>> >
>> > ISO metadata found
>> >
>> > cool
>> >
>> > Start of alignment for : REFDATABASE
>> >
>> > Recoverable error
>> >
>> > XTRE0540: Ambiguous rule match for /
>> >
>> > Matches both "document-node()" on line 5 of
>> >
>> >
>> >
>>
file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913
>> >
>> > 9.xsl
>> >
>> > and "document-node()" on line 30 of
>> >
>> >
>> >
>>
file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.
>> >
>> > xsl
>> >
>> > ArcSDE harvest finished
>> >
>>
-------------------------------------------------------------------------------------------------------------------------------------
>> > An no metadata is harvested, any clue/help can be rewarded with a beer
>> in
>> > Barcelona ;-).
>> >
>> > Many thanks,
>> >
>> > Walter & Maria Jose
>> >
>> > --
>> > --
>> > ================================================
>> > Walter Simonazzi
>> > European Topic Centre on Land Use and Spatial Information Torre C5-S,
>> 4ª
>> > planta
>> > Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
>> > 08193 Bellaterra (Barcelona)
>> > Spain
>> > ================================================
>> > E-mail: walter.simonazzi (at) uab.es
>> > Direct phone: +34 93 581 3868
>> > Fax: +34 93 581 3545
>> > Secretary: +34 93 581 3518
>> > http://terrestrial.eionet.europa.eu
>> >
>>
------------------------------------------------------------------------------
>> > _______________________________________________
>> > GeoNetwork-users mailing list
>> > GeoNetwork-users@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>> > GeoNetwork OpenSource is maintained at
>> > http://sourceforge.net/projects/geonetwork
>> >
>> >
>>
>> --
>> View this message in context:
>> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html
>> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>>
>>
>>
------------------------------------------------------------------------------
>> _______________________________________________
>> GeoNetwork-users mailing list
>> GeoNetwork-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>>
>
------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
>
http://n2.nabble.com/file/n3336435/utilities_storm.xml utilities_storm.xml
--
View this message in context:
http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3336435.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus
on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

andrew_walsh · August 5, 2009, 3:01am

Hi Heikki, Francois, PatB, List,

I have done some more investigation on ESRI's ARCSDE metadata
(we have ESRI 9.3 here) and run some metadata from a point feature type dataset (see file attached)
through the GN2.4 xsl translators ARC_TO_ISO19115 and then ISO19115_TO_ISO19139.

I found very similar data and validation issues that Heikki found ie:
- had to add a 'mdContact' element to ISO19115
-added missing idCitation element with resTitle element to ISO19115 (the title)
-added missing idAbs element to ISO19115 (the abstract)
-added missing Topic category 'tpCat' to ISO19115
-Edited dates in YYYYMMDD in ISO19115 to gco:datetime type YYMMDDThh:mm:ssZ valid for ISO19139

I did not see a root element 'MD_Metadata' stored in the metadata column in the ESRI 9.3 database tables.
Not sure where MD_Metadata comes from?, perhaps it comes from a conversion tool that ESRI supply
to convert data to ISO19139.

I studied the ESRI metadata and compared it to the DTD they supply (see http://www.esri.com/metadata/
and click on link 'esriprof80.dtd' to download). The metadata has additional elements not in the
DTD so it not possible to validate against this DTD. Also note the following as stated on ESRI's metadata
page:

"NOTE: DTDs require the elements in an XML file to be in a specific order. ArcCatalog cannot guarantee that the elements in the metadata files that it maintains will be in the correct order. A separate operation must order the elements in the XML file before it can be validated by the DTD"

For those who want more detail on the ESRI metadata format see this link
http://support.esri.com/index.cfm?fa=knowledgebase.whitepapers.viewPaper&PID=43&MetaID=239
and download the pdf.

Failing DTD validation it is possible to do some validation against ESRI's metadata using a tool called 'mp'
supplied by the US Geological Survey (see http://geology.usgs.gov/tools/metadata/).
'mp' can take into account extended elements that ERSI have added to the FGDC standard if
one specifies an extension file on the command line. I downloaded and installed 'mp' and got the extension file from
http://geology.usgs.gov/tools/metadata/tools/ext/ and click on link 'esriprof80.ext.
I managed to run mp on the file attached and this was useful as it highlighted some missing mandatory elements and also
some new elements ESRI has added recently e.g DataProperties/lineage/process but not documented
in the .ext file.

After looking at the ISO_19139 data that comes out the xsl translations I have following comments
about the translation process:

* translation didn't extract 'citation->title' and 'abstract' elements but these are existing in the ESRI metadata.

* Topic category doesn't exist in ESRI metadata - I guess we could specify this in the harvest parameters.

* Contact information seems to be in the ESRI metadata in metadata/metainfo/metc/cntinfo but its
not getting put across to the ISO19139 MD_Metadata/contact element.

*Dates are stored as YYYYMMDD in ESRI so they need to be converted to YYYY-MM-DD and stored
as gco:Date type in ISO19139

*There is other useful information in the ESRI metdata which could be extracted and stored in ISO19139.
For example there is attribute information for features in metadata/eainfo/detailed/attr/attrlabl. This information
could be listed in MD_Meatadata/gmd:featureAttribute. Also MD_Metadata/gmd:FeatureType could store the
feature type e.g point, line or polygon.

Regarding the harvestor setup page it could be enhanced as follows:

-Could specify a Topic category

-Could provide more control over what the harvestor extracts. Currently the harvestor pulls out all the databases/feature classes
in the collection but user may want to harvest by SQL criteria. To do this user could specify an 'Owner' and/or 'Name' and/or 'DatasetType'.
(Note: These are database column names in the ESRI metadata table named SDE.GDB_USERMETADATA. There is also
a column called 'DatabaseName' but if you use Oracle this is null)

Given the above there is definitely some more work needed on the harvester xsl translation from ESRI to ISO19139.
I think this should be a 1 step process i.e don't need the intermediate step from ESRI to ISO19115.

Andrew Walsh

----- Original Message ----- From: "heikki" <tropicano@anonymised.com>
To: "PatB" <pbarnhart@anonymised.com>
Cc: <geonetwork-users@lists.sourceforge.net>
Sent: Tuesday, July 28, 2009 9:29 PM
Subject: Re: [GeoNetwork-users] ArcSDE harvester problems

hi Pat and all other (potential) ArcSDE harvesters,
thanks for posting the ESRI output, yes that worked
I've run this through the transformations in ArcSDEHarvester, with thefollowing results :
(1) ARC_TO_ISO19115_TRANSFORMER_LOCATION :
produces well-formed XML, however it's not completely valid w.r.t. theISO19115 XSD in GeoNetwork's<install-dir>/web/geonetwork/xml/schemas/iso19115/schema.xsd.
Now I'm not too sure what ISO19115 XSD that actually is, because as far as Iunderstood the ISO19139 XSD is the schema for the ISO19115 spec. I think itwas some ad-hoc XSD made for ISO19115 from before the existence of theISO19139 XSD, but as I said I'm not sure about this.
In any case, I repaired the output to make it valid for that ISO19115 XSD(had to add an <mdContact> element, an <idCitation> element, and a <tpCat>element).
Using this modified ISO19115, I ran it through the second transformation :
(2) ISO19115_TO_ISO19139_TRANSFORMER_LOCATION
Again this resulted in a well-formed XML document, but not totally validw.r.t. to the ISO19139 XSD (I used GeoNetwork's copy in<install-dir>/web/geonetwork/xml/schemas/iso19139/schema.xsd).
To make it valid ISO19139 I had to change the format ofMD_Metadata/dateStamp/gco:DateTime to an actual date-time format (the valueproduced in the transformation was something like "20040901"), and I had toadd a value toMD_Metadata/identificationInfo/MD_DataIdentification/citation/CI_Citation/date/CI_Date/date/gco:DateTime(it did not have a value).

So I think we can make these empirical observations:
* ArcSDE appears to put out FGDC-ish metadata, but there are also caseswhere it appears to put out ISO19139 ;
* The FGDC output from version 9.3 is not transformed correctly byARC_TO_ISO19115_TRANSFORMER_LOCATION. This can be due to incomplete FGDCoutput from ArcSDE, or due to some missing steps in that transformation. Orit can be that the ISO19115 XSD in GeoNetwork is not 100% appropriate ;
* The ISO19139 output from the transformation inISO19115_TO_ISO19139_TRANSFORMER_LOCATION does not transform to a correctISO19139 format. This seems due to wrong assumptions about the input XML.

Now to make the ArcSDEHarvester work like a charm for all cases, I proposethe following :
(I) it should scan for "MD_Metadata" in the output, as it does now; thesemetadata should not be subjected to the chain of transformations to turnFGDC into ISO19139, obviously. It should be saved in GeoNetwork as it is,assuming that this really is valid (or at least not too badly misformed)ISO19139 ;
(II) it should also scan for "metadata" in the output. These metadata shouldgo through the chain of transformations to turn it into ISO19139, as that isGN's preferred format. The transformations need to be made a bit more robustto avoid putting out invalid formats -- even if the input XML may beincomplete or a bit malformed.
(III) if the community really insists on it I could make it optional totransform things to ISO19139, if you must save it in FGDC format
(IV) we should find out what exactly is that ISO19115 XSD in GeoNetwork. Whomade this, and for what purpose ? And it would surely perform better (and beless error-prone) if we transformed the FGDC directly to ISO19139, in just asingle step. Should we aim for that ?
(V) we should find out if there are differences between various versions ofArcSDE, for both the FGDC and ISO output, that we should take into account.So I'd like to ask everybody who has some version of ArcSDE, could youplease share some example metadata output from your catalog ? Or is there anESRI employee in the audience who can shed light on this ?
(VI) as for accessing only your targeted database schema, I'll take a lookagain at the documentation of ESRI's plug-in, but I do not have any ArcSDEavailable for testing. Anyone who does, willing to help me test this ?

Looking forward to your reactions,
kind regardsHeikki Doeleman

On Mon, Jul 27, 2009 at 11:21 PM, PatB <pbarnhart@anonymised.com> wrote:

Hi Heikki,>> I agree completely! It’d be awesome to have geonetwork harvest metadata> from arcsde regardless of the version or format.>> I uploaded (or at least tried to, let me know if it didn't work) a dummy> metadata record exported from ArcCatalog 9.3. The esri documentation> (http://www.esri.com/metadata/) mentions several times that ‘by default’,> catalog stores metadata as fgdc with the additional tags added by esri – it> doesn’t mention how you would do it ‘not by default’. There are also a> number of stylesheets within catalog so you can display/export metadata in> whatever format you prefer (they’re on the page referenced above) but the> records are still stored as fgdc esri. I haven’t done much with metadata> in> 9.2 but from the original post I answered it sounds like ISO metadata was> being detected (it found ‘MD_Metadata’ in the xml blob and said cool so> maybe it depends on what format it was in when created. Not sure. They’ve> also started talking about a more detailed ESRI ISO format but I don’t> think> it’s being widely used (at least not yet).>> So anyway, the only other thing I ran into was that the ArcSDE query> returns> all records regardless of which tables the user has rights to. I’m not> sure> if this is an across the board problem or specific to my setup but I solved> it by adding a ‘Where Owner=…’ clause in the arcsdemetadataadapter.java> file.>> Oh, and I replaced intermap with openlayers but I’ve already seen some> posts> about that. In 2.4 RC1 the aoi search didn’t work, in RC2 it did but then> the search by category was broken, I haven’t checked it since the release> of> 2.4 but it might be worth looking into.>> Thanks!>>> Heikki Doeleman wrote:> >> > hi Pat,> >> > glad to hear to hear that you got it working, and sorry to hear that you> > had> > to struggle to get there.> >> > The reason for this chain of transformations in the ArcSDE harvester is> > that> > we try to turn things into ISO19139 ("preferred" format) for storage in> > GeoNetwork, as you correctly deduced.> >> > I think it would be in everybody's best interest to get this harvester> > working more smoothly, also across different versions of ESRI's catalog.> >> > From what you write I gather that version 9.3 returns a FGDC flavour> > that's> > not correctly handled by the transformation in> > FGDC_ISO_TRANSFORMER_LOCATION> > in the harvester code ?> >> > If you could provide me with a spec on what it does actually return, I> > could> > add a different transformation for that one to make the code work. If> it's> > different from ESRI catalog version 8, we could make the ESRI version a> > parameter in the harvesting settings, to get things right for both 8 and> > 9.3> > (and what about 9.2?)> >> > Thanks very much for sharing your analysis,> >> > kind regards> > Heikki Doeleman> >> >> > On Sat, Jul 25, 2009 at 6:17 PM, PatB <pbarnhart@anonymised.com>> wrote:> >> >>> >> I got it working with 9.3 - but it was a struggle. First thing I'd do> is> >> verify the metadata you're getting out of sde 9.2 is the format you're> >> expecting (ISO), by default 9.3 returns esri's flavor of fgdc.> >>> >> Next take a look at> >>> >>> ...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java> >>> >> around line 175 it has the following:> >>> >> // create JDOM element from String-XML> >> Element metadataElement = Xml.loadString(metadata, false);> >>> >> // transform ESRI output to ISO19115> >> Element iso19115 = Xml.transform(metadataElement,> >> FGDC_ISO_TRANSFORMER_LOCATION);> >>> >> // transform ISO19115 to ISO19139> >> Element iso19139 = Xml.transform(iso19115,> >> ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);> >>> >> So, if you track down the transformers and see what they're doing, it> >> goes> >> from ESRI Catalog 8 (which doesn't match anything I saw coming out of> >> 9.3)> >> to ISO19115 to ISO19139. I got around it by using esri's metaid as the> >> uuid> >> and commenting out the transformers above so the metadata is stored as> >> fgdc-std. I'm still surprised that it worked.> >>> >> Good luck! It sounds like you're close, you may also need to look at:> >> ...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java> >>> >> there's a line in there that sets ISO_METADATA_IDENTIFIER to look for> >> MD_Metadata, which is what identifies the type of incoming metadata. I> >> had> >> to change mine to "metadata" for fgdc.> >>> >> Hope this helps. (oh, and you have to rebuild the jars using ant if you> >> mess with the .java files)> >>> >>> >> WSimonazzi wrote:> >> >> >> > Dear all,> >> >> >> > We are testing the ArcSDE harvester functionality using GN 2.4.0> >> (final)> >> > /jetty/McKoi.> >> >> >> > ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).> >> >> >> > dbtune: SDEbinary storage,> >> >> >> > Metadata: ISO 19115 and 19139.> >> >> >> > This is what we get:> >> >> >> >> >>> ----------------------------------------------------------------------------------------------------------------------------> >> >> >> > Connected to ArcSDE> >> >> >> > start retrieve metadata> >> >> >> > ISO metadata found> >> >> >> > cool> >> >> >> > Start of alignment for : REFDATABASE> >> >> >> > Recoverable error> >> >> >> > XTRE0540: Ambiguous rule match for /> >> >> >> > Matches both "document-node()" on line 5 of> >> >> >> >> >> >> >>> file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913> >> >> >> > 9.xsl> >> >> >> > and "document-node()" on line 30 of> >> >> >> >> >> >> >>> file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.> >> >> >> > xsl> >> >> >> > ArcSDE harvest finished> >> >> >>> --------------------------------------------------------------------------------------------------------------------------------------> >> > An no metadata is harvested, any clue/help can be rewarded with a beer> >> in> >> > Barcelona ;-).> >> >> >> > Many thanks,> >> >> >> > Walter & Maria Jose> >> >> >> > --> >> > --> >> > ================================================> >> > Walter Simonazzi> >> > European Topic Centre on Land Use and Spatial Information Torre C5-S,> >> 4ª> >> > planta> >> > Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona> >> > 08193 Bellaterra (Barcelona)> >> > Spain> >> > ================================================> >> > E-mail: walter.simonazzi (at) uab.es> >> > Direct phone: +34 93 581 3868> >> > Fax: +34 93 581 3545> >> > Secretary: +34 93 581 3518> >> > http://terrestrial.eionet.europa.eu> >> >> >>> ------------------------------------------------------------------------------> >> > _______________________________________________> >> > GeoNetwork-users mailing list> >> > GeoNetwork-users@lists.sourceforge.net> >> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users> >> > GeoNetwork OpenSource is maintained at> >> > http://sourceforge.net/projects/geonetwork> >> >> >> >> >>> >> --> >> View this message in context:> >> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html> >> Sent from the GeoNetwork users mailing list archive at Nabble.com.> >>> >>> >>> ------------------------------------------------------------------------------> >> _______________________________________________> >> GeoNetwork-users mailing list> >> GeoNetwork-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/geonetwork-users> >> GeoNetwork OpenSource is maintained at> >> http://sourceforge.net/projects/geonetwork> >>> >> ------------------------------------------------------------------------------> > _______________________________________________> > GeoNetwork-users mailing list> > GeoNetwork-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users> > GeoNetwork OpenSource is maintained at> > http://sourceforge.net/projects/geonetwork> >> >> http://n2.nabble.com/file/n3336435/utilities_storm.xml utilities_storm.xml> --> View this message in context:> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3336435.html> Sent from the GeoNetwork users mailing list archive at Nabble.com.>>> ------------------------------------------------------------------------------> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day> trial. Simplify your report design, integration and deployment - and focus> on> what you do best, core application coding. Discover what's new with> Crystal Reports now. http://p.sf.net/sfu/bobj-july> _______________________________________________> GeoNetwork-users mailing list> GeoNetwork-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/geonetwork-users> GeoNetwork OpenSource is maintained at> http://sourceforge.net/projects/geonetwork>\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_GeoNetwork\-users mailing listGeoNetwork-users@anonymised.com://lists.sourceforge.net/lists/listinfo/geonetwork-usersGeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

(attachments)

metadataGN.xml (17.6 KB)

[GeoNetwork-users] ArcSDE harvester problems

-- --

Walter Simonazzi European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª planta Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona 08193 Bellaterra (Barcelona) Spain

-- --

Walter Simonazzi European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª planta Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona 08193 Bellaterra (Barcelona) Spain

--
--

Walter Simonazzi
European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª
planta
Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
08193 Bellaterra (Barcelona)
Spain

--
--

Walter Simonazzi
European Topic Centre on Land Use and Spatial Information Torre C5-S, 4ª
planta
Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
08193 Bellaterra (Barcelona)
Spain