[GeoNetwork-devel] Adding DB2 support to geonetwork

GeoNetwork GeoNetwork Developer
,
1

Hi, I am a geotools/geoserver developer and want to contribute some code to geonetwork. My biggest customer is IBM minded and uses DB2. Since we have to participate in the EU INSPIRE project we decided to use a geonetwork/geoserver combination.

One of my geotools modules is the DB2 datastore. Within the last 12 months I downloaded some versions of geonetwork and always had to adapt geonetwork to work with DB2. I am little bit tired on this procedure :slight_smile:

1)
The first questions is if the community has some interest to integrate DB2 support the same way as it has been done for Oracle, mysql, postgres. I would maintain the DB2 stuff in the future.

I did an svn checkout from trunk and developed DB2Panel.java for the connection parameters. So far so good. I am unsure about the following.

2)
There are some scripts data-db-xxx.sql and create-db-xxx.sql. I created data-db-db2.sql and create-db-db2.sql manually. Is this correct or are you using some tool to generate these batch files ?

3)
There is a file
gast/setup/database.druid
What is the usage of this file ?

Christian

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

2

Hi Christian,

On Thu, Nov 25, 2010 at 10:59 AM, <christian.mueller@anonymised.com> wrote:

Hi, I am a geotools/geoserver developer and want to contribute some
code to geonetwork. My biggest customer is IBM minded and uses DB2.
Since we have to participate in the EU INSPIRE project we decided to
use a geonetwork/geoserver combination.

One of my geotools modules is the DB2 datastore. Within the last 12
months I downloaded some versions of geonetwork and always had to
adapt geonetwork to work with DB2. I am little bit tired on this
procedure :slight_smile:

The first questions is if the community has some interest to integrate
DB2 support the same way as it has been done for Oracle, mysql,
postgres. I would maintain the DB2 stuff in the future.

Yes for sure contributions are very welcomed.
Moreover, that could be nice ton integrate support for DB2 in Geonetwork. Thatā€™s a nice contribution.

I did an svn checkout from trunk and developed DB2Panel.java for the
connection parameters. So far so good. I am unsure about the following.

There are some scripts data-db-xxx.sql and create-db-xxx.sql. I
created data-db-db2.sql and create-db-db2.sql manually. Is this
correct or are you using some tool to generate these batch files ?

In my opinion this is correct, or anyone could use whatever he likes to generate them but most of the time they are modified manually.
But I let Jose confirm you that point because I did them first.

There is a file
gast/setup/database.druid
What is the usage of this file ?

It was used to open a druid project with Geonetwork configuration when we used to generate default mckoi database using druid script files (ddf files for database druid format). Here you have more information about Druid [1]

Feel free to open a ticket for enhancement in the trac [2] and then attach your contribution to it.

Thanks in advance,

Best Regards,

Mathieu

[1] http://druid.sourceforge.net/
[2] http://trac.osgeo.org/geonetwork/

3

Last week I did a svn checkout from trunk and tried a

mvn clean install

The build fails because the following files are not UTF-8 encoded

web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/da-DK.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/ca.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/sv-SE.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/nb.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en-CA.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en.js

Since I am working on a linux box I manually did a

recode oldEncoding..utf-8 filename

for all *.js files not encoded in utf-8.

After the recode, the build works.

Some questions:

Does the build work for other developers ?.

Should I make a patch file and open an issue ?

Should I do the same on the 2.6.1 branch ?

Cheers
Christian

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

4

The table called Settings is created as

CREATE TABLE Settings
   (
     id int not null,
     parentId int,
     name varchar(32) not null,
     value CLOB(1G),
     primary key(id)
   );

The type of the "value" attribute is db specific and is a type for a long text field.

only in the script for sql server I found correct insert statements.

INSERT INTO Settings VALUES (701,700,'mdmode','1');
INSERT INTO Settings VALUES (702,700,'tokentimeout','3600');
INSERT INTO Settings VALUES (703,700,'cachesize','60');

All others use

INSERT INTO Settings VALUES (701,700,'mdmode',1);
INSERT INTO Settings VALUES (702,700,'tokentimeout',3600);
INSERT INTO Settings VALUES (703,700,'cachesize',60);

passing integer values for the last column.

What is correct ?. DB2 complains about int values for CLOB attributes.

Cheers
Christian

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

5

Hi Christian,

On Tue, Nov 30, 2010 at 3:22 PM, <christian.mueller@anonymised.com> wrote:

The table called Settings is created as

CREATE TABLE Settings
(
id int not null,
parentId int,
name varchar(32) not null,
value CLOB(1G),
primary key(id)
);

The type of the ā€œvalueā€ attribute is db specific and is a type for a long text field.

The type CLOB is never used for now in our scripts, we try to use native type (such as text for Postgresql or longtext for MySQL).
Maybe we should use CLOB for Oracle also ?

only in the script for sql server I found correct insert statements.

INSERT INTO Settings VALUES (701,700,ā€˜mdmodeā€™,ā€˜1ā€™);
INSERT INTO Settings VALUES (702,700,ā€˜tokentimeoutā€™,ā€˜3600ā€™);
INSERT INTO Settings VALUES (703,700,ā€˜cachesizeā€™,ā€˜60ā€™);

All others use

INSERT INTO Settings VALUES (701,700,ā€˜mdmodeā€™,1);
INSERT INTO Settings VALUES (702,700,ā€˜tokentimeoutā€™,3600);
INSERT INTO Settings VALUES (703,700,ā€˜cachesizeā€™,60);

passing integer values for the last column.

What is correct ?. DB2 complains about int values for CLOB attributes.

Maybe the type CLOB needs quotes but it seems that other natives types such as text or longtext do not need.
Or at least they are not complaining about thatā€¦

Does anybody else confident with that?

Cheers,

Mathieu

6

We can turn it the other way around. There are integer values like "port" and "subversion" in the script. These integers are quoted for all db engines. Only the three settings I pointed at are not quoted. Making the scripts consistent will be the best way, having these 3 values quoted in all scripts and anything should work well.

Quoting Mathieu Coudert <mathieu.coudert@anonymised.com>:

Hi Christian,

On Tue, Nov 30, 2010 at 3:22 PM, <christian.mueller@anonymised.com> wrote:

The table called Settings is created as

CREATE TABLE Settings
(
   id int not null,
   parentId int,
   name varchar(32) not null,
   value CLOB(1G),
   primary key(id)
);

The type of the "value" attribute is db specific and is a type for a long
text field.

The type CLOB is never used for now in our scripts, we try to use native
type (such as text for Postgresql or longtext for MySQL).
Maybe we should use CLOB for Oracle also ?

only in the script for sql server I found correct insert statements.

INSERT INTO Settings VALUES (701,700,'mdmode','1');
INSERT INTO Settings VALUES (702,700,'tokentimeout','3600');
INSERT INTO Settings VALUES (703,700,'cachesize','60');

All others use

INSERT INTO Settings VALUES (701,700,'mdmode',1);
INSERT INTO Settings VALUES (702,700,'tokentimeout',3600);
INSERT INTO Settings VALUES (703,700,'cachesize',60);

passing integer values for the last column.

What is correct ?. DB2 complains about int values for CLOB attributes.

Maybe the type CLOB needs quotes but it seems that other natives types such
as text or longtext do not need.
Or at least they are not complaining about that...

Does anybody else confident with that?

Cheers,

Mathieu

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

7

Hi Christian,

The build fails because the following files are not UTF-8 encoded

web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/da-DK.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/ca.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/sv-SE.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/nb.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en-CA.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en.js

Since I am working on a linux box I manually did a

recode oldEncodingā€¦utf-8 filename

for all *.js files not encoded in utf-8.

After the recode, the build works.

Some questions:

Does the build work for other developers ?.

The build works for me on linux (ubuntu) and windows.

However, youā€™re right the encoding of these files is not in UTF-8 for all of them.

#web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang$ file *
ca.js: ISO-8859 C program text
cs-CZ.js: UTF-8 Unicode C program text
da-DK.js: ISO-8859 C program text
de.js: UTF-8 Unicode C program text
en-CA.js: ASCII C program text
en.js: ASCII C program text
es.js: UTF-8 Unicode C program text
fr.js: UTF-8 Unicode C program text
it.js: UTF-8 Unicode C program text
nb.js: ISO-8859 C program text
nl.js: UTF-8 Unicode C program text
pt-BR.js: UTF-8 Unicode C program text
sv-SE.js: ISO-8859 C program text
zh-CN.js: UTF-8 Unicode C program text
zh-TW.js: UTF-8 Unicode C program text

What do you think, should we move them to the UTF-8 encoding?

Is anybody else encounters the same trouble building the trunk?

Cheers,

Mathieu

8

Christian,

On Tue, Nov 30, 2010 at 4:49 PM, <christian.mueller@anonymised.com> wrote:

We can turn it the other way around. There are integer values like ā€œportā€ and ā€œsubversionā€ in the script. These integers are quoted for all db engines. Only the three settings I pointed at are not quoted. Making the scripts consistent will be the best way, having these 3 values quoted in all scripts and anything should work well.

Youā€™re right! We should make the scripts consistent.
Do you have any patch for that or is it only these three values?

Feel free to open a new ticket for that and to attach the patch if any.

Cheers,

Mathieu

9

Hi Mathieu

The same problems for the 2.6.x branch.

I will work with geonetwork on different plattforms
Ubunutu 64 Bit
Open SUSE 32 Bit
SLES 10 on IBM Power PC Servers
SLES 10 on IBM mainframes

For the last two, there is no SUN java, I have to use IBM java.

If I have a vote I would vote for encode all problematic files into utf-8, on 2.6.x and on trunk.

Just my 2 cents

Quoting Mathieu Coudert <mathieu.coudert@anonymised.com>:

Hi Christian,

The build fails because the following files are not UTF-8 encoded

web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/da-DK.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/ca.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/sv-SE.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/nb.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en-CA.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en.js

Since I am working on a linux box I manually did a

recode oldEncoding..utf-8 filename

for all *.js files not encoded in utf-8.

After the recode, the build works.

Some questions:

Does the build work for other developers ?.

The build works for me on linux (ubuntu) and windows.

However, you're right the encoding of these files is not in UTF-8 for all of
them.

#web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang$ file *
ca.js: ISO-8859 C program text
cs-CZ.js: UTF-8 Unicode C program text
da-DK.js: ISO-8859 C program text
de.js: UTF-8 Unicode C program text
en-CA.js: ASCII C program text
en.js: ASCII C program text
es.js: UTF-8 Unicode C program text
fr.js: UTF-8 Unicode C program text
it.js: UTF-8 Unicode C program text
nb.js: ISO-8859 C program text
nl.js: UTF-8 Unicode C program text
pt-BR.js: UTF-8 Unicode C program text
sv-SE.js: ISO-8859 C program text
zh-CN.js: UTF-8 Unicode C program text
zh-TW.js: UTF-8 Unicode C program text

What do you think, should we move them to the UTF-8 encoding?

Is anybody else encounters the same trouble building the trunk?

Cheers,

Mathieu

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

10

I checked on 2.6.x, same situation.

Should I open 2 tickets, for trunk and 2.6.x

Cheers

Quoting Mathieu Coudert <mathieu.coudert@anonymised.com>:

Christian,

On Tue, Nov 30, 2010 at 4:49 PM, <christian.mueller@anonymised.com> wrote:

We can turn it the other way around. There are integer values like "port"
and "subversion" in the script. These integers are quoted for all db
engines. Only the three settings I pointed at are not quoted. Making the
scripts consistent will be the best way, having these 3 values quoted in all
scripts and anything should work well.

You're right! We should make the scripts consistent.

Do you have any patch for that or is it only these three values?

Feel free to open a new ticket for that and to attach the patch if any.

Cheers,

Mathieu

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

11

On Wed, Dec 1, 2010 at 10:03 AM, <christian.mueller@anonymised.com> wrote:

I checked on 2.6.x, same situation.

Should I open 2 tickets, for trunk and 2.6.x

One is enough, with version 2.6.1 and milestone 2.6.2 (~due date).

Cheers,
Mathieu

12

hi,

in my opinion, to avoid any problems it is best to have all GeoNetwork files encoded in UTF-8, without BOM.

My 2Ā¢,

Heikki Doeleman

On Wed, Dec 1, 2010 at 10:00 AM, <christian.mueller@anonymised.com> wrote:

Hi Mathieu

The same problems for the 2.6.x branch.

I will work with geonetwork on different plattforms
Ubunutu 64 Bit
Open SUSE 32 Bit
SLES 10 on IBM Power PC Servers
SLES 10 on IBM mainframes

For the last two, there is no SUN java, I have to use IBM java.

If I have a vote I would vote for encode all problematic files into
utf-8, on 2.6.x and on trunk.

Just my 2 cents

Quoting Mathieu Coudert <mathieu.coudert@anonymised.com>:

Hi Christian,

The build fails because the following files are not UTF-8 encoded

web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/da-DK.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/ca.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/sv-SE.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/nb.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en-CA.js
web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en.js

Since I am working on a linux box I manually did a

recode oldEncodingā€¦utf-8 filename

for all *.js files not encoded in utf-8.

After the recode, the build works.

Some questions:

Does the build work for other developers ?.

The build works for me on linux (ubuntu) and windows.

However, youā€™re right the encoding of these files is not in UTF-8 for all of
them.

#web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang$ file *
ca.js: ISO-8859 C program text
cs-CZ.js: UTF-8 Unicode C program text
da-DK.js: ISO-8859 C program text
de.js: UTF-8 Unicode C program text
en-CA.js: ASCII C program text
en.js: ASCII C program text
es.js: UTF-8 Unicode C program text
fr.js: UTF-8 Unicode C program text
it.js: UTF-8 Unicode C program text
nb.js: ISO-8859 C program text
nl.js: UTF-8 Unicode C program text
pt-BR.js: UTF-8 Unicode C program text
sv-SE.js: ISO-8859 C program text
zh-CN.js: UTF-8 Unicode C program text
zh-TW.js: UTF-8 Unicode C program text

What do you think, should we move them to the UTF-8 encoding?

Is anybody else encounters the same trouble building the trunk?

Cheers,

Mathieu

This message was sent using IMP, the Internet Messaging Program.

Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

13

Another 2 cents, :slight_smile:

At least the files which are subject to internationalization.

Quoting heikki <tropicano@anonymised.com>:

hi,

in my opinion, to avoid any problems it is best to have all GeoNetwork files
encoded in UTF-8, without BOM.

My 2Ā¢,

Heikki Doeleman

On Wed, Dec 1, 2010 at 10:00 AM, <christian.mueller@anonymised.com> wrote:

Hi Mathieu

The same problems for the 2.6.x branch.

I will work with geonetwork on different plattforms
Ubunutu 64 Bit
Open SUSE 32 Bit
SLES 10 on IBM Power PC Servers
SLES 10 on IBM mainframes

For the last two, there is no SUN java, I have to use IBM java.

If I have a vote I would vote for encode all problematic files into
utf-8, on 2.6.x and on trunk.

Just my 2 cents

Quoting Mathieu Coudert <mathieu.coudert@anonymised.com>:

> Hi Christian,
>
>> The build fails because the following files are not UTF-8 encoded
>>
>> web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/da-DK.js
>> web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/ca.js
>> web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/sv-SE.js
>> web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/nb.js
>> web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en-CA.js
>> web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang/en.js
>>
>> Since I am working on a linux box I manually did a
>>
>> recode oldEncoding..utf-8 filename
>>
>> for all *.js files not encoded in utf-8.
>>
>> After the recode, the build works.
>>
>> Some questions:
>>
>> Does the build work for other developers ?.
>>
>
> The build works for me on linux (ubuntu) and windows.
>
> However, you're right the encoding of these files is not in UTF-8 for all
of
> them.
>
> #web/src/main/webapp/scripts/openlayers/lib/OpenLayers/Lang$ file *
> ca.js: ISO-8859 C program text
> cs-CZ.js: UTF-8 Unicode C program text
> da-DK.js: ISO-8859 C program text
> de.js: UTF-8 Unicode C program text
> en-CA.js: ASCII C program text
> en.js: ASCII C program text
> es.js: UTF-8 Unicode C program text
> fr.js: UTF-8 Unicode C program text
> it.js: UTF-8 Unicode C program text
> nb.js: ISO-8859 C program text
> nl.js: UTF-8 Unicode C program text
> pt-BR.js: UTF-8 Unicode C program text
> sv-SE.js: ISO-8859 C program text
> zh-CN.js: UTF-8 Unicode C program text
> zh-TW.js: UTF-8 Unicode C program text
>
> What do you think, should we move them to the UTF-8 encoding?
>
> Is anybody else encounters the same trouble building the trunk?
>
> Cheers,
>
> Mathieu
>

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

14

Hi all, last week I submitted 2 patches

http://trac.osgeo.org/geonetwork/ticket/394
http://trac.osgeo.org/geonetwork/ticket/396

I would be happy to have this patches applied to 2.6.x and trunk to show my customer that he has not paid for nothing :slight_smile:

There is an additional issue with some files not utf-8 encoded breaking the build on some environments.
Should I add an additional patch ?

A cautious question:

Looking into the future I assume I will have to take care of the DB2 stuff. Additionally my build environments are IBM SDK on IBM P6 machines and Linux on zOS (mainframe).

To illustrate the problem, please take a look at

http://gis.linux4all.at/hudson64/
http://gis.linux4all.at/hudson32/

The URLs point to two virtual machines acting as 32 bit and 64 bit hudson build servers for geotools and geoserver using different Java SDKs. Until know I never succeeded in getting all circles blue :frowning:

Some years ago, the got commit access to the following repos

https://svn.codehaus.org/geoserver/
http://svn.osgeo.org/geotools/

Do you see a possibility to give me commit access to the geonetworks repo ?. Life would be easier.

Cheers
Christian

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.