[GeoNetwork-users] GeoentWork performance problem

GeoNetwork GeoNetwork User
1

Hi,
We have a question about GeonetWork Performance. Currently, 440,000
datasets are stored in GN, but the search performance is quite slow. If we
search by nothing, it takes around 15 seconds to get the first 50 datasets.
If we search by a keyword such as “water”, it takes 8 seconds.
Most of them are from NASA ECHO who has made some modification to iso19115
format, so some attributes cannot be recognized by GN. We are suspecting
this is the reason for the low performance.

Best,

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

2

Hi Kai,

The problem relates to extracting the permissions/ownership stuff from the database when processing the records returned from the search (before returning for display). I have a fix I'll commit shortly - see buildExtraMetadataInfo in DataManager slows search in html5ui for larger catalogues · Issue #846 · geonetwork/core-geonetwork · GitHub

Cheers,
Simon
________________________________________
From: Kai Liu [kliu.gis@anonymised.com]
Sent: Friday, 20 March 2015 7:09 AM
To: Geonetwork-Users@anonymised.com
Subject: [GeoNetwork-users] GeoentWork performance problem

Hi,
We have a question about GeonetWork Performance. Currently, 440,000
datasets are stored in GN, but the search performance is quite slow. If we
search by nothing, it takes around 15 seconds to get the first 50 datasets.
If we search by a keyword such as “water”, it takes 8 seconds.
Most of them are from NASA ECHO who has made some modification to iso19115
format, so some attributes cannot be recognized by GN. We are suspecting
this is the reason for the low performance.

Best,

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

3

I think Jesse and François has improved the data bases access during the
search, few monthes ago in develop.
Simon you should check with them that you are not doing agin the same thing
:wink:

Cheers

On Fri, Mar 20, 2015 at 5:29 AM, <Simon.Pigot@anonymised.com> wrote:

Hi Kai,

The problem relates to extracting the permissions/ownership stuff from the
database when processing the records returned from the search (before
returning for display). I have a fix I'll commit shortly - see
buildExtraMetadataInfo in DataManager slows search in html5ui for larger catalogues · Issue #846 · geonetwork/core-geonetwork · GitHub

Cheers,
Simon
________________________________________
From: Kai Liu [kliu.gis@anonymised.com]
Sent: Friday, 20 March 2015 7:09 AM
To: Geonetwork-Users@anonymised.com
Subject: [GeoNetwork-users] GeoentWork performance problem

Hi,
We have a question about GeonetWork Performance. Currently, 440,000
datasets are stored in GN, but the search performance is quite slow. If we
search by nothing, it takes around 15 seconds to get the first 50 datasets.
If we search by a keyword such as “water”, it takes 8 seconds.
Most of them are from NASA ECHO who has made some modification to iso19115
format, so some attributes cannot be recognized by GN. We are suspecting
this is the reason for the low performance.

Best,

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for
all
things parallel software development, from weekly thought leadership blogs
to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
geonetwork-users List Signup and Options
GeoNetwork OpenSource is maintained at
GeoNetwork - Geographic Metadata Catalog download | SourceForge.net

--
Florent Gravin
Camptocamp - Chambéry
0479444492

4

2015-03-20 9:34 GMT+01:00 Florent Gravin <florent.gravin@anonymised.com>:

I think Jesse and François has improved the data bases access during the
search, few monthes ago in develop.
Simon you should check with them that you are not doing agin the same thing
:wink:

We did on develop, not 2.10.x

Francois

Cheers

On Fri, Mar 20, 2015 at 5:29 AM, <Simon.Pigot@anonymised.com> wrote:

> Hi Kai,
>
> The problem relates to extracting the permissions/ownership stuff from
the
> database when processing the records returned from the search (before
> returning for display). I have a fix I'll commit shortly - see
> https://github.com/geonetwork/core-geonetwork/issues/846
>
> Cheers,
> Simon
> ________________________________________
> From: Kai Liu [kliu.gis@anonymised.com]
> Sent: Friday, 20 March 2015 7:09 AM
> To: Geonetwork-Users@anonymised.com
> Subject: [GeoNetwork-users] GeoentWork performance problem
>
> Hi,
> We have a question about GeonetWork Performance. Currently, 440,000
> datasets are stored in GN, but the search performance is quite slow. If
we
> search by nothing, it takes around 15 seconds to get the first 50
datasets.
> If we search by a keyword such as “water”, it takes 8 seconds.
> Most of them are from NASA ECHO who has made some modification to
iso19115
> format, so some attributes cannot be recognized by GN. We are suspecting
> this is the reason for the low performance.
>
> Best,
>
> --
>
> Kai Liu
> Ph.D. student
> Department of Geography and GeoInformation Science
> George Mason University
>
>
>
------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub
for
> all
> things parallel software development, from weekly thought leadership
blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

--
Florent Gravin
Camptocamp - Chambéry
0479444492

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for
all
things parallel software development, from weekly thought leadership blogs
to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

5

Thanks Simon and all.

We are using the v2.10.4 release version. That is probably why. And we are
using PostgreSQL + PostGIS.

I am wondering if there is a way to just replay some classes in the
release version.

Thanks and best regards,

On Fri, Mar 20, 2015 at 4:37 AM, Francois Prunayre <fx.prunayre@anonymised.com>
wrote:

2015-03-20 9:34 GMT+01:00 Florent Gravin <florent.gravin@anonymised.com>:

> I think Jesse and François has improved the data bases access during the
> search, few monthes ago in develop.
> Simon you should check with them that you are not doing agin the same
thing
> :wink:
>

We did on develop, not 2.10.x

Francois

>
> Cheers
>
> On Fri, Mar 20, 2015 at 5:29 AM, <Simon.Pigot@anonymised.com> wrote:
>
> > Hi Kai,
> >
> > The problem relates to extracting the permissions/ownership stuff from
> the
> > database when processing the records returned from the search (before
> > returning for display). I have a fix I'll commit shortly - see
> > https://github.com/geonetwork/core-geonetwork/issues/846
> >
> > Cheers,
> > Simon
> > ________________________________________
> > From: Kai Liu [kliu.gis@anonymised.com]
> > Sent: Friday, 20 March 2015 7:09 AM
> > To: Geonetwork-Users@anonymised.com
> > Subject: [GeoNetwork-users] GeoentWork performance problem
> >
> > Hi,
> > We have a question about GeonetWork Performance. Currently, 440,000
> > datasets are stored in GN, but the search performance is quite slow. If
> we
> > search by nothing, it takes around 15 seconds to get the first 50
> datasets.
> > If we search by a keyword such as “water”, it takes 8 seconds.
> > Most of them are from NASA ECHO who has made some modification to
> iso19115
> > format, so some attributes cannot be recognized by GN. We are
suspecting
> > this is the reason for the low performance.
> >
> > Best,
> >
> > --
> >
> > Kai Liu
> > Ph.D. student
> > Department of Geography and GeoInformation Science
> > George Mason University
> >
> >
> >
>
------------------------------------------------------------------------------
> > Dive into the World of Parallel Programming The Go Parallel Website,
> > sponsored
> > by Intel and developed in partnership with Slashdot Media, is your hub
> for
> > all
> > things parallel software development, from weekly thought leadership
> blogs
> > to
> > news, videos, case studies, tutorials and more. Take a look and join
the
> > conversation now. http://goparallel.sourceforge.net/
> > _______________________________________________
> > GeoNetwork-users mailing list
> > GeoNetwork-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
>
>
>
> --
> Florent Gravin
> Camptocamp - Chambéry
> 0479444492
>
>
------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub
for
> all
> things parallel software development, from weekly thought leadership
blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for
all
things parallel software development, from weekly thought leadership blogs
to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

6

I tried to create the installer the source code following the instructions
on http://geonetwork
-opensource.org/manuals/trunk/eng/developer/development/index.html.

but problems came one by one when I use the command "mvn clean install".

First, it has the error:

[ERROR] Child module
/Users/kailiu/Downloads/core-geonetwork-develop/geoserver/pom.xml of
/Users/kailiu/Downloads/core-geonetwork-develop/pom.xml does not exist

Then I commented out the geoserver module, it comes:

[ERROR] Failed to execute goal on project csw-server: Could not resolve
dependencies for project
org.geonetwork-opensource:csw-server:jar:3.0.0-SNAPSHOT: Could not find
artifact org.geonetwork-opensource:core:jar:tests:3.0.0-SNAPSHOT in
core-maven-repo (
https://raw.githubusercontent.com/geonetwork/core-maven-repo/master) ->
[Help 1]

Do you have any suggestions to create the installer using the develop
version?

Thanks and best regards,

On Fri, Mar 20, 2015 at 11:46 AM, Kai Liu <kliu.gis@anonymised.com> wrote:

Thanks Simon and all.

We are using the v2.10.4 release version. That is probably why. And we are
using PostgreSQL + PostGIS.

I am wondering if there is a way to just replay some classes in the
release version.

Thanks and best regards,

On Fri, Mar 20, 2015 at 4:37 AM, Francois Prunayre <fx.prunayre@anonymised.com>
wrote:

2015-03-20 9:34 GMT+01:00 Florent Gravin <florent.gravin@anonymised.com>:

> I think Jesse and François has improved the data bases access during the
> search, few monthes ago in develop.
> Simon you should check with them that you are not doing agin the same
thing
> :wink:
>

We did on develop, not 2.10.x

Francois

>
> Cheers
>
> On Fri, Mar 20, 2015 at 5:29 AM, <Simon.Pigot@anonymised.com> wrote:
>
> > Hi Kai,
> >
> > The problem relates to extracting the permissions/ownership stuff from
> the
> > database when processing the records returned from the search (before
> > returning for display). I have a fix I'll commit shortly - see
> > https://github.com/geonetwork/core-geonetwork/issues/846
> >
> > Cheers,
> > Simon
> > ________________________________________
> > From: Kai Liu [kliu.gis@anonymised.com]
> > Sent: Friday, 20 March 2015 7:09 AM
> > To: Geonetwork-Users@anonymised.com
> > Subject: [GeoNetwork-users] GeoentWork performance problem
> >
> > Hi,
> > We have a question about GeonetWork Performance. Currently, 440,000
> > datasets are stored in GN, but the search performance is quite slow.
If
> we
> > search by nothing, it takes around 15 seconds to get the first 50
> datasets.
> > If we search by a keyword such as “water”, it takes 8 seconds.
> > Most of them are from NASA ECHO who has made some modification to
> iso19115
> > format, so some attributes cannot be recognized by GN. We are
suspecting
> > this is the reason for the low performance.
> >
> > Best,
> >
> > --
> >
> > Kai Liu
> > Ph.D. student
> > Department of Geography and GeoInformation Science
> > George Mason University
> >
> >
> >
>
------------------------------------------------------------------------------
> > Dive into the World of Parallel Programming The Go Parallel Website,
> > sponsored
> > by Intel and developed in partnership with Slashdot Media, is your hub
> for
> > all
> > things parallel software development, from weekly thought leadership
> blogs
> > to
> > news, videos, case studies, tutorials and more. Take a look and join
the
> > conversation now. http://goparallel.sourceforge.net/
> > _______________________________________________
> > GeoNetwork-users mailing list
> > GeoNetwork-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
>
>
>
> --
> Florent Gravin
> Camptocamp - Chambéry
> 0479444492
>
>
------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub
for
> all
> things parallel software development, from weekly thought leadership
blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub
for all
things parallel software development, from weekly thought leadership
blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

7

Hi,

2015-03-26 3:07 GMT+01:00 Kai Liu <kliu.gis@anonymised.com>:

I tried to create the installer the source code following the
instructions on http://geonetwork
-opensource.org/manuals/trunk/eng/developer/development/index.html.

but problems came one by one when I use the command "mvn clean install".

First, it has the error:

[ERROR] Child module
/Users/kailiu/Downloads/core-geonetwork-develop/geoserver/pom.xml of
/Users/kailiu/Downloads/core-geonetwork-develop/pom.xml does not exist

Then I commented out the geoserver module, it comes:

instead you should probably do
git submodule init
git submodule update

[ERROR] Failed to execute goal on project csw-server: Could not resolve
dependencies for project
org.geonetwork-opensource:csw-server:jar:3.0.0-SNAPSHOT: Could not find
artifact org.geonetwork-opensource:core:jar:tests:3.0.0-SNAPSHOT in
core-maven-repo (
https://raw.githubusercontent.com/geonetwork/core-maven-repo/master) ->
[Help 1]

Do you have any suggestions to create the installer using the develop
version?

It should work well AFAIK.

Francois

Thanks and best regards,

On Fri, Mar 20, 2015 at 11:46 AM, Kai Liu <kliu.gis@anonymised.com> wrote:

Thanks Simon and all.

We are using the v2.10.4 release version. That is probably why. And we
are using PostgreSQL + PostGIS.

I am wondering if there is a way to just replay some classes in the
release version.

Thanks and best regards,

On Fri, Mar 20, 2015 at 4:37 AM, Francois Prunayre <fx.prunayre@anonymised.com
> wrote:

2015-03-20 9:34 GMT+01:00 Florent Gravin <florent.gravin@anonymised.com
>:

> I think Jesse and François has improved the data bases access during
the
> search, few monthes ago in develop.
> Simon you should check with them that you are not doing agin the same
thing
> :wink:
>

We did on develop, not 2.10.x

Francois

>
> Cheers
>
> On Fri, Mar 20, 2015 at 5:29 AM, <Simon.Pigot@anonymised.com> wrote:
>
> > Hi Kai,
> >
> > The problem relates to extracting the permissions/ownership stuff
from
> the
> > database when processing the records returned from the search (before
> > returning for display). I have a fix I'll commit shortly - see
> > https://github.com/geonetwork/core-geonetwork/issues/846
> >
> > Cheers,
> > Simon
> > ________________________________________
> > From: Kai Liu [kliu.gis@anonymised.com]
> > Sent: Friday, 20 March 2015 7:09 AM
> > To: Geonetwork-Users@anonymised.com
> > Subject: [GeoNetwork-users] GeoentWork performance problem
> >
> > Hi,
> > We have a question about GeonetWork Performance. Currently, 440,000
> > datasets are stored in GN, but the search performance is quite slow.
If
> we
> > search by nothing, it takes around 15 seconds to get the first 50
> datasets.
> > If we search by a keyword such as “water”, it takes 8 seconds.
> > Most of them are from NASA ECHO who has made some modification to
> iso19115
> > format, so some attributes cannot be recognized by GN. We are
suspecting
> > this is the reason for the low performance.
> >
> > Best,
> >
> > --
> >
> > Kai Liu
> > Ph.D. student
> > Department of Geography and GeoInformation Science
> > George Mason University
> >
> >
> >
>
------------------------------------------------------------------------------
> > Dive into the World of Parallel Programming The Go Parallel Website,
> > sponsored
> > by Intel and developed in partnership with Slashdot Media, is your
hub
> for
> > all
> > things parallel software development, from weekly thought leadership
> blogs
> > to
> > news, videos, case studies, tutorials and more. Take a look and join
the
> > conversation now. http://goparallel.sourceforge.net/
> > _______________________________________________
> > GeoNetwork-users mailing list
> > GeoNetwork-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
>
>
>
> --
> Florent Gravin
> Camptocamp - Chambéry
> 0479444492
>
>
------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub
for
> all
> things parallel software development, from weekly thought leadership
blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join
the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub
for all
things parallel software development, from weekly thought leadership
blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University

--

Kai Liu
Ph.D. student
Department of Geography and GeoInformation Science
George Mason University