[GeoNetwork-users] General question about recording metadata on feature counts

Hi All,

I'm working on a GeoNetwork implementation where the client is keen to
record additional metadata on the count of different features in the
dataset. Not a count of different geometries, but totals of different
features based on some attribute. The metadata will be stored in Gemini 2.3
format, so ISO19139-based, but I've been looking in both the Feature
Catalog and Data Quality standards to see if I can find an existing
metadata element for storing this information and I'm not getting anywhere.
Does anyone have an example of this being done in practice? If there's even
an informal standard for recording this sort of information then I'd be
keen to use that rather than making something up.

Thanks

Jo

--
*Jo Cook*
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent
responses at that time, please visit support.astuntechnology.com or phone
our office on 01372 744009

--
--
*Sign up to our mailing list
<https://astuntechnology.com/company/#email-updates&gt; for updates on news,
products, conferences, events and training*
*
*

Astun Technology Ltd, 120
Manor Green Road, Epsom, Surrey, KT19 8LN, UK
t:+44 1372 744 009 w:
astuntechnology.com <http://astuntechnology.com/&gt; twitter:@astuntech
<https://twitter.com/astuntech&gt;

iShare - enterprise geographic
intelligence platform <https://astuntechnology.com/ishare/&gt;
GeoServer,
PostGIS and QGIS training <https://astuntechnology.com/training-courses/&gt;
Helpdesk and customer portal
<https://astuntech.atlassian.net/wiki/spaces/ISHAREHELP/pages/364970043/Astun+Technology+Support+Portal&gt;

Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.

Hi guys,
we are currently trying to solve the problems with uploading and downloading large data.
Upload size could be set upwards: either as suggested in earlier versions in the source code under services in the config-spring-geonetwork.xml file
or with new parameter api.params.maxUploadSize in config.properties in the newer versions.
Now we have the problem to download larger files via geonetwork. When clicking on a link to a 7.3 GB file under Download and links, an error message comes up immediately:
<code>runtime_exception</code>.
<description>
Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Required array size too large
</description>
<message>NestedServletException</message>.
The memory parameters are currently:
CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g -Dgeonetwork.dir=/var/lib/gn_data"
Tomcat8 can only be reached locally and uses NGinx Proxy for external communication via HTTPS protocol. The server.xml looks like this:
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
address="127.0.0.1"
redirectPort="443" />
the nginx configuration for the domain contains the section:
location / {
proxy_pass http://127.0.0.1:8080;
...
}
Do we have a configuration error? Is all data routed through Tomcat8 and cannot be sent through an external web server?
We want a portal that provides the metadata as well as the actual data and takes care of user rights, and we find the geonetwork software good in this regard.

best regards

Viktor

Viktor,

Geonetwork uses byte arrays to store the data before streaming it to the user.

https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261

So I believe the limit should be around 2GB

________________________________
From: Viktor Stroh <stroh@anonymised.com>
Sent: February 24, 2021 11:26 AM
To: geonetwork-users@lists.sourceforge.net <geonetwork-users@anonymised.comrge.net>
Subject: [GeoNetwork-users] Download large files

Hi guys,
we are currently trying to solve the problems with uploading and
downloading large data.
Upload size could be set upwards: either as suggested in earlier
versions in the source code under services in the
config-spring-geonetwork.xml file
or with new parameter api.params.maxUploadSize in config.properties in
the newer versions.
Now we have the problem to download larger files via geonetwork. When
clicking on a link to a 7.3 GB file under Download and links, an error
message comes up immediately:
<code>runtime_exception</code>.
<description>
Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
Required array size too large
</description>
<message>NestedServletException</message>.
The memory parameters are currently:
CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
-Dgeonetwork.dir=/var/lib/gn_data"
Tomcat8 can only be reached locally and uses NGinx Proxy for external
communication via HTTPS protocol. The server.xml looks like this:
<Connector port="8080" protocol="HTTP/1.1"
                connectionTimeout="20000"
                address="127.0.0.1"
                redirectPort="443" />
the nginx configuration for the domain contains the section:
location / {
             proxy_pass http://127.0.0.1:8080;
             ...
             }
Do we have a configuration error? Is all data routed through Tomcat8 and
cannot be sent through an external web server?
We want a portal that provides the metadata as well as the actual data
and takes care of user rights, and we find the geonetwork software good
in this regard.

best regards

Viktor

_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Hi,
so that means, that it is not possible to provide download for files bigger than 2 GB? Or is this just for the bulitin datastore?
Regards
Andreas

Am 25.02.2021 um 01:36 schrieb Ian Allen:

Viktor,

Geonetwork uses byte arrays to store the data before streaming it to the user.

https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261

So I believe the limit should be around 2GB

________________________________
From: Viktor Stroh <stroh@anonymised.com>
Sent: February 24, 2021 11:26 AM
To: geonetwork-users@lists.sourceforge.net <geonetwork-users@anonymised.comforge.net>
Subject: [GeoNetwork-users] Download large files

Hi guys,
we are currently trying to solve the problems with uploading and
downloading large data.
Upload size could be set upwards: either as suggested in earlier
versions in the source code under services in the
config-spring-geonetwork.xml file
or with new parameter api.params.maxUploadSize in config.properties in
the newer versions.
Now we have the problem to download larger files via geonetwork. When
clicking on a link to a 7.3 GB file under Download and links, an error
message comes up immediately:
<code>runtime_exception</code>.
<description>
Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
Required array size too large
</description>
<message>NestedServletException</message>.
The memory parameters are currently:
CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
-Dgeonetwork.dir=/var/lib/gn_data"
Tomcat8 can only be reached locally and uses NGinx Proxy for external
communication via HTTPS protocol. The server.xml looks like this:
<Connector port="8080" protocol="HTTP/1.1"
                 connectionTimeout="20000"
                 address="127.0.0.1"
                 redirectPort="443" />
the nginx configuration for the domain contains the section:
location / {
              proxy_pass http://127.0.0.1:8080;
              ...
              }
Do we have a configuration error? Is all data routed through Tomcat8 and
cannot be sent through an external web server?
We want a portal that provides the metadata as well as the actual data
and takes care of user rights, and we find the geonetwork software good
in this regard.

best regards

Viktor

_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
--------------------------------------------------------------------------------------------
            
Andreas Böhme
Techniker/Technician
Landschaftsökologie/Computational Landscape Ecology
        
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 1058
andreas.boehme@anonymised.com, www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: N.N.
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Georg Teutsch
Administrative Geschäftsführerin/Administrative Managing Director:
Dr. Sabine König

Vermeiden Sie unnötige Ausdrucke./Think before printing.

Informationen aus der Wissenschaft (z.B. Veranstaltungshinweise, Projektangaben, Publikationen,
Bücher)

--------------------------------------------------------------------------------------------

Hi, also if you have a lot of files to distribute for download it may be
easier to rely on a data store where you can easily push large datasets to
publish the files and which may provide different access protocols.

For example, EEA provides datasets in various format using different
protocols (HTTP, Webdav, CIFS, FTPS depending on you network access) and
owncloud allow to download data easily eg.
https://sdi.eea.europa.eu/catalogue/srv/eng/catalog.search#/metadata/4b3a3319-4db3-4a33-b18d-2ba55b3fe2ce

Using the GeoNetwork datastore requires to ZIP format based on multiple
formats like shapefile and requires uploading all files to the internal
store.

Francois

Le jeu. 25 févr. 2021 à 11:49, Andreas Böhme <andreas.boehme@anonymised.com> a
écrit :

Hi,
so that means, that it is not possible to provide download for files
bigger than 2 GB? Or is this just for the bulitin datastore?
Regards
Andreas

Am 25.02.2021 um 01:36 schrieb Ian Allen:
> Viktor,
>
> Geonetwork uses byte arrays to store the data before streaming it to the
user.
>
>
https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261
>
> So I believe the limit should be around 2GB
>
>
> ________________________________
> From: Viktor Stroh <stroh@anonymised.com>
> Sent: February 24, 2021 11:26 AM
> To: geonetwork-users@lists.sourceforge.net <
geonetwork-users@lists.sourceforge.net>
> Subject: [GeoNetwork-users] Download large files
>
> Hi guys,
> we are currently trying to solve the problems with uploading and
> downloading large data.
> Upload size could be set upwards: either as suggested in earlier
> versions in the source code under services in the
> config-spring-geonetwork.xml file
> or with new parameter api.params.maxUploadSize in config.properties in
> the newer versions.
> Now we have the problem to download larger files via geonetwork. When
> clicking on a link to a 7.3 GB file under Download and links, an error
> message comes up immediately:
> <code>runtime_exception</code>.
> <description>
> Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
> Required array size too large
> </description>
> <message>NestedServletException</message>.
> The memory parameters are currently:
> CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
> -Dgeonetwork.dir=/var/lib/gn_data"
> Tomcat8 can only be reached locally and uses NGinx Proxy for external
> communication via HTTPS protocol. The server.xml looks like this:
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> address="127.0.0.1"
> redirectPort="443" />
> the nginx configuration for the domain contains the section:
> location / {
> proxy_pass http://127.0.0.1:8080;
> ...
> }
> Do we have a configuration error? Is all data routed through Tomcat8 and
> cannot be sent through an external web server?
> We want a portal that provides the metadata as well as the actual data
> and takes care of user rights, and we find the geonetwork software good
> in this regard.
>
> best regards
>
> Viktor
>
>
>
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork
>
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--

--------------------------------------------------------------------------------------------

Andreas Böhme
Techniker/Technician
Landschaftsökologie/Computational Landscape Ecology

Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 1058
andreas.boehme@anonymised.com, www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: N.N.
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Georg Teutsch
Administrative Geschäftsführerin/Administrative Managing Director:
Dr. Sabine König

Vermeiden Sie unnötige Ausdrucke./Think before printing.

Informationen aus der Wissenschaft (z.B. Veranstaltungshinweise,
Projektangaben, Publikationen,
Bücher)

--------------------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

Hi Ian,
thank you. Good to know, that explains a lot.

Viktor,

Geonetwork uses byte arrays to store the data before streaming it to the user.

https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261

So I believe the limit should be around 2GB

I've created this issue [1] about this topic. It seems quite easy to fix
it and stream the result instead of copying the file content in memory.

[1] https://github.com/geonetwork/core-geonetwork/issues/5449

On Thu, Feb 25, 2021 at 3:36 PM Viktor Stroh <stroh@anonymised.com> wrote:

Hi Ian,
thank you. Good to know, that explains a lot.

> Viktor,
>
> Geonetwork uses byte arrays to store the data before streaming it to
> the user.
>
>
https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261
>
> So I believe the limit should be around 2GB

_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--

*Vriendelijke groeten / Kind regards,Juan Luis Rodríguez.
<http://www.geocat.net/&gt;Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664>Please consider the environment before printing
this email.*