[Geoserver-devel] Question on maven artifacts mirroring

GeoServer GeoServer Developer
1

Hi guys,
sorry for cross-posting.

I’m not sure if I have already asked that in the past (probably, but it was a couple of years ago but I didn’t find anything in my email history).

How the Osgeo artifacts mirroring works?
Long story short:
I had to manually add an artifact to geosolutions repository several days ago.
http://maven.geo-solutions.it/me/steinborn/libdeflate-java-core/0.1.0-beta/

A GT PR is failing due to a

java.lang.NoClassDefFoundError initialization error.

I suspect that it’s due to the artifact not being mirrored on Osgeo yet so the relevant class not being found.I cannot see it indeed in the index of artifacts.
https://repo.osgeo.org/#browse/browse:release

Is anyone familiar with the mirroring logic?
Is there any manual intervention that we can do to copy that artifact into Osgeo maven repository?

Please, let me know.
Regards,
Daniele

–

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

2

I set up the osgeo nexus server, and any project PSC should have admin ability (or talk to me to get such access).

The care and feeding instructions are here https://wiki.osgeo.org/wiki/SAC:Repo

This geo-solutions-cache is configured to proxy https://repo.osgeo.org/repository/geo-solutions-cache/
(It is available for direct online access so you can try the cache directly if you like).

Reviewing:

  • How did you “manually add an artifact to geosolutions repo”? I ask because the cache policy is set to strict “Validate that all paths are maven artifact or metadata paths”.

  • The cache is setup as a mirror of http://maven.geo-solutions.it/ (no https yet?)

  • Auto blocking enabled "

  • if remote peer is detected as unreachable/unresponsive

  • Cache is for 1 day, not found cache is 1 hour

  • No cleanup policy is setup (as that manages to clean up too many old snapshots we liked), instead we setup some kind of

  • Routing rules, geosolutions-artifacts, used to check it/geosolutions/jaiext, it/geosolutions/imageio-ext, org/lib/jpegturbo, com/oracle, opendap

Let me add “me/steinborn” to the routing rules…

···

–
Jody Garnett

3

To answer your question, you can deploy third party dependences to the appropriate project repo at any time.

See developers guide https://docs.geotools.org/latest/developer/procedures/add.html

···

–
Jody Garnett

4

Thanks for the feedbacks, Jody.

I set up the osgeo nexus server, and any project PSC should have admin ability (or talk to me to get such access).

The care and feeding instructions are here https://wiki.osgeo.org/wiki/SAC:Repo

This geo-solutions-cache is configured to proxy https://repo.osgeo.org/repository/geo-solutions-cache/
(It is available for direct online access so you can try the cache directly if you like).

Reviewing:

  • How did you “manually add an artifact to geosolutions repo”? I ask because the cache policy is set to strict “Validate that all paths are maven artifact or metadata paths”.

I have used a command like this:

mvn deploy:deploy-file -Durl=ftp://demo.geo-solutions.it -DrepositoryId=geosolutions -Dfile=myartifact.jar -DgroupId=me.steinborn -DartifactId=libdeflate-java-core -Dversion=0.1.0-beta -Dpackaging=jar -DgeneratePom=true

The repository configured in imageio-ext pom.xml is actually using https so I think that it can get updated.

  • Auto blocking enabled "
  • if remote peer is detected as unreachable/unresponsive
  • Cache is for 1 day, not found cache is 1 hour
  • No cleanup policy is setup (as that manages to clean up too many old snapshots we liked), instead we setup some kind of
  • Routing rules, geosolutions-artifacts, used to check it/geosolutions/jaiext, it/geosolutions/imageio-ext, org/lib/jpegturbo, com/oracle, opendap

Let me add “me/steinborn” to the routing rules…

Will that automatically trigger a mirroring for the missing artifact or will it work only from now on, instead? (I suppose the latest)
If not, do you have any chance to put a copy of that artifact in the mirror since my working day is almost done?

(no problem if you can’t. I will retrieve my OSGeo credentials tomorrow and retry on my own).

Regards,
Daniele

···

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

5

Comments inline:

Reviewing:

  • How did you “manually add an artifact to geosolutions repo”? I ask because the cache policy is set to strict “Validate that all paths are maven artifact or metadata paths”.

I have used a command like this:

mvn deploy:deploy-file -Durl=ftp://demo.geo-solutions.it -DrepositoryId=geosolutions -Dfile=myartifact.jar -DgroupId=me.steinborn -DartifactId=libdeflate-java-core -Dversion=0.1.0-beta -Dpackaging=jar -DgeneratePom=true

Okay, so ftp site :slight_smile:

The procedure to deploy to geotools releases or geoserver releases (for example) is similar.

Let me add “me/steinborn” to the routing rules…

Will that automatically trigger a mirroring for the missing artifact or will it work only from now on, instead? (I suppose the latest)
If not, do you have any chance to put a copy of that artifact in the mirror since my working day is almost done?

(no problem if you can’t. I will retrieve my OSGeo credentials tomorrow and retry on my own).

Yes that is my understanding, please try!

Can you give me your osgeo user id and I can make sure you have access to repo.osgeo.org.

Jody

6

Hi Jody.
My user is danieleromagnoli.
I have tried with the manual deploy using similar maven deploy command with dav but that didn’t work.
Please, let me know if you can allow my osgeo user to manually upload to Nexus.

Regards,
Daniele

···

Regards,

Daniele Romagnoli

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Daniele Romagnoli
Senior Software Engineer

GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272

https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

7

I would hope that it can now fetch and cache the jar (should be no need for manual upload now that /me/steinborn/* is in the routing rules to be cached?).

As for deploy, you provided credentials in your ~/.m2/settings.xml file yes? What does “that didn’t work” look like…

LDAP user danieleromagnoli has been granted GeoServer-Admin and GeoTools-Admin permissions; but once again manual upload should not be required.

···

–
Jody Garnett

Regards,

Daniele Romagnoli

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Daniele Romagnoli
Senior Software Engineer

GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272

https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

8

I would hope that it can now fetch and cache the jar (should be no need for manual upload now that /me/steinborn/* is in the routing rules to be cached?).

Yep. me too but it seems it’s still not there.

I have also tried to redeploy again to geosolutions, hoping the routing rules would catch that but it’s still not there on nexus.

As for deploy, you provided credentials in your ~/.m2/settings.xml file yes?

Yep.

What does “that didn’t work” look like…

Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy-file (default-cli) on project imageio-ext: Failed to deploy artifacts: Could not find artifact me.steinborn:libdeflate-java-core:jar:0.1.0-beta in osgeo (dav:http://download.osgeo.org/upload/geotools) → [Help 1]

LDAP user danieleromagnoli has been granted GeoServer-Admin and GeoTools-Admin permissions; but once again manual upload should not be required.

I’ll take a look again on tomorrow.
Thanks for your follow up so far.

Regards,
Daniele

···

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

9

This worked first try for me :slight_smile:

  1. I searched repo.osgeo.org to see if it was there https://repo.osgeo.org/#browse/search=keyword%3D%22libdeflate-java-core%22
    It was not.

  2. I edited modules/plugins/image/pom.xml with:

    me.steinborn
    libdeflate-java-core
    0.1.0-beta

  3. Tried it out: mvn compile

  4. It downloaded as expected:
    Downloaded from osgeo-snapshot: https://repo.osgeo.org/repository/snapshot/org/geotools/gt-opengis/27-SNAPSHOT/gt-opengis-27-20220419.191623-120.jar (0 B at 0 B/s)
    Downloading from osgeo-release: https://repo.osgeo.org/repository/release/me/steinborn/libdeflate-java-core/0.1.0-beta/libdeflate-java-core-0.1.0-beta.jar
    Downloaded from osgeo-release: https://repo.osgeo.org/repository/release/me/steinborn/libdeflate-java-core/0.1.0-beta/libdeflate-java-core-0.1.0-beta.jar (47 kB at 57 kB/s)

  5. I searched repo.osgeo.org to see if it was there https://repo.osgeo.org/#browse/search=keyword%3D%22libdeflate-java-core%22
    It is now listed.

···

Is that not what you wanted? Or did I miss something…

Jody Garnett

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.