[GeoNetwork-devel] No good very bad day for repo.osgeo.org release (thanks to geonetwork-cache)

GeoNetwork GeoNetwork Developer
1

Wow that was exciting - we as the GeoNetwork team managed to break a lot of java projects around the world :slightly_smiling_face:

How we did it:

  1. In an earlier life (say GeoNetwork 2.x) a maven repository was included as a submodule for things that could not be found officially online.
    https://github.com/geonetwork/core-maven-repo

  2. Then at some point builds referenced the raw GitHub endpoint:
    https://raw.githubusercontent.com/geonetwork/core-maven-repo/master

  3. Jeroen got a warning from GitHub two years ago, and we setup repo.osgeo.org geonetwork-cache to cache the above location

  4. Yesterday GitHub cut off access to the raw location (maybe they sent a letter)

  5. This caused geonetwork-cache to start collecting timeouts, and thus zero byte files
    For everyone using repo.osgeo.org “release”, so they are all broken this morning

I could not figure out a way to remove such files from repo.osgeo.org geonetwork-cache. But I was able to remove geonetwork-cache from repo.osgeo.org release.

It is my hope that this will fix everything for the java ecosystem, however I have not yet got confirmation if this step has helped.
There is a risk all the other repositories have cached zero sized “releases” and will be stuck. If so this will be a very expensive mistake (and we should issue an apology.)

It is my hope that this limits problems to the core-geonetwork community only:

  1. Chances are developers that already have stuff in their local repository will not notices
  2. Build servers that start with a fresh local repository should be broken if they ask anything unique / old
    (For example like our custom Saxon patches)

Action:

  1. We can apologize to everyone, these things happen.

  2. GeoNetwork team should collect anything we care about from an old revision:
    https://github.com/geonetwork/core-maven-repo/tree/56d8d6e6d373b0f13306e09519c3ee1508a1636c

    And upload it to geonetwork-releases:

  3. You can upload using the repo.osgeo.org GUI and your OSGeoID username;
    Or use the command line (running from your) core-geonetwork checkout which as the distribution repositories configured.
    https://docs.geotools.org/latest/developer/procedures/add.html

Reference:

···

–
Jody Garnett

2

Testing without geonetwork-cache:

main (4.4.x)
mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests

[ERROR] Failed to execute goal on project gn-core: Could not resolve dependencies for project org.geonetwork-opensource:gn-core:jar:4.4.0-SNAPSHOT: Could not find artifact jzkit:jzkit-z3950-plugin:jar:3.r260.geonetwork.patch.1 in osgeo-release (https://repo.osgeo.org/repository/release/) → [Help 1]

3.12.x

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork-3.12/m2/repository -DskipTests

ERROR] Failed to execute goal on project core: Could not resolve dependencies for project org.geonetwork-opensource:core:jar:3.12-SNAPSHOT: Could not find artifact jzkit:jzkit-z3950-plugin:jar:3.r260.geonetwork.patch.1 in osgeo-release (https://repo.osgeo.org/repository/release/) → [Help 1]

I assume earlier releases are also broken.

It becomes now important to recover the missing content.

···

–
Jody Garnett

–
Jody Garnett

3

To follow up on this:

It becomes now important to recover the missing content.

I am going to take notes in this email as I am unlikely do complete the activity before the end of my work day; and I expect many core-geonetwork developers will be unable to work tomorrow until this is resolved.

Reviewing:

https://repo.osgeo.org/#browse/browse:geonetwork-releases

We can only really build / fail / upload to determine what is actually missing:

Could not find artifact jzkit:jzkit-z3950-plugin:jar:3.r260.geonetwork.patch.1 in osgeo-release (https://repo.osgeo.org/repository/release/)

Checking geonetwork-releases:

  • The folder structure is there, but no jar (no idea why)

Checking geonetwork-caches:

Going to download jars and pom and then upload to geonetwork-releases.

  • Oh no - each link says “repository offline”!

So two options:

a) Recover from revision:

Merge branch ‘master’ of github.com:geonetwork/core-maven-repo 56d8d6e6d373b0f13306e09519c3ee1508a1636c

fxprunayre committed on Jan 24, 2019

b) Team effort to recover from everyone’s local repository

Checking from command line examples:

Adapting to publish from ~/.m2/repository

~/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar

~/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.pom

mvn deploy:deploy-file
-DpomFile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.pom \

-Dfile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar
-DrepositoryId=nexus
-Durl=https://repo.osgeo.org/repository/geonetwork-releases/

Cannot deploy artifact from the local repository: /Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar

Well I know how to fix that:

mvn deploy:deploy-file
-DpomFile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.pom
-Dfile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar
-DrepositoryId=nexus
-Durl=https://repo.osgeo.org/repository/geonetwork-releases/ \

-Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository

BUILD SUCCESS

Back to building:

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests -ntp

[ERROR] Failed to execute goal on project gn-core: Could not resolve dependencies for project org.geonetwork-opensource:gn-core:jar:4.4.0-SNAPSHOT: jzkit:jzkit-z3950-plugin:jar:3.r260.geonetwork.patch.1 was not found in https://repo.osgeo.org/repository/release/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of osgeo-release has elapsed or updates are forced → [Help 1]

Build with -U:

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests -ntp -U

Failed to execute goal on project gn-web-app: Could not resolve dependencies for project org.geonetwork-opensource:gn-web-app:war:4.4.0-SNAPSHOT: Could not find artifact jakarta:jakarta-slide-webdavlib:jar:2.1 in osgeo-release (https://repo.osgeo.org/repository/release/) → [Help 1]

mvn deploy:deploy-file
-DpomFile=/Users/jgarnett/.m2/repository/jakarta/jakarta-slide-webdavlib/2.1/jakarta-slide-webdavlib-2.1.pom
-Dfile=/Users/jgarnett/.m2/repository/jakarta/jakarta-slide-webdavlib/2.1/jakarta-slide-webdavlib-2.1.jar
-DrepositoryId=nexus
-Durl=https://repo.osgeo.org/repository/geonetwork-releases/ \

-Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository

Resume build:

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests -rf :gn-web-app -ntp -U

BUILD SUCCESS

Okay GeoNetwork main builds again, please use the above notes to get 3.12.x building.

Thanks for your patience today everyone,
Jody

4

Hello Jody,

Thank you for taking the time to fix this issue.

Following up, building some branches from scratch, without any local maven cache:

  • 3.12.x → Success, all dependencies found.

  • 4.0.x → Success, all dependencies found.

  • 4.2.x → Success, all dependencies found.

  • main → Success, all dependencies found.

Going to download jars and pom and then upload to geonetwork-releases.

  • Oh no - each link says “repository offline”!

This could be because I set the geonetwork-cache repository as offline when we detected the problem in our CI tool yesterday morning.

What puzzles me is why this has happened now if two years ago we removed the contents of https://github.com/geonetwork/core-maven-repo/. I don’t think Github has stopped serving contents from there. Maybe the contents were still cached in https://repo.osgeo.org/geonetwork-cache and somehow they have been evicted / cleared from there. Anyway, I think it’s safe to delete geonetwork-cache repository from repo.osgeo.org since all the dependencies are already in the release repo and there are instructions for legacy GeoNetwork versions to update the configuration in case anybody needs to build and old version.

Best regards,

Juan Luis.

Juan Luis Rodriguez Ponce

E-mail: juanluisrp@anonymised.com…437…

https://www.geocat.net

Veenderweg 13

6721 WD Bennekom

The Netherlands

Tel: +31318416664

---- On Fri, 08 Sep 2023 04:48:16 +0200 Jody Garnett via GeoNetwork-devel geonetwork-devel@lists.sourceforge.net wrote —

To follow up on this:

It becomes now important to recover the missing content.

I am going to take notes in this email as I am unlikely do complete the activity before the end of my work day; and I expect many core-geonetwork developers will be unable to work tomorrow until this is resolved.

Reviewing:

https://repo.osgeo.org/#browse/browse:geonetwork-releases

We can only really build / fail / upload to determine what is actually missing:

Could not find artifact jzkit:jzkit-z3950-plugin:jar:3.r260.geonetwork.patch.1 in osgeo-release (https://repo.osgeo.org/repository/release/)

Checking geonetwork-releases:

  • The folder structure is there, but no jar (no idea why)

Checking geonetwork-caches:

Going to download jars and pom and then upload to geonetwork-releases.

  • Oh no - each link says “repository offline”!

So two options:

a) Recover from revision:

Merge branch ‘master’ of github.com:geonetwork/core-maven-repo 56d8d6e6d373b0f13306e09519c3ee1508a1636c

fxprunayre committed on Jan 24, 2019

b) Team effort to recover from everyone’s local repository

Checking from command line examples:

Adapting to publish from ~/.m2/repository

~/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar

~/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.pom

mvn deploy:deploy-file \

-DpomFile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.pom \

-Dfile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar \

-DrepositoryId=nexus \

-Durl=https://repo.osgeo.org/repository/geonetwork-releases/

Cannot deploy artifact from the local repository: /Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar

Well I know how to fix that:

mvn deploy:deploy-file \

-DpomFile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.pom \

-Dfile=/Users/jgarnett/.m2/repository/jzkit/jzkit-z3950-plugin/3.r260.geonetwork.patch.1/jzkit-z3950-plugin-3.r260.geonetwork.patch.1.jar \

-DrepositoryId=nexus \

-Durl=https://repo.osgeo.org/repository/geonetwork-releases/ \

-Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository

BUILD SUCCESS

Back to building:

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests -ntp

[ERROR] Failed to execute goal on project gn-core: Could not resolve dependencies for project org.geonetwork-opensource:gn-core:jar:4.4.0-SNAPSHOT: jzkit:jzkit-z3950-plugin:jar:3.r260.geonetwork.patch.1 was not found in https://repo.osgeo.org/repository/release/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of osgeo-release has elapsed or updates are forced → [Help 1]

Build with -U:

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests -ntp -U

Failed to execute goal on project gn-web-app: Could not resolve dependencies for project org.geonetwork-opensource:gn-web-app:war:4.4.0-SNAPSHOT: Could not find artifact jakarta:jakarta-slide-webdavlib:jar:2.1 in osgeo-release (https://repo.osgeo.org/repository/release/) → [Help 1]

mvn deploy:deploy-file \

-DpomFile=/Users/jgarnett/.m2/repository/jakarta/jakarta-slide-webdavlib/2.1/jakarta-slide-webdavlib-2.1.pom \

-Dfile=/Users/jgarnett/.m2/repository/jakarta/jakarta-slide-webdavlib/2.1/jakarta-slide-webdavlib-2.1.jar \

-DrepositoryId=nexus \

-Durl=https://repo.osgeo.org/repository/geonetwork-releases/ \

-Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository

Resume build:

mvn clean install -Dmaven.repo.local=/Users/jgarnett/dev/geonetwork/core-geonetwork/m2/repository -DskipTests -rf :gn-web-app -ntp -U

BUILD SUCCESS

Okay GeoNetwork main builds again, please use the above notes to get 3.12.x building.

Thanks for your patience today everyone,

Jody

GeoNetwork-devel mailing list

GeoNetwork-devel@anonymised.comrge.net

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

5

Perfect, thanks for checking other branches Juan.

I expect we were only missing those two jars. If anymore are lost we can recover from git history.

I will remove the geonetwork-cache from repo.osgeo.org and close the ticket.

···

–
Jody Garnett