[OSGeo] #3116: repos.osgeo.org needs admin attention

#3116: repos.osgeo.org needs admin attention
--------------------------±--------------------------
Reporter: peterafrigis | Owner: sac-tickets@…
Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin | Keywords: repo
--------------------------±--------------------------
Automated [Workflow runs · geoserver/geoserver · GitHub GeoServer
builds] are failing for the past 3-4 days with the error message:

Error: 4,838 [ERROR] Failed to execute goal on project gs-grib: Could not
resolve dependencies for project org.geoserver.extension:gs-
grib:jar:2.25-SNAPSHOT: Failed to collect dependencies at org.geotools:gt-
grib:jar:31-SNAPSHOT → edu.ucar:cdm:jar:4.6.15 →
edu.ucar:httpservices:jar:4.6.15: Failed to read artifact descriptor for
edu.ucar:httpservices:jar:4.6.15: The following artifacts could not be
resolved: edu.ucar:httpservices:pom:4.6.15 (absent): Could not transfer
artifact edu.ucar:httpservices:pom:4.6.15 from/to osgeo-releases
(Repository - Sonatype Nexus Repository): status code: 500, reason
phrase: Server Error (500) → [Help 1]

Jody believes one of the repos is having problems; we access via a cache
at repos.osgeo.org

Could the repo admin please take a look at the problem to resolve it?

If you need to contact a GeoServer developer to help test,
https://app.gitter.im/#/room/#geoserver_geoserver:gitter.im or geoserver-
devel@lists.sourceforge.net or me personally: gs@smythe.co.za

Thank you

Peter

Ticket URL: https://trac.osgeo.org/osgeo/ticket/3116
OSGeo https://osgeo.org/
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
--------------------------+----------------------------
Reporter: peterafrigis | Owner: sac-tickets@…
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: repo |
--------------------------+----------------------------
Comment (by robe):

Replying to [comment:1 jive]:
> Andrea collected some details - with the conclusion that it may be
corrupted storage on the repo server? He indicated clearing the cache and
allowing it to redownload the artifacts did help.
>
> Checking the pom.xml file causing problems:
>
> * Visiting Sonatype Nexus Repository
cache:edu%2Fucar%2Fhttpservices%2F4.6.15 it appears listed
>
> * Other things in that cache are present:
https://repo.osgeo.org/repository/ucar-
cache/edu/ucar/cdm/4.4.0/cdm-4.4.0.pom
>
> * The original content is here
Sonatype Nexus Repository
releases:edu%2Fucar%2Fhttpservices%2F4.6.15
>
> * And the original pom.xml file can be downloaded
Sonatype Nexus Repository
releases:edu%2Fucar%2Fhttpservices%2F4.6.15%2Fhttpservices-4.6.15.pom
>
> So I expect this is a problem with repo.osgeo.org storage / cache.

@jive,

Is this something you need help with or you have it under control?
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

I do not have it under control, I may have time to look at it next week.

It feels like Andrea did the things that could be done with application configuration (clearing cache and letting it try and fetch again). Which is all I know how to do …

Is there any sign of trouble with disk / storage?

Oops you replied to discourse :slight_smile: instead of adding to trac. Because you are an admin and I can’t block admins from replying. That said, I’m going to take this opportunity to reply to you back on discourse.

No I don’t see any signs of disk corruption. I’m worried it might have had something to do with the upgrade a couple of weeks ago, maybe something got corrupted during the upgrade process. You know how long this has been an issue?

Just a few days, Andrea would know more or I can check the build server …

https://build.geoserver.org/view/geotools/job/geotools-main/

Looks like Jan 1 was the last successful build
The first failure was on Jan 4th

Not sure what to do about reply vs discorse?

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Changes (by robe):

* cc: jive (added)
* owner: sac-tickets@… => jive
* component: SysAdmin => SysAdmin/Repo

Comment:

@jive,

Okay I guess it's best we continue on here instead of the discourse feed.
So as you mentioned, the last successful run was Jan 1st

geotools-main [Jenkins]

So I think that rules out the major upgrade I did on #2932 as that was 3
weeks ago.

I'm going to check the errors in the logs and do a more intensive disk
check if I can't determine from the logs.
Might also be something about how docker volumes work that something is
locked and maybe the container just needs a reboot.
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by robe):

@jive,

One more question, from the above, I'm not sure how to tell if anything is
even wrong.

Can you give me some command or web call I can use to replicate the issue,
all the above links when I go there show something, so I think I'm missing
something.
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by robe):

I do see here - Sonatype Nexus Repository that

System - Repository Health Check: ucar-cache

has been failing. The other cache healthchecks return okay. So I guess
that gets to your point about the ucar-cache being corrupt. Though that
job doesn't say how long it's been failing and it runs every hour.

I also do notice it looks like the docker container restarted 2 weeks ago,
and I can't remember restarting it, though it's possible I did. So perhaps
something happened during that restart or it restarted on it's own because
of some corruption.
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by jive):

To check:

1. Visit to artifact for which a problem is reported:
Sonatype Nexus Repository
cache:edu%2Fucar%2Fhttpservices%2F4.6.15

2. Click on the download link

3. Success is a small text file; failure is 500 error
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by jive):

Sorry that was the link to the folder, the specific pom.xml file is at
Sonatype Nexus Repository
cache:edu%2Fucar%2Fhttpservices%2F4.6.15%2Fhttpservices-4.6.15.pom
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by peterafrigis):

I am attempting to follow along. The "download link" is on the right hand
side, under Summary > Path > click this link (e.g.
https://repo.osgeo.org/repository/ucar-
cache/edu/ucar/httpservices/4.6.15/httpservices-4.6.15.pom)

I still get a 500 Internal Server error
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by jive):

I now see a ucar-all-cache (so maybe someone deleted and remade the cache
to point to Repository - Sonatype Nexus Repository )?

It still as "unidata-artifacts" routing rule so will only be used to cache
"^/edu/ucar/.*"
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: new
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution:
Keywords: repo |
---------------------------+------------------------
Comment (by jive):

Okay build is now managing netCDF module so I think this can be closed.

I found that Andrea had done the change described above (thanks Andrea)
--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

#3116: repos.osgeo.org needs admin attention
---------------------------+------------------------
Reporter: peterafrigis | Owner: jive
     Type: defect | Status: closed
Priority: major | Milestone: Unplanned
Component: SysAdmin/Repo | Resolution: fixed
Keywords: repo |
---------------------------+------------------------
Changes (by jive):

* status: new => closed
* resolution: => fixed

--
Ticket URL: <#3116 (repos.osgeo.org needs admin attention) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.