[SAC] [OSGeo] #2180: geotools maven repository missing?

Committees SAC mailing list
1

#2180: geotools maven repository missing?
---------------------------+-------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Keywords:
---------------------------+-------------------
The geotools maven repository URL is 404!

* http://download.osgeo.org/webdav/geotools/

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

2

#2180: geotools maven repository missing?
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

Experimenting shows the content is available here:

* http://download.osgeo.org/webdav//geotools/

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:1&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

3

#2180: geotools maven repository missing?
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

I think this was working on June 28th or June 29th?

Many builds will not notice until they build with a "-U" update
dependencies options at which point they will get broken.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:2&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

4

#2180: geotools maven repository missing?
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by robe):

I got rid of the rewrite rule which I think jef had added and that seems
to have fixed it.

Maybe jef can look into this again.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:3&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

5

#2180: geotools maven repository missing?
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

I think this is working, not really sure if the build tools are smart
enough to handle http --> https redirect.

Going to mark as closed and reopen if their is any further feedback.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:4&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

6

#2180: geotools maven repository missing?
---------------------------+---------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone:
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------
Changes (by jive):

* status: new => closed
* resolution: => fixed

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:5&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

7

#2180: geotools maven repository missing?
---------------------------+-----------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: reopened
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+-----------------------
Changes (by jive):

* status: closed => reopened
* resolution: fixed =>

Comment:

It was not smart enough to follow https redirect. Can we have both http
and https at the same time?

This will give us time to change our documentation and get the word out.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:6&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

8

#2180: geotools maven repository missing?
---------------------------+-----------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: reopened
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+-----------------------

Comment (by robe):

Jody,

I think the https redirect is happening on the browser. All of
download.osgeo.org is redirecting and I don't see any redirect in configs
to cause that. When I use wget, I was still getting a 302 move just for
webdav (but it wasn't redirecting to https just another http) and the
others weren't redirecting.

I did still have in place a left over REWRITE rule for the regular non
https webdav which might have been what you were experiencing.

https://git.osgeo.org/gitea/sac/download-apache-
config/commit/b2c5ddce447dd33bbc75486aea53c5a4e23caf7d

I've taken that out, and also enabled webdav permissions (those were
remarked out on the https and not on the http) and they seem to control
who can do webdav posts

https://git.osgeo.org/gitea/sac/download-apache-
config/commit/da3a29bf64983a0218986b155738ecaa6b95fb60

Can you see if you can still do webdav posts and also download.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:7&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

9

#2180: geotools maven repository missing?
---------------------------+-----------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: reopened
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+-----------------------

Comment (by jef):

What ever is using this is constantly requesting non-existing stuff, which
get logged in download_error_log:

{{{
# head -1 /var/log/apache2/download_error_log
[Thu Jul 05 09:39:01 2018] [error] [client X.X.X.X] File does not exist:
/osgeo/download/webdav/geotools/org/springframework/spring-core/2.0.5
# wc -l /var/log/apache2/download_error_log
197851 /var/log/apache2/download_error_log
# fgrep /osgeo/download/webdav/ /var/log/apache2/download_error_log | wc
-l
194565
}}}

ie. 197851 total errors in less than 24h - 194565 webdav requests.

{{{
# b=; while sleep 1; do a=$(wc -l </var/log/apache2/download_error_log); [
-n "$b" ] && echo $a $(( a-b )); b=$a; done
203651 15
203663 12
203687 24
203704 17
203716 12
203727 11
203735 8
203747 12
203757 10
203761 4
203769 8
203788 19
203809 21
203816 7
}}}

more than ~10 per second… Any idea why that happens? These requests also
get logged in access.log:

{{{
# fgrep webdav /var/log/apache2/download_access_log|wc -l; grep "webdav.*
404 " /var/log/apache2/download_access_log | wc -l
636603
593517
}}}

so most of the requests are 404 from webdav. But this includes the time
while the rule was in place - I can't check the previous log as I just re-
enabled rotation and download_access_log.1 is 22 GB (**twenty two
gigabytes** since Feb 11th).

That's also why I addressed this at all - / was full.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:8&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

10

#2180: geotools maven repository missing?
---------------------------+-----------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: reopened
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+-----------------------

Comment (by jive):

Replying to [comment:7 robe]:
> Jody,
>
> I think the https redirect is happening on the browser. All of
download.osgeo.org is redirecting and I don't see any redirect in configs
to cause that. When I use wget, I was still getting a 302 move just for
webdav (but it wasn't redirecting to https just another http) and the
others weren't redirecting.

I was not using a browser, but using the repository from the maven build
tool.

Changing maven pom.xml repository definition
https://download.osgeo.org/webdav/geotools/ worked, while using
http://download.osgeo.org/webdav/geotools/ did not.

> I did still have in place a left over REWRITE rule for the regular non
https webdav which might have been what you were experiencing.
>
> https://git.osgeo.org/gitea/sac/download-apache-
config/commit/b2c5ddce447dd33bbc75486aea53c5a4e23caf7d
>
> I've taken that out, and also enabled webdav permissions (those were
remarked out on the https and not on the http) and they seem to control
who can do webdav posts
>
> https://git.osgeo.org/gitea/sac/download-apache-
config/commit/da3a29bf64983a0218986b155738ecaa6b95fb60
>
> Can you see if you can still do webdav posts and also download.

1. The use of maven build tool is now working with the "http" repository
reference.

2. The use of web dev with both "http" and "https" is now working

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:9&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

11

#2180: geotools maven repository missing?
---------------------------+-----------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: reopened
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+-----------------------

Comment (by jive):

Replying to [comment:8 jef]:

> What ever is using this is constantly requesting non-existing stuff,
which get logged in download_error_log:
>
> {{{
> # head -1 /var/log/apache2/download_error_log
> [Thu Jul 05 09:39:01 2018] [error] [client X.X.X.X] File does not exist:
/osgeo/download/webdav/geotools/org/springframework/spring-core/2.0.5
> # wc -l /var/log/apache2/download_error_log
> 197851 /var/log/apache2/download_error_log
> # fgrep /osgeo/download/webdav/ /var/log/apache2/download_error_log | wc
-l
> 194565
> }}}
>
> ie. 197851 total errors in less than 24h - 194565 webdav requests.
>

The tool used is

The functionality is actually working as expected

1. For each project a number of repositories is configured.

2. Each dependency is represented by a "pom" (project object model) and
"jars", "javadocs", "source", etc... as required. The "pom" file describes
the dependency and what it needs in turn.

3. Maven uses this to build up a tree like a package manager and downloads
all the components needed as part of the build process.

4. For each dependency it asks each repository if they have it, and then
downloads from one of them.

{{{
      Downloading from osgeo:
https://download.osgeo.org/webdav/geotools/org/codehaus/plexus/plexus-
io/2.0.3/plexus-io-2.0.3.pom
      Downloading from boundless:
https://repo.boundlessgeo.com/snapshot/org/codehaus/plexus/plexus-io/2.0.3
/plexus-io-2.0.3.pom
      Downloading from sonatype-snapshots:
https://oss.sonatype.org/content/repositories/snapshots/org/codehaus/plexus
/plexus-io/2.0.3/plexus-io-2.0.3.pom
      Downloading from central:
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/2.0.3
/plexus-io-2.0.3.pom
      Downloaded from central:
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/2.0.3
/plexus-io-2.0.3.pom (1.7 kB at 54 kB/s)
}}}

5. Because our repository is popular it is used by not only the geotools
build, but by all the projects that make use of geotools world wide. Each
of those projects is going to talk to us .. all the time.

There are two ways out:

a) Use a maven repository application (rather than web dev) for hosting
artifacts. Artififactory is used by boundless, Nexus is used by
LocationTech

b) Deploy on to "maven central" rather than out own repository.

The part of the maven infrastructure we are not making use of is called
"maven central", where each group responsible for a project can upload
their jars, which are then mirrored around the world.

My understanding is we are unable to unable to upload our jars because we
do not meet maven central policies ... our software depends on non open
source software ... Java Advanced Imaging.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:10&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

12

#2180: geotools maven repository missing?
---------------------------+-----------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: reopened
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+-----------------------

Comment (by robe):

I'm not sure we want to consider changing how its setup right now.

However since we are planning to move download.osgeo.org to OSGeo7, then
we should add this as part of our on-going OSGeo7 spec

https://wiki.osgeo.org/wiki/Osgeo7

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:11&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

13

#2180: geotools maven repository missing?
---------------------------+---------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone:
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------
Changes (by jive):

* status: reopened => closed
* resolution: => fixed

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:12&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

14

#2180: geotools maven repository missing?
---------------------------+---------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone:
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------

Comment (by jive):

With respect to OSGeo7, I have spoken with James Hughes and he will see if
he can help GeoTools release to maven central. If we can do this then a
lot of pressure on our server will be lifted.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2180#comment:13&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.