#606: Downloads.osgeo Upgrade/Replacement
---------------------------+------------------------------------------------
Reporter: wildintellect | Owner: sac@…
Type: task | Status: new
Priority: major | Milestone:
Component: Systems Admin | Keywords: download, infrastructure
---------------------------+------------------------------------------------
I was hoping to have an upgrade for Download.osgeo by the next release of
the LiveDVD(August), looks like the current issues with the Blades has
made this a more urgent matter. Below are some notes on ideas we might
want to implement in the replacement service.
1. Finish ticket:336 - Antivirus scan new uploads (unclear if it was
finished)
1. For non-SAC members with access use rssh to limit accounts to SFTP
only. This can be used to prevent shell access, port forwarding and all
sorts of other ways hacker might try use a compromised account.
1. Consider implementing a mirroring system, with something like
http://www.mirrorbrain.org/ or https://fedorahosted.org/mirrormanager/
* These applications keep an eye on the mirrors and can be configured so
md5 etc are hosted on the main site but files can be selective mirrored on
other servers. They also scan for status and verify that files match the
main mirror.
* The url to download is still the same, the manager automatically
routes the download with a redirect to geographically closer mirrors, less
impacted mirrors, etc.
* Mirrors do not have to be complete mirrors they can be partial, and
TOS rules can limit bandwidth on mirrors with limitations.
* simgislab has offered a mirror(Ukraine), I may be able to offer a
mirror (US-West) and a mirror in Asia and Australia(Lisasoft?) would
immensely increase international download speeds.
* This may also solve ticket:515 as it will route requests to multiple
mirrors
1. A Newer OS than the current is needed, something that allows for
larger than 2 GB file downloads(Possible Apache issue). Consider XFS
filesystem or EXT4 for increased speed.
Please add additional ideas to the ticket so we can discuss them when we
discuss the resolution of the current Blade hosted services.
--
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/606>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.
#606: Downloads.osgeo Upgrade/Replacement
---------------------------+------------------------------------------------
Reporter: wildintellect | Owner: sac@…
Type: task | Status: new
Priority: major | Milestone:
Component: Systems Admin | Keywords: download, infrastructure
---------------------------+------------------------------------------------
Comment(by warmerdam):
ticket #336 was not completed. I experimented with clamav but did not get
anything automated running.
Use of rssh to limit most access to sftp might be prudent if we are
particularly concerned about a lot of people having login access, though
it is not clear if that has been a serious problem.
I'm not sure why the proposed mirroring approach is better than the simple
rsync based mirroring we already had in place, and can be setup fairly
easily?
I'm presuming John will install something newish OS wise when he scrubs
the blade.
I'm honestly still not so keen on livedvd's sloshing around on the
download site, but I imagine I'm just going to have to live with it.
There has been some discussion with Chris about whether download.osgeo.org
ought to live at OSU OSL with telascience hosting a live backup/mirror.
I'm not sure how that decision will be resolved.
--
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/606#comment:1>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.
#606: Downloads.osgeo Upgrade/Replacement
---------------------------+------------------------------------------------
Reporter: wildintellect | Owner: sac@…
Type: task | Status: new
Priority: major | Milestone:
Component: Systems Admin | Keywords: download, infrastructure
---------------------------+------------------------------------------------
Comment(by wildintellect):
I should clarify our previous conversation on the topic. MirrorBrain does
not do the mirroring, that would still be left to rsync. What it does do
is allow us to seamlessly allow for many mirrors. Basically we don't have
to choose OSL or Telescience, it could be both plus 2-3 more donated
mirrors around the world but to the end user it's still only 1 url.
This lets us increase our download capacity, load balance between mirrors,
selectively redirect things like the Live DVD downloads to specific
mirrors, ensure integrity of mirrors, and keep logs of all downloads
traffic even if they get redirected to a mirror.
[http://www.mirrorbrain.org/features/ More details]
--
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/606#comment:2>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.
#606: Downloads.osgeo Upgrade/Replacement
---------------------------+------------------------------------------------
Reporter: wildintellect | Owner: sac@…
Type: task | Status: new
Priority: major | Milestone:
Component: Systems Admin | Keywords: download, infrastructure
---------------------------+------------------------------------------------
Comment(by hamish):
Replying to [comment:1 warmerdam]:
> I'm honestly still not so keen on livedvd's sloshing around
> on the download site, but I imagine I'm just going to have
> to live with it.
fwiw I notice on the Server Status wiki page that xblade-15 is listed for
ISO downloads...?
that does not solve the reported (AFAIU) ">2gb files from apache2 needing
a 64bit OS" problem, but as I just saw that I thought I'd mention it as
another option for the future.
for my part I don't really care where it goes as long as we have a semi-
official, semi-restricted, and mostly-static (for years) place to stash
stuff, including things like
large binary sample datasets & custom installer packages which do not
belong in the build script Subversion. (I'm talking about max 100mb files
used in the build, not the final ISOs & overhead of backing up the multi-
gigabyte files)
it is perhaps nice to be able to chmod any md5sum listing on the download
server to be go-w, but in the past I've mostly had to fix it the other way
when the uploader did not make it g+w so it required more effort to
maintain things...
regards,
Hamish
--
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/606#comment:3>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.
#606: Downloads.osgeo Upgrade/Replacement
--------------------------------------+-------------------------------------
Reporter: wildintellect | Owner: sac@…
Type: task | Status: closed
Priority: major | Milestone:
Component: Systems Admin | Resolution: worksforme
Keywords: download, infrastructure |
--------------------------------------+-------------------------------------
Changes (by wildintellect):
* status: new => closed
* resolution: => worksforme
Comment:
1. Still todo
2. not deemed critical at this time
3. mirrorbrain being tested by wildintellect on a non-osgeo server in
conjunction with the Live DVD downloads, download server is one of the
mirrors.
4. solved with fresh vm
--
Ticket URL: <http://trac.osgeo.org/osgeo/ticket/606#comment:4>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.