[SAC] [OSGeo] #458: revive geo-torrents?

Committees SAC mailing list
1

#458: revive geo-torrents?
---------------------------------------+------------------------------------
Reporter: hamish | Owner: sac@lists.osgeo.org
    Type: task | Status: new
Priority: normal | Component: SAC
Keywords: geodata, livedvd, torrents |
---------------------------------------+------------------------------------
Hi,

since ER Mapper is no more their geo torrents.org site has gone away
leaving a void in the large geodata dataset distribution world.

Due to an apache bug we can't serve the un-`split` live-dvd ISO in one
piece. (>2gb)

A bright spark in the liveDVD mailing list has figured out how to use
openbittorrent.com to act as the tracker,
  http://lists.osgeo.org/pipermail/live-demo/2009-September/000569.html

has uploaded the .torrent to download.osgeo.org, and is now distributing
the latest liveDVD .iso build with that. Only problem is that it still
relies on his home computer as the primary seed.

if download.osgeo.org were running a bittorrent client, OSGeo's fat pipes
& uptime could take over that seeding responsibility. There's also the
option that other impatient types could start downloading the ISO before
it finished uploading (which takes about 4 hours), but it may be
preferable to have the osgeo server only allow torrent data to flow
outwards.

I'm really not sure what goes into setting this up, before doing any
research on the matter I thought I'd check if the geodata crowd would want
on board, what their needs were, and if the SAC had any reservations or
ideas about such a thing.

?

cheers,
Hamish

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/458&gt;
OSGeo <http://www.osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

2

#458: revive geo-torrents?
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: sac@lists.osgeo.org
      Type: task | Status: new
  Priority: normal | Component: SAC
Resolution: | Keywords: geodata, livedvd, torrents
---------------------+------------------------------------------------------
Comment (by wildintellect):

I'm on the SAC mailing list now so we can properly discuss this. Sure long
term we need to upgrade Apache to get rid of this 2GB. In the mean time
and possibly a long term helpful idea I suggest we start seeding torrents
from an osgeo server or 2.

I've narrowed down the software options to
[http://www.transmissionbt.com/ transmission]
or
[http://libtorrent.rakshasa.no/wiki rtorrent]

Both would work well on a headless server and can handle auto torrenting
from a watched folder. Meaning the rules could be that if it's in the
torrent folder it will be seeded, remove it from said folder to stop
seeding. Not sure if symlinks work on that.

The other big decision would be what to cap the upload rate at. Note: this
could actually be used to help upload too for some of our associates in
far off places, the occasionally have to restart big uploads, but if they
seeded instead md5 would be checked automatically and others could help
pull chunks to closer machines.

See
http://wiki.osgeo.org/wiki/GISVM_Build#Build_the_Live_DVD_ISO_image_build
for some open free trackers that we're using.

Side notes:
rtorrent does included a tracker, but we might all consider hosting a
mirror of the OpenTracker project - they provide hardware if we have
space(or something like that)
transmission has an optional web management interface

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/458#comment:1&gt;
OSGeo <http://www.osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

3

#458: revive geo-torrents?
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: sac@lists.osgeo.org
      Type: task | Status: new
  Priority: normal | Component: SAC
Resolution: | Keywords: geodata, livedvd, torrents
---------------------+------------------------------------------------------
Comment (by hamish):

Replying to [comment:1 wildintellect]:
> Both would work well on a headless server and can handle
> auto torrenting from a watched folder. Meaning the rules
> could be that if it's in the torrent folder it will be seeded,
> remove it from said folder to stop seeding. Not sure if
> symlinks work on that.

mmm, it is more work to admin but I more favor an explicit approach to
distribution. makes it harder to mess up really really badly.

> The other big decision would be what to cap the upload rate at.

why have a cap at all? AFAIK bandwidth is "not really a problem" for the
SDSC.

> Side notes:
> rtorrent does included a tracker, but we might all consider
> hosting a mirror of the OpenTracker project - they provide
> hardware if we have space(or something like that)

[we can (and do) use the opentracker already which is grand; the less
stuff we have to admin the better]

as for hosting a mirror of it... um, I'm not really sure what else is
tracked by that site (AFAICT from their website neither do/can they) but
the U.S. in general and California in particular are perhaps not the legal
jurisdictions you want to test that in. The state is pretty much bankrupt
and Hollywood does bring a few tax dollars I imagine. That, and our hosts
might get grumpy about all the OT traffic.

> transmission has an optional web management interface

is that much of a plus over command line reports? I guess it would be a
bit like sitestats apache log viewers to show you what's going on? If so,
could be an interesting feature. But if it's just a webform crutch for
people who are afraid of the command line I don't see much gain.

obviously we don't want this abused by 3rd parties in any way, so the more
ways we can find to lock it down the better :slight_smile:

Hamish

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/458#comment:2&gt;
OSGeo <http://www.osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

4

#458: revive geo-torrents?
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: sac@lists.osgeo.org
      Type: task | Status: new
  Priority: normal | Component: SAC
Resolution: | Keywords: geodata, livedvd, torrents
---------------------+------------------------------------------------------
Comment (by wildintellect):

The bandwidth cap is to prevent the torrents from making the server
unusable. Torrents have been know to essentially prevent ssh, and take up
all the system resources if you let them. Seems reasonable to also ensure
that other downloads from the server continue as is.

The web GUI as I understand it will let you see the health, seeders,
leachers and speed of the torrents much in the way most torrent GUIs on
the desktop do. I would say firewall the port it's on and require it to be
fowarded over ssh if you want to see it. Might be easier to use than the
cli for reviewing status...not sure though since I haven't used either.

I'm not concerned about legal issues, OSGeo does not host or seed anything
of questionable nature.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/458#comment:3&gt;
OSGeo <http://www.osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

5

#458: revive geo-torrents?
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: sac@lists.osgeo.org
      Type: task | Status: new
  Priority: normal | Component: SAC
Resolution: | Keywords: geodata, livedvd, torrents
---------------------+------------------------------------------------------
Comment (by wildintellect):

Did some more research on the option, short story rtorrent looks harder to
use.
Basically now just need SAC to decide which and best install approach.

Transmission[[BR]]
Pro
  * Can create torrents
  * Web interface can restrict requests by IP address, default is 127.0.0.1
  * CLI takes args and displays results, does not run by default but will
stay running till stopped
  * Better Docs
Con
  * Requires RPC daemon for watch folder feature, this enables it's web
interface, so need to make sure to firewall the port.
  * CLI and Daemon are 2 separate apps
  * Needs a startup script written if you want it auto on boot up
  * [https://admin.fedoraproject.org/pkgdb/packages/name/transmission Maybe
theres a package for FC4] Not sure what version

rTorrent[[BR]]
Pro
  * Inlcudes a Tracker (If we want one)
  * Can do Superseeding (Jump start torrents quickly)
Con
  * NCurses interface, not clear how to exit and keep torrents running
  * Version with Super Seeding not available as
[http://dries.ulyssis.org/apt/packages/rtorrent/info.html rpm]

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/458#comment:4&gt;
OSGeo <http://www.osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

6

#458: revive geo-torrents?
---------------------------+------------------------------------------------
Reporter: hamish | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Keywords: geodata, livedvd, torrents
---------------------------+------------------------------------------------

Comment(by hamish):

Perhaps the Ibiblio people would be interested in hosting a Geo Channel on
their new torrent site.

Hamish

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/458#comment:5&gt;
OSGeo <http://www.osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.