[OSGeo] #3098: Downloading videos from PeerTube fails

OSGeoTrac · January 6, 2024, 5:32pm

#3098: Downloading videos from PeerTube fails
-------------------------------+-----------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: SysAdmin/Peertube | Keywords:
-------------------------------+-----------------------
{{{
$ yt-dlp pr-2521-simplify-dijkstra-code - OSGeo
[generic] Extracting URL: pr-2521-simplify-dijkstra-code - OSGeo
[generic] qC7HNRU783f6aqpysigF87: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] qC7HNRU783f6aqpysigF87: Extracting information
[PeerTube] Extracting URL: peertube:video.osgeo.org:qC7HNRU783f6aqpysigF87
[PeerTube] qC7HNRU783f6aqpysigF87: Downloading video JSON
[info] qC7HNRU783f6aqpysigF87: Downloading 1 format(s): 720p
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 14.7% of 604.68MiB at 472.32KiB/s ETA 18:37[download] Got
error: Downloaded 93453056 bytes, expected 634057781 bytes. Retrying
(1/10)...
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 18.8% of 604.68MiB at 471.13KiB/s ETA 17:47[download] Got
error: Downloaded 119154544 bytes, expected 634057781 bytes. Retrying
(2/10)...
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 51.1% of 604.68MiB at 581.76KiB/s ETA 08:40^C
ERROR: Interrupted by user
}}}

(it does say 581 KB/s, but it's practically 0)
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/3098>
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 2:13am

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Changes (by robe):

* milestone: Unplanned => Sysadmin Contract 2024-I

Old description:

{{{
$ yt-dlp pr-2521-simplify-dijkstra-code - OSGeo
[generic] Extracting URL:
pr-2521-simplify-dijkstra-code - OSGeo
[generic] qC7HNRU783f6aqpysigF87: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] qC7HNRU783f6aqpysigF87: Extracting information
[PeerTube] Extracting URL:
peertube:video.osgeo.org:qC7HNRU783f6aqpysigF87
[PeerTube] qC7HNRU783f6aqpysigF87: Downloading video JSON
[info] qC7HNRU783f6aqpysigF87: Downloading 1 format(s): 720p
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 14.7% of 604.68MiB at 472.32KiB/s ETA 18:37[download] Got
error: Downloaded 93453056 bytes, expected 634057781 bytes. Retrying
(1/10)...
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 18.8% of 604.68MiB at 471.13KiB/s ETA 17:47[download] Got
error: Downloaded 119154544 bytes, expected 634057781 bytes. Retrying
(2/10)...
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 51.1% of 604.68MiB at 581.76KiB/s ETA 08:40^C
ERROR: Interrupted by user
}}}

(it does say 581 KB/s, but it's practically 0)

New description:

{{{
$ yt-dlp pr-2521-simplify-dijkstra-code - OSGeo
[generic] Extracting URL: pr-2521-simplify-dijkstra-code - OSGeo
[generic] qC7HNRU783f6aqpysigF87: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] qC7HNRU783f6aqpysigF87: Extracting information
[PeerTube] Extracting URL: peertube:video.osgeo.org:qC7HNRU783f6aqpysigF87
[PeerTube] qC7HNRU783f6aqpysigF87: Downloading video JSON
[info] qC7HNRU783f6aqpysigF87: Downloading 1 format(s): 720p
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 14.7% of 604.68MiB at 472.32KiB/s ETA 18:37[download] Got
error: Downloaded 93453056 bytes, expected 634057781 bytes. Retrying
(1/10)...
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 18.8% of 604.68MiB at 471.13KiB/s ETA 17:47[download] Got
error: Downloaded 119154544 bytes, expected 634057781 bytes. Retrying
(2/10)...
[download] Destination: pr-2521-simplify-dijkstra-code
[qC7HNRU783f6aqpysigF87].mp4
[download] 51.1% of 604.68MiB at 581.76KiB/s ETA 08:40^C
ERROR: Interrupted by user
}}}

(it does say 581 KB/s, but it's practically 0)

--
Comment:

When I'm watching online the speed seems fine, no abruptness.

When I use yt-dlp, I'm getting 5MiB/s to 7MiB/s and it took 01:59 total to
dnwoload the 604.68Mib file.

I'm wondering if maybe it's an issue with our network provider. I'm
testing from Comcast Boston, so connectivity might be better there.

Do you have similar issue when downloading files from download.osgeo.org?
Like try https://download.osgeo.org/proj/proj-data-1.16.zip

This file downloads at 14.19MiB/s and completes for me in takes 00:51 s to
donwload the 694.72MiB file.

Download I would expect to be faster cause it has bonded nics so should
have twice the thruput of osgeo9
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 8:29am

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Comment (by lnicola):

> Do you have similar issue when downloading files from
download.osgeo.org?

In the first 2-3 minutes it goes slowly to about 2.3 MB/s, then hovers
around that. Afterwards, it gets less stable and drops to < 200 KB/s, then
hovers around 1.5 MB/s. Average is 1605 KB/s for the entire file.

With yt-dlp it's a little different, I get a minute or two at 2-3 MB/s,
then it drops sharply to ~0.
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 8:38am

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Comment (by lnicola):

From a Nuremberg Hetzner VM: `proj-data-1.16.zip` averages out at 16.7
MB/s.

yt-dlp starts at a bit over 7 MB/s and doesn't fail, but still slows down
after a while. The good part is that the download completes (100% of
668.73MiB in 02:14, roughly 5 MB/s).

So part of this must be a peering issue.
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 3:35pm

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Comment (by robe):

Okay so still sounds like our network backbone, probably worth reporting
to OSUOSL as we've had similar issues in the past. So when you are getting
horrible speed which network provider are you on?
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 3:38pm

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Comment (by robe):

Replying to [comment:2 lnicola]:
> > Do you have similar issue when downloading files from
download.osgeo.org?
>
> In the first 2-3 minutes it goes slowly to about 2.3 MB/s, then hovers
around that. Afterwards, it gets less stable and drops to < 200 KB/s, then
hovers around 1.5 MB/s. Average is 1431 KB/s for the entire file.
>
> With yt-dlp it's a little different, I get a minute or two at 2-3 MB/s,
then it drops sharply to ~0.
>

I think nginx has some default throttling, not sure about our backbone and
some nginx configs on domains I might have tweaked, but the peertube one I
jsut took whatever they suggested is the best nginx config as I recal.
But I think the nginx one is IP based, so we should be experiencing the
same behavior unless you've been doing nothing but downloading stuff or
you are on some sort of proxy that's been hogging resources.
> Is there any throttling involved?
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 3:43pm

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Comment (by lnicola):

I'm on my home network. There's no proxy involved on my side and I don't
think nginx throttles by default (it does buffer the responses, but that
can't be the case here).
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 7, 2024, 6:56pm

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution:
Keywords: |
-------------------------------+---------------------------------------
Comment (by robe):

Okay so it's a network issue between network providers most likely. We'll
have to report to osuosl. I forget what network checks they have us do to
confirm where the issue is.
--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

OSGeoTrac · January 18, 2025, 8:46pm

#3098: Downloading videos from PeerTube fails
-------------------------------+---------------------------------------
Reporter: lnicola | Owner: robe
Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Peertube | Resolution: worksforme
Keywords: |
-------------------------------+---------------------------------------
Changes (by robe):

* resolution: => worksforme
* status: new => closed

--
Ticket URL: <#3098 (Downloading videos from PeerTube fails) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.