Add support for using Invidious through a HTTP Proxy

[syeopite]

Partially addresses #301

To use just set at least the host and port configuration here in the Invidious configs.

##
## Configuration for using a HTTP proxy
##
## If unset, then no HTTP proxy will be used.
## 
http_proxy:
  user:
  password:
  host:
  port:

[syeopite]

Now that there's "proper" support for HTTP proxies, I suppose Invidious should also be removing its own proxy implementation at HTTPClient and HTTPProxy

[SamantazFox]

Won't that collide with the code in src/invidious/yt_backend/proxy.cr?
It might be a good idea to remove all of that proxy code (the IP list hasn't been updated in ages anyway) and let the instance owner use their own.

[ms-jpq]

I think it might be helpful to add the http proxy for pulling in reddit comments as well, provided it s not alot more work on top of this.

Reddit just recently IP banned alot of VPS IP ranges, so currently the easiest work around is to tunnel things either via TOR or cloudflare warp

[syeopite]

Reddit comments should already be fetched through the proxy in this PR

[mk-pmb]

Is this patch meant to also address the install step that says

Checking player dependencies, this may take more than 20 minutes... If it is stuck, check your internet connection.

?

Edit: It might. At least this time the error is about npm, and I can probably figure out how to configure npm's proxy settings inside.
Edit 2: Seems we don't use proper npm but instead download directly in fetch-player-dependencies.cr line 86.
Edit 3: With exotic network setups we shoud just use proper npm for downloading: #4361

[syeopite]

The part that fetches the videojs dependencies is technically not a part of the main Invidious program so requests in that don't go through the configured proxy.

[unixfox]

The part that fetches the videojs dependencies is technically not a part of the main Invidious program so requests in that don't go through the configured proxy.

Maybe we should? If one configure a proxy he expect all the requests to go through the proxy.

[unixfox]

Please use the edit button, don't triple post...

Please don't go offtopic, create a separate issue for your npm discussion.

[mk-pmb]

I researched a bit more about how to best pass proxy settings to the build process and found that doing it securely (not revealing proxy hostname and credentials in the built image) is more complicated than I had hoped. So build-time proxy stuff should best be left for a 3rd-party project.

@syeopite, I found a few more things that I'd prefer switched around in git history, and some whitespace that seems accidential. Please allow me to propose this alternative history for the same effect as your current PR. The last commit is just so you can easily verify that files in both branches are otherwise identical. If you like, feel free to hard-reset your branch to my proposal's 2nd commit.

Does Crystal have an elegant way to use a common function to de-duplicate the conn.proxy = … assignments?
A project-wide function to set default config for all HTTP(S) connections might also turn out to be useful if one day we want to configure more aspects than just proxy.

[SamantazFox]

Could you please rebase your changes?

[syeopite]

It seems like the upstream library no longer works on Crystal 1.12

[ChunkyProgrammer]

@syeopite https://github.com/mamantoha/http_proxy/blob/master/CHANGELOG.md#0102 I think you might be able to update the library to a newer version for the crystal 1.12 support

[syeopite]

Oops! I didn't see the newest update 😅

[Sommerwiesel]

Hello, I just tested this PR on my (was public, now blocked by YT) instance to check whether I can use it to bypass the IP-blockage.

I found out that it works for the most part but there are problems with the amount of connections to the HTTP proxy server (used a simple privoxy) on a instance with many users, especially the pubsub feeds update caused problems and hasn't (yet) fully worked using this PR. I had to set privoxy to accept a ridiculously high amount of connections to even get the videos to load...
I also tested setting up proxychain and it was much more reliable. I think what's causing my issues is this: mamantoha/http_proxy#31

I think we should wait until this implementation can correctly use a connection pooler.

[mk-pmb]

I'd say let's get this rolling for people whose proxy can accept insane amounts of connections, and upgrade to the pooling solution as soon as it becomes available.
In cases where the proxy is weak (or reasonably limited) but it's ok to refuse excessive connections, tcpfwd-maxconc may help. I use that to tame npm.

[Sommerwiesel]

I was able to mitigate my problems by using some more proxies and putting a HAProxy as a load balancer and connection pooler in between. DASH is still somewhat unreliable but everything else works perfectly.

[unixfox]

I vote on merging this very awaited PR. Since the YouTube stream management will soon be dedicated to a separate app, the issue described in #4270 (comment) won't be an issue anymore.

[mooleshacat]

I vote on merging this very awaited PR. Since the YouTube stream management will soon be dedicated to a separate app, the issue described in #4270 (comment) won't be an issue anymore.

Please do. I have 4 patched instances that each have their own proton-privoxy. Playback works & stable for me (incl. dash) as long as you have sig_helper and some method to update po_tokens. No more problems for me 👍 Merge to master means less patching to get proxy support 👍

[SamantazFox]

@Sommerwiesel @mooleshacat thanks for your feedback :)

[SamantazFox]

Thank you for this important contribution!